LinkedIn Kafka with Nacho Solis (Repeat)
Play • 57 min

Originally published October 18, 2019

Apache Kafka was created at LinkedIn. Kafka was open sourced in 2011, when the company was eight years old. By that time, LinkedIn had developed a social network with millions of users. LinkedIn’s engineering team was building a range of externally facing products and internal tools, and many of these tools required a high-throughput system for publishing data and subscribing to topics.

Kafka was born out of this need. Over time, Kafka’s importance within LinkedIn has only grown. Kafka plays a central role for services, log management, data engineering, and compliance. LinkedIn might be the biggest user of Apache Kafka in the entire software industry. Kafka has many use cases, and it is likely that they are almost all on display within LinkedIn.

Nacho Solis is a senior software engineering manager at LinkedIn, where he helps teams build infrastructure for Kafka, as well as Kafka itself. Nacho joins the show to discuss the history of Kafka at LinkedIn, and the challenges of managing such a large deployment of Kafka. We also talk about streaming, data infrastructure, and more general problems in the world of engineering management.

Full disclosure: LinkedIn is a sponsor of Software Engineering Daily.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post LinkedIn Kafka with Nacho Solis (Repeat) appeared first on Software Engineering Daily.

The Cloudcast
The Cloudcast
Cloudcast Media
Evolution of Commercial OSS
Joseph “JJ” Jacks (@asynchio, Founder/General Partner OSS Capital) talks about how Commercial OSS has evolved, coopetition with cloud providers, and what's next for Commercial OSS business models and communities.  *SHOW: *492 *SHOW SPONSOR LINKS:* * CloudZero - Cloud Cost Intelligence for Engineering Teams * BMC Wants to Know if your business is on its A-Game * BMC Autonomous Digital Enterprise * Datadog Security Monitoring Homepage - Modern Monitoring and Analytics * Try Datadog yourself by starting a free, 14-day trial today. Listeners of this podcast will also receive a free Datadog T-shirt. *CLOUD NEWS OF THE WEEK *- http://bit.ly/cloudcast-cnotw *CHECK OUT OUR NEW PODCAST - **"CLOUDCAST BASICS"* *SHOW NOTES:* * OSS Capital Partners and Advisors * Commercial Open-Source Software Company Index (COSSI) * OSS Capital to launch an ETF (with NASDAQ) of OSS Companies in Summer 2021 * Open Consensus - Data Driven Perspectives on Open Source Software * COSS Community / Open Core Summit  * The Kubernetes State of the Community (Eps.272) * Exploring the Business Side of Open Source Software (Eps.358) * Server Side Public License *Topic 1 *- Welcome to the show. For those that don’t already know you, tell us a little bit about your background, and some of the things you’re focused on today.  *Topic 2* - You’ve been tracking the commercialization of open-source projects for quite a while now. What big trends have you seen evolve over the last two decades (from Red Hat to MongoDB)  *Topic 3 *- Even in the face of new OSS-centric offerings from the cloud providers, we still continue to see companies getting funded. What is the sentiment in the VC-communities about what the new competitive landscape looks like? Are there new rules in the game?* * *Topic 4 *- We’ve recently seen MongoDB and Elastic changing their licensing model to SSPL. The stock of both companies continues to rise. Is what they are doing a short-term “fix” to a competitive threat, or a critical mistake? Does licensing need to evolve as a company matures?  *Topic 5* - Are there fundamental shifts in how OSS companies are created and eventually operationalized happening now?  *Topic 6* - Where do you see commercial OSS trending over the next 5 years, and what big changes need to happen to make those realities happen? *FEEDBACK?* * Email: show at thecloudcast dot net * Twitter: @thecloudcastnet
47 min
Software Daily
Software Daily
SoftwareDaily.com
Multi-Prem Software Delivery and Management with Grant Miller
Modern SaaS products are increasingly delivered via the cloud, rather than as downloadable, executable programs. However, many potential users of those SaaS products may need that software deployed on-prem, in a private network. Organizations have a variety of reasons for preferring on-prem software, such as security, integration with private tools, and compliance with regulations. The cost of setting up a bespoke on-prem version of a SaaS offering was often prohibitive for both the vendor and potential users. Replicated leverages the portability of containers to help SaaS vendors ship an on-prem or multi-prem version of their software. Replicated gives SaaS vendors a suite of ready-made components to help install and manage an instance of their software on-prem. Replicated has seen explosive growth in its six-year lifespan: today, 50 of the Fortune 100 companies manage apps with Replicated. Grant Miller is the Founder and CEO of Replicated. Replicated recently closed a Series B fundraising round to help scale their work in the cloud-native space, including the launch of their Kubernetes-Off-the-Shelf platform. He joins the show today to talk about the next generation of tools on the Replicated platform, how Kubernetes is changing enterprise IT, and why the on-prem software market isn’t going away anytime soon. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices
54 min
Google Cloud Platform Podcast
Google Cloud Platform Podcast
Google Cloud Platform
Cloud Spanner Revisited with Dilraj Kaur and Christoph Bussler
Mark Mirchandani and Stephanie Wong are back this week as we learn about all the new things happening with Google Cloud Spanner. Our guests this week, Dilraj Kaur and Christoph Bussler, describe Cloud Spanner as a fully managed relational database that boasts unlimited scaling and advanced consistency and availability. Unlimited scaling truly means unlimited, and Chris explains why Cloud Spanner offers this feature and how it’s making database design and development easier. Dilraj and Chris tell us all about the cool new features Spanner has developed, like generated columns and foreign keys, and how customer needs influenced these developments. Chris walks us through the process of using some of these new features, including how developers can monitor their database systems. Managed backups and multi-region configuration are additional recent additions to Cloud Spanner, and our guests explain how these are used by current enterprise clients. Dilraj and Chris explain the automatically managed features of Spanner versus the customer managed features and how people set up and manage database projects. We hear examples of companies using Cloud Spanner and how it has improved their businesses. Dilraj Kaur Dilraj Kaur is an Enterprise Customer Engineer with specialization in Data Management. She has been with Google for about 2.5 years and is based in Atlanta. Christoph Bussler As a Solutions Architect Chris is focusing on databases, data migration and data integration in enterprise customer settings. See his professional work and background on his website. Cool things of the week * New to Google Cloud? Here are a few free trainings to help you get started blog * Start your skills challenge today site * Service Directory is generally available: Simplify your service inventory blog Interview * Google Cloud Spanner site * GCP Podcast Episode 62: Cloud Spanner with Deepti Srivastava podcast * Using the Cloud Spanner Emulator docs * Cloud Spanner Ecosystem site * Cloud Spanner Qwiklabs site * Google Cloud Platform Community On Slack site * Creating and managing generated columns docs * WITH Clause docs * Foreign Keys docs * Numeric Data Type docs * Information schema docs * Overview of introspection tools docs * Backup and Restore docs * Multi-region configurations docs * ShareChat: Building a scalable data-driven social network for non-English speakers globally site * Blockchain.com: Streamlining infrastructure for the world’s most dynamic financial market site * What is Cloud Spanner? video What’s something cool you’re working on? Mark has been working on budgeting blog posts, including Protect your Google Cloud spending with budgets. Stephanie is working on her data center animation series
41 min
Streaming Audio: A Confluent podcast about Apache Kafka
Streaming Audio: A Confluent podcast about Apache Kafka
Confluent, original creators of Apache Kafka®
Becoming Data Driven with Apache Kafka and Stream Processing ft. Daniel Jagielski
When it comes to adopting event-driven architectures, a couple of key considerations often arise: the way that an asynchronous core interacts with external synchronous systems and the question of “how do I refactor my monolith into services?” Daniel Jagielski, a consultant working as a tech lead/dev manager at VirtusLab for Tesco, recounts how these very themes emerged in his work with European clients.  Through observing organizations as they pivot toward becoming real time and event driven, Daniel identifies the benefits of using Apache Kafka® and stream processing for auditing, integration, pub/sub, and event streaming. He describes the differences between a provisioned cluster vs. managed cluster and the importance of this within the Kafka ecosystem. Daniel also dives into the risk detection platform used by Tesco, which he helped build as a VirtusLab consultant and that marries the asynchronous and synchronous worlds. As Tesco migrated from a legacy platform to event streaming, determining risk and anomaly detection patterns have become more important than ever. They need the flexibility to adjust due to changing usage patterns with COVID-19. In this episode, Daniel talks integrations with third parties, push-based actions, and materialized views/projects for APIs. Daniel is a tech lead/dev manager, but he’s also an individual contributor for the Apollo project (an ICE organization) focused on online music usage processing. This means working with data in motion; breaking the monolith (starting with a proof of concept); ETL migration to stream processing, and ingestion via multiple processes that run in parallel with record-level processing. EPISODE LINKS * Building an Apache Kafka Center of Excellence Within Your Organization ft. Neil Buesing  * Risk Management in Retail with Stream Processing * Event Sourcing, Stream Processing and Serverless * It’s Time for Streaming to Have a Maturity Model ft. Nick Dearden * Read Daniel Jagielski's articles on the Confluent blog * Join the Confluent Community * Learn more with Kafka tutorials, resources, and guides at Confluent Developer * Live demo: Kafka streaming in 10 minutes on Confluent Cloud * Use *60PDCAST* to get an additional $60 of free Confluent Cloud usage (details)
48 min
Level-up Engineering
Level-up Engineering
Coding Sans
Leadership Soft Skills: Master Your Own Mind to Lead Your Team to Success
https://codingsans.typeform.com/to/L5OvgflD (Click here to take the State of Software Development 2021 survey!) Interview with Gergely Hodicska "Felhő", VP of Engineering at Bitrise. We take a deep dive with him into leadership soft skills and his story about finding his weaknesses and turning them into superpowers. We cover common weaknesses, methods to improve yourself and others around you, and more. In this interview we're covering: Importance of soft skills in engineering leadership Essential leadership soft skills Story about building up leadership soft skills Practices to improve leadership soft skills Common challenges for engineering managers Helping direct reports improve their leadership skills Integrating leadership soft skills into your engineering culture Excerpt from the interview: "I had a heated argument with a friend of mine at Ustream, and he told me, “This is why I don’t like working with you.” At first, I thought, I’m just trying to find the best solution, but really I was defending my ego in that conversation. Moments like that made me realize that I should improve, and research why I react the way I do in certain situations. That’s when I started to learn about my psychological drivers. The first step I took was learning about personality types. This didn’t give me a lot of answers, but it helped me understand why others are stupid..." https://codingsans.com/blog/leadership-soft-skills (Click here to read the full interview!)
1 hr 6 min
AWS TechChat
AWS TechChat
Shane Baldacchino
Episode 81 - re:Invent 2020 - AI/ML Special
In this episode of AWS TechChat, we close out our four parts of AWS re:Invent 2020 series with an AI/ML special. We cover Amazon Sagemaker, Amazon Kendra, Amazon Elastic MapReduce (EMR), Amazon QuickSight, and some brand new services. We talk about AWS HealthLake and how it makes sense of health data. AWS customers can use Kendra’s Google Drive connector to ingest and manage content from Google Docs and Google Slides. We introduce AWS Panorama which will help improve your operations with computer vision at the edge. We continue with a raft of new Amazon SageMaker updates: • Amazon SageMaker Feature Store - A fully managed repository for machine learning features • Amazon SageMaker Clarify - Bias Detection and Explainability • Amazon SageMaker Debugger - Optimize ML models with real-time monitoring of training metrics and system resources • Amazon SageMaker Model Monitor - Detect drift in model quality, model bias, and feature importance • Amazon SageMaker Pipelines - First purpose-built CI/CD service for machine learning • Amazon SageMaker Jumpstart - Simplifies Access to Pre-built Models and Machine Learning Solutions Before wrapping out, we share two more AI/ML updates - Amazon EMR Studio is the integrated development environment (IDE) for applications written in R, Python, Scala, PySpark, and Jupyter notebooks now gives you the option to deploy on Amazon Elastic Kubernetes Service (EKS). Amazon QuickSight allows you to ask Natural Language Query (NLQ) about your data and get answers in seconds. Speakers: Shane Baldacchino - Edge Specialist Solutions Architect, ANZ, AWS Shai Perednik - Solutions Architect, AWS Pallavi Nargund - Solutions Architect, AWS
1 hr 4 min
Software Defined Talk
Software Defined Talk
Software Defined Talk LLC
Episode 286: Press the turbo button on that one
This week we discuss the demise of the blameless post mortem, a $500 Million mistake and some forgiveness for Red Hat. Plus, a live update on the Texas Winter Apocalypse. Rundown Citi Can’t Have Its $900 Million Back (https://www.bloomberg.com/opinion/articles/2021-02-17/citi-can-t-have-its-900-million-back) Brian Armstrong on the Crypto Economy (Ep. 115) (https://conversationswithtyler.com/episodes/brian-armstrong/) Operating Systems CentOS Stream: Why it’s awesome (https://jaymzh.medium.com/centos-stream-why-its-awesome-5c45d944fb22) The world’s second-most popular desktop operating system isn’t macOS anymore (https://arstechnica.com/gadgets/2021/02/the-worlds-second-most-popular-desktop-operating-system-isnt-macos-anymore/) Relevant to your interests ‘Millions’ of Ford cars to be powered by Android in major Google deal (https://www.siliconrepublic.com/machines/ford-google-connected-cars-cloud) Miami Pushes Crypto With Proposal to Pay Workers in Bitcoin (https://www.bloomberg.com/news/articles/2021-02-11/miami-mayor-pushes-crypto-with-offer-to-pay-workers-in-bitcoin) Online workspace startup Notion hit by outage, citing DNS issues – TechCrunch (https://techcrunch.com/2021/02/12/notion-outage-dns-domain-issues/) Penpot | Design Freedom for Teams (https://penpot.app/) Taiga: Your opensource agile project management software (https://www.taiga.io/) Facebook Meets Apple in Clash of the Tech Titans—‘We Need to Inflict Pain’ (https://www.wsj.com/articles/facebook-meets-apple-in-clash-of-the-tech-titanswe-need-to-inflict-pain-11613192406) CEOs of Reddit and Robinhood and ‘Roaring Kitty’ slated to testify in GameStop hearing (https://www.theverge.com/2021/2/13/22281698/ceo-reddit-robinhood-roaring-kitty-testify-gamestop-hearing-congress-stocks) Building a tool to measure real-time behavior of Wikipedia users (https://medium.com/apache-pinot-developer-blog/analyzing-wikipedia-in-real-time-with-apache-kafka-and-pinot-4b4e5e36936b) Excel Is The World’s Most Used “Database” (http://jasonlbaptiste.com/startups/microsoft-excel-is-the-worlds-most-used-database/) Code With Me Beta: Support for Audio and Video Calls (https://blog.jetbrains.com/blog/2021/02/16/code-with-me-beta-support-for-audio-and-video-calls/) The four reasons AWS succeeded, according to Andy Jassy (https://twitter.com/pmddomingos/status/1361789872432771073?s=20) The Mars Relay Network Connects Us to NASA’s Martian Explorers (https://www.jpl.nasa.gov/news/the-mars-relay-network-connects-us-to-nasas-martian-explorers) Elon Musk's SpaceX raised $850 million, jumping valuation to about $74 billion (https://www.cnbc.com/2021/02/16/elon-musks-spacex-raised-850-million-at-419point99-a-share.html) This Cloud Computing Billing Expert Is Very Funny. Seriously. (https://www.nytimes.com/2021/02/17/technology/corey-quinn-amazon-aws.html?referringSource=articleShare) Changes to Sharing and Viewing News on Facebook in Australia - About Facebook (https://about.fb.com/news/2021/02/changes-to-sharing-and-viewing-news-on-facebook-in-australia/?utm_source=newsletter&utm_medium=email&utm_campaign=newsletter_axioslogin&stream=top) Security Logging startups are suddenly hot as CrowdStrike nabs Humio for $400M (https://techcrunch.com/2021/02/18/logging-startups-are-suddenly-hot-as-crowdstrike-nabs-humio-for-400m/) Datadog bolsters app security and observability data with Sqreen and Timber acquisitions (https://venturebeat.com/2021/02/12/datadog-bolsters-app-security-and-observability-data-management-with-sqreen-and-timber-acquisitions/) The Long Hack: How China Exploited a U.S. Tech Supplier (https://www.bloomberg.com/features/2021-supermicro). Passwords LastPass Free Accounts Will Now Work on Either Your Phone or Computer, Not Both (https://www.vice.com/en/article/pkd88v/lastpass-free-accounts-will-now-work-on-either-your-phone-or-computer-not-both?utm_source=newsletter&utm_medium=email&utm_campaign=newsletter_axioscodebook&stream=technology) Apple releases Chrome extension for iCloud passwords (https://www.theverge.com/2021/1/31/22259720/apple-icloud-passwords-chrome-browser-extension-released) Hardware highlights…? Backblaze Hard Drive Stats for 2020 (https://www.backblaze.com/blog/backblaze-hard-drive-stats-for-2020/). Microsoft, Google, and Qualcomm are reportedly nervous about Nvidia acquiring Arm (https://www.theverge.com/2021/2/12/22280262/qualcomm-microsoft-google-nvidia-arm-acquisition-investigations-concerns) Audio is the future? Clubhouse’s Inevitability (https://stratechery.com/2021/clubhouses-inevitability/) The new media mogul: Andreessen Horowi (https://www.axios.com/the-new-media-mogul-andreessen-horowitz-969145da-43f0-4153-8da2-0a35f2f21632.html) Nonsense 90-year-old man spends $10,000 on Wall Street Journal ads to shame AT&T (https://nypost.com/2021/02/12/man-90-spends-10k-on-wall-street-journal-ads-to-shame-att/) Elon Musk predicts Austin, Texas, will be 'the biggest boomtown that America has seen in 50 years' (https://www.businessinsider.com/elon-musk-austin-joe-rogan-biggest-boom-town-50-years-2021-2) Sponsors strongDM — Manage and audit remote access to infrastructure. Start your free 14-day trial today at: strongdm.com/SDT (http://strongdm.com/SDT) Listener Feedback Andy wants you to work at BookingLive as DevOps Engineer (https://bookinglive.zohorecruit.com/recruit/PortalDetail.na?digest=iHU1EAOPeO@465g4gK.nYDgjwjkyaz8ZkMQbQdbLaAs-&iframe=true&jobid=297951000002327006&widgetid=297951000000072311&embedsource=CareerSite) (UK based) Conferences DevOpsDay Texas on March 2nd. (https://devopsdays.org/events/2021-texas/welcome/) SpringOne.io (https://springone.io), Sep 1st to 2nd - CFP is open until April 9th (https://springone.io/cfp). Two SpringOne Tours: (1.) developer-bonanza in for NA, March 10th and 11th (https://tanzu.vmware.com/developer/tv/springone-tour/0014/), and, (2.) EMEA dev-fest on April 28th (https://tanzu.vmware.com/developer/tv/springone-tour/0015/). SDT news & hype Join us in Slack (http://www.softwaredefinedtalk.com/slack). Send your postal address to stickers@softwaredefinedtalk.com (mailto:stickers@softwaredefinedtalk.com) and we will send you free laptop stickers! Follow us on Twitch (https://www.twitch.tv/sdtpodcast), Twitter (https://twitter.com/softwaredeftalk), Instagram (https://www.instagram.com/softwaredefinedtalk/) and LinkedIn (https://www.linkedin.com/company/software-defined-talk/). Brandon built the Quick Concall iPhone App (https://itunes.apple.com/us/app/quick-concall/id1399948033?mt=8) and he wants you to buy it for $0.99. Use the code SDT to get $20 off Coté’s book, (https://leanpub.com/digitalwtf/c/sdt) Digital WTF (https://leanpub.com/digitalwtf/c/sdt), so $5 total. Become a sponsor of Software Defined Talk (https://www.softwaredefinedtalk.com/ads)! Recommendations Matt: HDMI LCD controllers (https://www.aliexpress.com/item/1005001623726553.html?spm=a2g0s.12269583.0.0.5aad98bbnSzTR4). Brandon: Fake Famous (https://www.hbo.com/documentaries/fake-famous). Coté: Susan Sontag’s first book, Against Interpretation (https://www.goodreads.com/book/show/52374.Against_Interpretation_and_Other_Essays).
1 hr 5 min
Syntax - Tasty Web Development Treats
Syntax - Tasty Web Development Treats
Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers
React Query + More React with Tanner Linsley
In this episode of Syntax, Scott and Wes talk with Tanner Linsley about React Query, how it works, why you might want to use it, and more! Deque - Sponsor Deque’s free axe browser extension helps developers instantly catch 50% of accessibility bugs while they code. It’s lightweight, easy-to-use, and has zero false positives. Get started for free at deque.com/axe. Sentry - Sponsor If you want to know what’s happening with your errors, track them with Sentry. Sentry is open-source error tracking that helps developers monitor and fix crashes in real time. Cut your time on error resolution from five hours to five minutes. It works with any language and integrates with dozens of other services. Syntax listeners can get two months for free by visiting Sentry.io and using the coupon code “tastytreat”. Mux - Sponsor Mux Video is an API-first platform that makes it easy for any developer to build beautiful video. Powered by data and designed by video experts, your video will work perfectly on every device, every time. Mux Video handles storage, encoding, and delivery so you can focus on building your product. Live streaming is just as easy and Mux will scale with you as you grow, whether you’re serving a few dozen streams or a few million. Visit mux.com/syntax. Guests * Tanner Linsley Show Notes 01:56 - What do you do? * nozzle.io * React Query 06:04 - What is React Query? 24:19 - How does React Query use dev tools? 31:20 - What about React Context? 36:59 - Server-rendered components? 42:40 - Thoughts on static sites? 50:38 - What is the stack? Links * Redux * Axios * Syntax 206: State Machines, CSS and Animations with David K Piano * RTK Query * Relay * Remix * Twin.macro * Tailwind CSS * React Static * Next * TanStack ××× SIIIIICK ××× PIIIICKS ××× * Tanner: React Query - Essentials Course * Scott: Govee TV LED Backlights * Wes: 1Password Shameless Plugs * Tanner: Nozzle.io * Scott: Testing with Cypress - Sign up for the year and save 25%! * Wes: All Courses - Use the coupon code ‘Syntax’ for $10 off! Tweet us your tasty treats! * Scott’s Instagram * LevelUpTutorials Instagram * Wes’ Instagram * Wes’ Twitter * Wes’ Facebook * Scott’s Twitter * Make sure to include @SyntaxFM in your tweets
1 hr 5 min
More episodes
Search
Clear search
Close search
Google apps
Main menu