Software Misadventures
Ronak Nathani, Guang Yang
Subscribe
A show about not just the technologies, but the people and stories behind them. In every episode, Ronak and Guang sit down with engineers, founders, and investors to chat about their paths, lessons they’ve learned and of course, the misadventures along the way.
Available episodes
Newest first
Oldest first
6 days ago
Nathan Marz - On changing the economics of building large-scale software with Rama - #23
What does it mean to change the economics of software development? Nathan Marz joins the show to share how they reduced the cost of building Mastodon at Twitter-scale by 100X and the 10 years journey to build Rama, a new programming platform that made this feat possible.
Nathan is the founder of Red Planet Labs. Prior to RPL, he led engineering for BackType which was acquired by Twitter in 2011. Nathan created the Apache Storm project and wrote the book Big Data: Principles and best practices of scalable realtime data systems. Outside of working, Nathan is a private pilot, loves going to stand-up comedy shows, and is forever trying to teach his dog new tricks.
Show Notes:
* Nathan’s Twitter: https://twitter.com/nathanmarz
* What is Rama? https://redplanetlabs.com/learn-rama
* Reducing the cost of building Mastodon at Twitter-scale by 100X: https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/
Stay in touch:
✉️ Subscribe to our newsletter: https://softwaremisadventures.substack.com
👋 Send feedback or say hi: softwaremisadventures@gmail.com
Segments:
[0:00] flying
[0:07] inefficiencies of backend software development
[0:17] suffering oriented programming
[0:23] AI programming?
[0:25] RAMA’s programming model
[0:33] deployment & monitoring with RAMA
[0:36] building a twitter clone at scale with RAMA
[0:43] migrations with RAMA
[0:54] driving adoption for RAMA
[1:01] fundraising
[1:15] building a fully remote team
playlist_add
Aug 3, 2023
Kelsey Hightower - On retiring as Distinguished Engineer from Google at 42 (Part 2)
Kelsey Hightower was a Distinguished Engineer at Google, where he worked on Google Cloud Platform. In this second part of the conversation, we focus on Kelsey’s retirement - the financial planning that enabled him to retire at 42, how he got started advising startups and his perspectives on compensation, turning down a substantial offer from Microsoft and meeting Satya Nadella in person. And, of course, plans for the future.
playlist_add
Jul 24, 2023
Kelsey Hightower - On retiring as Distinguished Engineer from Google at 42 (Part 1)
Kelsey Hightower was a Distinguished Engineer at Google, where he worked on Google Cloud Platform. In this first part of the conversation, we delve into pivotal moments in Kelsey’s career journey ranging from buying his first car by working at mcdonald’s after school, to starting his own computer store that turned into a music studio after 6pm, to hacking on python infrastructure with the core developers. Through these stories, we learned a ton about how Kelsey thinks about acquiring new skills - getting paid for it, breaking into the world of open source, navigating corporate politics, building trust within a team, and much more.
playlist_add
Jun 26, 2023
Julie Amundson - Career breaks, job search amidst hiring freezes, positioning yourself and much more - #20
Julie Amundson is a Sr Staff Software Engineer at Google working on Machine Learning Infrastructure. Prior to Google, she was the Director of Machine Learning Infrastructure at Netflix. Julie decided to take a career break last year when she was affected by mass layoffs. In this conversation, we talk to her about what it was like to find a job during hiring freezes, what it was like to position herself in this market, whether the interviewers cared about the career break she took and how the career break changed her perspective towards work and life.
playlist_add
Jun 3, 2022
Chris Pruett - On deciding to leave LinkedIn and co-founding Jam, values based decision making and compassionate leadership - #19
Chris Pruett is the CTO and Co-founder of Jam - a new way to share and listen to bite-sized audio. Prior to Jam, Chris spent 9+ years at LinkedIn growing from an engineering manager to VP of Engineering. During his tenure at LinkedIn, he worked on almost all aspects of the app and towards the end, led an org of 500+ engineers working on Feed, Messaging, Identity and Search. In this episode, we discuss how he made the decision to leave his leadership position at LinkedIn and co-found Jam. We also spoke about his time at LinkedIn and how he developed the practice to make value based decisions both in professional and personal life.
playlist_add
Mar 25, 2022
Software Misadventures Update and Plans for 2022
Short episode about reflections on the past year and plans for 2022.
playlist_add
Feb 25, 2022
Kailash Nadh - On being an absurdist and building the tech team at Zerodha, India's largest stock broker - #18
Kailash is the CTO at Zerodha, the largest stock broker in India. In this conversation, we speak with him about absurdism - a philosophy that guides his personal and professional worldview. We discuss how he built Zerodha’s tech team, their team culture and how the team operates so efficiently while being so lean. We also discuss why Zerodha self-hosts all of their tech stack, what they look for when hiring engineers and how their systems scaled when the user base grew from 2 to 8 million in 18 months.
playlist_add
Jan 14, 2022
Michael Lynch - On quitting google for indie hacking, bootstrapping to $450K+ ARR in public, writing personal retrospectives and more - #17
Michael Lynch is the founder of TinyPilot. After doing software engineering at Microsoft and Google for 7 years, Michael decided in 2018 to quit and start working for himself by building small software businesses. From years of negative profit to now building a $450K+ ARR hardware business, Michael joins the show to chat about what made him quit his cushy job at Google, how he builds in public with monthly retrospectives, what he has learned over the 3 years indie hacking and much more.
playlist_add
Nov 12, 2021
Cory Watson - Leading observability teams at Twitter & Stripe, how to succeed in a new org, effective ways to advocate for your team and more - #16
Cory is currently a Solutions Engineer at Jeli.io and very well known in the community for his work on Observability. His career in observability began at Twitter where he managed the observability team and then he joined Stripe, where he created and led the observability team, this time around as a Principal Engineer. We talk to him about how he got his start in customer support and the role it played in the later part of his career. We discuss his time at Twitter where there was a power outage in the data center on the day he joined and how once he had to stay up all night dealing with file handle leaks. We also discuss how he created and led the observability team at Stripe as an individual contributor, how one can succeed in a new org, how to navigate information asymmetry in the workplace, what are some effective ways to advocate for your team and how we all are just humans trying to get stuff done.
playlist_add
Oct 12, 2021
Ashwin Kumar - On learning new things by breaking them down, the secret to winning >$100k from hackathons, the art of storytelling, and much more - #15
Ashwin is a Startup Partnership Lead at Stripe. From web development to co-founding a YC startup, to deep learning, Ashwin has a knack for picking up new skills extremely quickly. In this episode, we chat about the methods he employed to successfully make these transitions, learnings/tips from winning 30+ hackathons in a row, and what engineers can gain from better story-telling.
playlist_add
Sep 12, 2021
Bruno Connelly - Building and leading the global SRE org at LinkedIn - #14
Bruno Connelly is a VP of Engineering at LinkedIn. He leads the Site Engineering org responsible for LinkedIn's production infrastructure. He joins the show to talk about his journey in tech - from teaching himself how to code at a young age, building, maintaining and reverse engineering software as a teenager, building ISPs in the early part of his career (there are some fun stories that involve sleeping in the data center) to leading the SRE org at LinkedIn over the last decade. He talks about the early days at LinkedIn that involved a lot of firefighting to keep the site up, how the team built technical stability and scaled the platform. We also dive into how he grew the SRE org globally and overcame challenges that came with the growth. Throughout the conversation, he shares various nuggets of wisdom - like how to stay calm under pressure and how to make people feel at ease - as he describes his leadership style, people who have influenced him and what he thinks is a positive way t…
playlist_add
Aug 14, 2021
Lorin Hochstein - On how Netflix learns from incidents, software as socio-technical systems, writing persuasively and more - #13
With 5+ years of experience building resilient systems at the Netflix scale, Lorin joins the show to chat about his favorite incident story, the path that led him to doing chaos engineering (and later away from it), and advocating for a dedicated analyst to talk to people after an incident. Throughout the conversation, Lorin shares his philosophy and tips on how to learn from incidents, what engineers can gain from writing better, and why some metrics may not be as useful as you think.
playlist_add
Jul 9, 2021
Spoons (Daniel Spoonhower) - On building Lightstep, being customer focused, developing systems at Google scale and much more - #12
Spoons is the Co-founder and Chief Architect of Lightstep. He joins the show to talk about building systems at Google scale and various aspects that make Google a weird place than other companies. We talked about Spoons's journey of leaving Google and deciding to join Lightstep as a co-founder. We dig into the challenges during the early days of Lightstep and discuss the importance of speaking to customers to build the right product. We talk about what it's like to start a family and run a startup and how one can be intentional about building a company’s culture. As always, we go through some of the misadventures and one of them involves a cable being cut under the English channel.
playlist_add
Jun 11, 2021
Emmanuel Ameisen - On production ML at Stripe scale, leading 100+ ML projects, iterating fast, and much more - #11
Having led 100+ ML projects at Insight and built ML systems at Stripe scale, Emmanuel joins the show to chat about how to build useful ML products and what happens next when the model is in production. Throughout the conversation, Manu shares stories and advice on topics like the common mistakes people make when starting a new ML project, what’s similar and different about the lifecycle of ML systems compared to traditional software, and writing a technical book.
playlist_add
May 6, 2021
Todd Underwood - On lessons from running ML systems at Google for a decade, what it takes to be a ML SRE, challenges with generalized ML platforms and much more - #10
Todd is a Sr Director of Engineering at Google where he leads Site Reliability Engineering teams for Machine Learning. Having recently presented on how ML breaks in production, by examining more than a decade of outage postmortems at Google, Todd joins the show to chat about why many ways that ML systems break in production have nothing to do with ML, what’s different about engineering reliable systems for ML, vs traditional software (and the many ways that they are similar), what he looks for when hiring ML SREs, and more.
playlist_add
Apr 23, 2021
Evan Estola - On recommendation systems going bad, hiring ML engineers, giving constructive feedback, filter bubbles and much more - #9
Evan Estola (https://twitter.com/estola) is a Director of Engineering at Flatiron Health where he's leading software engineering teams focused on building Machine Learning products. Throughout this episode, Evan shares various stories when recommendation systems didn’t work as expected, like this one time when members saw mathematically worst recommendations for meetups near them. He also shares why Schenectady, NY pops up on some lists of most popular cities and the story behind the Wall Street Journal article titled 'Orbitz steers Mac users to pricier hotels'. We also discuss skills Evan looks for when hiring ML engineers, how to give constructive feedback, filter bubbles and much more.
playlist_add
Apr 9, 2021
Uma Chingunde - On managing migrations, growing engineering teams and much more - #8
Uma is a VP of Engineering at Render. In this episode, she shared with us her insights on how to successfully manage infrastructure migrations. We discussed the importance of communicating the "why" behind a migration, identifying success metrics, creating a culture where migrations are identified as highly impactful projects and much more. Uma also shared stories where parts of a migration didn’t go as planned, how the team fixed the issue and the kind of engineers she thinks would make good tech leads for these projects.
We had a great time speaking with Uma! Our major focus in this episode was large scale infrastructure migrations and Uma shared many insights on how to manage them successfully. We discussed the importance of communicating the “why” behind a migration, identifying success metrics, creating a culture where migrations are identified as highly impactful projects and much more. Uma also shared stories where parts of a migration didn’t go as planned, how the team…
playlist_add
Mar 20, 2021
Charity Majors - On database outages, journey as a co-founder, thriving under pressure and growing as an engineer - #7
Charity Majors (https://twitter.com/mipsytipsy) is the co-founder and CTO of Honeycomb.io. Before this she worked at Facebook, Parse and Linden Lab on infrastructure and developer tools, and always seemed to wind up running the databases. She is the co-author of Database Reliability Engineering book and also has an amazing blog at charity.wtf. We love the content in her blogs and have learned a lot from them.
We had a lot of fun speaking with Charity in this lively conversation! We learned about her journey from being an engineer to co-founding Honeycomb, what it was like being on-call when she was only 17, and staying calm during production incidents. We talked about various production outages throughout the episode and our favorite involved driving to a datacenter to flip a DB switch. Charity also shares what it takes to build an awesome engineering culture, the engineer/manager pendulum, and qualities Charity looks for when hiring senior engineers.
playlist_add
Mar 7, 2021
Tammy Bryant Butow - On failure injection, chaos engineering, extreme sports and being curious - #6
Tammy Bryant Butow is a Principal SRE at Gremlin where she works on Chaos Engineering. In this episode, we discuss how her curiosity led her to the world of infrastructure engineering, an outage from her early days where a core switch took down half the datacenter, her experience running a disaster recovery test and how it taught her about the importance of injecting failures into a system to make it more resilient. We also touch on advanced failure injection techniques, how chaos engineering is evolving and how extreme sports help Tammy keep calm under pressure. Lastly, Tammy has some great advice for teams looking to get started with chaos engineering.
playlist_add
Feb 19, 2021
Oliver Leaver-Smith - On how "just a monitoring change" took down the entire site and resilience engineering - #5
Oliver Leaver-Smith, better known as Ols, is a Senior Devops Engineer at Sky Betting and Gaming. In this episode, we discuss how a seemingly simple monitoring change ended up taking down the entire site. We also talk about chaos and resilience engineering. We discuss how the team at Sky Betting and Gaming conducts fire drills (chaos engineering exercises) where they not only test the resiliency of their software systems but also their people systems. We walk through a recent example of a fire drill, how they have evolved over the past few years and the lessons learned in the process.
playlist_add
Feb 6, 2021
Ryan Underwood - On debugging the Linux kernel - #4
Ryan Underwood is a Staff SRE and tech lead on the Helix and Zookeeper SRE team at LinkedIn. Prior to LinkedIn, he was an SRE at Machine Zone and Google. Apart from his regular responsibilities, Ryan’s interest and expertise include debugging production kernel, I/O and containerization issues. His opinion about not treating software as a black box and his persistent approach to debugging complex problems are truly inspiring.
On several occasions, Ryan’s colleagues have leaned on him to solve an esoteric problem that everyone thought was insurmountable. Our main focus today is one such problem that Ryan and team ran into while upgrading machines to 4.x kernel that resulted in elevated 99th percentile latencies. We dive into what the problem was, how it was identified and how it was fixed. We discuss some of the tools and practices that are helpful in debugging system performance issues. And we also talk about Ryan’s background and how his curiosity landed him a career in Site Rel…
playlist_add
Jan 23, 2021
David Henke - On building a culture of "Site Up" at LinkedIn and Yahoo! - #3
David is LinkedIn’s former SVP of Engineering and Operations. He came out of retirement to join LinkedIn in 2009 during a time of rapid growth. After 4 years at LinkedIn, he retired in 2013. Throughout his career, David has been in multiple leadership positions and has been recognized as one of the best Operations Executives.
This was an extremely fascinating conversation. David shares insightful stories from early days at LinkedIn and what it took to develop the culture of “Site Up and Secure”. He shares one of the most severe outages he has experienced in his career - this one was at Yahoo!, which he calls the 10g massacre. We talk about David’s 3 retirements throughout his career, his advice on developing operational excellence and lessons on being an effective leader. Throughout this conversation you’ll also hear various nuggets of wisdom from David, better known as Henkeisms. Please enjoy this highly entertaining and deeply insightful conversation with David Henke.
Web…
playlist_add
Jan 6, 2021
Julia Evans - On kubernetes scheduler bugs, TCP performance regressions and debugging tips - #2
In this episode, we speak with Julia Evans. Julia runs a programming zines business, called Wizard Zines (https://wizardzines.com/), where she creates comics about various programming concepts. She has been creating zines, when she was still a software engineer at Stripe. Her zines are extremely approachable and highly educational. In addition to creating zines, Julia is a prolific blogger and has around 500 posts on her blog at jvns.ca. Her blogs are another great source to learn about fundamental programming concepts.
We had a lot of fun speaking with Julia for this episode. We discuss two bugs she came across at Stripe. We talk about how she identified and fixed a bug in Kubernetes Scheduler and how her understanding of TCP helped her fix a performance regression. We also cover other topics like blogging, zines, debugging and learning new things. Please enjoy this fun conversation with the amazing Julia Evans!
Website link: https://softwaremisadventures.com/julia Links:
https:/…
playlist_add
Dec 4, 2020
Kelsey Hightower - On ways kubernetes can break, being an effective leader and much more - #1
In this episode, we speak with Kelsey Hightower who is currently a Principal Developer Advocate at Google and one of the most influential individuals in the Kubernetes community. He is also an author and a keynote speaker, with a knack for demystifying complex topics, doing live demos and enabling others to succeed.
In this insightful conversation, we cover wide ranging topics from his role at Google to the art of storytelling. We get into some very interesting details of how Kubernetes can break in production and practices that work for Kelsey in being an effective leader.
Links:
https://twitter.com/kelseyhightower
https://github.com/kelseyhightower
Music Credits:
Vlad Gluschenko — Forest
License: Creative Commons Attribution 3.0
Unported: https://creativecommons.org/licenses/by/3.0/deed.en
playlist_add
Nov 28, 2020
Introducing Software Misadventures Podcast - #0
In this episode, Ronak, Austin and Guang share the origin story - who they are, what this podcast is about and why they are doing this.
They've seen first hand how stressful it is when something breaks in production but also found it to be the best opportunity to learn about a system more deeply. They started this podcast to have in-depth conversations with software and devops experts and hear their stories from the trenches about how software breaks in production.
In upcoming conversations, they discuss the principles and practical tips to build resilient software as well as advice to grow as technical leaders.
Learn more at https://softwaremisadventures.com.
Music Credits:
Vlad Gluschenko — Forest
License: Creative Commons Attribution 3.0
Unported: https://creativecommons.org/licenses/by/3.0/deed.en
playlist_add