#225 Can subinterpreters free us from Python's GIL?
Play • 1 hr 11 min
Have you heard that Python is not good for writing concurrent asynchronous code? This is generally a misconception. But there is one class of parallel computing that Python is not good at: CPU bound work running the Python layer.

What's the main problem? It's Python's GIL or Global Interpreter Lock of course. Yet, the fix for this restriction may have been hiding inside CPython since version 1.5: subinterpreters.

Join me to talk about PEP 554 with core developer Eric Snow.

Links from the show

Eric on Twitter: @ericsnowcrntly
Eric's "Multi-core Python" project: github.com/ericsnowcurrently/multi-core-python
Blog post (2016): ericsnowcurrently.blogspot.com
Dave Beazley's talk on concurrency (performance): dabeaz.com
PEP 554 ("Multiple Interpreters in the Stdlib"): python.org
CSP: wikipedia.org
Original notes for PEP 554: docs.google.com
CAPI: python.org
Python benchmarks: github.com
Slides from Language Summit 2018: docs.google.com
Slides from Language Summit 2019: docs.google.com

Talk at PyCon US 2019, "to GIL or not to GIL: the Future of Multi-Core (C)Python"
Video: youtube.com
Slides: docs.google.com

Sponsors

Ting
TopTal
Talk Python Training
Python Bytes
Python Bytes
Michael Kennedy and Brian Okken
#221 Pattern matching and accepting change in Python with Brett Cannon
Sponsored by Datadog: pythonbytes.fm/datadog Special guest: Brett Cannon Brian #1: Keeping up with Rich * Will McGugan has been building Rich * It looks like it’s on its way to becoming a full fledged TUI (text user interface) * December: Live view: no blog post on that, I don’t think. * January: Tree view: Rendering a tree view in the terminal with Python and Rich * February: Layouts: Building Rich terminal dashboards * fun fullscreen.py example, uses Live view * Also, python -m rich will display a demo screen that shows tons of the stuff that Rich can do * Many of the features also have a stand alone demo built in, like: $ python -m rich.layout $ python -m rich.tree $ python -m rich.live * Although I haven’t figured out how to kill the live demo. it doesn’t seem to time out, and it eats Ctrl-C in my terminal. * I’d really like to use Rich for interactive stuff, like keyboard interrupts and arrow keys and tab and such. It’d be fun. * Which brings me to the bottom right corner of the python -m rich output. It includes a GitHub Sponsor link for Will. * Also, Will, unless it’s a contradiction to RTD TOS, I think you should include a Sponsor link in the Rich documentation. * Let’s convince Will to make Rich a full TUI. Michael #2: 12 requests per second * If you take a look around the blogosphere at various benchmarks for Python web frameworks, you might start to feel pretty bad about your own setup. * The incredible work of the guys at magic stack, getting 100,000 requests per second from uvloop in a single thread. * There’s the FastAPI benchmarks * Even more mind-blowing is Japronto which claims an insane 1.2 million requests per-second in a single thread * But what about your “boring” Flask or Django app? And how realistic are these benchmarks? Usually, not very. * Here’s an article diving into this for a “proper” ORM style app. * 12 - 80 requests per sec: Both our sync workers are now hitting a blazing 12 requests per second 😅 Using async workers seems to help a lot, but oddly Sanic struggles here. * Be sure to run in prod on a “real” server setup (nginx + gunicorn or whatever) * Compare this to Talk Python Training - Python 3, uWSGI, Pyramid, MongoDB, $20/mo server * Get around 125 requests/sec @ 100ms response time on a single server. * More realistically, we can handle about 10,000-20,000 concurrent “realistic” users before performance suffers. Brett #3: Python Launcher for Unix reaches RC (probably 😉) * Exclusive! 😁 * Started right after PyCon US 2018 * Implemented in Rust (it’s my “good size” Rust learning project) * The Python Launcher for Windows works by: * Checking the shebang line * If VIRTUAL_ENV is set * Find the newest pythonX.Y version/command on $PATH * Can specify specific versions via e.g. -3 or -3.9 * PY_PYTHON and PY_PYTHON3 environment variables supported * --list also works * Can use PYLAUNCH_DEBUG to see what the tool is doing * --help covers all of this * Unix version differs in the preference of shebang versus VIRTUAL_ENV over shebang * Figure on Unix you will chmod the executable bit if you truly care about the shebang * I also assume at this point people will use entry points if they really want to tie something to an interpreter version * How often do people peg their scripts to a specific Python version instead of python2 or python3? * What do people think of this logic swap (hence the “probably”)? * Unix bonus feature: will automatically use any virtual environment found in .venv in the working directory (and no, what directory is considered is not configurable 😁) * All of this has made it py my preferred way of running Python on my machine * Really useful with Starship and its Python support (does away with the big “Tip” box they have for the python_binary setting) Michael #4: Build a text editor with Python and curses * [curses](https://docs.python.org/3/library/curses.html) is a library to avoid having to deal with low level issues like efficiently painting to the terminal screen and receiving user input. * a barebones curses application import curses def main(stdscr): while True: k = stdscr.getkey() if k == "q": sys.exit(0) if __name__ == "__main__": curses.wrapper(main) * Clear the screen with stdscr.erase() * Adding text (a line of text) to the screen: stdscr.addstr(row, 0, line) * The article covers interesting topics like a “view” into the file that fits the screen and a cursor you move around with arrow keys Brett #5: Pattern matching and accepting change in Python * The “5-barrel foot-gun” in the room (to use Brian’s words from the last episode 😉) * Usual places have people commenting from “I like this” to screaming bloody murder * I think there are many “new” people to the language who don’t know Python prior to Python 3, so they don’t realize how much things used to regularly change in the language * Pattern matching was very much debated publicly, so this wasn’t a secret (and I’m sorry if you didn’t have the time to participate, but Python development doesn’t even always wait for me and I’m on the SC, so …) * The 2020 SC also announced publicly the possibility of this back in December with their recommendation to accept the PEP(s) * Also usual comments of “why did they waste their time on that?!? Go fix packaging!” (and it’s almost always packaging 🙄) * This is open source: some people wanted to put their personal time and effort into trying to get pattern matching into Python, so that’s what they did * If you want to help with Python’s packaging ecosystem, you can do so but trying to tell people what they “need” or “should” do with their time is simply rude * History repeats itself: every change is unwelcome unless it solves your problem * Pattern matching very much opens up opportunities for certain problems that were not easily possible before, e.g. parsers and compilers are classics (and hence why they are so often implemented in functional languages) * I don’t think you will see this in nearly every code base like you do e.g. list comprehensions * E.g. I’m sure data scientists aren’t saying any of this since they got @, right? 😉 * People also claiming it isn’t Pythonic need to note that Guido helped drive this * Do you know what is Pythonic better than Guido? 😉 * He might not be BDFL anymore, but that doesn’t mean he still doesn’t have good design sense, i.e. if you like Python up to this point then trust Guido’s gut that this is a good thing * “In Guido We Trust” (you can even get it on a mug 😉) * If you use pattern matching in real-world code and have feedback to provide with enough time to consider it before b1, then please let python-dev know * E.g. there is a chance to change the meaning of _ if that is truly your biggest hang-up * This will all probably become a blog post * Running title is “The Social Contract of Open Source” * These kinds of attitudes against people trying their best to make things better for folks is what led to Guido retiring from being the BDFL in the first place, me having to take a month off from open source every year, etc. * Aside: more influenced by Scala than by Haskell (not sure where Michael and some other people I’ve seen online got the idea Haskell played into this) * Did you know we got list comprehensions from Haskell? Brian #6: A Quick Intro to Structural Pattern Matching in Python * aka the “switch” statement. I mean, the “match” statement. * Also known as PEP 636, Appendix A — Quick Intro * courtesy Guido van Rossum * This finally helps me to get my head around simple uses of the new syntax for 3.10 * simple form: def http_error(status): match status: case 400: return "Bad request" case 401: return "Unauthorized" cas…
59 min
Google Cloud Platform Podcast
Google Cloud Platform Podcast
Google Cloud Platform
Cloud SQL Insights with Nimesh Bhagat
This week on the podcast, Mark Mirchandani and Gabi Ferrara talk with Nimesh Bhagat about Cloud SQL Insights. This powerful tool enables developers to diagnose database issues for faster, smoother performance. Nimesh tells us the inspiration for Cloud SQL Insight’s development and describes its biggest benefits. One of the important aspects of Insights is the ability for developers to gain an application-centric view by allowing them to tag database queries with SQL comments. These tags are aggregated in Insights and give developers a visual of the database queries. Here, developers can see load patterns and use that information to improve database efficiency. Cloud SQL Insights offers managed database analysis that helps developers understand the past and predict the future. Simplifying the journey of database debugging, Nimesh explains, was the goal of creating Cloud SQL Insights. He takes us through the process of using the software, pointing out the improvements Insights makes over the old way. Cloud SQL Insights only launched in January, but it’s already helping numerous clients with their projects. Nimesh describes these real-world uses, including Major League Baseball experience as part of Insights Early Access Program. Nimesh Bhagat Nimesh is a product manager at Google Cloud, he leads Cloud SQL Insights. He has worked across engineering and product roles, building highly available and high performance enterprise infrastructure used by Fortune 500 companies. His passion lies in combining powerful infrastructure with simple user experience so that every business and developer can build software at scale and velocity. Cool things of the week * A new collaboration with Google Cloud blog * Don’t fear the authentication: Google Drive edition blog Interview * Cloud SQL Insights docs * Cloud SQL Documentation docs * GCP Podcast Episode 163: Cloud SQL with Amy Krishnamohan podcast * Google Cloud Monitoring site * Database observability for developers: introducing Cloud SQL Insights blog * Introduction to Cloud SQL Insights codelab * Boost your query performance troubleshooting skills with Cloud SQL Insights blog * Introducing Sqlcommenter: An open source ORM auto-instrumentation library blog * Introducing Cloud SQL Insights video * Cloud SQL Github site What’s something cool you’re working on? Gabi is working on several things, including Schema Migrations with CI/CD pipelines. She is always available on Twitter and she offers free office hours! Sound Effects Attribution * “Small Audience Laugh” by Tim Kahn of Freesound.org
33 min
Kubernetes Podcast from Google
Kubernetes Podcast from Google
Adam Glick and Craig Box
Multi-Cluster Services, with Jeremy Olmsted-Thompson
This week we talk multi-cluster services with Jeremy Olmsted-Thompson, co-chair of the Kubernetes Multicluster SIG, and tech lead on the Google Kubernetes Engine platform team. Guest host Tim Hockin shows us the way. Do you have something cool to share? Some questions? Let us know: * web: kubernetespodcast.com * mail: kubernetespodcast@google.com * twitter: @kubernetespod Chatter of the week * Episode 41, with Tim Hockin * The Machete Order * John Boyega on Star Wars News of the week * Istio 1.9 * IstioCon 2021 - February 22-26 * Mayadata spins out Chaos Native * Cilium Network Policy editor * Kubernetes network policy explained by Dominik Tornow * Trend Micro write-up on container-escaping malware * Dynatrace Cloud Automation and native log support * Episode 119, with Alois Reitbauer * Shipa 1.2 * New GKE, EKS and AKS releases * Tanzu Build Service 1.1 * Kubernetes 101 Retrospective by Jeff Geerling * CFP for the eight KubeCon EU pre-days * Designing for SaaS on Kubernetes at Teleport by Virag Mody * Comparing OPA/Gatekeeper and Kyverno by Chip Zoller Links from the interview * Anthos on VMware * SIG Multicluster * Federation v2 update * Multi-Cluster Services KEP * Namespace sameness * Gateway API (formerly known as Service APIs) * Istio RFC * Introducing GKE multi-cluster services * Multi-cluster Ingress * Cluster API * Cluster ID KEP * Jeremy Olmsted-Thompson on Twitter and GitHub
26 min
More episodes
Search
Clear search
Close search
Google apps
Main menu