Python Bytes
Python Bytes
Oct 16, 2020
#203 Scripting a masterpiece for Python web automation
Play • 41 min

Sponsored by DataDog: pythonbytes.fm/datadog

Michael #1: Introducing DigitalOcean App Platform

  • Reimagining PaaS to make it simpler for you to build, deploy, and scale apps.
  • Many of our customers have come to DigitalOcean after their PaaS became too expensive, or after hitting various limitations.
  • You can build, deploy, and scale apps and static sites by simply pointing to your GitHub repository.
  • Built on DigitalOcean Kubernetes, the App Platform brings the power, scale, and flexibility of Kubernetes without exposing you to any of its complexity.
  • App Platform is built on open standards providing more visibility into the underlying infrastructure than in a typical closed PaaS environment.
  • You can also enable ‘Autodeploy on Push,’ which automatically re-deploys the app each time you push to the branch containing the source code.
  • To efficiently handle traffic spikes (planned or unplanned), the App Platform lets you scale apps horizontally (i.e., add more instances that serve your app) and vertically (beef up the instances with more CPU and memory resources). (with zero downtime)
  • What can you build with the App Platform? Web apps, Static sites, APIs, Background workers

Brian #2: Announcing Playwright for Python

  • playwright-python
  • playwrignt-pytest
  • it’s a Microsoft thing
  • the pitch: “With the Playwright API, you can author end-to-end tests that run on all modern web browsers. Playwright delivers automation that is faster, more reliable and more capable than existing testing tools.”
  • timeout-free automation
    • automatically waits for the UI to be ready
  • Intended to stay modern
    • emulation of mobile viewports
    • geolocation
    • web permissions
    • can automate scenarios across multiple pages
  • cross browser
    • Chromium (Chrome and Edge), WebKit (Safari), and Firefox
    • Safari rendering even works on Windows and Linux
  • pytest compatible
  • Django compatible
  • Can work within CI/CD, even GH actions.

Michael #3: Asynchronously Opening and Closing Files in asyncio

  • Article by Chris Wellons
  • asyncio has support for asynchronous networking, subprocesses, and interprocess communication. However, it has nothing for asynchronous file operations — opening, reading, writing, or closing.
  • If a file operation takes a long time, perhaps because the file is on a network mount, then the entire Python process will hang.
  • Let’s build it!
  • The usual way to work around the lack of operating system support for a particular asynchronous operation is to dedicate threads to waiting on those operations. By using a thread pool, we can even avoid the overhead of spawning threads when we need them. Plus asyncio is designed to play nicely with thread pools anyway.
  • open() uses with so build an aopen() to have async with. Here’s the tasty bit:
        def __aenter__(self):
            def thread_open():
                return open(*self._args, **self._kwargs)
            loop = asyncio.get_event_loop()
            self._future = loop.run_in_executor(None, thread_open)
            return self._future

Brian #4: Excel: Why using Microsoft's tool caused Covid-19 results to be lost

  • this article was on bbc.com, but it was in several places
  • Nearly 16,000 coronavirus cases went unreported in England.
  • Logs pulled together from data from commercial testing firms (filed as csv files) was combined in a Excel xls template so that it could then be uploaded to a central system and made available to the NHS Test and Trace team, as well as other government computer dashboards.
  • XLS was one problem. Limit is about 65k rows.
  • XLSX increases that limit by about 16 times.
  • But still, …. Excel for this?
  • Comment from Prof Jon Crowcroft from the University of Cambridge:
    • "Excel was always meant for people mucking around with a bunch of data for their small company to see what it looked like.”
    • “And then when you need to do something more serious, you build something bespoke that works - there's dozens of other things you could do.”
    • "But you wouldn't use XLS. Nobody would start with that."
  • In short: Best practices in computing don’t always make it into the rest of the world. Much of the world still runs on Excel.
  • What does this have to do with Python? Well.. Big datasets should use databases and Python.
  • Check out the Talk Python free webcast on moving from Excel to Python: talkpython.fm/excel-webcast

Michael #5: locust.io

  • via Prayson Daniel
  • locust.io is awesome tool to simulate users hammering your endpoint. Quite handy.
  • An open source load testing tool: Define user behavior with Python code, and swarm your system with millions of simultaneous users.
  • Usage: after installing it via pip, you can map your local endpoint locust --host=http://localhost:5000 and open http://localhost:8089 to access the locust web ui to simulate usage
  • Features:
    • Define user behavior in code: No need for clunky UIs or bloated XML. Just plain code.
    • Distributed & scalable: Locust supports running load tests distributed over multiple machines, and can therefore be used to simulate millions of simultaneous users
    • Proven & battle tested: Locust has been used to simulate millions of simultaneous users. Battlelog, the web app for the Battlefield games, is load tested using Locust, so one can really say Locust is Battletested ;).
  • Example:
    from locust import HttpUser, between, task

    class WebsiteUser(HttpUser):
        wait_time = between(5, 15) 

        def on_start(self):
            self.client.post("/login", {
                "username": "test_user",
                "password": ""
            })

        @task
        def index(self):
            self.client.get("/")
            self.client.get("/static/assets.js")

        @task
        def about(self):
            self.client.get("/about/")

Brian #6: Fixing Hacktoberfest

  • various sources
  • Hacktoberfest is an interesting idea sponsored by Digital Ocean, and other sponsors.
    • Overall, it’s a good idea. Encourage people to contribute by bribing them with a t-shirt and other swag.
  • Problem and some solutions outlined well by Anthony Sottile in what’s (wrong with) hacktoberfest?
    • There’s always been some spam associated with hacktoberfest.
      • Tiny bizarre PRs, PRs to unmaintained repos, etc.
    • This year has been worse
    • A fairly popular YouTuber posted a video showing people how to get a free t-shirt by doing things like adding “- an awesome project” or expanding “It’s” to “It is” to the readme, then submitting it as “improved docs”.
  • Changes:
    PRs count if:
    > Submitted during the month of October AND (
    >   The PR is labelled as hacktoberfest-accepted by a maintainer OR
    >   Submitted in a repo with the hacktoberfest topic AND (
    >     The PR is merged OR
    >     The PR has been approved
    >   )
    > )
- The deadline for completions, merging, labeling, and approving is November 1.
- I applaud DO and whoever else is working on hacktoberfest for reacting quickly to this.

Extras:

Michael:

  • PyCascades 2021 will take place on Saturday, February 20th from many locations across the Pacific Northwest and beyond.
  • Call for Proposals 📣 PyCascades has been lucky to give our stage to incredible speakers with wonderful talks over the last three years. We are really looking forward to showcasing our community again next year. Our Call for Proposals (CFP) opens today and closes at the end of the day on Tuesday, November 10th, 2020 Anywhere on Earth.
  • Patricio Reyes, a researcher at Barcelona Supercomputing Center (virtual tour):

Joke: More Classical Programmer Paintings

Kubernetes Podcast from Google
Kubernetes Podcast from Google
Adam Glick and Craig Box
Cilium, with Thomas Graf
Thomas Graf is the inventor of Cilium and the co-founder of Isovalent. Cilium is a container networking plugin built on top of eBPF, bringing modern SDN technologies to accelerate your pods. Adam and Craig also discuss the many uses of Christmas trees. Do you have something cool to share? Some questions? Let us know: * web: kubernetespodcast.com * mail: kubernetespodcast@google.com * twitter: @kubernetespod Chatter of the week * Christmas trees: * Keep clear (mostly) * Culinary uses * Discussed in episodes 104 and 111 News of the week * Google grants $3m to the CNCF to run the Kubernetes infrastructure * AWS Managed Grafana and Prometheus * In partnership with Grafana Labs * Red Hat acquires Stackrox * Windows Containers GA in OpenShift 4.6 * CNCF Annual Report * KubeCon NA 2020 Transparency Report * Rancher announces Harvester * I’ll give you the key * Kubernetes 1.20 feature deep-dives: * Pod impersonation an short-lived volumes * Third-party device metrics GA * More granular control of storage permission * Sonobuoy goes beyond conformance * Project Contour security audit * Pulse: stats from Envoy Mobile * Crossplane 1.0 * Project Karavi from Dell Technologies * Cluster API provider for Microsoft Azure * Vitess project journey report * Tanzu Gemfire * Kubernetes Security Essentials from the CNCF Links from the interview * Chains and tables * Berkeley Packet Filter * eBPF * Episode 91: eBPF and Falco, with Leonard Di Donato * High level languages for kernel developers * eBPF Summit 2020 * Cilium * Is it DNS? * Is it a series of tubes? * BGP * Hubble * Accelerating Envoy and Istio with Cilium * Episode 128: Antrea, with Antonin Bas * Bringing Cilium to GKE with Dataplane v2 * Maglev load balancing connection scheduling * Isovalent * Notes on A16Z’s investment * Thomas Graf on Twitter
41 min
Azure DevOps Podcast
Azure DevOps Podcast
Jeffrey Palermo
Jérôme Laban on Multi-Platform DevOps - Episode 123
This week, Jeffrey is joined by Jérôme Laban, CTO of the open-source Uno Platform, and a 4x recipient of the Microsoft MVP award. The Uno Platform is a framework that aims to improve the development cycle of cross-platform apps using Windows, iOS, Android, and WebAssembly using Mono and Xamarin. It is also Open Source (Apache 2.0) and available on GitHub. In this conversation, Jérôme shares their DevOps success story and all of the thought that went into creating a complete DevOps environment for a platform that targets a multitude of computing environments. He also shares details of its creation, what developers should know about it, gives advice, and shares invaluable resources. Topics of Discussion: [:38] Be sure to visit AzureDevOps.Show for past episodes and show notes. [1:02] About The Azure DevOps Podcast and Jeffrey’s offer to speak at virtual user groups. [1:10] Clear Measure is hiring! Be sure to check out the link in the show notes. [1:33] About today’s guest, Jérôme Laban! [1:49] Jeffrey welcomes Jérôme to the podcast. [1:58] About Jérôme’s career background and what has led him to become the CTO of the Uno Platform. [4:03] Regarding the Uno Platform, what should people be looking for now vs. what they should be looking for in the future for cross-platform and mobile development? [8:00] Jérôme walks listeners through the creation of the Uno Platform. [13:44] Jérôme elaborates on the design of the Uno Platform and the branching strategy that they put in place at the front-end. [15:08] The Uno Platform has enabled automated builds upon pull request creation. How many tests are they able to fit into that and what duration does that pull request build take on the feature branch? [16:28] Is there a short cycle build for smaller issues such as a spelling error? [17:41] Jérôme explains what happens in the environment after the pull request is accepted and merges into master. [20:25] A word from Azure DevOps Podcast’s sponsor: Clear Measure. [20:56] Do you fully deploy to target platforms on the pull request build process? [22:08] Jérôme discusses Calculator.Platform.Uno. [24:11] Jérôme received the codebase for the calculator from the Windows team. Did he also receive the test cases for it as well? And did those port over? [25:28] With Uno, will WinForms applications and WPF desktop applications just be able to be “slid” into WebAssembly and URL launched? [27:09] With Uno, how many different types of test frameworks are there and what are they? [30:24] Is the state-of-the-art for web still Selenium? What about mobile? [31:05] Does the Xamarin UI test cover Android and iOS? [31:13] What would you use for UI testing for WebAssembly? [32:38] If people are interested in this cross-platform UI testing do they need to use Uno Platform? [33:33] For developers that are developing new applications now, what technologies and frameworks should they be investing in and which should they be letting go as we look ahead into the future? [36:13] Jeffrey thanks Jérôme for joining the podcast. [36:45] Where to get in touch with Jérôme and learn more about the Uno Platform. Mentioned in this Episode: Azure DevOps Clear Measure (Sponsor) .NET DevOps for Azure: A Developer's Guide to DevOps Architecture the Right Way, by Jeffrey Palermo — Available on Amazon! bit.ly/dotnetdevopsebook — Click here to download the .NET DevOps for Azure ebook! Jeffrey Palermo’s Youtube Jeffrey Palermo’s Twitter — Follow to stay informed about future events! The Azure DevOps Podcast’s Twitter: @AzureDevOpsShow Uno Platform Uno Platform Discord Community @UnoPlatform on Twitter Jérôme Laban’s Twitter @jlaban Jérôme Laban’s Blog Blazor .NET 5.0 NuGet Xamarin Calculator.Platform.Uno Selenium GitVersion Mergify Want to Learn More? Visit AzureDevOps.Show for show notes and additional episodes.
38 min
Software Defined Talk
Software Defined Talk
Software Defined Talk LLC
Episode 281: That’s a thing, I don’t need to read about it
This week we discuss VMware CEO Pat Gelsinger jumping to Intel and what is going on with DevSecOps. Plus, lots advice on picking movies both you and your partner will enjoy. Rundown VMware CEO → Intel What’s the latest Solarwinds hack news? Coté is figuring out “DevSecOps” - or is it “DevOpsSec”? Relevant to your interests M&A Red Hat to Acquire Kubernetes-Native Security Leader StackRox (https://www.redhat.com/en/about/press-releases/red-hat-acquire-kubernetes-native-security-leader-stackrox) F5 to acquire @Volterra_ (https://twitter.com/f5/status/1347291942363811841?s=21) VMware/Intel VMware has the strategy and culture to thrive after CEO Pat Gelsinger's exit to Intel (https://www.theregister.com/2021/01/14/pat_gelsinger_vmware_legacy/) VMware Names Zane Rowe As Interim CEO (https://www.crn.com/news/virtualization/vmware-names-zane-rowe-as-interim-ceo) Prodigal Son Gelsinger Returns As Intel CEO (https://go.forrester.com/blogs/prodigal-son-gelsinger-returns-as-intel-ceo/) Bitcoin and Blockchain Is blockchain coming to your bank? (https://thehustle.co/01082021-blockchain-banks/) 85% of Italian Banks Are Exchanging Interbank Transfer Data on Corda - CoinDesk (https://www.coindesk.com/85-of-italian-banks-are-exchanging-interbank-transfer-data-on-corda) Lost Passwords Lock Millionaires Out of Their Bitcoin Fortunes (https://www.nytimes.com/2021/01/12/technology/bitcoin-passwords-wallets-fortunes.html) Don’t Forget Your Bitcoins (https://www.bloomberg.com/opinion/articles/2021-01-12/don-t-forget-your-bitcoins) RISC-V BeagleBoard BeagleV (https://beagleboard.org/beaglev) Raspberry PI Why We Love the Raspberry Pi (https://www.nytimes.com/wirecutter/reviews/raspberry-pi/) Buy a Raspberry Pi Zero W – Raspberry Pi (https://www.raspberrypi.org/products/raspberry-pi-zero-w/) Solarwinds Details on SolarWinds Hack (https://twitter.com/briankrebs/status/1348828797966147584?s=21) SolarWinds hires former Trump cyber security chief Chris Krebs (https://www.ft.com/content/df641e33-9150-4846-b4f7-db4e3175d290) NYT JetBrains story, I'm calling it - the story was irresponsibly released. (https://twitter.com/MalwareJake/status/1347650824416227331) Deplatforming Parler accuses Amazon of breaking antitrust law in suspending hosting services. (https://www.nytimes.com/2021/01/11/business/parler-amazon.html) Parler Finds Refuge With the Far-Right's Favorite Webhost (https://www.vice.com/en/article/4ad7dp/parler-finds-refuge-with-the-far-rights-favorite-webhost) Parler loses data (https://twitter.com/salmeron_manny/status/1348604719934173185) Parler’s amateur coding could come back to haunt Capitol Hill rioters (https://arstechnica.com/information-technology/2021/01/parlers-amateur-coding-could-come-back-to-haunt-capitol-hill-rioters/) The balkanization of the cloud is bad for everyone (https://www.technologyreview.com/2020/12/17/1014967/balkanization-cloud-computing-bad-everyone/) Decentralization Is A Necessity Now (https://pomp.substack.com/p/decentralization-is-a-necessity-now) It happened. Twitter shut down @realDonaldTrump for good (https://thehustle.co/01112021-Twitter-Trump/) State of the World 2021 (https://people.well.com/conf/inkwell.vue/topics/510/State-of-the-World-2021-page01.html) Goodreads plans to retire API access, disables existing API keys | Joe's website (https://joealcorn.co.uk/blog/2020/goodreads-retiring-API) ‘Your Cock Is Mine Now:’ Hacker Locks Internet-Connected Chastity Cage, Demands Ransom (https://www.vice.com/en/article/m7apnn/your-cock-is-mine-now-hacker-locks-internet-connected-chastity-cage-demands-ransom) Video games have replaced music as the most important aspect of youth culture | Sean Monahan (https://www.theguardian.com/commentisfree/2021/jan/11/video-games-music-youth-culture) Intel CEO Bob Swan to be replaced with VMWare's Pat Gelsinger (https://finance.yahoo.com/news/intel-ceo-bob-swan-reportedly-set-to-depart-in-february-to-be-replaced-with-vm-wares-pat-gelsinger-142520631.html?guccounter=1) WhatsApp gives users an ultimatum: Share data with Facebook or stop using the app (https://arstechnica.com/tech-policy/2021/01/whatsapp-users-must-share-their-data-with-facebook-or-stop-using-the-app/) First Oracle said it powered Zoom. Then AWS claimed it. Now Zoom says it uses co-located kit (https://www.theregister.com/2021/01/13/zoom_prospectus_reveals_colo_infrastructure/) Poland plans to make censoring of social media accounts illegal (https://www.theguardian.com/world/2021/jan/14/poland-plans-to-make-censoring-of-social-media-accounts) Dropbox to cut workforce by 11% (https://www.axios.com/dropbox-to-cut-workforce-by-11-75f8c050-29bb-46ea-a7dc-d9fa972dcee2.html?utm_source=newsletter&utm_medium=email&utm_campaign=newsletter_axioslogin&stream=top) Nonsense I'm glad you have passed the certification! I just don't want to see it. (https://github.com/antonbabenko/you-have-passed-the-certification) Coté in TikTok (https://www.tiktok.com/@drunkandretired/video/6917223569395174658?lang=en). Sponsors strongDM — Manage and audit remote access to infrastructure. Start your free 14-day trial today at: strongdm.com/SDT (http://strongdm.com/SDT) Now Hiring Jordi wants you to work at Tricentis (https://www.tricentis.com/company/careers/all/) Michael wants to be a Solution Architect at Cloudbees (https://boards.greenhouse.io/cloudbees/jobs/2582391) More info in SDT Slack (https://www.softwaredefinedtalk.com/slack) see channel #jobs Conferences Call for Papers (https://sessionize.com/devopsdays-texas-2021/) ends on Jan. 31st for DevOpsDay Texas on March 2nd. (https://devopsdays.org/events/2021-texas/welcome/) SDT news & hype Join us in Slack (http://www.softwaredefinedtalk.com/slack). Send your postal address to stickers@softwaredefinedtalk.com (mailto:stickers@softwaredefinedtalk.com) and we will send you free laptop stickers! Follow us on Twitch (https://www.twitch.tv/sdtpodcast), Twitter (https://twitter.com/softwaredeftalk), Instagram (https://www.instagram.com/softwaredefinedtalk/) and LinkedIn (https://www.linkedin.com/company/software-defined-talk/). Brandon built the Quick Concall iPhone App (https://itunes.apple.com/us/app/quick-concall/id1399948033?mt=8) and he wants you to buy it for $0.99. Use the code SDT to get $20 off Coté’s book, (https://leanpub.com/digitalwtf/c/sdt) Digital WTF (https://leanpub.com/digitalwtf/c/sdt), so $5 total. Become a sponsor of Software Defined Talk (https://www.softwaredefinedtalk.com/ads)! Recommendations Matt: Apple TV’s Ted Lasso (https://tv.apple.com/au/show/ted-lasso/umc.cmc.vtoh0mn0xn7t3c643xqonfzy) Brandon: Possessor (https://www.rottentomatoes.com/m/possessor_uncut). Coté: Descript (https://www.descript.com), so far. Bear app (https://bear.app). Don’t spike your mind during family time with inserting work. Photo Credit (https://unsplash.com/photos/ny-lHmsHYHk) Photo Credit (https://unsplash.com/photos/eM6WUs4nKMY)
1 hr 2 min
The Cloudcast
The Cloudcast
Cloudcast Media
An Event-Driven Apps Look Ahead for 2021
James Urquhart (@jamesurquhart, Global Field CTO @VMware, O’Reilly Author) talks about event-driven application architectures, how it's changing real-time business models, and technology stack driven the evolution.  *SHOW: *483 *SHOW SPONSOR LINKS:* * BMC Wants to Know if your business is on its A-Game * BMC Autonomous Digital Enterprise * Datadog Security Monitoring Homepage - Modern Monitoring and Analytics * Try Datadog yourself by starting a free, 14-day trial today. Listeners of this podcast will also receive a free Datadog T-shirt. * Okta - You should not be building your own Auth * Learn how Okta helped Cengage improve student success rates during COVID. *CLOUD NEWS OF THE WEEK *- http://bit.ly/cloudcast-cnotw *CHECK OUT OUR NEW PODCAST - **"CLOUDCAST BASICS"* *SHOW NOTES:* * Flow Architectures - The Future of Event Streaming & Event-Driven Integration * The Cloudcast Eps.43 - James Urquhart * The Cloudcast Eps.344 - Bringing AI to the Edge (Swim.ai) * The Cloudcast Eps.334 - The Future of Edge Computing (Derek Collison, Synadia) *Topic 1 *- Welcome back to the show. We’ve known you for quite a while, going back to working together on very early Cloud stuff. You’ve always enjoyed being focused on complex, distributed systems. Tell us what you’re focused on these days.  *Topic 2 *- Let’s talk about this concept of “event-driven” and flow. Where did it come from, what does it do, why is it valuable to application designers? *  * *Topic 2a *- What is a “flow” and how is it related to event-driven? *Topic 3 *- Events are data. We’ve had relational databases for data, and then we had NoSQL or eventually-consistent databases for data. Are events a new type of data, or a new way to deal with data in a different context?  (channels, replays, etc.) *Topic 4 *- Can we talk through an example of an event-driven application, or an event-driven integration between multiple organizations? How is it new/different? What unique capabilities does it bring now?  (Kafka, IoT, API Gateways, etc.) *Topic 5 *- Cloud made IT self-service. Serverless made Ops become on-demand. If I’m a business leader, what does event-driven give us?*  * *Topic 6 *- Where are we in the maturity of event-driven architectures? What might be some of the next stages coming in 2021 or 2022?   *FEEDBACK?* * Email: show at thecloudcast dot net * Twitter: @thecloudcastnet
43 min
More episodes
Search
Clear search
Close search
Google apps
Main menu