Hex: Data Project Sharing with Caitlin Colgrove and Barry McCardel
Play • 49 min

Data science is a collaborative field. Collaboration requires sharing the artifacts that data scientists are working on, such as Jupyter Notebooks and SQL tables. Hex is a platform for improving sharing across data science workflows. Caitlin Colgrove and Barry McCardel are founders of Hex and they join the show to discuss what they have built.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Hex: Data Project Sharing with Caitlin Colgrove and Barry McCardel appeared first on Software Engineering Daily.

The Backend Engineering Show with Hussein Nasser
The Backend Engineering Show with Hussein Nasser
Hussein Nasser
The 2021 Slack Outage (Detailed analysis)
On Jan 4th 2021, Slack experienced a global outage that prevented customers from using the service for nearly 5 hours. Slack has released the Root cause analysis incident report which I’m going to summarize in the first part of this video. After that Ill provide a lengthy deep dive of the incident so make sure to stick around for that. If you are new here, I make backend engineering videos and also cover software news, so make sure to Like comment and subscribe if you would like to see more plus it really helps the channel, lets jump into it. So This is an approximation of Slack’s architecture based on what was the described in the reports. Clients connects to load balancers, load balancers distribute requests to backend servers and backend servers finally make requests to database servers which is powered by mysql through vitess sharding. All of those are connected by routers in cross boundary network. Around 6AM jan 4 , the cross network boundary routers setting between LB and backend and backend to DB started to drop packets. This lead to the load balancers slowly marking backends as unhealthy and removing them from the fleet Which compounded the amount of requests The number of failed requests eventually triggered the provisioning service to start spinning an absurdly large number of backend servers However the provisioning service couldn’t keep up with the huge demand and shortly started to time out for the same networking reasons and eventually ran out of maximum open file handles. Eventually Slack’s cloud provider increased the networking capacity and backend servers went back to normal around 11 AM PST This was a summary of the slack outage, Now set back, grab your favorite beverage and lets go through the detailed incident report! 0:00 Outage Summary 2:00 Detailed Analysis Starts 5:20 The Root Cause 30:00 Corrective Actions --- Send in a voice message: https://anchor.fm/hnasr/message
44 min
Streaming Audio: A Confluent podcast about Apache Kafka
Streaming Audio: A Confluent podcast about Apache Kafka
Confluent, original creators of Apache Kafka®
Change Data Capture and Kafka Connect on Microsoft Azure ft. Abhishek Gupta
What’s it like being a Microsoft Azure Cloud advocate working with Apache Kafka® and change data capture (CDC) solutions? Abhishek Gupta would know! At Microsoft, Abhishek focuses his time on Kafka, Databases, Kubernetes, and open source projects. His experience in a wide variety of roles ranging from engineering, consulting, and product management for developer-focused products has positioned him well for developer advocacy, where he is now. Switching gears, Abhishek proceeds to break down the concept of CDC starting off with some of the core concepts such as "commit logs." Abhishek then explains how CDC can turn data around when you compare it to the traditional way of querying the database to access data—you don't call the database; it calls you. He then goes on to discuss Debezium, which is an open source change data capture solution for Kafka. He also covers some of the Azure connectors on Confluent, Azure Data Explorer, and use cases powered by the Azure Data Explorer Sink Connector for Kafka. EPISODE LINKS * Streaming Data from Confluent Cloud into Azure Data Explorer * Integrate Apache Kafka with Azure Data Explorer * Change Data Capture with Debezium ft. Gunnar Morling * Tales from the Frontline of Apache Kafka DevOps ft. Jason Bell * MySQL CDC Source (Debezium) Connector for Confluent Cloud * MySQL, Cassandra, BigQuery, and Streaming Analytics with Joy Gao * Join the Confluent Community Slack * Learn more with Kafka tutorials, resources, and guides at Confluent Developer * Live demo: Kafka streaming in 10 minutes on Confluent Cloud * Use *60PDCAST* to get an additional $60 of free Confluent Cloud usage (details)
43 min
Software Defined Talk
Software Defined Talk
Software Defined Talk LLC
Episode 281: That’s a thing, I don’t need to read about it
This week we discuss VMware CEO Pat Gelsinger jumping to Intel and what is going on with DevSecOps. Plus, lots advice on picking movies both you and your partner will enjoy. Rundown VMware CEO → Intel What’s the latest Solarwinds hack news? Coté is figuring out “DevSecOps” - or is it “DevOpsSec”? Relevant to your interests M&A Red Hat to Acquire Kubernetes-Native Security Leader StackRox (https://www.redhat.com/en/about/press-releases/red-hat-acquire-kubernetes-native-security-leader-stackrox) F5 to acquire @Volterra_ (https://twitter.com/f5/status/1347291942363811841?s=21) VMware/Intel VMware has the strategy and culture to thrive after CEO Pat Gelsinger's exit to Intel (https://www.theregister.com/2021/01/14/pat_gelsinger_vmware_legacy/) VMware Names Zane Rowe As Interim CEO (https://www.crn.com/news/virtualization/vmware-names-zane-rowe-as-interim-ceo) Prodigal Son Gelsinger Returns As Intel CEO (https://go.forrester.com/blogs/prodigal-son-gelsinger-returns-as-intel-ceo/) Bitcoin and Blockchain Is blockchain coming to your bank? (https://thehustle.co/01082021-blockchain-banks/) 85% of Italian Banks Are Exchanging Interbank Transfer Data on Corda - CoinDesk (https://www.coindesk.com/85-of-italian-banks-are-exchanging-interbank-transfer-data-on-corda) Lost Passwords Lock Millionaires Out of Their Bitcoin Fortunes (https://www.nytimes.com/2021/01/12/technology/bitcoin-passwords-wallets-fortunes.html) Don’t Forget Your Bitcoins (https://www.bloomberg.com/opinion/articles/2021-01-12/don-t-forget-your-bitcoins) RISC-V BeagleBoard BeagleV (https://beagleboard.org/beaglev) Raspberry PI Why We Love the Raspberry Pi (https://www.nytimes.com/wirecutter/reviews/raspberry-pi/) Buy a Raspberry Pi Zero W – Raspberry Pi (https://www.raspberrypi.org/products/raspberry-pi-zero-w/) Solarwinds Details on SolarWinds Hack (https://twitter.com/briankrebs/status/1348828797966147584?s=21) SolarWinds hires former Trump cyber security chief Chris Krebs (https://www.ft.com/content/df641e33-9150-4846-b4f7-db4e3175d290) NYT JetBrains story, I'm calling it - the story was irresponsibly released. (https://twitter.com/MalwareJake/status/1347650824416227331) Deplatforming Parler accuses Amazon of breaking antitrust law in suspending hosting services. (https://www.nytimes.com/2021/01/11/business/parler-amazon.html) Parler Finds Refuge With the Far-Right's Favorite Webhost (https://www.vice.com/en/article/4ad7dp/parler-finds-refuge-with-the-far-rights-favorite-webhost) Parler loses data (https://twitter.com/salmeron_manny/status/1348604719934173185) Parler’s amateur coding could come back to haunt Capitol Hill rioters (https://arstechnica.com/information-technology/2021/01/parlers-amateur-coding-could-come-back-to-haunt-capitol-hill-rioters/) The balkanization of the cloud is bad for everyone (https://www.technologyreview.com/2020/12/17/1014967/balkanization-cloud-computing-bad-everyone/) Decentralization Is A Necessity Now (https://pomp.substack.com/p/decentralization-is-a-necessity-now) It happened. Twitter shut down @realDonaldTrump for good (https://thehustle.co/01112021-Twitter-Trump/) State of the World 2021 (https://people.well.com/conf/inkwell.vue/topics/510/State-of-the-World-2021-page01.html) Goodreads plans to retire API access, disables existing API keys | Joe's website (https://joealcorn.co.uk/blog/2020/goodreads-retiring-API) ‘Your Cock Is Mine Now:’ Hacker Locks Internet-Connected Chastity Cage, Demands Ransom (https://www.vice.com/en/article/m7apnn/your-cock-is-mine-now-hacker-locks-internet-connected-chastity-cage-demands-ransom) Video games have replaced music as the most important aspect of youth culture | Sean Monahan (https://www.theguardian.com/commentisfree/2021/jan/11/video-games-music-youth-culture) Intel CEO Bob Swan to be replaced with VMWare's Pat Gelsinger (https://finance.yahoo.com/news/intel-ceo-bob-swan-reportedly-set-to-depart-in-february-to-be-replaced-with-vm-wares-pat-gelsinger-142520631.html?guccounter=1) WhatsApp gives users an ultimatum: Share data with Facebook or stop using the app (https://arstechnica.com/tech-policy/2021/01/whatsapp-users-must-share-their-data-with-facebook-or-stop-using-the-app/) First Oracle said it powered Zoom. Then AWS claimed it. Now Zoom says it uses co-located kit (https://www.theregister.com/2021/01/13/zoom_prospectus_reveals_colo_infrastructure/) Poland plans to make censoring of social media accounts illegal (https://www.theguardian.com/world/2021/jan/14/poland-plans-to-make-censoring-of-social-media-accounts) Dropbox to cut workforce by 11% (https://www.axios.com/dropbox-to-cut-workforce-by-11-75f8c050-29bb-46ea-a7dc-d9fa972dcee2.html?utm_source=newsletter&utm_medium=email&utm_campaign=newsletter_axioslogin&stream=top) Nonsense I'm glad you have passed the certification! I just don't want to see it. (https://github.com/antonbabenko/you-have-passed-the-certification) Coté in TikTok (https://www.tiktok.com/@drunkandretired/video/6917223569395174658?lang=en). Sponsors strongDM — Manage and audit remote access to infrastructure. Start your free 14-day trial today at: strongdm.com/SDT (http://strongdm.com/SDT) Now Hiring Jordi wants you to work at Tricentis (https://www.tricentis.com/company/careers/all/) Michael wants to be a Solution Architect at Cloudbees (https://boards.greenhouse.io/cloudbees/jobs/2582391) More info in SDT Slack (https://www.softwaredefinedtalk.com/slack) see channel #jobs Conferences Call for Papers (https://sessionize.com/devopsdays-texas-2021/) ends on Jan. 31st for DevOpsDay Texas on March 2nd. (https://devopsdays.org/events/2021-texas/welcome/) SDT news & hype Join us in Slack (http://www.softwaredefinedtalk.com/slack). Send your postal address to stickers@softwaredefinedtalk.com (mailto:stickers@softwaredefinedtalk.com) and we will send you free laptop stickers! Follow us on Twitch (https://www.twitch.tv/sdtpodcast), Twitter (https://twitter.com/softwaredeftalk), Instagram (https://www.instagram.com/softwaredefinedtalk/) and LinkedIn (https://www.linkedin.com/company/software-defined-talk/). Brandon built the Quick Concall iPhone App (https://itunes.apple.com/us/app/quick-concall/id1399948033?mt=8) and he wants you to buy it for $0.99. Use the code SDT to get $20 off Coté’s book, (https://leanpub.com/digitalwtf/c/sdt) Digital WTF (https://leanpub.com/digitalwtf/c/sdt), so $5 total. Become a sponsor of Software Defined Talk (https://www.softwaredefinedtalk.com/ads)! Recommendations Matt: Apple TV’s Ted Lasso (https://tv.apple.com/au/show/ted-lasso/umc.cmc.vtoh0mn0xn7t3c643xqonfzy) Brandon: Possessor (https://www.rottentomatoes.com/m/possessor_uncut). Coté: Descript (https://www.descript.com), so far. Bear app (https://bear.app). Don’t spike your mind during family time with inserting work. Photo Credit (https://unsplash.com/photos/ny-lHmsHYHk) Photo Credit (https://unsplash.com/photos/eM6WUs4nKMY)
1 hr 2 min
The Angular Show
The Angular Show
E045 - Micro Frontends in Angular
Micro-what?!? What is a micro frontend, what does it have to do with Angular, why would I want to use this strategy, and how do I go about implementing this in my organization? These are all good questions. Whether you know what a micro frontend is, have never heard of it, or are currently implementing this strategy in your organization, this is an episode of the Angular Show that you do not want to miss. Join panelists Aaron Frost, Brian Love, and Jennifer Wadella as they start their journey in learning about Micro Frontends in Angular. We had the privilege to sit down with Zama Khan Mohamed, who is an expert in Micro Frontends. Zama teaches us the basics of what a micro frontend is and why we would consider this strategy for our applications. Then, we dive into the details on how Angular developers and organizations using Angular, as well as a mix of other frameworks, can leverage Micro Frontends in their architecture. We also discuss some of the benefits and costs of this approach, and why you might want to consider using micro frontends. Finally, we also touch on Webpach version 5's new support for federated modules and the impact this has on how we build modern micro frontends. Phew - that sounds like a lot, but have no fear, your friendly panelists are here to walk with you through this new landscape of micro frontends in the JavaScript ecosystem. Show Notes: https://github.com/single-spa/single-spa https://webpack.js.org/concepts/module-federation/ Connect with us: @mohamedzamakhan @likeOMGitsFEDAY @brian_love @aaronfrost
1 hr 9 min
Azure DevOps Podcast
Azure DevOps Podcast
Jeffrey Palermo
Charles Flatt on Learning as a Developer - Episode 124
Charles Flatt is joining the podcast today! He has been a software developer since 1994 and has helped over a dozen organizations succeed on over fifty projects, both small and large. Charles has an unusual breadth of business and personal experience from foodservice and retail to music, business management, hardware installation, and of course, software development. In this episode, Charles talks about his learning as a developer and some of his successes, big lessons, and key takeaways from the course of his career. He shares actionable advice for developers, teams, and organizations on how to improve; his favorite resources and books for further learning; the metrics that matter the most; and what he sees as being the key components of what makes a DevOps organization successful. Topics of Discussion: [:38] Be sure to visit AzureDevOps.Show for past episodes and show notes. [1:00] About The Azure DevOps Podcast, Clear Measure, and Jeffrey’s offer to speak at virtual user groups. [1:24] Clear Measure is hiring! Be sure to check out the link in the show notes. [1:34] About today’s guest, Charles Flatt! [1:57] Jeffrey welcomes Charles to the podcast. [2:34] Charles shares his career journey before software development and how he began his career in software. [6:49] Charles speaks about where he has worked and what he has been working on in the last decade. [11:48] Charles shares some of the big lessons and key takeaways from the course of working on over fifty projects in software development. [18:21] Charles and Jeffrey discuss their favorite books on DevOps and give their recommendations on what you should be reading as a developer today. [20:50] A word from Azure DevOps Podcast’s sponsor: Clear Measure. [21:22] What Charles sees as needing to change within an organization in order to become more successful. [23:01] Charles gives some actionable advice on how to begin improving as a developer, as a team, and as an organization. [28:06] Charles and Jeffrey discuss the metrics that matter the most. [29:18] Jeffrey and Charles discuss the importance of continuous integration and what it really means to do continuous integration. [32:32] Charles recommends some go-to resources to check out after today’s podcast! [34:33] Jeffrey thanks Charles for joining the podcast! Mentioned in this Episode: Azure DevOps Clear Measure (Sponsor) .NET DevOps for Azure: A Developer's Guide to DevOps Architecture the Right Way, by Jeffrey Palermo — Available on Amazon! bit.ly/dotnetdevopsebook — Click here to download the .NET DevOps for Azure ebook! Jeffrey Palermo’s Youtube Jeffrey Palermo’s Twitter — Follow to stay informed about future events! The Azure DevOps Podcast’s Twitter: @AzureDevOpsShow Charles Flatt’s LinkedIn Azure DevOps Podcast Ep. 33: “Rockford Lhotka on Software Architecture” Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations, by Nicole Forsgren PhD The Phoenix Project (A Novel About IT, DevOps, and Helping Your Business Win), by Gene Kim The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations, by Gene Kim, Patrick Debois, John Willis, and Jez Humble Continuous Integration: Improving Software Quality and Reducing Risk, by Paul M. Duvall, Steve Matyas, and Andrew Glover Good Habits, Bad Habits: The Science of Making Positive Changes That Stick, by Wendy Wood Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs, by John Doerr Want to Learn More? Visit AzureDevOps.Show for show notes and additional episodes.
35 min
Python Bytes
Python Bytes
Michael Kennedy and Brian Okken
#216 Container: Sort thyself!
Sponsored by Datadog: pythonbytes.fm/datadog Special guest: Jousef Murad, Engineered Mind podcast (audio, video) Watch on YouTube Brian #1: pip search. Just don’t. * pip search [query] is supposed to “Search for PyPI packages whose name or summary contains [query]” * The search feature looks like it’s going to be removed and the PyPI api for it removed. * Alternative, and better approach, just manually look at pypi.org and search for stuff. * Right now it does this: $ pip search pytest ERROR: Exception: Traceback (most recent call last): ... [longish traceback ommited] --- xmlrpc.client.Fault: [Fault -32500: "RuntimeError: PyPI's XMLRPC API has been temporarily disabled due to unmanageable load and will be deprecated in the near future. See https://status.python.org/ for more information."] * The Python Infrastructure status page says, as of Jan 12: “Update - The XMLRPC Search endpoint remains disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again.” * This started becoming a problem in mid December. * The endpoint was just never architected to handle the scale it’s getting now. * There’s a current issue “Remove the pip search command”, open on pip. * The commend thread is locked now, but you can read some of the history. * I personally don’t understand the need to hammer search with a CI system or other. * Probably should be using a local cache or local pypi mirror for an active/aggressive CI system. * If you have scripts or jobs that run pip search , it ain’t gonna work, so probably best to remove that. Michael #2: QPython - Scripting for Android with Python * Python REPL on Android - interesting * Scripting Android tasks with Python - more interesting * Free, open source app that is ad supported. * Some people have commented that their phone is their only “computer” * With SL4A features, you can use Python programming to control Android work: * Android Apps API, such as: Application, Activity, Intent & startActivity, SendBroadcast, PackageVersion, System, Toast, Notify, Settings, Preferences, GUI * Android Resources Manager, such as: Contact, Location, Phone, Sms, ToneGenerator, WakeLock, WifiLock, Clipboard, NetworkStatus, MediaPlayer * Third App Integrations, such as: Barcode, Browser, SpeechRecongition, SendEmail, TextToSpeech * Hardwared Manager: Carmer, Sensor, Ringer & Media Volume, Screen Brightness, Battery, Bluetooth, SignalStrength, WebCam, Vibrate, NFC, USB Jousef #3: Thesis: Deep Learning assistant for designers/engineers * PyTorch (3D) / TensorFlow * The thesis: what is it actually about & goal of the thesis * Libraries mainly used: numpy, pandas * (Reinforcement Learning & GANs) Brian #4: sortedcontainers * Thanks to Fanchen Bao for the topic suggestion. * Pure-Python, as fast as C-extensions, sorted collections library. >>> from sortedcontainers import SortedList >>> sl = SortedList(['e', 'a', 'c', 'd', 'b']) >>> sl SortedList(['a', 'b', 'c', 'd', 'e']) >>> sl *= 10_000_000 >>> sl.count('c') 10000000 >>> sl[-3:] ['e', 'e', 'e'] >>> from sortedcontainers import SortedDict >>> sd = SortedDict({'c': 3, 'a': 1, 'b': 2}) >>> sd SortedDict({'a': 1, 'b': 2, 'c': 3}) >>> sd.popitem(index=-1) ('c', 3) >>> from sortedcontainers import SortedSet >>> ss = SortedSet('abracadabra') >>> ss SortedSet(['a', 'b', 'c', 'd', 'r']) >>> ss.bisect_left('c') 2 * “All of the operations shown above run in faster than linear time.” * Types: * SortedList * SortedKeyList (like SortedList, but you pass in a key function, similar to key in Pythons sorted function.) * SortedDict * SortedSet * Great documentation and tons of performance metrics in the docs. Michael #5: Łukasz Langa Typed Twitter Thread * Let’s riff on typing for a bit. * Here is my philosophy: If I have to type more than three characters to complete a symbol in my editor, something is wrong. * e.g. to go from email_service. → email_service.send_account_email() I should only need to type .sae then tab/enter. These types of things are vastly better because of type hints. * Python type hints are more malleable than even TypeScript. * Lukasz is addressing this comment: Controversial take: Types in a Python code-base are a net negative. * Points * put enough annotations and tooling connects the dots, making plenty of errors evident. * The most common to me at least is when a None creeps in. * The second bug often caught by type checkers is on the "return" boundary: one of your code paths forgets a return. * squiggly lines in your editor * Microsoft is now developing powerful type checking and code completion for Python in VSCode. This effort employs a member of the Python Steering Council, and possibly also the creator of Python himself soon. You think they would settle for "illusion of productivity"? Jousef #6: * Point Cloud operations → open3d Extras: Michael: * via Francisco Giordano Silva: On Brian's ref to using numpy all for array element-wise comparison, also please check out numpy.allclose method. Allows you to compare two arrays based on a given tolerance. Brian: * Just this: 2021 is exhausting so far. * Test & Code has shifted to every other week to allow time for other projects I’m working on. * This is probably a short term change. But I don’t know for how long. It’s definitely not going away though. Just slowing down a bit. Jousef: Engineered Mind podcast
36 min
More episodes
Clear search
Close search
Google apps
Main menu