Dec 25, 2021
001. MQs, storages, and dataframes
A few hot updates
* Apache Geode 1.12.5
* enterprise edition is known as gemfire
* geodistributed storage
* has native clients in Java, C#, and C++ (!)
* JTA compliant transaction support
* Pinot released 0.9.0
* Added Segment Merge and Rollup
* Rollup is a technique for tree-like groupby example: city, streets, houses
* General info about pinot
* Made by guys from LinkedIn and Uber has zookeeper as deps
* column-oriented database
* It's an OLAP tool for real-time analytics
* there are BI tools focused on dashboards and reports used by analists etc
* this is more for data exploration for de / ds folks
* Near real-time ingestion from streams (Kafka, Kinesis, and batch ingestion from Hadoop/S3 and the likes)
* It has built-in UI for SQL edits and general BI for exploration focus on realtime analytics
* You can connect Pinot to various BI tools such Superset, Tableau, or PowerBI to visualize data in Pinot
* RocketMq 4.9.0 / 4.9.2 Comparison Table
* Based on ActiveMQ
* Does not need ZooKeeper
* Has conecept of strict message order
* Has focus on perfect configuration OOTB
* Rich web interface
* SQLite 3.37
* new STRICT table definition and ANY type works as cast on write
* cli client update:
* multiple connections from the same client
* security mode with `-safe`
* author is well-known as a supporter of flexible typing, have a read https://sqlite.org/flextypegood.html Ligthning
* Superset 1.3.2
* bugfixes
* if you never saw what's 1.3.0 has to offer, check it out: they have funnels
* also revised treemap viz
* Beam 2.34.0
* NiFi new release (1.15.0)
* main feature is parameter context inheritance
* Apache Ratis release
* Raft
* Airflow 2.2.2
* bugfixes 🤷♀️
* Nats 2.6.5 recent release
* bugfixes Discussion: Are dataframes necessary?
Kotlin DataFrame
Music by https://t.me/red_hands Dr. Igor Mosyagin
Data Engineer @ Klarna
Igor identifies himself as a pragmatic engineer with strong academic background. A theoretical physicist by training, he eventually assumed he had enough PhDs and left Academia to work with Data-* related things. As of 2022, Igor works as a Data Platform Engineer at Klarna. On top of that, he’s a huge fan of cephalopods, math rock, and quantum mechanics. He also hates baked carrots so much he decided to mention it in this bio
* Visit Website (opens in a new tab)
* Visit Twitter account (opens in a new tab)
* Visit LinkedIn account (opens in a new tab)
* Email Pasha Finkelshteyn
Developer advocate @ JetBrains
Having 14 years of experience in IT, Pasha went through a fire in water, from technical support to developer, team lead, and data engineer. Now Pasha works as a developer advocate for Data Engineering at JetBrains. He helps develop the Big Data Tools plugin, gives talks on Kotlin and various aspects of data engineering, and work with data. Also, he is the author and maintainer of Kotlin API for Apache Spark.
* Visit Website (opens in a new tab)
* Visit Twitter account (opens in a new tab)
* Visit Facebook account (opens in a new tab)
* Visit Instagram account (opens in a new tab)
* Visit LinkedIn account (opens in a new tab)
* Visit GitHub account (opens in a new tab)
* Email