DEbrief: Recent Episodes
DEbrief
This episode was delayed due to ongiong situation in Ukraine. Thank for understanding.
Hot updates
- Pulsar is updated
- Apache Kylin
"Extreme OLAP Engine for Big Data"
- Three main versions: 2.4, 3.0 and most recent is 4.0.1 v4 released in the autumn 2021
- Brings OLAP back to data
- Been around since 2015, brought to you by eBAY
- Not a friend to HBase, but likes parquet
- Web Interface for all data steps
- Official python client
- with pandas support
- Ambari is killed (put in the attic)
- Apache Hop 1.1
- https://www.leanwithdata.com/blog/2022/02/hop-1.1.0/
- At January, 18th graduated from Incubator
- Apache Hop Sucks!
- Dolphin Scheduler
Lightning news
- Apache Arrow for Rust
- Apache Iceber 0.13.0
- Hudi 0.10.1
- Apache HBase 2.4.9
- Apache Seatunnel easy-to-use ultra-high-performance distributed data integration platform that supports real-time synchronization of massive data
- Apache ORC 1.6.13
- Apache Beam 2.35.0
- Apache Airflow 2.2.3
Discussion: DataSecOps
- OWASP
- Data debiasing
- Data anonymization
Dr. Igor MosyaginData Engineer @ Klarna
Igor identifies himself as a pragmatic engineer with strong academic background. A theoretical physicist by training, he eventually assumed he had enough PhDs and left Academia to work with Data-* related things. As of 2022, Igor works as a Data Platform Engineer at Klarna. On top of that, he’s a huge fan of cephalopods, math rock, and quantum mechanics. He also hates baked carrots so much he decided to mention it in this bio
- Visit Website (opens in a new tab)
- Visit Twitter account (opens in a new tab)
- Visit LinkedIn account (opens in a new tab)
Pasha FinkelshteynDeveloper advocate @ JetBrains
Having 14 years of experience in IT, Pasha went through a fire in water, from technical support to developer, team lead, and data engineer. Now Pasha works as a developer advocate for Data Engineering at JetBrains. He helps develop the Big Data Tools plugin, gives talks on Kotlin and various aspects of data engineering, and work with data. Also, he is the author and maintainer of Kotlin API for Apache Spark.
- Visit Website (opens in a new tab)
- Visit Twitter account (opens in a new tab)
- Visit Facebook account (opens in a new tab)
- Visit Instagram account (opens in a new tab)
- Visit LinkedIn account (opens in a new tab)
- Visit GitHub account (opens in a new tab)
Hot updates
- dbt 1.0.0 released
- dbt is gaining popularity
- Great instrument which solves really existing problem
- RedisJSON is out for public preview https://redis.com/blog/redisjson-public-preview-performance-benchmarking/
- Need to have Redis 6.x or later probably a good point to talk once again that
- RedisJSON* is faster than MongoDB and ElasticSearch on direct read, write, and update workloads.
- available in Redis Cloud or you can always buuild it yourself
- Basically a bunch of JSON commands for "native" json experience:
- JSON.SET
- JSON.GET
- JSON.NUMINCRBY
- Client libraries for Go/Node.js/Python/Java/.NET/PHP/Ruby only Java and Python are official libs
- You can index your JSON documents using RediSearch, and you can set it up to update indexes on every write
- Check the benchmarks
- Why community is dissing MongoDB recently? Their SSPL license is to blame
- FerretDB: relevant interesting solution — MongoDB interface over PostgreSQL
-
Neo4j 4.4 is out last december
- user impersonation is the main new feature
- All cloud providers have their own graph db
- GCP and AWS provide AuraDB: neo4j managed cloud service
- Amazon Neptune
- Azure CosmosDB
- Neo4j has two main language engines: cypher, gremlin gremlin is a Java API for Graph DB. Techincally, Gremlin itself is a database engine
.... cypher is a term used to describe freestyle rap in a group setting, which might be something to consider when you search online for cypher tips + Some say cypher looks very Lua-ish + Neo4j Desktop and a browser interface additional plugins for easier visualizations/explorations + "I wonder if we could solve this year's AoC problems with some DB like that" + O'Relly often gives out their Neo4j book for free + Why would you ever need a graph database?
Lightning news
- Apache IoTDB 0.12.4
- group by multi level
- Lots of major verndor release updates due to well-known log4j vulnerabilities Do not forget to update log4j if you didn't yet
- Calcite 1.29.0
- Log4j obviously
- Apache Beam new release
- Minor relase, 3 braking changes
- Last time we mention it here because they release new versions weekly
- LakeFS new releases
- They just never stop:
- performance improvements
- new OpenAPI method
- security check
- They just never stop:
- Apache ORC 1.7.2 released
- It's just good to know that this format is still alive and is being developed.
- row level filtering in columnar storage format
- now row-level predicates will work on rows (at the reader level)!
Discussion: ETL and Reverse ETL
Dr. Igor MosyaginData Engineer @ Klarna
Igor identifies himself as a pragmatic engineer with strong academic background. A theoretical physicist by training, he eventually assumed he had enough PhDs and left Academia to work with Data-* related things. As of 2022, Igor works as a Data Platform Engineer at Klarna. On top of that, he’s a huge fan of cephalopods, math rock, and quantum mechanics. He also hates baked carrots so much he decided to mention it in this bio
- Visit Website (opens in a new tab)
- Visit Twitter account (opens in a new tab)
- Visit LinkedIn account (opens in a new tab)
Pasha FinkelshteynDeveloper advocate @ JetBrains
Having 14 years of experience in IT, Pasha went through a fire in water, from technical support to developer, team lead, and data engineer. Now Pasha works as a developer advocate for Data Engineering at JetBrains. He helps develop the Big Data Tools plugin, gives talks on Kotlin and various aspects of data engineering, and work with data. Also, he is the author and maintainer of Kotlin API for Apache Spark.
- Visit Website (opens in a new tab)
- Visit Twitter account (opens in a new tab)
- Visit Facebook account (opens in a new tab)
- Visit Instagram account (opens in a new tab)
- Visit LinkedIn account (opens in a new tab)
- Visit GitHub account (opens in a new tab)
A few hot updates
- Apache Geode 1.12.5
- enterprise edition is known as gemfire
- geodistributed storage
- has native clients in Java, C#, and C++ (!)
- JTA compliant transaction support
- Pinot released 0.9.0
- Added Segment Merge and Rollup
- Rollup is a technique for tree-like groupby example: city, streets, houses
- General info about pinot
- Made by guys from LinkedIn and Uber has zookeeper as deps
- column-oriented database
- It's an OLAP tool for real-time analytics
- there are BI tools focused on dashboards and reports used by analists etc
- this is more for data exploration for de / ds folks
- Near real-time ingestion from streams (Kafka, Kinesis, and batch ingestion from Hadoop/S3 and the likes)
- It has built-in UI for SQL edits and general BI for exploration focus on realtime analytics
- You can connect Pinot to various BI tools such Superset, Tableau, or PowerBI to visualize data in Pinot
- Added Segment Merge and Rollup
- RocketMq 4.9.0 / 4.9.2 Comparison Table
- Based on ActiveMQ
- Does not need ZooKeeper
- Has conecept of strict message order
- Has focus on perfect configuration OOTB
- Rich web interface
- SQLite 3.37
- new STRICT table definition and ANY type works as cast on write
- cli client update:
- multiple connections from the same client
- security mode with
-safe
- author is well-known as a supporter of flexible typing, have a read https://sqlite.org/flextypegood.html
Ligthning
- Superset 1.3.2
- bugfixes
- if you never saw what's 1.3.0 has to offer, check it out: they have funnels
- also revised treemap viz
- Beam 2.34.0
- NiFi new release (1.15.0)
- main feature is parameter context inheritance
- Apache Ratis release
- Raft
- Airflow 2.2.2
- bugfixes 🤷♀️
- Nats 2.6.5 recent release
- bugfixes
Discussion: Are dataframes necessary?
Kotlin DataFrame
Music by https://t.me/red_hands
Dr. Igor MosyaginData Engineer @ Klarna
Igor identifies himself as a pragmatic engineer with strong academic background. A theoretical physicist by training, he eventually assumed he had enough PhDs and left Academia to work with Data-* related things. As of 2022, Igor works as a Data Platform Engineer at Klarna. On top of that, he’s a huge fan of cephalopods, math rock, and quantum mechanics. He also hates baked carrots so much he decided to mention it in this bio
- Visit Website (opens in a new tab)
- Visit Twitter account (opens in a new tab)
- Visit LinkedIn account (opens in a new tab)
Pasha FinkelshteynDeveloper advocate @ JetBrains
Having 14 years of experience in IT, Pasha went through a fire in water, from technical support to developer, team lead, and data engineer. Now Pasha works as a developer advocate for Data Engineering at JetBrains. He helps develop the Big Data Tools plugin, gives talks on Kotlin and various aspects of data engineering, and work with data. Also, he is the author and maintainer of Kotlin API for Apache Spark.
- Visit Website (opens in a new tab)
- Visit Twitter account (opens in a new tab)
- Visit Facebook account (opens in a new tab)
- Visit Instagram account (opens in a new tab)
- Visit LinkedIn account (opens in a new tab)
- Visit GitHub account (opens in a new tab)
Hot Updates
- Spark 3.2 with pandas API support
- Apache Beam 2.33.0
- Arrow 6
- Airflow 2.2.1
Lightning news
- Streamlit cloud released
- TestContainers Cloud).
- Greenplum 0.16 released
Discussion
Classical CI/CD vs GitOps
Dr. Igor MosyaginData Engineer @ Klarna
Igor identifies himself as a pragmatic engineer with strong academic background. A theoretical physicist by training, he eventually assumed he had enough PhDs and left Academia to work with Data-* related things. As of 2022, Igor works as a Data Platform Engineer at Klarna. On top of that, he’s a huge fan of cephalopods, math rock, and quantum mechanics. He also hates baked carrots so much he decided to mention it in this bio
- Visit Website (opens in a new tab)
- Visit Twitter account (opens in a new tab)
- Visit LinkedIn account (opens in a new tab)
Pasha FinkelshteynDeveloper advocate @ JetBrains
Having 14 years of experience in IT, Pasha went through a fire in water, from technical support to developer, team lead, and data engineer. Now Pasha works as a developer advocate for Data Engineering at JetBrains. He helps develop the Big Data Tools plugin, gives talks on Kotlin and various aspects of data engineering, and work with data. Also, he is the author and maintainer of Kotlin API for Apache Spark.
- Visit Website (opens in a new tab)
- Visit Twitter account (opens in a new tab)
- Visit Facebook account (opens in a new tab)
- Visit Instagram account (opens in a new tab)
- Visit LinkedIn account (opens in a new tab)
- Visit GitHub account (opens in a new tab)