The GeekNarrator: Recent Episodes
Kaivalya Apte
The GeekNarrator podcast is a show hosted by Kaivalya Apte who is a Software Engineer and loves to talk about Technology, Technical Interviews, Self Improvement, Best Practices and Hustle.
Connect with Kaivalya Apte https://www.linkedin.com/in/kaivalya-apte-2217221a
Tech blogs: https://kaivalya-apte.medium.com/
Wanna talk? Book a slot here: https://calendly.com/speakwithkv/hey
Enjoy the show and please follow to get more updates. Also please don’t forget to rate and review the show.
Cheers
In this episode of The Geek Narrator podcast, hosted by Kaivalya Apte, we welcome a special guest, Kishore Gopalakrishna from StarTree, co-author of Apache Pinot and other notable projects. Kishore shares his extensive experience in building real-time analytics and streaming systems, including Apache Pino, Espresso, Apache Helix, and Third Eye. The episode delves into the motivations and challenges behind creating these systems, the innovations they brought to distributed systems, and the impact of community on open-source projects. Kishore also discusses the evolution of testing methodologies, cost optimizations in transactional and analytical systems, and key considerations for companies evaluating real-time analytics solutions. Don't miss this in-depth conversation packed with valuable insights for both seasoned developers and tech enthusiasts!Chapters:00:00 Introduction 03:13 Building Distributed Systems at LinkedIn08:57 Testing and Challenges in Distributed Systems30:50 Advantages of Columnar Storage33:04 The Importance of Upserts34:24 Building a Strong Open Source Community41:10 Challenges and Lessons in System Design51:35 Real-Time Analytics: Do You Need It?StarTree: https://startree.ai/Apache Pinot: https://pinot.apache.org/If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#distributedsystems #kafka #s3 #streaming #realtimeanalytics #database #pinot #startree
In this episode of The GeekNarrator podcast, host Kaivalya Apte interviews Ryan and Richie, the founders of WarpStream. They discuss the architecture, benefits, and core functionalities of WarpStream, a drop-in replacement for Apache Kafka. The conversation covers their experience with Kafka, the design decisions behind WarpStream, and the operational challenges it addresses. They also delve into the seamless migration process, the scalability, and cost benefits, the integration with the Kafka ecosystem, and potential future features. This episode is a must-watch for developers and tech enthusiasts interested in modern, distributed data streaming solutions.Chapters:00:00 Introduction02:27 Introducing Warpstream: A Kafka Replacement11:07 Deep Dive into Warpstream's Architecture35:42 Exploring Kafka's Ordering Guarantees36:52 Handling Buffering and Compaction38:44 Efficient Data Reading and File Caching44:06 WarpStream's Flexibility and Cost Efficiency01:06:59 Future FeaturesLinks:WarpStream : https://www.warpstream.com/Blog: https://www.warpstream.com/blogX: Ryan: https://x.com/ryanworlRichard Artoul: https://x.com/richardartoulKaivalya Apte: https://x.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#distributedsystems #kafka #s3 #streaming
Exploring XTDB with Jeremy Taylor & Malcolm Sparks: An In-Depth Dive into Immutability and Database InternalsIn this episode of the Geek Narrator Podcast, host Kaivalya is joined by Jeremy Taylor and Malcolm Sparks from Juxt to explore XTDB, an immutable database designed to handle complex historical and financial data with precision. They delve into the architecture, internal mechanics, and use cases while discussing the importance of immutability. This episode covers everything you need to know about XTDB and its capabilities. Whether you're a developer interested in databases or someone curious about data management and history tracking, this discussion offers invaluable insights.Chapters:00:00 Introduction02:51 Challenges with General Purpose Databases11:50 XTDB: A New Approach to Databases31:56 Understanding Kafka and XTDB Integration36:06 Querying and Indexing in XTDB40:31 Temporal Data Management and Use Cases54:52 Deployment and User ExperienceXTDB: https://xtdb.com/XTDB Github: https://github.com/xtdb/xtdbJuxt: https://www.juxt.pro/Juxt Github: https://github.com/juxtIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#sql #kafka #datastorage #immutable
In this episode of The GeekNarrator podcast, host Kaivalya Apte dives into the complexities of testing distributed systems with Will Wilson from Antithesis. If you’re grappling with the challenges of testing databases, micro-services, and distributed systems, this episode is a must-watch. Will Wilson demystifies the concept of deterministic simulation testing, shares insights about its advantages over conventional testing methods, and explains how Antithesis helps developers ensure software reliability. Learn about the various strategies and techniques used to identify and resolve bugs, and explore how deterministic simulation can transform your software testing approach. Perfect for developers, engineers, and tech enthusiasts who are keen on improving their testing methodologies for complex systems.Chapters:00:00 Introduction03:04 Limitations of Conventional Testing Methods04:09 Understanding Deterministic Simulation Testing08:07 Implementing Deterministic Simulation Testing14:30 Real-World Example: Chat Application19:56 Antithesis Hypervisor and Determinism27:06 Defining Properties and Assertions38:34 Optimizing Snapshot Efficiency40:44 Understanding Isolation in CI/CD Pipelines43:39 Strategies for Effective Bug Detection47:59 Exploring Program State Trees51:17 Heuristics and Fuzzing Techniques01:01:56 Mocking Third-Party APIs01:05:54 Handling Long-Running Tests01:09:06 Classifying and Prioritizing Bugs01:15:35 Future Plans and Closing RemarksReferences:Hypervisor: https://antithesis.com/blog/deterministic_hypervisor/AFL : https://github.com/google/AFLAntithesis website: https://antithesis.com/Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#distributedsystems #databases #microservices #antithesis #fuzzer #testing
Exploring Turso with Glauber Costa: Insights on SQLite for ProductionIn this episode of The GeekNarrator podcast, host Kaivalya Apte interviews Glauber Costa, founder and CEO of TursoDB. They discuss the inception of TursoDB, Glauber's background in Linux kernel development, and the journey from unikernel projects to founding a database company. Glauber explains TursoDB's enhancements to SQLite for production use, including native replication, schema management, and vector search capabilities. The conversation dives deep into use cases, architecture, and the benefits of a multi-tenant database design. Learn about TursoDB’s future plans and essential insights for developers.Chapters:00:00 Introduction05:05 The Birth of Turso08:02 Challenges and Pivot to libSQL17:12 SQLite for Production: Enhancements and Features22:02 Replication and Backup Solutions23:38 Enterprise-Level Features and Multi-Tenancy25:55 User Experience and Simplicity of TursoDB33:14 Handling Network Failures and Monitoring36:35 Native Replication in SQLite37:52 Virtualizing the Write-Ahead Log39:20 Replication Mechanisms41:31 Primary and Replica Dynamics46:51 Multi-Tenancy and Scalability53:33 Schema Changes and Migrations58:51 Vector Search Capabilities01:02:13 Future Roadmap and FeaturesIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#distributedsystems #databases #sqlite #sql
Deep Dive into Serverless Databases with Neon: Featuring Heikki LinnakangasIn this episode of the Geek Narrator podcast, host Kaivalya Apte is joined by Heikki Linnakangas, co-founder of Neon, to explore the innovative world of serverless databases. They discuss Neon's unique approach to separating compute and storage, the benefits of serverless architecture for modern applications, and dive into various compelling use cases. They also cover Neon's architectural features like branching, auto-scaling, and auto-suspend, making it a powerful tool for both developers and enterprises. Whether you're curious about multi-tenancy, fault tolerance, or developer productivity, this episode offers insightful knowledge about leveraging Neon's capabilities for your next project.00:00 Introduction 00:53 The Birth of Neon: Why It Was Created02:16 Understanding Serverless Databases07:06 Neon's Architecture: Separation of Compute and Storage09:59 Exploring Branching in Neon18:21 Auto Scaling and Handling Spikes in Traffic20:17 The Challenge of Multiple Writers in Distributed Systems22:51 Auto Suspend: Cost-Effective Database Management26:02 Optimizing Cold Start Times27:14 Balancing Cost and Performance28:52 Replication and Durability30:32 Understanding the Storage Layer34:02 Custom LSM Tree Implementation36:21 Fault Tolerance and Failover07:00 Developer Productivity and Use Cases42:56 Migration and Tooling48:35 Future Roadmap and User Experience50:28 Conclusion and Final ThoughtsNeon website: https://neon.tech/Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#PostgreSQL #SQL #RDBMS #NEON
In this video I speak with Felix GV, who is a Principal Staff Engineer at Linkedin, and has done major contributions to the data infrastructure and Linkedin, including VeniceDB.This episode will give you a good understanding of why we need a new database for storing "Derived Data" in a low latency, high performance manner, which is very important for Machine Learning workloads.Chapters:00:00 Introduction 01:42 The Evolution of LinkedIn's Databases03:15 Challenges with Voldemort and the Birth of VeniceDB08:42 Understanding Derived Data13:33 Planet-Scale Applications and Multi-Region Support17:40 Writing Data into VeniceDB22:53 Merging Data in VeniceDB40:31 Understanding the Architecture40:47 Components of the Write Path41:56 Leader and Follower Architecture43:58 Partitioning and DaVinci Client47:57 Read Patterns and Client Options54:25 Fault Tolerance and Recommender Systems01:01:19 Kafka Integration and Deployment01:06:56 Roadmap and Future ImprovementsImportant links:VeniceDB blog: https://www.linkedin.com/blog/engineering/open-source/open-sourcing-venice-linkedin-s-derived-data-platformVeniceDB docs: https://venicedb.org/Qcon: https://youtu.be/pJeg4V3JgYo?si=vblGUxp5fNdKPHoCFollow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#kafka #linkedin #venicedb #Rocksdb
In this video I speak with Philippe Noël, about ParadeDB, which is an Elasticsearch alternative built on Postgres, modernizing the features of Elasticsearch's product suite, starting with real-time search and analytics.I hope you will enjoy and learn about the product.Chapters:00:00 Introduction 01:12 Challenges with Elasticsearch and the Need for ParadeDB02:29 Why Postgres?06:30 Technical Details of ParadeDB's Search Functionality18:25 Analytics Capabilities of ParadeDB24:00 Understanding ParadeDB Queries and Transactions24:22 Application Logic and Data Workflows25:14 Using PG Cron for Data Migration30:05 Scaling Reads and Writes in Postgres31:53 High Availability and Distributed Systems34:31 Isolation of Workloads39:38 Database Upgrades and Migrations41:21 Using ParadeDB Extensions and Distributions43:02 Observability and Monitoring44:42 Upcoming Features and Roadmap46:34 Final ThoughtsImportant links:Links:GitHub: https://github.com/paradedb/paradedbWebsite: https://paradedb.comDocs: https://docs.paradedb.com/Blog: https://blog.paradedb.comFollow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#postgresql #datafusion #parquet #sql #OLAP #apachearrow #database #systemdesign #elasticsearch
In this video I speak with Andrew Lamb, Staff Software Engineer @Influxdb. We discuss FDAP (Flight, DataFusion, Arrow, Parquet) stack for modern OLAP database system design. Andrew shared some insights into why the FDAP stack is so powerful in designing and implementing a modern OLAP database. Chapters:00:00 Introduction01:48 Understanding Analytics: Transactional vs Analytical Databases04:41 The Genesis and Goals of the FDAP Stack09:31 Decoding FDAP: Flight, Data Fusion, Arrow, and Parquet12:40 Apache Parquet: Revolutionizing Columnar Storage17:18 Apache Arrow: The In-Memory Game Changer23:51 Interoperability and Migration with Apache Arrow27:10 Comparing Apache Parquet and Arrow28:26 Exploring Data Mutability in Analytic Systems29:19 Handling Data Updates and Deletions29:24 The Role of Immutable Storage in Analytics30:42 Optimizing Data Storage and Mutation Strategies34:20 Introducing Flight: Simplifying Data Transfer35:02 Deep Dive into Flight's Benefits and SQL Support39:20 Unpacking Data Fusion's SQL Support and Extensibility46:12 The Interplay of FDAP Components in Analytics51:49 Future Directions and Innovations in Data Analytics56:04 Concluding Thoughts on FDAP and Its ImpactFDAP Stack: https://www.influxdata.com/glossary/fdap-stack/FDAP Blog: https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/InfluxDB: https://www.influxdata.com/Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#datafusion #parquet #sql #OLAP #apachearrow #database #systemdesign
In this video I and Pratim Bhosale, Developer Advocate at SurrealDB, talk about SurrealDB, a multi-model database which aims to make Developer’s life easier by letting them focus mainly on the business logic and not on the Database choice. Following chapters will help you understand what is a multi-model database and how SurrealDB shines. Chapters:00:00 Introduction01:48 The Genesis of SurrealDB03:59 SurrealDB's Mission and Use Cases07:34 Understanding Multi-Model Databases10:30 Deep Dive into SurrealDB's Architecture33:09 Deployment and Getting Started with SurrealDB34:31 Future Developments and Use Case Considerations43:51 Final Thoughts and How to Get StartedImportant links:Install SurrealDBhttps://sdb.li/4bqwn38SurrealDB Docs: https://sdb.li/3wxjoxxSurrealDB Website: https://sdb.li/3JMK7JISurrealist:https://sdb.li/4b7wcdhSurrealDB GitHub: https://sdb.li/3JRPNlEFollow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#surrealdb #elasticsearch #search #vectorsearch #acid #databases #sql #joins #indexes #graphdatabase
In this video, I talk to Dhruba, CTO @Rockset about search and realtime analytics. We discussed deep internals of Rockset, its architecture and why is it a great fit for search and realtime analytics use cases. Chapters: 00:00 Introduction02:45 The Evolution of Data Systems: From Hadoop to Rockset07:30 Understanding Rockset: Real-Time Analytics and Search Defined12:01 The Technical Edge: Rockset vs. Elasticsearch18:16 Deep Dive into Rockset's Architecture and Internals28:21 Partitioning, Hashing, and Data Distribution in Rockset36:56 Exploring Hot Storage and Cache Layers37:40 Why Hot Storage is Essential for Low Latency39:05 Optimizing Data Storage with Compression and Delta Encoding39:49 Balancing Cost and Performance in Data Storage41:50 The Power of Converged Indexing in Rockset45:50 Efficient Query Execution and Index Management54:51 Leveraging Mutability for Real-Time Analytics59:24 Deep Dive into Query Processing and Optimization01:04:21 Understanding Joins and Reporting Queries in Rockset01:12:23 Future Directions and Vector Search InnovationsIndex Conference: https://rockset.com/index-conf/Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#rockset #elasticsearch #search #vectorsearch #realtime #databases #sql #joins #indexes
In this episode we explore how to Rapidly Simulate Production Traffic with Michael Drogalis, using his creation ShadowTraffic. I am sure you will be able to relate to all the different problems mentioned in this episode and like how ShadowTraffic aims to solve those problems.I hope you like this conversation.Chapters:00:00 Welcome to The Geek Narrator Podcast: Exploring Deep Tech00:18 The Challenge of Simulating Production Traffic00:59 Introducing Shadow Traffic: A Solution to Data Simulation02:34 Understanding the Problem Space of Data Simulation06:03 How Shadow Traffic Works: A Deep Dive08:17 The Power of Declarative Data Generation with Shadow Traffic10:40 Shadow Traffic's Architecture and Deployment13:02 Configuring Load Testing and Throttling with Shadow Traffic15:47 Testing and Validation in Shadow Traffic20:42 Mimicking Production Data Distribution with Shadow Traffic26:48 Innovative Features for Stream Processing Testing28:47 Shadow Traffic: Adding Faults to Data for Robust Testing29:04 Antithesis and Shadow Traffic: A Synergistic Approach32:46 The Challenge of Generating Realistic Test Data40:04 Enhancing Observability in Data Generation41:50 Customer-Driven Roadmap and Future Vision45:27 Closing ThoughtsShadowTraffic: https://shadowtraffic.io/Contact Michael: https://shadowtraffic.io/contact.htmlFollow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#kafka #s3 #postgres #testing #streamprocessing #loadtesting #chaostesting #demo
If you're involved in the Java space, chances are you've come across #GraalVM. And for those active in the tech community, you might have heard about the recent 1BRC challenge initiated by Gunnar Morling.GraalVM truly showcased its capabilities in this challenge, sparking my curiosity. That's why I reached out to Alina to delve deeper into GraalVM, exploring its features and uncovering how it excels in such endeavors. And here we are talking about GraalVMChapters:00:00 Introduction01:47 GraalVM's Impact on the 1BRC Challenge and Its Features04:34 Exploring GraalVM's Core Features and Benefits08:34 Real-World Success Stories: GraalVM in Action16:18 Understanding Native Image Compilation with GraalVM20:34 Framework Compatibility and GraalVM Integration25:04 Testing and Integration with GraalVM25:26 Exploring Testing and Development with GraalVM25:58 Best Practices for Developing with GraalVM28:11 Migrating to GraalVM: Strategies and Considerations31:25 Performance Optimization in GraalVM35:15 Building and Resource Considerations for GraalVM38:45 Expanding Horizons: Polyglot Programming with GraalVM43:15 Future Directions and Limitations of GraalVM47:40 Engaging the Java Community: GraalVM's Impact50:21 Getting Started with GraalVM: Resources and RecommendationsReferences and Links:- The GraalVM website with docs, downloads, guides: https://www.graalvm.org/- Nicolai Parlog's "Modern Java in Action" demo: https://github.com/nipafx/modern-java-demo- My native version of Nicolai's demo: https://github.com/alina-yur/native-modern-java-demo- For news, follow GraalVM: https://twitter.com/graalvmFollow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#Java #jvm #graalvm #highperformance #JITcompiler #AOT #nativeimage #security #rust #c++
In this episode I am talking to Javier Ramirez from QuestDB, about everything QuestDB. This episode is a great resource to understand how QuestDB works, its architecture, what is it optimised for and whats upcoming as per the roadmap.If you have timeseries data and need a simple yet highly scalable solution, #QuestDB is a great option. Chapters:00:00 Introduction03:04 Understanding QuestDB: Origins and Use Cases09:21 Deep Dive into QuestDB's Architecture and Data Ingestion19:07 Optimizing Data Reads and Writes in QuestDB28:40 Exploring Data Granularity and Partitioning in QuestDB29:29 Optimizing Query Performance with Partition Strategies30:26 Handling Data Ingestion and Query Efficiency32:58 In-depth Look at Data Duplication and Ingestion Performance34:55 Understanding Compression and Its Impact on Performance38:51 Replication and Data Distribution Strategies47:10 Observability and Metrics in QuestDB50:57 Future Developments and Enhancements in QuestDB58:45 Closing Remarks Links:QuestDB: https://questdb.io/Github: https://github.com/questdb/questdb===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#questdb #sql #timeseries #timeseriesanalysis #databases #highscale #scaleup #performance #parquet #S3 #replication #writeaheadlog #wal #durability #columnstore
In this episode I talk to Andras Gerlits, who founded omniledger.io. Andras has a very interesting view on how Distributed Consistency should work that can get rid of several bottlenecks when it comes to maintaining Distributed consistency. He argues how getting rid of a global wall clock and using causality to approach Distributed consistency helps you build resilient, simple and performant systems. We have gone deeper into how that can be achieved and how the product works. Chapters: 00:00 Introduction00:52 Andras's Journey into Distributed Consistency03:04 The Evolution of Data Consistency in Banking and Beyond08:04 Introducing Client-Centric Consistency10:36 Exploring the Standard Model of Distributed Consistency16:01 Redefining Strong Consistency with a Relativistic Approach34:25 Practical Implications of Client-Centric Consistency in Banking36:20 Mitigating Latencies and Partitions in Distributed Systems41:08 Exploring System Reliability and Availability41:52 Tuning System Properties for Specific Use Cases43:07 Comparing Standard and New Models for Data Management45:08 Understanding Local Progress and Mutex-Free Updates47:23 Deep Dive into Token-Based Ordering and Global Calibration58:30 Introducing OmniLedger: A New Approach to Distributed Consistency01:02:41 Performance Optimizations and Tunable Consistency01:08:20 Ideal Use Cases and Potential Limitations of OmniLedger01:14:30 Future Directions and Closing ThoughtsLinks:Our website:https://omniledger.ioA long-form essay on the thinking behind our model:https://medium.com/p/5e397cb12e63A demo of transactionalityhttps://www.youtube.com/watch?v=XJSSjY4szZEI think my blog in general might be interesting to somehttps://medium.com/@andrasgerlitsThe science-paper with all its mathematical rigour:https://www.researchgate.net/publication/359578461_Continuous_Integration_of_Data_Histories_into_Consistent_Namespaces===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#databases #sql #consistency #distributedsystems
In this video I talk to Semih Salihoglu about KuzuDB : A highly scalable, extremely fast, easy to use embeddable Graph Database.Chapters:00:00 Introduction00:40 The Genesis of KuzuDB: From Academic Research to Startup06:40 Graph Databases 101: Understanding the Basics and Beyond10:24 When to Opt for a Graph Database: Use Cases and Advantages19:16 KuzuDB vs. Traditional Databases: A Comparative Analysis24:39 Inside KuzuDB: Optimizations and Data Ingestion Explained31:08 Exploring Query Optimizations in Graph Databases31:34 The Relational Nature of Graph Databases33:33 Factorization: A Key Optimization Technique38:50 Integrating New Data Sources and Handling Joins43:39 Optimizing Write Operations and Index Management50:23 Comparing Kuzu with Other Graph Databases58:50 Future Developments and Vision for KuzuImportant links:- History of DBMSs and the IDS, which is the first database in history, which had a graph-based model: https://dl.acm.org/doi/abs/10.1145/1147376.1147382 is a good paper by CS historian on this history and a must read for everyone interested in the birth of databases as a field. - https://blog.kuzudb.com/post/what-every-gdbms-should-do-and-vision/ blog on the what every GDBMS should do and vision of Kùzu. - The user survey paper that got Semih into GDBMSs. https://arxiv.org/pdf/1709.03188.pdf- Blog on factorization https://blog.kuzudb.com/post/factorization/- Kùzu's RDFGraphs feature https://docs.kuzudb.com/rdf-graphs/===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!
In this video, I talk to Stephan Ewen from Restate, who is popularly known from the world of Apache Flink. We have talked about the problems in the world of Distributed systems and the complex solutions developers have to deal with. This complexity makes the architecture so complex that it eventually creates reliability, Observability and delivery velocity problems. Restate aims to solve it by making resilience and durability for your services, functions and RPC a lot simpler. Chapters: 00:00 Introduction00:45 Introducing Restate: A Solution for Distributed System Challenges01:22 Deep Dive into Restate with Stefan: From Apache Flink to Building Resilient Systems06:04 The Complexities of Distributed Systems and How Restate Addresses Them15:49 The Vision of Restate: Simplifying Developer Experience in Distributed Systems24:42 Integrating Restate into Your Architecture: A User's Perspective33:16 Exploring Restate: The Durable Service Mesh33:32 The Power of Restate in Handling Transactions34:26 Restate's Role in Service Communication and Durability35:40 Deep Dive into Restate's Mechanisms and Benefits38:04 Practical Example: Email Pipeline with Restate39:40 Understanding Restate's Log and Event Handling58:43 Restate's Unique Features and Programming Model01:04:22 Final Thoughts on Restate's Impact and DeploymentRestate: https://restate.dev/===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#distributedsystems #faulttolerance #reliability #resilience
In this episode of The GeekNarrator podcast, our host Kaivalya talks to Seeta Somagani from Volt Active Data, a low latency stream processing platform. They discuss fascinating topics about what low latency stream processing means, the different guarantees that Volt Active Data provides, and the various problems it can solve. They delve into the evolution of VoltDB to Volt Active Data, real-time data processing use cases, the high-level architecture, and how the platform effectively addresses high-concurrency challenges. This is a must-listen for anyone interested in understanding latency critical applications, data processing, and high performance computing.Chapters:00:00 Welcome to The GeekNarrator Podcast with Special Guest from Volt Active Data00:41 Introduction01:45 The Evolution of VoltDB to Volt Active Data06:13 Exploring Real-Time Data Processing and Use Cases08:25 Addressing High-Concurrency Challenges in Various Industries12:57 High-Level Architecture of Volt Active Data19:26 Understanding Stored Procedures and Data Processing in Volt22:48 Practical Application: Tracking Data Usage with Volt Active Data25:16 Diving into Replicated and Partitioned Tables25:44 Exploring Event Processing and Exporting26:57 Understanding Stored Procedures and Performance29:03 Partitioning Strategies and Recommendations31:39 Ensuring Determinism in Stored Procedures35:02 Handling Complex Requirements with Compound Procedures37:25 Fault Tolerance and Data Replication Strategies40:44 Exploring Use Cases for VoltActiveData43:30 The Future of Streaming and VoltActiveData's Role47:05 Closing Remarks and How to Learn MoreVolt Active Data: https://www.voltactivedata.com/use-cases/activesd-streaming-data/===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#sql #streamprocessing #java #acid
In an enlightening episode of the GeekNarrator Podcast, host Kaivalya Apte and TigerBeetle's CEO, Joran, delve deep into the world of online transaction processing (OLTP). They discuss the origin, unique architecture, and innovative methodologies behind TigerBeetle, a database tailored to efficiently handle high-volume transaction systems. The podcast explores the system's key features such as efficient scalability, performance-oriented design, and optimized memory usage, demonstrating its robustness in handling business transactions and accounting. It also elucidates TigerBeetle’s adaptability to various domains beyond finance, like energy management and gaming, while highlighting the rigorous testing it undergoes for impeccable quality assurance.Chapters:00:00 Introduction01:19 Joran's Journey into Databases03:59 Understanding Financial Transaction Databases07:41 The Evolution of OLTP and OLAP16:13 The Need for a New Database: TigerBeetle16:53 Performance and Safety Features of TigerBeetle28:49 The Importance of Safety in Financial Transactions36:49 Changing Developer Experience with TigerBeetle41:43 Understanding the CPU and Memory Bandwidth42:12 The Importance of Data Format Language43:27 The Concept of Serialization and its Impact46:23 The Architecture of TigerBeetle46:29 The Role of Replicated State Machine48:18 The Importance of Consensus in Replication50:20 The Structure of TigerBeetle50:37 The Importance of Log in Systems50:51 Understanding the State in Replicated State Machine52:55 The Role of LSM in TigerBeetle53:55 The Impact of Compaction Process on Performance57:06 The Importance of Predictability in Software01:06:15 The Read and Write Path in TigerBeetle01:14:46 Potential Use Cases for TigerBeetle01:17:09 Understanding the Limitations of TigerBeetle===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!#tigerbeetledb #databases #acid #olap #oltp #postgres #mysql
In this episode, we dive deep into the world of clean coding with none other than the master and pioneer of the field, Uncle Bob. We explore the nuances and the art behind writing effective and efficient scripts. This conversation covers the nitty-gritty of writing and editing scripts, from understanding how to break down large functions, to discussing principles like 'Single Responsibility Principle', 'Dependency Inversion Principle' and how to balance the 'DRY' (Don't Repeat Yourself) principles. Uncle Bob also shares valuable insights on testing, handling errors, naming conventions and how to work with different types of duplication in coding. He shares recommended resources and books that every coder should read.Chapters:00:00 Introduction and Welcome00:06 The Importance of Code Quality00:29 Introducing Robert Martin (Uncle Bob)01:39 Uncle Bob's Journey in Programming02:34 Discussion on Functional Design and New Book03:52 The Evolution of Software Development04:28 Revisiting the Clean Code Book04:49 The Impact of Hardware Changes on Software06:13 The Evolution of Programming Languages07:33 The Importance of Code Structure and Organization09:07 The Impact of Microservices and Open Source11:14 The Role of Modular Programming22:07 The Importance of Naming in Code26:31 The Role of Functions in Code34:12 The Role of Switch Statements in Code42:36 The Importance of Immutability51:00 Dealing with Complex Steps in Programming51:21 Implementing State Machines in Programming51:46 The Pragmatic Approach to Programming53:01 Understanding Error Handling in Programming54:08 The Challenge of Exception Handling57:27 The Importance of Log Messages in Debugging01:03:05 The Dilemma of Code Duplication01:05:51 The Intricacies of Error Handling01:07:40 The Role of Abstraction in Programming01:13:55 The Importance of Testing in Programming01:19:43 The Challenges of Mocking in Testing01:25:11 The Essence of Programming: Discipline, Ethics, and StandardsBook Recommendations:Tidy First: https://www.oreilly.com/library/view/tidy-first/9781098151232/Design Patterns: https://www.amazon.de/-/en/Erich-Gamma/dp/0201633612Analysis Pattern: https://martinfowler.com/books/ap.htmlStructured Analysis and System Specification: https://www.amazon.de/-/en/Tom-Demarco/dp/0138543801Fundamental Algorithms: https://www.amazon.com/Art-Computer-Programming-Vol-Fundamental/dp/0201896834Sorting and Searching: https://www.amazon.de/-/en/Donald-Knuth/dp/0201896850Structure and Interpretation of Computer Programs: https://web.mit.edu/6.001/6.037/sicp.pdf===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!
In this episode of the Geek Narrator Podcast, Kaivalya Apte engages in an invigorating discussion with Dominik Tornow, the founder and CEO of Resonate. They explore Durable Async-Await, an interesting concept in distributed systems world, along with other nuances of distributed programming. Dominik also talks about the development and execution of Resonate to simplify distributed systems with a focus on observability, usability, and it's future direction. The conversation concludes with a discussion of different concurrency models and the future of distributed systemsChapters:00:00 Introduction and Guest Background02:44 Understanding Async Await10:25 Challenges with Current Async Await Model12:53 Introducing Resonate: A Solution for Distributed Async Await13:34 Practical Application: E-commerce Example24:57 Understanding the Role of the Platform in Distributed Systems30:12 Dealing with Partial Failures in Distributed Systems39:44 Getting Started with Resonate40:40 Introduction to Resonate and its Simplicity41:09 Getting Started with Resonate: Installation and Setup42:22 Understanding the Durability Aspect of Resonate42:49 Exploring the Resonate Durable Promise Server44:10 Scaling Up: Introducing Workers into the System48:35 The Importance of Open Standards in Resonate50:17 Exploring the Integration Capabilities of Durable Promises01:04:31 Understanding the Role of Timeouts in Durable Promises01:07:29 The Future of Resonate: Challenges and Upcoming Features01:13:04 Understanding the Limitations of Durable Promises01:14:51 Wrapping Up: Final Thoughts on Resonate and Durable PromisesReferences:A note on Distributed Systems: https://scholar.harvard.edu/files/waldo/files/waldo-94.pdfThinking in Distributed Systems: https://dtornow.gumroad.com/l/distributed-systemsMcCarthy's paper: https://www-formal.stanford.edu/jmc/recursive/recursive.html===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!
Join host Kaivalya Apte in this episode of The Geek Narrator Podcast as he discusses observability engineering with field CTO at Honeycomb, Liz Fong-Jones. They delve into the importance of observability for software engineers, the role of Honeycomb in popularizing this concept, and how observability has evolved over the years. Liz shares her experiences transitioning from being an SRE at Google to advocating for observability at Honeycomb and walking the journey from developer advocate to Field CTO. They discuss the definitions and misconceptions surrounding observability and elucidate on Service-Level Objectives (SLOs) & indicators (SLIs) and challenges they solve. Tune in for an informative and in-depth conversation on observability engineering.Chapters:00:00 Introduction 00:08 Understanding Observability Engineering00:37 Guest Introduction: Liz Fong Jones00:53 Liz's Journey to Field CTO at Honeycomb27:38 Understanding Site Reliability Workbook Materials27:57 Identifying Critical User Journeys29:49 Different Types of Services and Their SLOs33:05 Setting Up SLOs: Granularity and Number42:42 Understanding Service Level Indicators (SLIs)50:26 Common Mistakes in Setting Up SLOs52:09 Cultivating an Observability-Driven Development CultureReferences:Observability Engineering: https://www.oreilly.com/library/view/observability-engineering/9781492076438/ @Google SRE book: https://sre.google/books/===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!
In this video I talk about Apache Pulsar with Matteo Merli, CTO at StreamNative. This episode will provide you good insight about how Apache Pulsar works and more importantly differs with the most popular Pub/Sub and streaming platform Apache Kafka. Things like, what enables possibility of 1 million topics? Why is rebalancing not required? How does decoupled storage and compute architecture works? How it uses the concept of Subscriptions to avoid retaining data unnecessarily? And much more...Chapters:00:00 Introduction and Guest Introduction00:08 Understanding Apache Pulsar and its Origin01:22 The Problem Apache Pulsar was Designed to Solve02:35 The Evolution of Apache Pulsar05:15 Understanding Basic Concepts of Apache Pulsar09:27 Deep Dive into Apache Pulsar's Architecture21:16 Understanding the Flow of Data in Apache Pulsar28:54 Understanding Subscriptions in Apache Pulsar31:57 Understanding End-to-End Latency and Subscription Creation32:32 Broker's Role and Handling Metadata33:05 Memory Management and Consumer Handling34:07 Message Processing and Flow Control34:32 Message Storage and Retrieval36:00 Comparing Pulsar with Kafka43:52 Understanding Multi-Tenancy in Pulsar49:17 Exploring Tiered Storage and Future DevelopmentsImportant links:StreamNative: https://streamnative.io/Apache Pulsar: https://pulsar.apache.org/Matteo Merli: https://twitter.com/merlimat===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!
Deep Dive into Victoria Metrics with Alex and RomanJoin the insightful discussion with Vitoriametrics creators, Alex and Roman, in the Geekneritor podcast hosted by Kaivalya Apte. This episode explores the internals of Victoria Metrics - a highly scalable monitoring solution and time series database. Discover the origins of Victoria Metrics, understand how it evolved, and learn about its unique architecture and functionality. From the concept of time series, the usage of consistent hashing in data distribution to real-world applications, it's all packed into this engaging conversation.00:00 Introduction01:52 The Genesis of VictoriaMetrics02:18 The Journey from Postgres to Clickhouse03:19 The Transition from Prometheus to Victoria Metrics05:08 The Birth and Evolution of Victoria Metrics13:01 The Architecture of Victoria Metrics20:10 Data Ingestion and Integration in Victoria Metrics29:15 Understanding the Vector Metric Architecture30:30 Comparing Shared Storage and Object Store31:00 Designing the VictoriaMetrics Architecture32:01 The Role of Object Storage36:15 The Importance of Indexing43:19 Understanding the Ingestion Process45:46 Exploring the Select Process55:55 Future Plans for Victoria MetricsImportant Links:1. Architecture Overview: https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#architecture-overview2. How ClickHouse Inspired Us to Build a High Performance Time Series Databasehttps://altinity.com/wp-content/uploads/2021/11/How-ClickHouse-Inspired-Us-to-Build-a-High-Performance-Time-Series-Database.pdf3. Frequently asked questions.https://docs.victoriametrics.com/FAQ.html===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!
Join us on a deep dive into the intricacies of TiDB with Li Shen from PingCap. In this episode, Li Shen provides a comprehensive exploration of TiDB, its unique features, and how it tackles scalability and reliability issues commonly associated with MySQL. If you're dealing with struggles in your MySQL cluster and seeking a more dependable and scalable system, TiDB might be the solution for you. This conversation touches on various aspects of this cutting-edge database, its operational mechanism, use case scenarios, and how it's optimized for different workloads. Key topics include: the architecture of TiDB, the journey of data from API to storage node, embracing analytical use cases, the importance of database reliability, and the process of migrating to TiDB. Dive in now!00:00 Introduction and Welcome02:47 Defining TIDB: A Disputed SQL Database04:55 The Role of MySQL Compatibility in TIDB05:54 Primary Use Cases for TIDB09:38 Understanding the Data Ingestion Process in TIDB16:52 Understanding Indexing in TIDB23:01 Pushing Down Table Scans and Partial Aggregation24:39 Introduction to Columnary Extension: Flash24:54 Understanding Data Replication and Learner Nodes26:23 Ensuring Strong Consistency in Data27:12 Balancing Transactional and Analytical Use Cases27:57 Understanding Data Replication and Consistency Model28:42 Exploring Ty Flash Storage Layer28:54 Understanding High Concurrency Insert and Update32:09 Exploring the Read Path and Caching Mechanism37:50 Understanding the Importance of High Reliability43:01 Exploring Migration from Other Databases48:01 Comparing TiDB with Other Distributed SQL Databases52:21 Identifying Use Cases Where TiDB Might Not Be the Best ChoiceStay Curios! Keep Learning!
In this video I discuss Database tuning and Optimisation with Andy Pavlo, OtterTune. Andy is an Associate Professor with Indefinite Tenure of Databaseology in the Computer Science Department at Carnegie Mellon University. My research interest is in database management systems, specifically main memory systems, self-driving / autonomous architectures, transaction processing systems, and large-scale data analytics.00:00 Introduction and Welcome01:31 Understanding Database Optimization05:48 Understanding When Database Tuning is Needed08:45 Understanding Database Optimization Difficulties16:16 Understanding Default Settings in Databases22:35 Role of Machine Learning in Database Tuning22:38 Introduction to Ottertune28:36 Data Collection for Machine Learning Model35:25 Deployment and Data Collection Process38:03 Admitting the Limitations of Current Model38:53 Challenges in Predicting Performance Improvements39:28 The Importance of Data Collection Over Time39:52 Avoiding Weekend and Holiday Tuning40:05 Introducing New Features for Database Comparison42:09 Provisioning Recommendations and Performance Predictions43:03 The Importance of Telemetry in Understanding Database Performance44:01 Handling Dramatic Changes in Database Workloads44:48 Preparing for Predictable Traffic Spikes48:13 The Importance of Testing in Database Optimization53:33 The Future of Database Optimization55:50 Common Mistakes in Database Management01:09:15 The Future of Holistic Database TuningLinks:Ottertune: https://ottertune.com/Andy Pavlo: https://www.cs.cmu.edu/~pavlo/CMU youtube: https://www.youtube.com/@UCHnBsf2rH-K7pn09rb3qvkA Resources:CMU: https://15799.courses.cs.cmu.edu/spring2022/schedule.htmlOttertune blog: https://ottertune.com/blog===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Stay Curios! Keep Learning!
Deep Dive into DuckDB with CTO Mark RaasveldtDecode the insights of databases with Geek Narrator podcast. In this episode, host Kaivalya Apte converses with Mark Raasveldt, the CTO of DuckDB labs, discussing his journey from being a database enthusiast to creating DuckDB. They delve into how DuckDB, an analytical database, differs from other databases, the design decisions, its internal mechanisms, and much more. The episode also highlights the advantages of DuckDB in analytics, the motivation behind its ACID compliance, and how DuckDB handles ingestion, transaction isolation, mutations, and queries. Join in to learn how your data workloads can benefit from DuckDB.00:00 Introduction and Guest Introduction00:44 Guest's Journey into Databases03:40 The Birth of DuckDB04:30 Challenges with Existing Databases05:15 Technical Difficulties05:16 Why Existing Databases Fall Short for Data Scientists09:16 The Role of SQLite and Its Limitations13:59 Defining DuckDB16:48 Comparing DuckDB with Other Analytical Databases19:50 Deployment Models for DuckDB22:47 Data ingestion into DuckDB22:51 Data Ingestion in DuckDB30:24 How DuckDB Handles Updates and Mutations35:35 Understanding Column Granularity and Rewrites35:58 Implications of Compression on Data Updates36:38 Trade-offs in Row Group Size37:32 Benefits of Column Storage Model38:15 Row Groups and Parallelism39:02 Choosing Row Group Size: An Experimental Approach40:00 Handling Data Type Changes in Columns41:00 Internal Data Structures in DuckDB42:21 Reading Data: Point Lookups, Aggregations, and Joins47:22 Optimization for Full Table Scans53:49 Understanding ACID Compliance in DuckDB55:49 Multi-Version Concurrency Control (MVCC) in DuckDB59:50 Use Cases and Applications of DuckDB01:01:42 The Story Behind DuckDB's Name01:02:34 Future Vision for DuckDBReferences:DuckDB: https://duckdb.org/Mark's blog: https://mytherin.github.io/===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!Cheers,The GeekNarrator
In this episode we talk about ScyllaDB internals with Felipe Mendes.Chapters:0:00 ScyllaDB internals with Felipe Mendes07:51 Write Path - API to Storage11:40 What makes it faster than Cassandra?13:39 Optimisations: Sea Star, shard per core architecture15:49 Optimisations: No Garbage collection and Custom Cache Implementation18:15 Optimisations: Scheduling groups and IO priority classes20:07 Optimisations: IO scheduler22:55 Benefits of shard per core architecture30:16 Write path - Hows is a coordinator chosen?38:20 Read path39:27 Read path optimisations - Index Caching41:48 Shard vs Partition43:10 Shard per core architecture tradeoff44:03 Observability of DatabaseReferences:ScyllaDB architecture: https://opensource.docs.scylladb.com/stable/architecture/Sea star: https://seastar.io/ScyllaDB Caching: https://www.scylladb.com/2018/07/26/how-scylla-data-cache-works/Shard per core architecture: https://www.scylladb.com/product/technology/shard-per-core-architecture/Database performance at Scale: https://www.scylladb.com/2023/10/02/introducing-database-performance-at-scale-a-free-open-source-book/===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!Cheers,The GeekNarrator
In this episode I talk to Michael Hunger from Neo4j about Graph Database Internals (Neo4J)Chapters:0:00 Introduction and historical context20:51 Data Modelling25:16 Problem with SQL for Graph Model26:21 Cypher - Query Language28:23 Write Path31:36 Neo4J Storage Layer33:51 Graph API on top of Relational Model vs Native Graph Databases37:05 Create Node Relationships40:42 What makes Graph Database's performance better?46:00 Partitioning Strategy53:20 Read path59:27 Schema Migration01:04:41 Graph database use cases===============================================================================For discount on the below courses:Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.===============================================================================Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!Cheers,The GeekNarrator
In this episode I talk to Micah Wylde about why #Rust could be the best choice for writing distributed systems and how does it compare to #C++, #Java and #Go.Chapters:00:00 Introduction03:48 History of Systems Programming09:42 Is C++ coming back?13:31 Problems with C++16:24 Problems with Java25:18 Problems with Go31:21 Why did you choose Rust?35:19 What makes Rust better?41:49 Rust cannot save you from logical bugs44:02 Problems in the context of Stream Processing48:10 Challenges with Rust51:28 Learning Rust54:10 Future of Rust56:41 A SummaryBlog mentioned in the discussion: https://www.arroyo.dev/blog/rust-for-data-infraFor the courses mentioned use the following links:Coupon code: "geeknarrator"Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount.Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.Database internals series: https://youtu.be/yV_Zp0Mi3xsPopular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNStay Curios! Keep Learning!Cheers,The GeekNarrator
Hello Everyone, In this podcast I have invited John Crickett, who has been a Software Engineer since 27 years, having vast experience in variety of tech stacks. He is known for his newsletter "Coding Challenges" that helps developers build real world applications and becomming a better engineer.00:00 Introduction01:17 What made you start Coding Challenges?03:21 What made you start learning Rust?04:08 How should Software Engineers Prioritise learning? What should they learn? How would they know?12:20 How to become a better engineer?14:05 Knowing your passion? but how?17:43 Should LeetCode be part of interviews? When does (and not) it make sense ?25:39 System Design interviews29:38 Building as a community.More about Coding Challenges : https://codingchallenges.fyiJoin the discord server: https://discord.com/invite/zv4RKDcEKVConnect with John : https://www.linkedin.com/in/johncrickett/Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarratorIf you like this episode, please hit the like button and share it with your network.
Cheers,
The GeekNarrator
Hey Everyone, In this video I talk to Franck Pachot about internals of YugabyteDB. Franck has joined the show previously to talk about general database internals and its again a pleasure to host him and talk about DistributedSQL, YugabyteDB, ACID properties, PostgreSQL compatibility etc.Chapters:00:00 Introduction01:26 What does Cloud Native means?02:57 What is Distributed SQL?03:47 Is DistributedSQL also based on Sharding?05:44 What problem does DistributedSQL solves?07:32 Writes - Behind the scenes.10:59 Reads: Behind the scenes.17:01 BTrees vs LSM: How is the data written do disc? 25:02 Why RocksDB?29:52 How is data stored? Key Value?33:56 Transactions: Complexity, SQL vs NoSQL42:51 MVCC in YugabyteDB: How does it work?45:08 Default Transaction Isolation level in YugabyteDB51:57 Fault Tolerance & High Availability in Yugabyte56:48 Thoughts on Postgres Compatibility and Future of Distributed SQL01:03:53 Usecases not suitable for YugabyteDBPrevious videos:Database Internals: Part1: https://youtu.be/DiLA0Ri6RfY?si=ToGv9NwjdyDE4LHOPart2: https://youtu.be/IW4cpnpVg7E?si=ep2Yb-j_eaWxvRwcGeo Distributed Applications: https://youtu.be/JQfnMp0OeTA?si=Rf2Y36-gnpQl18yjPostgres Compatibility: https://youtu.be/2dtu_Ki9TQY?si=rcUk4tiBmlsFPYzYI hope you liked this episode, please hit the like button and subscribe to the channel for more. Popular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_dModern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsNFranck's Twitter and Linkedin: https://twitter.com/FranckPachot and https://www.linkedin.com/in/franckpachot/Connect and follow here: https://twitter.com/thegeeknarrator and https://www.linkedin.com/in/kaivalyaapte/Keep learning and growing.Cheers,The GeekNarrator
Hey Everyone, In this video I talk to Gilad Kleinmann, CEO and Co-Founder of epsio.io, about Epsio and how it helps companies to run queries faster and cheaper.
Chapters: 00:00 Introduction 02:09 Defining the problem statement 07:17 What is Epsio ? 09:58 How does Epsio change my architecture? 12:59 Use of CDC 14:05 Where is the query result stored ? (Foreign data wrappers) 15:40 What permissions does Epsio needs? 16:43 How does Epsio parses a query and creates a virtual table? 24:15 Consistency model of Epsio 27:48 How do I know if Epsio is suitable for me? 31:41 How does it compare with Caching? 35:59 What metrics are available with Epsio? 38:32 What other databases does Epsio support? (will support) 40:47 How to know more about Epsio? 41:37 Pricing model of Epsio
Read more about epsio: https://www.epsio.io/ Docs: https://docs.epsio.io/ Foreign data wrappers: https://wiki.postgresql.org/wiki/Foreign_data_wrappers
Other playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-
Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17
Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d
Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN
I hope you like this episode, please hit the like button if you did and subscribe to the channel if you haven't.
Cheers, The GeekNarrator
Hey Everyone,
In this episode I talk to Anjul Sahu, Founder of CloudRaft, about all things Kubernetes. There are many common questions that we try to answer. If you ever wondered if you should adopt Kubernetes as a company? or learn Kubernetes as a Developer? or why people fail adopting Kubernetes? Or even if Kubernetes is worth the hype? Then this episode will bring a ton of clarity.
Chapters 00:00 Introduction 02:37 History and Background 06:06 k8s use cases 08:24 k8s features 09:34 Some basics of k8s 11:54 Common problems people face with k8s 15:48 k8s - challenges 18:59 Should you use k8s? 24:02 Alternatives of k8s 26:54 Top reasons why people fail with k8s 31:38 As a dev - should I learn k8s? Is it good for my career?
Do give this episode a like and share it with your network. Also please subscribe to the channel for content like this.
Other playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-
Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17
Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d
Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN
Anjul Sahu: https://anjul.dev/ Cloudraft: https://www.cloudraft.io/
Our website: www.geeknarrator.com
References: Kubernetes course: https://kodekloud.com/courses/kubernetes-for-the-absolute-beginners-hands-on/ Kubernetes Hard Way: https://github.com/kelseyhightower/kubernetes-the-hard-way Kind: https://kind.sigs.k8s.io/ k3s: https://k3s.io/ k0s: https://k0sproject.io/
Cheers, The GeekNarrator
In this video I talk to David Jacot who works as a Staff Software Engineer at @Confluent and has been a long time Kafka user, committer and PMC member. We covered how Kafka works internally in great depth.
We use Kafka for various use cases and it works great, but going one level below the abstraction and truly understanding the protocols, techniques and algorithms used is a fun ride.
Chapters: 00:00 Kafka Internals with David Jacot 03:33 Defining Kafka 05:16 Kafka Architecture(s) 11:39 Write Path - Producer sending data 18:35 How does replication work? 25:47 How do we track replication progress? 30:42 Failure Modes: Leader fails 38:18 Consumers: Push vs Pull 40:54 Consumers: How does fetch works? 49:03 Consuming number of bytes vs records 50:50 Optimising consumption 01:00:21 Offset management and choosing partitions 01:09:10 Ending notes
I hope you like this episode and more importantly you learnt some amazing techniques Kafka uses to ensure durability, low latency, simplicity and scalability in its architecture.
Do give this episode a like and share it with your network. Also please subscribe to the channel for content like this.
Other playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-
Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17
Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d
Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN
Other episodes: KsqlDB: https://youtu.be/2yE86P6uD_0 Exactly once semantics: https://youtu.be/twgbAL_EaQw
David's Linkedin: https://www.linkedin.com/in/davidjacot/
our website: www.geeknarrator.com
Cheers, The GeekNarrator
In this episode I talk to @gkcs on the Practical system design for Senior Engineers. Gaurav brings in a ton of experience in Competitive Programming, Teaching System Design and working on real world problems at companies like @Uber and Directi. Currently he is running his startup called @interviewready_ that helps folks perform better at Interviews. Do check him out if you haven't.
I truly believe System Design in the real world requires ton of other skills along with great technical skills. It becomes even more critical at Senior levels. So in this episode we discuss our view and experience on the same while providing some concrete examples from our real world experience.
I hope this episode brings in some clarity and adds some value to folks at all levels. At junior level, you would know what to expect and how to approach, At senior level, you would know what you might be missing or can do more to get to the next level.
Chapters: 00:00 System Design as Senior Engineers - Things that no one talks about about. 04:11 Fundamentals 08:32 Do you get to choose the Tech stack? 16:47 Thinking about the future while being fast 21:18 Real world vs Interviews 23:45 Importance of Communication Skills 30:50 IC does not mean Isolated Contributor 35:20 Senior Engineering is a journey not a Certificate. 39:17 Volatility, Uncertainity, Complexity, Ambiguity 42:18 Importance of Decision Making 47:49 Reducing the blast radius 53:10 Leading by example
Give it a watch and share with your network. Hit the like button and don't forget to subscribe to the channel.
Cheers, The GeekNarrator
Hey Everyone,
In the 43rd episode I speak with Tim Berglund on Realtime Analytics with Apache Pinot.
Chapters:
00:00 Introduction 01:22 What do we mean by analytics and realtime analytics? 05:35 Can we define realtime in millis, seconds or minutes? 08:54 What is the fundamental difference between traditional analytics systems and Apache Pinot? 12:19 Was Kafka one of the reasons Apache Pinot could reach its full potential? 16:50 E-commerce Application example - How do I get my data in? 20:07 How is data stored (structured) on the disk? 23:31 Are joins available in Apache Pinot? 26:07 Joins vs pre-computing at ingestion 27:15 How is historical data ingested into Apache Pinot? 28:14 Types of indexes available in Apache Pinot 35:42 Do indexes cause write amplification? Is that a problem in Apache Pinot? 40:02 Point lookups in Apache Pinot 42:54 Anamoly Detection 45:51 Coming up in Apache Pinot
Links: StarTree https://startree.ai/ Apache Pinot: https://pinot.apache.org/ Joins in Pinot: https://startree.ai/blog/apache-pinot... Apache Pinot Indexes: https://docs.pinot.apache.org/basics/...
Other playlists: Distributed systems: • Distributed Syste...
Modern Databases: • Modern Databases
Serverless Architecture: • Serverless Archit...
Software Engineering: • Software Engineering
I hope you like the episode. Like, share and subscribe to the channel.
Cheers, The GeekNarrator
In this video I talk to Philip Fried from Estuary about Batch vs Realtime Stream Processing. Philip brings a ton of experience in the world of data processing and has shared some of the best practices in implementing these systems. We dive deep into the world of data processing, covering batch and streaming systems, their challenges, tradeoffs and use cases.
Chapters: 00:00 Batch vs Realtime Stream Processing 03:25 What is Batch and Reatlime processing? 18:29 How does Batch and Realtime compare in terms of Latency and Throughput? 27:24 Where is the cost saving coming from? Compute?Storage? or Network? 31:38 Moving from Batch to Stream processing 37:50 How is Idempotency implemented in Streaming systems? 48:50 How do we approach Schema evolution in Batch and Streaming systems? 57:16 Summary - key points to keep in mind
Do checkout Estuary if you deal with a ton of data, and don't want to deal with the painful operations, infrastructure management, schema migrations etc and only want to focus on building highly scalable and resilient applications.
References: Estuary: https://estuary.dev/ Flow documentation: https://docs.estuary.dev
If you like this video please hit the like button, share it with your network (whoever works with a ton of data) and subscribe to the channel.
Feel free to watch related episodes in the playlist: • Distributed Syste...
Modern Databases: • Modern Databases
Software Engineering: • Software Engineering
Distributed Systems: • Distributed Systems
Cheers, The GeekNarrator
Hey Everyone,
In this video I talk to Patrick McFadin from DataStax. We uncovered the new features in Cassandra 5.0 and discussed how ACID transactions are achieved in the new version. This is a deep dive into the features of Cassandra, consensus protocols and how Accord is different as compared to Paxos, RAFT, Spanner and Calvin.
Chapters: Cassandra 5.0 - ACID transactions and Vector Search 00:00 Introduction 01:45 List of features in the new Cassandra 04:51 Who needs ACID properties? 07:20 Why didn't Cassandra have ACID properties so far? 10:35 Why is Accord consensus protocol well suited for Cassandra? 16:40 Lets take a gaming example to see how Transactions work 21:55 Whats happening behind the scenes in a Transactions? 27:44 What happens when there are failures? 33:41 How is upgrade to the new version going to look like? 35:48 How is the latency impacted because of transactions? 40:23 What was missing in lightweight transactions? 42:24 Vector Search - What is it? How does it work?
Previous episode on Cassandra: • Apache Cassandra ...
Other playlists to watch:
Distributed systems and Databases: • Distributed Syste...
Software Engineering: • Software Engineering
Distributed systems: • Distributed Systems
Modern Databases: • Modern Databases
Patrick's Linkedin: https://www.linkedin.com/in/patrick-m... Astra: astra.datastax.com Cassandra: https://cassandra.apache.org/_/index....
I hope you liked this episode. If you did, please hit the like button, share it with your network and subscribe to the channel.
Cheers, The GeekNarrator
Hey Everyone,
In this episode I discuss Twisp's journey with Mike Parsons, CTO and Co-Founder of Twisp. We have discussed how they ended up building a cloud native distributed ledger database on top of DynamoDB, why they chose DynamoDB? what were the challenges they faced and many highly technical and very interesting topics.
Chapters: 00:00 Introduction - Twisp - CTO and Co-Founder Mike Parsons 03:01 What do we mean by a Distributed Ledger Database? 07:13 Is it similar to Blockchain? 08:32 Challenges faced with Relational Database setup 19:39 Why DynamoDB was the right choice? 24:40 What was the scale you were building for? 26:05 Who uses a Ledger database? 29:30 What are the basic primtives in a ledger database? 33:55 Is every primitive Immutable? How is it stored? 38:07 Impact of Immutability on Data storage 41:33 How does idempotency work in Twisp? 44:53 How did you provide ACID guarantees on top of DynamoDB? 52:30 How do you ensure correctness? 57:13 Interesting real world usecases where Twisp is used. 01:02:33 Are you planning to build even higher level primitives?
References: Clocks: https://cse.buffalo.edu/tech-reports/... and http://rystsov.info/2018/10/01/tso.html TicToc: https://people.csail.mit.edu/sanchez/... Hekaton Paper: https://www.microsoft.com/en-us/resea... Hermitage Test suite: https://github.com/ept/hermitage Online Async Schema Change in F1 : https://static.googleusercontent.com/... Designing Data intensive Applications: https://www.oreilly.com/library/view/... P Programming language: https://github.com/p-org/P Twisp: https://www.twisp.com/ AWS Dynamo DB: https://aws.amazon.com/pm/dynamodb/?t...
Connect with Mike: https://www.linkedin.com/in/parsnips/
Follow me: Twitter: https://twitter.com/thegeeknarrator Linkedin: https://www.linkedin.com/in/kaivalyaa...
Watch other playlists: Databases: • Modern Databases Distributed systems: • Distributed Systems Software Engineering: • Software Engineering Sofk skills: • Soft Skills
Like, share and subscribe to the channel folks. See you in the other episode.
Cheers, The GeekNarrator
In this episode I talk to Yan Cui, who is an AWS Serverless Hero, all about Serverless technologies.
Chapters: 00:00 Serverless Architecture with Yan Cui 01:58 What do we mean by Serverless Architecture? 05:42 What is the core problem Serverless solves? 11:06 Do we need to think differently to be able to use Serverless? 15:27 What is the difference between serverless and managed services? 19:17 Is Vendor Lock-in really a problem? 27:42 Multicloud - Is it really worth it? 33:46 Is ColdStart a real problem? What kind of apps get impacted? 43:25 Monitoring serverless applications 48:22 Usecases when serverless may not be the best solution 54:27 Future of serverless 57:31 How should a developer learn about serverless?
I hope you enjoy the discussion and learn from it. Please hit the like button, share it with your network and also subscribe to the channel.
References: Yan Cui - https://theburningmonk.com Courses - https://productionreadyserverless.com/ Corey Quinn on MultiCloud - • Corey Quinn: The ... Linkedin Yan - https://www.linkedin.com/in/theburnin... Twitter Yan - https://twitter.com/theburningmonk
Other playlists to watch: Distributed Systems and Databases - • Distributed Syste...
Software Engineering - • Software Engineering
Distributed systems practices - • Distributed Systems
Cheers, The GeekNarrator
It was an honour to host Dr. Venkat Subramaniam on the podcast.
Dr. Venkat Subramaniam is an award-winning author, founder of Agile Developer, Inc., and an instructional professor at the University of Houston. He has mentored tens of thousands of software developers in the US, Canada, Europe, and Asia, and is a regularly-invited speaker at several international conferences. Venkat helps his clients effectively apply and succeed with agile practices on their software projects.
In this episode, Dr. Venkat has shared his experience on becoming productive, managing time effectively, learning and Software Engineering. There is so much to learn from the way Venkat approaches learning and keeping up with the pace. No wonder, how he is able to do so many different things.
Chapters: 00:00:00 Introduction into Venkat's Journey 00:04:23 How is Venkat able to do so many things? 00:15:23 Saying NO and prioritisation 00:44:42 Keeping up with the pace and learn 00:53:55 Engineering practices every engineer must follow 01:14:40 Ending notes, resources etc..
This episode is full of stories, experiences and learnings from Dr. Venkat's vast experience. The analogies he has shared are amazing.
I hope you like this discussion and also learn a ton of things that matter.
Venkat Subramaniam: https://twitter.com/venkat_s Agile Developer: https://agiledeveloper.com/
If you like this discussion, please like, share, subscribe and press the bell icon. Keep supporting and more amazing discussions coming your way.
Cheers, The GeekNarrator
Hey Everyone,
In this video I was joined by Daniel Ciocîrlan from https://rockthejvm.com and we discussed Functional Programming concepts in great detail.
Chapters: 00:00 Introduction 03:11 What is Functional Programming? How can I start learning? 11:40 Object Oriented Programming and Functional Programming 16:08 How should a beginner approach learning Functional Programming? 21:58 Handling Side Effects (Pure functions) 28:04 Immutability and Recursion 35:59 Concurrency Models (Futures, Co-routines, Actors) 42:47 Project Loom 46:42 Scala vs Kotlin 49:01 Current Limitations and Future
RockTheJVM : https://rockthejvm.com
I hope you like the episode, please like, share and subscribe for more videos.
Cheers, The GeekNarrator
Hello Everyone,
Time Management is a critical skill for Software Engineers and most of us struggle with it everyday. So I and Arpit decided to discuss and share our learnings and experience on time management for Software Engineers.
We have discussed about: - How to balance Context Switching? - How to say NO? - How and when to ask for help? - Prioritisation of tasks - Visibility of your work - Documentation and helping others - Getting better at Time Management
I hope this casual discussion will help you figure out gaps in your approach and help you get better at managing your time well :)
If you like the episode, please like, share and subscribe to the channel. Also don't forget to press the bell icon to get notified on any new video added to the series.
Arpit Bhayani : https://www.linkedin.com/in/arpitbhay... and https://arpitbhayani.me/
softwareengineering #timemanagement
Cheers, The GeekNarrator
Hey Everyone,
In this episode I was joined by Francesco Tisiot who is a Developer Advocate @Aiven. He has shared some interesting insights into how you can build robust data platforms. The approach we have discussed is also described here: https://aiven.io/blog/a-soft-methodology-to-define-robust-data-platforms
References:
SOFT : https://aiven.io/blog/a-soft-methodology-to-define-robust-data-platforms
Aiven: https://aiven.io/Francesco
Tisiot https://twitter.com/FTisiot and https://www.linkedin.com/in/francescotisiot/
I hope you like the episode. Please subscribe to the channel and share with your network.
Cheers, The GeekNarrator
Hey Everyone,
In this episode I and Matthias talk about KsqlDb. We have covered the topic in great depth talking about its history, architecture, different concepts, use cases, limitations, comparison to Kafka Streams and so on.
References:
ksqlDB - https://ksqldb.io/
exactly once semantics podcast: https://youtu.be/twgbAL_EaQw
Matthias Sax: https://twitter.com/MatthiasJSax and https://www.linkedin.com/in/mjsax/
Cheers, The GeekNarrator
Hey Everyone,
In this episode I have Denis Magda from YugabyteDb with me.
We have explained how to build Geo-Distributed Applications in depth.
I am sure this will help you make your applications more scalable, highly resilient, compliant and highly performant. If you liked the video please subscribe to the channel and give me a like.
Further reading:
https://www.yugabyte.com/blog/what-is...
https://cloud.google.com/load-balancing
AWS outage: https://aws.amazon.com/message/41926/
Connect with Denis: https://www.linkedin.com/in/dmagda/ and https://twitter.com/denismagda
Connect with me: https://www.linkedin.com/in/kaivalya-...
Cheers, The GeekNarrator
Hey Everyone,
In this episode Gunnar Morling, who is the former project lead for Debezium and has recently started working for Decodable, has joined us to talk everything about Change Data Capture with Debezium.
We discussed basic and advance topics and features of Debezium. If you are doing dual writes, Debezium could be a great tool for you.
I hope you like the episode.
Cheers,
The GeekNarrator
Hey Everyone,
In this episode I am joined by Richard Startin who works with DataDog on Java profiling. Richard has an amazing experience with profiling and benchmarking Java applications and he has explained us what profiling means, how one can get started, different concepts in the world of profiling, tools available, limitations and the future of profiling.
I hope you like the episode.
Cheers,
The GeekNarrator
Hey Everyone,
In this episode I am joined by Matthias Sax, who works with Confluent to build the amazing world of Kafka.
We have discussed in real depths of Kafka Streams and how Exactly once semantics is implemented.
This episode will give you all the details you need to understand how beautifully Kafka imeplements EOS.
I hope you like the episode.
Cheers,
The GeekNarrator
Hey Everyone,
In this episode Alex Debrie joined us to dive deep into the world of one of the most popular database, i.e. AWS DynamoDB.
This is a true masterclass where we go deep into different concepts, architecture and design choices of DynamoDB. We talked about CAP and PACELC, dynamo streams and many other advanced topics as well.
I hope you like this masterclass.
Cheers,
The GeekNarrator
Hey Everyone,
In this episode Paul Christmann has joined us to demystify the world of Non Functional Requirements.
We started with very basics and discussed on top of it, about how one can approach defining Non Functional requirements for a system.
I hope you like the discussion.
Cheers,
The GeekNarrator
Hey Everyone,
In this episode I invited Arpit Bhayani (#asliengineering) to talk about the famous Bitcoin whitepaper.
We have discussed in depth the architecture, design choices, challenges and suggested solutions.
I hope you like the episode.
Cheers,
The GeekNarrator
Hey Everyone,
This is the part2 of our episode with Franck Pachot.
In this episode I talk to Franck Pachot from YugabyteDB who has vast experience in the world of Databases.
We have talked about how modern databases work internally. BTrees, LSM trees and various design choices.
Cheers,
The GeekNarrator
Hey Everyone,
This is the part1 of our episode with Franck Pachot.
In this episode I talk to Franck Pachot from YugabyteDB who has vast experience in the world of Databases.
We have talked about how modern databases work internally. BTrees, LSM trees and various design choices.
Cheers,
The GeekNarrator
Hey everyone,
This is the part-2 of our episode with Tim Berglund. We have covered some advanced topics on Kafka, Kafka Streams and Apache Pinot.
I hope you like the discussion.
Cheers,
The GeekNarrator
Hey Everyone,
In this episode I talked to Tim Berglund about his vast experience with Kafka, realtime analytics and Apache Pinot.
I hope you like the episode. Do watch the part-2.
Cheers,
The GeekNarrator
In this podcast I have invited Neha Pawar, who is one of the Founding Engineers are StarTree (the company powering Apache Pinot). We talked about how StarTree has implemented Tiered storage and how it differs from other available implementations. Note: Currently tiered storage is available only in StarTree’s Pinot and not available in the open source version. But its only about time.
Chapters: 00:00 Introduction 03:28 What does Tiered Storage mean? 05:51 How many tiers are typically supported? 07:30 Is it mainly about Cost Optimisation? How do I compare the cost savings vs performance hit? 15:41 What is mmap and how does it help? 16:45 How do I implement/approach Tiered Storage? What are the challenges? 23:00 What is Apache Pinot? When we say low latency, how low it is? 25:00 How is it implemented in StarTree (Apache Pinot)? 36:45 What happens when I query for more number of (or all) columns? How is that optimised? 47:10 What are the failure modes? 50:15 How can we test and validate Tiered Storage as a feature? 54:30 How would bloom filter false positives affect performance and correctness? 56:15 Can I move back my data from Cold storage to Hot Storage? 57:45 What other cloud storage services are supported other than S3? 58:35 What is the future of Tiered Storage?
In this episode I invited Curtis Einsmann to join us and share his knowledge and experience on how to do code reviews effectively.
Curtis has created a course called "Master the Code Review" which has some great testimonials and amazing feedback.
We have discussed various suggestions on making code reviews more effective for teams and individuals.
Chapters:
00:00 Introduction
10:36 How should teams appoach Code Reviews?
18:19 How should individuals approach Code Reviews?
25:10 Does Code Reviews slow down the development process?
28:15 Common mistakes done by Authors and Reviewers
35:20 Are nitpicks useful?
37:45 Where does Pair Programming fit into this process?
40:50 Practical examples
Links: Curtis Einsmann Twitter: https://twitter.com/curtiseinsmann
Curtis Einsmann Linkedin: https://www.linkedin.com/in/curtisein...
Master the code review course: https://curtiseinsmann.gumroad.com/l/...
Here is the team license: https://curtiseinsmann.gumroad.com/l/...
Hope you get amazing insights from the episode and make your code review process better.
If you like the video please don't forget to hit the like button and share your feedback in the comments section.
Also please press the bell icon and subscribe to The GeekNarrator for more content on Software Engineering.
Cheers,
The GeekNarrator
Hey Everyone,
In this episode I had a fun conversation with Pratik Pandey who is currently working as a Senior Software Engineer at Booking.com and has vast experience in the Software Industry specialising in designing systems for large scale workloads.
We have covered System Design in Depth and destroyed the myths around System Design in general.
Agenda:
00:00 Introduction
04:30 System Design Framework
19:33 Requirements Gathering
28:00 Capacity Planning and Estimations
31:50 Driving System design with Capacity planning
42:21 Choosing the right tech stack
58:45 Real world System Design vs System Design Interviews
01:13:00 Must have components in a System Design Interviews
01:17:32 How do I prepare for System Design?
01:22:45 How do I gain experience in System Design?
01:26:00 Resources to prepare for System Design
I hope you like our discussion and learn more about System Design interviews and how it differs with the real world system design. Do hit the like button and subscribe to the channel if you like such content.
Cheers, The GeekNarrator
Hey Everyone,
In this episode I invited Patrick McFadin who is an expert in the world of Cassandra and Data Modelling.
Patrick currently works for DataStax as a VP Of Dev Rel. Patrick has given several techtalks on Cassandra and the ecosystem around it. We have covered the architecture of Cassandra in depth.
Heres what we have covered:
00:00 Introduction
04:00 History of Cassandra
07:18 Patrick Apache Cassandra?
14:30 How writes work in Cassandra?
21:30 How many copies are written on a single write?
25:44 How does replication work?
32:00 How do reads work? (Read consistency levels)
39:00 Why is Allow Filtering not recommended?
43:00 Data Modelling in Cassandra
50:45 Modeling a Chat Application
01:05:00 How does CAP theorem fits Cassandra?
01:07:06 New features in Cassandra?
References:
Patrick McFadin: https://www.linkedin.com/in/patrick-m...
Kaivalya Apte: https://www.linkedin.com/in/kaivalya-...
Astra: astra.datastax.com
Cassandra: https://cassandra.apache.org/_/index....
Webinar on Data Modeling: https://www.youtube.com/watch?v=4D39w...
Playlist on Distributed Systems and Databases: https://www.youtube.com/playlist?list...
I hope you enjoyed our discussion and learned from it. Please like, share and subscribe to the channel and keep supporting.
Cheers, The GeekNarrator
In this episode I talk to Luis Mineiro, Senior Director at Delivery Hero about Site Reliability Engineering.
We have busted a lot of myths around Site Reliability Engineering as a concept.
We have talked about the following:
00:00 Introduction
02:00 What is SRE? What is NOT SRE?
10:00 DevOps vs SRE vs Developers
13:10 When do we need a dedicated SRE team?
20:00 Traits of a Strong SRE culture
29:00 Observability vs Monitoring vs Alerting
41:40 Adding Observability to a Sandiwch Shop
53:00 How to add distributed tracing to the Sandwich shop?
57:10 How do we define SLO's for the Sandwich shop?
01:01:00 How to I define timeouts between services?
01:05:00 How do I determine cost of adding 9's to my SLOs?
01:15:00 How do I transition from a Developer to SRE?
References: SRE Books from Google: https://sre.google/books
Luis Linkedin: https://www.linkedin.com/in/lmineiro/
The GeekNarrator Page: https://www.linkedin.com/company/the-...
Kaivalya Apte Linkedin: https://www.linkedin.com/in/kaivalya-...
I hope you like the video and learn a lot about SRE.
Cheers, The GeekNarrator
In this episode I speak with Arasu Shankher, who is working as a CTO at KBX Digital. Arasu has a vast experience as an Engineer, Manager and Technology Leader. He has worked for companies like Microsoft, Google and Amazon and has built many high performance teams.
We cover extremely important topics for Software Engineers and Managers at all levels.
Chapters:
00:00 Introduction
02:30 Traits of a Successful IC and Manager
07:11 Depth for ICs?
09:05 Do you miss being an IC? what is it that you miss?
10:10 How do you stay up to date? How do you keep learning?
16:30 What are some resources that you use to learn?
21:20 Productivity and Time Management
28:20 Dealing with toxic people?
37:10 How has the Tech space changed over the years?
41:40 What is the next big thing in the Tech Space?
48:00 Buy vs Build? How do we make a good decision?
54:10 How to manage Tech Debt effectively?
57:00 Attaching risk level to Tech Debt. Is it a good model?
59:20 Strategies to mitigate technical risks
01:01:20 One tip for Junior, Mid and Senior Engineers and Managers.
I hope you like this episode. Please hit the like button if you did and subscribe to the channel for more content like this.
Cheers, The GeekNarrator
Hey Everyone,
In this episode I am talking to Mayank Shrivastava who has vast experience into building and maintaining high scale distributed systems. He was in the team that originally built Apache Pinot at Linkedin and is now working at StarTree as the Head of Core Data Engineering.
He has shared some amazing insights from his experience and there is a lot to learn from our discussion.
We discuss about the following:
00:00 Introduction
04:20 Practices to follow while designing and developing Distributed Systems
05:47 What do we mean by Solid Scalable Design? How do we approach that?
09:00 Safety Nets for developing Distributed systems
10:21 When is the right time to do performance benchmarking?
17:00 What is release certification?
21:00 Deploying to Production
24:45 Example when Canary Deployment might not be a good strategy?
26:00 Example when Canary Deployment a good strategy?
27:30 Post Deployment - how do we observe our system?
33:30 How do we avoid on-call(alerting) noise?
42:00 Maintaining a Large scale Distributed system
47:15 Scaling up/down for stateful systems
51:30 Handling Failures in Production (Disaster Recovery)
01:00:30 Runbooks - How do we keep them updated?
References:
The GeekNarrator Linkedin page: https://www.linkedin.com/company/86276626
Kaivalya Apte: https://www.linkedin.com/in/kaivalya-apte-2217221a/
Geeknarrator website: www.geeknarrator.com
Mayank Shrivastava: https://www.linkedin.com/in/mayankshriv/
StarTree: https://www.startree.ai/
Apache Pinot: https://pinot.apache.org/
Hope you enjoy the discussion and learn from it. Please hit the like button if you liked my discussion with Mayank and please subscribe to the channel for more content like this.
Cheers, The GeekNarrator
Hey Everyone, Welcome to the API design masterclass with our guest Deepak Patil, who works as a Principal Architect at iCertis and has decades of experience building and architecting mission critical experience. This is the Part-2 of the masterclass and we are going to continue where we left and talk about more practical stuff. 00:00 Sync, Async, Bulk, Composite APIs 10:16 Timeouts, Retries and Idempotency 17:08 Structure of an API 21:00 Versioning 26:45 How to choose REST vs GraphQL vs gRPC 36:30 API Lifecycle and Governance 38:50 Security, Scalability, Availability, Fault Tolerance, performance and Monitoring I hope you enjoyed the discussion in the two parts series and now know a lot about API Design. Please like, share and subscribe to the channel for more content like these. Cheers, The GeekNarrator
Hey Everyone,
Welcome to the API design masterclass with our guest Deepak Patil, who works as a Principal Architect at iCertis (https://www.linkedin.com/in/01patilde...) and has decades of experience building and architecting mission critical experience.
We talk about:
Basic API Discussion
00:00 Introduction
03:50 What is an API?
07:10 Why do we want to implement APIs?
10:22 How do we plan to execute the API program?
14:45 Practical API Design
24:40 Dominant Styles of API Entity vs Behaviour oriented
31:50 Fine Grained vs Coarse grained APIs
43:45 Pagination
I hope you enjoyed the discussion in the part-1, please do listen to part-2 for more advanced topics and more practical examples.
Cheers,
The GeekNarrator
Hey All,
In this episode, we have Maheep Bhalla with us, who has been a Senior Leader in the Product Management space for many years and has worked at amazing Tech companies who have built successful products over the years.
We talked about Product Management in general and everything around it.
We start with defining what is Product Management and start diving deep into answering all the confusing things in the Product Management world. Are you aspiring to be a successful product manager? Then this Podcast is for you.
Are you a developer who thinks how product management relates to you? Then this Podcast is for you.
If you are a product manager and want to reach that next level, then this podcast is for you.
And last but not the least, if you love Product Management like our guest Maheep, then this podcast is for you.
Some key topics we discussed:
00:00 A Joke on Dev and PM relationship
03:01 What is Product Management?
07:00 How to transition into a Product Management Role?
12:40 Are we solving the problem for majority of the customers?
18:45 Why it might be sometimes important to solve for the 1% customers?
19:10 How Technical should Product Managers be?
30:07 How can developers work effectively with PMs?
43:00 Asking "Why" style questions is also very important.
45:10 How to focus on the right problem to solve?
53:10 How to measure product's success? and Examples of Great Software Products?
There is so much to talk about in this space, and for that we will be collaborating more in the future to bring you more insights into Product Management. I hope you enjoy the podcast, please like, share and subscribe and let us know how do you like this podcast. Also please feel free to share more questions that you need answers for.
Cheers, The GeekNarrator
Hey 👋 ,
In this podcast we are talking about Distributed Stream Processing at High scale with Maximilian Michels (https://www.linkedin.com/in/maximilianmichels/) , who is an expert in Data intensive applications and an open source maintainer for Apache Flink, Beam and other cool technologies.
We have covered an introduction on Stream Processing at high scale:
-
What is stream processing?
-
How does it compare with Batch Processing and also event processing?
-
What tools are available to perform stream processing?
-
Use cases where we need Stream processing.
-
Flink vs Kafka and some factors to decide between the two. https://docs.confluent.io/platform/current/streams/index.html and https://flink.apache.org/
-
Why Exactly once processing is a Hard problem and how Kafka and Flink approach that problem? Is it similar to exactly once delivery?
-
Do we really need exactly once?
-
Why data locality matters?
-
Should we make network calls from the stream processor OR have a join instead? How do decide? And so many other important considerations based on real production experience in operating stream processing pipelines at a very high scale.
I hope you like the podcast. Please like, share and subscribe 🙏 The GeekNarrator channel needs your love and support 😃
Regards, The GeekNarrator
In this video I (Kaivalya Apte) talk to Max Linden, CEO and Co-Founder of Lemon.markets, where he shares his insights into entrepreneurship, product mindset, setting goals and making HUGE impact. -
-
What is lemon.markets?
-
Why do we need a trading API like lemon.markets?
-
What it takes to build a trading API like lemon.markets?
-
Why is it named lemon.markets?
-
How to be more productive?
-
Setting stretch goals
-
Getting Customer Feedback and solving problems for customers.
-
Biggest lesson from Max's entrepreneurship journey
To try lemon markets - https://www.lemon.markets
How to use Gmail more effectively https://klinger.io/posts/dont-drown-i...
Follow The GeekNarrator below: Anchor: https://anchor.fm/kaivalya-apte
Spotify: https://open.spotify.com/show/2peu9Op...
Apple: https://podcasts.apple.com/us/podcast...
Google: https://podcasts.google.com/feed/aHR0...
Please Like, Share and Subscribe to the GeekNarrator.
Cheers, The GeekNarrator
Hey 👋 People,
In this conversation we talk to Ananth about his experience with Microservices and Monolithic architectures. We also look at various examples of a good/bad monolith, good/bad Microservices.
We also talk about our personal experience with different architectural styles and share pros/cons.
Org structure must align with the way we develop software and we briefly touch about how business leadership and technical leadership should align on Software Architecture.
Having the right tooling in place is critical for Microservices architecture. We talk about the challenges of Microservices architecture when such tooling is missing.
We end the episode with talking about SLOs/SLIs and SLAs, and how a customer focused approach can help us.
Please like, share and follow the podcast.
Cheers,
The GeekNarrator
Get more insights into the company before making a decision. Interview is a great time to ask questions to get an insider’s view. Not sure what questions to ask? In this episode I share some tips based on my experience on both sides
In this episode I have shared some techniques that I have found extremely useful to be more productive and do effective work. Let me know how it works for you. Thank you for listening 😀
In this episode I talk about why it is important for software engineers to follow different approaches to learning depending on the use case. I also share some examples from my personal experience.
An approach that I found very useful to propose technical ideas with lot of nuances to a team and take it to the end successfully.
We share some tips with Jack (imaginary friend) to be able to select projects that have high impact and visibility to accelerate his career growth.
In this episode I share some tips to follow while onboarding to new teams. It is a structured approach that has helped me to onboard so many different teams. I am excited for you to follow this approach and share experience. Enjoy
In this episode we look at Bob’s journey and why he ended up with a bad performance review. We also recommend some tips for him to get better and avoid such situations in future.
Talks about key things to become a strong software engineer. Specially for engineers that are just starting their journey or engineers who just want take a step back and validate their approach and skills. Enjoy