Summary

One of the critical components for modern data infrastructure is a scalable and reliable messaging system. Publish-subscribe systems have been popular for many years, and recently stream-oriented systems such as Kafka have been rising in prominence. In this episode, Rajan Dhabalia and Matteo Merli discuss the work they have done on Pulsar, which supports both options, in addition to being globally scalable and fast. They explain how Pulsar is architected, how to scale it, and how it fits into your existing infrastructure.

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you start by explaining what Pulsar is and what the original inspiration for the project was?
• What have been some of the most challenging aspects of building and promoting Pulsar?
• For someone who wants to run Pulsar, what are the infrastructure and network requirements that they should be considering and what is involved in deploying the various components?
• What are the scaling factors for Pulsar and what aspects of deployment and administration should users pay special attention to?
• What projects or services do you consider to be competitors to Pulsar and what makes it stand out in comparison?
• The documentation mentions that there is an API layer that provides drop-in compatibility with Kafka. Does that extend to also supporting some of the plugins that have developed on top of Kafka?
• One of the popular aspects of Kafka is the persistence of the message log, so I’m curious how Pulsar manages long-term storage and reprocessing of messages that have already been acknowledged?
• When is Pulsar the wrong tool to use?
• What are some of the improvements or new features that you have planned for the future of Pulsar?

Contact Info

• Matteo
  • merlimat on GitHub
  • @merlimat on Twitter
• Rajan
  • @dhabaliaraj on Twitter
  • rhabalia on GitHub

This podcast was originally published on Data Engineer Podcast.