Klaviyo Data Science Podcast: Recent Episodes
Michael Lawson
This podcast is intended for all audiences who love data science--veterans and newcomers alike, from any field, we’re all here to learn and grow our data science skills. New episodes monthly. Learn more about Klaviyo at www.klaviyo.com!
Internationalizing your product
There are many aspects of product growth — reaching new heights for peak volume, reaching new levels of sustained daily volume, growing your feature set and the complexity of your code based, and many others. Dealing with growth in an intelligent and forward-looking way is never easy, but this month we deal with a type of growth that presents its own unique set of challenges: international growth, i.e. expanding the range of countries and languages your products are natively available in.
This month, we talked with multiple members of the internationalization effort here at Klaviyo, from teams across our organization. You’ll hear about:
- How strings can be far more complex than they seem
- Why changing languages doesn’t just mean changing words
- Where in the world your assumptions about language may break
For the full show notes, including who's who, see the Medium writeup.
How real marketers use data science
We spend a lot of time on this podcast talking about how to build data science solutions. Implicit in many of those conversations is perhaps the most fundamental truth of product design and development: we build data science solutions because people use them. We aren’t doing this just for fun — the reason we spend so much time, effort, and energy to refine our solutions is that it actually matters to real people.
This month, we talk to some of those people. In particular, we sat down with two members of the team at Made In Cookware (http://madeincookware.com/) to discuss what makes their business unique, how they approach understanding and marketing to their customers, and how data science and AI help them do all of that. You’ll hear about:
- What kitchen knives can teach you about product design and development
- Which type of pan you should use to cook a steak, and how that can help you understand customer segmentation
- How AI saves real marketers real time while also giving them better results
About Made In
Made In Cookware (Made In) is a premium cookware brand based in Austin, TX. Founded in 2017 but born of a 4th-generation, family-owned kitchen supply business, Made In creates best-in-class cookware developed in partnership with the world’s finest chefs and foremost craftsmen. Today, you’ll find Made In products in more than 2,000 restaurants, in the hands of James Beard Award-winning chefs at Michelin-starred restaurants across the country, and in the kitchens of home cooks everywhere. Made In products have garnered over 100,000 5-star reviews, and the company was named one of Inc. Magazine’s best workplaces and Newsweek’s best online shops of 2024.
For the full show notes, including who's who, see the Medium writeup.
An Introduction to ML Ops
Building data science products requires many things we’ve discussed on this podcast before: insight, customer empathy, strategic thinking, flexibility, and a whole lot of determination. But it requires one more thing we haven’t talked about nearly as much: a stable, performant, and easy-to-use foundation. Setting up that foundation is the chief goal of the field of machine learning operations, aka ML Ops.
This month on the Klaviyo Data Science Podcast, we give a brief but thorough introduction to the field of ML Ops. You’ll hear about:
- How ML Ops is different from the similar fields of data science and DevOps
- What skills a successful ML Ops developer should have, and what an ML Ops developer’s day-to-day looks like
- Why concepts like “velocity” and “stability” have their own special nuances in the world of ML Ops
For the full show notes, including who's who, see the Medium writeup.
In many ways, 2023 was the year of AI in tech, which is a double-edged sword. On the one hand, the basic technology is straightforwardly exciting — but on the other hand, with seemingly every technology solution scrambling to integrate a thin wrapper around ChatGPT, it’s hard to stand out in a saturated environment. This month on the Klaviyo Data Science Podcast, we dive into a case study of how to build AI products, SegmentsAI, and discuss the principles that go into making sure your AI-powered product shines — and, more importantly, actually helps your customers. You’ll hear about:
- How to know when AI is the right solution for the problem
- The unique technical challenges that come with building an AI product, from user testing to validation
- The answer to the AI chicken-and-egg problem
“Why do this, why build another LLM feature? It seems like every website is rushing to get their name next to AI... How you break through the noise is to actually provide value to people, not novelty. Being able to help customers speed up or generate new, interesting segments that they otherwise wouldn’t? I think that’s valuable.”— Rob Huselid, Senior Data Scientist
For the full show notes, including who's who, see the Medium writeup.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Equity, Diversity, and Inclusion
Equity, diversity, and inclusion (EDI) are more than just central principles of successful teams in data science and beyond — they’re also a rich field that presents interesting and challenging data science problems. This episode, we chat with two EDI specialists at Klaviyo about EDI, the data that powers it, and the challenges that come with using that data. You’ll hear about:
- Why EDI is a core part of both processes and products
- How to work with self-reported data — and, sometimes, work around the fact that you don’t actually have the data you want
- Examples of EDI work in action
For the full show notes, including who's who, see the Medium writeup.
2023 Year in Review
As the new year starts, we take a look back at 2023. We spoke to 11 data scientist and people who work closely with data scientists, and we asked them all the question we ask every year: what is the coolest data science thing you learned about in 2023? You’ll hear a wide range of answers, including:
- How data science moving to peripheral devices and becoming more accessible has huge implications for the future of the field
- Peculiarities of working with large language models, both in terms of the tasks they can carry out and how the process of working with them is more complicated than it seems at first
- How powerful simple techniques can be at even highly complex tasks
“You don’t have to have a PhD any longer to do data science. And I think that’s amazing and powerful, and it’s going to mean that the future is… where everybody is allowed to do data science stuff without having lots and lots of education.”
— Wayne Coburn, Director, Product Management
For the full show notes, including stories mentioned in the episode and who's who, see the Medium writeup.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Knowing your customers
Customers are all unique, whether you’re building a data science product or selling an ecommerce product. In an ideal world, we’d be able to think about all of them on a truly one-on-one basis. Most of us can’t keep track of that many people in our brains, though, which is where the topic of today’s episode comes in: what is the best way to summarize an entire population of customers into a number of groups that is small enough to intuit but fine-grained enough to actually be useful in practice?
Listen along to learn more about:
- Why understanding your customers is the superpower that drives any successful product
- How simple-sounding concepts like recency can be trickier than expected
- How to build a technical solution that draws on a vast number of data stores
For the full show notes, including resources mentioned in the episode and who's who, see the Medium writeup.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
When Things Break
Welcome to the November episode of the Klaviyo Data Science Podcast for this year! November is a unique month for ecommerce, which makes it a unique month for any software solution built for ecommerce; it’s a tradition on this podcast to take the opportunity to celebrate some of those unique challenges.
In an ideal world, software and data science products would never break. We do not live in an ideal world, though, so an important question to answer is: what should you do when things do break? This month, we discuss incidents, incident response, and getting things back on track as quickly and effectively as possible to continue delivering value to your customers.
Listen along to learn more about:
- Why not all ways of recognizing something has gone wrong are created equal
- How to cut through disagreements when the stakes are at their highest
- What sorts of unique challenges data science breakages and incidents pose
For the full show notes, including resources mentioned in the episode and who's who, see the Medium writeup.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Off the Happy Path
In most discussions about data science and data science features on this podcast, we make a basic, foundational assumption: the users whose data we are thinking about and customer experience we are trying to improve are, generally speaking, trying to use the platform in a way we recognize and approve of. Not all users of an application have this intention, and the data science behind detecting users who misuse a platform— and even abuse it — constitutes a complex and vast field of study.
Listen along to learn more about:
- Different types of human behaviors motivating platform misuse, and how that translates to different types of data
- What makes many-to-one problems so challenging
- Why keywords alone are not enough
For the full show notes, including who's who, see the Medium writeup.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Presenting your work for fun and profit
Presenting technical work is not something you automatically learn how to do — just like the technical skills themselves, it has to be learned and practiced, and opportunities to practice it can be hard to find. This episode, we discuss one opportunity that Klaviyo put together for its R&D teams this summer: the Klaviyo R&D Science Fair. Listen along to hear about:
- How, much like software development, explaining technical work is an iterative process
- The best ways to engage a crowd and get them interested in what you have to say
- The unique and powerful allure of scissors and glue guns
“We put together a little game: try to find all of the accessibility problems in this form, without using the tool that we built…. And then when they react, ‘oh my God, like that one was impossible, I don’t know how you expected me to find that,’ that’s when we can say: exactly! That’s why we needed this feature!”— Maya Nigrin, Senior Software Engineer
For the full show notes, including photos of the event, see the Medium writeup.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
An introduction to production
What comes after you finish building a data science model? If you’re working on a software project, the answer likely involves that model serving customers in production. Understanding production is crucial for any data scientist or software engineer, so we spend this episode learning about best practices from three experienced Klaviyo engineers.
Listen along to learn more about:
- How to make sure your code is “battle-ready,” whether you’re working on a data science project or not
- Why error messages you think are safe to ignore may not actually be safe to ignore
- One key lesson for safely deploying your code, no matter what environment you work in
“That’s stuck with me through the years: there are these knock-on effects between things. Even if it’s not your code, you should still try to understand how it’s working and whether it can have a ripple effect that comes back and affects your code.”— Chris Conlon, Lead Software Engineer
Check out the full show notes on Medium!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Research is a core part of data science. But data science is far from alone in that respect — other fields rely on research just as heavily, and they have their own set of hypotheses, methods, complications, and concerns. This month, we talk to three Klaviyos about research they did before joining the team — both data science research and other kinds — to see what we can learn about conducting effective data science research.
Listen along to learn more about:
- What tiny iron meteorites teach us about the importance of using your results to tell a compelling story
- What data science research into commerce and policy teaches us about iterating on your research questions
- What rubber beams teach us about the importance of getting feedback early
“Everybody has a unique perspective could be the one that opens up a brand new door. You’re looking at doing specific algorithms, you’re looking at doing the research a specific way, but there could be an alternative path.”
- Mike Galli, Data Scientist
See the full writeup on Medium!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Few parts of your product, application, or webpage are more crucial than the very initial experience. In a web application like Klaviyo, that means the home page. Everyone sees it every time they log on to do anything, and interactions with that page set the tone for everything that follows. Meaning: if you’re going to change the home page, you need to really know what you’re doing.
This month, we talk with the Klaviyo engineering team that did just that. We discuss many aspects of that redesign, including:
- How to get buy-in from teams you depend on without taking away your own independence
- The unique difficulties that come with large front-end engineering projects and smart data visualization
- How to filter through the noise when evaluating the success of a feature
“There are very few features ever been released in Klaviyo that have seen that sort of change… At the end of the day, if we can help our users complete tasks faster and more effectively, that’s our highest priority.”
- Griffin Drigotas, Senior Product Designer
See the full writeup on Medium!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
The question is slightly tongue-in-cheek, but only slightly. Data science is a new field — while many people today are graduating with degrees in data science, the same was not true a decade ago. Many of the people who work (and will work) as data scientists were not classically trained as a data scientist, but as something else. This month, we examine that process: the process of working in a field that’s distinct from data science and becoming a data scientist.
We discuss several parts of that journey, including:
- What attracts someone to data science in the first place
- How to approach gaining the technical skills you need to get a data science job
- How similar some parts of the data scientist job are to washing dishes
Where do data scientists come from?“You really need to practice using these tools. I did my best to come up with excuses to use data science techniques in all my projects… maybe instead of trying to automate a workflow in Excel VBA, I’d try to automate it in python instead.”
- Steven Her, Data Scientist
Read the full writeupon Medium!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Back by popular demand: data science is a broad, deep field with an extraordinary amount to learn, and we’re here to help you learn it. We asked four members of the Data Science team at Klaviyo what one of their favorite data science books was, and we got four different answers. Listen on if you’ve wanted to know more ways to learn about:
- How to think about and employ the Bayesian framework (and corgis)
- Learning intro-to-intermediate coding skills necessary for data science work
- The theory that drives natural language processing
- The mindset of a data scientist in general
“it gives you a different lens to apply to different problems. And sometimes taking that different lens, suddenly a problem that was really hard to formulate using traditional frequentist statistics or machine learning techniques, suddenly it can be really easy to frame in this other way” - Tommy Blanchard, Senior Data Science Manager
Read the full writeupon Medium!
Listen to the full episode on Anchor, or in your favorite podcast distribution platform!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Starting from scratchWe’ve talked about a lot of aspects of data science on this podcast — building software features, conducting research, learning new methods and skills, recruiting new members — but there’s one we’ve always avoided: building a new team from the ground up. A large reason for that is personnel — while your cohosts may be intrepid, they are not experts in this area.
This month, we bring on two people who are: Eric Silberstein and Ezra Freedman, who founded the Data Science team at Klaviyo. We draw on their wealth of experience, knowledge, and lessons learned the hard way while founding a young team.
As you might expect, these lessons extend beyond data science teams in particular — whether you’re founding another team or starting a new business, or looking to join a team in its early stages, you might be able to learn from our discussions, such as:
- How setting concrete goals is key for a new team
- How to think about your first hire, and your next five
- How to steer a team through large organizational changes while maintaining its culture and essence
“When you view the world, do you think of it as ‘if-then’ statements, or do you tend to think of it as some sort of function to optimize? Our team needed both.”
- Eric Silberstein, VP of Data Science
Read the full writeup on Medium!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
When the data science world changesWhen you work in data science, it’s inevitable that the world will change for you. Sometimes it’s due to global events, macroeconomic trends, or sudden shifts in consumer behavior. Other times it’s due to new features added by a commonly-used piece of software. When your lifeblood is data, all of these can be equally shocking and disruptive.
This month, we discuss one of the latter cases: the changes to the world of email marketing data brought about by the iOS 15 privacy updates. We bring on a panel of product managers, data scientists, and software engineers to discuss:
- How one software update can drastically alter your data landscape
- How to do research while the world is changing, and how to test your conclusions while the ground truth is still in flux
- How using different sources of data can help you adapt your product to a new reality
“That was the biggest thing I came out of this with. Being first doesn’t really matter if what you’re delivering isn’t the right solution for your customers”— Nader Fotouhi, Lead Product Manager
Read the full writeup on Medium!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
2022 Year in ReviewAs the new year starts, we take a look back at 2022. We spoke to 8 data scientist and people who work closely with data scientists, and we asked them all the same question: what is the coolest data science thing you learned about in 2022? You’ll hear about fascinating data science topics, including:
- Advances in AI, NLP, and data science in general in 2022
- How understanding data science and ML operations makes a recruiter’s job easier
- New ways to classify models, visualize data, and play video games
“I think these models have been really good for a long while. It’s a snowball effect of people realizing they’ve been good and seeing how much cool stuff can be built with them.”
- Robert Huselid, Data Scientist
Read the full writeup on Medium!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Tools of the TradeWe talk a lot on this podcast about the results of data science and software engineering work. We even talk about the process of doing data science and software engineering work. But one thing we haven’t shed much light on, until this month, is: what specific tools help a Data Science team — or any developer or data scientist similarly engaged in building a scalable and intelligent system — actually do their work? We asked several data scientists, machine learning engineers, software engineers, designers, and product managers the same question: what is your favorite tool that helps you do your job?You’ll hear all their answers in this episode, including:
- Why some well-known tools fully deserve the hype
- Specialized packages for specialized purposes
- How to slow down and really force yourself to think about the problem
- How to avoid analysis paralysis
“I like banging out unstyled web forms as much as the next back-end developer, but when you have the experience of spending all day in a tool, those ‘tiny’ things like icon consistency really pay off.”
— Zac Bentley, Lead Site Reliability Engineer II
Read the full show notes, meet this month's guests, and learn more about Klaviyo in our Medium writeup!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Anomaly Detection It’s our third November on the Klaviyo Data Science Podcast, and if you work in ecommerce, you know that November means one thing: Black Friday and (usually) Cyber Monday, i.e. the month of the year where everything changes. Traditionally, we’ve talked about things that help prepare builders of software for when the world is about to change, such as infrastructure, readiness, scale-out testing, and other things along those lines. This year, we’re approaching it from another angle: ecommerce stores go through the exact same struggle every year. How can a platform like Klaviyo help prepare them for the unexpected? One answer: by automatically figuring out when unexpected things are happening, i.e., by detecting anomalous behavior. You’ll hear all about anomaly detection on this episode, including:
- How to pivot your research when your idea is valuable but your results aren’t providing that value
- How to label large swaths of data efficiently
- How to design algorithms for an extraordinarily diverse base of end users
“Imagine a sneaker company who does product drops compared to a regular ecommerce brand. Then imagine customers who are just starting up, with very low traffic…. It was definitely a challenge to generalize to the entire Klaviyo customer base.”
— Harsh Mehta, Senior Machine Learning Engineer
Read the full show notes on Medium!
I’ll let you in on a secret: this podcast does not cover everything. We cover a wide array of projects, go into detail on a variety of aspects of them, and speak to a diverse panel of data scientists and people related to the data science world, but we still can’t cover everything. This month, to give you a taste of what we haven’t been able to showcase on this podcast, we’re asking six Klaviyos who work on or with the Data Science team one simple question: what is your favorite data science project you’ve worked on? You’ll hear about all of the following and more:
- How data science and product management can work together to maximize their strengths
- How two different viewpoints on the same project can illuminate different, equally fascinating parts of it
- An unexpectedly powerful way to use data about first names
“As a data scientist, you have to be curious and you have to be really agile, you have to pivot and ask a new question…. I learned a lot about how to leverage that to maximize outcomes when we work together.”
— Alexandra Edelstein, Director of Product Management
See the full show notes on Medium!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Using NLP to communicate at scale Last episode, we discussed the history and practice of natural language processing, or NLP. This month, we’re here to discuss an exciting and cutting-edge application: using NLP to help businesses converse with their customers at scale. See the power of NLP in action as we talk with NLP experts on the Conversation AI team at Klaviyo about:
- How NLP enables a qualitative shift in how businesses communicate
- What intent classification is and why it matters
- Tips on tailoring NLP to a highly specific use case
“There’s a lot of ways to think about the term ‘intent’. One way is what is the customer saying, and you can assign some sort of value to that. But the real intent that we’re interested in is what response are they hoping to get.”
- David Lustig, Data Scientist
See the full show notes on Medium!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
What’s the deal with natural language? Natural language processing, or NLP, is one of the dominant forces in modern data science, and it’s produced a host of data science-powered products many people take for granted as a basic fact of life. It hasn’t always been so powerful or pervasive, though — NLP has a long and interesting history, and some of the advances powering today’s technology would have seemed like science fiction only decades ago. This month, we dive into the history and foundations of NLP, examining:
- Why natural languages are so difficult to work with in the first place
- Early attempts by mathematicians and data scientists to use natural languages, and why they failed
- What distinguishes today’s cutting edge models and allows them to succeed
“Language is the natural medium for humans to communicate in. So if you want to build a really immersive, interesting product, for especially non-data scientists to interact with, it almost has to involve NLP.”
- Robert Huselid, Data Scientist
See the full show notes, including resources to learn more, on Medium.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Thinking big-picture with A/B testing We’ve discussed A/B testing multiple times on this podcast, for good reason. But there’s an important angle we have yet to cover: in the life of a researcher or marketer, there’s no such thing as an A/B test. There’s an entire system of A/B tests run for specific purposes over time. What is the best way to construct a system of A/B tests to help you learn, improve, and grow over time? How does that translate into tenets to hold while building software to help people run A/B tests? We’ve brought on three members of the data science team at Klaviyo, and you’ll hear about A/B tests in a variety of ways, including:
- Real data-driven trends observed by successful A/B testers on Klaviyo
- Why up-front thinking and vision translate into long-term success
- Why dad jokes might be far more powerful than you think
“The more experimental you can be, the more creative you can be, the more you can learn about your customers to really deliver authentic experiences and see return on your investment.”
- Woody Austin, Senior Machine Learning Engineer
Check out the full show notes on Medium for more information!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Using data science to help people write Using machine learning models to generate text, images, and other creative objects is, as they say, a bit of a hot topic right now. There are examples of models like this in action all across the internet and across different fields and disciplines. Today, we discuss one of those fields in more depth: marketing. In particular, the Klaviyo data science team recently released the Subject Line Assistant tool, which helps marketers craft better subject lines. We take a close look at that tool, how it works, and the thinking behind it to examine what it looks like to use AI to help a human write. We’ve brought on four experts from Klaviyo, and you’ll hear about subject lines from a variety of angles, including:
- What a subject line is, and why it’s arguably the most important part of an email
- What holds people back from writing great subject lines and how the team went about solving those problems
- How a specialized human-in-the-loop model for a highly specific context can look
“Subject lines are a very unique type of text generation problem. We’re not asking for a short story where there’s a lot of leeway to really hit a home run — you have a limited amount of space to communicate a brand message, communicate what the email is communicating, make a connection with your audience, and encourage them to interact.”
- Josh Villarreal, Data Scientist
Head over to the full show notes to see all the information about this episode!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Writing code for computers and people No matter what sort of data science work you do, it’s fairly inevitable that you’ll have to write code to accomplish your goals. For substantial projects, it’s also fairly inevitable that you’ll have to work with other people to see them to completion. As anyone who’s dived into a legacy code base can tell you, writing code that other people (and yourself in the future) can understand is both an essential skill to have and a difficult practice to master. This episode, we talk specifics about improving your coding skills. We’ve brought on four software engineering experts from Klaviyo, and you’ll hear about writing good code from a variety of angles, including:
- What exactly is good code?
- The biggest misconceptions that come with writing code
- How to prepare for your first code review
- Our panel’s top tips for improving your coding skills, tailored to your level of experience
“You don’t have to make a perfect work of art. It doesn’t have to be bug-free. But it should absolutely be an act of polite and intelligible communication for the next person who will interpret what you create.”
- Zac Bentley, Lead Site Reliability Engineer
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
What are data privacy and security? Data privacy and security are huge and hugely important topics — in all likelihood, you already know a little about them if you’re reading this intro. But they are both crucial to any good data science work, and this month we explore the fundamentals of both topics: why data privacy and security are necessary to deliver the value you promise your customers, who they matter the most to, and how to build privacy and security into your own data science work. The panel includes some of the foremost experts on the topics at Klaviyo from data science, engineering, and security and risk governance, so you’ll get to hear about these topics from a variety of angles, including:
- How approaches to data privacy that seem intuitive can fail, and fail spectacularly
- The consequences of not taking privacy and security carefully enough
- How to make people actively want to work within the security environment you set up
“The worst case is that you violate your customers’ trust. And if you think about personal relationships you have where someone has violated your trust, it’s really hard to build that back.”
- Dom Lombardi, Security Risk and Compliance Manager
Learn More * Privacy and security failures mentioned in the episode
— The SWIFT hack of the Bank of Bangladesh
— The CafePress data breach * Differential privacy
—Overview: A non-technical primer from Nissim et al.
— Example: Apple’s DP Sketch algorithm
— Example: Google’s RAPPOR * Data Privacy
— The Harvard Business Review’s New Rules of Data Privacy
For the full show notes, see the writeup on Medium.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Customer-focused research This month, we focus on research — but specifically research that’s aimed at your customers, delivering the sort of insight they would try to glean by running experiments and analysis using their own data. In particular, we dive into two different case studies drawn from the recent topics explored by the Klaviyo data science team. You’ll hear about:
- Why customer-focused research can be some of your highest-impact work
- Whether or not to use emojis when you’re sending out an email
- How to react when you encounter surprising results in your research
“It was startling. It was the type of number that when you see it, you think: oh, what did I do wrong?”
- Mike Galli
See the full writeup, including links to the blog posts we mention, in the show notes on Medium.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
2021 Year in Review Once again, as the new year starts, we begin by recapping the old. Instead of diving deep into a specific topic, I asked 7 members of the Klaviyo data science team to give their personal highlight for 2021 as a year in data science. You’ll hear about fascinating data science topics, including:
- How companies used domain knowledge to hyper-charge their ranking algorithms
- Powerful estimating methods that account for covariance
- How 2021 provided new opportunities — and pitfalls— for state-of-the-art experimental analysis techniques
Be sure to check out the show notes in Medium to learn more about the topics we discuss in this episode!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Customer research: your secret weapon You can study as much mathematical theory, invent as sophisticated a machine learning model, or write as clean production-ready code as you want — if you don’t make sure you’re solving the right problems to begin with, all that effort could be for nothing. It’s not a topic you learn about in most data science coursework, but understanding your end customer is a crucial part of being an effective data scientist. We spend this whole episode describing why and how to do great customer research. Topics include:
- Why customer research is such a big deal in the first place
- How talking with customers can drastically change your thinking
- How to run the perfect customer call
Be sure to check out the show notes in Medium to learn more about the topics we discuss in this episode!
If you have any questions, comments, or concerns, please contact me on Twitter.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Fuel for the Creative Fire It’s no secret: being creative is hard. Creativity requires time and energy, at the bare minimum, and lacking creativity can spiral into writer’s block and other such conditions. That may be okay if you’re just sending out a tweet here or there — but what if your core user base consists of people who need to be creative, day in and day out? The Creative team at Klaviyo recently tackled the problem of helping users get inspired to create content, and I sat down to discuss the thinking that went into the resulting feature, Showcase. You’ll hear about the development process for Showcase, but also about the underlying problems that Showcase is trying to solve and the process of coming up with a solution like Showcase. Specific topics include:
- Using data science to answer questions that seem simple… even when they aren’t
- Ensuring data privacy in solutions that have to scale
- Controversial sandwiches, and why they make great marketing tools
“There are actually a lot of sites where you can subscribe to literally every single email that a company sends out… but you have no sense of: did these emails do well? What about them was good? Is this something I should copy? It’s just throwing out a bunch of data with no context or insight whatsoever.”
— Charlie Natoli, Senior Data Scientist
See the full episode writeup, including links and who's who, on Medium.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Slow Problems, Quick Solutions We’ve devoted quite a bit of time on this podcast to robust, carefully tuned, and vetted-in-a-thousand-ways solutions. This episode, we venture beyond the land of neatly trimmed hedges and into the unknown, where scrappy solutions may be the only ones that are feasible — or even possible. And we’ll hear about settings where a quick calculation on a napkin can be the difference between success and failure — including the biggest weekend of the ecommerce year. You’ll hear about all that and more, including:
- How to solve Fermi problems (and possibly get put on a watch list)
- When quick calculations can save hours of painstaking work
- How even the simplest math can help you prepare for the most complex engineering challenges of the year
“We really are talking huge surges here… The systems you really want to watch out for, between the hours of 9 to 11 a.m. on Black Friday, move as much data as they had to move in the month of June.”
— Zac Bentley, Lead Site Reliability Engineer
See the full show notes, including the statistical explanations of the paradoxes we discuss, on Medium.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Solving difficult problems with data science This month, we talk with Shane Suazo, the founder of Plytrix Analytics, about using data science to drive efficient business growth. Shane and Plytrix work with Vital Proteins, and we dive deep into their story and highlight the places where using specific — and powerful — data science techniques helped accelerate a growth opportunity into a growth story. You’ll hear about all that and more, including:
- Establishing a single source of truth as a foundation for advanced analyses
- Preventing churn with minimal cost
- The most important advice for translating general data science techniques to the reality of a specific business
“It’s enabled Vital Proteins to send more timely messages with more relevant offers — offers that are better-tailored to our high-value customers specifically.”
— Shane Suazo, Plytrix
Links * Learn more about Plytrix Analytics (Medium, LinkedIn, Twitter, Facebook) * Full show notes on Medium
About Klaviyo Klaviyo empowers creators to own their own destiny and helps growth-focused ecommerce brands drive more sales with super-targeted, highly relevant email, SMS, Facebook, and Instagram marketing. Interested? We’re always looking for great people to join our team.
Who’s who * Michael Lawson, Senior Data Scientist * Shane Suazo, Founder, Plytrix
Edited by: Michael Lawson
Logo by: Griffin Drigotas, Ally Hangartner from Klaviyo Design
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
(More) required reading for data science A question we frequently get asked is: what books should I read to be a better data scientist/machine learning engineer? This may not surprise you, but there isn’t just one answer — in fact, we spent an entire episode talking about three ways to level up your data science knowledge and skills. This month, we’re back with three more:
- One of the foremost foundational texts for understanding machine learning models in a statistical way
- A survey course for a broad variety of machine learning models, with the opportunity to go in depth on topics like deep learning
- A foundational text in designing and analyzing experiments — both in ideal scenarios and in cases where the standard assumptions aren’t met
Mentioned this episode We discuss the following books and courses in this episode:
- The Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani, and Jerome Friedman: https://web.stanford.edu/~hastie/ElemStatLearn/
- Kirill Eremenko’s A-Z courses on data science, machine learning, artificial intelligence, and deep learning
- Field Experiments: Design, Analysis, and Interpretation by Alan Gerber and Donald Green: https://wwnorton.com/books/9780393979954
About Klaviyo Klaviyo helps growth-focused ecommerce brands drive more sales with super-targeted, highly relevant email, Facebook, and Instagram marketing. Interested? We’re always looking for great people to join our team.
Who’s who * Michael Lawson, Senior Data Scientist * Nuvan Rathnayaka, Statistician at NoviSci * Chad Furman, Senior Software Engineer * David Lustig, Data Scientist
Edited by: Michael Lawson
Logo by: Griffin Drigotas, Ally Hangartner from Klaviyo Design
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Getting real value from data science This week, we talk with Ben Knox from Super Coffee and Gina Perrelli from Lunar Solar Group about using data science to motivate the growth of a business. No hypothetical business cases this week — Super Coffee is a real business with a real growth story, and we’re here to showcase the ways that they have partnered with Lunar Solar Group and used inquisitive problem-solving methods to answer questions core to Super Coffee’s business needs. You’ll hear about all that and more, including:
- How expert insight translates into valuable questions
- Dealing with findings that stumped even the experts
- The data science feature that has helped Super Coffee the most
Links * Learn more about Super Coffee * Learn more about Lunar Solar Group
Who’s who * Michael Lawson, Senior Data Scientist * Ben Knox, SVP Digital, Super Coffee * Gina Perrelli (LinkedIn, Website), Co-Founder, Lunar Solar Group
Edited by: Michael Lawson
Logo by: Griffin Drigotas, Ally Hangartner from Klaviyo Design
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Making your product experiments count We’ve talked about quite a few aspects of data science on this podcast, but one that’s perhaps conspicuously absent so far is running experiments on your product. It’s no secret that experiments provide extraordinarily high-quality data to help you make decisions, but it’s also no secret that you only get good experimental results if you run good experiments. You’ll hear about running a good experiment and more, including:
- How experimentation fits into the design cycle
- What sorts of changes can drive unexpectedly large growth
- How to understand and adapt to counterintuitive results
Resources * Evan Miller’s A/B testing guide: https://www.evanmiller.org/ab-testing/
Who’s who * Michael Lawson, Senior Data Scientist * Eric Gravlin, Lead Product Designer * Hannah McGrath, Product Analyst II
Edited by: Michael Lawson, Aaron Goeglein
Logo by: Griffin Drigotas, Ally Hangartner from Klaviyo Design
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Recruiting for a data science team Most of us reading this writeup have probably had at least one interaction with a recruiter. Most of us reading this writeup probably don’t have a deep knowledge of recruiting — what recruiters do, how they help teams scale, and what the other 90% of the iceberg you don’t see as a candidate consists of. Recruiters are on the front lines of attracting talent and making sure that a team grows the right way, and this episode we talk about how to make sure that happens. You’ll hear about all that and more, including:
- Common misconceptions about recruiting
- The most difficult aspects of scaling a team
- Why some recruiters hate the one-page résumé
Full show notes: https://medium.com/klaviyo-data-science/klaviyo-data-science-podcast-ep-12-how-data-science-teams-should-grow-d1c7005b1dc8
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Required reading for data science A question we frequently get asked is: what books should I read to be a better data scientist/machine learning engineer? This may not surprise you, but there isn’t just one answer — depending on the skills you have, your knowledge base, the point of your career that you’re in, and many other factors, there are many books you could read that will help you learn more. This month, we cover several ways to improve the skills you need to contribute to a data science team. You’ll hear about all that and more, including:
- Object-oriented programming, how to think about it practically, and how it can help anyone on a data science team
- The ethics of machine learning and AI, and why understanding AI ethics is one of your most powerful tools
- How Pac-Man delivers some of the most powerful data science insights of our time
Mentioned this episode Some more reading or viewing that we mention in this episode:
- Practical Object-Oriented Design in Ruby by Sandi Metz: https://www.poodr.com/
- Sandi Metz’s keynote: https://www.youtube.com/watch?v=8bZh5LMaSmE
- Weapons of Math Destruction by Cathy O’Neil: https://weaponsofmathdestructionbook.com/
- Northeastern CS 4100: https://www.ccs.neu.edu/home/jwvdm/teaching/cs4100/fall2019/
- UC Berkeley CS 188: https://inst.eecs.berkeley.edu/~cs188/pacman/home.html
Contact us
The best place to reach the podcast is by messaging me on Twitter: https://twitter.com/lawson_m_t.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Understanding your Customer Lifetime Value This is a math-heavier episode than usual — we’re going to dive into probabilistic distributions and talk about systems of estimators. Even if that’s not your background, though, you should still find this episode useful. That discussion is all based in something crucial to real-life businesses around the world: customer lifetime value, or CLV. What exactly does CLV tell you, how exactly is it calculated and predicted, and why exactly does it matter to your business? You’ll hear about all that and more, including:
- Statistical approaches to modeling customer behavior
- Difficulties that arise when customers don’t act exactly like they’re modeled
- How the humble Tungsten cube can teach us about the entire customer journey
Contact me
The best place to reach the podcast is by messaging me on Twitter: https://twitter.com/lawson_m_t.
Benchmarks: what are they and why? You’ve probably heard of benchmarks. You’ve probably even used them. But what exactly are benchmarks, how are they useful, and how can you go about building a system to make benchmarks in your own industry? You’ll hear about all that and more, including:
- How to use benchmarks to make informed decisions about improving your business
- Why the humble stoplight served as a key insight for making complex math understandable
- How to assess your personal levels of spice intake
Mentioned this episode Some more reading or viewing that we mention in this episode:
- 2020 in review using benchmarks: https://www.klaviyo.com/blog/ecommerce-benchmarks-2020-email-marketing
- The benchmarks feature announcement video, featuring the one and only Data Science Santa: https://youtu.be/Ge5zvOQVLDc?t=298
- The Klaviyo benchmarks feature page: https://www.klaviyo.com/features/benchmarks
- The Klaviyo benchmarks announcement post: https://www.klaviyo.com/blog/benchmarks-ecommerce-performance
About Klaviyo Klaviyo helps growth-focused ecommerce brands drive more sales with super-targeted, highly relevant email, Facebook, and Instagram marketing. Interested? We’re always looking for great people to join our team.
Contact me The best place to reach the podcast is by messaging me on Twitter: https://twitter.com/lawson_m_t.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
2020 Year in Review We have a bit of a different episode this month. Instead of diving deep into a specific topic, I asked 14 members of the Klaviyo data science team to give their personal highlight for 2020 as a year in data science. You’ll hear about a bunch of fascinating data science topics, including:
- Using machine learning to take a quantum leap in drug discovery
- Discovering methods from much older years that are still relevant for the problems of today
- How 2020 provided some new opportunities — and some sobering real-world stress tests — for data science
Full Episode Notes
See https://medium.com/klaviyo-data-science/klaviyo-data-science-podcast-ep-8-2020-a-data-science-year-in-review-88be9b534183.
Contact Me
Contact me on twitter: @lawson_m_t
Corrections This podcast was recorded in January 2021, before Abigail Thorn publicly came out as transgender. It currently refers to her by her former name, but will soon be edited. Congrats to Abigail!
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Engineering Challenges in Data Science All data science work at scale rests on a solid foundation of engineering. We discuss how to establish that foundation — from what goes into software engineering to begin with to the specifics of how to prepare for big seasonal events like Black Friday and Cyber Monday. You’ll hear from software engineers on the team about:
- Why building software is a lot like running a bar
- How to make things be — or at least seem — real-time
- How a tiny leak can crash a whole system
Full show notes available at https://medium.com/@michael-lawson-96765/klaviyo-data-science-podcast-ep-7-laying-a-stable-engineering-foundation-ba6462aa0db.
Contact us: @lawson_m_t on Twitter.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Seasonality in e-commerce As the calendar changes, so do the right steps to take for your e-commerce business. We wade into the waters of seasonal changes in behavior, data, and logistics, and we take a deeper look at how to navigate them. You’ll hear from data scientists and product analytics about:
- The fact that gardening is sometimes more powerful than the biggest holiday of the year
- Why you should stop worrying and love the error bars
- How to immortalize yourself as a data science meme
Full show notes are available at https://medium.com/@michael-lawson-96765/klaviyo-data-science-podcast-ep-6-navigating-seasonality-in-e-commerce-1bac11b8bf13.
Contact us: @lawson_m_t on Twitter.
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
Recommender systems: how do they work?
We get recommendations for all sorts of things today: routes to take when we drive, places to eat, books to read, petitions to sign, and of course, things to buy. We take a deeper look at the task of making the data science and software systems that dispense useful recommendations at scale, with a special focus on recommending ecommerce products. You’ll hear from data scientists and engineers about:
- The best (and worst) recommendations we’ve ever gotten
- How the recommendation systems you take for granted actually work under the hood
- Why dining room tables are basically the same thing as people’s favorite colors
Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…
What makes a report good? Data-centric teams likely take it as a given that good reporting is a key to living a happy life, but what exactly makes a report good? We dive into the topic of reporting and discuss ways to make a report exceed expectations. You’ll hear from data scientists and product designers about:
- Why bad reports may be worse than no reports
- Why good reports may be more like the movie Inception than you think
- How to design reports with your target audience in mind
In this episode, we take a deep dive into a recent feature the team built, signup form A/B testing, to give you a taste of what it’s like to build software for data science. You’ll hear from data scientists, product designers, and software engineers. We discuss:
- The reason multi-arm bandits are called multi-arm bandits
- How to distill a two-minute explanation into a single word
- The way people think about randomness, and what that means when you’re designing a feature that involves randomness
Questions, comments, clarifications, or concerns? Reach out to Michael Lawson!
In this episode, we discuss how our careers in data science began, lessons we’ve learned along the way, and mistakes we’ve made and learned from. You can expect to hear:
- When we stopped wanting to be astronauts and started wanting to be data scientists
- Advice we’d give to anyone just starting in the field
- Advice we’d give to anyone currently at the helm of a dinosaur-based movie franchise
Resources We mention a few books and other resources in the course of this episode. Check them out here:
- The Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani, and Jerome Friedman: https://web.stanford.edu/~hastie/ElemStatLearn/
- Statistical Rethinking by Richard McElreath: https://xcelab.net/rm/statistical-rethinking/
- Linear algebra: Foundations to Frontiers on edX: https://www.edx.org/course/linear-algebra-foundations-to-frontiers
- Blog posts: http://www.terran.us/talks/201808_successful_project.pdf, https://www.fast.ai/2018/07/12/auto-ml-1/, https://eng.lyft.com/whats-in-a-name-ce42f419d16c
We’re excited to unveil the first episode of the Klaviyo Data Science podcast! This podcast is intended for all audiences who love data science--veterans and newcomers alike, from any field, we’re all here to learn and grow our data science skills.
We’re jumping right into the action with this episode. This is a deep dive into research in action. We’ll learn about what’s happening in the world of ecommerce in the wake of COVID-19, and more importantly how we figured out what’s happening. We’ll dig into the whole research funnel, from forming a hypothesis, to analyzing and learning, to taking what you’ve learned and iterating again.
Also in this episode:
- Mastering dinner table conversations with friends and family
- Using automation to win friends and influence marketers
- How to smell good in quarantine
Want to learn more about Klaviyo? Check us out at www.klaviyo.com!