Experiment Nation: The Podcast: Recent Episodes
Experiment Nation
We interview Experimenters from around the world to explore how they got into Experimentation, learn from their experiences, and hopefully have some fun along the way.
In this episode of Experiment Nation, host Jason Gossit sits down with Ameliano Blanca, a senior growth and product leader with 10+ years of experience driving revenue through data-driven experimentation.They dive deep into:How to build experimentation programs from the ground upThe balance between qualitative and quantitative insightsExperimentation in startups vs. mature organizationsCRO frameworks and the power of iterationHow to avoid "random testing mode" and adopt structured frameworksIf you’re a growth, product, or CRO professional, this episode is packed with actionable insights to help you scale experimentation programs, improve conversion rates, and create sustainable growth strategies.👉 Get our new book: Prove It or Lose It – The (Mostly) No-Nonsense Guide to Surviving Experimentation Program Drama at experimentnation.com and all major bookstores.📌 Chapters:0:00 – Why tests should be rerun after 12 months0:19 – Welcome & introduction with Jason Gossit0:35 – Meet Ameliano Blanca: growth & product leader1:32 – First steps in building experimentation programs2:42 – Measuring conversion funnels & revenue impact3:46 – Tools for gathering quantitative & qualitative insights5:05 – Heatmaps, surveys, and finding hidden gems6:00 – Drafting test ideas & prioritization7:19 – Using and adjusting the ICE framework9:50 – The “three strikes” rule for failed experiments11:16 – Experimentation in startups vs. mature companies13:00 – Why rerunning startup tests is critical14:57 – When to pause or rerun losing experiments15:29 – Complementing quantitative data with customer interviews17:16 – Running experiments at scale in larger organizations18:18 – Balancing speed vs. rigor in CRO testing20:04 – Defaulting experiments & tracking long-term results20:11 – CRO frameworks and the power of iteration22:26 – Why iteration is underrated in experimentation23:44 – Adapting frameworks to your company & industry25:18 – Advice for teams stuck in random testing mode27:39 – How CRO perspectives have evolved over 10 years28:39 – Growth, insights, and the rise of experimentation in strategy30:38 – Real-world example: financing simulator boosts conversions
In this episode of Experiment Nation, host Jason sits down with Namata Agarwal, a seasoned product designer with over 17 years of experience building digital products, websites, and apps.Together, they explore:- Ethical testing: how to balance user needs with business goals.- Leaks vs. puddles framework: a practical way to identify urgent vs. non-urgent product problems.- The role of design in CRO and experimentation.- Frameworks, guardrails, and collaboration strategies for building an experimentation culture.Whether you’re a CRO specialist, product manager, or experimentation nerd, this episode is packed with actionable insights you can bring to your own team.👉 Don’t forget to like, comment, and subscribe to stay updated on the latest in experimentation and growth!📖 Check out Prove It or Lose It: The (Mostly) No-Nonsense Guide to Surviving Experimentation Program Drama .⏱️ Chapters with Timestamps0:00 – Why design must be involved early0:22 – Welcome & intro to Jason0:43 – Guest intro: Namata Agarwal0:55 – Topics: Ethical testing & Leaks vs. Puddles1:18 – What is ethical testing?2:22 – Balancing business needs vs. user needs2:54 – Handling stakeholder pushback4:01 – Misaligned metrics and real solutions5:17 – The Leaks vs. Puddles analogy explained6:32 – Why better hypothesis building matters7:48 – Using Airtable & frameworks for test prioritization8:40 – Why experimentation must be collaborative9:24 – Reframing experiments around the user journey10:00 – Unpredictable user journeys & design challenges11:12 – Involving guest collaborators in experimentation11:50 – Mapping user journeys with Miro & qualitative research12:50 – Jobs To Be Done & other qualitative frameworks14:44 – Turning qualitative insights into hypotheses15:32 – How product design elevates CRO strategies16:11 – The importance of forms in experimentation17:36 – Why design should be included from day one18:20 – Design is not just visuals—it’s about impact19:11 – Governance, responsibility, and inclusivity in testing19:46 – Leak vs. Puddle framework deep dive22:58 – Running collaborative workshops25:45 – The “Stinky Fish” workshop format27:21 – Micro vs. macro frameworks for problem-solving28:05 – Balancing bold ideas vs. safe bets29:19 – Planning tests like an investment portfolio30:13 – Testing across 13 sites at scale30:26 – What Namata would A/B test in daily life31:57 – Art, painting, and creative balance32:42 – Martial arts & keeping calm32:56 – Wrap up & final thoughts33:14 – How to connect with Namata
In this episode of the Experiment Nation Podcast, host Ward Vanestra sits down with Ryan Lux, one of the first team members at Eppo and now Experimentation Evangelist at Datadog.They dive into:What Eppo is and how it helps companies experiment at scaleWhy data warehouses are critical for trustworthy resultsThe recent Datadog acquisition of Eppo and what it means for the futureAdvanced methods like CUPED (noise-cancelling for experiments) and contextual banditsHow to scale experimentation culture and get promoted by driving adoption, efficiency, and trustworthinessWhether you’re a CRO, growth leader, or experimentation enthusiast, this episode is packed with insights from someone who’s been shaping experimentation for over a decade.📘 Check out Experiment Nation’s new book: Prove It or Lose It: The Mostly No-Nonsense Guide to Surviving Experimentation Program Drama → experimentnation.com⏱️ Chapters0:00 – Why win rates don’t really increase0:20 – Introduction to Ryan Lucht & Eppo1:20 – What Eppo does and who it’s for2:18 – Experimentation at smaller companies3:18 – Datadog’s acquisition of Eppo3:59 – The data warehouse advantage6:03 – CUPED explained: noise-cancelling for experiments9:13 – Segments, personalization & pitfalls11:09 – Heterogeneous treatment effects (HTEs)12:06 – Personalization with contextual bandits14:00 – Why Datadog acquired Eppo16:22 – Data observability & product analytics17:52 – Experimentation and site performance19:46 – Content, culture, and experimentation career paths22:10 – Every experiment is valuable (win, lose, or flat)24:17 – OKRs for experimentation leaders25:47 – Three key OKRs: adoption, efficiency, trustworthiness28:14 – Knowledge building and learning plans29:34 – The vision: experiments as the ultimate insight system30:24 – Ryan’s Substack “Everything is an Experiment”31:02 – Closing thoughts & where to connect
In this episode of the Experiment Nation Podcast, host Richard Joe welcomes Manuel Da Costa, founder of Effective Experiments and now Efestra, to unpack a critical issue in the experimentation industry: the trust gap between practitioners and business decision-makers.🎯 Manuel shares how running more experiments isn’t the goal—making better decisions is. He introduces the Learning Loop, explains why most experimentation programs stall, and gives practical strategies for connecting experimentation insights to executive decisions.🔍 Whether you’re in CRO, product experimentation, or experimentation governance, this is a must-watch for moving beyond vanity metrics and toward real impact.📌 Connect with Manuel:📍 LinkedIn: https://www.linkedin.com/in/manueldacosta/#CRO #Experimentation #ABTesting #EffectiveExperiments #ProductManagement #DigitalMarketingChapter00:00 – Experiments don't matter unless they drive decisions 00:30 – Meet Manuel Da Costa, founder of Effective Experiments 01:45 – Manuel’s journey: from web design to CRO and Lean Startup 03:00 – How experimentation became his product 05:20 – The limitations of spreadsheets in experimentation 06:45 – Building Effective Experiments to streamline the workflow 09:10 – What is the "Trust Gap" in experimentation? 11:30 – Why 88% of experimentation programs aren’t transformative 13:00 – The survival mindset vs. strategic experimentation 16:00 – The echo chamber of experimentation advice 17:40 – Why velocity and win rate are not the real success metrics 19:00 – The problem with how learnings are shared 21:00 – How to communicate insights in one powerful slide 23:00 – Building organizational learning with the “Learning Loop” 25:00 – Step-by-step breakdown of the Learning Loop framework 27:15 – Why experimentation needs executive alignment 29:00 – How to reset a tactical experimentation program 30:40 – Introducing Efestra: a governance platform for experimentation 32:00 – Final thoughts and how to get in touch with Manuel 33:14 – About the book: Prove It or Lose It
In this episode, we dive into Speero’s shift from CRO to Experiment-Led Growth, how they structure their testing systems, where to start with a new client, and what makes a good growth bet. Plus, we discuss the power of insights, the rise of EXL meetups and the future of Speero.Chapters----00:00 - Introduction 00:18 - Meet the Hosts & Guest 01:11 - From CRO to Growth Experimentation 02:55 - Building a System for Experimentation 05:00 - Organizational Buy-in and Culture Shift 06:02 - Client Reactions and Collaboration 08:00 - Applying Experimentation Across the Business 10:00 - Strategy vs. Tactics 11:30 - Why Create a Strategy? 13:02 - Making Better Bets with Experimentation 15:05 - Breaking Down Big Initiatives 16:41 - Landing Pages, Messaging & Paid Spend 18:20 - Irreversible Risks and Thought Experiments 20:15 - Research Frameworks: Qual & Quant 21:47 - Copy Testing and Messaging Insights 23:48 - Combining Qualitative and Quantitative Testing 25:49 - Using Ads to Accelerate Learning 26:33 - EXL Meetups: Community and Format 28:00 - How to Host an EXL Meetup 30:04 - Upcoming Global Meetups 31:35 - How to Connect with Paul Randall 32:40 - Outro#cro #experimentnation #abtesting
This episode features Achille Vaulot, a CRO expert, discussing the evolution of conversion rate optimization (CRO) and the steps to building a mature program. He highlights the shift from basic A/B testing to a data-driven, strategic approach encompassing research, thorough analysis (including post-test analysis of both primary and secondary KPIs), and effective client communication. Achille emphasizes the importance of planning for both successful and unsuccessful tests, managing client expectations, and fostering a culture of experimentation within the client’s organization. He stresses the value of qualitative data alongside quantitative results and the importance of aligning CRO strategies with broader business goals. The conversation underscores the crucial role of education and collaboration in scaling a successful CRO program.
Ekaterina (Shpadareva) Gamsriegler, a seasoned growth and marketing expert, shared her insights on mobile app experimentation. Here are 5 key takeaways:Mobile experimentation is more complex: Release cycles are longer, and dependencies on app versions can slow down learning.Think beyond the app: Experimentation should encompass ads, store listing pages, and the app itself.Borrow insights from other areas: Use data from paid advertising to inform store listing optimizations.Interconnectivity is key: Changes in one area can impact others. Strong stakeholder management is crucial.Document everything: Detailed documentation helps track results and avoid surprises.Check out the full conversation on Experiment Nation’s YouTube channel!LinkedIn: https://www.linkedin.com/in/ekaterina-shpadareva-gamsriegler/#mobilemarketing #appmarketing #experimentation #growthmarketing #experimentnation
Tracy Laranjo shares learnings from years of growing her career with Suchita Srinivasan, here are some key takeaways:1. CRO vs Experimentation: CRO focuses on driving performance and results, while experimentation centers around decision-making and validating ideas.2. Agency vs In-House: Tracy’s journey highlights the benefits of agency experience for rapid skill development and exposure to diverse challenges.3. Building Your Experimental Brand: Don’t be afraid to be yourself online, even if it’s not the typical thought leader persona.4. Adapting to Constant Change: Focus on fundamentals and building your skills; stay curious and open to new developments, but don’t fear the future.5. CRO is for Everyone: While AB testing requires scale, CRO encompasses qualitative research and user interviews, making it accessible for businesses of all sizes.
This episode of Experiment Nation features an interview with Dr. Else van der Berg, a product management consultant for B2B SaaS startups and scale-ups. Dr. Else van der Berg discusses her process for helping companies become more outcome-driven, focusing on understanding the ideal customer profile (ICP) and its associated value. She also shares insights on common challenges faced during the testing phase, including participant recruitment, data analysis, and differentiating signal from noise.Highlights🎯 Dr. Else van der Berg emphasizes the importance of defining a narrow and well-defined ICP, emphasizing the need to focus on a specific customer segment.🔎 She believes in a layered approach to testing, using a variety of methods, including interviews, landing page tests, and moderated user tests.🗣️ Empathy plays a crucial role in interpreting user feedback, recognizing that data interpretation can be subjective and require a “gut feeling” alongside evidence.📈 Dr. Else van der Berg stresses the importance of finding a mentor within the field and learning on the job to master product management techniques.👩💻 She encourages aspiring product managers to reach out to her via LinkedIn to learn more and potentially connect for career advice.
We recently had the pleasure of chatting with Rishi, a Shopify product expert. Here are 5 key takeaways for optimizing your product pages:1) Product pages are the Grand Central Station of conversions. They’re where buyers make the decision to buy or leave.2) Pop-ups aren’t as effective as you think. Focus on valuable content, not annoying interruptions.3) Content is king (but don’t be afraid to go long!). Detailed product descriptions are more effective than short ones.4) Build trust above the fold. Establish your company’s credibility before highlighting product features.5) Understand the filters buyers use. Address common concerns like “too good to be true” to build confidence.Want to learn more? Check out Rishi’s website, frictionlesscommerce.com!#ecommerce #productpages #conversionrateoptimization #marketing #experimentation
Black Friday is coming, and the pressure is on. But don’t worry, Ilan Hurwitz has some expert advice to help you navigate the chaos.Here are five key takeaways from his chat with Tracy Laranjo:1) Start now! Don’t wait until the last minute to get buy-in and start testing.2) Speak the language of profit. Showcase the ROI of CRO to your stakeholders, focusing on increasing conversions and profits.3) Don’t be afraid to experiment. Test offers, product placements, and even email campaigns outside of standard website A/B testing.4) Focus on the customer journey. Don’t just look at your website. Analyze customer service tickets, reviews, and LTV to identify areas for improvement.5) Think beyond discounting. Explore creative offers like gift with purchase, cross-sells, and subscription incentives to boost sales.Get ready to conquer Black Friday!#BlackFriday #CRO #ecommerce #marketing #experimentation
Experimentation is a journey, not a destination. Don’t expect to build a culture of experimentation overnight.Cross-functional teams are critical. Breaking down silos and bringing together engineers, product managers, and designers will create a unified vision and drive faster iteration.Leadership buy-in is essential. A top-down approach to embedding experimentation ensures everyone is on board and resources are allocated strategically.Don’t underestimate the importance of organizational rhythms. Establish clear processes and structures to ensure smooth collaboration and avoid friction.Embrace the growing pains. Change is rarely easy. Be prepared for challenges and celebrate milestones along the way.
In this podcast episode of Experiment Nation, hosted by Charlotte Bomford, guest Iqbal Ali, an experienced web designer, researcher, and comic book writer, discusses the concept of solution bias and the importance of identifying underlying problems before diving into solutions. Ali emphasizes using AI to help understand problem landscapes, avoiding hastily jumping to solutions, and improving experimentation processes. The conversation covers techniques like premortems and frameworks to minimize potential issues and ensure experiments address real user problems. Ali highlights the collaborative potential of AI tools in maintaining the connection between problem statements and solutions, advocating for a more thoughtful approach to A/B testing and problem-solving in product development.
Ruben de Boer, a lead consultant at Online Dialogue and the founder of Conversion Ideas, shares his approach to conversion rate optimization (CRO) and experimentation. With 15 years of experience, Ruben emphasizes the importance of data-driven decision-making and understanding human behavior. He highlights the shift from focusing solely on digital products to also optimizing organizational processes. Ruben provides insights into the challenges of implementing experimentation in organizations, the need for higher management support, and the significance of aligning with colleagues' goals and motivations. He advocates for a research-driven approach, continuous feedback, and fostering an experimentation culture within teams to drive better results.
Tymur Donets, a seasoned Growth PM and experimentation consultant, shared some powerful insights during his Experiment Nation talk. Here are five key takeaways:Growth PMs are holistic: Don’t be siloed! Growth PMs work across the entire customer journey, ensuring marketing and product teams are aligned.Commercial awareness is key: Both Growth PMs and CROs must understand the business impact of their work and how it contributes to the bottom line.Empathy is vital: CROs need to understand the pressures their PM colleagues face and proactively offer support.Leverage your expertise: If you’re a CRO, consider specializing in Growth PM to expand your impact and skillset.Transparency is key: Clear communication between teams helps everyone understand their roles and collaborate effectively.Find Tymur on LinkedIn: https://www.linkedin.com/in/tim-donets-2358364aChapters:0:00 - Introduction and The Role of a Growth PM0:58 - Growth PM vs. Product Manager2:40 - Tymur’s Journey and How He Became a Growth PM5:23 - Importance of Business Savvy and Data Literacy8:30 - Bridging the Gap Between Marketing and Product Teams12:42 - Experimenters as Facilitators16:38 - The Complexity of a Product Manager’s Role19:50 - CRO Responsibilities and Building Relationships with PMs24:48 - Transitioning from CRO to PM29:00 - Finding Growth PM Opportunities and Wrap-Up#growthmarketing #experimentation #cro #productmanagement #collaboration
Kelly Wortham dropped some serious knowledge bombs in our recent Experiment Nation chat. Here’s what stood out:- CRO isn’t just for marketers: Lawyers, project managers, even developers can thrive in this field. It’s all about a love of learning and adaptability.- Don’t underestimate the power of mentorship: Kelly has THREE mentors, each focusing on different aspects of her career. Find your own advisors to help you grow!- Certifications are nice, but experience trumps all: While CXL is great, hands-on testing platform experience is the real deal.- UX psychology is CRUCIAL: Understand your users’ motivations and behaviors to create truly effective tests.
Simar, CEO and Founder of OptiPhoenix, shared invaluable insights on A/B testing and CRO at Experiment Nation. Here are 5 key takeaways:Don’t rely solely on macro goals: Micro goals reveal user behavior, driving better testing strategies and informing future hypotheses.Iterative testing trumps scrapping: Learn from failed tests by adjusting hypotheses and running new experiments, boosting your win rate.QA is critical, but often neglected: Invest time in thorough QA, ensuring your test is accurate and integrated correctly.Outsourcing AB test development can be beneficial: Consider outsourcing for faster scale-up, access to expert resources, and reduced development burden.Prioritize experimentation within your workflow: Make A/B testing an integral part of your product development process, leading to better decisions and faster results.
Avishek Basu Malik, a Senior PM at Cheapoair talks about:- Experimentation at scale: Even with high traffic, rigorous experimentation is crucial.- Idea generation: User research, NPS surveys, and internal stakeholders are valuable sources of experiment ideas.- Communicating progress: Regular updates with leadership build trust and ensure alignment.- Personalization is nuanced: Distinguish between customization, segmentation, and true personalization.- Generative AI’s potential: Explore its use for simplifying the travel planning experience.
Michael St Laurent, a seasoned CRO expert, shared his insights on large-scale testing Here are 5 key takeaways for marketers:Prioritization is key: Large-scale testing requires prioritizing tests based on potential impact and aligning with overall business goals.AI is the future, but humans guide it: Mike emphasizes the power of AI in automating processes and analyzing data, but highlights the importance of human expertise and guidance.Data quality is crucial: AI models rely heavily on the quality of data. Invest in building a robust data infrastructure to ensure accurate results.People are the biggest challenge: Navigating internal politics and managing stakeholder expectations is often the biggest hurdle for large-scale testing.Building a testing culture is a win: The ultimate goal of large-scale testing is to create a data-driven culture that fosters continuous improvement and better user experiences.
Ellie Hughes, a top CRO expert and Experimentor of the Year, shared some incredible insights in her recent Experiment Nation interview. Here are 5 takeaways to boost your experimentation game:- Start Small and Stay Curious: Ellie emphasizes that experimentation is a journey, not a destination. Start small, stay curious, and never stop learning!- Prioritize Value, Not Tasks: Focus your time on activities that truly drive client value, like uncovering insights and ideation, rather than spending hours on tedious tasks.- Set Clear Expectations: Clearly define reporting timelines and communicate about peeking, managing client expectations, and setting the stage for success.- Embrace Iteration: Experimentation is an iterative process. Be open to changing your process, adapting to new learnings, and continuously improving.- Maximize Impact Through Big Ideas: Larger changes can be easier to detect, resulting in faster results and more impactful growth. Encourage clients to think big!#experimentation #CRO #conversionrateoptimization #marketing #digitalmarketing #experimentnation
Join us for a deep dive into conversion rate optimization on low-traffic sites with these actionable insights Chris MercerMeasure before optimizing: Understand the current user journey and behaviors to identify areas for improvement.Test sequentially: Iterate faster on low-traffic websites by testing changes sequentially instead of using AB tests.Focus on the right metrics: Prioritize actions that drive meaningful outcomes, such as lead generation or revenue.Listen to the market: Gather user feedback to inform your optimization strategy and meet their needs.Automate measurement: Leverage Google Analytics 4 to collect and analyze data effectively to guide decision-making.#CRO #Experimentation #ConversionOptimization
Discover the power of experimenting to learn and win! 💪Nick Murphy shares his expertise on building learning libraries. Here are 5 key takeaways:- Learning libraries capture the “why” behind successful and unsuccessful tests.- Organize your learning library with consistent and digestible fields.- Use “broad” and “specific” learning columns for a layered understanding.- Encourage cross-team collaboration by sharing insights company-wide.- Learning libraries foster a culture of experimentation and growth.
In this episode, senior customer experience leader, Courtney LeBlanc, reveals the ideal partnership between CX and experimenters and shares invaluable insights on:Forging powerful partnerships between CX and experimentersNavigating the daily demands of a CX leaderSimplifying CX workflows with customer insights... and so much more!Don't miss out on this enlightening conversation from an expert in the CX game!Find Courtney on LinkedIn ⤵LinkedIn: https://www.linkedin.com/in/courtney-leblanc-426920b3/00:00 Introduction and Mutual Appreciation00:27 Meet Courtney: Customer Support and CX Expert03:38 Building an Alliance with Customer Support06:11 Challenges and Strategies in CX Collaboration14:25 Data Analysis and Insights Sharing23:01 The Ideal Partnership Between CX and Experimentation24:47 Balancing Customer Insights and Business Priorities26:46 The Emotional Impact of Customer Support28:29 Communicating Customer Stories Effectively30:04 Leveraging NPS Data for Product Improvement32:08 Empowering Teams with Data Access33:06 The Importance of Experiment Communication37:53 Fostering Collaboration Between CX and Product Teams48:58 Reflecting on Growth and Collaboration50:22 Conclusion and Final Thoughts
How do you get Product Managers to adopt Experimentation? Shagun Aulakh, the Director of Product Management - Experimentation at American Express shares with Rommil here tips. Here are just a few of the takeaways:1- Leadership advocacy is critical for embedding experimentation practices in a highly matrixed organization.2 - Prioritize education and internal PR to influence and persuade teams to embrace experimentation.3- Define success metrics for your experimentation program and map out a long-term plan for evolving maturity.5 - Identify champions within the organization to inspire and motivate other teams to adopt experimentation practices.5 - Democratize insights and share success stories to showcase the value and impact of experimentation.Watch Shagun here:Experimentation Elite in Birmingham UK: https://experimentationelite.com/#spe...Growth Marketing Summit in Frankfurt, Germany: https://www.growthmarketingsummit.com/en
In this episode, we have Joey Herzberg who shares his thoughts on how AI fits in the experimentation life cycle, the utility of keeping easily referenced records of test results, and the importance of experimentation strategy with regard to sample ratios.
Learn about human-first engineering in this in-depth interview where Shawn David discusses strategies to organize work and to break down tasks into smaller digestible sessions. He explained the most effective ways for teams to discover the work that should be automated the most by focusing on the desires and strengths of individuals, to remove work that is menial specially to them. We explored the impacts on company culture that menial tasks can cause and discussed some of the pitfalls with new technologies as well as lessons to keep in mind while reviewing new tools and technologies.
Chapters
00:00 Introduction to Human First Engineering
00:55 Meet Sean David: The Pioneer of Human First Engineering
01:34 Breaking Down Tasks for Efficiency
09:56 The Importance of Time Tracking in Various Professions
16:51 Discovering Automation Opportunities
25:34 Maximizing Productivity: The Power of Uninterrupted Focus Time
26:59 Introducing Roman Santiago & Experiment Nation
27:13 The Impact of Menial Tasks on Team Mental Load
27:31 Exploring Company Culture and Employee Well-being
00:00 The Role of Menial Tasks in Shaping Company Culture
32:24 The Evolution of Work: From Henry Ford to Jeff Bezos
36:32 Navigating the Hype Around AI and Tools in the Workplace
47:45 Human-First Approach: Automating to Enhance Productivity
50:44 Closing Thoughts and Where to Find More
Jyoti Malik, the founder of Blue Bagels, built a successful CRO agency by only working with ethical brands. In this chat with Tracy Laranjo, she shares: How to run CRO programs with purpose instead of random guesses- Her favourite underrated user research methods- Tips for creating a CRO strategy that succeeds in the long-term... And much more.Contact Jyoti:Website: https://www.bluebagels.com/LinkedIn: https://www.linkedin.com/in/jyotiisinghEmail: jyoti@bluebagels.comChapters00:00 Introduction to Experiment Nation and Guest Introduction00:48 Jyoti Malan's Journey and the Essence of Blue Bagels02:26 The Philosophy Behind Purpose-Driven Brands05:43 Crafting Purposeful CRO Programs08:32 Identifying and Correcting Off-Track CRO Programs17:36 Unconventional Research Methods in CRO21:06 Adapting and Evolving CRO Strategies24:41 Upcoming Projects and Closing Thoughts
Join Nima Yassini as he dives into experimentation culture, the journey of building a brand UX and digital marketing agency focused on conversion rate optimization, the integration of digital marketing and experimentation cultur, the struggles and benefits of adopting a culture of experimentation, and the need for leadership and reinvestment in the experimentation industry.Connect with Nima here: https://www.linkedin.com/in/nimayassini/Chapters0:00 Opening Remarks and Guest Introduction00:48 The Evolution of a Digital Marketing Agency01:27 Merging UX Design with Experimentation03:56 The Power of Data in Design Decisions09:13 The Culture of Experimentation in Agencies vs. In-House11:49 The Challenge of Embedding a Culture of Experimentation13:36 The Future of the Experimentation Industry25:12 Advice for Experimentation Practitioners28:29 Closing Thoughts and Offers to Help the Industry
CEO and founder of Decision Layer, Solomon Kahn chats with Rommil Santiago about why he thinks that most companies should not be A/B testing and shares what we should instead. He also discusses his recent debate with Ron Kohavi on LinkedIn on this same topic.Solomon's LinkedIn: https://www.linkedin.com/in/solomonkahnSolomon's email: solomon.kahn@gmail.comOriginal YouTube Video: https://www.youtube.com/watch?v=vqPylhO3ePUOriginal LinkedIn Post: https://www.linkedin.com/posts/solomonkahn_the-case-against-ab-testing-yes-i-know-activity-7184577272685719552-hFABRon's reply: https://www.linkedin.com/posts/ronnyk_abtesting-experimentguide-abtestingissues-activity-7185060367054565376-cAU9
It's that time of year again! It's time for Experiment Nation's 2024 CRO Salary Report. In this episode, we explore some of the report's highlights with Tracy Laranjo as well as ask her for her thoughts on how CROs and Experimenters can get paid fairly.Find the CRO Salary report here: https://experimentnation.com/experiment-nations-2024-cro-salary-report/Tracy's LinkedIn: https://www.linkedin.com/in/tracylaranjo/Tracys email: hello@tracylaranjo.com
Zach Lebovics shares insights on experimentation and product management. He discusses the importance of documenting assumptions, testing the best-case scenario first, and understanding that experiment failure isn't solely tied to the outcome. Zach also shares personal experiences and learnings from working in companies like Zynga, Ritual, and Tonal, providing valuable tips for product managers.Episode linksZach's Etsy shop link: www.mindfulmix.coZach's Team Hub on Coda link: https://coda.io/@zach-lebovics/tonals-product-team-hubChapters00:00 Unlocking the Power of Experimentation: Insights and Strategies00:23 Meet Zach: A Journey from Product Manager to Experimentation01:56 The Art of Optimization: From Personal Life to Professional Projects05:33 Diving Deep into Experimentation: Lessons from Zynga to Tonal12:21 The Three Pillars of Successful Experimentation30:26 Wrapping Up: Key Takeaways and Future Directions
Shiva Manjanath, discusses with Ana Kerkovic the significance of collaboration with stakeholders and stakeholder management, optimizing CRO processes, transitioning from testing to experimentation, the impact of copy testing, and the need for research-backed strategies in CRO.Chapters-----00:00 Unlocking Business Growth with Data00:24 Introducing Experiment Nation and Guest Shiva Manjunath00:45 Diving Deep into the World of CRO and Experimentation02:52 The Optimizer's Breakfast: A Routine for Success04:12 Anna Kerkovic Shares Her Journey and Passion for Psychology in CRO05:34 The Art of Persuasion: Manipulation or Motivation?06:29 Strategizing Beyond Spaghetti Testing: A Structured Approach to Experimentation14:15 Building Trust and Navigating Frustration in the CRO Process23:41 From Testing Ideas to Strategic Experimentation: The Journey of Learning31:06 The Importance of Copy Testing and the Future of CRO35:44 Wrapping Up: An Optimistic Outlook on CRO
Media.Monk's Juliana Jackson shares the truth about app development, optimization, and how web and app should work together.
Fractional Head of CRO, Tracy Laranjo talks about the power of Mentorship for Conversion Rate Optimizers (CROs). We cover: Why would CROs need mentorship?How do you find one?When are you ready to be a mentor and how do you get that word out?Would Tracy be a mentor for our listeners?GrowthMentor: https://app.growthmentor.com/mentees/sign_up?ref=970f4f6f79Contact Tracy:LinkedIn: https://www.linkedin.com/in/tracylaranjo/Email: hello@tracylaranjo.comChapters:00:00 Introduction02:34 Tracy's Journey to Self-Employment06:01 The Value of CRO Mentorship09:45 Finding the Right Mentor13:09 The Mentor-Mentee Relationship18:14 Transitioning from Mentee to Mentor23:06 Tracy's Offer to Mentor
🔥 Check out this insightful interview with Bryce York, Director of Product Management at Tatari, on Experiment Nation’s Conference! Here are 5 key takeaways from the interview:The similarities and differences between CRO and product management.The importance of overlapping skills and work in both areas, especially in smaller companies.Bryce’s background and journey from entrepreneurship to product management.The significance of qualitative and quantitative data in experiment design and analysis.The value of documentation and knowledge sharing in building an experiment-driven culture.Watch the video to gain valuable insights into CRO, product management, and experimentation! Chapters00:00 - Introduction and Background12:30 - Product Management vs. CRO18:45 - Quantitative and Qualitative Data25:20 - Minimum Viable Experiment vs. Minimum Viable Product31:34 - Prioritization and Risk Assessment34:38 - Getting Smart vs. Getting Lucky38:54 - Experiment Log and Knowledge Sharing#CRO #ProductManagement #Experimentation #MarketingSkills #DataDrivenMindset
Documentation. The unsung, yet incredibly important part of any experimentation program. Great documentation allows you to look back at past experiments and find patterns of what were the most effective levers to inform your testing roadmap and levers you probably can deprioritize. In today's episode of Experiment Nation, our own Charlotte April Bomford recently spoke to Edmund Beggs about the "Levers Framework" which allows Convert.com to deliver results for their clients in shorter amounts of time. If you'd like to learn more about this interesting framework, watch the full interview.
🚀 Exciting Insights from Simon Girardin about learning from failure!🚀📊 Takeaways from the conversation with Simon Girardin, CRO Program Manager at Conversion Advocates:1️⃣ Speak Stakeholder Language: Report results in terms of uplifts, decreases, and revenue. Ensure clarity for stakeholders who may not be as versed in the technical aspects.2️⃣ Learning from Tests: Emphasize the importance of learning from every test. Capture value by reviewing data, segmenting results, and analyzing user behavior through session recordings and heatmaps.3️⃣ Strategic Iteration: When faced with a winning result in a specific segment, be cautious about immediate implementation. Consider the long-term impact on the user experience and potential tech debt. Iteration should be intentional and aligned with overall business goals.🔗 Follow Simon Gerarden on LinkedIn for more insightful content on strategic testing and experimentation.🌐 Connect with Tracy Lano and Experiment Nation for more interviews and discussions with top conversion rate optimizers.Chapters00:00 Introduction and Welcome00:23 Understanding the Importance of Learning from Past Experiments00:56 The Role of Stakeholders in Experimentation01:29 The Importance of Testing Strategically02:34 The Process of Iterative Testing04:34 The Importance of Segmentation in Testing05:39 Interpreting Test Data06:58 The Value of Inconclusive or Losing Tests07:49 The Challenge of Iterating on a Test09:32 The Importance of Prioritizing Hypotheses11:40 The Process of Reviewing Test Results15:00 The Role of Metrics in Testing18:02 The Challenge of Implementing Test Wins25:51 Conclusion and Final Thoughts#ConversionOptimization #Experimentation #CRO #MarketingInsights
Koalatative's Gerda Vogt-Thomas asks Rommil Santiago questions about how he started Experiment Nation and how he manages it while working a full-time job at Optimizely.
Data-backed web design refers to the approach of utilizing empirical evidence and user insights to inform the design and optimization of websites. By analyzing data such as user behavior, click-through rates, conversion rates, and heatmaps, designers can make informed decisions about layout, navigation, content placement, and visual elements. A data-backed approach helps identify user preferences, pain points, and areas for improvement, resulting in enhanced user experiences. A/B testing, user surveys, and analytics tools are commonly employed to gather data and validate design choices. By leveraging data, web designers can create user-centric interfaces that drive engagement, conversions, and overall business success.
In this insightful video, Ishan Goel, transitioning from a software engineer to a researcher, delves into crucial CRO statistical concepts. He sheds light on statistical significance in experimentation, explores A/B testing intricacies, emphasizes the efficiency of sequential testing, and discusses challenges in stopping tests prematurely. The talk concludes by highlighting the intersection of experimentation and statistics in the realm of Conversion Rate Optimization.
In this episode, Mike Green discusses the use of diary studies in UX research, particularly in the context of public sector user research. Mike Green, a freelance user research lead based in the UK, shares his experiences using Google Forms, structuring incentives, and addressing challenges in recruitment for diary studies. The discussion emphasizes the longitudinal nature of diary studies, contrasting them with traditional usability tests, and highlights their effectiveness in capturing user feedback over a period of time. Mike also touches upon the importance of understanding user behavior, attitudinal responses, and the ethical considerations in dealing with vulnerable groups. The planning and design stage for diary studies is emphasized, stressing the need to align the research goals with the chosen methodology. Overall, the video aims to provide insights and tips for conducting impactful UX research using diary studies in the public sector.Chapters00:00 Introduction to Diary Studies with Mike Green03:21 Understanding the Purpose of a Diary Study06:45 Recruiting Participants for Your Diary Study10:12 Analyzing Data from Diary Studies14:30 Key Considerations for Diary Studies Success #UXResearch #DiaryStudies #UserExperience #ResearchablePodcast
Learn from André Morys, CEO of KonversionsKRAFT, on gaining crucial stakeholder buy-in for experiments. André highlights the challenges faced by optimizers and advocates for understanding team dynamics through stakeholder mapping. He emphasizes approaching stakeholders, using empathy to build relationships, and overcoming objections. André shares practical insights on conflict management, including a stakeholder map with color-coded power dynamics. Discover the significance of regular check-ins, fostering trust, and achieving tangible results in experimentation. Master the art of securing support for your experiments effectively.About growth marketing Summit:June 19th, 2024 in Frankfurt: www.growthmarketingSUMMIT.com - with 700-800 attendees it is Europe's #1 real-life conference with a handpicked lineup of great people you will usually not see at one conference.Andre's strategy illustrated:http://experimentnation.com/wp-content/uploads/2023/12/andre-stakeholder-strategy.pngChapters00:00 - Introducing André Morys and the importance of stakeholder buy-in04:12 - Understanding the stakeholder map and its color-coded quadrants09:37 - The impact of stakeholders on experimentation in different company sizes13:45 - Importance of building relationships and trust with stakeholders18:20 - Recognizing objections and crafting strategies to address them24:10 - Navigating implicit forces in larger organizations24:43 - Learn about André Morys’ upcoming Growth Marketing Summit
Will Laurenson shares his journey from startup to becoming a CRO (Conversion Rate Optimization) expert. Will discusses the significance of usability, anxiety, and motivation in CRO, drawing from his extensive experience in running tests and optimizing various businesses. Will's approach to CRO involves deep insights into user behavior and a focus on aligning products with customer needs, challenging the common pitfalls of assuming product-market fit based solely on initial traction or advertising success.Chapters:0:00 - Introduction to Will Laurenson’s CRO Journey2:30 - Transitioning from startups to founding Customers Who Click6:15 - Understanding the usability, anxiety, and motivation in CRO10:45 - Onboarding process and working with clients at Customers Who Click15:20 - The UAM Method: Usability, Anxiety, and Motivation in CROWill's podcast: https://customerswhoclick.carrd.co/Will on LinkedIn: https://www.linkedin.com/in/willlaurenson/
You need to invest in ad optimization.Effective performance marketing / advertising optimization relies on continuous experimentation, data-driven decision-making, audience insight, and refined targeting. It's not just about numbers; creativity, a deep understanding of advertising channels, and ethical targeting are crucial. Brand trust and performance marketing are interconnected, with customer reviews and testimonials playing a significant role. Embracing experimentation and learning from failures is essential for long-term success in performance marketing. Transparency with clients, a focus on the bigger picture, and purposeful action are keys to building trust. Joe discusses the launch of AD Scientists and encourages connecting with other industry professionals.If you want to follow Joe, check him out here:https://www.adscientist.ca/https://www.linkedin.com/in/joefitzpatrick1/
Maximizing results in ad network optimization requires a systematic approach. Start with A/B testing on Google Ads and gradually introduce more complex campaigns, balancing technical and creative aspects. Analyze your channel's performance, ensuring sufficient impressions and conversions for statistically significant results. Shape your product to align with the ad network's requirements and consider user experience. Prioritize velocity in testing initially, gradually enhancing quality, and encourage collaboration across different platforms for valuable insights in the quest for optimization.
Lucas, a senior conversion specialist, shares his journey in the media industry, emphasizing the transformative impact of A/B testing. He discusses myths about experimentation - debunking them along the way as well as highlights the importance of validation, integration into essential projects, and showcasing negative results. Lucas encourages testing the obvious and unexpected, questioning assumptions, and showing the value of experimentation to reduce risk and aid decision-making, making it an asset for senior managers and specialists.
Personalization timing mattersEric Melchor, an evangelist at Optimonk, emphasizes the importance of focusing on the product page for effective conversion rate optimization (CRO). He suggests treating website visitors as individuals and creating a personalized and enjoyable customer experience in real-time. Melchor also discusses the significance of capturing visitor interest early on and leveraging smart CRO tactics and personalization timing throughout the customer journey. Additionally, he mentions the value of optimizing the product page, as it can yield significant returns on investment.
We explore various angles of CRO Organizational Structure. We discussed primarily the differences between centralized CRO teams vs a company-wide experimentation culture, and how this is best looked at as a continuum. Companies starting out might benefit from a centralized CRO team whereas more mature optimization programs can benefit more from working experimentation into all facets of the company. How can a company who does 0 experimentation introduce and adopt an experimentation process/culture? We also chatted about experimentation at the customer level, what this means, and how it could be the future of optimization.
Heatmaps to the rescue! Creating an impactful website hero section above the fold is vital for user engagement. Clear value propositions and compelling calls to action are essential, as users often don't scroll past this section. Heat mapping, combined with other techniques like exit polls and A/B testing, provides a comprehensive understanding of user behavior. While urgency and scarcity tactics can be effective, users have become more skeptical of overused strategies like countdown timers. Understand your audience through analytics data and tailor your optimization strategies accordingly, whether it's for B2C, B2B, or other sectors. Mobile interactions may have imprecise heat map data, so use it alongside other sources for a well-rounded view.
Experimentation Culture for Startups is so important.In the realm of Conversion Rate Optimization (CRO), engaging all stakeholders, from executives to product managers and customer service, is pivotal for success. This inclusive approach fosters collaboration, early alignment on technical aspects, and efficient experimentation. It stands in stark contrast to tactical CRO, which tends to focus on short-term projects and lacks a comprehensive strategy. Strategic CRO, on the other hand, entails research, testing, iteration, and transparent communication, forming a continuous and value-driven process. When selecting a CRO agency, clients should prioritize those that emphasize strategic approaches, effective utilization of test results, and well-structured workflows. This approach transforms CRO into an integral part of an organization's culture and operations, creating enduring value.
Having an experimentation roadmap is crucial for conversion rate optimization and Hypotheses Testing because it provides a structured and strategic approach to improving conversion rates on a website or marketing campaign. Here are a few key reasons why it is important:
Systematic Testing: An experimentation roadmap helps in organizing and prioritizing the different experiments that need to be conducted. It ensures that testing is done in a systematic manner, focusing on one element at a time, and allows for clear tracking and analysis of results.
Data-Driven Decision Making: By following a roadmap, you can collect valuable data through controlled experiments. This data helps in making informed decisions based on actual user behavior and preferences, rather than relying on assumptions or guesswork. It enables you to understand what works and what doesn't, leading to effective optimization strategies.
Resource Allocation: An experimentation roadmap helps allocate resources efficiently. By planning ahead and setting clear goals and timelines, you can allocate resources such as time, budget, and personnel effectively. It prevents random testing or haphazard changes that may waste resources without generating significant improvements.
Continuous Improvement: Conversion rate optimization is an ongoing process. An experimentation roadmap allows for continuous improvement by setting up a cycle of testing, learning, and iterating. It ensures that insights gained from previous experiments are applied to future iterations, leading to cumulative improvements over time.
Alignment and Collaboration: A roadmap provides a shared vision and plan for the conversion rate optimization efforts. It aligns stakeholders, such as marketers, designers, developers, and executives, towards a common goal. Collaboration becomes easier as everyone understands the testing priorities, expected outcomes, and their roles in the process.
In summary, an experimentation roadmap is essential for conversion rate optimization as it provides a structured and data-driven approach to testing, decision making, resource allocation, continuous improvement, and collaboration. It maximizes the effectiveness and efficiency of optimization efforts, leading to better conversion rates and overall business success.
Landing pages are, generally speaking, a page that users “land on” after clicking on a link in an email or an ad. What we’re talking about today, however, are standalone pages that are specifically designed to motivate, build trust, create urgency, and ultimately convert visitors. They matter because, when used correctly, they can significantly improve the effectiveness of ad campaigns and maximize the ROI of ad spend.00:00 - Introductions1:19 - What else is going on1:47 - What is a landing page?2:28 - What happens when you don't use landing pages?3:40 - How should eCommerce companies leverage landing pages8:27 - Common mistakes9:15 - Ways to improve landing pages10:15 - How to reach Claire
Don't test everything. Experimentation isn't about proving how clever you are. Mistakes in testing ideas from your own head are common; personal bias clouds results. High-performing experimentation needs executive buy-in and integrated roles. Successful teams engage diverse players with vested interest, ensuring hypotheses formation and metric selection are comprehensive. Valuable experimentation saves time by testing what not to do. Rather than just chasing conversion lifts, recognize when improvements won't make a difference. Cultural shifts, humility, and prioritizing efficient use of time are key components of effective experimentation programs. Test what makes sense.
In this podcast episode, Mark Eltsefon, a senior data scientist and experimentation evangelist, discusses building an experimentation culture in different companies. He shares his journey from starting as a software developer to becoming passionate about data science and experimentation. He contrasts the experimentation culture at TikTok, where data-driven decisions were central and democratization was encouraged, with his current role at an e-commerce company specializing in print-on-demand. He emphasizes the importance of educating stakeholders about data-driven decision-making, and he highlights the need for strong experimentation infrastructure and high-quality hypothesis testing. He believes that while democratization is valuable, it's important to balance it with maintaining the quality of ideas and statistical rigor.
Find Mark on Medium and LinkedIn: - https://medium.com/@markeltsefon - https://www.linkedin.com/in/mark-eltsefon/
Juliana Amorim emphasizes that personalization is not a silver bullet and recommends its use when aligned with strong hypotheses and business goals. She co-founded Croct, a personalization experimentation platform, focusing on adapting user journeys to context. Examples include tailoring websites for diverse buyer personas within B2B companies and customizing content for users at different stages, like newcomers or existing customers. AI-based personalization and rule-based personalization can coexist. Brands should prioritize personalizing their websites and consider first-party data support.
Building a personal brand as a conversion rate optimizer is crucial in today's competitive digital landscape. Your personal brand serves as a powerful tool to establish credibility, showcase expertise, and attract new opportunities. By cultivating a strong brand identity, you position yourself as an authority in the field, gaining trust from potential clients or employers. A well-developed personal brand helps differentiate you from others, highlighting your unique approach and methodologies. It opens doors to collaborations, speaking engagements, and thought leadership opportunities. Additionally, a strong personal brand creates a lasting impression, leading to increased visibility, a larger professional network, and ultimately, a steady stream of clients or job offers.
Ideation strategies are essential to building an experimentation culture. Companies should encourage employees to generate new ideas and provide a platform for sharing them. This can be done through brainstorming sessions, hackathons, or online idea-sharing platforms. It's important to involve stakeholders from different departments and levels of the organization to ensure diverse perspectives are considered. By implementing ideation strategies and fostering a culture of experimentation, companies can drive innovation and stay ahead of the curve.
Building an experimentation culture is crucial for companies that want to innovate and stay ahead of the competition. It starts with creating an environment where experimentation is encouraged and failure is seen as an opportunity to learn and improve. Teams should be given the freedom to try new ideas and test them quickly using small-scale experiments. It's important to collect and analyze data from these experiments to make informed decisions about what works and what doesn't.
Lucas Vos interviews Lucia van den Brink in this great AMA session.
Eddie Aguilar, a developer and CRO specialist, shares his experience with experimentation and optimization. He discusses how he started as a developer at a young age and eventually shifted to optimization. He recounts a specific incident while working with Envision App, where he made a global change to the website that caused a significant error, resulting in a spike in error logs. Eddie emphasizes the importance of communication and collaboration in such situations. Despite the error, the test yielded impressive results, with a 227% revenue lift and 70% LTV lift.
David Mannheim, author and expert on personalization, discusses the importance of personalization in marketing. He emphasizes the need for brands to focus on building relationships with customers based on trust and care. Mannheim distinguishes personalization from segmentation, highlighting personalization as a communication principle rather than just a marketing tactic. He also mentions the challenges faced in implementing personalization, such as the pressure for immediate returns and the balance of metrics.
In this video, Riccardo Vandra discusses the importance of asking powerful questions for conversion rate optimization (CRO). He emphasizes that the quality of our questions determines the quality of our insights and ultimately our success. Riccardo proposes a three-step questioning framework for building effective questions.
We sat down with Jason Chappel, founder of Defiant, a specialist Customer Value Optimization agency, to discuss what sets CVO apart from CRO. Jason shares his insight on the importance of understanding and optimizing the entire customer journey, from acquisition to retention, and how CVO can help businesses increase customer lifetime value and drive sustainable growth in your business strategy.
Mixing quantitative and qualitative research methods is essential for understanding human motivation. While quantitative methods provide statistical data and measurable insights, qualitative methods capture nuanced experiences and emotions. By combining both approaches, researchers gain a comprehensive understanding of the complex interplay between factors such as social, psychological, and cultural influences on motivation.
The best Product Managers and Experimenters possess a unique skill: they know how to tell good stories that capture the hearts and minds of stakeholders and gain buy-in. Through powerful storytelling, they paint a vivid picture of the product's vision, value, and potential impact. They tap into emotions, using anecdotes and examples to create relatable narratives that resonate with their audience. They highlight the problem, solution, and user benefits in a way that is clear, concise, and memorable. Their stories inspire trust and confidence, compelling stakeholders to align with the product's vision and goals. In this way, storytelling becomes an invaluable tool for Product Managers to generate buy-in and drive successful product outcomes.
Optimizing forms during checkout is crucial for a seamless user experience. By simplifying and streamlining the process, businesses reduce cart abandonment rates and increase conversions. Shortening form length, utilizing autofill options, and providing clear instructions enhance efficiency, minimize user frustration, and ultimately boost customer satisfaction, loyalty, and revenue.
Nicoleta walks us through a case-study where she explores a 1 column form vs a 3 column form in checkout.
checkout #forms
Learn A/B Testing Statistics, how to use Overall Evaluation Criterion, Twyman's law, how much traffic do you need to run A/B tests, and other statistical concepts all Experimenters must know.
Watch the video version here: https://www.youtube.com/watch?v=tH4tjpbziCo#statistics #abtesting #cro
Are you looking to replace Google Optimize? On Experiment Nation's recent podcast, Charlotte April Bomford chats with Kenneth Kutyn about replacing Google Optimize, how to select a new vendor, and whether it's worth building your own Experimentation Platform (spoiler alert: the answer is usually no).#googleoptimize #vendor #experimentation
Tim Thijsse, a senior CRO (Conversion Rate Optimization) consultant at Orange Valley, discusses AI and the future of CRO, and the importance of incorporating qualitative research into company-wide processes. He emphasizes the challenge of linking user behavior and mindset to knowledge and information. Tim shares his background, starting as an interaction designer and later transitioning to UX design and CRO. He highlights the evolution of CRO, the need for a culture of experimentation, and the role of qualitative insights in optimization strategies. Tim also discusses the challenges companies face in adopting an experimentation mindset, such as aligning strategy, operations, and technical aspects, and the importance of having a common metric and collaboration among teams.
Ayat Shukairy emphasizes the importance of addressing the functional, social, and emotional aspects of customer needs when designing websites. She shares seven lessons learned from her experience in conversion rate optimization.
Georgi Georgiev performed a meta-analysis of over 1000 A/B tests and has a great deal to share about:- What you should expect your win rate to be- The benefits of sequential testing- Why you should report on confidence intervals
Many businesses approach conversion optimisation by hiring a single in-house practitioner and believing that they can run all aspects of the discipline. But what you also need is Executive buy-inMost CROs are generally very multiskilled because that have had to learn to be so, however this approach typically leaves them frustrated because they lack the buy-in from an executive, support of specialist resources required to take their craft to the next level.In truth, optimisation requires a team of very different skills and ways of thinking. This doesn't mean you have to hire a big internal team, but you do need to forward plan where and how you are going to get these skills from when you need them, and need them you will.You will learn about the comprehensive set of skills and resources needed to scale a data-driven approach.About JonnyJonny is Conversion Director at Journey Further, a performance marketing agency working with the world's leading brands, including the likes of Liberty London, Krispy Kreme and Lick Home. During his 15 year career of improving websites with strategy, experimentation and data, he build and led the conversion team at Sky, as well overseeing the ecommerce strategy and operations at Principal Hotel Company.Chapters---0:00 - Introduction3:49 - How can experimenters tackle bigger business questions?4:41 - 2 key ways that experimentation goes wrong7:38 - What do companies usually use CRO and Experimentation?9:45 - What is the ideal situation?10:21 - Replicating the scientific approach in a business context12:41 - A concrete example17:38 - Recap
Experimenters obsess about many A/B Testing Statistics concepts - but not all of them are worth worrying about. Statsig's Timothy Chan shares his thoughts about test isolation, interaction effects and confidence levels - some of his thoughts may surprise you.#experimentation #product #statistics
Client Relationship Tips and secrets behind a thriving relationship between a CRO agency and the client - anybody can learn how to get started in CRO but it takes years of experience to know how to run a CRO agency and how to scale while keeping the same standards of quality. In my presentation, I will walk you through what you need to know to build a strong, successful relationship with your clients and with your team. You will learn: - how to set expectations with your client from Day 1- how to leverage operating procedures and automate your work- how to communicate with clients and get stakeholder buy-in every timeChapters---0:00 - Introduction1:52 - The testing Masterfile3:37 - The testing sheet template
10 lessons that helped John Ekman in Launching a CRO agency. John learned many lessons when launching his agency - he wants to share them with you.
After decades of acquisition marketing, companies are waking up to the fact that optimising the entire lifecycle of the customer is the way to go. That happens due to rising acquisition costs, cookie deprecation, siloes, company mentalities, etc.In my session, I will share the insights that I got during my 12 years journey, transitioning from CRO - website optimization to increase the conversion rate to CVO - experimenting towards improving the CLV.About the speaker:I am the CEO & Founder at Omniconvert, a solution that helps eCommerce companies all over the world become customer-centric and thrive in a cookie-less future. I am also the founder of Omniconvert CVO Academy where he helps marketers become certified in CVO through his teaching of the importance of customer lifetime value.On top of this, he is writing, blogging, podcasting, and advising for brands. I have a genuine calling to help others and share the shortcuts I discover on my journey.Chapters---0:00 - Introduction3:59 - Experimentation fatigue4:53 - A/B Testing is finding the one size that fits all5:29 - The dominant numbers in eCommerce8:50 - Aha moment10:07 - RFM customer segmentation12:50 - The soulmates14:12 - Qualitative and Quantitative17:23 - The CVO Methodology20:06 - The conversion rate can be a misleading metric in eCommerce21:53 - Customer value optimization22:20 - The 4 types of CVO campaigns24:53 - 4 reasons why the CRO experts are in pole position to nail CVO
Message-Market fit is the idea that a message, whether it be a product, service or idea, needs to resonate with a specific target market in order to be successful. It involves finding the right message that effectively communicates the value proposition of a product or service to a particular audience, in a way that resonates with them.This process involves identifying the unique needs and preferences of the target audience, understanding their pain points and motivations, and crafting a message that speaks directly to them. The message should be clear, concise, and compelling, and should effectively communicate the benefits of the product or service.Achieving message-market fit requires a deep understanding of the target audience and ongoing feedback and iteration to refine the message until it resonates with the market. When a message effectively resonates with a target market, it can lead to increased engagement, loyalty, and ultimately, growth.It is important to note that message-market fit is not a one-time event, but rather an ongoing process that requires continual testing, refinement, and adaptation to changing market conditions. Businesses that prioritize achieving message-market fit are more likely to create products or services that are successful in the long term.About Daphne Tideman:Teaching D2C Startups how to grow faster | 10x the subscriber base of a startup in 18 months | Worked with 200+ brands | Eco-friendly and wellness brands specialist | AuthorLinkedin: https://www.linkedin.com/in/daphnetideman/#marketing #growth #messaging #experimentation
A/B testing plays a critical role in decision-making - but so does AB Testing Statistics Mistakes. In data-driven companies, A/B tests not only prove positive impacts of product features (treatments) but also provide evidence to safely roll out neutral features that are not expected to move business metrics. When rolling out a neutral feature, we need to test if the treatment is non-inferior to control, and conventional A/B testing is not suitable in this case. In my talk, I present when conventional A/B testing would fail, what the non-inferiority test is, and how is non-inferiority test better when testing treatment is not worse than control. We've published this topic here https://bit.ly/3vmFsYpAbout the speaker:Max is a senior data scientist at Wish where he focuses on experimentation (A/B testing) and machine learning. He has been improving the A/B testing platform at Wish on various fronts, including infrastructure, statistical testing, usability, etc. His passion is to empower data-driven decision-making through the rigorous use of data. Max earned his Ph.D. in Statistical Informatics from the University of Arizona.
Add Exploration and Validation phases to your testing program - because most of your wins are fake.#experimentation #statistics #testing
Being an Experimentation leader requires a different set of skills. Vistaprint's Cristina Pujol walks through the lessons she's learned.
0:00 - Introduction1:03 - How Vistaprint organizes for Experimentation2:22 - The Ambassador role4:19 - Experimentation process for domains8:18 - Let’s take a real example11:36 - Ideas to share learnings and celebrate success
leadership #program #experimentation
Our industry can be damning of those doing things ""imperfectly"" - but it seems not to draw the distinction between those doing things imperfectly through ignorance (who can be educated) and those doing things imperfectly through lack of a better option (who need to be helped to show the value to decision-makers). This attitude has a negative impact on new entrants to the industry and sets completely the wrong tone for the next generation. The greater the participation in the industry, the stronger it will become - the wider that participation, the more ideas (and the more diverse those ideas) will come through. And that is only a good thing for us all longer-term. About Matt: Matt Scaysbrook, Founder & Director of Optimisation at WeTeachCRO Matt has been optimising online experiences across B2B & B2C industries for over 10 years. He has a great passion for guiding & educating businesses on the value of Conversion Rate Optimisation. He places the understanding of human behaviour at the heart of all of his work, knowing that that breaks all problems down to the lowest common denominator – us. Clients include Nando’s, GoDaddy and Sage. Chapters --- 0:00 - Introduction 0:55 - The quality of our community is a function of quantity and diversity 2:53 - The purist 8:09 - The pragmatist 12:58 - So what? 16:36 - We need the best of both
In this session, I want to cover the most strategic actions that a new Experimentation Strategist (or Program Manager) can execute during their first three months in the role to increase trust among the stakeholders and clients. I will share this based on my experience working with clients such as Miro, Procore, The Bouqs, and Gympass, among others. About the speaker Carlos has over nine years of experience working in online marketing and mass media. As an agency founder, he has led web development and conversion optimization projects for more than 80 brands in multiple regions and verticals. It has provided him with broad expertise in conversion optimization, user research, ecommerce strategy, digital psychology, and data analysis that let him contribute to driving innovation that delivers more profits. As an Experimentation Strategist (former Program Manager) at Speero, he has been actively involved in running the Experimentation Programs of fast-growth companies such as Miro, Procore, The Bouqs, and Gympass, among others.
You don't have to A/B test everything. Be smart by having guiding principles. #conversionrateoptimization #experimentation #ecommerce
Many optimizers are good at using data to craft hypotheses. Great optimizers, however, build data-driven segments, analyze by business goals (not just test KPIs), and importantly, propose how to take action on campaign results. About the speaker Alexandra leads Product Marketing at Kameleoon. She loves storytelling as a way of demonstrating a product’s values and benefits to customers. Chapters --- 0:00 - Introduction 3:41 - Great experimenters consolidate their customer data for precise and persistent segmentation 6:40 - Great experimenters analyze by meaningful business goals and draw actionable conclusions 8:05 - Great experimenters take action on their experimentation results 10:29 - Coming to full circle
Having mature conversations with immature markets, the problem with the word, “Hypothesis”, the importance of HiPPOs with Nima Yassini #experimentation #conversionrateoptimization #growth
Life optimization involves setting goals and creating plans for reaching them. It also involves improving physical, mental, and emotional well-being. Ruben de Boer shares with our own Richard Joe how he leverages a CRO mindset to engage in activities that bring purpose, satisfaction, and joy to life.
Creating a customer journey map can help stakeholders better understand the customer’s experience, and helps ensure that all stakeholders are working towards the same goal. It can also help identify potential areas of improvement, and provide insights into how customers interact with your product, service, or organization.
Chapters --- 0:00 - Introduction 2:02 - This session is for you if… 2:42 - Pre-requisites for a successful job hunt in CRO 4:27 - It’s ok to job-hop 8:31 - Get clear on your shit 9:41 - Know what experience you want 10:56 - Know what you stand for 11:32 - Know your non-negotiable 13:28 - Show what sets you apart 18:20 - Apply thoughtfully 20:19 - Own the interview 23:22 - Seal the deal
Richard talks to Alun Lucas from Zuko Analytics about Form Optimization #analytics #experimentation
Teams are horrible at this! They have no idea it should happen, and if they do, they're usually lost and the approach is terrible. When you break it down, it's really not difficult. This shouldn't be a black box anymore. I will cover the following: (1) What each of the test planning aspects are and how to do them including MDE and duration estimates, (2) CXL's pre-test calculator unless our new Optimizely one is live by the time I record, (3) how to turn pre-test calculations into a testing roadmap, and (4) how to determine when a business can't test or when a swimlane isn't viable so teams can stop wasting time. Chapters --- 0:00 - Introduction 1:50 - Imagine you’re with your team building an experimentation program 2:30 - Common struggles 6:26 - What is pre-test planning? 6:56 - Primarily…MDE and Duration estimates 26:28 - Roadmapping 28:38 - Should we test everywhere? Should we test at all? 31:00 - Example scenario 32:20 - Wrap up
Chapters --- 0:00 - Introduction 2:06 - What is skeptical? 3:08 - Why first-time buyers? 4:15 - So how do we convert these skeptical first-time buyers? 5:44 - People are skeptical of “too good to be true” 6:36 - They find expertise sexy 7:30 - They root for people who beat the odds 8:35 - They are fascinated by surprising details 10:16 - They are visual animals 10:54 - They need motivation to break habits 12:10 - They love personalized experiences 12:25 - They like knowing they’ve stumbled onto something rare 13:08 - We must resolve their negative thoughts 14:20 - How we target skeptical first-time buyers?
Our new podcast host, Richard Joe, asks Ton about his career and dives into the culture of Experimentation, and how we have to be more inclusive.
There’s a lot of talk about how to find things to test, how to analyze tests and the importance of iterating on tests. This week, I was thinking about how I could add something a little different to the conversation and decided to share my thoughts about when to run a test - particularly when building products.
Curiosity needs to be a driver in every experimenter’s toolkit. What is going on in the head of my customer? What is really their job-to-be-done? When buying flowers, we typically think of the standard occasions: Mother’s Day, Valentine’s Day, new baby & sympathy. But if we were to structure our ecommerce website on those occasions alone, we may be addressing the majority, but we’d be missing out on the real motivation behind the purchase as well as other reasons people buy flowers. By deconstructing the customer’s journey of buying flowers, we discovered a piece of data that was sitting in front of us the entire time… the card message. These wonderfully written messages were our customers literally telling us the reason behind their purchase. It drastically changed how we approached our ecommerce website, now centered around the gifter’s journey.
Even if you are the best Experimenter at your company, unless you are able to nurture a culture of Experimentation - you’ll end up constantly fighting uphill battles. Here are a few quick pointers on how to be a better Experimentation leader.
Pineapple on pizza. If you ever want to have a heated debate with a group of people, ask them if pineapple belongs on pizza. Now let’s replace pizza with CRO and pineapple with “best practices” (or whatever topic CROs are angry about on LinkedIn these days).
About this series: Rommil Santiago, the founder of Experiment Nation shares his most recent thoughts about the industry and the craft of Experimentation in less than 5 minutes.
Based on a brand new obsession (thanks to the pandemic) for homemade pizza, I would like to talk about how optimisation is just like cooking that perfect pie. You can have all of the tools in the world, but do you have the experienced chef and the right ingredients. Sure, you can cook up a quick Margherita in under an hour, but does it taste good. But, what if you left the dough to rise for a few days? Well, you’d get a much better flavour, and a smooth experience. Optimisation and Pizza Making are the same and I will help you understand how and why it is beneficial to let your dough prove for longer.
About this series: Rommil Santiago, the founder of Experiment Nation shares his most recent thoughts about the industry and the craft of Experimentation in less than 5 minutes.
Florent Buisson is a decade-long behavioral economist who published a practical guide to behavioral data analysis in business. It's the book he wished he had when he started in analytics.
In this episode, he shares:
Plus other statistical concepts explained in a way even a five-year-old could understand.Why intuition can actually be a good thing in data analytics
If you need to know R or Python to understand your users
Why and when causal analytics are important
--
Florent Buisson is a behavioral economist with 10 years of experience in business, analytics, and behavioral science. He most recently started and led for four years the behavioral science team of one of the largest insurance companies in the US; he then published Behavioral Data Analysis with R and Python (O'Reilly), the book he wished he had had when he started. Florent holds a PhD in behavioral economics from the Sorbonne university in Paris.
Find Florent on LinkedIn: linkedin.com/in/florentbuisson
Read Florent's content on Medium: medium.com/@florent.buisson
About this series: Rommil Santiago, the founder of Experiment Nation shares his most recent thoughts about the industry and the craft of Experimentation in less than 5 minutes.
A/B Testing can be intimidating to the uninitiated, but it doesn’t have to be! I want to introduce testing concepts via a real-world scenario that we can all relate to: Dessert. I’ll cover what an MVT is, hypotheses, independent variables, dependent variables, results (of course), and design flaws. And if you need any more convincing, I have four words for you: Monte Cristo Bread Pudding.
Matt Green and Moshe Mikanovsky of Product for Product podcast help you navigate the plethora of A/B testing tools out there.
Whether you're in product or marketing, listen for a beginner-friendly comparison of tools like Google Optimize, Optimizely, and more.
This episode, nor those of Product for Product, are endorsements of any of the discussed tools.
Listen to Product for Product: spreaker.com/show/product-for-product
Find Matt on LinkedIn: /in/mattgreenanalytics
Find Moshe on LinkedIn: /in/mikanovsky
You can do better than testing button colors, but how do you decide what to test when, and what type of test to run? Roadmapping. This session teaches you to be the GPS navigator, helping your team optimize experimentation routes. On this journey, you will identify how each path can drive desired impacts by turning a formidable trek into a series of attainable milestones.
Presented by
- Kenya Davis, Microsoft Digital Analytics
- Planning Program Manager, and Evolytics Decision Science Director Krissy Tripp.
Experiment Nation: The Podcast is a show on which we share Interviews with Experimenters from around the world, opinion pieces, and recent presentations with you - the experimentation community. We hope you enjoy the show, let’s get to it.
If you enjoy our content, consider signing up for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ
Tracy is a Toronto-based growth marketer with a knack for conversion optimization. In a past life she’s QA tested “adult” eBooks and helped make it easier for Canadians to write their wills online. Tracy is also one of the co-hosts of the Experiment Nation podcast where she chats with optimizers at the top of their game.
Beyond the 9-5, you can catch her knitting, crushing empires in Civ6, or napping with her cats.
About this series: Rommil Santiago, the founder of Experiment Nation shares his most recent thoughts about the industry and the craft of Experimentation in less than 5 minutes.
eCommerce CRO veteran Johann Van Tonder shares what usually goes wrong when coming up with test ideas and how to fix it. We also discuss the merits of quantity vs. quality of ideas, the problem with brainstorming, and what to do instead for more effective ideation sessions.
Find Johann on Twitter: @amaclickclick
Learn more about AWA Digital: awa-digital.com
Anil is a Digital Marketing and Analytics professional with 18+ years of experience. He has worked with companies ranging from startups to Fortune 500 and has helped them improve marketing results as well as develop data driven culture. He has vast experience in Web, Email, Social, Display, Search, Paid Search, and CRM. He holds a B. Tech in Electronics and Communication Engineering from India and an MBA from the University of Washington, Seattle. He is the founder of https://academy.optizent.com – A Digital Analytics training and education company.
About this series Rommil Santiago, the founder of Experiment Nation shares his most recent thoughts about the industry and the craft of Experimentation in less than 5 minutes.
When experimenting with our products, we need to find our users but it’s not always that simple! Where do we find the users? In this session Moshe will go over different types of products and different ways to find users for each of them. Coming out of this session, attendees will have some ideas on where to start looking for their users.
About this series
Rommil Santiago, the founder of Experiment Nation shares his most recent thoughts about the industry and the craft of Experimentation in less than 5 minutes.
Although the two sample proportion test is well known within the marketing experimentation community, there are a host of categorical data analysis techniques that are amenable to the analysis of A\B test data. This presentation provides a quick overview of Chi-square analysis and a discussion of its use cases in marketing experimentation. Specifically, we discuss how to 1) test a\b\n experiments without inflating Type I error after multiple comparisons, 2) analyze tests with more than dichotomous outcome variables and 3) the process for calculating goodness-of-fit and tests-of-independence. Simple guidelines will be provided regarding test assumptions and use case scenarios.
Sina Fak is the Head of Optimization & Business Intelligence at ConversionAdvocates, the #3 top CRO agency in the world in 2020. Sina shares what businesses get wrong when they hire a shiny new in-house CRO strategist, what to look out for before taking an in-house CRO role, and when a business should consider going agency vs. In-house for CRO.
Learn more about ConversionAdvocates: https://conversionadvocates.com/
Find Sina Fak on LinkedIn: https://www.linkedin.com/in/sinafak/
If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ
We know that the key skillsets you need for any successful CRO / Experimentation program are: Research, Design, Development and QA. Whilst larger companies who can afford to hire all of those skilled people, for smaller companies, it becomes a big challenge. So much so, many companies exhaust their options of running smaller tests and then stop doing all kind of experimentation work. In this session, I will be covering the areas where the companies can reduce their cost of experimentation, things that they will need to consider and increase their profitability of running a successful experimentation program.
If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ
Rishi Rawat is the product page guy. With his 13 years of running A/B tests, Rishi shares what makes an effective product description, his go-to method for segmenting product pages, and how to stand out to your visitors during their most “promiscuous state”.
Learn more about Rishi's work at frictionless-commerce.com.
A lot of research and data analysis has gone in to answering this question. But yet, there’s no clear, definitive answer. In attempt to resolve this question once-and-for-all, this session will present and interesting and surprising answer. Applying the results from a real-life A/B test case study, digital marketers will be prompted to think not only about email send time, but also their audience and how both aspects impact conversions.
---
If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ
Finally, remember to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ
Welcome to "Where I Started", the interview mini-series in which we get behind the scenes and find out how some of today's experimenters got into CRO. This should be a fun little project that Ontrack Digital started with Experiment Nation to give you a bit of a getaway from all those test results. But also it can be a source of inspiration for those of you that want to get started in CRO and don't really know which path to take.
If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ
This session is about the conundrum that “data is everything” – a mentality circulating at large the past couple years. There’s so much data, and we often find ourselves simply staring at metrics looking for “+” or “-” to make decisions. The objective of this session is to draw on thinking strategies from physics, of how to approach CRO differently, without any real “data”, to then gleam positive impacts to CRO.
Finally, remember to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ
The success of your testing program depends on the quality of your hypotheses. But too many experimentation programs lack the processes and frameworks to move from spaghetti testing to strategic experimentation. Haley shares two practical frameworks to help you form better hypotheses and objectively rank test ideas.
If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ
Aditi Gupta covers the creation of a growth roadmap/ how to build alignment with stakeholders/ creating an org-wide impact.
---
If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ
Welcome to "Where I Started", the interview mini-series in which we get behind the scenes and find out how some of today's experimenters got into CRO. This should be a fun little project that Ontrack Digital started with Experiment Nation to give you a bit of a getaway from all those test results. But also it can be a source of inspiration for those of you that want to get started in CRO and don't really know which path to take.
If you liked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ
Speero's Director of CX Optimization, Edgar Spongolts, dives deep into the influential agency's Experimentation Maturity Audit and why maturity matters. Learn about the four pillars of a successful experimentation program and what it takes to reach the highest levels of maturity. Take the audit yourself here: https://speero.link/ExperimentNation
If you looked this episode, don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ
Too often experiments are squandered. But if you put customer problems first and design the right type of test you can ensure every experiment is valuable to your business. The following talk will give your experimentation programs guardrails to make sure you’re not testing for testing sake. In this session Annika and Paul will share; How to map out business goals and customer problems. How to pick the right type of experiment. e.g when to use tree testing, copy testing, 5 second tests, A/B tests etc. How to gain insights faster and deliver more effective experiments.
Don't forget to signup for Experiment Nation's bi-weekly newsletter here: https://buff.ly/31qO0AQ
Experiment Nation: The Conference Reloaded
Everybody is doing it, so you have to do it too, right? As my mom used to say: you’re not everyone! Best practices are not experimentation and you should know that. I want to prove to you this is the easy way, but it’s not the right way for a healthy, profitable and experiment-oriented business. These tactics don’t last long. In this session, we will discuss why, statically, best practices don’t work. It’s fun to know about the competitors, their results and ideas. It may generate new ideas and grab your attention when you’re starting to experiment, but that’s the only good thing. To have a great experiment culture, you need to learn from your tests. Let’s talk more about this?
If you liked this interview, signup for Experiment Nation's bi-weekly newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://lnkd.in/g6rM8w25
Experiment Nation: The Conference Reloaded
CRO should be the foundation that all of your other digital marketing efforts rests upon. Through what we call our ‘Ripple Benefits’, I’d love to showcase just how CRO not only improves your conversion rates, but your SEO, Content Marketing, Paid Ads, and more. Testing is key and should be the foundation that drives where your efforts are focused. -
If you liked this interview, signup for Experiment Nation's bi-weekly newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://lnkd.in/g6rM8w25
Petra Lylander is a growth marketer who developed the experimentation process at Helsinki-based SaaS company, Aiven. In this episode she talks about process documentation for experimentation and how to get other teammates to care about it.
If you liked this interview, signup for Experiment Nation's bi-weekly newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://lnkd.in/g6rM8w25
Experiment Nation: The Conference Reloaded
AKA Why you should approach research like an Anthropologist How ethnographic research helps you understand why people think the way they think (not just what they think or how they respond to triggers) to create more meaningful hypotheses and experiments.
------
If you liked this interview, signup for Experiment Nation's bi-weekly newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox:https://lnkd.in/g6rM8w25
Experiment Nation: The Conference Reloaded
Experimentation post-conversion or how to switch from buying journey to customer journey.
Experiment Nation: The Conference Reloaded
A quick and dirty guide to pointing out bad practices that create friction to the customer when brands practice the company journey instead of the customer journey.
Nesse episódio nosso convidado André Gillich, Head of CRO, compartilha sua experiência na Jussi crescendo a cultura de experimentação, começando pela nova abordagem em que “CRO não é um profissional e sim um processo.”Exploramos o impacto da cultura “Agil” e o desafio do foco em outputs não outcomes.Fechamos o podcast de forma prática compartilhando iniciativas tomadas em nossas empresas para acelerar a flywheel de experimentação.
- Edição: João Pedro AlencarReferências citadas:
- Mestre GP https://www.mestregp.com.br/
- Product arena https://productarena.io/
- It takes a flywheel to fly https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/it-takes-a-flywheel-to-fly-kickstarting-and-keeping-the-a-b-testing-momentum/
- Cursos PM3 cursospm3.com.br
- Naval at Tim Ferris https://open.spotify.com/episode/1yTaZvBKLTwVQHcNTvZ19i?si=wF_acs3ZT7qEGTCa4JUDlw
- Naval monologue https://open.spotify.com/episode/1TCwItjO5uj294IZ7MoZ10?si=3uWYsGhQStumCcRYBFJm5w
- How to get rich (without getting lucky) https://twitter.com/naval/status/1002103360646823936?lang=en
- Guess the test inspiration https://guessthetest.com/
Steph walks us through the challenges of growing Specsavers’ CRO team and how she overcame them, her experimentation red flags, and her take on if CRO certifications are really worth it. Plus: A comparison of the CRO job market between Canada vs. the UK, including how well you can expect to be paid as an optimizer.
Experiment Nation: The Conference Reloaded
CRO, experimentation, A/B testing, research, analytics: these things are all parts of a whole which never really gets discussed. What we’re doing is replicating the scientific method in a business context, and yet we can’t see the wood for the trees. Our industry is focused on the individual elements of what we do and how we do it, but not why we do it and the overall philosophy for it. This is part of the reason that businesses don’t give it the traction it deserves; senior business leaders are not interested in the execution, they’re interested in the overall benefit. Let’s take a step back and think about how to go beyond the detail and drive a more mature outlook on what we do!
We'd love your feedback. Tell us what you thought of the session and of our conference for a chance to win a free Customer Value Optimization course from Omniconvert.com and a chance to be featured on ExperimentNation.com Send us your feedback here: https://forms.gle/Mzo9h33RZ94d471Q7
Experiment Nation: The Conference Reloaded
We'd love your feedback. Tell us what you thought of the session and of our conference for a chance to win a free Customer Value Optimization course from Omniconvert.com and a chance to be featured on ExperimentNation.com Send us your feedback here: https://forms.gle/Mzo9h33RZ94d471Q7
Experiment Nation: The Conference Reloaded
We'd love your feedback. Tell us what you thought of the session and of our conference for a chance to win a free Customer Value Optimization course from Omniconvert.com and a chance to be featured on ExperimentNation.com Send us your feedback here: https://forms.gle/Mzo9h33RZ94d471Q7
Experiment Nation: The Conference Reloaded
As conversion specialists and data analysts, we strive for experimentation- and validation-driven culture. To be successful, it is essential to understand that your job is not just to run A/B tests. A large proportion of our work should be dedicated to change management. However, change is hard. In his presentation, Ruben will discuss the importance of change management guided by an understandable model that you can start using today. He will explain the model step-by-step, using the knowledge you already have. You will find many examples of things you should and should not do to cultivate a culture of experimentation in your organization.
We'd love your feedback. Tell us what you thought of the session and of our conference for a chance to win a free Customer Value Optimization course from Omniconvert.com and a chance to be featured on ExperimentNation.com Send us your feedback here: https://forms.gle/Mzo9h33RZ94d471Q7
Experiment Nation: The Conference Reloaded
We'd love your feedback. Tell us what you thought of the session and of our conference for a chance to win a free Customer Value Optimization course from Omniconvert.com and a chance to be featured on ExperimentNation.com Send us your feedback here: https://forms.gle/Mzo9h33RZ94d471Q7
Tim shares his tips for running low and high maturity experimentation programs, what it’s like to optimize in-house vs. agency, when it is (and is not) ok to make compromises, and to how make your program’s results actually drive action. Plus: Tim’s take on those controversial UI polls on LinkedIn.
Experiment Nation: The Conference Reloaded
Shiva will run through WHY you want to be running an experimentation program based on insight generation and not simply ‘winning’. He’ll also show you how to structure tests to support the growth of an insight-driven experimentation program.
Experiment Nation: The Conference Reloaded
The average success rate of most AB testing programs hovers around 12%. Yep, for every 100 experiments you run, only 12 of them will generate an increase in conversions. That is worse than a coin toss. Too many companies start AB testing hoping that it will deliver a great increase in conversions. A few months later, they are bitterly disappointed. In this session, I will share the lessons we learned by analyzing over 22,000 AB tests on 700 CRO projects. These lessons paint the common patterns that helped us deliver upwards of 50% success rates on AB testing programs. The lessons provide a strategic roadmap for a CRO project as well as tactical details needed while running the day-to-day operations. Key takeaways Attendees will learn what it really takes to run a successful CRO program (Time, resource, management, etc). Attendees will learn the right balance between strategy and tactics when running an experimentation program. Attendees will see an actual project management board running a CRO program
Hoje começamos um novo papo em nosso podcast: como migrar para a área de experimentação? Nosso convidado especial é Fabricio Migues, CRO Manager na Small World Financial Services, conta pra gente como foi sair da área de jornalismo e relações públicas e adentrar na experimentação, como aproveitar seus aprendizados e conhecimentos da área de comunicação dentro de CRO e como isso o tem ajudado.
Edição: João Pedro Alencar Referências citadas no episódio:
Referência citada da minha parte
- https://www.exp-growth.com/
- https://open.spotify.com/episode/4d3Prnlc0VIf6dQSdWdGmb?si=ziE3zbnwSKSmW8P34oQIZg&dl_branch=1
Experiment Nation: The Conference Reloaded
Why is it that all through school, girls outperform boys but that equation is flipped in the corporate world? Are there factors in our nurture (and possibly nature) that play against us? How do these factors play out in the experimentation space? In this session – recommended for women in experimentation and those working with them – Bithika will help identify blind spots and hopes to help women and teams reach their true potential.
Tracy Laranjo chats with Shopify’s Experimentation & Analysis Lead, Shanelle Mullin, about tough buy-in conversations, her take on the “I don’t dream of labour” trend, making experimentation more representative of diverse voices, and how we can empower the next generation of optimizers.
Follow Shanelle on Twitter: @shanelle_mullin
There's no single answer as Conversion Optimization and Experimentation continue to evolve and find fit in the digital era. What resonates with us is, if you're constantly testing and celebrating performance instead of delivery, you're likely to be on the right path of doing it right. In this episode, Marion Ravel, Eden Bidani, and Positive John shared different angles, from e-commerce, copywriting, and product, to structuring and thinking through CRO.
Books mentioned:
- Emotional Design — https://www.amazon.com/Emotional-Design-Love-Everyday-Things/dp/0465051367
- Working Backwards: https://www.amazon.com/Working-Backwards-Insights-Stories-Secrets/dp/1250267595
Neste terceiro episódio contamos com a presença de André Vieira, fundador da Looptimize. Ele nos conta um pouco mais da sua visão sobre:
- Qual o ponto de partida para começar a investir em CRO?
- Popularização da experimentação: heurísticas e boas práticas — pontos positivos e negativos
- A qual área CRO melhor pertence dentro de uma empresa?
Edição do podcast por João Pedro (https://www.linkedin.com/in/joaopedroalencar/)
Referências citadas ao longo do ep:
- Looptimize: https://looptimize.com.br/
- Online Dialogue: https://onlinedialogue.com/
- How to measure anything: https://www.amazon.com/How-Measure-Anything-Intangibles-Business-ebook/dp/B00INUYS2U#:~:text=Written%20by%20recogni
- Post CRO vs Experimentation: https://www.linkedin.com/posts/joaoostrowski_cro-vs-experimentation-fight-activity-6830566831628124160-Ia-z
- GrowthMentor: https://www.growthmentor.com/?ref=johnostrowski
Nesse segundo episódio, falamos sobre gatilhos que fazem empresas procurarem o CRO e desafios para se começar a trabalhar com a área. Discutimos sobre entendimento geral de experimentação em uma empresa sobre o produto "estar pronto" para ser otimizado. Fiquem à vontade para contribuir.
- Editor: https://www.linkedin.com/in/jo%C3%A3o-pedro-de-queiroz-alencar-1852a518b/
- Referências: Agende uma mentoria com John no GrowthMentor: https://www.growthmentor.com/?ref=johnostrowski
- Podcast com Ron Kohavi: https://www.abtasty.com/blog/1000-experiments-club-ronny-kohavi/
- Auditoria de maturidade de experimentação com base no artigo The Evolution of Continuous Experimentation in Software Product Development: https://exp-platform.com/Documents/2017-05%20ICSE2017_EvolutionOfExP.pdf
Tracy Laranjo asks Dexter's Joris Bryon about
- How we need to stop saying Best practices and start saying Common Practices
- His tips on starting a CRO agency and working with a partner
- And how to spot a CRO imposter
Nesse primeiro episódio da série, vamos discutir como caímos no mundo de Otimização de Conversão (CRO), algumas experiências até posições atuais e diferentes perspectivas entre exatas/humanas e seus benefícios.
Editor: João Pedro Alencar https://bit.ly/3jt4BLu
Materiais comentados no episódio:
- Learning how to learn from the Awesome Dr. Barbara Oakley: https://bit.ly/3h2AAk1
- Making websites win, free audiobook on Spotify >> https://spoti.fi/3qzRYzX
- CXL Conversion Optimization minidegree >> https://bit.ly/3Aid9em
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing >> https://bit.ly/3AicXvE
- You should test that https://amzn.to/3x7gTNv
Rommil, Kenya, and Shiva discuss the challenges of speeding up a CRO program, as well as some of the ways they overcome them.
Positive John and Marion talk about their favorite CRO tools, how they evaluate them, and how to get clients to pay for them.
Jaya and Sid chat with Moshe Mikanovsky, a Product Manager with over 30 years of experience, about the importance of getting closer to the data by getting familiar with data tools and working with business intelligence.
This episode was edited by Jin Yang. She is the co-founder of Mindset Masters. Mindset Masters works with professionals in all industries to encourage high school and university students to develop a growth mindset. If you are interested in speaking in one of their online workshops, please reach out at any time at contact.mindsetmasters@gmail.com.
Jaya and Sid chat about building competency with and a fluency with data to get a better idea of what's happening with systems and how products are helping customers.
Mentioned in this episode: Ravi Mehta's Product Competency Toolkit: https://www.ravi-mehta.com/product-manager-skills/
In this episode, Rommil and Tracy interview Melanie Kyrklund from Specsavers on how to run a high-impact optimization program, how to align the program with a product's vision, and how she likes black licorice.
This week, Siobhan Solberg, Kenya Davis, Shiva Manjunath and Eddie Aguilar talk about how to address the lack of direction and primary calls to action on pages.
This episode was edited by Zohaib. Zohaib is an audio engineer, music producer and DJ from Karachi who now lives in Toronto. You can find this profiles here:
- https://soundcloud.com/saazbyzoi
- https://www.facebook.com/saazbyzoi
- https://www.instagram.com/zoi_music/?hl=en
In this episode, our panel discusses how to manage stakeholders and HiPPOs. What do we do with them when they don't agree with our results? How do we get ahead of issues?
On the panel:
- Shiva Manjunath
- Eddie Aguilar
- Kenya Davis
- Siobhan Solberg
Eden and Positive John speak about how recording your sessions with clients helps reduce the need for taking notes, how qualitative research is key to not just CRO but for all things business, and how no one likes pop-up surveys except for Positive John.
Useful resources mentioned in the episode: https://www.notion.so/User-Psych-framework-applied-on-LinkedIn-s-conversion-funnel-3bf3cdfc4aa54c2c9191f403157ed317
Jaya and Sid explore:
- What does a good product experiment look like
- Aligning the need to deliver and the need to learn with OKRs
- Why aren’t companies adopting the practice
On today’s panel, we have
- Siobhan Solberg from Raze
- Shiva Manjunath from Gartner
- Kenya Davis from Evolytics
- Eddie Aguilar from Circle media labs
In this episode the panel discusses:
- Balancing between optimizing the entire funnel vs focusing only on the website
- Bringing segmentation into the mix
- Their beef with Google optimize
On today's episode, we interview Twilio's Justin Coons and talk about:
- Not only can you run experiments for SEO, but you also should
- Which is better, short-form content or long-form content
- What metrics to optimize for when running SEO Experiments
On today's episode of Point/Counterpoint, Trish leads Patrick and Marion through a discussion about: Their CRO New Year's resolutions, and the pros and cons of several experiment prioritization frameworks such as
- PIE (https://www.widerfunnel.com/pie-framework/)
- PXL (https://cxl.com/blog/better-way-prioritize-ab-tests/)
- Hotwire's framework (https://blog.optimizely.com/2015/05/05/how-to-prioritize-ab-testing-ideas/)
Jaya and Siddharth explore how Experimentation fits into the Product Development Life Cycle, how to work with challenging stakeholders, share stories about how they've leveraged Experiments to generate results.
On today’s episode, John Ostrowski and Eden Bidani talk about:
- How they respond when someone asks them how many experiments they should run
- What they've done when a test breaks in production
- Why John keeps a personal list of shame
If you enjoy today’s episode. and you think we’ve earned it - please consider subscribing!
Today's episode features:
Eddie Aguilar Kenya Davis Shiva Manjunath Siobhan Solberg
The panel talks about:
How best practices in CRO should only apply to approaches and workflows and not necessarily what you should test. The danger of copying others’ "best practices".
Enjoy!
It's been one year since Experiment Nation started interviewing Experimenters from around the world, can you believe it? For this special episode, the tables are turned! Experiment Nation's new Canadian co-host, Shannon (aka the "Russian Giant"), interviews Rommil for a change of pace. We talk about escape rooms, raising children, and wearing lab coats. We hope you enjoy it! And yes, we promise to go back to talking more about Experimentation soon - pinky swear.
On that note - we're looking for more Interviewers and podcast co-hosts to join us. Shannon and I cannot cover the world alone. So if you're interested in helping us share the stories of Experimenters wherever they may be, check out our job posting on LinkedIn!
In this episode, we speak with Gartner's Shiva Manjunath about the world of CRO: what it means, what it doesn't mean, and the important role that humour and politics play in it. As a bonus, we learn how his lack of enthusiasm about medicine helped him become the person he is today and what he really thought about last season's NHL playoffs.
We're still working through the audio setup. We'll get it right one day, swear.
Experiment Nation finally has a podcast and we couldn't think of a better way to kick it off than having a great in-depth conversation with Munir Al-Dajani about how to run Experiments on Personalization, some of the ethical aspects of Personalization, and the differences between Machine Learning and AI.