Software Daily - SoundSage

View Details

Cloud applications continue to grow in popularity, but ensuring the security of these applications often presents a formidable engineering challenge. This challenge motivated the creation of Jit. Jit is a continuous security platform for developers, and seeks to enable every cloud app to start with minimum viable security, or MVS, without slowing development velocity. David Melamed is the Co-founder and CTO of Jit and he joins us in the episode to talk about his platform. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Convex: Get started on Convex for free today by visiting https://www.Convex.dev. Statsig: Visit https://statsig.com/softwaredaily to learn more. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Serverless backend platforms are cloud services that simplify the process of building a backend. These platforms are growing rapidly in popularity because they can greatly accelerate application development, and improve the developer experience. Convex is a real-time backend platform that uses 100% TypeScript and is designed with reactive UI frameworks in mind. The team behind Convex has built and maintained some of the most popular apps, backends and exabyte-scale storage projects at companies like Dropbox and Google. James Cowling is the Co-Founder and CTO at Convex, and he joins us to discuss how Convex offers a simpler full-stack developer experience. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Statsig: Visit https://statsig.com/softwaredaily to learn more. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Exafunction is a leader in deploying deep learning models at scale. One of their products is Codeium, a coding assistant for software developers based on Exafunction's deep learning technology. Codeium provides AI-assisted autocompletion in your IDE, making it easier for you to incorporate deep learning technology in your software development workflow. Varun Mohan is the CEO and Co-founder of Codeium, and he is our guest today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Fullstory: Visit https://fullstory.com to learn more. Statsig: Visit https://statsig.com/softwaredaily to learn more. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Tom Preston-Werner is a renowned software developer, inventor and entrepreneur. He co-founded GitHub and is the creator of the avatar service Gravatar, the TOML configuration file format, and the static site generator software Jekyll. Tom is currently working on the full-stack web framework, RedwoodJS. He joins us today to tell us the latest about RedwoodJS, including its new support for server-side rendering. Check out the show notes for a link to the RedwoodJS conference. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Incident management is the process of managing and resolving unexpected disruptions or issues in software systems, especially those that are customer-facing or critical to business operations. Implementing a robust incident management system is often a key challenge in technical environments. Rootly is a platform to handle incident management directly from Slack, and is used by hundreds of leading companies including Canva, Grammarly, and Cisco. Ashley Sawatsky leads Developer Relations at Rootly and previously led Shopify's Incident Communications team. Niall Murphy is Co-founder and CEO at Stanza. He has written extensively about reliability engineering and is the co-author of the best-selling book, Site Reliability Engineering. Ashley and Niall join us in this episode to discuss how engineers and their non-technical counterparts can successfully approach incident management together. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

AI-assisted software delivery refers to the utilization of artificial intelligence to assist, enhance, or automate various phases of the software development lifecycle. AI can be used in numerous aspects of software development, from requirements gathering to code generation to testing and monitoring. The overarching aim is to streamline software delivery, reduce errors and, ideally, reduce the time and costs associated with software development. Birgitta Böckeler is the Global Lead for AI-assisted Software Delivery at Thoughtworks and she joins us in this episode. We discuss how the latest advances in large language models are revolutionizing software development. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Buy Now, Pay Later model, or BNPL, is traditionally a business-to-consumer model that is gaining traction in the business-to-business domain. Adoption of BNPL in business-to-business scenarios offers benefits like improved cash flow management for buyers, and increased sales for suppliers, often leading to higher order value. However, this model introduces complexities, including the need for advanced technical platforms for managing such transactions. There are also risks associated with delayed or defaulted payments, and potential accumulation of interest and fees. As business-to-business entities consider BNPL, they must strike a balance between its potential advantages and its inherent risks and operational challenges. Tranch is a London-based startup that is focused on providing a payments-checkout platform for businesses that want to pay easier and get paid faster. Sohil Pandya is the Head of Engineering at Tranch and he joins us in this episode. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Database caching is a fundamental challenge in database management and there are hundreds of techniques to satisfy different caching scenarios. PolyScale is a fully automated database cache. It offers an innovative approach to database caching, leveraging AI and automated configuration to simplify the process of determining what should and should not be cached. Ben Hagen is the founder and CEO of PolyScale and he is our guest today. Full disclosure: PolyScale is a sponsor of Software Engineering Daily. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the realm of modern software development, clear and comprehensive documentation is essential for guiding users and contributors alike. Developers often struggle to create, manage, and maintain documentation that is both user-friendly and visually appealing. In this episode, we sit down with Sebastian Lorber, the lead maintainer of Docusaurus, a widely acclaimed open-source static site generator, purpose-built for documentation sites. Sebastian sheds light on how Docusaurus empowers developers to effortlessly build and deploy documentation websites that are not only functional, maintainable, and easy to brand. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Nearly all developers are interested in developing APIs to streamline the connection process for other companies to their services. APIs are ubiquitous and critical to building modern software. However, making an API that provides a great developer experience can be a time-consuming endeavor. As a result, API teams often leave the “final” mile of integration on the shoulders of the users. Speakeasy, a budding startup, aims to provide a product and a pipeline that makes it easy to create and consume any API. Sagar Batchu is the Co-foudner and CEO of Speakeasy and he joins us in this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The increasing adoption of GPT (Generative Pre-trained Transformer) models underscores the critical need for privacy and data protection. Using LLMs without appropriate safeguards can lead to sensitive data leakage. Skyflow GPT Privacy Vault prevents sensitive date from reaching GPTs. Amruta Moktali is the Chief Product Officer of Skyflow and she joins us today. We discuss about Generative AI, how its different from the other AI models and how we can use this technology in an ethical manner. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The CAP theorem, also known as Brewer's theorem, is a fundamental principle in distributed systems that states that it is impossible to simultaneously achieve three desirable properties in a distributed data system: Consistency, Availability, and Partition tolerance. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Companies have high hopes for Machine learning and AI to support real-time product offerings, prevent fraud and drive innovation. But there was a catch - training models require labeled data that machines can digest. As data volumes increase, the opportunity to get great ML results rises, but so does the problem of labeling all the data to get that excellent result. Enter Snorkel AI’s programmatic data labeling and MLops platforms like Snorkel Flow. Today we are interviewing Alex Ratner, one of the founders of Snorkel AI. Snorkel AI evolved from research Alex led as part of his Ph.D. research at Stanford, focused on programmatic data labeling to enable much faster and more accurate ML training and retraining. Alex is a born teacher who always has enthusiasm for the topic. Today he will share the newest evolutions of the product at Snorkel, shed light on why doing ML well requires programmatic data labeling, and talk about foundation models in actual enterprise settings and generally. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Elasticsearch is the most established solution today to search and analyze large amounts of logs. However, it can be costly and complex to manage. Quickwit searches large amounts of append only cloud data like logs or ledgers in a fraction time with significantly less cost than Elasticsearch. In this episode, we interview Paul Masurel, one of the founders of Quickwit and the creator and maintainer of Tantivy, the open source full text search engine written in Rust. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As Companies scale and their codebase becomes large, it can become difficult to measure team effectiveness. With so many moving pieces, and an increased lack of visibility, Engineering Leaders mostly rely on gut feelings to guide decisions and determine impact. Uplevel pulls meta-data from programs like Git, Slack, Jira and public calendar events and uses machine learning and organizational science to generate actionable insights. With Uplevel Engineering teams can increase velocity, decrease burnout and plan and execute more predictable releases. Christina Forney is the VP of Product at Uplevel and she joins us in this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Fullstory: Visit https://fullstory.com to learn more. CoxAutomotive : Visit https://COXAUTOTECH.COM today! Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Rudderstack is a warehouse-native customer data platform (CDP) that helps businesses collect, unify, and activate customer data from all their different sources. In today's episode, we're talking to Soumyadeb Mitra, the founder and CEO of Rudderstack. We discuss the importance of activating all your data, how Rudderstack can help you activate your data, the challenges of integrating different data sources and how to build a data-driven culture in your organization. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Hyperscalers refer to expansive cloud service providers capable of delivering enterprise-scale computing and storage services. Companies like Google, Amazon, Facebook that have huge data centers and are either running their own software or renting out this infrastructure can be called Hyperscalers. These Hyperscalers realized a long time back that the traditional Network, Storage & Compute server racks were not enough to fulfill the requirements of a modern enterprise at scale. So they built custom solutions for their data center requirements. For all of the other companies and organizations that have their own data centers, they are still running legacy old technology. Oxide Computer is a company aiming to democratize access to rack-scale technology and provide a seamless software stack to facilitate its implementation. Bryan Cantrill is the Co-Founder and CTO of Oxide Computer and he joins us in this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Starburst: Try Starburst Galaxy today: https://starburst.io/sedaily. CoxAutomotive : Visit https://COXAUTOTECH.COM today! Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Available as a cloud service, Red Hat Trusted Software Supply Chain provides a DevSecOps framework to create applications more securely. Vincent Danen is the VP of Product Security at Red Hat and he joins us in this episode. Red Hat has been a secure open source software provider for a long time. We discuss how the Red Hat Trusted Software Supply Chain product allows enterprises to adopt DevSecOps practices successfully, safely, consume open source code and third-party dependencies and build security into the software development lifecycle. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

CoxAutomotive : Visit https://COXAUTOTECH.COM today! Starburst: Try Starburst Galaxy today: https://starburst.io/sedaily. Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The state of Data inside most companies is chaotic. It takes significant time and investment to tame this chaos. When you are a platform provider you are gathering tons of data from the developers using your platform. These developers building products on your platform need insight into that data to better understand how their application is performing or to troubleshoot it. Most Platforms or SaaS application providers find it both difficult and expensive to build customer-facing analytics and data applications into their platforms. In fact most companies don’t know what to do with the data they are gathering and continually postpone future product roadmap features aimed to unlock this data. This data can be a crucial part of the developer experience and can empower your customers. It can save you countless hours of handling support tickets, and increase overall stickiness on the platform. Propel is a GraphQL API platform ideal for powering customer-facing analytics use cases, from customer dashboards and analytics APIs to product usage or in-product metrics. Tyler Wells is Co-founder and CTO at Propel and he joins us today. We discuss how the customer-centric experiences at Twilio lead his team to the journey they are on today.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The company Continue has a new take on the resale marketplace. By partnering with brands and pulling post-purchase data onto their platform, this innovative solution simplifies and enhances the resale process, empowering consumers to participate in a more sustainable and circular economy. Richie Ganney and Harry Riley are the Co-founders of Continue and they join us in this episode. We uncover how this startup is revolutionizing the way we consume and recycle products. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

CRaC (Coordinated Restore at Checkpoint) is a new technology that can improve startup and warmup times by orders of magnitude. It is a project of OpenJDK that was proposed and led by Azul. The CRaC Project defines public Java APIs that allow for the coordination of resources during checkpoint and restore operations. With CRaC, a checkpoint can be set at any point where an application can be safely paused. Azul is releasing a reference implementation of CRaC in JDK 17 with the Azul Zulu Build of OpenJDK for x86 64-bit Linux update. CRaC allows for an "instant" start at any point in the application lifecycle at an optimal speed. It also work's really well with Azul's ReadyNow! feature that optimizes warm-up. For the first time, developers will have a TCK-tested, CRaC-configured, production-ready JVM with commercial support available for their use. Gerrit Grunwald is a Java Champion & Principle Engineer at Azul and he joins us in this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Fullstory: Visit https://fullstory.com to learn more. CoxAutomotive : Visit COXAUTOTECH.COM today! Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

WebAssembly is a low-level binary format for the web that is compiled from other languages to offer maximized performance and is meant to augment the places where JavaScript isn’t sufficient. High Performance applications like AutoCAD, Figma and Photoshop are now leveraging the capabilities of WebAssembly to provide native experiences on the web. Matt Butcher is the CEO at Fermyon Technologies and he joins us in this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos CoxAutomotive : Visit COXAUTOTECH.COM today! Bloomberg: Learn more about the opportunities that await you by visiting https://Bloomberg.com/careers NGINX: Register for Microservices March at https://www.nginx.com/sedpodcast

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In this episode, we are talking to Robert Cooke, founder and CTO of 3forge. He has spent the last decade creating a full stack software platform that revolutionized enterprise real-time data management, visualization, and workflows through its inventive "high impact code" concept. With offices in New York, London, and Singapore, 3forge has been serving a global clientele including Tier 1 banks and large financial institutions. He talks about his life long passion for computing, the challenges of creating a complete high performance platform, and the scaling principles needed to exceed the requirements of the most demanding global firms. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Usage based pricing is becoming more and more popular. Led by the wild popularity of cloud service providers such as AWS, customers are demanding more visibility into usage information and more accountability for the dollars they spend on a service. But historically service oriented products have shied away from usage based pricing because of the complexity involved in managing the data involved. Amberflo is a service that provides a platform for building and deploying usage based pricing. Puneet Gupta is the CEO and Founder of Amberflo and he is our guest today. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Feature flags also known as feature toggles, release toggles or feature flippers are a way to enable or disable a particular feature from your app without making any changes to the source code. You can turn on or off a particular functionality without deploying new code. Feature flags can also be used to serve different features to different subset of users. The company Flagsmith provides you a platform for developing, implementing and managing your feature flags. Ben Rometsch is the Co-founder and the CEO of Flagsmith and he joins us today. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As companies depend more on data to improve digital products and make informed decisions, it's crucial that the data they use be accurate and reliable. MonteCarlo, the data reliability company, is the creator of the industry's first end-to-end data observability platform. Bar Moses and Lior Gavish are the founders of MonteCarlo and they join us today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Programming robotics software has traditionally been a specialized field. The software industry has seen rapid progress, the operating system that provides the foundation for our software applications is taken care of by companies like Google, Microsoft, and other players like Canonical, Amazon, etc. The robotics industry still needs that OS layer that handles the complexities so that engineers can build robots that serve their business needs. VIAM aspires to be the Operating System for robotics that makes it easy to turn great ideas into production robots. Eliot Horowitz is the Founder and CEO of VIAM and he joins us today. Eliot was previously the CTO at MongoDB, a database company. We discuss with him the different challenges one faces when trying to build a robot and how VIAM is trying to solve that. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Fullstory: Visit https://fullstory.com to learn more. CoxAutomotive : Visit COXAUTOTECH.COM today! Bloomberg: Learn more about the opportunities that await you by visiting https://Bloomberg.com/careers

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Monolithic relational databases are the traditional foundation of financial core ledger systems. Nevertheless, the process of building and operating mission critical financial ledgers on these databases, and implementing homegrown accounting models, is a journey fraught with engineering challenges.Twisp has set out to rethink the underlying technology for financial ledger systems by combining the operational and scaling characteristics of a distributed database, the correctness guarantees offered by relational databases, with pre-built accounting primitives, while fully leveraging the modern cloud. Twisp systems are repeatable, autoscaling, fully managed, and a boon to developer productivity.In this episode, we speak to Michael Parsons, co-founder & CTO of Twisp.To learn more about Twisp and get access to a sandbox ledger, go to: www.twisp.com/Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Java Applications became the go-to preference of most developers because of the write-once-run-anywhere advantage it gave over other languages. And it didn't take much time for Java to become the language for the enterprise. Consequently, most enterprises are still running legacy Java Monoliths on their infrastructure. Breaking up a monolith is not an easy process–nor is it something that every company should do just because they have a monolith. In some cases, a monolith is just fine. But sometimes, you do need to decompose a monolith as the complexity of the monolith grows and leads to longer release cycles or scalability issues. Breaking down into microservices is a natural way to shift legacy applications to the cloud. vFunction is an artificial intelligence platform that assesses, analyzes, helps you design microservices and then automatically creates those microservices for you. So it's an end-to-end platform from analysis to the actual creation of the code of those microservices with their respective APIs. While vFunction started with Java, they are expanding these capabilities to other platforms as well. Moti Rafalin who is the CEO of vFunction and Amir Rapson who is the CTO of vFunction join us today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

AWS Insiders: Click on https://link.chtbl.com/awsinsiders?sid=podcast.cloudengineering to get in on the fun. Redhat: check them out by clicking the link: https://link.chtbl.com/codecomments?sid=podcast.cloudengineering Privacy Dynamics: Sign up for a free account today at https://privacydynamics.io/sedaily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When Web Assembly was created it was supposed to be a compile target, where you could compile your favorite programming language and then execute it inside of a web browser. This made it possible for developers to choose a programming language like C++ for compute intensive applications. Fermyon is taking Web Assembly to the cloud. With Fermyon Cloud deploying and managing cloud-native WebAssembly applications becomes a breeze. Matt Butcher is the CEO at Fermyon Technologies and he joins us today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Privacy Dynamics: Sign up for a free account today at https://privacydynamics.io/sedaily. DataSet: Start today at https://www.dataset.com/sed/ Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

If you are a company with a large pool of physical servers, and compute resources sitting in the data center, and you want to use them efficiently, one way to do that would be to virtualize them. Then you can use those smaller virtual compute resources to run whatever workloads you need. VMware is a company that's known for virtualization. It's also the creator of Software Defined Data Center. In 2019, VMware acquired Heptio, this acquisition gave VMware the know-how to go full-scale on Kubernetes. Joe Beda and Craig McLuckie, one of the original creators of Kubernetes working at Heptio, also joined VMware as part of this acquisition. What followed this was VMware Tanzu, a division within VMware tasked to work on modern applications. VMware Tanzu is a modular, cloud-native application platform. It allows you to build, deliver, and operate cloud-native apps in a multi-cloud world, it also protects you from Vendor lock-in. Betty Junod is VP of Product Marketing at VMware and joins us today.

Get 25% on all Linux Foundation Trainings and Certifications https://training.linuxfoundation.org/ use code "SDT2022NovThx" valid only till November Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Redhat: check them out by clicking the link: https://link.chtbl.com/codecomments?sid=podcast.cloudengineering Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Flatiron School: Start today for free at https://flatironsdevelopment.com/products/fuse Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

With an increasing number of data breaches impacting customer trust, prioritizing data privacy and security is more important than ever. However, as you layer on security and privacy, the overall complexity of a system grows and the data gets harder to use. There’s a constant push and pull between securing and locking down the data but still being able to use it. Skyflow Data Privacy Vault isolates, secures, stores, and tightly controls access to manage and use sensitive data. With secure workflow support, developers are able to securely execute custom code, de-identity and transform data, and securely share data all within the secure compliant environment of the vault balancing data security and usability. Amruta Moktali, the Chief Product Officer of Skyflow, joins the show to discuss Skyflow’s Data Privacy Vault APIs, running secure workflows, and the engineering behind the technology. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos DataSet: Start today at https://www.dataset.com/sed/ Wix: Head over to https://Wix.com/Partners and reimagine what your agency can accomplish. AWS Insiders: Click on https://link.chtbl.com/awsinsiders?sid=podcast.cloudengineering to get in on the fun AZUL:Say goodbye to latency lag: https://download Azul Platform Prime for free. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The complexity of the software infrastructure has been increasing as companies have migrated towards kubernetes, containers, microservices and other distributed systems. However the tools around observability and monitoring have not seen much improvement. These tools are usually managed by teams distributed across different locations and time zones, which results in siloing of knowledge of your infrastructure across individuals. Reasoning about downtime involves bringing together all the pieces. While the teams can communicate through slack, screenshots and google docs, this has been still far from the kind of collaboration tools that other teams have enjoyed for example Figma is used by Design teams and Notion is used by product teams. Fiberplane is a collaborative notebook platform for DevOps and SRE. It puts a programmable SRE environment at the fingertips of engineers, redefining collaboration for infrastructure teams. Fiberplane offers collaborative investigation and documentation tools with real data, all hosted within a technical open-source notebook format. Micha Hernandez van Leuffen is the founder and CEO of Fiberplane and he joins us today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Flatiron School: Start today for free at https://flatironsdevelopment.com/products/fuse Privacy Dynamics: Sign up for a free account today at https://privacydynamics.io/sedaily. DataSet: Start today at https://www.dataset.com/sed/ Apple Card: https://www.apple.com/apple-card/ Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Presto/Trino project makes distributed querying easier across a variety of data sources. As the need for machine learning and other high volume data applications has increased, the need for support, tooling, and cloud infrastructure for Presto/Trino has increased with it. Starburst helps your teams run fast queries on any data source. With Starburst you get a single point of access to your data, no matter where it's stored and it supports high concurrency. Whether it's fast SQL queries on your data lake or faster queries across multiple datasets, Starburst helps your teams run analytics anywhere. Justin Borgman is the CEO of Starburst, and he joins us today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

DataSet: Start today at https://www.dataset.com/sed/ Privacy Dynamics: Sign up for a free account today at https://privacydynamics.io/sedaily. AZUL:Say goodbye to latency lag: https://download Azul Platform Prime for free. Merge: check out Merge today at https://www.merge.dev/daily Flatiron School: Start today for free at https://flatironsdevelopment.com/products/fuse Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Building and managing data-intensive applications has traditionally been costly and complex, and has placed an operational burden on developers to maintain as their organization scales. Todays’ developers, data scientists, and data engineers need a streamlined, single cloud data platform for building applications, pipelines, and machine learning models — without having to move or copy their data. Platforms like the Snowflake Data Cloud provides a unified tool for developers to easily build data applications with Python using Streamlit’s open source framework and Snowflake’s Native Application Framework, gain a streamlined architecture that natively supports users’ programming languages of choice including Java, Scala, SQL, and now Python with Snowpark, store and use transactional and analytical data together with Unistore, and more. Torsten Grabs is the Director of Product Management at Snowflake focused on Data Lake, Data Pipelines, and Data Science. He joins the show to dive into how Snowflake is disrupting application development, and how developers today can eliminate complexity with the Data Cloud.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Privacy Dynamics: Sign up for a free account today at https://privacydynamics.io/sedaily. Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos NORD VPN: Grab your EXCLUSIVE NordVPN deal by going to https://nordvpn.com/sed . Merge: check out Merge today at https://www.merge.dev/daily Flatiron School: Start today for free at https://flatironsdevelopment.com/products/fuse Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Companies are rapidly moving to the cloud, and modern businesses are operating faster with a myriad of SaaS tools in their day-to-day operations. Provisioning resources has been easier than ever. WIth a few clicks you can spin up resources in any part of the world. While we all love the cloud because of the scalability it offers, assets can quickly pile up with every user, bucket, repository and resource. Keeping resources secure and compliant grows in complexity with every cloud service and SaaS application utilized. Resmo is a continuous cloud and SaaS asset visibility, security and compliance solution. DevOps, DevSecOps, and security teams can leverage Resmo to automatically audit resources, mitigate risks and answer security and compliance questions in minutes. Serhat Can is the co-founder at Resmo and he joins us today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

AZUL:Say goodbye to latency lag: https://download Azul Platform Prime for free. Flatiron School: Start today for free at https://flatironsdevelopment.com/products/fuse Interval: Start building for free at https://Interval.com today University of Cincinnati: Go to https://Online.UC.edu to learn more about the MSIT program. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Mikko Hypponen is the CRO of WithSecure and a principal research Officer at F-Secure. In this interview with our host, Jordi Monn, he discusses the eventful and rocky early start of his software engineering career, hybrid warfare, reverse engineering, recent exploits like Log for Shell and heart bleed and more. This episode is packed with valuable career and life lessons on dealing with the unexpected. As well as fascinating stories from the frontiers of cyber security and geopolitics, many of which are discussed in more depth in Mikko's recent book. If it's smart, it's vulnerable. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

React is the most widely used front-end framework. It pioneered the component model and influenced the software design of all modern web applications. React is not an opinionated tool, allowing engineers to shape their applications in a way that fits the problems at hand. However, this freedom of expression can be very confusing to newcomers. Alex Kondov is a principal engineer who has worked with React ever since its inception. In his book, Tao of React, he shares a collection of fundamental architectural principles that aim to help front-end engineers build better web applications. Sponsorship inquiries: sponsor@softwareengineeringdaily.com University of Cincinnati: Go to https://Online.UC.edu to learn more about the MSIT program. Flatiron School: Start today for free at https://flatironsdevelopment.com/products/fuse Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Monte Carlo: Block off your calendar and register for the virtual conference today at https://ImpactSummit.com/2022 Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Observability is a critical aspect of modern digital applications. You can't operate an application at scale that satisfies your customer needs without understanding how the application is currently performing, whether it's understanding the current operating needs of the application, adjusting resource usage, detecting issues before they become serious or solving an ongoing technical issue as it's going on. Thundra is a performance monitoring company that specializes in application, performance monitoring, infrastructure monitoring, and most notably serverless monitoring. Berkay Mollamustafaoglu is the CEO of Thundra and he's our guest.

AZUL:Say goodbye to latency lag: https://download Azul Platform Prime for free. INTERVAL: Start building for free at https://Interval.com today Monte Carlo: Block off your calendar and register for the virtual conference today at https://ImpactSummit.com/2022 Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The classical computing power has doubled every two years - a pattern known as Moore’s law. However, the ability to fabricate more and more transistors in a computer chip is approaching saturation as we are approaching atomic dimensions. Quantum Computing is a promising technology to take us beyond this. A quantum Computer uses qubits to run multi-dimensional quantum algorithms. Quantum Computers are faster and can solve problems that are beyond the reach of even the most powerful classical supercomputers. Dmitri Maslov is the Chief Software Architect at IBM’s Quantum Computing Division. He joins the show to discuss how Quantum Computing is different from Classical Computing. How to get started in this space, How would Quantum Computing impact current encryption standards, we also went through practical applications, quantum supremacy and quantum algorithms. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

AZUL:Say goodbye to latency lag: https://download Azul Platform Prime for free. Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos NORD VPN: Grab your EXCLUSIVE NordVPN deal by going to https://nordvpn.com/sed . Flatiron School: Start today for free at https://flatironsdevelopment.com/products/fuse

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Java Virtual Machine is an abstract machine that makes it possible for you to write Java code once and run it across multiple devices and operating system types. While you can use the OpenJDK it comes with various issues like Security Vulnerabilities and compliance. Azul is a company that specializes in Java for the enterprise. It has the largest Java engineering team after Oracle. John Ceccarelli is the Senior Director of Product Management at Azul Systems responsible for Azul Platform Prime which is their hyper optimized build of OpenJDK.He joins the show to discuss Azul's optimization of OpenJDK, specifically the JIT compiler and the garbage collector. We also discussed the different use cases like the financial sector and other infrastructure areas like Cassandra and Kafka. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Datadog: Start monitoring your applications with a free trial and Datadog will send you a free Tshirt! softwareengineeringdily.com/datadog Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Monte Carlo: Block off your calendar and register for the virtual conference today at https://ImpactSummit.com/2022 Apple Card: https://www.apple.com/apple-card/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Passwordless Authentication is a technique in which users are given access to an environment without entering a password or answering a security question. This allows users to access an environment securely and protects organizations against attack vectors like Keylogging, Brute force methods, and phishing. The company SuperTokens provides secure login and session management for your apps in an open-core model. In this episode, we interviewed Advait Ruia and Rishabh Poddar from SuperTokens. We discussed Open source authentication, security considerations for Authentication, recipes for Authentication, and the future of passwords. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

AZUL:Say goodbye to latency lag: https://download Azul Platform Prime for free. Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Monte Carlo: Block off your calendar and register for the virtual conference today at https://ImpactSummit.com/2022 Flatiron School: Start today for free at https://flatironsdevelopment.com/products/fuse

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Monolithic relational databases are the traditional foundation of financial core ledger systems. Nevertheless, the process of building and operating mission-critical ledgers to track and reconcile payments and money movement is complex. Twisp is rethinking core accounting and financial orchestration on a cloud-native ledger system, so developers can stop re-invented the ledger and focus on building products their customer’s love. When you provision an instance of the Twisp accounting core, you get a transactions ledger for double-entry accounting, a chart of accounts to represent any economic activity, and layered balances for tracking settled, pending, and planned funds flows. All of this is accessible via a straightforward GraphQL API. In this episode, we speak with Jarred Ward, Co-founder & CEO of Twisp. Previously, Jarred was principal engineer at Simple, the first neobank, and later led the team building Open Platform at BBVA, one of the first banking-as-a-service products. To learn more about Twisp and get access to a sandbox ledger, go to: https://www.twisp.com/ Sponsorship inquiries: sponsor@softwareengineeringdaily.com

AZUL:Say goodbye to latency lag: download Azul Platform Prime for free. Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Monte Carlo: Block off your calendar and register for the virtual conference today at https://ImpactSummit.com/2022 Apple Card: https://www.apple.com/apple-card/ University of Cincinnati: Go to https://Online.UC.edu to learn more about the MSIT program.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

PostgreSQL is a free and open-source relational database management system. Postgres-based databases are widespread and are used by a variety of organizations, from Reddit to the International Space Station, and Postgres databases are a common offering from cloud providers such as AWS, Alibaba Cloud, and Heroku. Neon is a serverless open-source alternative to AWS Aurora Postgres. It separates storage and compute and substitutes the PostgreSQL storage layer by redistributing data across a cluster of nodes. Today, we spoke with Nikita Shamgunov of Neon. We discuss how Neon scales Postgres, how it saves cost and the engineering that makes it possible. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

University of Cincinnati: Go to https://Online.UC.edu to learn more about the MSIT program. Datadog: Visit https://softwareengineeringdaily.com/datadog to get started. Monte Carlo: Block off your calendar and register for the virtual conference today at https://ImpactSummit.com/2022 Apple Card: https://www.apple.com/apple-card/ Keeper Security: Get started by visiting https://www.keepersecurity.com/

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Today, we spoke with Daniel Situnayake of Edge Impulse. We discussed AI, machine learning, edge devices, TinyML and AI tool chain. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Merge: check out Merge today at https://www.merge.dev/daily Apple Card: https://www.apple.com/apple-card/ Keeper Security: Get started by visiting https://www.keepersecurity.com/ Wix: Head over to https://Wix.com/Partners and reimagine what your agency can accomplish. Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Today, we spoke with Toma Pujlak and Vedran Jukic of Codeanywhere. We discussed Cloud-based dev environments, Cloud-based IDEs, Infrastructure as code, Dev containers and Live collaboration. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Apple Card: https://www.apple.com/apple-card/ Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos AZUL:Say goodbye to latency lag: download Azul Platform Prime for free. Monte Carlo: Visit https://softwareengineeringdaily.com/montecarlodata to learn more. Cox Automotive: Visit https://coxautotech.com/ to find career opportunities at Cox. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The default configuration in most databases is meant for broad compatibility rather than performance. Database tuning is a process in which the configurations of a database are modified to achieve optimal performance. Databases have hundreds of configuration knobs that control various factors, such as the amount of memory to use for caches or how often the data is written to the storage. The problem with these knobs is that

they are not standardized (i.e., two databases may have a different name for the same knob),

not independent (i.e., changing one knob can impact others),

and not universal (i.e., what works for one application may be suboptimal for another).

In reality, information about the effects of the knobs typically comes only from (expensive) experience. OtterTune is automatic database tuning software that promises to overcome these problems. It uses machine learning to tune the configuration knobs of your database automatically to improve performance. In this episode, we interview Andy Pavlo. Andy is a Database Professor at Carnegie Mellon and Co-Founder of OtterTune. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Apple Card: https://www.apple.com/apple-card/ Keeper Security: Get started by visiting https://www.keepersecurity.com/ Monte Carlo: Visit https://softwareengineeringdaily.com/montecarlodata to learn more. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud computing provides tools, storage, servers, and software products through the internet. Securing these resources is a constant process for companies deploying new code to their cloud environments. It’s easy to overlook security flaws because company applications are very complex and many people work together to develop them. Wyze Labs, for example, had millions of users’ data stolen due to a mistake by a single employee. The company Bridgecrew is a cloud security platform helping to prevent mistakes like that from happening. Bridgecrew integrates into developer workloads to automatically find infrastructure errors in cloud accounts, workloads, and infrastructure as code. Their platform also monitors code reviews and build pipelines to prevent errors from being deployed into production. If an error is found then Bridgecrew’s software reverts that code back to its last known correct state. In today’s episode we talk with Guy Eisenkot, VP Product & Co-founder at Bridgecrew. Guy previously worked as a Principal Product Manager at RSA Security and as a Product Manager at Fortscale before that. We discuss Infrastructure as code, devsecops, cloud security, software supply chain and composition analysis. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Snyk: Sign up for free today at https://snyk.co/sedaily Monte Carlo: Visit https://softwareengineeringdaily.com/montecarlodata to learn more. Rookout: Try it at https://rookout.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published on January 1, 2022. Charlie Gerard is an incredibly productive developer. In addition to being the author of Practical Machine Learning in JavaScript, her website charliegerard.dev has a long list of really interesting side projects exploring the intersection of human computer interaction, computer vision, interactivity, and art. In this episode we touch on some of these projects and broadly explore how practical it is to bring interesting HCI concepts into one’s work. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software engineering can be a surprisingly grueling career. It is both physically and mentally demanding to sit in front of a screen for hours on end producing code. Andy Johns is our guest in this episode. Andy previously worked on growth at Facebook, Twitter, and Quora as an early employee. He’s also an investor in Robinhood, Webflow, and Reforge. Andy joins the show to discuss Mental health in tech, burnout and work/life balance. Check out the show notes to learn more about Andy’s work, specifically his substack blog on Mental Health. Show Notes Andy’s Substack Andy’s Linkedin Sponsorship inquiries: sponsor@softwareengineeringdaily.com Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Monte Carlo: Visit https://softwareengineeringdaily.com/montecarlodata to learn more. Apple Card: https://www.apple.com/apple-card/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data analytics technology and tools have seen significant improvements in the past decade. But, it can still take weeks to prototype, build and deploy new transformations and deployments, usually requiring considerable engineering resources. Plus, most data isn't real-time. Instead, most of it is still batch-processed. Tinybird Analytics provides an easy way to ingest and query large amounts of data in real-time, as well as to automatically create an API to consume those queries. This makes it easy to build fast and scalable applications that query your data; no backend needed! In this episode, we interview Jorge Sancha, Founder and CEO of Tinybird. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Rookout: Try it at https://rookout.com Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Monte Carlo: Visit https://softwareengineeringdaily.com/montecarlodata to learn more. Keeper Security: Get started by visiting https://www.keepersecurity.com/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Application security is usually done with a set of tools and services known as SIEM - Security Information and Event Management. SIEM tools usually try to provide visibility into an organization’s security systems, as well as event log management and security event notifications. The company Panther takes traditional SIEM security a step further. Panther processes and retains all of your security data with cloud-first workflows, identifies and alerts in real-time suspicious activity, enables building a high-fidelity alerting pipeline with Python, version control, unit tests, and CI/CD, and provides a security data lake where raw logs are structured for security at scale. In this episode we talk with Jack Naglieri, Founder and CEO at Panther Labs.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Maintaining availability in a modern digital application is critical to keeping your application operating and available and to keep meeting your customers growing demands. There are many observability platforms out there and certainly Prometheus is a popular open source solution for cloud native companies yet operating an observability platform, costs money, and all of the platforms are highly data intensive. Managing costs and data retention policies is critical to keeping your application operating healthy and operating available. Chronosphere is the leading observability platform for scaling cloud native applications. Focusing on managing costs by managing data. Martin Mao is the co-founder and CEO of Chronosphere and he is my guest today . Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Google DORA: Take the survey at https://cloud.google.com/devops LINKERD: Visit https://softwareengineeringdaily.com/linkerd Permit.io: Get started at https://Permit.io Datadog: Get started today with a free trial at https://softwareengineeringdaily.com/datadog Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As software engineering teams start to build products that become more and more mature. It becomes necessary to be able to debug complex issues with tooling that enables understanding of the full scenario. This can come from application scenarios where APIs are running multiple versions, where users are using your app from multiple devices, device types, OS versions, browser versions, things like that. You may also run into issues with network connectivity or a functionality that is intermittent based on sort of use cases and user interactions. And for this sort of scenario, we've come to start to use a class of tools for debugging called observability tools, observability tools, help developers to understand the full scenario of an application by digging into the data that is being used and recorded and can help to pipe data across various , solutions that are used within enterprises to read and discover what's going on from analytics. One such tool, one product used for this is Rookout . And we are here today, chatting with Liran Haimovitch . We'll talk a little bit about the product and his history, as well as the use case for out and sort of where the team is headed. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Google DORA: Take the survey at https://cloud.google.com/devops Permit.io: Get started at https://Permit.io Snyk: Sign up for free today at https://snyk.co/sedaily Keeper Security: Get started by visiting https://www.keepersecurity.com/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Web infrastructure has evolved from individual servers to shared hosting services to virtual machines and virtual functions. The future of the internet however is looking toward a much more distributed computation model blockchain technology is central to the future of this modern internet of blockchains are still in their infancy and the most people blockchains are intimately tied to crypto currencies but the use of blockchain is substantially larger than that of crypto. Auctions provide a model for distributed computation that allows no centralized ownership and no centralized control of large scale applications. Celestia is developing blockchain technology that enables these modern distributed applications. Nader dabit is it in developer relations at Celestia and is an expert in blockchain technology ,and he joins us today.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Rookout: Try it at https://rookout.com Keeper Security: Get started by visiting https://www.keepersecurity.com/ Wix: Head over to https://Wix.com/Partners and reimagine what your agency can accomplish. Cox Automotive: Visit https://coxautotech.com/ to find career opportunities at Cox. Google DORA: Take the survey at https://cloud.google.com/devops Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In today’s podcast, you’ll see how Prophecy’s low-code data engineering platform, built on Spark and Airflow, provides a complete solution for developing, deploying, and monitoring data pipelines. We’ll speak with Raj Bains, who is the founder & CEO of Prophecy. Raj has spent two decades building powerful tools - Microsoft Visual Studio, founding engineer for CUDA at NVIDIA, product manager for Hive at Hortonworks. He is passionate about making all organizations productive with data and making the lives of data engineers better by reducing toil and increasing the joy. Also, if you like what you hear, Prophecy has a special offer for our listeners. Go to Prophecy.IO/request-a-demo and put in “SED” for the referral code. If you decide to purchase within 3 months of the demo, you get 10% off. If you purchase after 3 months, you still get 5% off.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data is becoming a bank’s biggest asset. These complex enterprises have a huge opportunity ahead - to transform themselves to become a trusted hub of a much broader data ecosystem that goes beyond the financial industry and helps to form a new class of cross-industry experience architectures that are scalable and transparent. The data physics that is needed for such emerging systems runs on consent and privacy preservation rather than black-boxed data lakes. A foundation for making this happen lies in the ability to use distributed, heterogenous data effectively and transform it into experiences that are relevant to the customer. These new experience architectures are following design patterns that are more participatory and consent-based than blindly personalized. In this episode, we deep dive into the context-aware Flybits platform with founder & CEO, Hossein Rahnama, alongside their CTO, Petar Kramaric, and VP of Engineering, Justin Lam.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Rookout: Try it at https://rookout.com Permit.io: Get started at https://Permit.io Google DORA: Take the survey at https://cloud.google.com/devops Datadog: Get started today with a free trial at https://softwareengineeringdaily.com/datadog Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Enabling authorization policies across disparate cloud-native environments such as containers, microservices and modern application delivery infrastructure is complex and can be a roadblock for software engineering teams. Open Policy Agent, or OPA, is an open, declarative, policy-as-code approach to authorization that reduces security and compliance burden for engineering teams. Business context is translated into declarative policy statements. These policy statements are compiled into code and deployed as agents that can be injected into any process, such as an API gateway, Kubernetes provisioning service, public cloud access controls, or continuous delivery automation service. Styra created and contributed OPA to the Cloud Native Computing Foundation where it is a graduated project with over 130M downloads to date and is used by large companies. Styra's enterprise commercial offering, Declarative Authorization Service (DAS), is specifically designed for OPA and includes the ability to author policies, preview impacts of new policies, and document the history of old policies, all through a single view. Tim Hinricks, CTO and founder of Styra, joins the show today to discuss how to make authorization policies easier to author, distribute, and monitor. One note of disclosure to be aware of: Styra is a portfolio company of Capital One Ventures, the strategic investing arm of Capital One. Views and questions expressed in this podcast and related material are my own, or those of my guest, and do not reflect the views of Capital One Ventures or its respective affiliates. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

View Details

Cloud native applications utilizing microservice architectures has grown into one of the most popular application architectural patterns in recent years. The value of leveraging dynamic cloud resources, along with the flexibility and scalability of microservice architectures, creates a strong paradigm that's hard to miss. The strong adoption of Kubernetes has strengthened the pattern enormously. The unique structure and requirements of Kubernetes has led to an increased need for Kubernetes specific monitoring and diagnostics tools. There has been a large number of companies who have jumped at this opportunity. One of those companies is Komodor, a Kubernetes diagnostics platform that focuses on Kubernetes troubleshooting for the entire Kubernetes stack. Itiel Shwartz is the co-founder and CTO of Komodor, and he joins us today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Datadog: Visit https:// softwareengineeringdaily.com/datadog to get started. Permit.io: Get started at https://Permit.io Google DORA: Take the survey at https://cloud.google.com/devops LINKERD: Visit https://softwareengineeringdaily.com/linkerd Keeper Security: Get started by visiting https://www.keepersecurity.com/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Permissions are Hard! And they are becoming harder as we move more into the Cloud-native ecosystem. If we go back in time to the point where it was just a single monolith that you were building on your own. You'll probably have a framework to manage the permissions for you. But when you are working with distributed microservices, especially if you are a polyglot, you can't use those solutions anymore. So you end up having to sprinkle a bit of access control into every little microservice and component that you build. In addition, with the scale of modern applications, it's no longer just your services, there are a lot of third-party services that you have to connect to. Think about things like authentication, billing, analytics and other stuff that you combine from eternal services into what you are building. Permit.io empowers developers to bake in permissions and access control into any product in minutes and takes away the pain of constantly rebuilding them. Or Weis is the co-founder and CEO of Permit.io and joins us today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Cloud Bees: Visit https://www.cloudbees.com/ Wix: Head over to https://Wix.com/Partners and reimagine what your agency can accomplish. Google DORA: Take the survey at https://cloud.google.com/devops Datadog: Visit https://softwareengineeringdaily.com/datadog to get started. Keeper Security: Get started by visiting https://www.keepersecurity.com/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Companies that gather data about their users have an ethical obligation and legal responsibility to protect the personally identifiable information in their dataset. Ideally, developers working on a software application wouldn’t need access to production data. Yet without high-quality example data, many technology groups stumble on avoidable problems. Organizations need a solution to protect privacy while simultaneously preserving aspects of the data which are important. Tonic is automating data synthesis to advance data privacy. Their solution gives your production-like data for development and analytical purposes without compromising on data quality or privacy. In this episode, We interview Tonic’s CEO Ian Coe, and Head of Engineering Adam Kamor. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Keeper Security: Get started by visiting https://www.keepersecurity.com/ Google DORA: Take the survey at https://cloud.google.com/devops Rookout: Try it at https://rookout.com LINKERD: Visit https://softwareengineeringdaily.com/linkerd Merge: check out Merge today at https://www.merge.dev/daily Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Couchbase is a distributed NoSQL cloud database. Since its creation, Couchbase has expanded into edge computing, application services, and most recently, a database-as-a-service called Capella. Couchbase started as an in-memory cache and needed to be rearchitected to be a persistent storage system. In this episode, We interviewed Ravi Mayuram, SVP Products, and Engineering at Couchbase. To learn more about Couchbase, check out couchbase.com/sedaily. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Datadog: Visit https://softwareengineeringdaily.com/datadog to get started. New Relic: Sign up at https://newrelic.com/sedaily Wix: Head over to https://Wix.com/Partners and reimagine what your agency can accomplish. Merge: check out Merge today at https://www.merge.dev/daily Google DORA: Take the survey at https://cloud.google.com/devops

. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Remix is a full stack web framework that lets you focus on the user interface and work back through web fundamentals to deliver a fast, slick, and resilient user experience that deploys to any Node.js server and even non-Node.js environments at the edge like Cloudflare Workers. In this episode, we interviewed Ryan Florence, co-founder at Remix Software. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Snyk: Sign up for free today at https://snyk.co/sedaily LINKERD: Visit https://softwareengineeringdaily.com/linkerd Merge: check out Merge today at https://www.merge.dev/daily Google DORA: Take the survey at https://cloud.google.com/devops Wix: Head over to https://Wix.com/Partners and reimagine what your agency can accomplish. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Profiling has been a part of the software engineering toolbox since the 1970s, but it was long regarded to be too high in overhead, so it was only done ad hoc. The problem with profiling that way is that you never catch the moment in time where for example an incident happened. Continuous profiling is the systematic capturing of profiling at all times, Google has been doing it for over a decade, but it comes both with challenges of how to capture the data at low overhead and also storing and querying the enormous amount of data that continuous profiling brings with it. Polar Signals created the open-source project Parca to bring continuous profiling to the masses. Frederic Braczyk, Founder and CEO of Polar Signals joins the show to discuss the uses of continuous profiling as well as the challenges in building everything from the low overhead profilers using eBPF to purpose-built storage engines for the high volume data it entails. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

LINKERD: Visit http://softwareengineeringdaily.com/linkerd Keeper Security: Get started by visiting https://www.keepersecurity.com/ Cloud Bees: Visit https://www.cloudbees.com/ Snyk: Sign up for free today at snyk.co/sedaily Wix: Head over to https://Wix.com/Partners and reimagine what your agency can accomplish. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Enterprise-grade authentication is often an essential ingredient to virtually all applications in today's world. However, companies often have a hard time understanding the value of that authentication especially during the early stages of product development. And hardening of an application is often left as an afterthought. Add enterprise-level requirements such as single sign-on and two-factor authentication and other requirements that the once afterthought becomes are now major issues. Delaying launch schedules and new feature launches. WorkOS is dramatically shortening this development time by providing tools to allow in integrating complex enterprise standards with just a few lines of code, allowing developers to focus on core products and features. Michael Grinich is the CEO of WorkOS and is my guest today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Cox Automotive: Visit https://coxautotech.com/ to find career opportunities at Cox. Merge: check out Merge today at https://www.merge.dev/daily Keeper Security: Get started by visiting https://www.keepersecurity.com/ Capital one: Visit https://capitalone.com/ML Netapp: Check them out at https://spot.io/sedaily Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Build automation tools automate the process of building code, including steps such as compiling, packaging binary code, and running automated tests. Because of this, build automation tools are considered a key part of a continuous delivery pipeline. Build automation tools read build scripts to define how they should perform a build. Common build scripts include Makefile, Dockerfile, and bash. Earthly is a build automation tool that allows you to execute all your builds in containers. Earthly uses Earthfiles, which draws from the best features of Makefile and Dockerfile and provides a common layer between language-specific tooling and the CI build spec. Earthly builds are repeatable, isolated, and self-contained, and will run the same way across different environments such as a CI system or a developer’s laptop. In this episode we spoke with Adam Gordon Bell who works in developer relations at Earthly. Adam is also the host of the CoRecursive podcast Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Capital one: Visit https://capitalone.com/ML Netapp: Check them out at https://spot.io/sedaily SetApp : Head over to https://fullstory.com/sedailyory.com/sedaily to get in touch. New Relic: Sign up at https://newrelic.com/sedaily Merge: check out Merge today at https://www.merge.dev/daily Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Creating great creative tools is extremely difficult. There are thousands of paths a user could take, and every aspect of the user experience has to be carefully considered and optimized for performance. And when you try to add real-time collaboration on top of that, you can be sure that you’re left with almost no time to focus on the core user experience of your product. Liveblocks is a set of tools and APIs that helps you create amazing real-time collaborative experiences. With Liveblocks, you can make anything collaborative in minutes so that you can give your core product the attention it deserves. Guillaume Salles is the Co-founder and CTO of Liveblocks and joins the show to discuss what he and his team have built. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

RudderStack: Software Engineering Daily listener at https://rudderstack.com/dmdstack.com/dmds Capital one: Visit https://capitalone.com/ML Netapp: Check them out at https://spot.io/sedaily Keeper Security: Get started by visiting https://www.keepersecurity.com/ Cloud Bees: Visit https://www.cloudbees.com/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Merge: check out Merge today at https://www.merge.dev/daily RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Netapp: Check them out at https://spot.io/sedaily Capital one: Visit https://capitalone.com/ML SetApp : Head over to https://fullstory.com/sedailyory.com/sedaily to get in touch. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Mailchimp is an all-in-one marketing platform for growing businesses, empowering millions of customers around the world to launch, build, and grow their businesses with world-class marketing technology, award-winning customer support, and inspiring content. Eric Muntz is Mailchimp's CTO, responsible for the engineering teams that design, implement, and maintain Mailchimp's products and infrastructure. He joins the show to take us through the engineering behind Mailchimp’s infrastructure. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Netapp: Check them out at https://spot.io/sedaily Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily Capital one: Visit https://capitalone.com/ML Datadog: Get started today with a free trial at https://softwareengineeringdaily.com/datadog Cox Automotive: Visit https://coxautotech.com/ to find career opportunities at Cox. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Venture capital investment has continued to flow into technology startups. No one builds technology from scratch. There are cloud services, software libraries, 3rd party services, and software platforms that modern entrepreneurs must adopt to build their products efficiently and quickly. These layers of infrastructure are a key area for many investors. Tim is a partner at Menlo Ventures where he invests in cloud and data infrastructure, AI/ML and cyber security. He joins the show to discuss what goes into a smart investment. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Datadog: Get started today with a free trial at https://softwareengineeringdaily.com/datadog SetApp : Head over to https://setapp.com/developers to get in touch. Capital one: Visit https://capitalone.com/ML Netapp: Check them out at https://spot.io/sedaily Dynatrace: See why Dynatrace is radically different and try it free for 15 days at https://www.dynatrace.com/se-daily/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Between hyper financialisation and the growth of automation in the world, trade and investments have become an ever more prevalent and important tool for the mass market. But the relative knowledge of professional and hobbyist investors is different, and so is the access to opportunities. Coinrule empowers regular people to compete with professional traders by automating investments without having to learn a single line of code. Zdeněk Höfler is the co-founder of Coinrule and joins the show to discuss what he and his team have built. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Keeper Security: Get started by visiting https://www.keepersecurity.com/ Capital one: Visit https://capitalone.com/ML Netapp: Check them out at https://spot.io/sedaily Cloud Bees: Visit https://www.cloudbees.com/ Fullstory : Go to https://fullstory.com/sedailyory.com/sedaily to activate your free one month trial.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Jeromy Johnson is a distributed systems researcher focused on incentive mechanism design and trustless distributed system scalability. He joins us today to share how Protocol Labs is spearheading the innovation in web3. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Keeper Security: Get started by visiting https://www.keepersecurity.com/ Cloud Bees: Visit https://www.cloudbees.com/ Capital one: Visit https://capitalone.com/ML Netapp: Check them out at https://spot.io/sedaily Fullstory : Go to https://fullstory.com/sedaily to activate your free one month trial. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As crypto gains worldwide attention, the most scalable, resilient and performant systems are required for the industry to thrive. Products must be designed for a mix of technical and non-technical users. Blockchain.com's VP of Engineering Lewis Tuff takes us behind the scenes for a look at the architecture, programming languages and database choices required to build an open, accessible and fair financial future, one piece of software at a time. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Dynatrace: See why Dynatrace is radically different and try it free for 15 days at Dynatrace.com/SE-Daily. Dragon Ball Legends : https://twitter.com/DB_Legends Keeper Security: Get started by visiting https://www.keepersecurity.com/ Wix: Head over to https://Wix.com/Partners and reimagine what your agency can accomplish. Fullstory : Go to https://fullstory.com/sedaily to activate your free one month trial.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His most recent book, Architecting for Scale (O’Reilly Media) is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Subscribe at mdb.fm.

Follow Lee, at leeatchison.com. As companies move more and more workloads to the cloud, cost management in the cloud is becoming critically important. Modern application development methodology increasingly involves moving to service and microservice architectures enabled by the cloud and kubernetes is the backbone of this modern infrastructure trend. Managing costs of kubernetes clusters is therefore becoming increasingly important. Kubecost is the company focused on giving visibility into kubernetes resources, allowing you to reduce your overall kubernetes based infrastructure spend. Webb Brown is the founder of Kubecost and joins us today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

SetApp : Head over to https://setapp.com/developers to get in touch. Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily Capital one: Visit https://capitalone.com/ML Datadog: Get started today with a free trial at softwareengineeringdaily.com/datadog Dragon Ball Legends : https://twitter.com/DB_Legends

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Web3 promises to change the way we interact and transact online, but today’s dapps and wallets are hard to use. They are typically optimized for desktop machines and domain experts. In particular, it can be hard to bring value on-chain, and hard to know what to do once you’ve onboarded. Charlie Andrews-Jubelt is a software engineer who works on Valora, a mobile wallet, and Revo, a dapp that makes it easier to invest in DeFi and get compound interest. Charlie joins the show to talk about the engineering challenges of making web3 more accessible.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Capital one: Visit https://capitalone.com/ML Keeper Security: Get started by visiting https://www.keepersecurity.com/ Fullstory : Go to https://fullstory.com/sedaily to activate your free one month trial. Skyflow: Visit https://skyflow.com Netapp: Check them out at https://spot.io/sedaily Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

While Kubernetes has many benefits, there is often a need for teams to deploy a monitoring and observability stack to troubleshoot issues that happen within the cluster and the applications themselves. ContainIQ, an out-of-the-box solution, allows engineers to monitor the health of their cluster and troubleshoot issues faster. ContainIQ is unique in its approach in that it was built with eBPF in mind, and is able to provide an APM like experience without being an APM. Matt Lenhard is the Co-founder and CTO of ContainIQ and joins the show to discuss the future of K8s monitoring and observability, as well as the unique technological approach he is taking with eBPF. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

New Relic: Sign up at https://newrelic.com/sedaily Netapp: Check them out at https://spot.io/sedaily Skyflow: Visit https://skyflow.com Capital one: Visit https://capitalone.com/ML Dragon Ball Legends : https://twitter.com/DB_Legends

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Nearly all new tech companies build in a public cloud and established companies are rapidly migrating to the cloud from their on-prem data centers. But this move to the cloud can lead to a visibility problem. Cloud Providers offer not only Compute instances but also manage services like databases, blob storage, queues and more. It can be difficult for SRE teams and security departments to understand what is happening across a company's cloud accounts. Yevgeny Pats is the creator of CloudQuery, an open source cloud asset inventory powered by SQL. CloudQuery allows you to ingest and structure the resources in your cloud accounts so that you can query them using SQL. This allows SRE teams to understand the source of specific resources while security teams can ensure compliance with policies. In this episode we’ll discuss CloudQuery, Yevgeny’s entrepreneurial background and raising funding with an open source project Sponsorship inquiries: sponsor@softwareengineeringdaily.com.

Capital one: Visit https://capitalone.com/ML Skyflow: Visit https://skyflow.com Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily Netapp: Check them out at https://spot.io/sedaily Fullstory : Go to https://fullstory.com/sedaily to activate your free one month trial. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

How to build a rocket ship while flying it? In this episode, Cos returns to the show to discuss the technologies and approach Brex chose to create an engineering organization and innovation engine for the long term, while navigating shifts as the company continued to scale at a rapid pace. Cos and Jeff also cover the nuts and bolts of programming languages and databases and strategies to break through performance bottlenecks. Cos is the CTO of Brex, whose mission is to reimagine financial systems so growing businesses can achieve their full potential. Sponsorship inquiries: sponsor@softwareengineeringdaily.com.

SetApp : Head over to https://setapp.com/developers to get in touch. Fullstory : Go to https://fullstory.com/sedaily to activate your free one month trial. Capital one: Visit https://capitalone.com/ML Netapp: Check them out at https://spot.io/sedaily Cloud Bees: Visit https://www.cloudbees.com/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Convex makes storing your application's data as easy as using React state management. If you can use React hooks, you can also manage your backend data using Convex. James Cowling is a former Dropbox infrastructure engineer turned startup founder. James joins the show to discuss how Convex offers a simpler full-stack developer experience than the tools developers use today — without many of the compromises of other alternatives. This episode is guest hosted by Jackson Gabbard, a long-time Facebook product engineer who is now the CTO of Cord — the SDK for collaboration in any SaaS tool. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Cox Automotive: Visit https://coxautotech.com/ to find career opportunities at Cox. Netapp: Check them out at https://spot.io/sedaily Datadog: Get started today with a free trial at https://softwareengineeringdaily.com/datadog Wix: Head over to https://Wix.com/Partners and reimagine what your agency can accomplish. Capital one: Visit https://capitalone.com/ML Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Developers looking for read or write access to Ethereum, Polygon, IPFS or other Web3 networks in order to get their idea in the hands of users need reliable RPC endpoints they can count on, whether they are working on a hackathon project or running hybrid architecture for a production application. Infura provides a platform for developers to get access to those end points as well as tooling to help them build on Web3. With its origin being Ethereum, Infura has expanded to both EVM-compatible and non-EVM compatible blockchains, as well as decentralized storage. From wallets, DeFi, gaming, Metaverse, or NFT projects, Infura is relied upon by some of the most used and popular projects in Web3. With billions of daily requests handled, the team has built knowledge on how to scale with reliability. Co-Founder and General Manager of Infura, EG Galano joins the show to talk about Blockchain infrastructure and protocols. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Streaming data platforms like Kafka, Pulsar, and Kinesis are now common in mainstream enterprise architectures, providing low-latency real-time messaging for analytics and applications. However, stream processing - the act of filtering, transforming, or analyzing the data inside the messages - is still an exercise left to the receiving microservice or datastore, a custom programming exercise likely repeated over and over within an application. Stream processing tools such as Apache Flink and ksqlDB have been around for half a decade, but their complexity has hindered adoption. Decodable’s mission is to radically simplify processing on the stream with a SaaS platform based on Flink, and using only SQL which frees up developers to focus on what matters most. Eric Sammer is founder and CEO of Decodable and joins the show to discuss the potential of stream processing, its role in modern data platforms, and how it’s being used today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Fullstory : Go to https://fullstory.com/sedaily to activate your free one month trial. Netapp: Check them out at https://spot.io/sedaily Capital one: Visit https://capitalone.com/ML Skyflow: Visit https://skyflow.com Cloud Bees: Visit https://www.cloudbees.com/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Kubernetes ecosystem has drastically changed how development teams ship software. While Kubernetes has provided many advancements in cloud infrastructure, it has also left organizations with massive security blindspots. KSOC was created to give developers and security teams a single control plane to harden multi-cluster Kubernetes environments through event-driven analysis, least privilege enforcement, and remediation-as-code. Jimmy Mesta is the Co-Founder and CTO of KSOC and joins the show to discuss modern Kubernetes security challenges and how security teams need to prepare for a future where Kubernetes is the top attack target for adversaries. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Skyflow: Visit https://skyflow.com Capital one: Visit https://capitalone.com/ML Datadog: Get started today with a free trial at https://softwareengineeringdaily.com/datadog Netapp: Check them out at https://spot.io/sedaily Cloud Bees: Visit https://www.cloudbees.com/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In software engineering, telemetry is the data that is collected about your applications. Unlike logging, which is used in the development of apps to pinpoint errors and code flows, telemetry data includes all operational data including logs, metrics, events, traces, usage, and other analytical data. Companies usually visualize this information to troubleshoot problems and understand patterns and opportunities in how their applications are used. The company New Relic is a modern observability platform built to optimize your entire software stack from one place. New Relic includes a Telemetry Data Platform that acts as a single source of truth for telemetry data. Built on top of that are tools for full stack observability to visualize and troubleshoot your data in milliseconds. When a problem does occur, New Relic’s applied intelligence will detect and understand the problem to help resolve it faster. Nic Benders is New Relic’s chief architect and GVP of engineering. As part of the engineering team since the early days of the company, Benders has been involved with everything from Agent development to ZooKeeper deployments and all the pieces and products in between. In his role as chief architect, Benders looks after the long-term technical strategy behind New Relic’s product and the experience of all the engineering teams who build it. He joins the show to discuss the engineering behind New Relic. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Datadog: Get started today with a free trial at softwareengineeringdaily.com/datadog Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Capital one: Visit https://capitalone.com/ML Netapp: Check them out at https://spot.io/sedaily Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

One of the challenges with Microservices architecture is how you manage dependencies between your services when implementing workflows. Conductor is an open-source microservices and workflow orchestration platform. Boney Sekh co-founded Orkes Inc - a company focused on offering Conductor as a service. Boney joins the show to discuss how engineers leverage Conductor to build highly reliable distributed applications using. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Get Stream: Check it out at https://getstream.io/ Netapp: Check them out at https://spot.io/sedaily Cox Automotive: Visit https://coxautotech.com/ to find career opportunities at Cox. RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Capital one: Visit https://capitalone.com/ML Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Notifications have typically been an area of a product where building in-house has been the only option. However, building a best-in-class notifications system that delivers a great customer experience requires a significant investment when you start to layer in complexities like batching, cross channel orchestration, and user preferences. Chris Bell of Knock joins the show to talk through how Knock can solve the challenges with building an in-house product notification system, and free up engineering time in the process. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Get Stream: Check it out at https://getstream.io/ Netapp: Check them out at https://spot.io/sedaily Cox Automotive: Visit https://coxautotech.com/ to find career opportunities at Cox. Cloud Bees: Visit https://www.cloudbees.com/ Capital one: Visit https://capitalone.com/ML Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The advent of the cloud introduced a new form of technical debt in which organizations can lose track of what infrastructure they have and how it relates to the business. While the cloud's native APIs offer some transparency into your infrastructure, these offerings are often described as necessary but not sufficient. When companies have a non-trivial question about their environments, getting answers can have a big cognitive overhead to get a clear answer. In this episode, I interview Tyson Kunovsky, Co-Founder and CEO at AutoCloud, creators of the open source project Cloud Graph. CloudGraph is a free and open source GraphQL API for AWS, Azure, GCP, and Kubernetes. CloudGraph provides a much more intuitive graphQL based interface to ask questions about your infrastructure in context of the relationships between components.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Datadog: Get started today with a free trial at https://softwareengineeringdaily.com/datadog Capital one: Visit https://capitalone.com/ML RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily Skyflow: Visit https://skyflow.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Highlight is a tool that helps teams reproduce end-to-end user sessions to better understand their application. With Highlight, engineering teams can replay errors with high precision, which includes a complete session replay, outgoing network requests, dense stack traces and insight into the app's state management system (redux, apollo's cache, etc.). At the same time, product teams can analyze user activity, collaborate with engineering teams and find the reason for user drop offs to gain insight into UX. Jay Khatri is the CEO of Highlight and joins the show to talk about Browser Observability and what he and his team have built. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Datadog: Get started today with a free trial at https://softwareengineeringdaily.com/datadog Mongo DB: Just visit https://mongodb.com/world-sedaily RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Netapp: Check them out at https://spot.io/sedaily Capital one: Visit https://capitalone.com/ML Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Lee Atchison spent seven years at Amazon working in retail, software distribution and Amazon Web Services. He then moved to New Relic, where he spent eight years scaling the company's internal architecture. Lee has deep expertise in building and managing fast growing web applications, and he used this knowledge to write architecting for scale from O'Reilly. Technical debt is a common problem for engineering teams. Technical debt is additional long-term work required to evolve or maintain an application due to a short-term decision in designing or building that application. Too much technical debt can reduce product innovation and lead to lower employee retention. In this episode, we'll discuss all things technical debt based on Lee’s recent article, Technical Debt Will Sink You. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

There's an increasing regulatory and consumer pressure on companies to do a better job protecting sensitive customer data. Yet, despite this pressure, data breaches and compliance issues continue to plague the tech industry. Companies like Apple, Netflix, and Google have solved these challenges by pioneering a new type of technology, the data privacy vault. Skyflow is making this technology available to all companies through a simple API. Sean Falconer joins the show to discuss the features of a data privacy vault and the data security and compliance challenges it addresses.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily Capital one: Visit https://capitalone.com/ML Skyflow: Visit https://skyflow.com. Netapp: Check them out at https://spot.io/sedaily Cloud Bees: Visit https://www.cloudbees.com/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data-as-a-service is a company category type that is not as common as API-as-a-service, software-as-a-service, or platform-as-a-service. In order to vend data, a data-as-a-service provider needs to define how that data will be priced, stored, and delivered to users: streaming over an API or served via static files. Naqeeb Memon of Safegraph joins the show to talk through the mechanics of delivering data-as-a-service. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Skyflow: Visit https://skyflow.com Cloud Bees: Visit https://www.cloudbees.com/ RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Netapp: Check them out at https://spot.io/sedaily Capital one: Visit https://capitalone.com/ML Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Blockchains were the first systems to allow guaranteed permanent storage of public data. As cryptocurrency technology has advanced, a rich ecosystem of permanent storage and compute has developed as well. One of these is Arweave, a system for permanent information storage. Sam Williams is a founder of Arweave and joins the show to talk through how Arweave works and his motivation for starting it. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Mongo DB: Just visit https://mongodb.com/world-sedaily Cloud Bees: Visit https://www.cloudbees.com/ RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Netapp: Check them out at https://spot.io/sedaily Capital one: Visit https://capitalone.com/ML Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data labeling allows machine learning algorithms to find patterns among the data. There are a variety of data labeling platforms that enable humans to apply labels to this data and ready it for algorithms. Heartex is a data labeling platform with an open source core. Michael Malyuk joins the show to talk through the platform and modern usage of data labeling systems. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

View Details

Real-time analytics are difficult to achieve because large amounts of data must be integrated into a data set as that data streams in. As the world moved from batch analytics powered by Hadoop into a norm of “real-time” analytics, a variety of open source systems emerged. One of these was Apache Pinot. StarTree is a company based on Apache Pinot that provides fast, real-time data analytics. Chinmay Soman joins the show to discuss Apache Pinot in relation to other real-time analytics platforms, and what StarTree has built on top of Pinot. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Cox Automotive: Visit https://coxautotech.com/ to find career opportunities at Cox. Wix: Head over to https://Wix.com/Partners and reimagine what your agency can accomplish. Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily Netapp: Check them out at https://spot.io/sedaily Capital one: Visit https://capitalone.com/ML Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Web3 is powerful but difficult to work with. Deploying blockchain nodes, accessing data, and performing staking operations are non-trivial engineering actions. To simplify web3, Ankr hosts APIs for node deployment, RPC, and staking. Josh Neuroth from Ankr joins the show to talk about modern web3 infrastructure. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily Get Stream: Check it out at https://getstream.io/ Netapp: Check them out at https://spot.io/sedaily Skyflow: Visit https://skyflow.com today to learn how. Capital one: Visit https://capitalone.com/ML Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The terminal is a necessary tool for any software engineer. In order to work quickly, developers have always customized their terminals to work for their specific application workflow. Fig is a reimagined terminal product that adds autocomplete and an entire app ecosystem to the existing terminal you are familiar with. In a previous show we talked through the basics of Fig. In today’s show we take a deeper dive into the functionality and engineering behind the product with the founders of Fig, Brendan and Matt. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily JS Nation: https://jsnation.com Skyflow: Visit https://skyflow.com Netapp: Check them out at https://spot.io/sedaily Capital one: Visit https://capitalone.com/ML Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

WP Engine is a domain specific cloud provider that hosts high performance WordPress infrastructure. This website, Software Engineering Daily, runs on WP Engine. Scaling a domain specific cloud provider for WordPress includes complexities at the level of the database, application, load balancer, and other areas. Brandon DuRutte from WP Engine joins the show to talk through his work. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily Capital one: Visit https://capitalone.com/ML React Summit : https://reactsummit.com Netapp: Check them out at https://spot.io/sedaily Skyflow: Visit https://skyflow.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data loss can occur when large data sources such as Slack or Google Drive get leaked. In order to detect and avoid leaks, a data asset graph can be built to understand the risks of a company environment. Polymer is a data loss prevention product that helps companies avoid problematic data leaks. Yasir Ali is the founder of Polymer and joins the show to talk about the engineering and product vision of Polymer. https://www.polymerhq.io/ Sponsorship inquiries: sponsor@softwareengineeringdaily.com

React Summit : https://reactsummit.com Skyflow: Visit https://skyflow.com Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily Netapp: Check them out at https://spot.io/sedaily Puppet:You can join Puppet for the conversation by registering at https://events.puppet.com/fireside.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Cloud Bees: Visit softwareengineeringdaily.com/cloudbees to get a free trial. JS Nation: https://jsnation.com Skyflow: Visit https://skyflow.com Netapp: Check them out at https://spot.io/sedaily Puppet:You can join Puppet for the conversation by registering at https://events.puppet.com/fireside.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data integration infrastructure is not easy to build. Moving large amounts of data from one place to another has historically required developers to build ad hoc integration points to move data between SaaS services, data lakes, and data warehouses. Today, there are dedicated systems and services for moving these large batches of data. Airbyte builds open source data integration systems, and Michel Tricot from Airbyte joins the show to talk about the design and development of Airbyte.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Capital one: Visit https://capitalone.com/ML Netapp: Check them out at https://spot.io/sedaily Get Stream: Check it out at https://getstream.io/ Puppet:You can join Puppet for the conversation by registering at https://events.puppet.com/fireside.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern organizations eventually face data governance challenges. Keeping track of where data came from, what systems update it, in what ways updates can be made are just some of the issues to be tackled. Large organizations face additional challenges like training, onboarding, and capturing the institutional knowledge that leaves with the departure of key team members. As teams grow, these challenges often grow faster for unprepared organizations. Select Star helps companies unlock the full context of their data. Their solution automatically catalogs and documents your database tables and BI dashboards. In this episode I interview Shhinji Kim about the functionality of Select Star and how companies have achieved successful adoption.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily Wix: Head over to https://Wix.com/Partners and reimagine what your agency can accomplish. Netapp: Check them out at https://spot.io/sedaily RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Puppet:You can join Puppet for the conversation by registering at https://events.puppet.com/fireside.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The JavaScript supply chain includes numerous vulnerabilities due to its expansive nature and the long dependency chains. Socket is a new security company that can protect your most critical apps from supply chain attacks. They are taking an entirely new approach to one of the hardest problems in security in a stagnant part of the industry that has historically been obsessed with just reporting on known vulnerabilities. Feross is the Founder and CEO of Socket Security. He joins the show to talk about Socket’s approach to detecting and blocking supply chain attacks.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily Netapp: Check them out at https://spot.io/sedaily Puppet:You can join Puppet for the conversation by registering at https://events.puppet.com/fireside. Capital one: Visit https://capitalone.com/ML Monte Carlo: Visit https://softwareengineeringdaily.com/montecarlodata to learn more. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Loft is a platform for Kubernetes self-service and multi-tenancy. Loft allows you to control Kubernetes clusters with added multi-tenancy and self-service capabilities to get more value out of Kubernetes beyond simply cluster management. It allows for cost optimization, more efficient provisioning, and other features. Lukas Gentele joins the show to talk about Kubernetes multi-tenancy and the engineering behind Loft.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

View Details

Observability consists of metrics, logs, and traces. Lightstep is a company that builds distributed tracing infrastructure, which requires them to store and serve high volumes of trace data. There are numerous architectural challenges that come with managing this data. Ben Sigelman and Alex Kehlenbeck join the show to discuss the implementation of Lightstep.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Puppet:You can join Puppet for the conversation by registering at https://events.puppet.com/fireside. RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Strong DM: Start your free 14 day trial today at: https://strongdm.com/SEDaily Monte Carlo: Visit https://softwareengineeringdaily.com/montecarlodata to learn more. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The solution many turn to for capturing their streaming data is InfluxDB. In this episode, I interview Brian Gilmore, director of product management at Influx Data, about how real time applications achieve success built on top of InfluxDB. When most people hear the phrase Internet of Things, it typically evokes an image of connected devices we install in our homes. While this is a common use case, the true winner to date in IoT is probably industrial automation. While small improvements can yield big returns, and small errors can result in huge losses, it's critical to capture and elegantly handle telemetry data from industrial systems.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Netapp: Check them out at https://spot.io/sedaily Puppet:You can join Puppet for the conversation by registering at https://events.puppet.com/fireside. Get Stream: Check it out at https://getstream.io/ Capital one: Visit https://capitalone.com/ML Cox Communucations: Go to https://cox.career/techjobs to find out more about career opportunities at Cox Communications. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Cox Automotive: Visit https://coxautotech.com/ to find career opportunities at Cox. Puppet:You can join Puppet for the conversation by registering at https://events.puppet.com/fireside. Strong DM: Start your free 14 day trial today at: https://strongdm.com/SEDaily Monte Carlo: Visit https://softwareengineeringdaily.com/montecarlodata to learn more.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Terminals are a gateway to building and running software, but they have not been reimagined since their initial creation. Warp is a new kind of terminal with visual aids, autocomplete, documentation, customization, and other features. It includes GPU acceleration, its own UI framework, and other engineering systems to increase speed and performance. Zach Lloyd joins the show to talk through the creation of Warp and the future of developer tools.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

CODA: Head over to https://coda.io/sedaily Cox Communucations: Go to https://cox.career/techjobs to find out more about career opportunities at Cox Communications. RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Strong DM: Start your free 14 day trial today at: https://strongdm.com/SEDaily Monte Carlo: Visit https://softwareengineeringdaily.com/montecarlodata to learn more. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data infrastructure is a fast-moving sector of the software market. As the volume of data has increased, so too has the quality of tooling to support data management and data engineering. In today’s show, we have a guest from a data intensive company as well as a company that builds a popular data engineering product. James Densmore works at Hubspot, which produces tons of data, and Lior Gavish works at Monte Carlo Data, which sells a data quality product.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Puppet:You can join Puppet for the conversation by registering at events.puppet.com/fireside. CODA: Head over to https://coda.io/sedaily Cox Communucations: Go to https://cox.career/techjobs to find out more about career opportunities at Cox Communications. RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software engineering can be a surprisingly grueling career. It is both physically and mentally demanding to sit in front of a screen for hours on end producing code. During the pandemic, the strains of being an engineer became even more acute, as we were secluded in our homes, in some cases alone. Gabe Greenberg and Michelle Bakels work at G2i, a company that manages a large number of developers. G2i has a focus on developer health, which they believe is an underappreciated topic. They are also helping organize React Miami, a conference that takes place April 18-19, 2022.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Puppet:You can join Puppet for the conversation by registering at https://events.puppet.com/fireside. Work OS: To learn more and get started, go to https://softwareengineeringdaily.com/workos RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Strong DM: Start your free 14 day trial today at: https://strongdm.com/SEDaily CODA: Head over to https://coda.io/sedaily Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Crypto companies have cemented themselves as a company category that is not going away. Bitski is a company that makes web3 tooling and infrastructure, including a wallet and a collection of tools for creating and selling NFTs. Patrick Tescher joins the show to talk through the engineering behind crypto infrastructure and APIs.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Doppler: Sign up via https://doppler.com/sedaily. Cox Communucations: Go to https://cox.career/techjobs to find out more about career opportunities at Cox Communications. RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Strong DM: Start your free 14 day trial today at: https://strongdm.com/SEDaily CODA: Head over to https://coda.io/sedaily Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Running a database company requires expertise in both technical and managerial skills. There are deeply technical engineering questions around query paths, scalability, and distributed systems. And there are complex managerial questions around developer productivity and task allocation. Sam Lambert is the CEO of PlanetScale, which is building modern relational database infrastructure. Before PlanetScale he spent several years on infrastructure at GitHub. He joins the show to talk about his work at PlanetScale and the vision for the company

Sponsorship inquiries: sponsor@softwareengineeringdaily.com Puppet:You can join Puppet for the conversation by registering at https://events.puppet.com/event/fireside/ Cox Communucations: Go to https://cox.career/techjobs to find out more about career opportunities at Cox Communications. RudderStack: Software Engineering Daily listener at https://rudderstack.com/SED. Get Stream: Check it out at https://getstream.io/ New Relic: Sign up at https://newrelic.com/sedaily Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

GitOps is a deployment and infrastructure management strategy based around continuous delivery and Kubernetes, with Git at the center of deployment workflows. Policy management can be used to define permissions and rules around who can deploy and what constitutes safe deployment. There is a synergy between GitOps tooling and policy management tooling. Alexis Richardson is the co-founder and CEO of Weaveworks. Mohamed Ahmed founded Magalix, a security and compliance company that was acquired by Weaveworks. They join the show to talk about GitOps and Policy Management.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com Puppet:You can join Puppet for the conversation by registering at events.puppet.com/fireside. Influx Data: . To learn more and get started for free, visit https://influxdata.com/sedaily Strong DM: Start your free 14 day trial today at: http://strongdm.com/SEDaily RudderStack: Software Engineering Daily listener at http://rudderstack.com/SED. New Relic: Sign up at https://newrelic.com/sedaily Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A company’s response to an incident defines how that company responds to an adverse, unexpected scenario. Kintaba automates the incident management process so teams can quickly report, respond, resolve, and reflect on major incidents collaboratively. Kintaba's incident response platforms gives you instant access to incident management processes based on the practices of companies like Facebook, Google, Stripe, and other successful and resilient organizations. John Egan is Cofounder & CEO at Kintaba and joins the show to talk about Incident Response.

Attend The Incident Response Conference April 1, 2022

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Work OS: To learn more and get started, go to softwareengineeringdaily.com/workos Influx Data: . To learn more and get started for free, visit influxdata.com/sedaily Strong DM: Start your free 14 day trial today at: strongdm.com/SEDaily RudderStack: Software Engineering Daily listener at rudderstack.com/SED. New Relic: Sign up at newrelic.com/sedaily Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

SingleStore is a multi-use, multi-model database designed for transactional and analytic workloads, as well as search and other domain specific applications. SingleStore is the evolution of the database company MemSQL, which sought to bring fast, in-memory SQL database technology to market. Jordan Tigani is Chief Product Officer of SingleStore and joins the show to talk through the architecture and engineering of the SingleStore platform.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Work OS: To learn more and get started, go to softwareengineeringdaily.com/workos Influx Data: . To learn more and get started for free, visit influxdata.com/sedaily Cox Communucations: Go to cox.career/techjobs to find out more about career opportunities at Cox Communications. New Relic: Sign up at newrelic.com/sedaily Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Compute resources continue to trend towards being cheaper, easier to use, and faster. Despite these positives, more compute demands more energy and therefore an increasing carbon footprint. With many companies committing to controlling their net carbon emissions, tools are required for engineers to not only measure their cloud infrastructure, but to make informed choices about how to set up the infrastructure. Steren Giannini is a senior product manager. He works on Google Cloud Carbon Footprint. In this interview, we discuss how Google Cloud collects and exposes this data to users and how those users are responding in their infrastructure approaches

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

View Details

Vantage is a system for optimizing cloud costs. It provides tools and interfaces for developers to analyze how they are spending on AWS resources, and has recently expanded into GCP as well. Vantage users gain an easy interface into their costs that would otherwise be hard to analyze via the raw AWS console. Ben Schaechter is the founder of Vantage. In a previous show he gave an overview of Vantage. In today’s episode he returns to discuss the engineering internals of Vantage and provide a macro perspective on where people are wasting money in the cloud.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

New Relic: Sign up at newrelic.com/sedaily Influx Data: . To learn more and get started for free, visit influxdata.com/sedaily Get Stream: Check it out at https://getstream.io/ Monte Carlo: Visit softwareengineeringdaily.com/montecarlodata to learn more. RudderStack: Software Engineering Daily listener at rudderstack.com/SED. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At Lyft, Ketan Umare worked on Flyte, an orchestration system for machine learning. Flyte provides reliability and APIs for machine learning workflows, and is used at companies outside of Lyft such as Spotify. Since leaving Lyft, Ketan founded Union.ai, a company focused on productionizing Flyte as a service. He joins the show to talk about the architecture and usage of Flyte, as well as how he is formulating the company around it.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Cox Communucations: Go to cox.career/techjobs to find out more about career opportunities at Cox Communications. Work OS: To learn more and get started, go to softwareengineeringdaily.com/workos Get Stream: Check it out at https://getstream.io/ Strong DM: Start your free 14 day trial today at: strongdm.com/SEDaily RudderStack: Software Engineering Daily listener at rudderstack.com/SED. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

DuckDB is a relational database management system with no external dependencies, with a simple system for deployment and integration into build processes. It enables complex queries in SQL with a large function library, and provides transactional guarantees through multi-version concurrency control. Hannes Mühleisen works on DuckDB and joins the show to talk about query engines and OLAP system Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Rudder Stack:Visit datastackshow.com/live to register today. Influx Data: To learn more and get started for free, visit influxdata.com/sedaily Get Stream: Check it out at https://getstream.io/ Monte Carlo: Visit softwareengineeringdaily.com/montecarlodata to learn more. Cox Automotive: Visit COXAUTOTECH.COM to find career opportunities at Cox. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Substack is a platform for subscription-based content, covering newsletters, podcasts, and soon video. The design and speed of the platform are notable–with a minimalistic format that simply gets creators paid for its work, Substack has gained massive popularity. Chris Best is the CEO of Substack and joins the show to talk about the platform, architecture, and engineering of Substack.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Private Internet Access: .Go to https://privateinternetaccess.com/SEDaily InfluxData: To learn more and get started for free, visit influxdata.com/sedaily GetStream:Check it out at https://getstream.io/ MonteCarlo: Visit softwareengineeringdaily.com/montecarlodata to learn more. CoxAutomotive: visit COXAUTOTECH.COM to find career opportunities at Cox. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Customer data pipelines power the backend of many successful web platforms. In a customer data pipeline, data is collected from sources such as mobile apps and cloud SaaS tools, transformed and munged using data engineering, stored in data warehouses, and piped to analytics, advertising platforms, and data infrastructure. RudderStack is an open source customer data pipeline system that pulls together this disparate functionality. In a previous episode, we covered the basics of RudderStack. In today’s show, we dive deeper into the engineering of RudderStack with returning guest CEO Soumyadeb Mitra.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com InfluxData:To learn more and get started for free, visit influxdata.com/sedaily StrongDM: Start your free 14 day trial today at: strongdm.com/SEDaily GetStream: Check it out at https://getstream.io/ Monte Carlo: Visit softwareengineeringdaily.com/montecarlodata to learn more. Cox Automotive: visit COXAUTOTECH.COM to find career opportunities at Cox. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Managing Kubernetes nodes leads to operational complexity, security issues, and nodes that are perhaps more expensive to run than necessary. Deferring the node management to an underlying platform abstracts away these problems and can improve operations. Madhuri Yechuri runs Elotl, a nodeless Kubernetes platform. She joins the show to talk about the architecture and purpose of her company.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com Rudder Stack: Sign up today... or just get a free t-shirt for being a Software Engineering Daily listener at rudderstack.com/SED. Influxdata: To learn more and get started for free, visit influxdata.com/sedaily Monte Carlo: Visit softwareengineeringdaily.com/montecarlodata to learn more. Getstream: Check it out at https://getstream.io/ Work OS: To learn more and get started, go to softwareengineeringdaily.com/workos Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Retool is a company that allows customers to build complex internal tools using a high level GUI. Users configure the relationships between these different tools, giving them the ability to build applications even without much background in engineering. Of course, having some engineering expertise helps, as users can write JavaScript to interact with the higher level components. Snir Kodesh is the head of engineering at Retool, and he previously worked at Lyft as a director of engineering. He joins the show to talk about his work at Retool, how it compares to Lyft engineering, and the lower level challenges of building the platform.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com Get Stream: Check it out at https://getstream.io/ Cox Automative: Visit COXAUTOTECH.COM to find career opportunities at Cox. Influx Data: To learn more and get started for free, visit influxdata.com/sedaily Private Internet Access: .Go to https://privateinternetaccess.com/SEDaily Work OS: To learn more and get started, go to softwareengineeringdaily.com/workos Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The software supply chain consists of packages, imports, dependencies, containers, and APIs. These different components each have unique security risks. To ensure the security of their software supply chain, many developers use tools to analyze and scan their infrastructure for vulnerabilities. Barak Schoster works at Bridgecrew, a DevSecOps cloud security platform. He joins the show to talk about the risks of the modern software supply chain and what his company does to alleviate it. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Get Stream: Check it out at https://getstream.io/ Private Internet Access: .Go to https://privateinternetaccess.com/SEDaily Work OS: To learn more and get started, go to softwareengineeringdaily.com/workos Influx Data: To learn more and get started for free, visit influxdata.com/sedaily Cox Automotive: visit COXAUTOTECH.COM

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The data lake architecture has become broadly adopted in a relatively short period of time. In a nutshell, that means data in it’s raw format stored in cloud object storage. Modern software and data engineers have no shortage of options for accessing their data lake, but that list shrinks quickly if you care about features like transactions. Apache Hudi is a platform for building streaming data lakes that is optimized for lake engines and batch processing. In this episode, I interview Vinoth Chandar, creator of the Hudi Project and Founder and CEO at Onehouse. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Support for the show comes from StackOverflow stackoverflow.com/teams/sedaily Coda coda.io/sedaily Private Internet Access privateinternetaccess.com/SEDaily RudderStack datastackshow.com/live Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

National Instruments develops software and hardware for engineering in a wide variety of domains, from aerospace to government technology to application testing. The interface between hardware and software presents a variety of difficult engineering challenges. Luke Schreier is a Senior Vice President at National Instruments and joins the show to discuss the engineering and management of the company. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Support for the show comes from StackOverflow stackoverflow.com/teams/sedaily Coda coda.io/sedaily Private Internet Access privateinternetaccess.com/SEDaily Honeycomb softwareengineeringdaily.com/honeycomb RudderStack datastackshow.com/live Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Automating video search requires a data pipeline that extracts metadata from videos and allows users to annotate the video with information that correlates with that metadata. The video needs to be segmented into intervals that define the search space, and the search space needs to be queryable by a user. Sieve Data is an automated video search platform that enables developers to add video search and analytics to internal tools and applications. Mokshith Voodarla is the CEO of Sieve Data and joins the show to talk about video search and what he and his team have built. Sponsorship inquiries: sponsor@softwareengineeringdaily.com GetStream: Check it out at https://getstream.io/ CapitalOne: Visit capitalone.com/ML Mparticle: Visit mparticle.com MONTE CARLO: Visit softwareengineeringdaily.com/montecarlodata to learn more. TotallyBio: To learn more go to TotallyBio.io. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Increasingly, technology groups need to be strategic about the cloud services they adopt to ensure their vendor’s pricing is both fair and measured exactly right for the application’s unique access pattern. Definitely answering questions like these can take a significant amount of time and energy from your most valuable engineers. Taloflow is a company trying to make this decision making process easier for object storage and cloud cost management. In this episode, I interview LV Jadavji about the key questions companies are asking and how Taloflow helps people navigate them. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Totally Bio: To learn more go to TotallyBio.io. Influx Data:To learn more and get started for free, visit influxdata.com/sedaily CapitalOne: Visit capitalone.com/ML StrongDM: Start your free 14 day trial today at: strongdm.com/SEDaily WorkOS: To learn more and get started, go to softwareengineeringdaily.com/workos Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A data catalog provides an index into the data sets and schemas of a company. Data teams are growing in size, and more companies than ever have a data team, so the market for data catalog is larger than ever. Amundsen is a data catalog that came out of Lyft. We have previously explored the basics of Amundsen. In today’s episode, Mark Grover returns to the show to talk about the art and science of data catalogs. Sponsorship inquiries: sponsor@softwareengineeringdaily.com MONTE CARLO: Visit softwareengineeringdaily.com/montecarlodata to learn more. Mparticle: Visit mparticle.com CapitalOne: Visit capitalone.com/ML TotallyBio: To learn more go to TotallyBio.io. WorkOS: To learn more and get started, go to softwareengineeringdaily.com/workos Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Splunk is a monitoring and logging platform that has evolved over its 18 years of existence. In its modern focus on observability, it is focused on open source and AIOps. Observability has evolved with the growth of Kubernetes, and Splunk’s work around OpenTelemetry has kept parity with the open-source community of Kubernetes. Spiros Xanthos is the general manager of observability at Splunk. He joins the show to talk about Splunk’s modern product portfolio and his work on his own company prior to being acquired by Splunk. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Support for the show comes from: Totallybio: To learn more go to TotallyBio.io. Influx Data: To learn more and get started for free, visit influxdata.com/sedaily Mparticle: Visit mparticle.com Monte Carlo: Visit softwareengineeringdaily.com/montecarlodata to learn more Doppler : Sign up via doppler.com/sedaily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Francesco Cesarini founded Erlang Solutions in 1999 with a mission to help companies adopt Erlang. In this interview, I speak with Francesco and Gabor Olah from Erlang Solutions. We discuss the Erlang language, it's ecosystem, and features like concurrency, resilience, and scalability that motivate adoption. We use Java and the Java Virtual Machine as a comparison point for Erlang and it's virtual machine the BEAM. Lastly, we explore where Erlang fits best in contemporary software engineering projects. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Support for the show comes from: New Relic: Sign up at newrelic.com/sedaily Monte Carlo: Visit softwareengineeringdaily.com/montecarlodata to learn more Mparticle: Visit mparticle.com Work OS: To learn more and get started, go to softwareengineeringdaily.com/workos Strong DM: Sign up at strongdm.com/sedaily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Serious software projects require several environments. Your production environment is obviously mission critical. A staging environment is also necessary to perform validation and regression testing before taking the risk of pushing an update to production. Best practices and approaches for managing these and other environments vary from organization to organization. In some sense, different software systems should be expected to have unique needs. Yet certain commonalities and wisdom can be gained from observing high scale success stories. In this episode, I interview Senthil Padmanabhan about how eBay turned around an impeding staging environment into its biggest asset for developer productivity.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com Support for the show comes from: Influx Data: To learn more and get started for free, visit influxdata.com/sedaily Monte Carlo: Visit softwareengineeringdaily.com/montecarlodata WorkOS: To learn more and get started, go to softwareengineeringdaily.com/workos Totallybio: To learn more go to TotallyBio.io. Capital One: Visit capitalone.com/ML Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In contrast to other IDEs, the notebook interface offers software developers a unique environment idealized for data professionals. Despite the growth in popularity, a surprising learning curve still exists for setup and configuration. A siloed notebook offers no native collaboration tools. While one can connect to a SQL database programmatically, if you’re looking for an ideal ergonomic environment for some heavy duty SQL queries, many developers seek an external tool for that job. In this episode, Kyle interviews Barry McCardel and Caitlin Colgrove from Hex. Hex is a collaborative data workspace that makes it easy to go from idea to analysis to sharing. We talk about Hex’s offering and the evolving space of notebook solutions for going beyond some of the issues noted above. Sponsorship inquiries: sponsor@ softwareengineeringdaily.com Totally Bio : To learn more go to TotallyBio.io. Honey Comb :Use Honeycomb for free at softwareengineeringdaily.com/honeycomb. Mparticle : Visit mparticle.com Doppler : Sign up via doppler.com/sedaily. Monte Carlo : Visit softwareengineeringdaily.com/montecarlodata Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When writing code, test driven development is a common accepted methodology to ensure the development of high quality software. Your organization’s data, on the other hand, is an entirely different challenge. Data can be missing due to human error, a failure with a 3rd party provider, a botched release, or dozens of other issues. When not missing, data can still become corrupted or start exhibiting a trend in the wrong direction that isn’t obvious to notice. Anomalo is a complete data quality platform. It can monitor your enterprise data and alert you to problems that are automatically detected. In this episode, I interview Jeremy Stanley about the ways in which teams are using the platform to monitor and improve their data quality. Sponsorship inquiries : sponsor@softwareengineeringdaily.com

MAGIC MIND: Go to magicmind.co MPARTICLE: Visit mparticle.com DOPPLER: Sign up via doppler.com/sedaily. STRONG DM: Sign up at strongdm.com/sedaily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When you visit a web page, the creator’s intent is to present you a seamless experience that fills your browser window. That web page or web application is generally divided up in some meaningful way across navigation elements, content, ads, header, footer, and other components. Those components may represent the work of independent teams. Typically a web app is built in a single code base, pulling all those components into a monolithic software application. For backend software development, these monoliths are often split up in a refactoring towards microservice architecture. In this episode, I interview Luca Mezzalira, author of Building Micro-Frontends. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Support for this show comes from Cox Automotive: visit COXAUTOTECH.COM Influx Data: visit influxdata.com/sedaily Capital One: Visit capitalone.com/ML Work OS: To learn more and get started, go to softwareengineeringdaily.com/workos MParticle: Visit mparticle.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Support for this show comes from: Monte Carlo: Visit softwareengineeringdaily.com/montecarlodata mparticle: Visit mparticle.com Capital One: Visit capitalone.com/ML strongDM: strongdm.com/sedaily Honeycomb: Use Honeycomb for free at softwareengineeringdaily.com/honeycomb. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The creator economy has seen rapid growth, thanks largely to software solutions like Patreon that are enabling creators. As the creator economy grows, providers must be prepared for scalability issues and the challenges of maintaining and growing a software infrastructure and the team that evolves it. In this episode, I interview Utkarash Srivastava, SVP Engineering at Patreon. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

WorkOS: To learn more and get started, go to softwareengineeringdaily.com/workos Influx Data: visit influxdata.com/sedaily Capital One: Visit capitalone.com/ML mparticle: Visit mparticle.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Snyk is a platform for security that started with open source scanning and has expanded into container security, infrastructure as code, and other products. Snyk is a simple product to use, but has hidden complexities that build large data structures to manage and scan code dynamically. In a previous episode we discussed the core Snyk product. In today’s show, we talk about the engineering behind Snyk. CEO Guy Podjarny joins the show to talk through the architecture of Snyk and how the company has evolved to serve a variety of use cases. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Support for this show comes from: Mergify: Visit www.mergify.com mparticle: Visit mparticle.com Capital One: Visit capitalone.com/ML WorkOS: Go to softwareengineeringdaily.com/workos Doppler: Sign up via doppler.com/sedaily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Linkerd is a service mesh that runs efficiently with a low memory footprint. We have covered the details of Linkerd in previous episodes. Buoyant is the company that sells Linkerd as a service, and today’s show focuses on the engineering details of the company, and how Linkerd is architected in 2022. William Morgan is the CEO of Buoyant, and he joins the show to talk in detail about running a leading service mesh company.

Support for this show comes from: mParticle: Visit mparticle.com StrongDM: Sign up at strongdm.com/sedaily Capital One: Visit capitalone.com/ML WorkOS: To learn more and get started, go to softwareengineeringdaily.com/workos

Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Online poker was deemed illegal in the United States ten years ago. Since then, poker has decreased in popularity, then found new invigoration thanks to live streaming and a large volume of bored gamers looking for something to do during the pandemic. Poker is a strategy game that can be played even without the financial element, and Clubs Poker is a free-to-play web-based poker client that grew significantly over the past year. Taylor Crane is the founder of Clubs Poker, and he joins the show to talk about the past, present, and future of online poker, as well as the engineering around the platform. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Database product companies typically have a few phases. First, the company will develop a technology with some kind of innovation such as speed, scalability, or durability. The company will offer support contracts around that technology for a period of time, before eventually building a managed, hosted offering. PlanetScale is a database company built around the Kubernetes-based Vitess technology. Sugu Sougoumarane is the CTO of PlanetScale, and formerly worked at YouTube, where he built large-scale SQL databases. In today’s show, he talks about building out the core PlanetScale technology, tuning PlanetScale’s consensus model, and the development of the hosted cloud service offering of the company. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Couchbase is a distributed NoSQL cloud database. Since its creation, Couchbase has expanded into edge computing, application services, and most recently a database-as-a-service called Capella.

Couchbase started as an in-memory cache and needed to be rearchitected to be a persistent storage system. In this episode, I interview Ravi Mayuram, SVP Products and Engineering at Couchbase about the architecture and the history of Couchbase. To learn more about Couchbase, check out Couchbase.com/SEDaily. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Rackspace is a multi cloud solutions provider that has evolved beyond its cloud computing origins into a diverse set of services and support offerings. Customers work with Rackspace to adopt cloud application deployments, modern data analytics, and all the other opportunities offered by cloud computing. Much of this occurs through partnerships where Rackspace provides teams of engineers to work with the customer. Jeff DeVerter is the CTO of products and services at Rackspace. He’s also the host of the podcast Cloud Talk, a show about cloud trends and technologies. Jeff previously served as the CTO of Microsoft Technology at Rackspace, so our conversation began with a discussion of Microsoft-based migrations. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Historically, search engines made money by showing sponsored ads alongside organic results. As the idiom goes, if you’re not paying for something, you are the product. Neeva is a new take on search engines. When you search at neeva.com, you get the type of result you’d expect from a search engine minus any advertising. In this episode, I speak with Darin Fisher, Software Engineer at Neeva. We discuss the motivation, implementation, and mobile experience for searching with Neeva. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Tabnine is an AI assistant that helps software engineers write more efficient code. It’s been trained on a large corpus of source code or can be trained based on your specific codebase. Either way, the resulting model offers predictive completion of code that can make programmers more productive. In this episode, I interview Eran Yahav, professor at Technion - Israel Institute of Technology and CTO at Tabnine.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Protecting your customers begins with best practices for securely capturing, storing, and protecting the data you collect for or about them. When an organization has a large enough dataset, needs typically arise for doing analytical workloads or training machine learning models on this data. If you use random or mock data to generate a report or train a model, you arrive at an output that doesn’t reflect the true use case of the organization. Success on tasks like this seems to require production data. Alternatively, perhaps production-like data is good enough. In this episode, I interview Alex Watson, co-founder and chief product officer at gretel. We discuss their solution for privacy preserving synthetic data that remains representative of the underlying dataset. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When businesses share a common need such as payroll, commercial offerings can compete for market share with software solutions that easily adapt to a variety of businesses. Not all tasks can be easily commoditized or standardized. Take content moderation as an example. Every site that accepts user generated content is likely to have unique and nuanced processes that match their internal definitions of allowable content and procedures for what to do with non-allowable content. Many organizations create custom software solutions for their unique tasks. When they’re central to the core mission, that’s the right choice. When they’re not, such projects are often a distraction for an engineering team that is already stretched too thin. That’s where solutions like Flowdash come in. Flowdash is a low code / no code offering that enables businesses to rapidly develop custom solutions for tasks such as content moderation and others. In this episode, I interview Nick Gervasi, co-founder and CTO at Flowdash. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

If you're working on a proof of concept which you hope will help you raise funding, it's fine to take a few shortcuts. Use the tech stack you know the best, don't fall in love with your code, and when you start to experience growing pains, hopefully you'll have the time to thoughtfully and carefully identify the bottlenecks and limits of your tech stack applied to the specific industry problem you are solving. Another great strategy is to simply copy the tech stack of a larger company with the confidence that what works for a bigger company will likely work for you. But if you're a company like Uber, there's no larger company to copy. Worse still in comparison to most businesses, even a few minutes of downtime is pretty damaging for Uber. To successfully deliver a solution like theirs, one must identify bottlenecks and growing pains in advance, find solutions, and deliver the plan in a way that's invisible to customers. In this episode, I speak with Uday Kiran Medisetty, principal engineer at uber about steps taken in their core state machine design. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Writing software is an absolute joy. Getting software to build is a chore. Thus, build systems emerged as a solution to automate this chore. At some point software engineers either use or hear legends about make and makefiles. While perhaps being the historically known tool, a great deal of thought has gone into approaches to build systems since 1976 when make was introduced.

In today’s episode, we focus on a build system called Pants. Pants is a scalable, software build system. It can help you support all the modern challenges of a build system such as dependency resolution, testing, linting, and packaging. In this episode, I interview Benjy Weinberger, co-founder at Toolchain and contributor to the open source project Pants. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Infrastructure as code is a concept that has delighted software engineers, dev ops, and engineering management across the board. It’s neither fun nor efficient to configure the infrastructure and environments software teams require. Operating software at scale on a cloud, on-prem, or hybrid model is a problem of modernity that many enterprises find surprisingly challenging. Rob Hirschfeld is CEO and co-founder of RackN whose software helps IT teams maintain distributed, multi-vendor operations with consistent operational control. We discuss the approaches being adopted by modern enterprise teams for infrastructure management. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Pinterest is a visual discovery engine people use to find ideas in home, food, style, beauty, and more. The service grew quickly after its founding in 2010 and the company has grown to be a global team of thousands of professionals spanning time zones and continents. To deliver the smooth experience Pinterest users have come to expect requires an engineering team to keep the system running while continuously improving it. In this episode, I speak with three engineering leaders about their decision to work at Pinterest, the contributions they’ve made, and what makes Pinterest engineering different from other companies. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Charlie Gerard is an incredibly productive developer. In addition to being the author of Practical Machine Learning in JavaScript, her website charliegerard.dev has a long list of really interesting side projects exploring the intersection of human computer interaction, computer vision, interactivity, and art. In this episode we touch on some of these projects and broadly explore how practical it is to bring interesting HCI concepts into one’s work.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

TechLit Africa is a non-profit on a mission to lessen African poverty by leveraging the internet. Rural Africans lack digital skills and computers to gain from the digital economy, even though developed countries have an abundance of used computers. That’s where TechLit Africa comes in. They accept used computers, refurbish them with custom classroom code, ship them to Africa, and establish relationships with schools that provide space for the labs they create. In this episode, I interview Nelly Cheboi, one of the founders of TechLit Africa. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Getting a computer program to run the same in different environments has been a recurring problem since the earliest days of software systems. Software versioning, versions of dependencies, hardware configurations, and CPU instruction set differences are just a few examples of challenges engineers have faced to get their software to run in different settings. A core promise of the Java programming language has always been it's ability to run anywhere. If your system can run the Java virtual machine, it can run Java bytecode in a way that will be invariant to other factors. Java was created long before the modern mobile era and the JVM doesn't run on iOS devices. For Java and Kotlin developers looking for a consistent way to deploy their applications to Android and iphone devices, Codename One offers a solution. Codename One is an open-source cross-platform framework aiming to provide write once, run anywhere code for various mobile and desktop operating systems. In this episode I interview Steve Hannah about the project. https://twitter.com/codename_one Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

If you haven’t encountered a data quality problem, then you haven’t yet worked on a large enough project. Invariably, a gap exists between the state of raw data and what an analyst or machine learning engineer needs to solve their problem. Many organizations needing to automate data preparation workflows look to Trifacta as a solution. In this episode, I interview Joseph Hellerstein, professor of Computer Science at UC Berkeley and co-founder of Trifacta. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As the internet has grown, increasingly, we are consumers of services provided by corporations rather than owners and operators of our own systems. To many, this trend towards centralization is antithetical to the spirit of a free and open internet. Urbit is a new operating system and peer-to-peer network. There are several layers of novel ideas in this ambitious project. In this episode, I interview Galen Wolfe-Pauly about the Urbit project. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

InfluxDB is an open-source time-series database. It’s maintained by InfuxData who offers a suite of products that help organizations gain insights from time-series data. In this episode, I interview Zoe Steinkamp, software engineering and developer advocate at InfluxData. We explore some of the common use cases for time-series databases such as IoT and some recent announcements such as the ability to run flux queries right inside VS Code. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Robotic process automation or RPA refers to software robots constructed to automate some business process. Perhaps the most ubiquitous example is adding filters to your email inbox. I’ve worked with a lot of salespeople that configure complex email follow-up campaigns when inbound emails come in, but even that’s a fairly basic example compared to what’s becoming possible. UiPath is an automation platform. They offer a suite of solutions that empower developers and non-developers to construct effective software robots that can measurably improve business efficiency. In this interview, I speak with Boris Krumrey, Global VP Automation Innovations at UiPath. We overview the platform and discuss the current and future state of RPA. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Everyone is becoming increasingly aware of supply chains for physical goods. Software has its own supply chain. A supply of open source solutions exists as does a demand for these solutions by industry. Both have surely grown, but it would be nice to have a way of measuring by how much. The State of Software Supply Chain 2021 is an annual publication now in its 7th year. It’s released by Sonatype. In this interview, I speak with their Field CTO Ilkka Turunen. We review some of the highlights from the report including the state of open source and some particularly interesting statistics about supply chain attacks. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Relational databases have been a fixture of software applications for decades. They are highly tuned for performance and typically offer explicit guarantees like transactional consistency. More recently, there’s been a figurative cambrian explosion of other-than-relational databases. Simple key value stores or counters were an early win in this space. Managing a graph data structure is a more challenging task than key value stores. Asking questions of graphs demands a robust query language. Ideally, you’d also want to provide common graph algorithms to users out of box. MemGraph is an in-memory graph database that seeks to fill this need in a real-time fashion. It’s a performance driven solution that offers developers a mature option for managing and utilizing graph data. In this interview, I speak with Dominik Tomicevic, Founder and CEO of MemGraph. We discuss implementation and adoption details of MemGraph as well as the ways machine learning engineers use it for feature engineering in real time. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The lifeblood of most companies is their sales departments. When you’re selling something other than a commodity, it’s typically necessary to carefully groom the onboarding experience for inbound future customers. Historically, companies approached this in a one-size-fits-all manner, giving all customers a common experience. In today’s data-driven age, a better experience can be provided that is data-driven, personalized, and automated. Amplemarket leverages machine learning in the development of next-generation sales tools. In this episode, I interview João Batalha, Co-Founder, and CEO of Amplemarket. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As cloud providers enable greater levels of specificity and control, they empower compliance-driven enterprise companies. This level of parameterization is downright inhospitable to a new software engineer and can be a cognitive barrier to entry for a senior professional with a great idea but limited time. Developers want to focus on their code, algorithms, front end, and user experience. These concerns have created a huge vacuum in the market that companies like Render are filling. Render is a unified cloud experience to build and run all your apps and websites. It can auto-deploy for git and offers some really novel developer ergonomics. In this episode, I interview Anurag Goel, Founder, and CEO of Render.com about their approach to delivering a developer-centric all-in-one cloud solution. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Writing your application’s code is only half the battle. Getting it to run on your machine is a milestone, but it’s far from your code running in a production environment. There are an increasing set of options application designers have for helping to manage deployment, environments, and CI/CD. Encore is a backend engine for the Go language. One of its core features is the ability to turn any function into an API endpoint with just an annotation. André Eriksson is the founder of Encore. In this episode, we discuss his experience as a developer and explore the features and functionality Encore has to offer. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Microservice architecture has become a ubiquitous design choice. Application developers typically have neither the training nor the interest in implementing low-level security features into their software. For this and many other reasons, the notion of a service mesh has been introduced to provide a framework for service-to-service communication. Today’s guest is Zack Butcher. While working at Google, he was one of the earliest engineers contributing to Istio, an open platform-independent service mesh that provides traffic management, policy enforcement, and telemetry collection. Today he’s the head of product at Tetrate, working on products like the Tetrate Service Bridge and Tetrate Cloud. We discuss the need and implementation of a service mesh and how companies are leveraging tools like their service bridge. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Many software projects run the risk of evolving over time to a complex state that is inhospitable for new contributors to join. This is a dangerous place for a company to be. Either software needs to remain more accessible, or faster paths must be created to help them get on board. Today’s interview is with Kartik Agaram. We explore some of these topics and software development in general while also discussing some of the low-level software projects Kartik has open-sourced. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Climate modeling is increasingly important as supply chains, emergency management, and dozens of other efforts need to make predictions about future conditions and how they will impact business. Analyzing climate data requires geospatial systems, and those systems need a full-stack geospatial technology solution. Gopal Erinjippurath serves as CTO and Head of Product at Sust Global, a venture focused on geospatial analytics. We discuss the software of climate adaptation. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Once a machine learning model is trained and validated, it often feels like a major milestone has been achieved. In reality, it’s more like the first lap in a relay race. Deploying ML to production bears many similarities to a typical software release process, but brings several novel challenges like failing to generalize as expected or model drift. AI Quality management is the biggest challenge in AI today. In this episode, I interview Anupam Datta, the co-founder at TruEra. TruEra has a solution aimed at helping with AI performance, monitoring, and model explainability. We talk about some of the challenges of modern machine learning deployment in production and how companies are succeeding with ML Ops. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Internships can be an incredibly valuable resource to new professionals and are often the first professional work experience for many participants. It's often the case that internship programs are suboptimal. Employers don’t always provide a clear path to success for the intern. Interns in turn don’t always have a resource to reach out for help or even know that it’s ok to do so. Nikita Gupta is a Co-Founder & CTO at Symba, a company that provides an all-in-one platform for managing talent development programs. We talk about what makes a good internship, how Symba helps companies facilitate that, and her journey as an entrepreneur. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When creating a website, there’s no shortage of choices for how to do it. Builders must make strategic decisions about the language or framework they want to adopt. An important first consideration for many is selecting a web application framework like React or Vue. Motivated by a low page response time and good user experience, many developers want their site to be server-side rendered. Nuxt.js is a free and open-source web application framework based on Vue.js which, among other benefits, brings server-side rendering to Vue.js developers. In this episode, I interview Alex Lichter, founder of Development and Nuxt.js maintainer. We discuss the features and Nuxt and what role it can play in your next web application. Show Notes:

https://blog.lichter.io/

https://vitejs.dev/

https://twitter.com/thealexlichter

Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Application observability is a fairly mature area. Engineering teams have a wide selection of tools they can choose to adopt and a significant amount of thought leadership and philosophy already exists giving guidance for managing your application. That application is going to persist data. As you scale up, your system is invariably going to experience problems beyond common metrics like response time, CPU usage, and throughput. At some point, you’ll experience data issues that, in many organizations, are discovered manually during a root cause analysis which links a downstream problem to a change in the database that wasn’t caught. Metaplane is a monitoring tool for your data warehouse. It monitors your tables and raises alerts when issues such as anomaly detection occur. In this interview, I speak with Kevin Hu, Co-founder, and CEO at Metaplane. We discuss how their solution brings observability into the data warehouse. https://twitter.com/metaplane Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Consumers are increasingly becoming aware of how detrimental it can be when companies mismanage data. This demand has fueled regulations, defined standards, and applied pressure to companies. Modern enterprises need to consider corporate risk management and regulatory compliance. In this interview, I speak with Terry O’Daniels, Director of Engineering (Risk & Compliance) at Instacart. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Thanks to the amazing books, blogs, videos, quickstarts, frameworks, and other software-related resources, getting started as a software engineer is easier than ever. Although you can get started in a day, it can take years to become a master of the craft and most practitioners describe it as a profession of lifelong learning. Titus Winters is a Senior Staff Software Engineer at Google and the author of the book Software Engineering at Google, often known as “the flamingo book”. This book is not just tips for structuring, writing, and testing code. It’s a resource that outlines all the facets of the software engineering practice that apply in professional settings through the lens of lessons learned at Google. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

With a few impressive exceptions, software is rarely written by one person. It takes a team and as that team outgrows a single shared office, coordination and communication become emergent problems. There are lots of lessons to be learned from companies that have already found approaches that scale. In this episode, I interview Tramale Turner, Head of Engineering, Traffic, and Seattle Site Lead at Stripe. We discuss infrastructure, organizational structure, and some insights made at Stripe. It’s an in-depth conversation with useful advice for all stages of the journey of a modern software engineer. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Neural networks, in particular, deep neural networks have revolutionized machine learning. Researchers and companies have pushed on the efficiency of every aspect of the machine learning lifecycle. The impact of the trained models is particularly significant for computer vision and in turn for autonomous driving and security systems. In this episode, I interview Forrest Iandola, Head of Perception at Anduril Industries. We discuss their Lattice solution which can plug into a variety of devices such as ground vehicles, security cameras, and drones. It does AI and sensor fusion out-of-box. We discuss field deployments of these systems and how to run ML on the edge. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern businesses run on the cloud and increasingly so they run on multi-cloud infrastructure. As any growing company can tell you, cloud costs can easily run far out of control. Today’s enterprises are trying to deliver new products and services at a fast pace. That needs to be done in a cost-effective, ideally cloud-agnostic way. In this interview, I speak with Jake Reichert, VP of Engineering at Yotascale, we discuss Yotascale’s solution and how they’ve helped companies like Zoom navigate the challenges of rapid growth. Yotascale offers a comprehensive spectrum of cloud cost management solutions and we get into how modern enterprises are using these tools for optimization, governance, and more. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

To many people’s surprise tech sales is not much of an art. It’s actually a regimented science where reps have clear step-by-step processes to bring in new business. Each stage takes the customer closer to the end of the deal and consists of learning more about the customer’s needs. A CRM is a database reps use to track this customer journey allowing sales leaders to forecast their revenue, but the output is only as good as the input sales reps provide. Given the sales rep is the only person who talks to the customer, there’s no way for sales leaders to check the quality and completeness of the information reps are providing. This is where products like Gong come in. Today I met with a very special guest, Gong's Executive Vice President of Research and Development, Ohad Parush. Gong gives their customers insight into sales conversations to provide more accurate data to sales managers. This in turn allows sales managers to forecast revenue better and coach their teams with greater detail...turning sales data from a game of telephone into a coordinated team sport. This episode is hosted by David Cohen. David is a Software Engineering Lead at LinkedIn where he works on backend applications and APIs that power their enterprise data systems. In his free time, he is an AI enthusiast and enjoys talking about all things Software. You can contact him on LinkedIn or Twitter. Show Notes: Gong.io Gong Culture Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The banking industry uses technology that some modern software engineers may regard as out of date or old-fashioned. Entrepreneurs wanting to create products in the banking space historically faced a steep curve to build software that could integrate with established banking systems. Christopher Dean seeks to change that. He founded Treasury Prime, a company that offers a suite of APIs to embed a full range of banking services from cards to account opening to payments. In this episode, we talk about how that solution works behind the scenes and what APIs are available for developers. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The internet is a layer cake of technologies and protocols. At a fundamental level, the internet runs on the TCP/IP protocol. It’s a packet based system. When your browser requests a file from a web server, that server chops up the file into tiny pieces known as packets and puts them on the network labeled with your machine’s address as its destination. That system works incredibly well for receiving a file from a web server. If some of the packets arrive out of order, that’s not a problem. If one is lost, it can be sent again. There are no guarantees in a packet based system. If a flood of new packets show up, the system can slow down and you may experience a lag in the response time. This can be annoying when visiting a blog that is slow to load, but it’s not a ruined experience. Streaming video, on the other hand, does not degrade elegantly in this situation. No consumer wants to have their experience interrupted by a spinning wheel. Traffic can be spikey and unpredictable, especially around live events. The stakes are high for building a scalable, effective streaming video solution. Amit Mishra is a member team at FOX which is responsible for building platform-to-live stream content across all the FOX properties. In this episode we discuss some of the technical milestones on delivering this platform and why golang was the right choice. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning models must first be trained. That training results in a model which must be serialized or packaged up in some way as a deployment artifact. A popular deployment path is using Tensorflow.js to take advantage of the portability of JavaScript, allowing your model to be run on a web server or client. Gant Laborde is Chief Innovation Officer at Infinite Red, a React Native consulting team and the author of Learning TensorFlow.js: Powerful Machine Learning in JavaScript from O’Reilly. In this interview, we explore use cases for Tensorflow.js. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

It does not matter if it runs on your machine. Your code must run in the production environment and it must do so performantly. For that, you need tooling to better understand your application's behavior under different circumstances. In the earliest days of software development, all we had were logs, which are still around and incredibly useful. You’re likely to also consider Application Performance Monitoring tools or APM. Observability is an evolving and important feature of any software system. In this episode, we interview Christine Yen, co-founder, and CEO of Honeycomb.io. We talk about getting her start in software, meeting her co-founder while working at Parse, and how some of the experiences shared there shaped their vision for an idealized tool for figuring out what’s happening with your system. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Angular is a free and open-source web application framework. It’s maintained by the Angular team at Google. It’s used by millions of web applications and has a strong ecosystem of core contributors and library builders. In this episode, I interview Minko Gechev, Developer Relations Lead at Google. We explore several aspects of open-source software development, Tensorflow.js, Angular, and a few other things worth sticking around for. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

It wasn’t that long ago that companies scheduled downtime in order to release an updated version of the software running their website. That’s rare today. Most developers want continuous testing, integration, and deployment. While that comes with many benefits, it also places greater demands on quality engineers who can no longer gate all updates into a single infrequency release. Liliya Frye is the Director of QA Engineering at LeagueApps, a provider of Sports League & Team Management Software. In this episode, we discuss Liliya’s experiences and modern practices for successful enterprise strategies. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The manner in which users interact with technology has rapidly switched to mobile consumption. The devices almost all of us carry with us at all times open endless opportunities for developers to create location-based experiences. Foursquare became a household name when the introduced social check-ins. Today they’re a location data platform. Ankit Patel is the SVP of Engineering at Foursquare. In this episode, we discuss Foursquare’s services, the way customers leverage location intelligence, and opportunities for areas like augmented reality. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The React Framework has seen continuous growth of adoption since its launch. There are many reasons for that, but one reason is how relatively painless it is to use react-create-app or copy some boilerplate code and have a functioning, hot reloading, live demo up and running in minutes. There is, however, a long way to do between proof of concept and enterprise app. Paige Niedringhaus is a software engineer and creator of The Newline Guide to Modernizing an Enterprise React App. In this episode, we discuss best practices and strategies for structuring your React project.. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Welcome to Software Engineering Daily; I’m your guest host, Joey Baruch. I’m the CTO at Alvarez and Marsal Data Intelligence Gateway (A&M DIG), prior to which I co-founded and was CTO of HuMoov, a vertical SaaS. I’ve been a software engineer at PayPal, IBM Research Labs, and Qualcomm via the acquisition of Wilocity. Joining me is Fernando Gómez-Baquero, the Director of the Runway and Spinouts Program at the Jacobs Technion-Cornell Institute at Cornell Tech, which is the technology campus of Cornell University located on Roosevelt Island in New York City. Fernando is a nanomaterials scientist and serial entrepreneur. He’s built companies around composite aerospace materials, technologies for energy storage and has mentored 100s of companies - mostly with deep tech innovations at their core. He also manages a portfolio of 82 companies created at Cornell Tech and the Jacobs Institute, with a combined enterprise value of over half a billion USD. Fernando joins us to talk about his experience, speaking to the engineer with the entrepreneurial spirit, encouraging the journey, and laying out the core attributes of a process that can be taught, which will help the entrepreneur build a great company. He reminds us of what’s mission-critical and what can be left for another day. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Show NotesCornell Tech: https://tech.cornell.edu/ Runway Program at the Jacobs Technion-Cornell Institute: https://tech.cornell.edu/programs/phd/startup-postdocs/ Lunchclub: https://lunchclub.com/ NewLab: https://newlab.com/ Company: https://company.co/ Activate: https://www.activate.org/ Tech.NYC: https://www.technyc.org/ Aren.ai : https://aren.ai/ Nanit: https://www.nanit.com/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

According to builtwith.com, more than 10 million websites are powered by React framework. Of the top 10k sites by traffic, 44.7% of those are built with React. This javascript framework is capable of powering a wide array of modern applications and remains fairly beloved by developers that use it. In this episode, I interview Kent C. Dodds, Software Engineer Educator. We discuss Kent's journey learning React and keeping up with the changes it's taken on as the framework evolves and how those lessons culminated into his Epic React course. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern business applications are complex. It's not enough to have raw logs or some basic telemetry. Today's enterprise organizations require an application performance monitoring solution or APM. Today's applications are complex distributed systems whose performance depends on a wide variety of factors. Every single line of code can affect production and teams need insights into the health of the system and how to improve them. In this episode, I interview Omri Sass and Hugo Kaczmarek from Datadog, a provider of cloud monitoring as a service. We discuss the APM space and the challenges faced by modern enterprise teams. We also get into some detail about their new live search option and why that solution was an important addition to their product suite. Full disclosure: Datadog is a sponsor of Software Engineering Daily. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

One of the most painful parts of getting started on a new development team is getting one’s environment set up. Whether it’s undocumented steps, overly complex setups, or simply the challenges of understanding how the pieces fit together, getting a dev environment up often feels like a chore to be suffered through in order to do what I want: contribute to the codebase. Gitpod seeks to solve these and other common challenges. With Gitpod, you can spin up fresh, automated dev environments, in the cloud, in seconds. In this episode, I interview Sven Efftinge, CEO of Gitpod. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The notebook paradigm of coding is relatively new in comparison to REPLs and IDEs. Notebooks run in your browser and give you discrete cells for running segments of code. All the code in a single cell runs at once, but cells run independently. Cells can be re-run, which is a blessing and a curse. The ability to run cells out of order can make it difficult for users to have a clear understanding of what else they might want to re-compute. The NB Safety project is an easy-to-install tool for automated management of notebook states which can help you catch bugs early. Stephen Mackey is a Ph.D. student in the Data and Information Systems laboratory at UIUC. In this episode, we discuss Jupyter Notebooks, the development of a custom kernel, and how NB Safety can help notebook users. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Virtual meetings were growing in popularity before the need accelerated as a result of the pandemic. Gather is a place where you can create a space for your community today. Users who join find themselves in a shared virtual space that offers the ability to interact with other users as well as interact with the environment itself. In this episode, I interview Phillip Wang, CEO at Gather about the platform, online communities, and bridging the gap between people, technology, and connecting with each other. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Imagine a world where you own some sort of building whether that’s a grocery store, a restaurant, a factory... and you want to know how many people reside in each section of the store, or maybe how long did the average person wait to be seated or how long did it take the average factory worker to complete their assembly task. Currently today these systems today are either not using AI and instead use a mix of sensors and buttons to track certain actions or they do use AI but in a way that’s highly specific to their use case and hard to easily modify for new use cases that come down the line. This is where BrainFrame comes in. BrainFrame is a tool that connects to all your on-prem cameras and lets you easily leverage AI models and business logic. Alex Thiele is the CTO of Aotu the company that makes BrainFrame and he joins me today to talk about BrainFrame and the vision for a future where computer vision can be run by anyone. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Show Notes: BrainFrame

OpenCV BrainFrame Capsules

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The gig economy involves independent contractors engaging in flexible jobs. Today gig workers often get work from centralized platforms that facilitate the process of connecting workers with employers in exchange for a fee. Some workers find the relationship between worker and platform to be adversarial in nature since the platform can establish and enforce rules at its own discretion. In this episode, I interview Adam Jackson, Founder & CEO of Freelance Labs, builders of Braintrust. We discuss the state of the gig economy and his vision for how Braintrust can create a new kind of marketplace. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Venture capital investment has continued to flow into technology startups. No one builds technology from scratch. There are cloud services, software libraries, 3rd party services, and software platforms that modern entrepreneurs must adopt to build their products efficiently and quickly. These layers of infrastructure are a key area for many investors. In this episode, I interview Tim Chen, managing partner of Essence VC, a venture fund on a mission to help highly technical founders go from zero to one. We discuss his approach to investment in infrastructure companies, developer tools, and similar areas for early-stage investments. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Whether you love them or hate them, share them or ignore them, you encounter memes all over the internet. Those that are popular can often take off and spawn a long history of remixes, variants, derivatives, and inspired works. In this episode, we interview Johan Unger, the founder of meme.com. They’re creating a platform for Meme Explorers to track these Memes and earn rewards along the way. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The last 15 years have seen the emergence of cloud-based developer APIs and services as dominant components of the developer toolchain. As a result, there has never been more power at developers’ fingertips. But making that power usable and accessible is a challenge that is shared between the providers and the consumers of these services. Google’s Developer Relations team has spent the past 15 years bridging the gap between Google’s internal engineering teams and the broader developer community that consumes their APIs and services. Luke Mahé is a long-time engineering manager at Google DevRel. We talk about the history of Google’s developer offering and the critical role DevRel plays in shaping and promoting that offering. This episode is hosted by Yaniv Bernstein. Yaniv Bernstein was until recently COO of Airtasker and is now a startup coach, adviser, and public speaker. He is the author of the People Engineering newsletter and is active on LinkedIn and Twitter. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The expression firing on all cylinders dates back to the early 1900s and refers to a function of the internal combustion engine. This expression poetically applies to successful businesses as well. Each department must operate at peak performance and the couplings between departments need optimization as well. In this episode, I interview business coach Jon Dwoskin about a variety of topics related to entrepreneurship and how to engineer a successful organization. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Infrastructure as Code is an approach to machine provisioning and setup in which a programmer describes the underlying services they need for their projects. However, this infrastructure code doesn’t compile a binary artifact like traditional source code. The successful completion of running the code signals that the servers and other components described in the configuration file have been created automatically by the tools being used. In this episode, I speak with Christian Tragesser, DevOps Consultant with World Wide Technology. We discuss the tools and processes that are helping modern development groups solve technical problems. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As our guest today points out, most enterprise software applications are essentially forms for collecting data. The tag and related components started appearing in HTML fairly early and those same concepts are still in use with modern web browsers. However, the technology for capturing state, validating input, and providing other common services for the management of form data has continued to evolve in many languages and frameworks. Erik Rasmussen is the author of many popular open-source libraries including Redux Form and React Final Form. In this interview, we discuss the need for those tools to fill in the market and some of the modern approaches for form state management. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Show Notes: Express, Webpack, SSR, Redux example repo

Redux Form

Final Form

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

By most accounts, demand for software engineers exceeds supply. Not just anyone can develop this skill set to the level required to deliver enterprise-grade production code. For those that can, companies are incentivized to take extra measures to ensure software engineers are as productive as possible. The pace of business is often throttled by the pace of software releases. In today’s episode, we speak with Utsav Shah about developer productivity in the context of the monolith, CI/CD, and best practices for growing teams. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The first industrial deployments of machine learning and artificial intelligence solutions were bespoke by definition and often had brittle operating characteristics. Almost no one builds custom databases, web servers, or email clients. Yet technology groups today often consider developing homegrown ML and data solutions in order to solve their unique use cases. Today’s modern data stack is a patchwork of interconnecting tools built to suit a variety of personas that need to interact with the data in notably different ways. In this episode, I speak with Leigh Marie Braswell, an Investor with Founders Fund. We have a wide-ranging discussion about the technology landscape of data and machine learning solutions and the modern enterprise data stack. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Phishing attacks, malware, and ransomware are just some of the major threats everyone connected to the internet faces. For companies, the stakes are especially high. Setting up a secure infrastructure is difficult. Your adversary only needs to find one flaw to get in. Vancord is a private cybersecurity company, based in Connecticut, that was founded and built by security engineers to specialize in incident resilience and response. In this episode, I interview Jason Pufahl and Russell Jancewicz from Vancord. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In a version control system, a Monorepo is a version control management strategy in which all your code is contained in one potentially large but complete repository. The monorepo is in stark contrast to an alternative approach in which software teams independently manage microservices or deliver software as libraries to be imported in other projects. The monorepo strategy has been followed by noteworthy companies such as Google, Facebook, and Microsoft. Derrick Stolee is a software engineer working at GitHub. He joins us today to talk about strategies for monorepos and innovations to the way git works in order to better support this style of the repository. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Companies that gather data about their users have an ethical obligation and legal responsibility to protect the personally identifiable information in their dataset. Ideally, developers working on a software application wouldn’t need access to production data. Yet without high-quality example data, many technology groups stumble on avoidable problems. Organizations need a solution to protect privacy while simultaneously preserving aspects of the data which are important. Tonic is automating data synthesis to advance data privacy. Their solution gives your production-like data for development and analytical purposes without compromising on data quality or privacy. In this episode, I interview Tonic’s CEO Ian Coe, and Head of Engineering Adam Kamor. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Applications write data to persistent storage like a database. The most popular database query language is SQL which has many similar dialects. SQL is expressive and powerful for describing what data you want. What you do with that data requires a solution in the form of a data pipeline. Ideally, these analytical workflows can follow similar best practices to those handled in application code. DBT is a transformation workflow that lets teams deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation. Users who know SQL can build production-grade data pipelines. In this episode, I interview Tristan Handy, CEO, and founder of DBT Labs. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The way we write, compile, and run software has continued to evolve since computer programming began. The cloud, serverless, no-code, and CI/CD are all contemporary ideas introduced to help software engineers spend more time on their application and less time on the chores of running it. Darklang is a new way of building serverless backends. It’s a cloud native programming language with several novel features including being not just serverless but deployless as well. In this episode, I interview Paul Biggar, founder and CEO of Dark. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Tedious, repetitive tasks are better handled by machines. Unless these tasks truly require human intelligence, repetitive tasks are often good candidates for automation. Implementing process automation can be challenging and technical. Increasingly, engineers are seeking out tools and platforms to facilitate faster, more reliable automation. In this episode I talk to Yaseer Sheriff, Co-Founder and CEO at Axiom about no-code solutions, process automation, and some of the challenges in developing the software powering those services. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Financial technology or fintech has always been a hot topic. This is increasingly true in recent years as disruptive companies enter the market to give better alternatives and solutions to consumers. Current is focused on creating better financial outcomes. In addition to providing banking services, their app has many tools and reminders to help users learn and execute better money management strategies. In this episode, I interview Trevor Marshall, CTO of Current about starting a fintech company, how banking software is written, and what opportunities exist for disruption. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The dream of machines with artificial general intelligence is entirely plausible in the future, yet well beyond the reach of today’s cutting edge technology. However, a virtual agent need not win in Alan Turing’s Imitation Game to be useful. Modern technology can deliver on some of the promises of narrow intelligence for accomplishing specific tasks. PeopleReign has created a virtual agent for IT and HR employee service. This agent’s goal is not to replace a human agent but to augment them by handling some requests and elegantly handing off to a human in other cases. In this episode, I speak with Dan Turchin, CEO of PeopleReign about their virtual agent and the future of work. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A developer’s core deliverables are individual commits and the pull requests they aggregate into. While the number of lines of code written alone may not be very informative, in total, the code and metadata about the code found in tracking systems present a rich dataset with great promise for analysis and productivity optimization insights. LinearB is a systematic approach to engineering improvement. Their WorkerB Slack Bot connects with teams on an individual level to help with productivity and collaboration. In this episode, I speak with Dan Lines, Co-Founder, and COO. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Show Notes:

LinearB.io

Learn more about WorkerB

Check out the Dev Interrupted Podcast

Save your spot for the INTERACT conference

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Interest in autonomous vehicles dates back to the 1920s. It wasn’t until the 1980s that the first truly autonomous vehicle prototypes began to appear. The first DARPA Grand Challenge took place in 2004 offering competitors $1 million dollars to complete a 150-mile course through the Mojave desert. The prize was not claimed. Since then, rapid progress has begun in autonomous driving fueled by advances in sensor technology, software, and the hardware which runs it. Infrastructure has become a serious consideration for autonomous vehicle companies. In this episode, I speak with Vinoj Kumar about infrastructure at Cruise, the company helping Walmart do all-electric, self-driving grocery delivery. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Money laundering is not a new crime. However, the growth of digital communications has greatly expanded the opportunity for money launderers to find innovative new ways to hide their true intent. Some estimates suggest it could be as high as 2-5% of the world’s GDP. Unit21 is a customizable no-code platform for risk and compliance operations. They offer a simple API and dashboard for detecting and managing money laundering and fraud. Today on the show, I speak with Clearance Chio, co-founder and CTO of Unit21. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Web applications often have some sort of login system, and once a user creates an account, they have access to features anonymous users can’t see. In time, application designers will often add an admin level of access for special users. This is often a slow trickle of technical debt. Proper execution of a programmatic authorization system requires concepts like roles, resources, departments, and organizations. OSO describes itself as batteries included authorization. It’s an open source library used by companies like Intercom and Wayfair whicseh allows them to manage authorization in a robust and standardized framework without reinventing the wheel. In this episode we speak with Sam Scott, CTO at OSO. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As developers hone their craft, becoming more productive often means learning utilities and tools at the command line. The right combination of various parsing commands chained together through pipes can enable engineers to quickly and efficiently automate many adhoc data processing tasks. In this episode I speak with Adam Gordon Bell about some of his favorite command line tools. We also discuss his role as a developer advocate for Earthly, a powerful tool for building software in a repeatable and understandable way. Adam is also the host of the Co-Recursive podcast. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern companies leverage dozens or even hundreds of software solutions to solve specific needs of the business. Organizations need to collect all these disparate data sources into a data warehouse in order to add value. The raw data typically needs transformation before it can be analyzed. In many cases, companies develop homegrown solutions, thus reinventing the wheel and possibly planting deep rooted seeds of technical debt. Mozart Data helps you collect all of your data sources in under an hour. They provide managed data pipelines, data warehousing, and transformation automation. In this episode, I interview CEO Peter Fishman and CTO Dan Silberman about the modern data stack. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

An application network is a way to connect applications, data and devices through APIs that expose some or all of their assets and data on the network. That network allows other consumers from other parts of the business to come in and discover and use those assets (mulesoft.com). The company Tetrate provides the tools necessary for a highly efficient application (aware) network. Tetrate helps connect and manage applications across clusters, clouds, and data centers. It supports integrating traditional workloads into your cloud-native application infrastructure, defining access control and editing rights or teams on shared infrastructure, and out of the box conformance with NIST standards for microservices security. In this episode we talk to Varun Talwar, Co-Founder of Tetrate. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A software engineer will make many mistakes on their career journey. In time, engineers learn to make smaller mistakes, recognize them faster, and build with appropriate guardrails. The demands of delivering software in a timely and efficient fashion often demands developers carefully optimize tradeoffs to deliver solutions to the problems at hand. Software Mistakes and Tradeoffs: How to Make Good Programming Decisions is the book by Tomasz Lelek and Jon Skeet. In this episode, we interview Tomasz about his experiences as a software engineer and sample the advice found in this book. Listeners interested in a copy can use the special discount code sedlelek35 at manning.com. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

By most accounts, the first databases came on line in the 1960s. This class of software has continued to evolve alongside the technology it runs on and the applications it supports. In the early days, databases were typically closed source commercial products. Today, databases run in the cloud on distributed systems. Increasingly, the leading tools are open source yet frequently supported by a related commercial entity offering managed services and white glove support. In this episode, we interview Jonathan Ellis, CTO of DataStax and Spencer Kimball, CEO of Cockroach Labs about the current state of distributed databases and the open source ecosystem. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Instabase is a technology platform for building automation solutions. Users deploy it onto their own infrastructure and can leverage the tools offered by the platform to build complex workflows for handling tasks like income verification and claims processing. In this episode we interview Anant Bhardwaj, founder of Instabase. He describes Instabase as an operating system. We explore what he means by that and discuss the types of use cases Instabase powers. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Israeli Tech Radar is an opinionated map of the latest technologies and trends in the Israeli tech industry. Now in its fifth edition, the Tech Radar was built in collaboration with Monday, Wix, Riskified, Netapp, Tabula, and other tech companies. Lior Kanfi is the CEO of Tikal. In this episode, we interview Lior about the Tech Radar as well as his thoughts for addressing the current tech talent shortage. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

This episode is hosted by Kyle Polich of the Data Skeptic podcast. We're glad to welcome Kyle to the Software Daily team. Becoming a contributor to an existing software project can be a daunting task for an engineer. A common convention is to add a README file to your repository to serve as a trailhead which gives new visitors step by step instructions for running, exploring, and understanding the structure of the codebase. The Missing Readme is the recently published book which prepares new software engineers to both survive and succeed. Learning to code in school, in a bootcamp, or independently can prepare you to write software. This book prepares you to do it effectively in a professional setting. In this episode, we talk to authors Chris Riccomini and Dmitriy Ryaboy. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Whether organizing projects, working from home, or conducting business, you need to use many necessary apps and cloud services to do the job. Despite these apps being necessary, switching between them and keeping them interconnected and updated is a challenge. They very easily become disorganized which makes using them less efficient. The company ClickUp solves this problem by providing a platform that provides everything a business needs to efficiently operate, all in one place! ClickUp uses a unique hierarchy to see the big picture without missing details, and offers everything from docs, to tasks, to imports and integrations. With everything in one place and easily searchable and organized, ClickUp makes work much easier. In this episode we talk to Zeb Evans, Founder and CEO of ClickUp. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes is an open source container orchestration service released by Google in 2014. It has quickly grown into a platform with a huge community of enthusiasts and professionals. Besides becoming the de facto standard for container orchestration, it has fostered an ecosystem of related tools and services with increasing power and sophistication (opensource.com). Argo, a project developed by the company Applatix and acquired by Intuit, is a set of essential Kubernetes-native tools for deploying and running jobs and applications on Kubernetes. All the Argo tools are implemented as controllers and custom resources. Some of these tools are Argo Workflows, Argo Events, Argo CD, and Argo Rollouts (intuit.com). In this episode we talk to Alex Collins, Principal Software Engineer at Intuit, about using Argo to manage Kubernetes applications. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

ETL stands for “extract, transform, load” and refers to the process of integrating data from many different sources into one location, usually a data warehouse. This process has become especially important for companies as they use many different services to collect and manage data. The company Grouparoo provides an open source framework that helps you move data between your data warehouse and all of your cloud-based tools. This process of moving data back from the data warehouse to the applications is called reverse ETL, and is important for things like marketing campaigns and customer service. It easily integrates with your developer’s tools and is free and easy to install. In this episode we talk with Brian Leonard, CEO at Grouparoo. This interview will also be published as a video episode on our YouTube channel. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the late 1970s a printer at MIT kept jamming, resulting in regular pileups of print jobs in the printer’s queue. To solve this problem, some computer scientists wrote a software program that alerted every user in the backed up queue “The printer is jammed, please fix it.” When a man named Richard Stallmen was refused a copy of the program code, he resolved to create a publicly available operating system and the open source movement was born (opensource.com). Over 50 years later, open source has become a coding philosophy practiced by millions of software engineers around the world. Why is open source so popular? What difference has it really made in software engineering, and what major projects are open source? In this episode we talk to William Morgan, CEO at Buoyant and creator of the open source service mesh Linkerd. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google uses automated programs called spiders, or crawlers, to index and rank web pages. Then, when a user searches for something, it uses a special algorithm to determine the order of results to display (howstuffworks). This process, of course, applies to web pages on the internet. There are 2 major projects, worked on by the company Edge & Node, that do what Google does for the web, but for the blockchain. The first is called The Graph, which is an indexing protocol for organizing blockchain data and making it easily accessible. The second is Everest, a universally shared projects registry of onchain data. In this episode we talk to Nader Dabit, Developer Relations Engineer at Edge & Node. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

“In October 1958, Physicist William Higinbotham created what is thought to be the first video game. It was a very simple tennis game, similar to the classic 1970s video game Pong, and it was quite a hit at a Brookhaven National Laboratory open house” (aps.org). 63 years have passed, and video games are prolific. The company Pragma provides a backend game engine that enables developers to create multiplayer and social games. Pragma’s features are divided into three broad categories: cross platform accounts to connect players, Game Loop, which is lobby, matchmaking, game allocation, and end of match processing, and storing, saving, and updating accounts, inventory, progression, and related data. In this episode we talk with Eden Chen, CEO and Co-Founder of Pragma. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Time series data are simply measurements or events that are tracked, monitored, downsampled, and aggregated over time. This could be server metrics, application performance monitoring, network data, sensor data, events, clicks, trades in a market, and many other types of analytics data (influxdata.com). The platform InfluxData is designed for building and operating time series applications. InfluxData is engineered for growth with enterprise-grade security, ingests metrics, events and logs in a high-performing time series database, and platform analytics for detecting and resolving problems. In this episode we talk to Russ Savage, Director of Product Management at InfluxData. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Domain names are the address of your website that people type into the browser URL bar. Once purchased, a domain name is stored on your behalf by custodians like Google domains. Blockchain domains, on the other hand, are similar to regular domain names except they are stored and controlled in your cryptocurrency wallet. The company Unstoppable Domains provides blockchain domains that enable users to do things like keep a universal username across apps and websites, act as a website URL, act as a payment address for wallets, and much more. Replace cryptocurrency addresses with human readable names! In this episode we talk to Matthew Gould, Founder and CEO of Unstoppable Domains. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Pay-as-you-go pricing has become a strong selling point for modern SaaS companies as well as cloud-based companies. Public cloud providers, for example, typically only charge you for only what you use. But implementing this option is challenging because it requires advanced platform analytics. The company Octane is a drop-in metered billing system that gives you flexibility to bill how you want. They offer expressive price plans that bill on usage, include base rates, add-ones, discounts, and trials. Their services can be integrated easily with simple APIs and SDKs, and generate detailed breakdowns of customer usage and spending with customizable reports and dashboards. In this episode we talk to Akash Khanolkar, Co-founder at Octane. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Whether sending messages, shopping in an app, or watching videos, modern consumers expect information and responsiveness to be near-instant in their apps and devices. From a developer’s perspective, this means clean code and a fast database. Apache Druid is a database built to power real-time analytic workloads for event-driven data, like user-facing applications, streaming, and anything else that requires instant data visibility. Druid offers lower latency for OLAP-style queries, time-based partitioning, fast search and filter, and out-of-the-box integration with Apache Kafka, AWS Kinesis, HDFS, AWS S3, and more (Druid.apache.org). In this episode we talk with Eric Tschetter, Field CTO at Imply, Fellow at Splunk, and experience developing large swaths of backend infrastructure largely focusing on Druid. We discuss the use cases and power of Apache Druid. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

This episode was published on the GeekWire podcast. Subscribe to GeekWire for more great content. Facebook CEO Mark Zuckerberg told employees recently that the company’s long-term goal is to “bring the metaverse to life” — helping to create an interconnected world of physical, virtual and augmented reality spaces that will reshape the way we work, interact with each other, create new things, and generally experience life. So how exactly will Facebook approach such an audacious plan? A new book called “Move Fast: How Facebook Builds Software” doesn’t delve into the metaverse, specifically. But in looking at Facebook’s engineering practices — the way the company makes stuff — the book examines the digital DNA of the social network, sheds new light on its most infamous motto, and explains the inner workings of a company that wants to reshape the human experience, again. Facebook influences the engineering culture and economy not just in its hometown of Menlo Park, Calif., but also in its development offices in the Seattle area, where it employs 7,000 people. And of course, ultimately, Facebook’s internal practices end up influencing people around the world who use its products. On this episode of the GeekWire Podcast, we talk with the author of the book, Jeff Meyerson, the longtime host of the Software Engineering Daily podcast, about the ways Facebook, Microsoft, Amazon and Google make things, and what those different approaches tell us about where they’re taking us. Listen above, or subscribe to GeekWire in any podcast app. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Auren Hoffman is the CEO of SafeGraph. In this episode we discuss data as a service and more. This interview was also recorded as a video podcast. Check out the video on the Software Daily YouTube channel. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Serverless computing is a cloud computing solution that lets developers deploy applications to containers without managing the servers themselves. Servers and resources are provisioned automatically, pay only for what you use, and experience little to no errors or downtime (ionos). Google Cloud Run is a managed compute platform that enables you to run containers that are invocable via requests or events. Cloud Run is serverless, transparent, and easy to use. Common use cases include web services like REST APIs, data processing, automation, and modernization (cloud.google.com). In this episode, we talk with Steren Giannini, Senior Product Manager at Google. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

According to Fugue’s new State of Cloud Security 2020 report, cloud misconfiguration remains the top cause of data breaches in the cloud, and millions of database servers are currently exposed across cloud providers. Some of the leading reasons are a lack of adequate oversight and too many APIs and interfaces to govern. (securityaffairs.co). Argos Security is a SaaS that detects and remediates misconfigurations of cloud assets within minutes. Argos detects and pinpoints exploitable issues in cloud environments, delivers a complete, real-time view of your cloud security posture, gives meaningful alerts and few false flags, and integrates in minutes with AWS, GCP, and Azure. In this episode, we talk with David O’Brien, Founder of Argos Security. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Latency is the time it takes to get from point A to point B. In programming, this might be the time from a user selecting their photos library to the pictures reaching their computer screen from the database. Fly.io is a simple platform for running full-stack apps and databases close to your users. Some available features include provisioning CPU, memory and storage based on your app’s demand, zero-configuration private networking and global load balancing, metrics and alerting, and managing SSL certificates at any scale. Fly.io beta is launching highly available PostgreSQL clusters with a single command and creating read replicas in different cities. In this episode, we talk with Kurt Mackey, Social Media Manager at Fly.io. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Direct-to-Consumer companies sell their products without going through a traditional middleman like an outlet store or wholesaler. By posting content on platforms like TikTok, Facebook, and YouTube, companies can reach millions of users.TikTok, for example, has an estimated 1 billion monthly active users (wallaroomedia). The company Kindred Studios helps direct-to-consumer companies create consistent, beautiful content that holds viewers’ attention and generates sales. They showcase your products’ fine details and selling points with product videos, mood photography, stop motion videos, texture photography, and many other content categories. In this episode, we talk to Daniel Snow, Co-Founder of Kindred Studios, as well as The Snow Agency. He specializes in the eCommerce space and joins us to discuss social media and content creation. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ryan Noon is the CEO of Material Security. This interview was also recorded as a video podcast. Check out the video on the Software Daily YouTube channel. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The term “boilerplate code” refers to code sections that are repeated across many projects with little to no variation. Every developer is familiar with boilerplate code, whether it be pom.xml files in Java or setting up React.js applications, tweaking boilerplate code for every project is inevitable. Actually, the company Wasp believes writing boilerplate code doesn’t have to be a part of web app development. Wasp is an open-source, declarative DSL for developers who want modern frontend, backend, and deployment experiences without writing boilerplate code. With Wasp, you can describe high-level features with Wasp DSL and write the rest of your logic in React, Node.js, and Prisma. In this episode, we talk to Matija Sosic, CEO and Co-Founder at Wasp. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes is an open-source container orchestration system. It makes managing container clusters possible as well as deploying code changes to these containers. Microservice architecture is widely used today in large part because of Kubernetes. However, using it can require a large time commitment due to its learning curve. The company Okteto empowers developers to innovate and deliver cloud-native applications faster than ever. Okteto CLI lets developers deploy realistic replicas of their stack on Kubernetes and updates it for continuous deployments. It also manages different code environments, self-service access, and container scaling automatically. In this episode, we talk with Ramiro Berreleza, Founder and CEO of Okteto, about managing Kubernetes clusters with Okteto. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Blockchain protocols like Bitcoin and Ethereum have changed the cyber world dramatically in the last decade. They’ve created communities of like-minded developers, generated new financial markets, and popularized “decentralization” in computer networks. However, they require large resources to operate which makes scaling difficult and transactions expensive. Hedera is a decentralized public network that takes the previous innovations a step further. The Hedera Consensus and Token service enables developers to make tokens and create publicly verifiable and scalable timestamps for any application at an incredibly fast speed, highest grade security, and extremely low bandwidth consumption. In this episode, we talk to Leemon Baird, Founder and Chief Scientist at Hedera Hashgraph. We discuss proof-of-stake public networks, the hashgraph consensus, and the future of blockchains. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Enterprise data warehouses store all company data in a single place to be accessed, queried, and analyzed. They’re essential for business operations because they support managing data from multiple sources, providing context, and have built-in analytics tools. While keeping a single source of truth is important, easily moving data from the warehouse to other applications is invaluable. The company Hightouch provides tools that easily move data from your warehouse to important business tools like Salesforce, Apache Airflow, Tableau and more. Hightouch uses SQL to move and transform your data and tracks every row to avoid moving data that hasn’t changed. Failed rows are retried, and all changes to rows are logged. In this episode, we talk to Tejas Manohar, Founder of Hightouch. We talk about reverse ETL, managing data across multiple systems, and how Hightouch helps companies operationalize their data warehouse. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In this episode we discuss venture capital and more with Preface Ventures Founder and General Partner Farooq Abbasi. This interview was also recorded as a video podcast. Check out the video on the Software Daily YouTube channel. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In today’s containerized world, it’s common to encounter similar issues with known solutions across multiple pods. For most people there are 2 solutions: go pod-by-pod finding and fixing the problem, or do that while also spending months trying to automate that process. This is significant time and manual labor. The company Shoreline orchestrates real-time debugging and automated repair across fleets. Shoreline makes it easy to define metrics, alarms, actions, and bots from the CLI that take action like draining and terminating nodes tagged for retirement or finding TLS certificates that are close to expiring. For unknown issues, Shoreline lets you debug across every pod from a single CLI rather than SSHing pod-by-pod, like running a command to grep for errors. In this episode, we talk with Anurag Gupta, Founder and CEO of Shoreline. Full disclosure: Shoreline is a sponsor of Software Engineering Daily. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Prophecy enables all your teams on Apache Spark with a unique low-code designer. While you visually build your Dataflows - Prophecy generates high-quality Spark code on Git. Then, you can schedule Spark workflows with Prophecy’s low-code Airflow. Not only that, Prophecy provides end-to-end visibility into your dataflows with Metadata Search and Column Level Lineage. For Enterprises, in addition to developing new workflows, data teams also need to migrate thousands of old proprietary ETL workflows to the Cloud. For that, Prophecy has built a Transpiler that automatically converts AbInitio, Informatica, SSIS and Alteryx workflows to high-quality Spark code. Learn more at www.prophecy.io. In this episode, we speak with Raj Bains, who is the founder & CEO of Prophecy. Previously, Raj was the product manager of Apache Hive at Hortonworks through the IPO. He also headed product management and marketing for a NewSQL database startup. Raj continues to actively code in compiler and database technologies. His engineering roles include developing a NewSQL database, building CUDA at NVIDIA as a founding engineer, and working as a compiler engineer for Microsoft Visual Studio. Full disclosure: Prophecy is a sponsor of Software Engineering Daily. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the previous episode, Pulsar Revisited, we discussed how the company DataStax has added to their product stack Astra Streaming, their cloud-native messaging and event streaming service that’s built on top of Apache Pulsar. We discussed Apache Pulsar and the added features DataStax offers like injecting machine learning into your data streams and viewing real-time analytics. In today’s episode we’re going to continue this conversation with Enrico Olivelli, a Senior software engineer at DataStax and an ASF Member with The Apache Software Foundation. Apache Pulsar has released a number of exciting upgrades and enhancements in their recent 2.8.0 release. How will these changes affect Astra Streaming and what can users look forward to in future Astra Streaming releases? Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally created at Yahoo! and now a top-level Apache Software Foundation project (pulsar.apache.org). Pulsar is used by many large companies like Yahoo!, Verizon media, Tencent, and Splunk. The company DataStax, an open, multi-cloud stack for modern data apps, has added to their product stack Astra Streaming Beta, their cloud native messaging and event streaming service that’s built on top of Apache Pulsar. Astra Streaming provides tools for streaming, queueing, and stream processing as a natural complement to their other services like Astra DB. You can integrate high quality video services with added features like injecting machine learning and viewing real-time analytics. In this episode we talk with Jonathan Ellis, CTO and co-founder at DataStax. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

If you’ve ever googled a CS or programming question, you likely found an answer (or many) on Stack Overflow. Founded in 2008 and named after a common computing error, Stack Overflow empowers the world to develop technology through collective knowledge. More than 100 million people visit Stack Overflow every month making it one of the 50 most-visited websites in the world. Stack Overflow’s products include its market-leading knowledge sharing and collaboration platform, Stack Overflow for Teams, in addition to Stack Overflow Reach & Relevance, which is focused on advertising. Stack Overflow for Teams is a knowledge sharing and collaboration solution that developers and managers already know and trust. It’s for companies who need to increase productivity, decrease cycle times, accelerate time to market, and protect institutional knowledge. In this episode we talk with Tom Limoncelli, a manager at Stack Overflow, author, and tech advocate. Full disclosure: Stack Overflow is a sponsor of Software Engineering Daily. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In 2003, Google developed a robust cluster management system called Borg. This enabled them to manage clusters with tens of thousands of machines, moving them away from virtual machines and firmly into container management. Then, in 2014, they open sourced a version of Borg called Kubernetes, or K8s. Now, in 2021, CockroachDB is a distributed database designed with Kubernetes architecture in mind. CockroachDB uses regular SQL and scales by automatically distributing data and workload demands. Their databases survive machine, datacenter, and region failures, and provide guaranteed ACID compliant transactions. In this episode, we talk to Spencer Kimball, CEO at Cockroach Labs, about distributed databases and containerization. Full disclosure: Cockroach Labs is a sponsor of Software Engineering Daily. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Governments, consumers, and companies across the world are becoming more aware and attentive to the risks and causes of climate change. From recycling to using solar power, people are looking for ways to reduce their carbon footprint. Markets like the financial sector, governments, and consulting are looking for ways to understand climate data to make smart decisions and manage risk. The company Sust Global was founded as a way to deliver sustainable change and climate resilient action. Sust Global uses an AI-powered platform that combines climate science, satellite-derived data, and geospatial data sets to quantify climate change. Companies can use this analysis to evaluate risk to assets, better understand future commodities like metal, and plan for future supply chain challenges and climate perils. In this episode we talk to Josh Gilbert, CEO at Sust Global. Josh explains Sust Global’s mission and product, and discusses how companies use Sust Global to prepare and respond to climate change. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Big data analytics is the process of collecting data, processing and cleaning it, then analyzing it with techniques like data mining, predictive analytics, and deep learning. This process requires a suite of tools to operate efficiently. Data analytics can save companies money, drive product development, and give insight into the market and customers. The company Imply provides the necessary tools for safe and efficient large scale analytic workloads. Their tools help minimize operational complexity so that deploying container clusters, or bare metal, or public and hybrid clouds is simple. Infra also monitors their performance, uptime, and scale. They also create fully interactive visualizations that update with real-time data, and plenty of other tools. In this episode, we talk with Jad Naous, VP of Engineering and Product at Imply, about big data analysis and real-world use cases. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ashmeet Sidana is the founder of Engineering Capital. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

DevOps has shortened the development life cycle for countless applications and is embraced by companies around the world. But managing and monitoring multiple environments is still a major pain point, particularly when companies need to mix cloud and legacy systems. Knowing when services go down and quickly pinpointing the cause is essential for continuous development. The company Better Stack provides services for DevOps teams. Their first service, Logtail, provides SQL-compatible structured log management. Query logs like you query a database. Their second service, Better Uptime, is an infrastructure monitoring platform that monitors uptime and, when issues occur, sends voice calls, SMS, and plenty other types of alerts with screenshots and error logs of the incident. Juraj Masar, Co-Founder and CEO at Better Stack, joins us to discuss DevOps and Better Stack. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data science is an interdisciplinary field that combines strong technical skills with industry knowledge to perform a large range of jobs. Data scientists solve business questions with hands-on work cleaning and analyzing data, building machine learning models and applying algorithms, and generating dynamic visuals and tools to understand the world from the data it generates. Amazon Web Services provides tools for storing data, moving it, analyzing it, and executing algorithms and models on it. In this episode we talk to Chris Fregly, author of the book Data Science on AWS. Chris works at AWS full time on AI and machine learning projects, and joins us to discuss his book and data science more broadly. What does it take to become a data scientist, who’s his book for, and what are the latest advancements in the field? Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In this episode we discuss the new Move Fast book, as well as many aspects of the current state of software engineering. Daliana Liu interviews Jeff Meyerson, host of Software Daily and author of Move Fast. This interview was also recorded as a video podcast. Check out the video on the Software Daily YouTube channel. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Big Data has exploded the past decade as cloud computing and more efficient hardware made scaling essentially limitless. Products like Uber revolve entirely around analyzing data to provide rides. According to an EMC/IDC study, there was approximately 5.2TB of data for every person in 2020. That estimate was made before the transition to remote work, which likely makes it much higher. The term “data lineage” refers to the collection, origin, storage, transfer, and use of data over time. Given the size of the Big Data industry and related industries, maintaining a thorough data lineage, even within small companies, can be very difficult. It becomes especially challenging at scale. What innovative tools make understanding all this information possible? Can data really continue growing at this rate? In this episode we talk with Julien Le Dem, CTO and Co-Founder at Datakin. We discuss the challenges, available tools, and future for big data and data lineage. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Here is the full audiobook for "Move Fast: How Facebook Builds Software". Continue listening to the end to hear my most recent album "Simulation". Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

We talk to a lot of exciting startups from all over the world about their tech products. Recently we’ve heard from people innovating in the blockchain space, cloud infrastructure, databases, and automation tools. However, in today’s episode, we’re going to talk about how these tech startups get investments to build their great products in the first place. When do teams begin seeking investments? How do they show the promise of their products? Today we’re talking with Sarah Guo, a Venture Capitalist and General Partner at Greylock. Greylock invests in market defining founders, like the teams at Airbnb, AppDynamics, Cloudera, Docker, Okta, and Dropbox. Sarah brings a lot of valuable experience in identifying companies with potential and we’re very excited to welcome her to today’s episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The company Dynatrace provides intelligent observability, continuous automation, and causation-based AI to help Cloud Ops, DevOps, and SRE teams transform faster, innovate more, and deliver better business outcomes. They offer application performance monitoring, infrastructure monitoring, cloud automation, application security, and much more. While being an industry leader in simplifying complex cloud applications, Dynatrace is continuously innovating and bringing a wide array of solutions to the market. Who is Dynatrace for and what can it really do for software development and DevOps teams? In this episode we’d like to get to know Dynatrace better with their VP, Chief Technical Strategist and Head of Innovation Lab, Aloïs Reitbauer. Aloïs drives product innovation at Dynatrace and knows more about their products than most. We discuss the use cases and benefits of Dynatrace as well as its involvement with open source projects. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In this episode we discuss coding bootcamps, fear, fitness, and more with Ruben Harris, CEO of Career Karma. This interview was also recorded as a video podcast. Check out the video on the Software Daily YouTube channel. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When I left Amazon, it was not out of resentment or dissatisfaction. It was out of education. At Amazon, one can learn how to build Amazon. Amazon was not built through master planning, but through incremental, adjacent expansion. Once you see how to expand from a book store to the backbone of the Internet, you see how to expand from anything to everything. And so I started a podcast. We are hiring for Software Daily and for a new company, Supercompute. If you are interested in working with us, send me an email: jeff@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

SOC 2 is a security audit to prove that SaaS companies have secured their company and customer data. It’s often considered the minimum audit necessary to sell software. HIPAA is a federal law regulating how sensitive medical information about patients must be handled. ISO 27001 is the global benchmark for demonstrating your information security management system. What do these things have in common? They are all security standards that companies need to maintain and renew to be trustworthy to customers. They also take intense preparation with months of work, and hundreds of screenshots to prove compliance with auditors. The company Vanta provides automation tools to monitor your applications and maintain compliance. Fix items on your Vanta to-do list, and when you’re ready a Vanta-trained CPA will perform an audit with you. In this episode we talk with Christina Cacioppo, CEO at Vanta. We discuss the accreditation process and security needs for various companies and how Vanta is keeping companies in compliance. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

There are over 4 billion people using email. Many people using email for business communicate quick questions to colleagues, send repetitive, template-based information to potential customers and freshly hired employees, and repeat a lot of the same phrases. We actually repeat phrases in a lot of written formats. How often do you copy and paste the same thing to multiple people? The company Text Blaze is making the workday a little faster, more productive, and convenient with their shortcut-to-snippet software product. With Text Blaze you can save any snippet of text or template, including templates that need fill-in-the-blank sections, to a keyboard shortcut. Then type that shortcut in Gmail, Google Docs, LinkedIn, or Salesforce, and wherever else you need to use your saved snippet. In this episode we talk to Scott Fortmann-Roe, CTO at Text Blaze. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Continuous integration is a coding practice where engineers deliver incremental and frequent code changes to create higher quality software and collaborate more. Teams attempting to continuously integrate new code need a consistent and automated pipeline for reviewing, testing, and deploying the changes. Otherwise change requests pile up in the queue and nothing gets integrated efficiently. The company LayerCI is a platform built to deliver a better remote infrastructure experience. It enables engineers to preview full stack staging environments for every commit and have a centralized CI/CD stack with full end-to-end testing. LayerCI can duplicate a fully provisioned environment so that end-to-end workflows can run in parallel and alongside unit tests. The result is faster code review, testing and deployment. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

ELT is a process for copying data from a source system into a target system. It stands for “Extract, Load, Transform” and starts with extracting a copy of data from the source location. It’s loaded into the target system like a data warehouse, and then it’s ready to be transformed into a usable format for things like modern cloud applications. The company Meltano provides code that manages ELT pipelines through an open-source, self-hosted, CLI-first, debuggable, and extensible process. Meltano projects manage your Singer tap and target configurations to easily select which entities and attributes to extract. These pipelines track their own incremental replication state so they can pick up where the previous run left off. Once your raw data is in its target source, Meltano helps you transform it into a usable format. These pipelines can run on a schedule and be fed to supported orchestrators like Apache Airflow. In this episode we talk to Douwe Maan, founder and CEO of Meltano, about their product-market fit and delivery plans. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In this episode we discuss plug and play auth, password management, and crypto with Sean Li, co-founder and CEO of Magic. This interview was also recorded as a video podcast. Check out the video on the Software Daily YouTube channel. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

For some data problems, you may be more concerned with the state of data at a particular point. A ticket is booked, or it’s not. How many poetry submissions were made to the contest? This is relational data. For other problems, you’re concerned with the change in data over time. Solar energy consumption, for example, or price behavior. This is time-series data. TimescaleDB resembles a traditional postgreSQL database, but is supercharged for time-series data. TimescaleDB has queries that are 10x faster, is optimized for time-series and advanced time-series analytics, has automated continuous aggregations, columnar storage, and uses best-in-class algorithms and memory efficient structures to compress your data so you can store more at a much cheaper price. And for your questions that are not time-series dependent, TimescaleDB is still an efficient and cost effective relational database. In this episode we talk to Mike Freedman, Co-Founder and CTO of TimescaleDB. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The cloud has delivered amazing benefits like on-demand infrastructure that’s easy to use, pay-as-you-go subscription plans, and effortless scaling of applications. This flexibility minimizes the growing pains for businesses and explains why today’s startups and established companies are both building apps on the cloud. However, the costs of using the cloud stack up once companies reach large scale. Dropbox, for example, shifted away from the cloud in 2016 to opt for a custom built infrastructure in co-location facilities. They saved $75 million over 2 years and increased their gross margins from 33% to 67% due to this change. Actually, recent research suggests major companies spend 75%-80% of cost of revenue on their cloud bills. Should large companies shift away from the cloud like Dropbox did? Is it too late for some companies to untangle themselves from it? In this episode we talk with Martin Casado, Partner at Andreessen Horowitz and an expert in IT Infrastructure. We discuss the costs of cloud for small and large companies, and the financial implications of cloud infrastructure at scale. Show NotesMartin Casado's article on cloud costs Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Uber is one of many examples we’ve discussed on this show that has changed the world with big data analysis. With over 8 million users, 1 billion Uber trips and people driving for Uber in over 400 cities and 66 countries, Uber has redefined an entire industry in a very short time frame. It’s difficult to find precise details about Uber’s big data infrastructure online, but we know they collect every possible data point about their drivers and riders. Matching riders and drivers, setting ride fares, predicting demand for cars - these are some examples of what Uber does with its data. In this episode we talk with Kevin Novak about Uber’s data science. What are some key details about their data infrastructure? What can people expect in the future from their data methodologies? How did a tech conference in Paris turn into one of the fastest growing, highest valued startups in the world? Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The quantity and quality of a company’s data can mean the difference between a major success or major failure. Companies like Google have used big data from its earliest days to steer their product suite in the direction consumers need. Other companies, like Apple, didn’t always use big data analytics to drive product design, but they do now. The company Axiom has created a large suite of advanced browser robots that perform difficult tasks like consolidating data across many web applications, extracting data from public sites or from behind logins, data entry, user interface automation, file management and spreadsheet automation. These powerful tools enable people and businesses to collect valuable data to inform their decisions. In this episode we talk to Yaseer Sheriff, Co-Founder and CEO at Axiom. We discuss the value of big data, the opportunities their products enable, and how people can use their tools to improve their data collection practices. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In this episode we discuss software investing, business, and the future with David Rosenthal, co-host of the Acquired podcast. This interview was also recorded as our very first video podcast. Check out the video on the Software Daily YouTube channel. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Complete information games are games where every player has information about the game sequence, strategies, and payoffs throughout gameplay. Playing chess, for example, relies on knowing the location of every piece everywhere on the board. In an incomplete information game like Minecraft, you continually gain new information during gameplay. Until very recently, incomplete information was nearly impossible on blockchains because every transaction and party is public. However, applied zero-knowledge cryptography on Ethereum has advanced greatly the past 18 months and is changing the transparency of blockchains. In this episode we talk to Brian Gu, a creator of the new game Dark Forest. Dark Forest is a space-conquest game built on Ethereum where players discover and conquer planets in an infinite, procedurally-generated, cryptographically-specified universe. It uses zero-knowledge proofs to validate game moves on Ethereum without sharing information to other players about the moves. We discuss the technology behind Dark Forest and the possibilities of a potentially less transparent blockchain future. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Proof of Work cryptocurrency mining, as used on the Ethereum and Bitcoin blockchains, requires huge amounts of energy to validate transactions and generate new tokens. The alternative, Proof of Stake, needs large deposits of assets to be staked up front in order to work. While both consensus protocols have their own drawbacks, they are the current industry standards. The company Spacemesh developed a new consensus protocol with the goal of powering an energy-efficient, decentralized, secure, and scalable smart contracts global computer and a cryptocurrency in the permissionless settings. They call their protocol Proof of Space Time which works on a blockmesh structure rather than a blockchain. The Spacemesh protocol allows newcomers to contribute to the security of the cryptocurrency network via unused storage space on their hard drives, driving down energy consumption and enabling anyone with a computer to contribute. In this episode we talk to Anton Learner, Core Team Lead at Spacemesh. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The company StreamSets is enabling DataOps practices in today’s enterprises. StreamSets is a data engineering platform designed to help engineers design, deploy, and operate smart data pipelines. StreamSets Data Collector is a codeless solution for designing pipelines, triggering CDC operations, and monitoring data in flight. StreamSets Transformer uses Apache Spark to generate insights about your data across multiple different platforms. Their Control Hub is the single hub for managing all of your data pipelines, data processing jobs, and execution engines. In this episode we talk to Arvind Prabhakar, CTO at StreamSets. Arvind is also an Official Member of the Forbes Technology Council, and a Member, PMC Chair/Member, Committer, Mentor, and Contributor to multiple projects with the Apache Software Foundation. He was previously a Director of Engineering at Cloudera, and a Software Architect at Informatica before that. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Delivering Saas products involves a lot more than just building the product. Saas management involves customer relationship management, licensing, renewals, maintaining software visibility, and the general management of the technology portfolio. The company Blissfully helps businesses manage their SaaS products from within a complete IT platform with organization, automation, and security built in. The Blissfully platform offers a system of record for creating and maintaining a single source of truth for technology, a workflows and automations feature for defining and executing consistent IT processes, an IT collaboration feature, and a security and compliance feature. These features come together to form a comprehensive IT management platform. In this episode we talk with Aaron White, a Founder and CTO at Blissfully. Aaron was previously a Co-Founder and Board Member of Price Intelligently, and Vice President at Venrock before that. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Amundsen was started at Lyft and is the leading open-source data catalog with the fastest-growing community and the most integrations. Amundsen enables you to search your entire organization by text search, see automated and curated metadata, share context with co workers, and learn from others by seeing most common queries on a table or frequently used data. Powered by Amundsen, the company Stemma is a fully managed data catalog that bridges the gap between data producers and data consumers. Stemma adds features to Amundsen like showing meaningful data to individual users, adding metadata to data automatically, and documenting data on the fly. Stemma integrates with all the major data sources like Snowflake, Redshift, Google BigQuery, and Apache Airflow. In this episode we talk to Mark Grover, Founder at Stemma. Mark co-created Amundsen and authored the book Hadoop Application Architectures. He was an engineer at Cloudera before joining Lyft as a Product Manager. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Coinbase is a very popular and well trusted cryptocurrency platform for buying and selling digital assets like Bitcoin, Ethereum, and many more. With Coinbase you can manage your portfolio of cryptocurrencies in 1 place like you would for other investments. There are added features like scheduling recurring purchases of assets, time-delayed withdrawals from digital vaults, and mobile apps with sleek UIs for mobile access to the markets. Coinbase has gained trust as a platform for storing digital assets in secure offline storage, using servers covered by insurance policies, by following industry best practices and supporting a variety of the most popular digital currencies. In this episode we talk to Brent Walter, a Senior Engineering Manager at Coinbase. Brent was previously an Advisor at With Labs and various other companies, and was a Director of Software Engineering at Western Digital before that. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you're looking to lower your AWS bill or negotiate a new contract with AWS, you can learn more about The Duckbill Group's services at https://www.duckbillgroup.com/. Corey is also the host and creator of Last Week in AWS, which publishes newsletters and podcasts covering topics to help you stay up to date on all things AWS and insightful conversations with experts in the world of cloud computing all delivered lovingly with Corey's infamous snark. Subscribe at https://www.lastweekinaws.com and follow Corey on Twitter @QuinnyPig. In this episode, Corey Quinn interviews Salman Paracha, Group Vice President, Cloud Engineering at Oracle. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you're looking to lower your AWS bill or negotiate a new contract with AWS, you can learn more about The Duckbill Group's services at https://www.duckbillgroup.com/. Corey is also the host and creator of Last Week in AWS, which publishes newsletters and podcasts covering topics to help you stay up to date on all things AWS and insightful conversations with experts in the world of cloud computing all delivered lovingly with Corey's infamous snark. Subscribe at https://www.lastweekinaws.com and follow Corey on Twitter @QuinnyPig. In this episode, Corey Quinn interviews John Allspaw, Founder and Principal of Adaptive Capacity Labs. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you're looking to lower your AWS bill or negotiate a new contract with AWS, you can learn more about The Duckbill Group's services at https://www.duckbillgroup.com/. Corey is also the host and creator of Last Week in AWS, which publishes newsletters and podcasts covering topics to help you stay up to date on all things AWS and insightful conversations with experts in the world of cloud computing all delivered lovingly with Corey's infamous snark. Subscribe at https://www.lastweekinaws.com and follow Corey on Twitter @QuinnyPig. In this episode, Corey Quinn interviews Liz Fong-Jones, who is Principal Developer Advocate for SRE and Observability at Honeycomb and who worked at Google prior to that. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you're looking to lower your AWS bill or negotiate a new contract with AWS, you can learn more about The Duckbill Group's services at https://www.duckbillgroup.com/. Corey is also the host and creator of Last Week in AWS, which publishes newsletters and podcasts covering topics to help you stay up to date on all things AWS and insightful conversations with experts in the world of cloud computing all delivered lovingly with Corey's infamous snark. Subscribe at https://www.lastweekinaws.com and follow Corey on Twitter @QuinnyPig. In this episode, Corey Quinn interviews Troy Hunt, who is a Microsoft Regional Director and MVP. Troy also runs "Have I Been Pwned" which checks to see if you have an email or password that has been compromised in a data breach. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you're looking to lower your AWS bill or negotiate a new contract with AWS, you can learn more about The Duckbill Group's services at https://www.duckbillgroup.com/. Corey is also the host and creator of Last Week in AWS, which publishes newsletters and podcasts covering topics to help you stay up to date on all things AWS and insightful conversations with experts in the world of cloud computing all delivered lovingly with Corey's infamous snark. Subscribe at https://www.lastweekinaws.com and follow Corey on Twitter @QuinnyPig. In this episode, Corey Quinn interviews Pete Cheslock, Chief Product Officer at Allma, about AWS. Prior to that Pete was Corey's colleague at The Duckbill Group. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Mark Saroufim is the author of an article entitled “Machine Learning: The Great Stagnation”. Mark is a PyTorch Partner Engineer with Facebook AI. He has spent his entire career developing machine learning and artificial intelligence products. Before joining Facebook to do PyTorch engineering with external partners, Mark was a Machine Learning Engineer at Graphcore. Before that he founded Yuri.ai. Mark has also published “The Robot Overlord Manual” which “will teach you all the software, math and ML you’ll need to start building robots at home.” In this episode we discuss machine learning subjects and his experience developing cutting edge software. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Next week Corey Quinn will be guest hosting on Software Engineering Daily, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you're looking to lower your AWS bill or negotiate a new contract with AWS, you can learn more about The Duckbill Group's services at https://www.duckbillgroup.com/. Corey is also the host and creator of Last Week in AWS, which publishes newsletters and podcasts covering topics to help you stay up to date on all things AWS and insightful conversations with experts in the world of cloud computing all delivered lovingly with Corey's infamous snark. Subscribe at https://www.lastweekinaws.com and follow Corey on Twitter @QuinnyPig. Amazon Web Services changed how software engineers work. Before AWS, it was common for startups to purchase their own physical servers. AWS made server resources as accessible as an API request, and has gone on to create higher-level abstractions for building applications. For the first few years of AWS, the abstractions were familiar. S3 provided distributed, reliable object storage. Elastic MapReduce provided a managed Hadoop system. Kinesis provided a scalable queue. Amazon provided developers with managed alternatives to complicated open source software. More recently, AWS has started to release products that are unlike anything else. A perfect example is AWS Lambda, the first function-as-a-service platform. Other newer AWS products include Ground Station, a service for processing satellite data and AWS DeepRacer, a miniature race car for developers to build and test machine learning algorithms on. As AWS has grown into new categories, the blog announcements of new services and features have started coming so frequently that it is hard to keep track of it all. Corey Quinn is the author of “Last Week in AWS”, a popular newsletter about what is changing across Amazon Web Services. Corey joins the show to give his perspective on the growing, shifting behemoth that is Amazon Web Services--as well as the other major cloud providers that have risen to prominence. He’s also the host of the Screaming in the Cloud podcast, which you should check out if you like this episode. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Platforms like Ethereum have billions of dollars of market cap and large developer communities. However, it is still a challenge to build widely adopted DApps on it because of current limitations. Blockchain Proof of Work transactions are typically slow, and Proof of Stake transactions trade off decentralization to achieve high throughput. Transaction fees get expensive, especially for high network load times, scalability is low and this creates a bad user experience. The company Polygon (previously Matic Network) solves some of these problems with their platform for Ethereum scaling and infrastructure development. Polygon combines the features of stand-alone blockchains (like sovereignty, scalability, and flexibility) and Ethereum (security, interoperability and developer experience). These features enable scalable solutions on Ethereum and support a multi-chain Ethereum ecosystem (polygon.technology). In this episode we talk to Denis Ermolin, a Senior Software Engineer at Polygon. Denis was previously a Senior Software Engineer at Animoca Brands, and CEO of Moonrealm Entertainment before that. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

AWS Outposts is a fully managed service that offers the same AWS infrastructure, AWS services, APIs, and tools to virtually any datacenter, co-location space, or on-premises facility for a truly consistent hybrid experience. AWS Outposts is ideal for workloads that require low latency access to on-premises systems, local data processing, data residency, and migration of applications with local system interdependencies (aws.amazon.com). In this episode we talk with Joshua Burgin, General Manager, AWS Outposts at Amazon Web Services. Joshua owns strategy, roadmaps, customer experience, pricing and demand generation for the AWS Edge/Hybrid Compute business, including full P&L responsibility. Joshua was previously a Senior Director, Technology Platform and Services at Zynga and a Senior Manager, Product Management at RPI before that. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Flutter is a UI toolkit developed by Google that helps developers build natively compiled applications for mobile, web, desktop, and embedded devices from a single code base. Development is fast because the screen “hot reloads” as you develop, the architecture is layered for fast and expressive designs, and its widgets incorporate all critical platform differences such as scrolling, navigation, icons and fonts. In this episode we talk about developing Flutter apps with Allen Wyma. Allen is a Founder of Plangora, a web and mobile development company that specializes in PHP, iOS with Swift, android with Java, Wordpress, and Flutter. Allen is also an Elixir Mix Panelist at Devchat.tv where he interviews members of the Elixir community. We discuss Native app development and developing with Flutter. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data exploration uses visual exploration to understand what is in a dataset and the characteristics of the data. Data scientists explore data to understand things like customer behavior and resource utilization. Some common programming languages used for data exploration are Python, R, and Matlab. Doris Jung-Lin Lee is currently a Graduate Research Assistant at the University of California, Berkeley, also earning a PhD in Information Management and Systems. Doris also did her undergrad at Berkeley, studying physics and astrophysics. She is currently developing Lux, a Python library for accelerating and simplifying the process of data exploration. Her research and work with Lux is aimed to make data science more intuitive and accessible to end users. In this episode Doris joins us to discuss data exploration and her research and development of Lux. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Arun Kumar is an Assistant Professor in the Department of Computer Science and Engineering and the Halicioglu Data Science Institute at the University of California, San Diego. His primary research interests are in data management and systems for machine learning/artificial intelligence-based data analytics. Systems and ideas based on his research have been released as part of the Apache MADlib open-source library, shipped as part of products from Cloudera, IBM, Oracle, and Pivotal, and used internally by Facebook, Google, LogicBlox, Microsoft, and other companies. Arun did his undergrad in Computer Science and Engineering at the Indian Institute of Technology, Madras, and then his MS and PhD in Computer Science at the University of Wisconsin-Madison, where his thesis research explores problems at the intersection of data management and machine learning, with a focus on problems related to usability, developability, performance, and scalability. In this episode he joins us to discuss data management systems and artificial intelligence. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud data warehouses are databases hosted in cloud environments. They provide typical benefits of the cloud like flexible data access, scalability, and performance. The company Firebolt provides a cloud data warehouse built for modern data environments. It decouples storage and compute to operate on top of existing data lakes like S3. It computes orders of magnitude faster performance from gigabyte to petabyte scale by using a columnar data structure, vectorized processing, just-in-time query compilation, and continuously aggregated indexing. Firebolt scales with data lakes by processing queries across clusters of nodes in parallel, providing consistently fast processing and granular control over resources. In this episode we talk with Eldad Farkash, Co-Founder and CEO of Firebolt. Eldad was previously a Venture Partner at Angular Ventures and a Founder, CTO and Board Member at Sisense before that. We discuss big data, data warehouses, and the unique benefits offered by Firebolt. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Running applications in containerized environments involves regularly organizing, adding and replacing containers. This complex job may involve managing clusters of containers in different geographic locations with different configuration requirements. Platforms like Kubernetes are great for managing this complexity, but include steep learning curves to efficiently get anything off the ground. The company Portainer provides a universal container management tool that works with Kubernetes, Docker, Docker Swarm, and Azure ACI. It enables managing containers without knowing platform-specific code and best practices. Instead, deploying containerized applications is done through a simple Graphical User Interface. Once deployed, you can observe and monitor the apps and govern security settings all through Portainer. In this episode, we talk to Neil Cresswell, a Co-Founder at Portainer. Neil is also CEO and Founder of CloudInovasi, and was CEO at Emerging Technology Partners before that. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Superset is an open-source, fast, lightweight and modern data exploration and visualization platform. It can connect to any SQL based data source through SQLAlchemy at petabyte scale. Its architecture is highly scalable and it ships with a wide array of visualizations. The company Preset provides a powerful, easy to use data exploration and visualization platform powered by Apache Superset. Preset enables team members with some to no programming experience to build interactive visualizations and dashboards with a no-code viz builder and SQL editor. It works directly on top of popular cloud data warehouses and leading data engines. Preset delivers all the data visualization power of Apache Superset through their complete, easy to consume, enterprise ready platform. In this episode we talk with Srini Kadamati, Senior Data Scientist / Developer advocate at Preset. Previously Srini worked as Head of Product at Dataquest.io and as a Data Scientist at Radius Intelligence before that. He is also a Committer to Apache Superset. We discuss data visualization, the power of big Data, and Preset. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Application Programming Interfaces (APIs) are interfaces that enable multiple software applications to send and retrieve data from one another. They are commonly used for retrieving, saving, editing, or deleting data from databases, transmitting data between apps, and embedding third-party services into apps. The company BaseTen helps companies build and deploy machine learning APIs and applications. Using pre-existing ML models, or choosing from BaseTen’s library of pretrained models, BaseTen helps you instantly deploy API endpoints powered by those models to use in your applications. These APIs easily scale and integrate with existing data sources. BaseTen’s serverless infrastructure enables chaining model outputs and pre- and post- processing code. They also use a drag-and-drop UI builder to create custom UI’s for the applications, all without learning React. In this episode, we talk with Tuhin Srivastava and Amir Haghighat, founders at BaseTen. Tuhin previously founded Shape, and also worked as a Data Scientist at Gumroad. We discuss machine learning API development, scaling ML-driven applications, and the capabilities of BaseTen’s technology. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The company Skynet Labs provides an open protocol for hosting data and web applications on the decentralized web. Skynet allows for decentralized, censorship-resistant, highly redundant storage and applications that are available around the globe. Developers don’t pay for their application’s storage, can launch apps with access to a user’s data right away, are free from corporations pulling access to their resources, and can maintain failover sites for when their primary site goes down. For users, you take your data with you without any corporate oversight, support developers and content creators by simply accessing their work, experience a web free of targeted ads, and never have to put your privacy or security at risk. Skynet is built on top of the Sia blockchain network, which is open-source and guided by the Sia Foundation. In this episode we talk with Matthew Sevey, Engineering Engineer at Skynet Labs. Before joining Skynet, Matthew was a Project Manager at Procter and Gamble and a web development fellow at Startup Institute before that. We discuss decentralized internet, the Sia blockchain network, and Skynet’s mission to build a better and more equitable ecosystem. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Show Notes:Skynet Website Skynet Jobs Skynet Discord Skynet Hackathon SkyChat SkyFeed Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Columnar databases store and retrieve columns of data rather than rows of data. Each block of data in a columnar database stores up to 3 times as many records as row-based storage. This means you can read data with a third of the power needed in row-based data, among other advantages. The company Altinity is the leading enterprise provider for ClickHouse - an open-source column-store analytic database, now a fully managed service developed and operated with Altinity.Cloud. Altinity only bills for the compute, storage, and support that is used. They provide enterprise support for analytic applications like tuning queries, Kafka support, and ClickHouse bugs, and their ClickHouse clusters run with out-of-the-box security and privacy. In this episode we talk with Robert Hodges, CEO at Altinity. Before becoming CEO at Altinity, Robert worked as a Senior Staff Engineer at VMWare and was the CEO of Continuent before that. We discuss databases and data warehousing, ClickHouse, and how Altinity helps customers create enterprise analytic applications. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Spark is a popular open source analytics engine for large-scale data processing. Applications can be written in Java, Scala, Python, R, and SQL. These applications have flexible options to run on like Kubernetes or in the cloud. The company Data Mechanics is a cloud-native Spark platform for data engineers. It runs continuously optimized Apache Spark workloads on a managed Kubernetes cluster within the user’s cloud account. They boast a 50%-75% cost reduction from cloud providers by dynamically scaling applications based on load and automatically tuning app configurations based on the historical Spark pipeline runs. Their Kubernetes clusters are deployed within user accounts so user data never leaves the environment and they handle the cluster management. In this episode we talk to Jean-Yves Stephan, Co-Founder and CEO at Data Mechanics. Jean-Yves previously worked as a Software Engineer then a Tech Lead Manager at Databricks. We discuss big data engineering in Spark and the unique advantages of using Data Mechanics to make Spark development easier and more cost effective. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Hudi is an open-source data management framework used to simplify incremental data processing and data pipeline development. This framework more efficiently manages business requirements like data lifecycle and improves data quality. Some common use cases for Hudi is record-level insert, update, and delete, simplified file management and near real-time data access, and simplified CDC data pipeline development (AWS.amazon.com). In this episode we speak to Vinoth Chandar, VP of Apache Hudi. Vinoth is the creator of the Hudi project at Uber. He continues to lead its evolution at the Apache Software Foundation. Previously he was a Principal Engineer at Confluent, and a Sr Staff Engineer/Manager at Uber before that. We discuss building large scale distributed and data systems. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

An application programming interface, API for short, is the connector between 2 applications. For example, a user interface that needs user data will call an endpoint, like a special URL, with request parameters and receive the data back if the request is valid. Modern applications rely on APIs to send data back and forth to each other and save, edit, delete, or retrieve data in databases. The number of APIs used in a single application is growing due to the increase of micro-services and distributed architectures. Understanding how your applications use APIs can increase their efficiency and stability and make debugging easier. The company Akita observes the structure of programs to visualize, map, and manage API behavior. By monitoring the APIs in your applications, Akita can catch code changes that may break production applications. While this work is normally labor-intensive, Akita automates it by analyzing the source code and logs. They check the observed behaviors against intended specs and contracts to provide clear oversight on all activity. This information can then be generated into maps that help you document and version your APIs across your entire service ecosystem. In this episode we talk with Jean Yang, Founder and CEO of Akita Software. Jean was previously an assistant professor at Carnegie Mellon University and a postdoctoral researcher and Harvard Medical School before that. We discuss modern APIs, their role in applications, and how Akita Software makes understanding and building APIs easier for developers. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The traveling salesman problem is a classic challenge of finding the shortest and most efficient route for a person to take given a list of destinations. This is one of many real-world optimization problems that companies encounter. How should they schedule product distribution, or promote product bundles, or define sales territories? The answers to these questions constantly change because business environments constantly change. The company Nextmv helps solve these problems with production-ready, commercial tools for solving optimization problems and simulating models with real company data. Their tool Hop encodes optimization strategies for dynamic environments. Hope can be deployed to routing, scheduling and assignment problems in multiple industries like on-demand delivery, e-commerce, and IT infrastructure management. Their tool Dash is a commercial-grade simulation engine that provides an environment to “A/B test” models online with real data. In this episode we talk to Carolyn Mooney, CEO at Nextmv. Carolyn was previously a Lead Systems Engineer at Grubhub, and a Decision System Analyst at Zoomer before that. We discuss optimization problems throughout different industries, machine learning strategies for solving them, and go into detail about how Nextmv helps companies become more profitable and efficient. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Microservice architecture has become very common over the past few years because of the availability of containers and container orchestrators like Kubernetes. While containers are overall positive for scaling apps and making them more available, they’ve also introduced hurdles like persisting data and state, and container restarts or pod failures. Development teams put significant work into designing applications that take these hurdles into account because without precautions you can lose valuable data or crash your app. The company Temporal provides tools for both building complex microservices as well as for apps that use microservices. They use 2 primary function types: workflow and activity. Workflow functions persist all local variables and threads so that if the server the app runs on crashes, it’s picked up on a different server where it left off, down to the line. Activity functions automatically initiate retry logic if the service the function invokes fails for something like its server being down. Temporal provides visibility into end-to-end workflows that can span multiple services. In this episode we talk to Ryland Goldstein, Head of Product at Temporal. Previously, Ryland was the Lead Product Manager at Reshuffle and a Software Engineer Lead at Parallel Machines. We discuss the challenges of managing state in micro services, orchestrating microservices and how Temporal simplifies this process for development teams. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Natural Language Processing (NLP) is a branch of artificial intelligence concerned with giving computers the ability to understand text and spoken words. “Understanding” includes intent, sentiment, and what’s important in the message. NLP powers things like voice-operated software, digital assistants, customer service chat bots, and many other academic, consumer and enterprise tools. The company Botpress provides open-source developer tools to create NLP tools for process and FAQ automation. They use the latest NLP models for domain-specific, contextual and goal-oriented conversations. This technology is free and available through simple API routes. They also maintain integrations with popular messaging services like Facebook Messenger, Slack, and Microsoft Teams. For other proprietary systems, they provide a raw Messaging API. In this episode we talk to Sylvain Perron, CEO of Botpress. Sylvain was previously a Director of Engineering at Protorisk Limited and a Software Developer at ArcBees before that. We discuss the current advances in Natural Language Processing and how NLP powers Botpress. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In software engineering, telemetry is the data that is collected about your applications. Unlike logging, which is used in the development of apps to pinpoint errors and code flows, telemetry data includes all operational data including logs, metrics, events, traces, usage, and other analytical data. Companies usually visualize this information to troubleshoot problems and understand patterns and opportunities in how their applications are used. The company New Relic is a modern observability platform built to optimize your entire software stack from one place. New Relic includes a Telemetry Data Platform that acts as a single source of truth for telemetry data. Built on top of that are tools for full stack observability to visualize and troubleshoot your data in milliseconds. When a problem does occur, New Relic’s applied intelligence will detect and understand the problem to help resolve it faster. In this episode we talk with Zain Asgar and Ishan Mukherjee. Zain is currently the GVP/GM - Pixie at New Relic, through the acquisition of Pixie Labs Inc where he was the co-founder and CEO. Ishan leads GTM for all New Relic products and is a co-founder and CPO of Pixie. We discuss big data, application observability, and how New Relic provides data observability and intelligence in one platform. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud computing provides tools, storage, servers, and software products through the internet. Securing these resources is a constant process for companies deploying new code to their cloud environments. It’s easy to overlook security flaws because company applications are very complex and many people work together to develop them. Wyze Labs, for example, had millions of users’ data stolen due to a mistake by a single employee. The company Bridgecrew is a cloud security platform helping to prevent mistakes like that from happening. Bridgecrew integrates into developer workloads to automatically find infrastructure errors in cloud accounts, workloads, and infrastructure as code. Their platform also monitors code reviews and build pipelines to prevent errors from being deployed into production. If an error is found then Bridgecrew’s software reverts that code back to its last known correct state. In today's episode we talk with Barak Schoster, CTO and co-founder at Bridgecrew. Barak previously worked as a senior software architect at RSA Security and as a software architect at Fortscale before that. We discuss cloud security, Infrastructure as Code, and big data architecture. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern applications are increasingly built as large, distributed systems. A distributed system is a program where its components are located on different machines that communicate with one another to create a single cohesive app. Components may exist as multiple instances across “nodes,” the computers hosting them, which form clusters of nodes that span across geographic regions. The platform Uber is built on distributed systems. The benefit of this architecture is higher availability, higher load capacity, and lower latency. This is essential for Uber because it needs to process up to thousands of requests per second. But this architecture also presents challenges. Some data, like payment requests, needs to be 100% accurate, which can be challenging for a system where its data is spread out across multiple nodes. The data needs to be durable as well-- if a node fails, the data it holds needs to be readily available on another. Their nodes also need to be up-to-date with the information from other nodes so they process the right requests. Any company using distributed systems faces these challenges. In this episode we talk with Gergely Orosz. Gergely is a software engineer and author of The Software Engineer’s Guidebook and his blog The Pragmatic Engineer. He previously worked as an engineering manager at Uber, working on their payments experience platform. We discuss distributed systems, payment technology, and the challenges he overcame working at Uber. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Geospatial technology impacts every person who uses a smartphone, drives a car, or flies in airplanes. It refers to all of the technology used to acquire and interpret geographic information. In more advanced settings, geospatial technology is used for constructing dynamic maps, 3D visualizations, and scientific and governmental simulations. The company Makepath specializes in geospatial technology and full-stack application development. Makepath helps companies to create beautiful and simple visualizations from mountains of complex data. Using open-source Python libraries and real-world validation, they create state-of-the-art analytics, web applications, and other robust automation processes. They are also passionate about the open source ecosystem and contribute to many ongoing projects. In this episode we talk with Brendan Collins from Makepath. Brendan is a founder and the principal of Makepath. He previously worked as the principal of Parietal and was a software developer at Anaconda, Inc before that. We discuss geospatial data science, the work they do at Makepath, and their broader open source projects. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Large portions of software development budgets are dedicated for testing code. A new component may take weeks to thoroughly test, and even then mistakes happen. If you consider software defects as security issues then the concern goes well beyond an application temporarily crashing. Although even minor bugs can cost companies a lot of time to locate the bug, resolve it, retest it in lower environments, then deploy it back to production. The company Speedscale provides an intelligent, Kubernetes-friendly testing toolkit that runs at build time. Their virtual SRE-bot works inside automated release pipelines to forecast and test real-world conditions the new code will encounter. This process requires no manual scripting because Speedscale uses existing traffic to generate tests and mocks. The feedback is immediate after every build and covers regression, performance, fuzzing, and chaos tests automatically. In this episode we talk with Ken Ahrens and Matt LeRay. Ken is a founder and the CEO at Speedscale. Previously Ken worked at New Relic as a senior director, solutions architects. Matt is a founder and CTO at Speedscale. He previously was the VP of Product at Observe, inc. We discuss testing in distributed environments, how Speedscale intelligently tests and mocks during builds, Kubernetes, and their future goals with Speedscale. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Encryption algorithms provide the means to secure and transfer sensitive information by taking input and transforming it into an unreadable output. Usually a special key, or multiple keys, are needed to unscramble the information back to the original input. These algorithms power the security of everything from our cell phone lock screens to Fortune 500 company servers. The company Skiff is protecting data privacy with their first product, the only end-to-end encrypted document collaboration platform with password protected folders, expiring links, and secure workplaces. Skiff’s document platform has all the traditional features of a typical document editor, making it feel familiar and comfortable. Their end-to-end encryption and built-in password protection are 2 of several methods that make collaborating on documents more safe and within user control than on any other platform. In this episode we talk with Andrew Milich, CEO of Skiff. Andrew was previously an associate product manager at Schmidt Futures. We discuss data privacy and security, the Skiff document collaboration platform, and potential future security products. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A liquid market enables individuals or groups to quickly buy and sell assets. Decentralized platforms can struggle to execute trades when their platform does not have much liquidity for a specific token. Newer tokens or tokens with limited supply are most often the least liquid because there might be an imbalance of buyers and sellers. You can’t sell token A for price X without a consenting buyer on the other end. The company Uniswap is a decentralized protocol for creating liquidity and trading ERC-20 tokens on Ethereum. Uniswap encourages users to be “liquidity providers” whereby they pool their assets into funds that enable people to complete trades without an opposite party. Instead, they swap against the liquidity pool created by the liquidity providers. Every swap incurs a small fee, which is distributed proportionately to liquidity providers when they decide to pull their funds. Uniswap prices coins based on the simple formula x*y=k. In this episode we talk with Noah Zinsmeister, engineering lead at Uniswap. Noah also maintains web3-react, a framework for building blockchain applications. We discuss cryptocurrency liquidity, the Ethereum blockchain, and how Uniswap is building a community of liquidity providers and traders. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Non-fungible tokens are proofs of authenticity that are stored on a blockchain. Unlike fungible tokens, such as cryptocurrencies which are interchangeable, non-fungible tokens aren’t inherently equivalent to any other token. Because they are unique, they can be used to represent any unique asset. Their presence on a blockchain enables an NFT owner to trade the asset and prove they are its owner. One such use case for NFTs is artwork, and the company Showtime is bringing artists and creators together to showcase their work. Crypto art is verified with NFTs and therefore can’t be authentically reproduced. This lets artists sell their original work to prospective buyers. With Showtime, you can discover and follow new crypto artists, browse the trending page and like, comment, and share artwork. The platform shows a variety of art styles and experience levels. In this episode we talk with Alex Masmejean, co-founder and CEO at Showtime. Before Showtime Alex worked as an advisor at TapIn and operations at MetaCartel DAO. We discuss non-fungible tokens, the world of crypto art and the growth and future of Showtime. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Volatility is the degree of fluctuation of something’s price. Highly volatile assets may see rapid and large price changes, while less volatile assets will maintain a steady price. This concept is important in decentralized finance because cryptocurrencies tend to be volatile assets. The company Synthetix provides assets called Synths that provide exposure to an asset without holding the underlying resource. For example, you can hold and trade Synths that track the price of USDs, synthetic gold and silver (measured by the ounce), and other currencies and commodities. Users use Synthetix to trade the price equivalents of real-world assets on Ethereum. This lets them diversify their investment portfolios with less volatile assets while staying on the blockchain and executing trades against smart contracts. In today’s episode we talk with Justin Moses, CTO at Synthetix. Previously Justin worked as CTO at Haven and as a Tech Advisor at blueshyft. We discuss derivatives trading in DeFi, the liquidity and volatility of synthetic assets, and the rewards and features available from using Synthetix. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A decentralized exchange, usually referred to as a DEX, is a platform for exchanging cryptocurrencies. Depending on trading volume for different coins, some DEXs are more liquid than others. On the one hand you can freely swap unlisted tokens and maintain full control over your private keys and wallet information. On the other hand, without the right supply and demand it’ll be difficult to swap a particular coin at the right price. 1inch Exchange is a decentralized exchange aggregator that can split a single trade transaction across multiple DEXs. Their Pathfinder API ensures users get the best price by using a discovery and routing algorithm to find the best possible paths for token swaps and then splits the swap across multiple exchanges and market depths of the same exchange. 1inch uses multiple liquidity sources, including private liquidity providers, to ensure there is sufficient liquidity for all swaps on their platform. If the rate of a trade becomes more expensive than the user has confirmed from the UI, the algorithm can cancel part of the route and simply return the unswapped tokens to the user’s wallet. In today’s episode we talk with Anton Bukov, co-founder of 1inch. Anton was previously a senior smart contract engineer at NEAR Protocol and chief blockchain engineer, consensus researcher at MultiToken. We discuss the pros and cons of decentralized exchanges, splitting cryptocurrency swaps across multiple exchanges with 1inch Exchange, and the growth of decentralized finance. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A data warehouse is a data management system that often contains large amounts of historical data and is used for business intelligence activities like analytics. It centralizes customer data from multiple sources to be an organization’s single source of truth. Getting the data from your data warehouse into the different applications used by your organization can be difficult. The company Census simplifies syncing your data warehouse with other applications. Census works on top of existing infrastructure and lets users pick destinations apps, map the data, and then handles maintaining live, in sync metrics. Because Census runs inside your data warehouse, data remains secure and is never stored on their servers. In today’s episode we speak with Boris Jabes, CEO of Census. Boris was previously a managing partner at Polynome and a Senior Director at LogMeln. We discuss the complexity of data warehouses, how the field of data analytics has grown over the years, and how Census makes data accessibility easy and secure. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

IT infrastructure are the components required to operate IT environments, like networks, virtual machines or containers, an operating system, hardware, data storage, etc…. As companies build out different deployment environments with infrastructure configurations, they must maintain the different environments, replicate them, and update them. The management of infrastructure, often automated to some extent, is referred to as Infrastructure as Code (IaC). The company Gold Fig Labs helps growing companies better understand their deployed infrastructure beyond the basic IaC principles. Gold Fig Labs developed 2 main tools to deliver the clearest view of infrastructure security and compliance. Their tool Checkup provides periodic security and best practices reports for AWS accounts. The report details specific, actionable, and relevant advice to improve security posture. Their other tool, Introspector, is a unique security and auditing tool that provides in-depth analysis of larger cloud deployments with complex regulatory requirements and custom internal policies. In this episode we talk with Vikrum Nijjar and Greg Soltis. Vikrum is co-founder and CEO at Gold Fig Labs. He was previously an angel investor with Angel and special advisor for Google’s onboarding acquisitions, compliance, and security. Greg is co-founder and CTO at Gold Fig Labs. Previously he was a senior software engineer at Google and a software engineer at Firebase. We discuss the principles of IaC, how Gold Fig Labs helps customers go beyond what IaC intends, and the complexity of cloud infrastructure security and regulatory compliance. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Decentralized applications, termed “dApps,” are applications that feel like normal apps but are actually deployed (mostly) on the Ethereum blockchain. This means dApps can’t be taken down, can’t be censored or blocked, typically use Ethereum accounts as identity, and would only experience downtime if Ethereum itself went down. There are a lot of things you can do with blockchain applications, particularly with decentralized finance. The company Compound develops protocols, built on the Ethereum blockchain, that establishes money markets. Money markets are pools of assets with algorithmically derived interest rates based on supply and demand. The Compound protocol represents assets as fungible ERC-20 token balances called cTokens. cTokens automatically increase in value from the amount of the initial underlying asset. The interest generated and managed through the Compound protocol can be used primarily for long-term investing in Ether and tokens as well as dApps and other entities. Compound provides lots of documents and discords for infusing interest and liquidity into dApps and related projects. This enables dApps to manage assets that generate interest and could lead to entirely new blockchain-based business models. In this episode we talk with Jared Flatow, Director of Protocol at Compound. Previously, Jared worked as a software engineer at Caffeine and founded the company Quasi Convex Union. We discuss the importance of liquidity and interest-earning assets in DeFi, how Compound is helping enhance dApps and the role and growth of dApps overall, and his goals for Compound going forward. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A ‘token’ can represent almost anything in Ethereum, according to Ethereum.org: Lottery tickets, points in an online platform, fiat currency, and much more. These tokens must follow a standard called ECR-20 to have the same type and value of any other token, and behave just like the ETH. The platform Opyn lets users buy and trade decentralized finance (DeFi) options on ETH and ECR20s. “Options” represent underlying assets with predefined (strike) prices and expiry dates that can be bought and sold. Opyn provides options protocols through smart contracts that are powerful and capital efficient. Using options helps secure against volatility and flash crashes, typically have lower margin requirements, can yield interest bearing collateral, and are noncustodial with Opyn. In this episode we speak with Aparna Krishnan, a co-founder of Opyn. We discuss options trading and how it differs from trading pure cryptocurrencies, developing and using smart contracts to define DeFi protocols, and the unique benefits of using Opyn for options trading. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In decentralized finance (DeFi) a liquidity pool is a collection of cryptocurrency funds created from the deposits of many users and usually multiple different currencies. There are 2 main types of pools: custodial and non-custodial. Custodial pools are controlled by a third party manager which contains information like the private keys and the funds. They are most commonly web-based. Non-custodial pools mean that each contributor has complete control of their private keys (their funds) and are often browser based or keys can be stored and funds traded using hardware devices. AAVE is a trusted, open-source, and non-custodial liquidity protocol on Ethereum. Participants use AAVE to either deposit or borrow funds. Depositors earn interest on their funds in the same currency they deposited: USD-USD, DAI-DAI, etc. When demand is high for the deposited currency, the earned interest increases automatically. Borrowers deposit collateral and then borrow from any currency in the protocol. Their entire platform, from their APIs to smart contracts, are open-source and reviewable by anyone. This has been the basis for many platform audits, providing high reliability to users and enabling integration into other services and products. In this episode we talk with Ernesto Boado, a full stack and blockchain developer currently co-leading the development of the AAVE protocol. We discuss decentralized finance, token economics, and how AAVE aTokens make their service unique. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A smart contract contains the “terms” of a blockchain transaction between a buyer and a seller as well as the capabilities to execute those terms. In order for smart contracts to include outside data from the world, such as stock market data, weather, sports data, etc…, the contract needs a third party service called an oracle. The industry standard blockchain oracle is Chainlink. Chainlink is a decentralized and open-source oracle network that connects to any blockchain with seamless API connections. Their nodes connect to trusted data with cryptographic proofs that make their network tamper-proof. Chainlink is used with smart contracts that secure billions of dollars of value for blockchain projects. Their oracle network greatly expands the value of smart contracts and is used to create cutting-edge, modern blockchain applications. Sergey Nazarov is the co-founder of Chainlink and joins us to talk about their platform. Sergey has also co-founded SmartContract, a smart contract middleware service, Secure Asset Exchange, which provides smart contracts with real-time revenue sharing and multi-signature escrow, and CryptaMail, a secure and 100% decentralized email service. He is an expert in smart contracts and discusses Chainlink, the role of oracles in the blockchain ecosystem, and the future of smart contracts and blockchains. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Using artificial intelligence and machine learning in a product or database is traditionally difficult because it involves a lot of manual setup, specialized training, and a clear understanding of the various ML models and algorithms. You need to develop the right ML model for your data, train the model, evaluate it, optimize it, analyze it for outliers and anomalies, assemble confidence ranges of the predictions and feature importance, and eventually deploy it to make predictions. An emerging field in AI, called Automated Machine Learning (AutoML), lowers these barriers to entry by using AI to automate much of this process. One of the market leaders in AutoML is MindsDB. Their service lets business users and developers make predictions on top of data at its source. Rather than make expensive copies of databases, a process that creates complex infrastructures, MindsDB trains and deploys models right inside the database. The results of their ML models can be queried with standard SQL statements and integrated into other applications as easily as querying any other database. In this episode we learn more about the progress that has been made in AutoML to simplify incorporating machine learning throughout organizations. We discuss the current features available from MindsDB, the difference their product has made for companies trying to leverage AI, and the future of AutoML and artificial intelligence generally. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The typical procedure many companies follow to reach production-level code is design the program, code and test it in different environments, and put it in a pipeline to deploy to production. Developers can make it pretty far into building their core features before inevitably breaking to include enterprise features and security standards like Single Sign On, Okta security and authorization for different user groups, Workday integrations, etc…. Including these requirements can put stress on a team with newer developers or limited resources. For some large projects, these standards need months to fully build out and test. The company WorkOS is dramatically shortening the time it takes to make applications enterprise ready so that developers and IT teams can focus on the application itself. WorkOS provides developer-minded tools like RESTful endpoints, JSON responses, framework-native SDKs, and a developer dashboard, among other tools, to integrate otherwise complicated enterprise standards in just a few lines of code. Their mission is to simplify building applications for enterprise users so that developers can focus on creating core features in a timely manner. In this episode, we discuss the process of building enterprise applications, the challenges of modern security and administrative requirements, and how WorkOS is solving those issues. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A major change in the software industry is the expectation of automation. The infrastructure for deploying code, hosting it, and monitoring it is now being viewed as a fully automatable substrate. Equinix Metal has taken the bare metal servers that you would see in data centers and fitted them with supreme automation and repeatability. This movement to modern metal-as-a-service has brought with it specialty hardware configurations and options for customizability for cloud native applications. Equinix has the largest collection of interconnected data centers in the world. Their existing networking capabilities bring an interconnectivity to its bare metal product that enables global scale. Their server plans are high-performing, secure, agile, and have flexible pricing plans. They are also extremely customizable to fit unique use-cases and were designed for an automated devOps experience. Nicole Hubbard is a principal engineer in the Equinix Metal delivery team. Nicole was involved in building a world class cloud control panel on Kubernetes, running atop Flatcar Linux, to solve the security and scalability of a control plane for the Metal platform. She has been greatly involved in the success of Equinix Metal so far and is here to tell us more about it. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cryptocurrencies like Bitcoin and Dogecoin are electronic currencies with a complete transaction history stored on a blockchain. A cryptocurrency blockchain is a linear record of all the transactions between users for a given currency. This record is public and distributed across thousands of computers, which makes falsifying a transaction nearly impossible because the hacker would need to alter over half of all copies. People use digital “contracts,” which is code that dictates the terms of the transaction and executes those terms, to perform trades. Understanding these technologies and how to use them used to be essential to trading and holding crypto. The platform dYdX is lowering these barriers to entry for people who want to invest in cryptocurrencies. Rather than learning to buy and hold cryptocurrencies manually, dYdX provides a detailed but easy to understand user interface for investing in popular cryptocurrencies. They recently released what they call Layer 2 protocol, which significantly reduces the gas cost of trading and therefore the fees and minimum trade sizes. There is no wait required to withdraw your funds with Layer 2 and the entire process has the highest grade security. Antonio Juliano is the founder of dYdX. Antonio has also founded Weipoint and worked as a software engineer at Uber and Coinbase for three years before launching dYdX in 2017. He joins us today to talk about the crypto community, their recent release of Layer 2 with dYdX, and the future of cryptocurrencies and blockchain. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Digital communities have exponentially grown in importance ever since most of the world went remote. Basically every popular online forum, message board, chat app, and other online social aggregators were created before this new normal. Many of these platforms lack sufficient organization or are just outdated for a fully remote environment. If society continues to work fully or hybrid remote, then digital communities need to be optimally designed for a remote experience. Tom Kleinpeter and Viraj Mody are 2 co-founders of Common Room, a platform made to bring communities together and developed in today’s remote environment. Common Room is in its early stages of development and its details have remained secretive. In today’s episode Tom discusses the challenges with current social platforms, what we can expect from the first version of Common Room, some of its underlying technology and unique features, and how his team plans to promote and grow Common Room with organizations and consumers. Previously, Viraj led engineering organizations at Convoy and Dropbox, helping both companies scale their teams and products. Viraj was also a founding engineer at Audiogalaxy, where he worked alongside Tom, which was acquired by Dropbox. Previously, Tom was a Principal Engineer at Dropbox, and before that the CTO at Audiogalaxy and FolderShare. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Creation Labs is helping bring Europe 1 step closer to fully autonomous long haul trucking. They have developed an AI Driver Assistance System (AIDAS) that retrofits to any commercial vehicle, starting with VW Crafters and MAN TGE trucks. Their system uses camera hardware mounted to the vehicle to capture video data that is processed with computer vision to understand the context on the road. This piece of the system was developed by the world’s leading experts in computer vision. While the computer interprets what is happening on the road, data is sent to a processing system that can control the vehicle’s break, throttle, and steering. The AIDAS system currently augments a driver’s role but does not replace the need for one yet. However, the difference between great drivers and bad drivers is around a 30% difference in fuel efficiency, according to Creation Labs. They have trained their systems with data from the best drivers in order to lower fuel costs for vehicles driven by their AIDAS. They’ve also built their system using the highest standards of safety. Jakub Langr is the CEO of Creation Labs and an Oxford-educated Data Scientist with 10 years of industry experience. He discusses Creation Labs’s vision for the future and the impact their incredible product is having on customers’ profit margins and emissions. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Product teams sometimes double as data teams. They struggle through import errors, scrub long and complicated data sheets for consistency, and map spreadsheet fields on step 3 in a long instruction document. Data structuring and synchronization is a very real problem that product teams regularly overcome. Flatfile uses AI-assisted data onboarding to eliminate repetitive work and make B2B data transactions fast, intuitive, and error-free. By drawing on the experiences of their thousands of users, Flatfile automatically learns how imported data should be structured and cleaned. Their product saves product teams a ton of time setting up data and syncing it across workspaces so people can focus on using their data instead of fixing it. David Boskovic is the co-founder and CEO of Flatfile. He was previously the Platform Architect at Envoy, and CTO and co-founder of the startup Rainmaker. Eric Crane is also a cofounder and the COO of Flatfile, after being the host of Customer Success Leader and Product Manager for Envoy. They join us to talk about data from a product team’s perspective and how leveraging AI with Flatfile is removing barriers between humans and data. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Right now, more than 10 million people use notebooks like Jupyter in their workflow. Notebooks are open-source tools for creating and sharing documents with live code, equations, visualizations and explanatory text. Notebooks like Jupyter have exploded in popularity the past 5 years to become the standard tool for data science teams. They became especially important in 2020 because they facilitate easy access to explore remote data. As data scientists continue to work remotely, the demand for a great notebook experience grows. Matthew Seal is the cofounder and CTO of Noteable, a modern and collaborative notebook platform. Noteable was built with enterprise-grade security and protection, secure deployment options, and the highest levels of availability. Most importantly, it delivers a positive experience for collaborating remotely. In this episode Matthew talks about how Noteable started, why it’s different from traditional notebooks, and what it means to collaborate in today’s mostly remote environment. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Many startups today begin their life as an open-source project. Open source projects allow early adopters of a technology to experiment, to contribute code and feedback, and to shape the evolution of the project in its early stages. When a “community maintainer” company emerges to provide service offerings based on that project, its early customer base often consists of contributors to the open source project who already have experience with the technology, but want a more fully featured offering. Orbit is a “community experience platform” that provides insights on activity for open-source and proprietary products. Orbit is founded on the “Orbit Model,” which it describes as a “framework for building high-gravity communities”- that is, developer communities with passionate and engaged contributors. The Orbit Model is offered as an alternative to traditional sales and marketing funnels that gives a focus on cultivating early adopters for a project. Patrick Woods and Josh Dzielak are the co-founders of Orbit. They join the show today to talk about the importance of an active and passionate community for projects, how Orbit helps Maintainers grow and shape their users’ experiences, and how they see Orbit evolving in the future. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

ELT, or “Extract, Load, and Transform,” is the process that modern data pipelines use to replicate data from a source and load it into a target system such as a cloud data warehouse. ELT is a more flexible evolution of the traditional “Extract, Load, Transform” workflow used in pre-cloud systems. The power of ELT relies on flexible integrations between data sources and their targets, called connectors. The wide variety of data sources available to a cloud application today means that an ELT platform needs to handle a vast, and growing set of use cases for its connectors. Airbyte is an open-source ELT platform built with the “long tail of integrations” in mind. Airbyte is secure, extensible, and simple to set up. Developers can use Airbyte’s platform to build the connectors they need for their specific use case, without worrying about scheduling, orchestrating, or monitoring. Michel Tricot and John Lafleur are co-founders of Airbyte. Previously, Michel was Head of Integrations at Liveramp building data ingestion connectors, while John was the co-founder of StreamNation, Anaxi, and CEO of CodinGame. Michel and John join the show today to talk about why ELT has changed the way organizations manage and store their data, what technical challenges exist in the world of data integration, and how Airbyte can give your infrastructure “superpowers.” Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The complexity of building web applications seems to have grown exponentially in the last several years. This added complexity may bring power, but it can also make applications brittle, costly, and difficult to maintain. Suborbital is an open-source project with a goal of making web application development simple. Its flagship project is Atmo, a platform that integrates three underlying projects also built by Suborbital. Vektor allows developers to write self-contained functions called Runnables to handle business logic, which are then built into WebAssembly. Atmo then automatically scales out a flat network of instances to handle traffic using Grav, a meshed message bus, and Reactr, an embedded job scheduler. Together, these projects make it simple to create powerful, WebAssembly-based server applications without worrying about infrastructure or writing boilerplate code. Connor Hicks maintains the Suborbital open source project. By day, he is a developer and Product Discovery Lead at 1Password. He joins the show today to talk about the exciting potential of WebAssembly, how Atmo is introducing new design patterns, and why we should think differently about complex webservices systems. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Observability is a key feature of a well-architected application. Because building an observability system for a cloud application can be challenging, especially at scale, many organizations elect to use third-party observability platforms rather than build internal tools. But these third-party provider contracts often charge by volume of data collected, which can be unpredictable and difficult to control. Organizations seeking to make their systems observable faced a tradeoff between convenience and control. Opstrace is building an observability platform that seeks to circumvent that tradeoff by matching costs to the users who get value from their service. Opstrace’s platform is open-source, and its clusters can be created easily via command line tools. Opstrace clusters talk directly to your cloud provider, and can store your data safely and inexpensively in S3 or GCS buckets. Sebastien Pahl is the co-founder and CEO of Opstrace. Prior to founding Opstrace, he was a Director of Engineering at Red Hat and Mesosphere. Sebastien joins the show today to talk about the evolution of the market for cloud observability tools, how Opstrace helps bring observability costs under control, and what the future has in store for the Opstrace open-source project. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As the volume and scope of data collected by an organization grow, tasks such as data discovery and data management grow in complexity. Simply put, the more data there is, the harder it is for users such as data analysts to find what they’re looking for. A metadata hub helps manage Big Data by providing metadata search and discovery tools, and a centralized hub which presents a holistic view of the data ecosystem. DataHub is Linkedin’s open-sourced metadata search and discovery tool. It is Linkedin’s second generation of metadata hubs after WhereHows. Pardhu Gunnam and Mars Lan join us today from Metaphor, a company they co-founded to build out the DataHub ecosystem. Pardhu and Mars, and the other co-founders of Metaphor, were part of the team at Linkedin that built the DataHub project. They join the show today to talk about how DataHub democratizes data access for an organization, why the new DataHub architecture was critical to Linkedin’s growth, and what we can expect to see from the DataHub project moving forwards. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The shift to microservices architectures and distributed systems has been a challenge for systems using conventional security practices, such filtering IP addresses using network policies. In addition, the increasing intersection of development and operations exemplified by the DevOps methodology has expanded the scope responsibilities in implementing secure systems. Part of CNCF, SPIFFE is a set of open-source specifications for issuing identity to services in heterogenous, distributed environments such as a cloud-native microservices architecture. Systems implementing SPIFFE bypass the need for application-level authentication and network-level ACL configuration. SPIRE, or the SPIFFE Runtime Environment, is a system that implements the SPIFFE standards to manage platform and workload attestation, providing an API for controlling policies, and coordinating certificate issuance and rotation. Derek Edwards is the head of engineering at Anthem.ai, and Ryan Turner is a software engineer at Uber. They join the show today to talk about the challenges of managing security in a distributed system, how adopting SPIFFE represented a paradigm shift in their authentication workflow, and how the SPIFFE and SPIRE projects are evolving to meet the needs of the next generation of cloud-native applications. HPE sponsored this podcast. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software-Defined Networking describes a category of technologies that separate the networking control plane from the forwarding plane. This enables more automated provisioning and policy-based management of network resources. Implementing software-defined networking is often the task of Site Reliability Engineers, or SREs. Site reliability engineers work at the intersection of development and operations by bringing software development practices to system administration. Equinix manages co-location data centers and provides networking, security, and cloud-related services to their clients. Equinix is leveraging its status as a market leader in on-prem networking capabilities to expand into cloud and IaaS offerings such as Equinix Metal, which has been referred to as “bare-metal-as-a-service,” and offers integrations with 3rd party cloud technologies with a goal of creating a seamless alternative to modern public clouds for organizations seeking the benefits of colocation. Tim Banks is a Principal Solutions Architect at Equinix and he joins the show to talk about what Equinix offers and how it differs from other cloud providers. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google Cloud, AWS, and Azure are the dominant cloud providers on the market today. But the market is still highly competitive, and there is significant overlap in the services offered by all three large providers. Since all three offer a broad range of services, developers looking to choose a platform for their application must focus on providers’ domains of relative excellence and how those align with their needs. One domain where Google Cloud Platform excels is with its database offerings. Google has data management baked into its organizational DNA, and has been the source of several innovative technologies in the data space such as Spanner, BigTable, and BigQuery. Andi Gutmans is a general manager and VP of engineering for databases at Google. He joins the show today to talk about how Google came to excel at databases and data management, how machine learning and Big Data users in particular can benefit from Google Cloud’s offerings, and how new features such as Database Migration Service are helping Google stay ahead of the curve in a competitive cloud landscape. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Vectors are the foundational mathematical building blocks of Machine Learning. Machine Learning models must transform input data into vectors to perform their operations, creating what is known as a vector embedding. Since data is not stored in vector form, an ML application must perform significant work to transform data in different formats into a form that ML models can understand. This can be computationally intensive and hard to scale, especially for the high-dimensional vectors used in complex models. Pinecone is a managed database built specifically for working with vector data. Pinecone is serverless and API-driven, which means engineers and data scientists can focus on building their ML application or performing analysis without worrying about the underlying data infrastructure. Edo Liberty is the founder and CEO of Pinecone. Prior to Pinecone, he led the creation of Amazon SageMaker at AWS. He joins the show today to talk about the fundamental importance of vectors in machine learning, how Pinecone built a vector-centric database, and why data infrastructure improvements are key to unlocking the next generation of AI applications. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Email has become such a routine feature of knowledge work that we often take it- and the email clients we use for it- for granted. While advancements such as intelligent spam filtering have improved the experience, many email clients retain the same basic structure and offer a largely similar experience. Superhuman is building a modern email client meant to reimagine email from the ground up. Superhuman is built to be fast, and seamlessly integrates insights from social networks such as Linkedin. It offers features such as undo send, AI triage, and mail status tracking. Superhuman even works offline, using Service Workers to serve cached assets when a network connection is not available. Emuye Reynolds is the Head of Engineering at Superhuman. She was formerly a Senior Software Developer at Apple and led the development of the UI for Apple TV UI. She joins the show today to talk about what’s lacking from traditional email clients, what engineering challenges her team faces when building across multiple platforms, and how Superhuman’s new features represent an evolutionary step forward for email client technology. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

WordPress is a free and open-source content management system, or CMS, written in PHP. Since its release in 2003, WordPress has become ubiquitous on the web. It is estimated that roughly 60 million websites use WordPress as a CMS. However, despite its popularity, Wordpress has limitations in its design. Wordpress sites are dynamic, and the front and back end are tightly coupled. A dynamic, full-stack application can be useful when handling complex functionality, but it also slows down the site and opens up security vulnerabilities. Strattic is a static site generator and hosting platform that specializes in converting WordPress sites into a static architecture. Static pages are isolated from the backing application, improving security against common Wordpress vulnerabilities. Modern web users have high expectations for speed and security, and Strattic helps WordPress sites achieve this without sacrificing the benefits of the Wordpress platform. Zeev Suraski is the CTO of Strattic. Zeev is one of the architects and principal authors of the PHP language, which is the foundation of WordPress. Zeev joins the show today to talk about the place of PHP in modern web development, and how Strattic helps Wordpress developers build modern, fast, and secure sites. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

AWS offers over 200 services as part of its IaaS platform, and that number continues to grow. Organizing all of these services, and tracking the costs they incur, can be a significant challenge, often requiring teams of AWS-certified sysadmins working together to get a handle on an enterprise-scale system. Vantage provides an alternative, streamlined AWS console that makes it easier to manage AWS services and track associated costs. Users link their AWS account to Vantage, and it automatically profiles all their services and aggregates the information into a dashboard. Users can customize how their Vantage console appears and allows users to break down service usage by region. Ben Schaechter is the co-founder of Vantage. Before founding Vantage he was a Senior Product Manager at AWS and DigitalOcean. Ben joins the show today to talk about how Vantage helps streamline the AWS experience, and why teams of all sizes can benefit from a better user experience on the AWS platform. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A data warehouse is a centralized repository that an enterprise may use to store selected data from production systems. Data is transformed into a structured form that makes it easily accessible for business intelligence or other operational users. SQL-compliant databases are frequently used for data warehouses due to the popularity of SQL as a tool in business data analytics. PostgreSQL is a free and open-source relational database management system. Postgres-based databases are widespread and are used by a variety of organizations, from Reddit to the International Space Station, and Postgres databases are a common offering from cloud providers such as AWS, Alibaba Cloud, and Heroku. Josh Drake and Thomas Richter are experts on Postgres data warehousing. They join the show today to talk about the staying power of Postgres, why Postgres is a good choice for data warehousing, and how cloud technology is changing relational database management systems. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Build automation tools automate the process of building code, including steps such as compiling, packaging binary code, and running automated tests. Because of this, build automation tools are considered a key part of a continuous delivery pipeline. Build automation tools read build scripts to define how they should perform a build. Common build scripts include Makefile, Dockerfile, and bash. Earthly is a build automation tool that allows you to execute all your builds in containers. Earthly uses Earthfiles, which draws from the best features of Makefile and Dockerfile and provides a common layer between language-specific tooling and the CI build spec. Earthly builds are repeatable, isolated, and self-contained, and will run the same way across different environments such as a CI system or a developer’s laptop. Vlad Ionescu is the Founder and CEO of Earthly Technologies. He was formerly the founder and chief architect at ShiftLeft.io. Vlad joins the show today to talk about why reproducible builds are important, how Earthly simplifies build scripts, and what the long-term vision for Earthly looks like. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Static analysis is a type of debugging that identifies defects without running the code. Static analysis tools can be especially useful for enforcing security policies by analyzing code for security vulnerabilities early in the development process, allowing teams to rapidly address potential issues and conform to best practices. R2C has developed a fast, open-source static analysis tool called Semgrep. Semgrep provides syntax-aware code scanning and a database of thousands of community-defined rules to compare your code against. Semgrep also makes it easy for security engineers and developers to define custom rules to enforce their organization’s policies. R2C’s platform has been adopted by industry leaders such as Dropbox and Snowflake, and recently received the “Disruptive Innovator” distinction at Forbes’ 2020 Cybersecurity Awards. Isaac Evans is the Founder and CEO of R2C. Before founding R2C he was an Entrepreneur in Residence at Redpoint Ventures and a computer scientist at the US Department of Defense. Isaac joins the show today to talk about how R2C is helping teams improve their cloud security, why static analysis is a natural fit for CI/CD workflows, and what to expect from R2C and the Semgrep project in the future. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern SaaS products are increasingly delivered via the cloud, rather than as downloadable, executable programs. However, many potential users of those SaaS products may need that software deployed on-prem, in a private network. Organizations have a variety of reasons for preferring on-prem software, such as security, integration with private tools, and compliance with regulations. The cost of setting up a bespoke on-prem version of a SaaS offering was often prohibitive for both the vendor and potential users. Replicated leverages the portability of containers to help SaaS vendors ship an on-prem or multi-prem version of their software. Replicated gives SaaS vendors a suite of ready-made components to help install and manage an instance of their software on-prem. Replicated has seen explosive growth in its six-year lifespan: today, 50 of the Fortune 100 companies manage apps with Replicated. Grant Miller is the Founder and CEO of Replicated. Replicated recently closed a Series B fundraising round to help scale their work in the cloud-native space, including the launch of their Kubernetes-Off-the-Shelf platform. He joins the show today to talk about the next generation of tools on the Replicated platform, how Kubernetes is changing enterprise IT, and why the on-prem software market isn’t going away anytime soon. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud platforms are often categorized as providing either Infrastructure-as-a-Service or Platform-as-a-Service. On one side of the spectrum are IaaS giants such as AWS, which provide a broad range of services for building infrastructure. On the other are PaaS providers such as Heroku and Netlify which abstract away the lower-level choices and focus on developer experience. Digital Ocean has carved out a sizable niche in the cloud hosting space by targeting the middle ground- a streamlined cloud platform built for developers, which still offers the ability to choose, customize, and manage infrastructure. The release of Digital Ocean’s App Platform takes this goal a step further. The App Platform allows users to build and deploy an app or static site directly from GitHub directly onto a DigitalOcean-managed Kubernetes cluster. Teams can access the power, scale, and flexibility of Kubernetes without having to worry about the complexity of managing a cluster themselves. The App Platform gives developers the choice of how much of their infrastructure they want to control, and how much they want to be provided by the platform. Cody Baker and Apurva Joshi work at Digital Ocean. They join the show today to talk about why Digital Ocean stands out in a competitive cloud hosting space, what is the value proposition for developers interested in the App Platform, and how the PaaS industry is evolving. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Yelp.com is a crowdsourced review platform focused on restaurants and local businesses. Originally created as an email-based recommendation service, Yelp re-launched in its modern form in 2005. At the time, its focus on user-created reviews and social interactions was fairly novel, and made it stand out from competitors such as Angie’s List and CitySearch. Since then, Yelp has become a worldwide brand, and as of 2021 it has over 171 million reviews on its site. The mid- to-late 2000s represented a time of explosive growth and profound change in the web application space. Industry leaders like Yelp had to adapt their technology stacks for the unprecedented scaling they were experiencing. At the same time, the rise of smartphones led Yelp and many others into the mobile application space. Michael Stoppelman was an engineer at Yelp during this turbulent time. He left Yelp in 2015, and now works as an angel investor. He joins the show today to talk about the engineering challenges Yelp’s team faced during this time, the profound changes that the industry as a whole went through, and how the history of Yelp can help us contextualize the startup landscape today. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Studies show that people in “maker” professions such as developers and writers are most productive when they can carve out dedicated time for focused work, without the frequent context-switching that comes with an irregular meeting schedule. Meetings and other non-development work are necessary parts of the job, but a team will be much more productive with deep work time in mind. Okay is an engineering metrics dashboard platform designed with the goal of maximizing time for deep work. Okay helps break down time slots into categories such as Maker Time, Meeting Load, and Friction Time based on data collected and feedback from the team. Okay organizes both quantitative and qualitative data into a single dashboard for team planning, and supports plug-and-play integrations with productivity tools such as GoogleCalendar, PagerDuty, and CircleCI, and more. Tomas Barreto is the CTO and co-founder of Okay. Before founding Okay, he worked with Sequoia Capital and Y Combinator, and was VP of engineering at Box. He joins the show today to talk about why Maker Time is so important for engineers, and how Okay helps teams make data-driven decisions to maximize productivity. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A “co-location” center is a data center that leases out networking and compute infrastructure to retail clients. Co-location centers host clients with a wide variety of infrastructure strategies, from small retail customers, to medium-size teams running hybrid cloud models, to large corporate clients who prefer not to incur the capital cost of building their own data center. While Equinix is already a market leader in co-location centers, they have expanded to provide a wide variety of services for their clients, including managed IaaS, disaster recovery, and integrations with cloud providers such as AWS and Google Cloud. Shaedon Blackman is a partner developer analyst at Equinix. As a partner developer analyst, Shaedon works to build Equinix’s network of corporate partners, while also advocating for diverse and inclusive human capital within the organization. Before he joined Equinix, he was a Core Fellow at Pursuit, a software engineering fellowship funded by Google, and the Chief Operating Officer of a non-profit youth program. He joins the show today to talk about a variety of important topics facing the tech industry today, including diversity, inclusion, and education, and also how Equinix is building partnerships and sponsoring open-source projects to achieve its goals. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Prediction Markets provide an exchange for trading based on the outcome of events. Most prediction markets are centralized- they operate like a casino, where betting takes place under the supervision of one central governing organization. This makes the market less efficient than it otherwise might be: the central organization is a business, and it makes money by extracting value from the trades the customers make. Augur is a prediction market built on the Ethereum blockchain. A trading network built on a blockchain can have a decentralized, permissionless transaction record without a centralized, governing body. Augur’s network is built to be transparent, low-cost, and free from interference. Joey Krug joins us today from Pantera Capital, a venture capital fund focused on Blockchain technology. Joey is also a co-founder of the Forecast Foundation, which contributes to the development of the Augur open-source project. We discuss what it takes to build a trustworthy decentralized market, how Augur is solving challenges such as the oracle problem, and why blockchain may be the key to democratizing finance. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the past several years, Kubernetes has become the de-facto standard for orchestrating containerized, stateless applications. Tools such as StatefulSets and Persistent Volumes have helped developers build stateful applications on Kubernetes, but this can quickly become difficult to manage as an application scales. Tasks such as machine learning, distributed AI, and big data analytics often require a distributed application to maintain some sort of state across services. KubeDirector is an open-source controller that helps streamline the deployment and management of complex stateful scale-out application clusters on Kubernetes. KubeDirector provides an application-agnostic deployment pattern and enables developers to run non-cloud native stateful applications on Kubernetes without modifying the code. KubeDirector aims to bring enterprise-level capabilities for distributed stateful applications to Kubernetes. Kartik Mathur is an engineer at HPE Developer, an open-source initiative within Hewlett-Packard Enterprise. HPE is an enterprise contributor to the KubeDirector open-source community. Kartik previously worked as senior software engineers at BlueData, which created the KubeDirector project before its acquisition by HPE. Kartik joins the show today to talk about why state is important for Big Data or Machine learning applications, how KubeDirector can help manage the complexity of stateful applications. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Over the past few years, the conventional wisdom around the value proposition of Big Data has begun to shift. While the prevailing attitude towards Big Data may once have been “bigger is better,” many organizations today recognize that broad-scale data collection comes with its own set of risks. Data privacy is becoming a hotly debated topic both in the technology industry and in regulatory agencies and governments. Bigger and less private datasets are more attractive targets for hackers, meaning that an organization must invest heavily in security as well to avoid a breach. Every organization faces a tradeoff between the value of the insights produced from large datasets versus increased storage costs and increasing privacy risks. Tonic is building a “synthetic data” platform to address these tradeoffs and help organizations mitigate data risk. Tonic takes in raw data, perhaps from a data lake, and transforms it into more manageable, de-identified data sets for ease of use and user privacy. Tonic can create statistically identical, structured datasets that allow software engineers and business analysts to extract the same useful insights that drive an organization’s progress, without the risk of working with identifiable, private user data. Ian Coe, Andrew Colombi, and Adam Kamor are co-founders of Tonic. Along with their fourth co-founder, Karl Hanson, Ian, Andrew, and Adam all worked together at Palantir Technologies where the idea for Tonic was born. They join the show today to talk about the value of synthetic data, the risks and rewards of big data, and how compliance, privacy, and security are driving innovation in the data management sector. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Serverless computing refers to an architectural pattern where server-side code is run on-demand by cloud providers, who also handle server resource allocation and operations. Of course, there is a server involved on the provider’s side, but administrative functions to manage that server such as capacity planning, configuration, or management of containers are handled behind-the-scenes, allowing the application developers to focus on business logic. This makes for highly elastic and scalable systems and can reduce development, testing, and iteration time due to reduced overhead. Function as a Service (FaaS) describes a model of serverless computing where services are decomposed into modular functions and deployed to a serverless platform. These functions are executed only when called and are typically stateless. Despite the benefits of elasticity and modularity that FaaS offers, it has drawbacks as well. Taking disaggregation of functionality to an extreme means that behavior that formerly required a method call now may require a network call to another function, increasing latency and making larger-scale operations inefficient. Cloudburst is a stateful FaaS platform built to combine the power of low-latency mutable state and communication in Python with the elasticity and scalability allowed by serverless architecture. Johann Schleier-Smith is an entrepreneur and engineer who is currently a board member of Sama. Johann was formerly the founder and CTO of if(we), a social network and incubator. He is the co-author of the paper Cloudburst: Stateful Functions as a Service, and joins the show today to talk about how Cloudburst addresses the drawbacks of current FaaS models, and what’s next for serverless computing. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Blockchain technology has a wide variety of potential applications. Fields such as finance, supply chain management, and even voting have seen innovations driven by the development of distributed applications built on blockchains, called DApps. However, developing a DApp on a blockchain often requires low-level knowledge about cryptographic protocols or particular networks. Since no one blockchain platform has emerged as dominant- and the field itself is rapidly evolving- there is a high opportunity cost for developers if they choose to invest significant time learning one blockchain paradigm or another. Reach provides a platform for developing DApps, complete with a high-level language based on Javascript. Reach allows developers to write one set of code to specify the DApp and all its components, and which can be deployed onto any blockchain implementation under the hood. Reach’s goal is to allow developers to focus on writing business logic for their DApps rather than worrying about low-level implementation details and aims to smooth the steep learning curve for developers new to the world of blockchain. Chris Swenor and Jay McCarthy are the founders of Reach. Chris was formerly the co-founder and CEO of Alacris Protocol, an operating system for blockchain applications, and he is currently a technologist in residence and mentor at Harvard. Jay has been a computer science professor for over a decade, and worked on the development of the Racket programming language. Chris and Jay join the show today to talk about the challenges of developing on blockchain, how Reach helps make blockchain developers more productive, and how the blockchain ecosystem might evolve in the future. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The incredible advances in machine learning research in recent years often take time to propagate out into usage in the field. One reason for this is that such “state-of-the-art” results for machine learning performance rely on the use of handwritten, idiosyncratic optimizations for specific hardware models or operating contexts. When developers are building ML-powered systems to deploy in the cloud and at the edge, their goals to ensure the model delivers the best possible functionality and end-user experience- and importantly, their hardware and software stack may require different optimizations to achieve that goal. OctoML provides a SaaS product called the Octomizer to help developers and AIOps teams deploy ML models most efficiently on any hardware, in any context. The Octomizer deploys its own ML models to analyze your model topology, and optimize, benchmark, and package the model for deployment. The Octomizer generates insights about model performance over different hardware stacks and helps you choose the deployment format that works best for your organization. Luis Ceze is the Co-Founder and CEO of OctoML. Luis is a founder of the ApacheTVM project, which is the basis for OctoML’s technology. He is also a professor of Computer Science at the University of Washington. Jason Knight is co-founder and CPO at OctoML. Luis and Jason join the show today to talk about how OctoML is automating deep learning engineering, why it’s so important to consider hardware when building deep learning systems, and how the field of deep learning is evolving. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A data-driven organization collects a wide variety of data to help in strategic decision-making. The cost of storing large amounts and variety of data has dropped dramatically in the last two decades, but too much unstructured data may not improve decision-making, and can even lead to “analysis paralysis.” Organizations react by extracting the most important, actionable data and placing it into a data warehouse, which has a predesigned structure meant to streamline the data in preparation for analysis. The key challenge with this approach is identifying what should be streamlined, and how to structure the data warehouse to focus on the most important, actionable items. This is especially important for organizations seeking to scale, as the necessary structure to generate the most relevant insights may change as the organization grows. Narrator is building data intelligence that uses a simple, proprietary Universal Data Model to help organizations streamline their data warehousing. Narrator is built on the belief that data tells the story of a system, and its platform empowers organizations to use those stories to make better decisions. Ahmed Elsamadisi is the founder and CEO of Narrator. Before founding Narrator, he spent several years working in data analysis and algorithm design for WeWork, Raytheon, and Cornell’s Autonomous Systems Laboratory. He joins the show today to talk about how Narrator generates the most actionable insights from a data warehouse, why a Universal Data Model is so important when scaling, and what makes Narrator’s approach to data analysis different. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Open source software is software distributed along with its source code, using a permissive license that allows anyone to view, use, or modify it. The term “open source” also refers more broadly to a philosophy of technology development which prioritizes transparency and community development of a project. Typically, development is managed by a governing body, whether a company, foundation, or just a group of passionate users, and work is done in public repositories like Github. Nearly every corner of the software engineering world has been impacted in some way by open source. Well-known open source projects include Linux, Kubernetes, and Wordpress. Kevin Xu is the author of Interconnected, a bilingual newsletter on tech, business, and U.S-China relations. He is an investor in open source startups at OSS Capital, and formerly served in the Obama White House. He joins the show today to talk about the benefits of open source in the public and private sectors, and how open source will be critical to the development of high-tech industry in our country as we pivot to facing some of the 21st century’s most pressing challenges. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Video calling over the internet has experienced explosive growth in the last decade. In 2010, surveys estimated that around 1 in 5 Americans had tried online video calling for any reason. By May of 2020, that number had nearly tripled. A significant factor in the growth of video calling has been an open-source project called WebRTC, or “Web Real-Time Communication.” WebRTC makes it possible to capture and stream audio or video data between browsers without the use of plugins or third-party software. Daily is a developer platform that builds on WebRTC to provide realtime video APIs for developers. Developers can easily add video call widgets to their code which come with a set of default configurations for functions such as bandwidth management and cross-browser support. Daily also offers a set of frontend libraries and REST APIs for developers who want to build a customized experience. Kwindla Hultman Kramer is a co-Founder at Daily, and he’s joined today by Wesley Faulkner, who handles developer relations. They join the show today to talk about the growth in demand for video calling services, building a developer-friendly video calling API, and what’s next for video calling applications. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cilium is open-source software built to provide improved networking and security controls for Linux systems operating in containerized environments along with technologies like Kubernetes. In a containerized environment, traditional Layer 3 and Layer 4 networking and security controls based on IP addresses and ports, like firewalls, can be difficult to operate at scale because of the volatility of the system. Cilium is eBPF, which is an in-kernel virtual machine which attaches applications directly to code paths in the kernel. In effect, this makes the Linux kernel “programmable” without changing kernel source code or loading modules. Cilium takes advantage of this functionality to insert networking and security functions at the kernel level rather than in traditional Layer 3 or Layer 4 controls. This allows Cilium to combine metadata from Layer 3 and Layer 4 with application-layer metadata such as HTTP method and header values in order to establish rules and provide visibility based on service, pod, or container identity. Isovalent, co-founded by the creator of Cilium, maintains the Cilium Open Source Project and also offers Cilium Enterprise, which is a suite of tools helping organizations adopt Cilium and overcome the hurdles of building a secure, stable cloud-native application. Dan Wendlant and Thomas Graf are the co-founders of Isovalent. Thomas, the firm’s CTO, was the original creator of the Cilium open-source project and spent 15 years working on the Linux kernel prior to founding Isovalent. Dan, Isovalent’s CEO, has also worked at VMWare and Nicira. They join the show today to talk about why Cilium and Cilium Enterprise are a great choice for organizations looking to build cloud-native applications. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In a distributed application, observability is key to handling incidents and building better, more stable software. Legacy monitoring methods were built to respond to predictable failure modes, and to aggregate high-level data like access speed, connectivity, and downtime. Observability, on the other hand, is a measure of how well you can infer the internal state of a system from its outputs in order to trace the cause. At its core, building a system with observability means using instrumentation to provide insights on how and why internal components within a system are performing a certain way. Developers and SREs can build on that data to proactively debug potential failure modes, set service-level objectives, and speed up incident response. New Relic has been an industry leader in the observability space for the better part of a decade. This year, they announced New Relic One, an evolution of their flagship platform that streamlines and simplifies all the functions available to help organizations achieve observability. New Relic One enhances the Full-Stack Observability Platform through AIOps with their Applied Intelligence, which draws insights from the observability data to help detect anomalies before they become incidents. Lew Cirne is the founder and CEO of New Relic. He joins the show today to talk about how New Relic One helps developers move beyond monitoring and embrace observability, and how he sees the future of software observability platforms. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Embedded Software Engineering is the practice of building software that controls embedded systems- that is, machines or devices other than standard computers. Embedded systems appear in a variety of applications, from small microcontrollers, to consumer electronics, to large-scale machines such as cars, airplanes, and machine tools. iRobot is a consumer robotics company that applies embedded engineering to build robots that perform common household tasks. Its flagship product is the Roomba, perhaps one of the most well-known autonomous consumer robots on the market today. iRobot’s engineers work at the intersection of software and hardware, and work in a variety of domains from electrical engineering to AI. Chris Svec is a Software Engineering Manager at iRobot. He started his career designing x86 chips and later moved up the hardware/software stack into embedded software. He joins the show today to talk about iRobot, the design process for embedded systems, and the future of embedded systems programming. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Security is more important than ever, especially in regulated fields such as healthcare and financial services. Developers working in highly regulated industries often spend considerable time building tooling to help improve compliance and pass security audits. While the core of many security workflows is similar, each industry and each organization may have its own idiosyncratic needs or particular regulatory requirements to meet. Sym is a platform for building security workflows that seeks to build on those core similarities while empowering developers with the tools they need to meet their application’s unique security and compliance needs. Sym believes in putting engineers in control of security, in the same way that DevOps put engineers in control of infrastructure. Yasyf Mohamedali is the CEO and co-founder of SymOps. Before SymOps, he was the CTO of Karuna Health. He joins the show today to talk about security and innovation in regulated industries and how Sym can help developers close the intent-to-implementation gap in application security. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Microservices are built to scale. But as a microservices-based system grows, so does the operational overhead to manage it. Even the most senior engineers can’t be familiar with every detail of dozens- perhaps hundreds- of services. While smaller teams may track information about their microservices via spreadsheets, wikis, or other more traditional documentation, these methods often prove unsuitable for the unique demands of a sprawling microservices system. A microservices catalog is a solution to this problem. A microservices catalog seeks to centralize information about the services in your software architecture, including the purpose of a service, its owner, and instructions for using it. A microservices catalog can also provide a centralized source of knowledge about a system, which can help on-call engineers diagnose issues and also provide resources for onboarding new team members. Larger companies sometimes devote significant internal resources toward developing in-house microservices catalogs, while smaller organizations may not have the resources at their disposal to do so. OpsLevel’s founders recognized that many teams were re-inventing the wheel building internal microservices catalogs, and set out to design a toolset that could meet the needs of users of all sizes. OpsLevel’s team has drawn from extensive experience working with industry leaders in DevOps to create a comprehensive toolset for managing microservices infrastructure. OpsLevel provides a “single pane of glass for operations,” integrating with a variety of tools such as Slack, git, CI/CD, incident management, and deployment systems. John Laban and Kenneth Rose are the co-founders of OpsLevel. Before John and Kenneth founded OpsLevel they worked together at PagerDuty, where John was the first engineer on the team. Kenneth, OpsLevel’s CTO, was also previously a senior developer at Shopify. John and Kenneth join the show today to talk about how OpsLevel can help developers manage their microservices better, and even transform how their team does DevOps. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Reinforcement learning is a paradigm in machine learning that uses incentives- or “reinforcement”- to drive learning. The learner is conceptualized as an intelligent agent working within a system of rewards and penalties in order to solve a novel problem. The agent is designed to maximize rewards while pursuing a solution by trial-and-error. Programming a system to respond to the complex and unpredictable “real world” is one of the principal challenges in robotics engineering. One field which is finding new applications for reinforcement learning is the study of MEMS devices- robots or other electronic devices built at the micrometer scale. The use of reinforcement learning in microscopic devices poses a challenging engineering problem, due to constraints with power usage and computational power. Nathan Lambert is a PhD student at Berkeley who works with the Berkeley Autonomous Microsystems Lab. He has also worked at Facebook AI Research and Tesla. He joins the show today to talk about the application of reinforcement learning to robotics and how deep learning is changing the MEMS device landscape. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

For startups that are still seeking product/market fit, pre-seed investments are critical to funding initial investments in the product and in the infrastructure needed to scale. Afore Capital is a pre-seed fund that invests in innovative companies across a wide variety of verticals. Afore focuses on startups with unique product insights and novel distribution approaches. Gaurav Jain is a co-Founder and Managing Partner at Afore Capital. Before that, he worked with the Android team at Google. Gaurav joins the show today to talk about the risks and rewards of pre-seed investing, and about how founders and investors can find opportunity in the current venture environment. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kafka has achieved widespread popularity as a popular distributed queue and event streaming platform, with enterprise adoption and a billion dollar company (Confluent) built around it. But could there be value in building a new platform from scratch? Redpanda is a streaming platform built to be compatible with Kafka, that does not require the JVM nor Zookeeper, both of which are dependencies that made Kafka harder to work with than perhaps necessary. Alexander Gallego is a core committer to Redpanda and joins the show to talk about why he started the project and its value proposition. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Companies can have a negative impact on the environment by outputting excess carbon. Many companies want to reduce their net carbon impact to zero, which can be done by investing in forests. Pachama is a marketplace for forest investments. Pachama uses satellites, imaging, machine learning, and other techniques to determine how much carbon is being absorbed by different forests. Diego Saez-Gil is a founder of Pachama, and joins the show to talk through how Pachama works and the long-term goals of the company. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Rust and Golang are two of the newest lower level languages for doing systems programming. They are often used for applications such as file systems, operating systems, and latency-sensitive applications. How do they compare in terms of safety, speed, and programming ergonomics? Linhai Song is an assistant professor and researcher at Penn State University, and joins the show to talk about his work researching Go and Rust. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

GraphQL has changed the common design patterns for the interface between backend and frontend. This is usually achieved by the presence of a GraphQL server, which interprets and federates a query from the frontend to the backend server infrastructure. Dgraph is a distributed graph database with native GraphQL support. Manish Jain is a founder of Dgraph, and joins the show to talk about its purpose and his vision for the future of the technology. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Agriculture infrastructure allows plants such as corn, soy, and wheat to move from large scale farms to consumers all around the world. The relevant players in the agricultural infrastructure includes growers, shippers, and planners. These individuals need new technology to interact more efficiently. Growers need to be able to connect more smoothly with buyers. Farmers need better management of their carbon credits. Microbial technology can allow plants to be better shielded from tough conditions. Agricultural health, transport, commerce, and logistics are all problems that Indigo Agriculture is focused on solving. David Potere is head of geoinnovation at Indigo, and joins the show to talk about the problems the company is solving and the engineering practices at Indigo. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Network discovery allows enterprises to identify what devices are on their network. These devices can include smartphones, servers, desktop computers, and tablets. Being able to index the devices on a network is crucial to figuring out the security profile of that network. HD Moore is a founder of Rumble Networks, a company focused on network discovery and asset inventory. He joins the show to talk about how network discovery works and his experience building Rumble. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Large technology companies are a new type of industry. Their power and reach is resistant to a comparison of previous generations, such as big oil. Alex Kantrowitz is a journalist who has covered big technology for much of his career, and he currently runs Big Technology, a newsletter and podcast about the biggest technology companies in the world. He’s also the author of “Always Day One: How The Tech Titans Plan To Stay On Top Forever”. Alex joins the show to talk about his work and share his thoughts on big tech. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cost management is growing in importance for companies that want to manage their significant cloud bill. Kubernetes plays an increasing role in modern infrastructure, so managing cost of Kubernetes clusters becomes important as well. Kubecost is a company focused on giving visibility into Kubernetes resources and reducing spend. Webb Brown is a founder of Kubecost and joins the show to talk about Kubernetes cost optimization and what he is building with Kubecost. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

TensorFlow Lite is an open source deep learning framework for on-device inference. TensorFlow Lite was designed to improve the viability of machine learning applications on phones, sensors, and other IoT devices. Pete Warden works on TensorFlow Lite at Google and joins the show to talk about the world of machine learning applications and the necessary frameworks and devices necessary to build them. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data lakes and data warehouses store high volumes of multidimensional data. Data sources for these pieces of infrastructure can become unreliable for a variety of reasons. When data sources break, it can cause downstream problems. One company working to solve the problem of data reliability is Monte Carlo Data. Barr Moses and Lior Gavish are founders of Monte Carlo and join the show to talk about data reliability and the overall landscape of data infrastructure. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When Tim Wagner worked at Amazon, he invented AWS Lambda. After working on the early serverless infrastructure, he joined Coinbase and worked as VP of Engineering. Since leaving Coinbase, he has started a new company called Vendia. Vendia combines his learnings from the serverless space with the innovations around blockchains to work on the problem of data sharing. Tim and David Wells join the show to discuss what they are working on with Vendia. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Serverless has grown in popularity over the last five years, and the space of applications that can be built entirely with serverless has increased dramatically. This is due to two factors: the growing array of serverless tools (such as edge-located key value stores) and the rising number of companies with serverless offerings. One of those companies is Fastly, which originally gained adoption for its CDN solution. Tyler McMullen is the CTO of Fastly and he joins the show to talk through how Fastly looks at edge computing today. This is Tyler’s third appearance on the show. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published July 30, 2019 “Internet of Things” is a term used to describe the increasing connectivity and intelligence of physical objects within our lives. IoT has manifested within enterprises under the term “Industrial IoT,” as wireless connectivity and machine learning have started to improve devices such as centrifuges, conveyor belts, and factory robotics. In the consumer space, IoT has moved slower than many people expected, and it remains to be seen when we will have widespread computation within consumer devices such as microwaves, washing machines, and lightswitches. IoT computers have different constraints than general purpose computers. Security, reliability, battery life, power consumption, and cost structures are very different in IoT devices than in your laptop or smartphone. One technology that could solve some of the problems within IoT is WebAssembly, a newer binary instruction format for executable programs. Jonathan Beri is a software engineer and the organizer of the San Francisco WebAssembly Meetup. He has significant experience in the IoT industry, and joins the show to discuss the state of WebAssembly, the surrounding technologies, and their impact on IoT. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published July 19, 2019 In 2011, Facebook had begun to focus its efforts on mobile development. Mobile phones did not have access to reliable, high bandwidth connections, and the Facebook engineering team needed to find a solution to improve the request latency between mobile clients and the backend Facebook infrastructure. One source of latency was recursive data fetching. If a mobile application client made a request to the backend for newsfeed, the backend API would return the newsfeed, but some components of that feed would require additional requests to the backend. In practice, this might result in a newsfeed loading partially on a phone, but having a delayed loading time for the comments of a newsfeed item. GraphQL is a solution that came out of this problem of recursive data fetching. A GraphQL server provides middleware to aggregate all of the necessary information to serve a complete request. GraphQL connects to backend data sources and federates the frontend request across these different data sources. GraphQL was open sourced in 2015, and has found many use cases in addition to simplifying backend data fetching for mobile clients. Today, GraphQL is used by PayPal, Shopify, Twitter, and hundreds of other companies. Lee Byron is the co-creator of GraphQL and he joins the show to tell the story of GraphQL, and how it fit into Facebook’s shift to mobile. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published September 13, 2019 Amazon Web Services first came out in 2006. It took several years before the software industry realized that cloud computing was a transformative piece of technology. Initially, the common perspective around cloud computing was that it was a useful tool for startups, but would not be a smart option for large, established businesses. Cloud computing was not considered economical nor secure. Today, that has changed. Every company that writes software is figuring out how to utilize the cloud. Software companies with on-prem servers are migrating old applications to the cloud, and most companies that have started in the last decade do not even have physical servers. Applications that are started on the cloud are referred to as “cloud-native.” The architecture of cloud-native applications is a newer topic of discussion, and some software patterns that became established in the pre-cloud era make less sense today. Cornelia Davis is VP of technology at Pivotal and the author of Cloud Native Patterns, a book about developing applications in the distributed, virtual world of the cloud. Cornelia was previously on the show to discuss Cloud Foundry. In today’s episode, our conversation centers on her book, and her perspective on the emerging patterns of cloud native software. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published May 29, 2020 Kubernetes has become a highly usable platform for deploying and managing distributed systems. The user experience for Kubernetes is great, but is still not as simple as a full-on serverless implementation--at least, that has been a long-held assumption. Why would you manage your own infrastructure, even if it is Kubernetes? Why not use autoscaling Lambda functions and other infrastructure-as-a-service products? Matt Ward is a listener of the show and an engineer at Mux, a company that makes video streaming APIs. He sent me an email that said Mux has been having success with self-managed Kubernetes infrastructure, which they deliberately opted for over a serverless deployment. I wanted to know more about what shaped this decision to opt for self-managed infrastructure, and the costs and benefits that Mux has accrued as a result. Matt joins the show to talk through his work at Mux, and the architectural impact of opting for Kubernetes instead of fully managed serverless infrastructure. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published April 17, 2019 Drishti is a company focused on improving manufacturing workflows using computer vision. A manufacturing environment consists of assembly lines. A line is composed of sequential stations along that manufacturing line. At each station on the assembly line, a worker performs an operation on the item that is being manufactured. This type of workflow is used for the manufacturing of cars, laptops, stereo equipment, and many other technology products. With Drishti, the manufacturing process is augmented by adding a camera at each station. Camera footage is used to train a machine learning model for each station on the assembly line. That machine learning model is used to ensure the accuracy and performance of each task that is being conducted on the assembly line. Krish Chaudhury is the CTO at Drishti. From 2005 to 2015 he led image processing and computer vision projects at Google before joining Flipkart, where he worked on image science and deep learning for another four years. Krish had spent more than twenty years working on image and vision related problems when he co-founded Drishti. In today’s episode, we discuss the science and application of computer vision, as well as the future of manufacturing technology and the business strategy of Drishti. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published April 7, 2017 Engineers in Silicon Valley see a world of constant progress. Our work is creative and intellectually challenging. We are building the future and getting compensated quite well for it. But what if we are actually achieving far less than what is possible? What if, after so many years of high margins, gourmet lunch, and self-flattery, we have lowered our standards for innovation? And if Silicon Valley has been lulled into complacency, what does that say about the rest of the United States? American exceptionalism has faltered and complacency has risen in its wake. Today’s guest Tyler Cowen is an economist and author. His new book The Complacent Class is the final book in a trilogy that describes a decline of American output and a decline in American mindset. Complacent America has lost its ability to assess risk. Children are prevented from playing tag for risk of injury. College students protest against speakers who might present challenging ideas. The number of Americans under 30 who own a business has fallen by 65% since the 1980’s--millennials are too busy going to business school to start businesses. In his books, Tyler weaves together history, philosophy, and contemporary culture. He presents hard data about many different fields, and theorizes about how the trends in those fields relate to each other. He also has a podcast, Conversations with Tyler, and in this episode I tried to mirror his interview style. If you like this episode, you should check out his show--he has interviewed people like Ezra Klein, Peter Thiel, and Kareem Abdul-Jabbar. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published March 6, 2020 ReactJS developers have lots of options for building their applications, and those options are not easy to work through. State management, concurrency, networking, and testing all have elements of complexity and a wide range of available tools. Take a look at any specific area of JavaScript application development, and you can find highly varied opinions. Kent Dodds is a JavaScript teacher who focuses on React, JavaScript, and testing. In today’s episode, Kent provides best practices for building JavaScript applications, specifically React. He provides a great deal of advice on testing, which is unsurprising considering he owns TestingJavaScript.com. Kent is an excellent speaker who has taught thousands of people about JavaScript, so it was a pleasure to have him on the show. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published June 21, 2019 Niantic is the company behind Pokemon Go, an augmented reality game where users walk around in the real world and catch Pokemon which appear on their screen. The idea for augmented reality has existed for a long time. But the technology to bring augmented reality to the mass market has appeared only recently. Improved mobile technology makes it possible for a smartphone to display rendered 3-D images over a video stream without running out of battery. Ingress was the first game to come out of Niantic, followed by Pokemon Go, but there are other games on the way. Niantic is also working on the Niantic Real World platform, a “planet-scale” AR platform that will allow independent developers to build multiplayer augmented reality experiences that are as dynamic and entertaining as Pokemon Go. Paul Franceus is an engineer at Niantic, and he joins the show to describe his experience building and launching Pokemon Go, as well as abstracting the technology from Pokemon Go and opening up the Niantic Real World platform to developers. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published July 7, 2017 Airbnb is a company that is driven by design. New user interfaces are dreamed up by designers and implemented for web, iOS, and Android. This implementation process takes a lot of resources, but it used to take even more before the company started using React Native. React Native allows Airbnb to reuse components effectively. React Native works by presenting a consistent model for the user interface regardless of the underlying platform, and emitting a log of changes to that user interface. The underlying platform translates those changes into platform specific code. Leland Richardson is an engineer at Airbnb. In today’s episode, he explains how Airbnb uses React Native, how React Native works, and the future of the platform. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published October 18, 2019 Apache Kafka was created at LinkedIn. Kafka was open sourced in 2011, when the company was eight years old. By that time, LinkedIn had developed a social network with millions of users. LinkedIn’s engineering team was building a range of externally facing products and internal tools, and many of these tools required a high-throughput system for publishing data and subscribing to topics. Kafka was born out of this need. Over time, Kafka’s importance within LinkedIn has only grown. Kafka plays a central role for services, log management, data engineering, and compliance. LinkedIn might be the biggest user of Apache Kafka in the entire software industry. Kafka has many use cases, and it is likely that they are almost all on display within LinkedIn. Nacho Solis is a senior software engineering manager at LinkedIn, where he helps teams build infrastructure for Kafka, as well as Kafka itself. Nacho joins the show to discuss the history of Kafka at LinkedIn, and the challenges of managing such a large deployment of Kafka. We also talk about streaming, data infrastructure, and more general problems in the world of engineering management. Full disclosure: LinkedIn is a sponsor of Software Engineering Daily. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published December 9, 2019 Machine learning algorithms have existed for decades. But in the last ten years, several advancements in software and hardware have caused dramatic growth in the viability of applications based on machine learning. Smartphones generate large quantities of data about how humans move through the world. Software-as-a-service companies generate data about how these humans interact with businesses. Cheap cloud infrastructure allows for the storage of these high volumes of data. Machine learning frameworks such as Apache Spark, TensorFlow, and PyTorch allow developers to easily train statistical models. These models are deployed back to the smartphones and the software-as-a-service companies, which improves the ability for humans to move through the world and gain utility from their business transactions. And as the humans interact more with their computers, it generates more data, which is used to create better models, and higher consumer utility. The combination of smartphones, cloud computing, machine learning algorithms, and distributed computing frameworks is often referred to as “artificial intelligence.” Chris Benson is the host of the podcast Practical AI, and he joins the show to talk about the modern applications of artificial intelligence, and the stories he is covering on Practical AI. On his podcast, Chris talks about everything within the umbrella of AI, from high level stories to low level implementation details. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published September 17, 2019 Ever since Apache Kafka was open sourced from LinkedIn, it has been used to solve a wide variety of problems in distributed systems and data engineering. Kafka is a distributed messaging queue that is used by developers to publish messages and subscribe to topics with a certain message type. Kafka allows information to flow throughout a company such that multiple systems can consume the messages from a single sender. In previous shows, we have covered design patterns within Kafka, Kafka streams, event sourcing with Kafka, and many other subjects relating to the technology. Kafka is broadly useful, and new strategies for using Kafka continue to emerge as the open source project develops new functionality and becomes a platform for data applications. In today’s episode, Tim Berglund returns to Software Engineering Daily for a discussion of how applications are built today using Kafka--including systems that are undergoing a refactoring, data engineering applications, and systems with a large number of communicating services. If you are interested in learning more about how companies are using Kafka, the Kafka Summit in San Francisco is September 30th - October 1st. Companies like LinkedIn, Uber, and Netflix will be talking about how they use Kafka. Full disclosure: Confluent (the company where Tim works) is a sponsor of Software Engineering Daily. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published January 25, 2019 When TensorFlow came out of Google, the machine learning community converged around it. TensorFlow is a framework for building machine learning models, but the lifecycle of a machine learning model has a scope that is bigger than just creating a model. Machine learning developers also need to have a testing and deployment process for continuous delivery of models. The continuous delivery process for machine learning models is like the continuous delivery process for microservices, but can be more complicated. A developer testing a model on their local machine is working with a smaller data set than what they will have access to when it is deployed. A machine learning engineer needs to be conscious of versioning and auditability. Kubeflow is a machine learning toolkit for Kubernetes based on Google’s internal machine learning pipelines. Google open sourced Kubernetes and TensorFlow, and the projects have users AWS and Microsoft. David Aronchick is the head of open source machine learning strategy at Microsoft, and he joins the show to talk about the problems that Kubeflow solves for developers, and the evolving strategies for cloud providers. David was previously on the show when he worked at Google, and in this episode he provides some useful discussion about how open source software presents a great opportunity for the cloud providers to collaborate with each other in a positive sum relationship. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published December 20, 2018 Ten years ago, there was a distinction between “backend” and “frontend” developers. A backend developer would be managing the business logic and database transactions using Ruby on Rails or Java. A frontend developer would be responsible for implementing designs and arranging buttons using raw HTML and JavaScript. Today, developers can build entire applications in JavaScript. Developers who spent their early career developing frontend JavaScript skills are finding themselves with a surprising amount of power. With NodeJS providing a backend framework and React, Vue, or Angular on the frontend, a single JavaScript developer can write all the code for a whole application—hence the rise of the “full stack developer”. At the same time, the cloud infrastructure is becoming easier to use. Backend-as-a-service simplifies the frustrations of deploying your application, and standing up a database. GraphQL improves the relationship between the frontend and the backend. And futuristic technologies like WebAssembly and web virtual reality are promising to make a JavaScript engineer’s life even more interesting. Adam Conrad is an engineer and a writer for Software Engineering Daily. In recent articles, he has documented the changing nature of the frontend, including JavaScript engines, virtual reality, and how mature corporations are using React and GraphQL. He joins the show to share his perspective on what is changing in the frontend—and how full stack JavaScript engineers can position themselves for future success in a quickly changing market. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published May 16, 2019 React is a set of open source tools for building user interfaces. React was open sourced by Facebook, and includes libraries for creating interfaces on the web (ReactJS) and on mobile devices (React Native). React was released during a time when there was not a dominant frontend JavaScript library. Backbone, Angular, and other JavaScript frameworks were all popular, but there was not any consolidation across the frontend web development community. Before React came out, frontend developers were fractured into different communities for the different JavaScript frameworks. After Facebook open sourced React, web developers began to gravitate towards the framework for its one-way data flow and its unconventional style of putting JavaScript and HTML together in a format called JSX. As React has grown in popularity, the React ecosystem has developed network effects. In many cases, the easiest way to build a web application frontend is to compose together open source React components. After seeing the initial traction, Facebook invested heavily into React, creating entire teams within the company whose goal was to improve React. Dan Abramov works on the React team at Facebook and joins the show to talk about how the React project is managed and his vision for the project. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published August 28, 2019 Kent Beck is a legendary figure in the world of software engineering. Kent was an early advocate of Test-Driven Development (TDD), and popularized the idea of writing unit tests before writing code that would satisfy those unit tests. A unit test isolates and tests a small piece of functionality within a large piece of software. Practitioners of Test-Driven Development write tens or hundreds of tests in order to cover a large variety of cases that could potentially occur within their software. When Kent Beck joined Facebook in 2011, he was 50 years old and thought he had seen everything in the software industry. During Facebook Boot Camp, Kent started to realize that Facebook was very different than any other company he had seen. Facebook Boot Camp is the six-week onboarding process that every new hire learns about the software practices of the company. After graduating Facebook Boot Camp, Kent began to explore Facebook’s codebase and culture. He found himself rethinking many of the tenets of software engineering that he had previously thought were immutable. Kent joins the show to discuss his time at Facebook, and how the company’s approach to building and scaling products thoroughly reshaped his beliefs about software engineering. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published April 3, 2017 A hedge fund is a collection of investors that make bets on the future. The “hedge” refers to the fact that the investors often try to diversify their strategies so that the direction of their bets are less correlated, and they can be successful in a variety of future scenarios. Engineering-focused hedge funds have used what might be called “machine learning” for a long time to predict what will happen in the future. Numerai is a hedge fund that crowdsources its investment strategies by allowing anyone to train models against Numerai’s data. A model that succeeds in a simulated environment will be adopted by Numerai and used within its real money portfolio. The engineers who create the models are rewarded in proportion to how well the models perform. Xander Dunn is a software engineer at Numerai and in this episode he explains what a hedge fund is, why the traditional strategies are not optimal, and how Numerai creates the right incentive structure to crowdsource market intelligence. This interview was fun and thought provoking--Numerai is one of those companies that makes me very excited about the future. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published March 31, 2017 Brendan Eich created the first version of JavaScript in 10 days. Since then JavaScript has evolved, and Brendan has watched the growth of the web give rise to new and unexpected use cases. Today Brendan Eich is still pushing the web forward across the technology stack with his involvement in the WebAssembly specification and the Brave browser. For all of its progress, JavaScript struggles to run resource-intensive programs like complex video games. With JavaScript falling short on its charge to be the “assembly language for the web” the four major browser vendors started collaborating on the WebAssembly project to allow programming languages a faster, lower level compile target when deploying to the web. Brendan is the CEO of Brave which aims to provide a faster and safer browsing experience by blocking ads and trackers by default in a new browser. The Brave browser is also helping publishers monetize in interesting new ways while also giving a share of ad revenue to its users. Caleb Meredith is the host of this show. He previously guest hosted a popular episode on Inferno, a fast, React-like JavaScript framework. As we bring on more guest hosts, please send us feedback. We want to know what every host is doing well, and what we can improve on. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published September 4, 2018 TIBCO was started in the 90’s with a popular message bus product that was widely used by finance companies, logistics providers, and other systems with high throughput. As TIBCO grew in popularity, the company expanded into other areas through products it developed in-house as well as through acquisitions. One acquisition was Jaspersoft, a business intelligence data platform. When TIBCO acquired Jaspersoft in 2014, the architecture was a monolithic Java application. Around this time, customer use cases were shifting from centralized reporting to real-time, embedded visualizations. The use case of the Jaspersoft software was becoming less centralized and less monolithic and the software architecture needed to change in order to reflect that. Jan Schiffman is a VP of engineering at TIBCO and Sherman Wood is a director at TIBCO. They join the show to discuss the process of migrating a large Java monolith to a composable set of services. Breaking up a monolith is not an easy process--nor is it something that every company should do just because they have a monolith. In some cases, a monolith is just fine. Jan and Sherman explain why the business use case for why the Jaspersoft monolith needed to be refactored, and their approach to the refactoring. We also talk through the modern use cases of embedded analytics and the interaction between business analysts and data engineers. At a higher level, we discuss the lessons they have learned from managing a large, complex refactoring. Full disclosure: TIBCO is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Osquery is a tool for providing visibility into operating system endpoints. It is a flexible tool developed originally at Facebook. Ganesh Pai is the founder of Uptycs, a company that uses Osquery to find threats and malicious activity occurring across nodes. Ganesh joins the show to talk about Osquery usage and his work on Uptycs. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data science is a collaborative field. Collaboration requires sharing the artifacts that data scientists are working on, such as Jupyter Notebooks and SQL tables. Hex is a platform for improving sharing across data science workflows. Caitlin Colgrove and Barry McCardel are founders of Hex and they join the show to discuss what they have built. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Border Gateway Protocol is a protocol designed for routing and reachability between autonomous systems on the internet. BGPmon is a tool for assessing the routing health of your network, which allows for a network administrator to understand network stability and risk of data. Andree Toonk is the founder of BGPmon and joins the show to talk about BGP, how to monitor routing data, and his work at Cisco. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Business intelligence is crucial for both internal and external applications at any company. There is a wide array of proprietary BI tools. Today, there is an increasing number of options for open source business intelligence, one of which is CubeJS. CubeJS is an open source analytical API platform for building BI. Artyom and Pavel from CubeJS join the show to talk about what they have built and their vision for the platform. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

For several years, we have had the ability to create artificially generated text articles. More recently, audio and video synthesis have been feasible for artificial intelligence. Rosebud is a company that creates animated virtual characters that can speak. Users can generate real or fictional presenters easily with Rosebud. Dzmitry Pletnikau is an engineer with Rosebud and joins the show to talk about the technology and engineering behind the company. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

An instruction set defines a low level programming language for moving information throughout a computer. In the early 1970’s, the prevalent instruction set language used a large vocabulary of different instructions. One justification for a large instruction set was that it would give a programmer more freedom to express the logic of their programs. Many of these instructions were rarely used. Think of your favorite programming language (or your favorite human language). What percentage of words in the vocabulary do you need to communicate effectively? We sometimes call these language features “syntactic sugar”. They add expressivity to a language, but may not improve functionality or efficiency. These extra language features can have a cost. Dave Patterson and John Hennessy created the RISC architecture: Reduced Instruction Set Compiler architecture. RISC proposed reducing the size of the instruction set so that the important instructions could be optimized for. Programs would become more efficient, easier to analyze, and easier to debug. Dave Patterson’s first paper on RISC was rejected. He continued to research the architecture and advocate for it. Eventually RISC became widely accepted, and Dave won a Turing Award together with John Hennessy. Dave joins the show to talk about his work on RISC and his continued work in computer science research to the present. He is involved in the Berkeley RISELab and works at Google on the Tensor Processing Unit. Machine learning is an ocean of new scientific breakthroughs and applications that will change our lives. It was inspiring to hear Dave talk about the changing nature of computing, from cloud computing to security to hardware design. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published July 27, 2018 React Native allows developers to reuse frontend code between mobile platforms. A user interface component written in React Native can be used in both iOS and Android codebases. Since React Native allows for code reuse, this can save time for developers, in contrast to a model where completely separate teams have to create frontend logic for iOS and Android. React Native was created at Facebook. Facebook itself uses React Native for mobile development, and contributes heavily to the open source React Native repository. In 2016, Airbnb started using React Native in a significant portion of their mobile codebase. Over the next two years, Airbnb saw the advantages and the disadvantages of adopting the cross platform, JavaScript based system. After those two years, the engineering management at Airbnb came to the conclusion to stop using React Native. Gabriel Peal is an engineer at Airbnb who was part of the decision to move off of React Native. Gabriel wrote a blog post giving the backstory for React Native at Airbnb, and he joins the show to give more detail on the decision. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

October 1, 2019 The development of self-driving cars is one of the biggest technological changes that is under way. Across the world, thousands of engineers are working on developing self-driving cars. Although it still seems far away, self-driving cars are starting to feel like an inevitability. This is especially true if you spend much time in downtown San Francisco, where you will see a self-driving car being tested every day. Much of the time, that self-driving car will be operated by Cruise. Cruise is a company that is building a self-driving car service. The company has hundreds of engineers working across the stack, from computer vision algorithms to automotive hardware. Cruise’s engineering requires engineers who can work with cloud tools as well as low-latency devices. It also requires product developers and managers to lead these different teams. The field of self-driving is very new. There is not much literature available on how to build a self-driving car. There is even less literature on how to manage a team of engineers that are building, testing, and deploying software and hardware for real cars that are driving around the streets of San Francisco. Mo Elshenawy is VP of engineering at Cruise, and he joins the show to talk about the engineering that is required to develop fully self-driving car technology, as well as how to structure teams to align the roles of product design, software engineering, testing, machine learning, and hardware. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published May 14, 2018 The Kubernetes ecosystem consists of enterprises, vendors, open source projects, and individual engineers. The Cloud Native Computing Foundation was created to balance the interests of all the different groups within the cloud native community. CNCF has similarities to the Linux Foundation and the Apache Foundation. CNCF helps to guide open source projects in the Kubernetes ecosystem--including Prometheus, Fluentd, and Envoy. With the help of the CNCF, these projects can find common ground where possible. KubeCon is a conference organized by the Cloud Native Computing Foundation. I attended the most recent KubeCon in Copenhagen. KubeCon was a remarkably well-run conference--and the attendees were excited and optimistic. As much traction as Kubernetes has, it is still very early days and it was fun to talk to people and forecast what the future might bring. At KubeCon, I sat down with Chris Aniszczyk and Dan Kohn, who are the COO and director of the CNCF. I was curious about how to scale an organization like the CNCF. In some ways, it is like scaling a government. Kubernetes is growing faster than Linux grew, and the applications of Kubernetes are as numerous as those of Linux. Different constituencies want different things out of Kubernetes--and as those constituencies rapidly grow in number, how do you maintain diplomacy among competing interests? It’s not an easy task, and that diplomacy has been established by keeping in mind lessons from previous open source projects. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published January 10, 2020 Slack is a messaging platform for organizations. Since its creation in 2013, Slack has quickly become a core piece of technology used by a wide variety of technology companies, groups, and small teams. The messages that are sent on Slack are generated at a very high volume, and are extremely sensitive. These messages must be stored on Slack’s servers in a way that does not risk a message from one company accidentally being accessible to another company. The messages must be highly available, and they also must be indexed for search. When Slack was scaling, the company started to encounter limitations in its data infrastructure that the company was unsure how to solve. During this time, Josh Wills was the director of data engineering at Slack, and he joins the show to retell the history of his time at Slack, and why the problem of searching messages was so hard. Josh also provides a great deal of industry context around how engineers from Facebook and Google differ from one another. When Slack was starting to become popular, the company quickly began to attract engineers from both of those companies. Facebook and Google have distinct solutions for how they have tackled the problems of data engineering. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

GitHub manages a large API surface for both internal and external developers. This API surface has been migrated from purely RESTful requests to GraphQL. GraphQL is a newer request language for data fetching with fewer round trips. Marc-Andre Giroux works at GitHub and is the author of Production Ready GraphQL. He joins the show to talk about GraphQL across the industry, and specifically at GitHub. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Infrastructure at Spotify runs at high speeds. Developers work autonomously, building and deploying services all the time. Backstage is an open source platform built at Spotify that allows developers to build portals for making sense of their infrastructure. Backstage developer portals are powered by a central service catalog, with centralized services and streamlined development. Stefan Alund joins the show to explain how Backstage works and their role in developing it. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The JavaScript ecosystem has millions of packages. How do you choose from those packages to find the best in breed for your projects? OpenBase is a system for searching and discovering JavaScript packages. OpenBase includes reviews, insights, and statistics around these JavaScript packages. Lior Grossman is a founder of OpenBase, and joins the show to talk about the JavaScript ecosystem and what he is building. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data leaks can cause privacy violations and other cloud security vulnerabilities. Visibility and control of cloud resources can help secure data and ensure compliance and governance. Open Raven is a system for discovering and classifying sensitive data in a public cloud, and assuring compliance and governance. Dave Cole is a founder of Open Raven, and he joins the show to talk through data protection and what he has built with Open Raven. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Banking and money management are at the core of many modern applications. Payment operations teams work to enable the transfer of funds between different bank accounts, and to track the movement of those funds effectively. Modern Treasury is a company that builds payment operations APIs. Sam Aarons works at Modern Treasury and joins the show to talk through the engineering at Modern Treasury. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Internal tools are often built with Ruby on Rails or NodeJS. Developers create entire full-fledged applications in order to suit simple needs such as database lookups, dashboarding, and product refunds. This internal tooling creates a drain on engineering resources. Retool is a low-code platform for creating internal tools. These internal tools can be written by bizops, marketing, or roles other than engineers. David Hsu is the founder of Retool and joins the show to talk through what he has built. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Microservices route requests between each other. As the underlying infrastructure changes, this routing becomes more complex and dynamic. The interaction patterns across this infrastructure requires operators to create rules around traffic management. Tobias Kunze Briseno is the founder of Glasnostic, a system for ensuring resilience of microservice applications. Tobias joins the show to talk about microservice routing and traffic management, and what he has built with Glasnostic. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Netlify is a cloud provider for JAMStack applications. To make those applications more performant, Netlify has built out capacity for edge computing--specifically “edge handlers”. Edge handlers can be used for a variety of use cases that need lower latency or other edge computing functionality. Matt Biilmann Christensen is the CEO of Netlify and joins the show to talk through the engineering behind edge handlers. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

DevOps practices are shared via community, and community manifests at conferences. Unfortunately, conferences are not possible right now due to COVID-19. The world has turned to virtual conferences. All Day DevOps is a 24 hour conference sharing learnings and software strategies around DevOps, starting November 12th. Derek Weeks and Mark Miller are organizers of the conference and they join the show to talk about modern DevOps. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Static analysis allows for the discovery of issues in a codebase without compiling. There have been many generations of static analysis tools. A newer static analysis tool is DeepSource, which automates code reviews, identifies bug risks, and generates pull requests to fix them. Jai Pradeesh and Sanket Saurav are founders of DeepSource, and join the show to talk through the creation of static analysis tooling, and their work on DeepSource. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data science requires data sets to be cataloged and indexed. The data sets are versioned and might be in CSV files in S3, a database, or another data storage system. Splitgraph allows the user to query this data catalog like it is a Postgres database, routing queries to any data set across your catalog. Artjoms Iškovs and Miles Richardson are the founders of Splitgraph and join the show to talk about data cataloging and what they are building with Splitgraph. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Sendbird is a company that makes chat, voice, and video APIs for developers. The biggest company in this category is arguably Twilio, but Sendbird works at a higher level of abstraction, with an emphasis on developer experience and visual components. John Kim is the CEO of Sendbird and he joins the show to discuss the engineering and competitive positioning of his company. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Newer machine learning tooling is often focused on streamlining the workflows and developer experience. One such tool is BentoML. BentoML is a workflow that allows data scientists and developers to ship models more effectively. Chaoyu Yang is the creator of BentoML and he joins the show to talk about why he created Bento and the engineering behind the project. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data labeling is a major bottleneck in training and deploying machine learning and especially NLP. But new tools for training models with humans in the loop can drastically reduce how much data is required. Humanloop is a platform for annotating text and training NLP models with much less labelled data. Raza Habib, founder of Humanloop, joins the show to to talk about NLP workflows and his work on Humanloop. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A customer data platform such as Segment allows developers to build analytics and workflows around customer data such as purchases, clicks, and other interactions. These customer data platforms (CDP) are often tightly coupled to an underlying data warehouse technology. Hightouch is a platform that provides an unbundled CDP--a platform that sits on top of your own data warehouse. The Hightouch team joins the show to talk about what they are building and the CDP ecosystem as a whole. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Fivetran is a company that builds data integration infrastructure. If your company is performing ELT or ETL jobs to move data from one place to another, Fivetran can help with that movement from source to destination. Once the data is moved into a data warehouse, a tool called DBT (data build tool) can be used to transform the data more effectively. We have done shows previously about Fivetran and DBT. In today’s episode, George Fraser of Fivetran returns to discuss the cross section of these two technologies, and what his company is doing around that integration point. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Staff engineer is a job title that suggests the engineer has deep expertise, and considerable experience. More and more companies are adopting a “staff engineer track” where an engineer can work to become a staff engineer. What is the role of staff engineer? Is it a management role or an individual contributor? What are the expectations and obligations of staff engineer? Will Larson is an experienced engineer who has worked at Stripe and other prominent tech companies. He joins the show to talk about the role of staff engineering, and the material he has written about it. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Salesforce Ecosystem has thousands of developers, designers, product people, and entrepreneurs engaging with each other. Salesforce exposes APIs and SDKs that allow people to build infrastructure on top of the Salesforce platform. In a previous episode, we explored how the ecosystem works as a whole. In today’s show, Chuck Liddell joins the show to talk about how developers themselves engage with Salesforce. Chuck is CEO of Valence, a Salesforce AppExchange ISV that adds native integration middleware to Salesforce. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Pair programming allows developers to partner on solving problems and learn from each other more effectively. Pair programming has become harder to do as remote work has become more prevalent. GitDuck is a tool to enable more effective pair programming. Dragos Fotescu and Thiago Monteiro are the founders of GitDuck, and they join the show to explain what they have built and their motivation behind it. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Effective data science requires clean data. As data moves through the data pipeline, there may be errors introduced. Errors can also arise from code changes, database migrations, and other forms of data movement. How can you ensure data quality within a fast moving, dynamic data system? Datafold is a company built around data quality management. It allows users to compare tables and databases, as well as automate data QA. Gleb Mezhanskiy is a founder of Datafold and joins the show to talk about the data quality space and what he is building with Datafold. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Federated learning is machine learning without a centralized data source. Federated Learning enables mobile phones or edge servers to collaboratively learn a shared prediction model while keeping all the training data on device. Mike Lee Williams is an expert in federated learning, and he joins the show to give an overview of the subject and share his thoughts on its applications. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

For all the advances in software development over the years, one area that has seen minimal improvement is the terminal. Typing commands into a black text interface seems antiquated compared to the dynamic, flashy interfaces available in web browsers and modern desktop applications. Fig is a visual terminal assistant with the goal of changing that. Fig sits next to the developer’s normal terminal and enhances the terminal experience. The founders of Fig, Brendan Falk and Matt Schrage, join the show today to discuss how Fig works and why it is useful to have an enhanced terminal. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud resources can get out of control if proper management constraints are not put in place. Cloud Custodian enables users to be well managed in the cloud. It is a YAML DSL that allows you to easily define rules to enable a well-managed cloud infrastructure giving security and cost optimization. Kapil Thangavelu works on Cloud Custodian and he joins the show to talk about modern cloud management and what he is building with Cloud Custodian. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Predicting the spread of COVID-19 is not easy. The best methods we have available require us to extrapolate trends from a large volume of data, and this requires the construction of large-scale models. Because of the expertise needed for developing these models, Silicon Valley engineers were brought in to help develop a maintainable model. Two of these engineers are Josh Wills and Sam Shah, and they join the show to talk about the engineering behind the COVID model, and their work to build it. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning models require training data, and training data needs to be labeled. Raw images and text can be labeled using a training data platform like Labelbox. Labelbox is a system of labeling tools that enables a human workforce to create data that is ready to be consumed by machine learning training algorithms. The Labelbox team joins the show today to discuss training data and how to label it. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Containers and virtual machines are two ways of running virtualized infrastructure. Containers use less resources than VMs, and typically use the runc open source container runtime. Sysbox is a containerization runtime that offers an alternative to runc, and allows for the deployment of Docker or Kubernetes within a container. Cesar Talledo is the founder of Nestybox, a company built around the Sysbox runtime. He joins the show to talk about container runtimes and his new company. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Firebase is well-known as a platform that makes it easy to build real-time applications quickly and easily. Firebase was acquired by Google, and has been turned into a large platform that runs on top of Google Cloud. Firebase is closed-source, which leads to a different ecosystem than open source platforms. Supabase is a new open source alternative to Firebase, built on Postgres and Elixir. Paul Copplestone is the founder of Supabase and he joins the show to talk through what he is building. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Development environments are brittle and hard to manage. They lack the kind of fungibility afforded by infrastructure-as-code. Gitpod is a company that allows developers to describe development environments as code to make them easier to work with, and enabling a more streamlined GitOps workflow. Johannes Landgraf and Sven Efftinge are creators of Gitpod and they join the show to discuss the product and the motivation for building it. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Training a computer vision model is not easy. Bottlenecks in the development process make it even harder. Ad hoc code, inconsistent data sets, and other workflow issues hamper the ability to streamline models. Roboflow is a company built to simplify and streamline these model training workflows. Brad Dwyer is a founder of Roboflow and joins the show to talk about model development and his company. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Databases are the source of truth for every company. Editing the data in the database normally requires writing a query in SQL or a domain specific querying language–languages that are only accessible to engineers and highly technical people. BaseDash is a tool for interfacing with a database without requiring the usage of a query language. It allows the user to interface with the database as easily as a spreadsheet. Max Musing is a founder of BaseDash, and he joins the show to talk about how it works and why he built it. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning models are only as good as the datasets they’re trained on. Aquarium is a system that helps machine learning teams make better models by improving their dataset quality. Model improvement is often made by curating high quality datasets, and Aquarium helps make that a reality. Peter Gao works on Aquarium, and he joins the show to talk through modern machine learning and the role of Aquarium. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ray is a general purpose distributed computing framework. Ray is used for reinforcement learning and other compute intensive tasks. It was developed at the Berkeley RISELab, a research and development lab with an emphasis on practical applications. Ion Stoica is a professor at Berkeley, and he joins the show to talk about the present and future of the Ray framework. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A private network connects servers, computers, and cloud instances. These networked objects are often separated by firewalls and subnets that create latency and complication. David Crawshaw is the CTO of Tailscale, a company that works to make private networks easier to build and simpler to configure and maintain. David joins the show to talk about private networks and the implementation of Tailscale. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Pachyderm is a system for data version control. Code has been version controlled for many years, but not data. In previous episodes with Joe Doliner, we explored the evolution of Pachyderm. In today’s show, we talk about the state of the company in 2020, as well as Pachyderm Hub, and end-to-end machine learning and data lineage product. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Deno is a runtime for JavaScript applications. Deno is written in Rust, which changes the security properties of it. Parts of Deno are also written in TypeScript, which are causing problems in the compilation and organization of Deno. Elio Rivero is an engineer who has studied Deno and TypeScript, and he joins the show to talk about the newer JavaScript runtime and the issues caused by TypeScript. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Developer tooling and infrastructure is a fruitful area for investing. A wide variety of technologies can have large investment outcomes based on the fact that there are lots of engineers and businesses are willing to pay for products that give those engineers a higher degree of leverage. Lee Edwards is a partner with Root Ventures. His focus is on hard problems within software, and he joins the show to talk about the thesis of his firm, as well as his personal beliefs on what makes a good investment. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Salesforce is a platform with a large number of developers, ISVs, and companies built on top of it. There is a thriving ecosystem of applications built and managed around Salesforce, leading to an important set of relationships and integration points between Salesforce and the other entities involved with the company. Kevin Poorman works at Salesforce as a developer evangelist, helping to strengthen the relationships in the Salesforce ecosystem. Kevin joins the show to talk about Salesforce and the applications that connect to it. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Twitter is a social media platform with billions of objects: people, tweets, words, events, and other entities. The high volume of information that gets created on Twitter everyday leads to a complex engineering problem for the developers building the Twitter search index. Nico Tonozzi is an engineer at Twitter. He joins the show to talk through the problem space of search at Twitter, as well as some recent challenges that he had to tackle in the continuously changing Twitter product. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Robinhood is a platform for buying and selling stocks and cryptocurrencies. Robinhood is complex, fast-moving, and financial, and together these things require high quality engineering in distributed systems, observability, and data infrastructure. Jaren Glover is an engineer at Robinhood, and he joins the show to talk about the problem space within Robinhood, as well as the specific DevOps and software engineering challenges. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Java ecosystem is maturing. The GraalVM high performance runtime provides a virtual machine for running applications in a variety of languages. TornadoVM extends the Graal compiler with a new backend for OpenCL. TornadoVM allows the offloading of JVM applications onto heterogeneous hardware. Juan Fumero works on TornadoVM. He joins the show to talk about the use case for TornadoVM, the design, and the engineering that underlies the system. We also talk about the overall Java ecosystem. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Investing in enterprise software has become a competitive business. Lots of venture capital firms compete for the good deals at every stage. This level of competition has driven more capital into the early stages. Ed Sim is a partner with Boldstart, an early stage enterprise investment firm. He joins the show to talk about modern enterprise investment strategy and his own varied personal experiences in working at funds. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Factories require quality assurance work. That QA work can be accomplished by a robot with a camera together with computer vision. This allows for sophisticated inspection techniques that do not require as much manual effort on the part of a human. Arye Barnehama is a founder of Elementary Robotics, a company that makes these kinds of robots. Arye joins the show to talk through the engineering of Elementary Robotics, and his vision for the future of the factory floor. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The most popular email client is Gmail, the web-based email client from Google. Gmail is dominant, but that dominance has come at a price, namely speed. Gmail caters to the lowest common denominator, serving a large ecosystem of use cases and plugins. This makes for a slow overall performance. Superhuman is an email client built for power users. Rahul Vohra is the founder of Superhuman, and joins the show to talk about the design and engineering of an email client that is made to be fast. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Internet Archive collects historical records of the Internet. The Wayback Machine is one tool from the Internet Archive which you may be familiar with. One project you may be unfamiliar with is book scanning. Internet Archive scans high volumes of books in order to digitize them. In today’s episode, Davide Semenzin joins the show to talk through the history of the Internet Archive and the engineering behind book digitization. We talk through OCR, storage, architecture, and scalability. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Biometric authentication uses signals from a human’s unique biology to verify identity. Forms of biometric authentication include fingerprints, eye patterns, and the way a person walks, otherwise known as gait. UnifyID is a company that builds systems for biometric authentication. John Whaley is the CEO of UnifyID, and he joins the show to talk through techniques for biometrics, and the implementation details that UnifyID has built to turn these into a reality. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Robotic process automation involves the scripting and automation of highly repeatable tasks. RPA tools such as UIPath paved the way for a newer wave of automation, including the Robot Framework, an open source system for RPA. Antti Karjalainen is the CEO of Robocorp, a company that provides an RPA tool suite for developers. Antti joins the show to talk through the definition of RPA, common RPA tasks, and what he is building with Robocorp. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

After working at VMware for 10 years, Jerry Chen developed an expertise in technology companies. Today, he works at Greylock, where he looks at deals in the infrastructure and developer tooling space. Jerry is an expert in go-to-market strategy and makes investments in technologies that have a good chance at becoming large and profitable businesses. In today’s episode, Jerry and I talk through the dynamics of modern infrastructure investing, including examples of deals such as Chronosphere and Rockset, both of which have been featured in previous episodes of the podcast. Jerry gives his perspective on deal terms, board dynamics, and everything else that goes into a smart investment. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

APIs within a company change all the time. Every service owner has an API to manage, and those APIs have upstream and downstream connections. APIs need to be tested for integration points as well as for their “contract”, the agreement between an API owner and the consumers of that API. Aidan Cuniffe is the founder of Optic, a product built for API change management. He joins the show to explain why there is an opportunity for such a product, and the market dynamics of the space of API testing and change management. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

WebAssembly allows for the execution of languages other than JavaScript in a browser-based environment. But WebAssembly is still not widely used outside of a few particular niches such as Dropbox and Figma. Nicolo Davis works on an application called Boardgame Lab, and he joins the show to explain why WebAssembly can be useful even for a simple application. Nicolo also shares his reflections on TypeScript, Rust, and the future of web development. He talks through the client/server interaction, performance, error handling, and the process of an actual migration. Sponsorship inquiries: sponsor@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Hyperparameters define the strategy for exploring a space in which a machine learning model is being developed. Whereas the parameters of a machine learning model are the actual data coming into a system, the hyperparameters define how those data points are fed into the training process for building a model to be used by an end consumer. A different set of hyperparameters will yield a different model. Thus, it is important to try different hyperparameter configurations to see which models end up performing better for a given application. Hyperparameter tuning is an art and a science. Richard Liaw is an engineer and researcher, and the creator of Tune, a library for scalable hyperparameter tuning. Richard joins the show to talk through hyperparameters and the software that he has built for tuning them. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Anduril is a technology defense company with a focus on drones, computer vision, and other problems related to national security. It is a full-stack company that builds its own hardware and software, which leads to a great many interesting questions about cloud services, engineering workflows, and management. Gokul Subramanian is an engineer at Anduril, and he joins the show to share his knowledge of how Anduril operates and what the company has built. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

CrowdFlower was a company started in 2007 by Lukas Biewald, an entrepreneur and computer scientist. CrowdFlower solved some of the data labeling problems that were not being solved by Amazon Mechanical Turk. A decade after starting CrowdFlower, the company was sold for several hundred million dollars. Today, data labeling has only grown in volume and scope. But Lukas has moved on to a different part of the machine learning stack: tooling for hyperparameter search and machine learning monitoring. Lukas Biewald joins the show to talk about the problems he was solving with CrowdFlower, the solutions that he developed as part of that company, and the efforts with his current focus: Weights and Biases, a machine learning tooling company. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As software permeates our lives, there are an increased number of situations where the legal system must be designed to account for that software. Whether the issues are open source licensing, cryptocurrencies, or worker classifications, software overlaps heavily with the law. Just as software is crafted by engineers, the legal structure around software is crafted by lawyers. There are large law firms that have built their business by knowing how to navigate these software and business questions. Mark Radcliffe is a lawyer who has been working with software companies for decades. He joins the show to talk about the intersection of software and the law, which we discuss from multiple points of view. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Code is version controlled through git, the version control system originally built to manage the Linux codebase. For decades, software has been developed using git for version control. More recently, data engineering has become an unavoidable facet of software development. It is reasonable to ask–why are we not version controlling our data? Dmitry Petrov is the founder of Iterative.ai, a company for collaborating and version controlling data sets. Dmitry joins the show to talk about how data version control works, and Iterative.ai, the company he is building around dataset management and collaboration. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every software company works off of several different development environments–at the very least there is staging, testing, and production. Every push to staging can be spun up as an application to be explored, tinkered with, and tested. These ad hoc spin-ups are known as release apps. A release app is an environment for engineers to play with, and potentially throw away or promote to production. Release apps have been made easier due to technologies such as infrastructure-as-code, continuous integration, and Kubernetes. Tommy McClung is the co-founder of Release App, a company that makes it easy to spin up release environments for your software. Tommy joins the show to discuss release workflows, and his work building Release App. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Chatbots are useful for developing well-defined applications such as first-contact customer support, sales, and troubleshooting. But the potential for chatbots is so much greater. Over the last five years, there have been numerous platforms that have arisen to allow for better, more streamlined chatbot creation. Dialogue software enables the creation of sophisticated chatbots. ParlAI is a dialogue platform built inside of Facebook. It allows for the development of dialogue models within Facebook. These chatbots can “remember” information from session to session, and continually learn from user input. Stephen Roller is an engineer who helped build ParlAI, and he joins the show to discuss the history of chatbot applications and what the Facebook team is trying to accomplish with the development of ParlAI. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Image annotation is necessary for building supervised learning models for computer vision. An image annotation platform streamlines the annotation of these images. Well-known annotation platforms include Scale AI, Amazon Mechanical Turk, and Crowdflower. There are also large consulting-like companies that will annotate images in bulk for you. If you have an application that requires lots of annotation, such as self-driving cars, then you might be compelled to outsource this annotation to such a company. SuperAnnotate is an image annotation platform that can be used by these image annotation outsourcing firms. This episode explores SuperAnnotate, and the growing niche of image annotation. Vahan and Tigran Petrosyan are the founders of SuperAnnotate, and join the show for today’s interview. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Business intelligence tooling allows analysts to see large quantities of data presented to them in a flexible interface including charts, graphs, and other visualizations. BI tools have been around for decades, and as the world moves towards increased open source software, the business intelligence tools are following that trend. Metabase is an open source business intelligence system that has been widely adopted by enterprises. It includes all the common tools that are expected from a business intelligence system: large-scale data ingestion, visualization software, and a flexible user interface. Sameer Al-Sakran is the CEO of Metabase and he joins the show to talk about Metabase’s design, engineering, and usage. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The US Army Cyber School is a training program which trains cyber soldiers and leaders to be adept in cyber military strategy and tactics. In order to teach these skills, the cyber school uses a system they call “courseware as code”, a workflow that allows updates to the curriculum in a reversion-friendly fashion similar to infrastructure-as-code. Ben Allison teaches at the US Army Cyber School and has put work into developing the training program and ongoing lesson plans. Ben joins the show to talk about how the US Army manages curriculum through courseware as code, and the work he has done to improve this workflow over time. Ben is also speaking at GitLab Commit 2020, GitLab’s upcoming conference. You can register for GitLab Commit yourself by going to softwareengineeringdaily.com/gitlabcommit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Logs are the source of truth. If a company is sufficiently instrumented, the logging data that streams off of the internal infrastructure can be refined to tell a comprehensive story for what is changing across that infrastructure in real time. This includes logins, permissions changes, other events that could signal a potential security compromise. Datadog is a company that was built around log management, metrics storage, and distributed tracing. More recently, they have also built tools for monitoring the security of an organization. Detecting security threats can be achieved by alerting on known security risks, or pieces of information that could be indicative of a vulnerability. Marc Tremsal works at Datadog, and joins the show to talk through security monitoring. Full disclosure: Datadog is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Dev.to has become one of the most popular places for developers to write about engineering, programming languages, and everyday life. For those who have not seen it, DEV is like a cross between Twitter and Medium, but targeted at developers. The content on DEV ranges from serious to humorous to technically useful. DEV contains a set of features which appeal to a developer community, such as the ability to embed code snippets in a post, but for the most part the entire app is generalizable to other types of communities. Hence, the motivation for “Forem”. Forem is an open source project to make it possible to spin up instances of communities that are like DEV, but for other communities such as mixed martial arts, or doctors. Ben Halpern is the creator of DEV and Forem, and he joins the show to talk about the DEV Community and his long-term goals for what the DEV team is building. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Drug trials can lead to new therapeutics and preventative medications being discovered and placed on the market. Unfortunately, these drug trials typically require animal testing. This means animals are killed or harmed as a result of needing to verify that a drug will not kill humans. Animal testing is unavoidable, but the extent to which testing needs to occur can be reduced by inserting machine learning models which simulate the effects of a drug on the human body. If the simulated effect is negative enough, animal testing doesn’t need to be run, thus no animals need to be harmed. Bryan Vicknair and Jason Walsh work at VeriSIM Life, a company which makes software simulations of animals. These simulations can be used to model drug testing, and change the workflow for drug trials. They join the show to talk through the mechanics of drug testing, and how VeriSIM Life fits into that workflow. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Across a company, there is a wide range of resources that employees need access to. Documents, S3 buckets, git repositories, and many others. As access to resources changes across the organization, a history of the changes to permissions can be useful for compliance and monitoring. Indent is a system for simplifying access management across infrastructure. Indent allows users within an organization to request access to resources, and keeps logs of the changes to who can access those resources. Fouad Matin and Dan Gillespie are the founders of Indent, and they join the show to talk through the application of access control management, and the architecture of Indent itself, which has numerous interesting engineering decisions within it. Indent job opportunities Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Acquisitions are part of the technology industry. A successful corporation will often have an “exit”, either going public or becoming acquired. And with each of these corporations, there is a set of stories that narrate the company from beginning to end. Acquired is a podcast that tells the stories of companies such as YouTube, Instagram, and PayPal. During each episode, the life of a company is explored from its beginning til the end. Media companies, chip companies, and software companies all take the center stage on various episodes. David Rosenthal and Ben Gilbert are the hosts of Acquired, and they join today’s show to talk about the podcast they started, a few business stories, and the podcast industry itself. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ray is a general purpose distributed computing framework. At a low level, Ray provides fault-tolerant primitives that support applications running across multiple processors. At a higher level, Ray supports scalable reinforcement learning, including the common problem of hyperparameter tuning. In a previous episode, we explored the primitives of Ray as well as Anyscale, the business built around Ray and reinforcement learning. In today’s episode, Richard Liaw explores some of the libraries and applications that sit on top of Ray. RLlib gives APIs for reinforcement learning such as policy serving and multi-agent environments. Tune gives developers an easy way to do scalable hyperparameter tuning, which is necessary for exploring different types of deep learning configurations. In a future show, we will explore Tune in more detail. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Pandas is a Python data analysis library, and an essential tool in data science. Pandas allows users to load large quantities of data into a data structure called a dataframe, over which the user can call mathematical operations. When the data fits entirely into memory this works well, but sometimes there is too much data for a single box. The Modin project scales Pandas workflows to multiple machines by utilizing Dask or Ray, which are distributed computing primitives for Python programs. Modin builds an execution plan for large data frames to be operated on against each other, which makes data science considerably easier for these large data sets. Devin Petersohn started the Modin project, and he joins the show to talk about data science with Python, and his work in the Berkeley RISELab. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A large codebase cannot be searched with naive indexing algorithms. In order to search through a codebase the size of Uber’s it is necessary to build a much more sophisticated indexing system than simple pure text search. SourceGraph is a system for universal code search. It allows developers to more easily onboard to a new codebase, make large refactors, and perform other tasks. SourceGraph can integrate with source control systems, IDEs, and other tools to fit comfortably into an engineer’s workflow. Beyang Liu is a co-founder of SourceGraph and he joins the show to talk about how codebases can become large and unwieldy, and the tooling that SourceGraph offers to make these codebases easier to work with. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Users do not use web applications in the way that you might expect. And it is not easy to get the data that is necessary to get a full picture. But a newer API within browsers does make this more possible by capturing DOM mutations. The change capture of these DOM mutations can be stored for replay in the future. After being stored, this change capture can be retrieved and replayed. That allows for comprehensive frontend monitoring, which has been built into a product called FullStory. Michael Morrissey is the CTO of FullStory, and he joins the show to talk about how session capture works, and the architecture of FullStory–how sessions get saved, stored and retrieved. In a previous show we talked about LogRocket, a product which does something similar. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Managing microservices becomes a challenge as the number of services within the organization grows. With that many services comes more interdependencies–downstream and upstream services that may be impacted by an update to your service. One solution to this problem: a dashboard and newsfeed system that lets you see into the health and changes across your services. With this kind of system, you can avoid accidentally shipping code that will impact other service owners. It can also help with testing, giving you an end-to-end picture for how a test can impact other services. Anish Dhar and Ganesh Datta are co-founders of Cortex, a system for managing your services. Anish and Ganesh join the show to talk about their work building Cortex, and the value that it provides to the companies that use it. In a previous show we covered a company called Effx, which does something similar. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

ADP has been around for more than 70 years, fulfilling payroll and other human resources services. Payroll processing is a complex business, involving the movement of money in accordance with regulatory and legal strictures. From an engineering point of view, ADP has decades of software behind it, and a bright future of a platform company used by thousands of companies. Balancing the maintenance of old code while charting a course with the new projects is not a simple task. Tim Halbur is the CTO of ADP, and he joins the show to talk through how engineering works at ADP, and how the organization builds for the future of the company while maintaining the code of the past. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software companies can be funded in a variety of ways: venture capital, self-funding, and debt, among others. In order to receive financing, a company is evaluated on its ability to generate future cash flows. After all, a valuation is a number that summarizes the present value of future cash flows. Determining that valuation number is a complicated, subjective process. If the valuation can be determined more intelligently and objectively, then smarter financing decisions can be made. This is the reasoning behind the company Capital, which aims to build a better modeling system for evaluating companies. Blair Silverberg and Chris Olivares are founders of Capital, and they join the show to explore the modeling process for valuations, and their strategy for doing this with their software models. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

GitHub has been a social network for developers for many years. Most social networks are centered around mobile applications, but GitHub sits squarely in a developer’s browser-based desktop workflow. As a result, the design of a mobile app for GitHub is less straightforward. GitHub did acquire a popular mobile client called GitHawk, which was developed by Ryan Nystrom. Since joining GitHub, Ryan has worked on a new mobile app for GitHub, along with a team of engineers including Brian Lovin. Ryan and Brian both join the show to discuss GitHub mobile, and how they designed, architected, and built the app. There is no company quite like GitHub–a social network combined with a version control system that provides a critical utility. All this made for an interesting episode about a one-of-a-kind mobile product. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A service mesh provides routing, load balancing, policy management, and other features to a set of services that need to communicate with each other. The mesh can simplify operations across these different services by providing an interface to configure them. There are lots of different vendors who offer service mesh technology: AWS has AppMesh, Google has Istio (which is open source), Buoyant has Linkerd (which is also open source), and HashiCorp has Consul Connect. Unfortunately, these service meshes do not all play well together. And at a large enough company, different teams will be setting up different service meshes. So it would be useful for services in those different meshes to be able to communicate with each other. Luke Kysow is an engineer at HashiCorp where he works on Consul Connect, and he joins the show to discuss service mesh usage, adoption, and possible strategies for maintaining multiple service meshes within a single organization. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Netflix runs all of its infrastructure on Amazon Web Services. This includes business logic, data infrastructure, and machine learning. By tightly coupling itself to AWS, Netflix has been able to move faster and have strong defaults about engineering decisions. And today, AWS has such an expanse of services that it can be used as a platform to build custom tools. Metaflow is an open source machine learning platform built on top of AWS that allows engineers at Netflix to build directed acyclic graphs for training models. These DAGs get deployed to AWS as Step Functions, a serverless orchestration platform. Savin Goyal is a machine learning engineer with Netflix, and he joins the show to talk about the machine learning challenges within Netflix, and his experience working on Metaflow. We also talk about DAG systems such as AWS Step Functions and Airflow. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

WordPress has been a dominant force in the world of online publishing for many years because of how battle-tested it is. WordPress is the definitive leader in CMS technology. But there have always been alternatives. Drupal, Ghost, and other open source CMSes. More recently, there has been an emergence of the headless CMS, such as Contentful, which decouples the CMS backend from the frontend presentation layer. Strapi is a popular open source headless CMS. Pierre Burgy is the founder of Strapi, and he joins the show to talk about the CMS category, the role that Strapi fills, and the technology behind Strapi. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

M3 is a scalable metrics database originally built to host Uber’s rapidly growing data storage from Prometheus. When Rob Skillington was at Uber, he helped design, implement, and deploy M3. Since leaving Uber, he has co-founded a company around a hosted version of M3 called Chronosphere. If you have access to a scalable metrics database, you might as well start accumulating as much data as possible, right? Not exactly. If your company generates enough data, you probably want to turn down the dials on how frequently you save a metric. Downsampling will reduce the amount of money that you pay for these hosted metrics. In today’s show, Rob discusses the engineering and deployment of M3, and how that work led him to founding Chronosphere, as well as the product offering of the company. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Developing machine learning models is not easy. From the perspective of the machine learning researcher, there is the iterative process of tuning hyperparameters and selecting relevant features. From the perspective of the operations engineer, there is a handoff from development to production, and the management of GPU clusters to parallelize model training. In the last five years, machine learning has become easier to use thanks to point solutions. TensorFlow, cloud provider tools, Spark, Jupyter Notebooks. But every company works differently, and there are few hard and fast rules for the workflows around machine learning operations. Determined AI is a platform that provides a means for collaborating around data prep, model development and training, and model deployment. Neil Conway is a co-founder of Determined, and he joins the show to discuss the challenges around machine learning operations, and what he has built with Determined. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

AWS has over 150 different services. Databases, log management, edge computing, and lots of others. Instead of being overwhelmed by all of these products, an engineering team can simplify their workflow by focusing on a small subset of AWS services–the defaults. Daniel Vassalo is the author of The Good Parts of AWS. An excerpt from the book: “The cost of acquiring new information is high and the consequence of deviating from a default choice is low, so sticking with the default will likely be the optimal choice. A default choice is any option that gives you very high confidence that it will work.” Having confidence in your workflow–even if it is a simple workflow–has advantages. S3, EC2, Elastic Load Balancers: for simple web applications, this is really all you need to build your business. Daniel Vassallo worked at AWS for more than 8 years before leaving to become an entrepreneur and author. He joins the show to talk about what the good parts of AWS are, and his strategy for building applications with that subset of services. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The modern release workflow involves multiple stakeholders: engineers, management, designers, and product managers. It is a collaborative process that is often held together with brittle workflows. A developer deploys a new build to an ad hoc staging environment and pastes a link to that environment in Slack. Other stakeholders click on that link, then send messages to each other in Slack, or make comments on the pull request in GitHub. This workflow is far from ideal. Collaborating around pull requests can be made easier with a dedicated set of tools for sharing and discussing those pull requests. This is the goal of FeaturePeek, a system for spinning up dedicated pull request environments, creating screenshots and comments, and reimagining the lifecycle of the release workflow. Eric Silverman is a co-founder of FeaturePeek and he joins the show to discuss release management, the interactions between different stakeholders, and the development of his company. Much like the previous show about Postman, in which we explored how API management has become a ripe space for collaboration, the same is true of pull requests. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Deepgram is an end-to-end deep learning platform for speech recognition. Unlike the general purpose APIs from Google or Amazon, Deepgram models are custom-trained for each customer. Whether the customer is a call center, a podcasting company, or a sales department, Deepgram can work with them to build something specific to their use case. Sound data is incredibly rich. Consider all the features in a voice recording: volume, intonation, inflection. And once the speech is transcribed, there are many more features that can be discovered from the text transcription. Scott Stephenson is the CEO of Deepgram, and he joins the show to talk through end-to-end deep learning for speech, as well as the dynamics of the business and the deployment strategy for working with customers. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

DynamoDB is a managed NoSQL database service from AWS. It is widely used as a transactional database to fulfill key-value and wide-column data models. In a previous show with Rick Houlihan, we explored how to build a data model and optimize the query patterns for a NoSQL database. Today’s show is about DynamoDB specifically: partitioning, indexing, query semantics, normalization, table design, and other subjects. We talk through how to be cost conscious, and how to integrate with event-based AWS Lambda triggers. Alex DeBrie is the author of The DynamoDB Book, a book whose title speaks for itself. Alex has comprehensive experience with DynamoDB, and he joins the show to share that experience through a detailed discussion of use cases and strategies related to DynamoDB. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As a user browses a webpage, that browser session generates events that need to be recorded, validated, enriched, and stored. This data is sometimes called customer data infrastructure, or CDI. This data requires a full stack of different tools: a system on the frontend to collect the data, middleware to transport the data, and backend systems for storing and loading that data into data warehouses and other analytical systems. Snowplow Analytics is a data collection platform for storing events. In Snowplow, modules called Trackers send data to Collectors. The data can then be validated and enriched, and then put into the user’s data warehouse via ETL. Alex Dean is the CEO of Snowplow, and he joins the show to talk through the business model, management, and engineering of Snowplow Analytics, as well as the overall data engineering landscape.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A software company manages and interacts with hundreds of APIs. These APIs require testing, performance analysis, authorization management, and release management. In a word, APIs require collaboration. Postman is a system for API collaboration. It allows users to test APIs with collections of requests, monitor the API responses, and visualize the query results. Users of Postman can collaborate with their team through Team Workspaces, sharing collections, environments, history, and more. Abhinav Asthana is the founder of Postman and he joins the show to talk about API management and collaboration. Abhinav started Postman as a hobby project, and it has grown into a large and successful business, far beyond the original product of API testing.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At a customer service center, thousands of hours of audio are generated. This audio provides a wealth of information to transcribe and analyze. With the additional data of the most successful customer service representatives, machine learning models can be trained to identify which speech patterns are associated with a successful worker. By identifying these speaking patterns, a customer service center can continuously improve, with the different representatives learning the different patterns. The same is true for other speech-based tasks, such as sales calls. Cresta is a company that builds systems to ingest high volumes of speech data in order to discover features that correlate with high performance human workers. Zayd Enam is a co-founder of Cresta, and joins the show to talk about the domain of speech data and what he and his team are building at Cresta.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published July 6, 2017. We are taking a few weeks off. We’ll be back soon with new episodes. React Native allows developers to reuse components from one user interface on multiple platforms. React Native was introduced by Facebook to reduce the pain of teams who were rewriting their user interfaces for web, iOS, and Android. Nader Dabit hosts React Native Radio, a podcast about React Native. Nader also trains companies to use React Native through his company React Native Training. In this episode, we explore what a developer can and cannot do with React Native, when a developer needs to use native APIs, and some speculation on the future of React Native.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published October 8, 2019. We are taking a few weeks off. We’ll be back soon with new episodes. Video surveillance impacts human lives every day. On most days, we do not feel the impact of video surveillance. But the effects of video surveillance have tremendous potential. It can be used to solve crimes and find missing children. It can be used to intimidate journalists and empower dictators. Like any piece of technology, video surveillance can be used for good or evil. Video recognition lets us make better use of video feeds. A stream of raw video doesn’t provide much utility if we can’t easily model its contents. Without video recognition, we must have a human sitting in front of the video to manually understand what is going on in that video. Veronica Yurchuk and Kosh Shysh are the founders of Traces.ai, a company building video recognition technology focused on safety, anonymity, and positive usage. They join the show to discuss the field of video analysis, and their vision for how video will shape our lives in the future.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published July 25, 2019. We are taking a few weeks off. We’ll be back soon with new episodes. Envoy is an open source edge and service proxy that was originally developed at Lyft. Envoy is often deployed as a sidecar application that runs alongside a service and helps that service by providing features such as routing, rate limiting, telemetry, and security policy. Envoy has gained significant traction in the open source community, and has formed the backbone of popular service mesh projects such as Istio. Envoy has been mostly used as a backend technology, but the potential applications of Envoy include frontend client applications as well. The goal of Envoy is to make the network easier to work with–and the network includes client applications such as mobile apps running on a phone. Envoy Mobile is a network proxy for mobile applications. Envoy Mobile brings many of the benefits of Envoy to the mobile client ecosystem. It provides mobile developers with a library that can simplify or abstract away many of the modern advances that have been made in networking in recent years, such as HTTP2, gRPC, and QUIC. Matt Klein is the creator of Envoy, and he joins the show to discuss Envoy Mobile. Matt describes how the networking challenges of mobile applications are similar to those of backend systems and cloud infrastructure. We discuss the advances in networking technology that Envoy Mobile helps bring to the mobile ecosystem, and also touch on the scalability challenges that Matt is seeing at Lyft.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published May 2, 2017. We are taking a few weeks off. We’ll be back soon with new episodes. A new programmer learns to build applications using data structures like a queue, a cache, or a database. Modern cloud applications are built using more sophisticated tools like Redis, Kafka, or Amazon S3. These tools do multiple things well, and often have overlapping functionality. Application architecture becomes less straightforward. The applications we are building today are data-intensive rather than compute-intensive. Netflix needs to know how to store and cache large video files, and stream them to users quickly. Twitter needs to update user news feeds with a fanout of the president’s latest tweet. These operations are simple with small amounts of data, but become complicated with a high volume of users. Martin Kleppmann is the author of Data Intensive Applications, an O’Reilly book about how to use modern data tools to solve modern data problems. His book includes high-level discussions about architectural strategy, and lower level discussions like how leader election algorithms can create problems for a data intensive application.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published December 20, 2019. We are taking a few weeks off. We’ll be back soon with new episodes. freeCodeCamp was started five years ago with the goal of providing free coding education to anyone on the Internet. freeCodeCamp has become the best place to begin learning how to write software. There are many other places that a software engineer should visit on their educational journey, but freeCodeCamp is the best place to start, because it is free, and there are no advertisements. For most people learning to code, the price of that education is important, because they are learning to code to build a new career. It’s also important that a new programmer learns from an unbiased source of information, because an ad-supported environment will educate the new programmer towards products that they might not need. freeCodeCamp has not been easy to build. Building freeCodeCamp has required expertise in software engineering, business, media, and community development. The donation-based business model of freeCodeCamp doesn’t collect very much money. Why would somebody build a non-profit when they could spend their time building a highly profitable software company? Quincy Larson is the founder of freeCodeCamp, and he joins the show for a special episode about his backstory and the journey to building the best place on the Internet for a new programmer to begin.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published April 14, 2017. We are taking a few weeks off. We’ll be back soon with new episodes. Facebook’s open source projects include React, GraphQL, and Cassandra. These projects are key pieces of infrastructure used by thousands of developers–including engineers at Facebook itself. These projects are able to gain traction because Facebook takes time to decouple the projects from their internal infrastructure and clean up the code before releasing them into the wild. Facebook has high standards for what they are willing to release. Tom Occhino manages the React team at Facebook and works closely with engineers to determine what projects make sense to open source. In this episode, Preethi Kasireddy interviews Tom about how Facebook thinks about open source–what went right with React, why it makes sense for Facebook to continue to release new open source projects, and how full-time employees at Facebook interact with that open source codebase.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published October 24, 2019. We are taking a few weeks off. We’ll be back soon with new episodes. Redis is an in-memory database that persists to disk. Redis is commonly used as an object cache for web applications. Applications are composed of caches and databases. A cache typically stores the data in memory, and a database typically stores the data on disk. Memory has significantly faster access times, but is more expensive and is volatile, meaning that if the computer that is holding that piece of data in memory goes offline, the data will be lost. When a user makes a request to load their personal information, the server will try to load that data from a cache. If the cache does not contain the user’s information, the server will go to the database to find that information. Alvin Richards is chief product officer with Redis Labs, and he joins the show to discuss how Redis works. We explore different design patterns for making Redis high availability, or using it as a volatile cache, and we talk through the read and write path for Redis data. Full disclosure: Redis Labs is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published November 21, 2019. We are taking a few weeks off. We’ll be back soon with new episodes. HTTP is a protocol that allows browsers and web applications to communicate across the Internet. Everyone knows that HTTP is doing some important work, because “HTTP” is at the beginning of most URLs that you enter into your browser. You might be familiar with the request/response model, and HTTP request methods such as GET, PUT, and POST. But unless you have had a reason to learn more about the details of HTTP, you probably don’t know much more than that. Julia Evans is a software engineer and writer who creates Wizard Zines, a series of easy-to-read online magazines that explain technical software topics. Julia’s zines include “Linux Debugging Tools”, “Help! I Have A Manager!”, and recently “HTTP: Learn your browser’s language”. Her zines are a creative, innovative format for describing the world of software engineering while also exploring her own artistic pursuits in writing, design, and illustration. Julia was previously on the show to discuss Ruby profiling, and she returns to the show to discuss HTTP, as well as her creative process and goals with Wizard Zines.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published June 13, 2019. We are taking a few weeks off. We’ll be back soon with new episodes. Machine learning allows software to improve as that software consumes more data. Machine learning is a tool that every software engineer wants to be able to use. Because machine learning is so broadly applicable, software companies want to make the tools more accessible to the developers across the organization. There are many steps that an engineer must go through to use machine learning, and each additional step inhibits the chances that the engineer will actually get their model into production. An engineer who wants to build machine learning into their application needs access to data sets. They need to join those data sets, and load them into a machine (or multiple machines) where their model can be trained. Once the model is trained, the model needs to test on additional data to ensure quality. If the initial model quality is insufficient, the engineer might need to tweak the training parameters. Once a model is accurate enough, the engineer needs to deploy that model. After deployment, the model might need to be updated with new data later on. If the model is processing sensitive or financially relevant data, a provenance process might be necessary to allow for an audit trail of decisions that have been made by the model. Rob Story and Kelley Rivoire are engineers working on machine learning infrastructure at Stripe. After recognizing the difficulties that engineers faced in creating and deploying machine learning models, Stripe engineers built out Railyard, an API for machine learning workloads within the company. Rob and Kelley join the show to discuss data engineering and machine learning at Stripe, and their work on Railyard.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published January 31, 2019 Artificial intelligence is reshaping every aspect of our lives, from transportation to agriculture to dating. Someday, we may even create a superintelligence–a computer system that is demonstrably smarter than humans. But there is widespread disagreement on how soon we could build a superintelligence. There is not even a broad consensus on how we can define the term “intelligence”. Information technology is improving so rapidly we are losing the ability to forecast the near future. Even the most well-informed politicians and business people are constantly surprised by technological changes, and the downstream impact on society. Today, the most accurate guidance on the pace of technology comes from the scientists and the engineers who are building the tools of our future. Martin Ford is a computer engineer and the author of Architects of Intelligence, a new book of interviews with the top researchers in artificial intelligence. His interviewees include Jeff Dean, Andrew Ng, Demis Hassabis, Ian Goodfellow, and Ray Kurzweil. Architects of Intelligence is a privileged look at how AI is developing. Martin Ford surveys these different AI experts with similar questions. How will China’s adoption of AI differ from that of the US? What is the difference between the human brain and that of a computer? What are the low-hanging fruit applications of AI that we have yet to build? Martin joins the show to talk about his new book. In our conversation, Martin synthesizes ideas from these different researchers, and describes the key areas of disagreement from across the field.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cruise is an autonomous car company with a development cycle that is highly dependent on testing its cars–both in the wild and in simulation. The testing cycle typically requires cars to drive around gathering data, and that data to subsequently be integrated into a simulated system called Matrix. With COVID-19, the ability to run tests in the wild has been severely dampened. Cruise cannot put so many cars on the road, and thus has had to shift much of its testing procedures to rely more heavily on the simulations. Therefore, the simulated environments must be made very accurate, including the autonomous agents such as pedestrians and cars. Tom Boyd is VP of Simulation at Cruise. He joins the show to talk about the testing workflow at Cruise, how the company builds simulation-based infrastructure, and his work managing simulation at the company.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Grafana is an open source visualization and monitoring tool that is used for creating dashboards and charting time series data. Grafana is used by thousands of companies to monitor their infrastructure. It is a popular component in monitoring stacks, and is often used together with Prometheus, ElasticSearch, MySQL, and other data sources. The engineering complexities around building Grafana involve the large number of integrations, the highly configurable ReactJS frontend, and the ability to query and display large data sets. Grafana also must be deployable to cloud and on-prem environments. Torkel Ödegaard is a co-founder of Grafana Labs, and joins the show to talk about his work on the open source project and the company he is building around it.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Airflow was released in 2015, introducing the first popular open source solution to data pipeline orchestration. Since that time, Airflow has been widely adopted for dependency-based data workflows. A developer might orchestrate a pipeline with hundreds of tasks, with dependencies between jobs in Spark, Hadoop, and Snowflake. Since Airflow’s creation, it has powered the data infrastructure at companies like Airbnb, Netflix, and Lyft. It has also been at the center of Astronomer, a startup that helps enterprises build infrastructure around Airflow. Airflow is used to construct DAGs–directed acyclic graphs for managing data workflows. Maxime Beauchemin is the creator of Airflow. Vikram Koka and Ash Berlin-Taylor work at Astronomer. They join the show to talk about the state of Airflow–the purpose of the project, its use cases, and open source ecosystem.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The life cycle of data management includes data cleaning, extraction, integration, analysis and exploration, and machine learning models. It would be great if all of this data management could be handled with automation, but unfortunately that is not an option. For most applications, data management requires a human in the loop. A human in the loop might be responsible for working in a spreadsheet, or labeling data as a mechanical turk, or creating an algorithm for data labeling in Snorkel. Data scientists and data analysts are humans in the loop, studying large data sets. Aditya Parameswaran is an assistant professor at UC Berkeley. He studies human-in-the-loop data analytics, and he joins the show to talk about the work and the projects that he is focused on, including DataSpread, an alternative to Excel, and OrpheusDB, a relational database versioning system.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes continues to mature as a platform for infrastructure management. At this point, many companies have well-developed workflows and deployment patterns for working with applications built on Kubernetes. The complexity of some of these deployments may be daunting, and when a new employee joins a company, that employee needs to get quickly onboarded with the custom dev environment. Environment management is not the only issue with Kubernetes development. When a service gets updated, that update needs to be live and usable as fast as possible. When Kubernetes-related errors occur, those problems need to be easily accessible in a UI for triage. Dan Bentley is the CEO of Windmill Engineering, a company that makes a set of Kubernetes tools called Tilt. Dan joins the show to talk about the workflow for deploying Kubernetes infrastructure and the role of Tilt, the product he has been working on.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Uber needs to visualize data on a range of different surfaces. A smartphone user sees cars moving around on a map as they wait for their ride to arrive. Data scientists and operations researchers within Uber study the renderings of traffic moving throughout a city. Data visualization is core to Uber, and the company has developed a stack of technologies around visualization in order to build appealing, highly functional applications. DeckGL is a library for high-performance visualizations of large data sets. LumaGL is a set of components that targets high performance rendering. These and other tools make up VisGL, the data visualization technology that powers Uber. Uber’s visualization team included Ib Green, who left Uber to co-found Unfolded.ai, a company that builds geospatial analytics products. He joins the show to discuss his work on visualization products and libraries at Uber, as well as the process of taking that work to found Unfolded.ai. Full disclosure: I am an investor in Unfolded.ai.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A frontend developer issuing a query to a backend server typically requires the developer to issue that query through an ORM or a raw database query. Prisma is an alternative to both of these data access patterns, allowing for easier database access through auto-generated, type-safe query building tailored to an existing database schema. By integrating with Prisma, the developer gets a database client that has query autocompletion, and an API server with less boilerplate code. Prisma also has a system called Prisma Migrate, which simplifies database and schema migrations. Johannes Schickling is CEO of Prisma, and he joins the show to talk about the developments of Prisma that have occurred since we last spoke, and where the company is headed.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning workflows have had a problem for a long time: taking a model from the prototyping step and putting it into production is not an easy task. A data scientist who is developing a model is often working with different tools, or a smaller data set, or different hardware than the environment which that model will be deployed to. This problem existed at Uber just as it does at many other companies. Models were difficult to release, iterations were complicated, and collaboration between engineers could never reach a point that resembled a harmonious “DevOps”-like workflow. To address these problems, Uber developed an internal system called Michelangelo. Some of the engineers working on Michelangelo within Uber realized that there was a business opportunity in taking the Michelangelo work and turning it into a product company. Thus, Tecton was born. Tecton is a machine learning platform focused on solving the same problems that existed within Uber. Kevin Stumpf is the CTO at Tecton, and he joins the show to talk about the machine learning problems of Uber, and his current work at Tecton.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Many data sources produce new data points at a very high rate. With so much data, the issue of data quality emerges. Low quality data can degrade the accuracy of machine learning models that are built around those data sources. Ideally, we would have completely clean data sources, but that’s not very realistic. One alternative is a data cleaning system, which can allow us to clean up the data after it has already been generated. HoloClean is a statistical inference engine that can impute, clean, and enrich data. HoloClean is centered around “The Probabilistic Unclean Database Model”, which allows for two systems–an “intension” and a “realizer” to work together to fill in missing fields and fix erroneous fields in data. HoloClean was created by Theo Rekatsinas, and he joins the show to talk about the problem of fast, unclean data, and his work with HoloClean. We also talk about other problems in machine learning and the engineering workflows around data.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Server infrastructure traditionally consists of monolithic servers containing all of the necessary hardware to run a computer. These different hardware components are located next to each other, and do not need to communicate over a network boundary to connect the CPU and memory. LegoOS is a model for disaggregated, network-attached hardware. LegoOS disseminates the traditional operating system functionalities into loosely-coupled hardware and software components. By disaggregating data center infrastructure, the overall resource usage and failure rate of server infrastructure can be improved. Yiying Zhang is an assistant professor of computer science at UCSD. Her research focuses on operating systems, distributed systems, and datacenter networking. She joins the show to discuss her work and its implications for data centers and infrastructure.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes has become a highly usable platform for deploying and managing distributed systems. The user experience for Kubernetes is great, but is still not as simple as a full-on serverless implementation–at least, that has been a long-held assumption. Why would you manage your own infrastructure, even if it is Kubernetes? Why not use autoscaling Lambda functions and other infrastructure-as-a-service products? Matt Ward is a listener of the show and an engineer at Mux, a company that makes video streaming APIs. He sent me an email that said Mux has been having success with self-managed Kubernetes infrastructure, which they deliberately opted for over a serverless deployment. I wanted to know more about what shaped this decision to opt for self-managed infrastructure, and the costs and benefits that Mux has accrued as a result. Matt joins the show to talk through his work at Mux, and the architectural impact of opting for Kubernetes instead of fully managed serverless infrastructure. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every software company is a distributed system, and distributed systems fail in unexpected ways. This ever-present tendency for systems to fail has led to the rise of failure testing, otherwise known as chaos engineering. Chaos engineering involves the deliberate failure of subsystems within an overall system to ensure that the system itself can be resilient to these kinds of unexpected failures. Peter Alvaro is a distributed systems researcher who has published papers on a range of subjects, including debugging, failure testing, databases, and programming languages. He works with both academia and industry. Peter joins the show to discuss his research topics and goals. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Brex is a credit card company that provides credit to startups, mostly companies which have raised money. Brex processes millions of transactions, and uses the data from those transactions to assess creditworthiness, prevent fraud, and surface insights for the users of their cards. Brex is full of interesting engineering problems. The high volume of transactions requires data infrastructure to support all those transactions coming through the platform. As a credit card company, Brex needs to integrate with credit card networks and banking systems. There are internal systems for applications such as dispute resolution. Cos Nicolaescu is the CTO at Brex. He joins the show to discuss engineering at Brex, the dynamics of a credit card company, and his strategies around management. It was an instructive look inside of a rapidly growing fintech company. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Devices on the edge are becoming more useful with improvements in the machine learning ecosystem. TensorFlow Lite allows machine learning models to run on microcontrollers and other devices with only kilobytes of memory. Microcontrollers are very low-cost, tiny computational devices. They are cheap, and they are everywhere. The low-energy embedded systems community and the machine learning community have come together with a collaborative effort called tinyML. tinyML represents the improvements of microcontrollers, lighter weight frameworks, better deployment mechanisms, and greater power efficiency. Zach Shelby is the CEO of EdgeImpulse, a company that makes a platform called Edge Impulse Studio. Edge Impulse Studio provides a UI for data collection, training, and device management. As someone creating a platform for edge machine learning usability, Zach was a great person to talk to the state of edge machine learning and his work building a company in the space. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

For the last five months, we have been working on a new version of Software Daily, the platform we built to host and present our content. We are creating a platform that integrates the podcast with a set of other features that make it easier to learn from the audio interviews. Software Daily includes the following features:

The world of software is large, and growing bigger every day. Software Daily is a place to explore this world of software companies and projects. If the podcast is a useful resource for you to learn about software, then Software Daily might also provide you with value. This post (and episode) is a brief description of the features that we have built into Software Daily. If you want to listen to Software Engineering Daily without ads, you can become a paid subscriber, paying $10/month or $100/year by going to softwaredaily.com/subscribe. We now have an RSS feed that paid customers can add to a podcast player like Overcast (on iOS) or Podcast Addict (on Android). You can also listen to the premium episodes using our apps for iOS or Android. Whether you are a listener who is fine with listening to ads, or you are a listener who pays to hear episodes without ads, we are happy to have you tuning in. Apple podcasts limits the number of episodes in an RSS feed to 300. The feed with the last 300 episodes is available by searching for Software Daily. In total, we have more than 1200 episodes in our back catalog. Listeners often want to find all our episodes on React, or Kubernetes, or serverless, or self-driving cars. We have been covering these topics for years, and much of the old content has retained its value. Software Daily allows you to easily find all the episodes relating to a subject that you are interested in. You can also find our most popular episodes, ranked by how people interact with them. Additionally, episode transcripts have interactive features with highlighting, commenting, and discussions. We want to create a Medium-like experience for the episodes. Software Daily is a place where listeners can write about the topics they are listening to. When you are listening to lots of episodes about a topic such as GraphQL, you may find it useful to write about that topic as a form of active learning. The topic pages also have a Q&A section. Post questions about a topic, or post an answer. Engage in the community dialogue surrounding a topic you are passionate or curious about. If there is a topic you want to write about, check out softwaredaily.com/write. We will be turning the best written content into short podcast episodes published on the weekends where we will read your contribution and mention your name. If you write something awesome, we want to turn it into audio for larger distribution. Every topic on Software Daily has a Q&A section. We have covered lots of niche software companies and open source projects, and on Software Daily we want to collect more information about the world of software with Q&A. If you want to write about a specific company or topic that you heard about on Software Daily, Q&A is also an option. Our goal with Q&A is to provide a companion experience to listening to the podcast. It is not always easy to retain what you hear in a podcast episode. Answering some questions after you listen to an episode can help with that retention. Are you looking to hire someone specific in the world of software? Post a job on the Software Daily jobs board. We will be announcing some of these jobs on the podcast, especially the more interesting postings, and ones that align with content we are producing. We appreciate you tuning into Software Daily. We would welcome your feedback, and hope you take the time to check out SoftwareDaily.com. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Over the last 5 years, web development has matured considerably. React has become a standard for frontend component development. GraphQL has seen massive growth in adoption as a data fetching middleware layer. The hosting platforms have expanded beyond AWS and Heroku, to newer environments like Netlify and Vercel. These changes are collectively known as the JAMStack. With the changes brought by the JAMStack, it raises the question: how should an app be built today? Can a framework offer guidance for how the different layers of a JAMStack app should fit together? RedwoodJS is a framework for building JAMStack applications. Tom Preston-Werner is one of the creators of RedwoodJS, as well as the founder of GitHub and Chatterbug, a language learning app. He joins the show to talk about the future of JAMStack development, and his goals for RedwoodJS. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Geospatial analytics tools are used to render visualizations for a vast array of applications. Data sources such as satellites and cellular data can gather location data, and that data can be superimposed over a map. A map-based visualization can allow the end user to make decisions based on what they see. ArcGIS is one of the most widely used geospatial analytics platforms. It is created by ESRI, the Environmental Systems Research Institute, which was started in 1969. Today, ESRI products have 40% of the global market share of geospatial analytics software. Max Payson is a solutions engineer at ESRI, and he joins the show to talk about applications of ArcGIS, and the landscape of GIS more broadly. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Customer data infrastructure is a type of tool for saving analytics and information about your customers. The company that is best known in this category is Segment, a very popular API company. This customer data is used for making all kinds of decisions around product roadmap, pricing, and design. RudderStack is a company built around open source customer data infrastructure. RudderStack can be self-hosted, allowing users to deploy it to their own servers and manage their data however they please. Soumyadeb Mitra is the creator of RudderStack, and he joins the show to talk about the space of customer data infrastructure, and his own company. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Matterport is a company that builds 3-D imaging for the inside of buildings, construction sites, and other locations that require a “digital twin.” Generating digital images of the insides of buildings has a broad spectrum of applications, and there are considerable engineering challenges in building such a system. Matterport’s hardware stack involves a camera built in-house by the company. The camera can take 360 degree scans of a room, stitch the imagery together, and make the digital twin available on the cloud. Japjit Tulsi works at Matterport, and he joins the show to discuss 3-D imaging, and his role as CTO of the company. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

There are many bad recipe web sites. Every time I navigate to a recipe website, it feels like my browser is filling up with spyware. The page loads slowly, everything seems broken, I can feel the 25 different JavaScript adtech tags interrupting each other. Whether I am searching for banana bread or a spaghetti sauce recipe, recipe sites usually make me lose my appetite. Anycart is a recipe platform that allows users to buy all of the ingredients for the recipe and have those ingredients delivered. It’s a vertically integrated content site and delivery system. It is also beautifully designed and extremely performant. I learned about it from Zack Bloom, who works at Cloudflare, as he mentioned it as a case study in performance. Rafael Sanches is a founder of Anycart, and he joins the show to talk about building a recipe delivery service, and the innovations in performance that were necessary to building it. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Amazon’s virtual server instances have come a long way since the early days of EC2. There are now a wide variety of available configuration options for spinning up an EC2 instance, which can be chosen from based on the workload that will be scheduled onto a virtual machine. There are also Fargate containers and AWS Lambda functions, creating even more options for someone who wants to deploy virtualized infrastructure. The high demand for virtual machines has led to Amazon moving down the stack, designing custom hardware such as the Nitro security chip, and low level software such as the Firecracker virtual machine monitor. AWS also has built Outposts, which allow for on-prem usage of AWS infrastructure. Anthony Liguori is an engineer at AWS who has worked on a range of virtualization infrastructure: software platforms, hypervisors, and hardware. Anthony joins the show to talk about virtualization at all levels of the stack. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A credit score is a rating that allows someone to qualify for a line of credit, which could be a loan such as a mortgage, or a credit card. We are assigned a credit score based on a credit history, which could be related to work history, rental payments, or loan repayments. One problem with the credit scoring system is that it is not internationalized. If I am coming from Brazil, I have a rental history of someone from Brazil. That information does not get naturally ported over to the United States. There needs to be a system for translating a foreign credit history to a US credit history. Nova Credit is a company that makes a credit passport–a system for allowing users in one geographic location to use the credit history that they have built up to have credit in another location, namely the United States. Brian Regan and Misha Esipov work at Nova Credit, and they join the show to talk about how the company works, and the problem it solves. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A large software company such as Dropbox is at a constant risk of security breaches. These security breaches can take the form of social engineering attacks, network breaches, and other malicious adversarial behavior. This behavior can be surfaced by analyzing collections of log data. Log-based threat response is not a new technique. But how should those logs be analyzed? Grapl is a system for modeling log data as a graph, and analyzing that graph for threats based on how nodes in the graph have interacted. By building a graph from log data, Grapl can classify interaction patterns that correspond to threats. Colin O’Brien is the creator of Grapl, and he joins the show to discuss security, as well as threat detection and response. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Infrastructure-as-code tools are used to define the architecture of software systems. Common infrastructure-as-code tools include Terraform and AWS CloudFormation. When infrastructure is defined as code, we can use static analysis tools to analyze that code for configuration mistakes, just as we could analyze a programming language with traditional static analysis tools. When a developer writes a program, that developer might use static analysis to parse a program for common mistakes–memory leaks, potential null pointers, and security holes. The concept of static analysis can be extended to infrastructure as code, allowing for the discovery of higher level problems such as insecure policies across cloud resources. Guy Eisenkot is an engineer with Bridgecrew, a company that makes static analysis tools for security and compliance. Guy joins the show to talk about cloud security and how static analysis can be used to improve the quality of infrastructure deployments. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Social distancing has been imposed across the United States. We are running an experiment unlike anything before it in history, and it is likely to have a lasting impact on human behavior. By looking at location data of how people are moving around today, we can examine the real-world impacts of social distancing. SafeGraph is a company that provides geospatial location data to be used by developers and researchers. Much of their data is aggregated from cell phone GPS pings which identify where anonymized users are in the world. This data set provides the basis for SafeGraph’s social distancing metrics, which measure how frequently people are coming into contact with one another. Ryan Fox Squire works at SafeGraph, and he returns to the show to discuss social distancing metrics and the research that has come out of studying these metrics. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Dropbox is a consumer storage product with petabytes of data. Dropbox was originally started on the cloud, backed by S3. Once there was a high enough volume of data, Dropbox created its own data centers, designing hardware for the express purpose of storing user files. Over the last 13 years, Dropbox’s infrastructure has developed hardware, software, networking, data center infrastructure, and operational procedures that make the cloud storage product best in class. Andrew Fong has been an engineer at Dropbox for 8 years. He joins the show to talk about how the Dropbox engineering organization has changed over that period of time, and what he is doing at the company today. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

“Data stream” is a word that can be used in multiple ways. A stream can refer to data in motion or data at rest. When a stream is data in motion, an endpoint is receiving new pieces of data on a continual basis. Each new data point is sent over the wire and captured by the other end. Another way a stream can be represented is as a sequence of events that have been written to a storage medium. This is a stream at rest. Pravega is a system for storing large streams of data. Pravega can be used as an alternative to systems like Apache Kafka or Apache Pulsar. Flavio Junquiera is an engineer at Dell EMC who works on Pravega. He joins the show to talk about the history of stream processing and his work on Pravega. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Redis is an in-memory object storage system that is commonly used as a cache for web applications. This core primitive of in-memory object storage has created a larger ecosystem encompassing a broad set of tools. Redis is also used for creating objects such as queues, streams, and probabilistic data structures. Machine learning systems also need access to fast, in-memory object storage. RedisAI is a newer module for supporting machine learning tasks. For serverless computing, RedisGears allows for the execution of functions close to your Redis instance. RedisEdge allows for edge computing with Redis. Alvin Richards returns to the show to discuss the expansion of Redis to becoming a broad suite of in-memory tools, as well as the resiliency properties of Redis and usage patterns for the tool. RedisLabs is a sponsor of Software Engineering Daily, and RedisConf is a virtual conference around Redis that runs May 12-13. If you are interested in Redis, you can check out RedisConf for free by going to RedisConf.com. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

For many applications, a transactional MySQL database is the source of truth. To make a MySQL database scale, some developers deploy their database using Vitess, a sharding system built on top of Kubernetes. Jiten Vaidya and Anthony Yeh work at PlanetScale, a company that focuses on building and supporting MySQL databases sharded with Vitess. Their experience comes from working at YouTube, which has a massive, rapidly growing database for storing the information about videos on the site. Sharding is not the only database problem that YouTube faced. Availability was another issue. At YouTube, the database operators want YouTube’s MySQL cluster to be resilient to the failure of an entire data center. Similarly, a developer deploying an important MySQL database to the cloud wants their database to be resilient to the failure of an entire cloud provider. Jiten and Anthony join the show to talk about their work building multicloud support for MySQL, and their process of deploying a consistent MySQL database in Azure, GCP, and AWS. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

We are all living in social isolation due to the quarantine from COVID-19. Isolation is changing our habits and our moods, ravaging the economy, and changing how we work. One positive change is that more people have been reconnecting with their friends and family over frequent calls and video chats. Isolation is not a normal way for humans to live. We are social animals, and we need social interaction. We’ve changed how we use Internet products. There has been an evolution of trends in online shopping, social networking, and video communication software. Courtland Allen is the founder of Indie Hackers and Anurag Goel is the founder of Render, a new cloud provider. Both Courtland and Anurag are friends of mine, and join this episode to talk about how their lives are changing as a result of social isolation. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A data warehouse is a system for performing fast queries on large amounts of data. A data lake is a system for storing high volumes of data in a format that is slow to access. A typical workflow for a data engineer is to pull data sets from this slow data lake storage into the data warehouse for faster querying. Apache Spark is a system for fast processing of data across distributed datasets. Spark is not thought of as a data warehouse technology, but it can be used to fulfill some of the responsibilities. Delta is an open source system for a storage layer on top of a data lake. Delta integrates closely with Spark, creating a system that Databricks refers to as a “data lakehouse.” Michael Armbrust is an engineer with Databricks. He joins the show to talk about his experience building the company, and his perspective on data engineering, as well as his work on Delta, the storage system built for the Spark ecosystem.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A content management system (CMS) defines how the content on a website is arranged and presented. The most widely used CMS is WordPress, the open source tool that is written in PHP. A large percentage of the web consists of WordPress sites, and WordPress has a huge ecosystem of plugins and templates. Despite the success of WordPress, the JAMStack represents the future of web development. JAM stands for JavaScript, APIs, and Markup. In contrast to the monolithic WordPress deployments, a JAMStack site consists of loosely coupled components. And there are numerous options for a CMS in this environment. TinaCMS is one such option. TinaCMS is an acronym for “Tina Is Not A CMS”, and it is a toolkit for content management. Scott Gallant, Jordan Patterson, and Nolan Phillips work on TinaCMS, and they join the show to explore the topic of content management on the JAMStack.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A data workflow scheduler is a tool used for connecting multiple systems together in order to build pipelines for processing data. A data pipeline might include a Hadoop task for ETL, a Spark task for stream processing, and a TensorFlow task to train a machine learning model. The workflow scheduler manages the tasks in that data pipeline and the logical flow between them. Airflow is a popular data workflow scheduler that was originally created at Airbnb. Since then, the project has been adopted by numerous companies that need workflow orchestration for their data pipelines. Jeremiah Lowin was a core committer to Airflow for several years before he identified several features of Airflow that he wanted to change. Prefect is a dataflow scheduler that was born out of Jeremiah’s experience working with Airflow. Prefect’s features include data sharing between tasks, task parameterization, and a different API than Airflow. Jeremiah joins the show to discuss Prefect, and how his experience with Airflow led to his current work in dataflow scheduling.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A relational database often holds critical operational data for a company, including user names and financial information. Since this data is so important, a relational database must be architected to avoid data loss. Relational databases need to be a distributed system in order to provide the fault tolerance necessary for production use cases. If a database node goes down, the database must be able to recover smoothly without data loss, and this requires having all of the data in the database replicated beyond a single node. If you write to a distributed transactional database, that write must propagate to each of the other nodes in the database. If you read from a distributed database, that read must return the same data that any other database reader would see. These constraints can be satisfied differently depending on the design of the database system. As a result, there is a vast market of distributed databases from cloud providers and software vendors. CockroachDB is an open source, globally consistent relational database. CockroachDB is heavily informed by Google Spanner, the relational database that Google uses for much of its transactional workloads. Peter Mattis is a co-founder of CockroachDB, and he joins the show to discuss the architecture of CockroachDB, the process of building a business around a database, and his memories working on distributed systems at Google. Full disclosure: CockroachDB is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Python is the most widely used language for data science, and there are several libraries that are commonly used by Python data scientists including Numpy, Pandas, and scikit-learn. These libraries improve the user experience of a Python data scientist by giving them access to high level APIs. Data science is often performed over huge datasets, and the data structures that are instantiated with those datasets need to be spread across multiple machines. To manage large distributed datasets, a library such as scikit-learn can use a system called Dask. Dask allows the instantiation of data structures such as a Dask dataframe or a Dask array. Matthew Rocklin is the creator of Dask. He joins the show to talk about distributed computing with Dask, its use cases, and the Python ecosystem. He also provides a detailed comparison between Dask and Spark, which is also used for distributed data science.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Chatbots became widely popular around 2016 with the growth of chat platforms like Slack and voice interfaces such as Amazon Alexa. As chatbots came into use, so did the infrastructure that enabled chatbots. NLP APIs and complete chatbot frameworks came out to make it easier for people to build chatbots. The first suite of chatbot frameworks were largely built around rule-based state machine systems. These systems work well for a narrow set of use cases, but fall over when it comes to chatbot models that are more complex. Rasa was started in 2015, amidst the chatbot fever. Since then, Rasa has developed a system that allows a chatbot developer to train their bot through a system called interactive learning. With interactive learning, I can deploy my bot, spend some time talking to it, and give that bot labeled feedback on its interactions with me. Rasa has open source tools for natural language understanding, dialogue management, and other components needed by a chatbot developer. Tom Bocklisch works at Rasa, and he joins the show to give some background on the field of chatbots and how Rasa has evolved over time.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Serverless computing is a way of designing applications that do not directly address or deploy application code to servers. Serverless applications are composed of stateless functions-as-a-service and stateful data storage systems such as Redis or DynamoDB. Serverless applications allow for scaling up and down the entire architecture, because each component is naturally scalable. And this pattern can be used to create a wide variety of applications. The functions-as-a-service can handle the compute logic, and the data storage systems can handle the storage. But these applications do not give the developer as much flexibility as an ideal serverless system might. The developer would need to use cloud-specific state management systems. Vikram Sreekanti is the creator of Cloudburst, a system for stateful functions as a service. Cloudburst is architected as a set of VMs that can execute functions-as-a-service that are scheduled onto them. Each VM can utilize a local cache, as well as an autoscaling key-value store called Anna which is accessible to the Cloudburst runtime components. Vikram joins the show to talk about serverless computing and his efforts to build stateful serverless functionality.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

NGINX is a web server that can be used to manage the APIs across an organization. Managing these APIs involves deciding on the routing and load balancing across the servers which host them. If the traffic of a website suddenly spikes, the website needs to spin up new replica servers and update the API gateway to route traffic to those new replicas. Some servers should not be accessible to outside traffic, and policy management is used to configure the security policies of different APIs. And as a company grows, the number of APIs also grows, increasing the complexity of managing routing logic and policies. Kevin Jones is a product manager with NGINX. He joins the show to discuss how API management has changed with the growth of cloud and mobile, and how NGINX has evolved over that period of time. Full disclosure: NGINX is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Web development has historically had more work being done on the server than on the client. The observability tooling has reflected this emphasis on the backend. Monitoring tools for log management and backend metrics have existed for decades, helping developers debug their server infrastructure. Today, web frontends have more work to do. Detailed components in frameworks such as React and Angular might respond quickly without waiting for a network request, with their mutations being processed entirely in the browser. This results in better user experiences, but more work is being done on the client side, away from the backend observability tools. Matt Arbesfeld is a co-founder of LogRocket, a tool that records and plays back browser sessions and allows engineers to look at those sessions to understand what kinds of issues are occurring in the user’s browser. Matt joins the show to talk about the field of frontend monitoring, and the engineering behind his company LogRocket.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Zoom video chat has become an indispensable part of our lives. In a crowded market of video conferencing apps, Zoom managed to build a product that performs better than the competition, scaling with high quality to hundreds of meeting participants, and millions of concurrent users. Zoom’s rapid growth in user adoption came from its focus on user experience and video call quality. This focus on product quality came at some cost to security quality. As our entire digital world has moved onto Zoom, the engineering community has been scrutinizing Zoom more closely, and discovered several places where the security practices of Zoom are lacking. Patrick Wardle is an engineer with a strong understanding of Apple products. He recently wrote about several vulnerabilities he discovered on Zoom, and joins the show to talk about the security of large client-side Mac applications as well as the specific vulnerabilities of Zoom.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook applications use maps for showing users where to go. These maps can display businesses, roads, and event locations. Understanding the geographical world is also important for performing search queries that take into account a user’s location. For all of these different purposes, Facebook needs up-to-date, reliable mapping data. OpenStreetMap is an open system for accessing mapping data. Anyone can use OpenStreetMap to add maps to their application. The data in OpenStreetMap is crowdsourced by users who submit updates to the OpenStreetMap database. Since anyone can submit data to OpenStreetMap, there is a potential for bad data to appear in the system. Facebook uses OpenStreetMap for its mapping data, including for important applications where bad data would impact a map user in a meaningfully negative way. In order to avoid this, Facebook builds infrastructure tools to improve the quality of its maps. Saurav Mapatra and Jacob Wasserman work at Facebook on its mapping infrastructure, and join the show to talk about the tooling Facebook has built around OpenStreetMap data.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

NGINX is a web server that is used as a load balancer, an API gateway, a reverse proxy, and other purposes. Core application servers such as Ruby on Rails are often supported by NGINX, which handles routing the user requests between the different application server instances. This model of routing and load balancing between different application instances has matured over the last ten years due to an increase in the number of servers, and an increase in the variety of services. A pattern called “service mesh” has grown in popularity and is used to embed routing infrastructure closer to individual services by giving them a sidecar proxy. The application sidecars are connected to each other, and requests between any two services are routed through a proxy. These different proxies are managed by a central control plane which manages policies of the different proxies. Alan Murphy works at NGINX, and he joins the show to give a brief history of NGINX and how the product has evolved from a reverse proxy and edge routing tool to a service mesh. Alan has worked in the world of load balancing and routing for more than a decade, having been at F5 Networks for many years before F5 acquired NGINX. We also discussed the business motivations behind the merger of those two companies. Full disclosure: NGINX is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Shopify is a platform for selling products and building a business. It is a large e-commerce company with hundreds of engineers and several different mobile apps. Shopify’s engineering culture is willing to adopt new technologies aggressively, trying new tools that might provide significant leverage to the organization. React Native is one of those technologies. React Native can be used to make cross-platform mobile development easier by allowing code reuse between Android and iOS. React Native was developed within Facebook, and has been adopted by several other prominent technology companies, with varying degrees of success. Many companies have seen improvements to their mobile development and release process. However, in a previous episode, we talked with Airbnb about their adoption of React Native, which was less successful. Farhan Thawar is a VP of engineering at Shopify. He joins the show to talk about Shopify’s experience using React Native, the benefits of cross-platform development, and his perspective on when it is not a good idea to use React Native.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ceph is a storage system that can be used for provisioning object storage, block storage, and file storage. These storage primitives can be used as the underlying medium for databases, queueing systems, and bucket storage. Ceph is used in circumstances where the developer may not want to use public cloud resources like Amazon S3. As an example, consider telecom infrastructure. Telecom companies that have their own data centers need software layers which make it simpler for the operators and developers that are working with that infrastructure to spin up databases and other abstractions with the same easy experience that is provided by a cloud provider by AWS. Sage Weil has been a core developer on Ceph since 2005, and the company he started around Ceph sold to Red Hat for $175 million. Sage joins the show to talk about the engineering behind Ceph and his time spent developing companies.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data analysts need to collaborate with each other in the same way that software engineers do. They also need a high quality development environment. These data analysts are not working with programming languages like Java and Python, so they are not using an IDE such as Eclipse. Data analysts predominantly use SQL, and the tooling for a data analyst to work with SQL is often a SQL explorer tool that lacks the kind of collaborative experience that we would expect in the age of Slack and GitHub. Rahil Sondhi is the creator of PopSQL, a collaborative SQL explorer. He created PopSQL after several years in the software industry, including 4 years at Instacart. Rahil joins the show to talk about the frictions that data analysts encounter when working with databases, and how those frictions led to the design of PopSQL.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a developer spins up a virtual machine on AWS, that virtual machine could be purchased using one of several types of cost structures. These cost structures include on-demand instances, spot instances, and reserved instances. On-demand instances are often the most expensive, because the developer gets reliable VM infrastructure without committing to long-term pricing. Spot instances are cheap, spare compute capacity with lower reliability, that is available across AWS infrastructure. Reserved instances allow a developer to purchase longer term VM contracts for a lower price. Reserved instances can provide significant savings, but it can be difficult to calculate how much infrastructure to purchase. Aran Khanna is the founder of Reserved.ai, a company that builds cost management tools for AWS. He joins the show to talk about the landscape of cost management, and what he is building with Reserved.ai.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning models require the use of training data, and that data needs to be labeled. Today, we have high quality data infrastructure tools such as TensorFlow, but we don’t have large high quality data sets. For many applications, the state of the art is to manually label training examples and feed them into the training process. Snorkel is a system for scaling the creation of labeled training data. In Snorkel, human subject matter experts create labeling functions, and these functions are applied to large quantities of data in order to label it. For example, if I want to generate training data about spam emails, I don’t have to hire 1000 email experts to look at emails and determine if they are spam or not. I can hire just a few email experts, and have them define labeling functions that can indicate whether an email is spam. If that doesn’t make sense, don’t worry. We discuss it in more detail in this episode. Braden Hancock works on Snorkel, and he joins the show to talk about the labeling problems in machine learning, and how Snorkel helps alleviate those problems. We have done many shows on machine learning in the past, which you can find on SoftwareDaily.com. Also, if you are interested in writing about machine learning, we have a new writing feature that you can check out by going to SoftwareDaily.com/write.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A workflow is an application that involves more than just a simple request/response communication. For example, consider a session of a user taking a ride in an Uber. The user initiates the ride, and the ride might last for an hour. At the end of the ride, the user is charged for the ride and sent a transactional email. Throughout this entire ride, there are many different services and database tables being accessed across the Uber infrastructure. The transactions across this infrastructure need to be processed despite server failures which may occur along the way. Workflows are not just a part of Uber. Many different types of distributed operations at a company might be classified as a workflow: banking operations, spinning up a large cluster of machines, performing a distributed cron job. Maxim Fateev is the founder of Temporal.io, and the co-creator of Cadence, a workflow orchestration engine. Maxim developed Cadence when he was at Uber, seeing the engineering challenges that come from trying to solve the workflow orchestration problem. Before Uber, Maxim worked at AWS on the Simple Workflow Service, which was also a system for running workflows. Altogether, Maxim has developed workflow software for more than a decade.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kafka is a distributed stream processing system that is commonly used for storing large volumes of append-only event data. Kafka has been open source for almost a decade, and as the project has matured, it has been used for new kinds of applications. Kafka’s pubsub interface for writing and reading topics is not ideal for all of these applications, which has led to the creation of ksqlDB, a database system built for streaming applications that uses Kafka as the underlying infrastructure for storing data. Michael Drogalis is a principal product manager at Confluent, where he helped develop ksqlDB. Michael joins the show to discuss ksqlDB, including the architecture, the query semantics, and the applications which might want a database that focuses on streams. We have done many great shows on Kafka in the past, which you can find on SoftwareDaily.com. Also, if you are interested in writing about Kafka, we have a new writing feature that you can check out by going to SoftwareDaily.com/write.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Building a game is not easy. The development team needs to figure out a unique design and gameplay mechanics that will attract players. There is a great deal of creative work that goes into making a game successful, and these games are often built with low budgets by people who are driven by the art and passion of game creation. A game engine is a system used to build and run games. Game engines let the programmer work at a high level of abstraction, by providing interfaces for graphics, physics, and scripting. Popular game engines include Unreal Engine and Unity, both of which require a license that reduces the amount of money received by the game developer. Godot is an open source and free to use game engine. The project was started by Juan Linietsky, who joins the show to discuss his motivation for making Godot. We have done some great shows on gaming in the past, which you can find on SoftwareDaily.com. Also, if you are interested in writing about game development, we have a new writing feature that you can check out by going to SoftwareDaily.com/write.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

V8 is the JavaScript engine that runs Chrome. Every popular website makes heavy use of JavaScript, and V8 manages the execution environment of that code. The code that processes in your browser can run faster or slower depending on how “hot” the codepath is. If a certain line of code is executed frequently, that code might be optimized to run faster. V8 is running behind the scenes in your browser all the time, evaluating the code in your different tabs and determining how to manage that runtime in memory. As V8 is observing your code and analyzing it, V8 needs to allocate resources in order to determine what code to optimize. This process can be quite memory intensive, and can add significant overhead to the memory overhead of Chrome. Ross McIlroy is an engineer at Google, where he worked on a project called V8 Lite. The goal of V8 Lite was to significantly reduce the execution overhead of V8. Ross joins the show to talk about JavaScript memory consumption, and his work on V8 Lite. We have done some great shows on JavaScript in the past, which you can find on SoftwareDaily.com. Also, if you are interested in writing about JavaScript, we have a new writing feature that you can check out by going to SoftwareDaily.com/write.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Serverless tools have come a long way since the release of AWS Lambda in 2014. Serverless apps were originally architected around Lambda, with the functions-as-a-service being used to glue together larger pieces of functionality and API services. Today, many of the common AWS services such as API Gateway and DynamoDB have functionality built in to be able to respond to events. These services can use Amazon EventBridge to connect to each other. In many cases, a developer does not need AWS Lambda to glue services together in order to build an event-driven application. Jeremy Daly is the host of the Serverless Chats podcast, a show about patterns and strategies in serverless architecture. Jeremy joins the show to talk about modern serverless development, and the new tools available in the AWS ecosystem.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cortico is a non-profit that builds audio tools to improve public dialogue. Allison King is an engineer at Cortico, and she joins the show to talk about the process of building audio applications. One of these applications was a system for ingesting radio streams, transcribing the radio, and looking for duplicate information across the different radio stations. In a talk at Data Council, Allison talked through the data engineering architecture for processing these radio streams, and the patterns that she found across the radio streams, including clusters of political leanings. Another project from Cortico is called Local Voices Network. The Local Voices Network is built around a piece of hardware called a “digital hearth”, a specialized device that records discussions among people in a community. These community discussions are made available to journalists, public officials, and political candidates, creating a listening channel that connects these communities and stakeholders. Much of our conversation is focused on the engineering of the digital hearth, this device that sits in the center of community discussions.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook Messenger is a chat application that millions of people use every day to talk to each other. Over time, Messenger has grown to include group chats, video chats, animations, facial filters, stories, and many more features. Messenger is a tool for utility as well as for entertainment. Messenger is used both on mobile and on desktop, but the size of the mobile application is particularly important on mobile. There are many users who are on devices that do not have much storage space. As Messenger has accumulated features, the iOS code base has grown larger and larger. Several generations of Facebook engineers have rotated through the company with the responsibility of working on Facebook Messenger, which has led to different ways of managing information within the same codebase. The iOS codebase had room for improvement. Project Lightspeed was a project within Facebook that had the goal of making Messenger on iOS much smaller. Mohsen Agsen is an engineer with Facebook, and he joins the show to talk about the process of rewriting the Messenger app. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern web development involves a complicated toolchain for managing dependencies. One part of this toolchain is the bundler, a tool that puts all your code and dependencies together into static asset files. The most popular bundler is webpack, which was originally released in 2012, before browsers widely supported ES Modules. Today, every major browser supports the ES Module system, which improves the efficiency of JavaScript dependency management. Snowpack is a system for managing dependencies that takes advantage of the browser support for ES Modules. Snowpack is made by Pika, a company that is developing a set of web technologies including a CDN, a package catalog, and a package code editor. Fred Schott is the founder of Pika and the creator of Snowpack. Fred joins the show to talk about his goals with Pika, and the ways in which modern web development is changing. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Food delivery apps have changed how the restaurant world operates. After seven years of mobile food delivery, the volume of food ordered through these apps has become so large that entire restaurants can be sustained solely through the order flow that comes in from the apps. This raises the question as to why you even need an “on-prem” restaurant. A cloud kitchen is a large, shared kitchen where food is prepared for virtual restaurants. These virtual restaurants exist only on mobile apps. There are no waiters, there are only the food delivery couriers who pick up the food from these warehouse-sized food preparation facilities. A virtual restaurant entrepreneur could open up multiple restaurants operated from the same cloud kitchen. The mobile app user might see separate restaurant listings for a pizza place, a cookie bakery, and a Thai food restaurant, when all of them are operated by the same restaurateur. Ashley Colpaart is the founder of The Food Corridor, a system for cloud kitchen management. Ashley joins the show to talk about the dynamics of virtual restaurants and the cloud kitchen industry. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Remote engineering work makes some elements of software development harder, and some elements easier. With Slack and email, communication becomes more clear cut. Project management tools lay out the responsibilities and deliverables of each person. GitHub centralizes and defines the roles of developers. On the other hand, remote work subtracts the role of nuanced conversation. There is no water cooler or break room. Work can become systematic, rigid, and completely transactional. Your co-workers are your allies, but they feel less like friends when you don’t see them every day. For some people, this can have a devastating long-term impact on their psyche. Managers have the responsibility of ensuring the health and productivity of the people that work with them. Managing an all-remote team includes a different set of challenges than an in-person team. Ryan Chartrand is the CEO of X-Team, a team of developers who work across the world and collaborate with each other remotely. X-Team partners with large companies who need additional development work. Ryan joins the show to talk about the dynamics of leading a large remote workforce, as well as his own personal experiences working remotely. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Programming languages are dynamically typed or statically typed. In a dynamically typed language, the programmer does not need to declare if a variable is an integer, string, or other type. In a statically typed language, the developer must declare the type of the variable upfront, so that the compiler can take advantage of that information. Dynamically typed languages give a programmer flexibility and fast iteration speed. But they also introduce the possibility of errors that can be avoided by performing type checking. This is one of the reasons why TypeScript has risen in popularity, giving developers the option to add types to their JavaScript variables. Sorbet is a typechecker for Ruby. Sorbet allows for gradual typing of Ruby programs, which helps engineers avoid errors that might otherwise be caused by the dynamic type system. Dmitry Petrashko is an engineer at Stripe who helped build Sorbet. He has significant experience in compilers, having worked on Scala before his time at Stripe. Dmitry joins the show to discuss his work on Sorbet, and the motivation for adding type checking to Ruby. We’re looking for new show ideas, so if you have any interesting topics, please feel free to reach out via twitter or email us at jeff@softwareengineeringdaily.com We realize right now humanity is going through a hard time with the Caronovirus pandemic, but we all have skills useful to fight this battle. Head over to codevid19.com to join the world’s largest pandemic hackathon! Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Datomic is a database system based on an append-only record keeping system. Datomic users can query the complete history of the database, and Datomic has ACID transactional support. The data within Datomic is stored in an underlying database system such as Cassandra or Postgres. The database is written in Clojure, and was co-authored by the creator of Clojure, Rich Hickey. Datomic has a unique architecture, with a component called a Peer, which gets embedded in an application backend. A Peer stores a subset of the database data in memory in this application backend, improving the latency of database queries that hit this caching layer. Marshall Thompson works at Cognitect, the company that supports and sells the Datomic database. Marshall joins the show to talk about the architecture of Datomic, its applications, and the life of a query against the database. We’re looking for new show ideas, so if you have any interesting topics, please feel free to reach out via twitter or email us at jeff@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A large cloud provider has high volumes of network traffic moving through data centers throughout the world. These providers manage the infrastructure for thousands of companies, across racks and racks of multitenant servers, and cables that stretch underseas, connecting network packets with their destination. Google Cloud Platform has grown steadily into a wide range of products, including database services, machine learning, and containerization. Scaling a cloud provider requires both technical expertise and skillful management. Lakshmi Sharma is the director of product management for networking at Google Cloud Platform. She joins the show to discuss the engineering challenges of building a large scale cloud provider, including reliability, programmability, and how to direct a large hierarchical team. We’re looking for new show ideas, so if you have any interesting topics, please feel free to reach out via twitter or email us at jeff@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Over the last fifteen years, there has been a massive increase in the number of new software tools. This is true at the infrastructure layer: there are more databases, more cloud providers, and more open-source projects. And it’s also true at a higher level: there are more APIs, project management systems, and productivity tools. ClickUp is a project management and productivity system for organizations and individuals. The goal of ClickUp is to create a system that integrates closely with other project management systems, popular SaaS tools, and the Google Suite of docs and spreadsheets. The company was started in 2016, and despite raising zero outside capital, it has grown as rapidly as many venture-backed companies. Zeb Evans and Alex Yurkowski are the founders of ClickUp. They join the show to talk about their experience building the company. We talk through their process of scaling the infrastructure, and their philosophy of moving fast. This episode has some useful strategic advice for anyone who is looking to take a product to market and iterate quickly–even if that product is bootstrapped. Full disclosure: ClickUp is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Infrastructure-as-code allows developers to use programming languages to define the architecture of their software deployments, including servers, load balancers, and databases. There have been several generations of infrastructure-as-code tools. Systems such as Chef, Puppet, Salt, and Ansible provided a domain-specific imperative scripting language that became popular along with the early growth of Amazon Web Services. Hashicorp’s Terraform project created an open source declarative model for infrastructure. Kubernetes YAML definitions are also a declarative system for infrastructure as code. Pulumi is a company that offers a newer system for infrastructure as code, combining declarative and imperative syntax. Pulumi programs can be written in TypeScript, Python, Go, or .NET. Joe Duffy is the CEO of Pulumi, and he joins the show to talk about his work on the Pulumi project and his vision for the company. Joe also discusses his twelve years at Microsoft, and how his work in programming language tooling shaped how he thinks about building infrastructure-as-code. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software investing requires a deep understanding of the market, and an ability to predict what changes might occur in the near future. At the level of core infrastructure, software investing is particularly difficult. Databases, virtualization, and large scale data processing tools are all complicated, highly competitive areas. As the software world has matured, it has become apparent just how big these infrastructure companies can become. Consequently, the opportunities to invest in these infrastructure companies have become highly competitive. When a venture capital fund invests into an infrastructure company, the fund will then help the infrastructure company bring their product to market. This involves figuring out the product design, the sales strategy, and the hiring roadmap. A strong investor will be able to give insight into all of these different facets of building a software company. Vivek Saraswat is a venture investor with Mayfield, a venture fund that focuses on early to growth-stage investments. Vivek joins the show to discuss his experience at AWS, Docker, and Mayfield, as well as his broad lessons around how to build infrastructure companies today. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A high volume of data can contain a high volume of useful information. That fact is well understood by the software world. Unfortunately, it is not a simple process to surface useful information from this high volume of data. A human analyst needs to understand the business, formulate a question, and determine what metrics could reveal the answer to such a question. Sisu is a system for automatically surfacing insights from large data sets within companies. A user of Sisu can select a database column that they are interested in learning more about, and Sisu will automatically analyze the records in the database to look for trends and relationships between that column and the other columns. For example, if I have a database of user purchases, including how much money those users spent on each purchase, I can ask Sisu to analyze the purchase price column, and find what kinds of attributes correlate with a high purchase price. Perhaps there will be correlations such as age and city that I can use to understand my customers better. Sisu can automatically surface these correlations and display them to me to help me make business decisions. Peter Bailis is the CEO of Sisu Data and an assistant professor at Stanford. Peter joins the show to give his perspective on the development of Sisu, which came out of his research on data-intensive systems, including MacroBase, an analytic monitoring engine that prioritizes human attention. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Physical places have a large amount of latent data. Pick any location on a map, and think about all of the questions you could ask about that location. What businesses are at that location? How many cars pass through it? What is the soil composition? How much is the land on that location worth? The world of web-based information has become easy to query. We can use search engines like Google, as well as APIs like Diffbot and Clearbit. Today, the physical world is not so easy to query, but it is becoming easier. Location data as a service is a burgeoning field, with some vendors offering products for satellite data, foot traffic, and other specific location-based domains. SafeGraph is a company that provides location data-as-a-service. SafeGraph data sets include data about businesses, patterns describing human movement, and geometric representations describing the shape and size of buildings. Ryan Fox Squire develops data products for SafeGraph, and he joins the show to talk about the engineering and strategy that goes into building a data-as-a-service company. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Descript is a software product for editing podcasts and video. Descript is a deceptively powerful tool, and its software architecture includes novel usage of transcription APIs, text-to-speech, speech-to-text, and other domain-specific machine learning applications. Some of the most popular podcasts and YouTube channels use Descript as their editing tool because it provides a set of features that are not found in other editing tools such as Adobe Premiere or a digital audio workstation. Descript is an example of the downstream impact of machine learning tools becoming more accessible. Even though the company only has a small team of machine learning engineers, these engineers are extremely productive due to the combination of APIs, cloud computing, and frameworks like TensorFlow. Descript was founded by Andrew Mason, who also founded Groupon and Detour, and Andrew joins the show to describe the technology behind Descript and the story of how it was built. It is a remarkable story of creative entrepreneurship, with numerous takeaways for both engineers and business founders. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Lyft is a ridesharing company that generates a high volume of data every day. This data includes ride history, pricing information, mapping, routing, and financial transactions. The data is stored across a variety of different databases, data lakes, and queueing systems, and is processed at scale in order to generate machine learning models, reports, and data applications. Data workflows involve a set of interconnected systems such as Kubernetes, Spark, Tensorflow, and Flink. In order for these systems to work together harmoniously, a workflow manager is often used to orchestrate them together. A workflow platform lets a data engineer have a high-level view into how data moves through the system, and can be used to reason about retries, resource utilization, and scalability. Flyte is a data processing system built and open-sourced at Lyft. Allyson Gale and Ketan Umare work at Lyft, and they join the show to talk about how Flyte works, and why they needed to build a new workflow processing system when there are already tools available such as Airflow. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud computing caused a fundamental economic shift in how software is built. Before the cloud, businesses needed to buy physical servers in order to operate. There was an up-front cost that often amounted to tens of thousands of dollars required to pay for these servers. Cloud computing changed the up-front capital expense to an ongoing operational expense, with businesses increasingly shifting to Amazon Web Services, Microsoft Azure, and Google Compute Platform. Although the initial motivation for moving onto cloud providers might have been decreased cost, over time the cloud providers have developed unique services that make software even easier to build than before. There has also been a proliferation of new software infrastructure companies that have been built on top of the cloud providers, giving rise to new databases, logging companies, and platform-as-a-service products. Danel Dayan is a venture investor with Battery Ventures and a co-author of the State of the OpenCloud 2019, a report that compiles a wide set of statistics and information on how cloud computing and open source are impacting the software industry. Danel joins the show to talk about his work as an investor, as well as his previous career at Google, where he worked on mergers and acquisitions. If you want to reach Danel you can email him at ddayan@battery.com or tweet at him via @daneldayan. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

GraphQL is a system that allows frontend engineers to make requests across multiple data sources using a simple query format. In GraphQL, a frontend developer does not have to worry about the request logic for individual backend services. The frontend developer only needs to know how to issue GraphQL requests from the client, and these requests are handled by a GraphQL server. GraphQL is mostly used to issue queries across internal databases and services. But many of the data sources that a company needs to query in modern infrastructure are not databases–they are APIs like Salesforce, Zendesk, and Stripe. These API companies might store a large percentage of the data that a given company needs to query, and executing queries, subscriptions, and joins against these APIs is not a simple task. OneGraph is a company that builds integrations with third-party services and exposes them through a GraphQL interface. Sean Grove is a founder of OneGraph, and he joins the show to explain the problem that OneGraph solves, how OneGraph is built, and some of the difficult engineering challenges required to design OneGraph. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A data warehouse serves the purpose of providing low latency queries for high volumes of data. A data warehouse is often part of a data pipeline, which moves data through different areas of infrastructure in order to build applications such as machine learning models, dashboards, and reports. Modern data pipelines are often associated with the term “ELT” or Extract, Load, Transform. In the “ELT” workflow, data is taken out of a source such as a data lake, loaded into a data warehouse, and then transformed within the data warehouse to create materialized views on the data. Data warehouse queries are usually written in SQL, and for the last 50 years, SQL has been the primary language for executing these kinds of queries. DBT is a system for data modeling that allows the user to write queries that involve a mix of SQL and a templating language called Jinja. Jinja allows the analyst to blend imperative code along with the declarative SQL. Tristan Handy is the CEO of Fishtown Analytics, the company that created DBT, and he joins the show to discuss how DBT works, and the role it plays in modern data infrastructure. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

ReactJS developers have lots of options for building their applications, and those options are not easy to work through. State management, concurrency, networking, and testing all have elements of complexity and a wide range of available tools. Take a look at any specific area of JavaScript application development, and you can find highly varied opinions. Kent Dodds is a JavaScript teacher who focuses on React, JavaScript, and testing. In today’s episode, Kent provides best practices for building JavaScript applications, specifically React. He provides a great deal of advice on testing, which is unsurprising considering he owns TestingJavaScript.com. Kent is an excellent speaker who has taught thousands of people about JavaScript, so it was a pleasure to have him on the show. Kent is also speaking at Reactathon, a San Francisco JavaScript conference taking place March 30th and 31st in San Francisco. This week we will be interviewing speakers from Reactathon, and if you are interested in JavaScript and the React ecosystem then stay tuned, and if you hear something you like, you can check out the Reactathon conference in person. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

JavaScript fatigue. This phrase has been used to describe the confusion and exhaustion around the volume of different tools required to be productive as a JavaScript developer. Frameworks, package managers, typing systems, state management, GraphQL, and deployment systems–there are so many decisions to make. In addition to the present-day tooling choices, a JavaScript developer needs to watch the emerging developments in the ecosystem. ReactJS is evolving at a rapid clip, and newer primitives such as React Hooks and React Suspense allow developers to handle concurrency and networking more robustly. Tejas Kumar works with G2i, a company that connects React developers with organizations that are looking for high-quality engineers. His role at G2i is head of vetting, which requires him to assess engineers for their competency in JavaScript-related technologies. Tejas joins the show to discuss the modern stack of technologies that a React developer uses to build an application. Full disclosure: G2i, where Tejas works, is a sponsor of Software Engineering Daily. Tejas is also speaking at Reactathon, a San Francisco JavaScript conference taking place March 30th and 31st in San Francisco. This week we will be interviewing speakers from Reactathon, and if you are interested in JavaScript and the React ecosystem then stay tuned, and if you hear something you like, you can check out the Reactathon conference in person. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Full-stack JavaScript applications have been possible since the creation of NodeJS in 2009. Since then, the best practices for building and deploying these applications have steadily evolved with the technology. ReactJS created consolidation around the view layer. The emergence of AWS Lambda created a new paradigm for backend execution. Serverless tools such as DynamoDB offer autoscaling abstractions. CDNs such as Cloudflare and Fastly can now do processing on the edge. Brian LeRoux is the founder of Begin.com, a hosting and deployment company built on serverless tools. He’s also the primary committer to Architect, a framework for defining applications to be deployed to serverless infrastructure. Brian joins the show to talk about his work in the JavaScript ecosystem and his vision for Begin.com. Brian is also speaking at Reactathon, a San Francisco JavaScript conference taking place March 30th and 31st in San Francisco. This week we will be interviewing speakers from Reactathon, and if you are interested in JavaScript and the React ecosystem then stay tuned, and if you hear something you like, you can check out the Reactathon conference in person. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

ReactJS began to standardize frontend web development around 2015. The core ideas around one-way data binding, JSX, and components caused many developers to embrace React with open arms. There has been a large number of educators that have emerged to help train developers wanting to learn React. A new developer learning React has numerous questions around frameworks, state management, rendering, and other best practices. In today’s episode, those questions are answered by Ryan Florence, a co-founder of React Training. React Training is a company devoted to helping developers learn React that trains large companies like Google and Netflix how to use React. Ryan has a strong understanding of how to be productive with React, and in today’s episode, he explains some of the fundamentals that commonly confuse new students of React. Ryan is also speaking at Reactathon, a San Francisco JavaScript conference taking place March 30th and 31st in San Francisco. This week we will be interviewing speakers from Reactathon, and if you are interested in JavaScript and the React ecosystem then stay tuned, and if you hear something you like, you can check out the Reactathon conference in person. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When ReactJS became popular, frontend web development became easier. But React is just a view layer. Developers who came to React expecting a full web development framework like Ruby on Rails or Django were required to put together a set of tools to satisfy that purpose. A full-stack JavaScript framework has numerous requirements. How does it scale? How does it handle server-side rendering versus client-side rendering? Should GraphQL be included by default? How should package management work? Guillermo Rauch is the creator of NextJS, a popular framework for building React applications. He is also the CEO of ZEIT, a cloud hosting company. Guillermo joins the show to discuss NextJS, and his vision for how the React ecosystem will evolve in the near future, as features such as React Suspense and Concurrent Mode impact the developer experience. Guillermo is also speaking at Reactathon, a San Francisco JavaScript conference taking place March 30th and 31st in San Francisco. This week we will be interviewing speakers from Reactathon, and if you are interested in JavaScript and the React ecosystem then stay tuned, and if you hear something you like, you can check out the Reactathon conference in person. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Slack is a messaging application with millions of users. The desktop application is an Electron app, which is effectively a web browser dedicated to running Slack. This frontend is built with ReactJS and other JavaScript code, and the application is incredibly smooth and reliable, despite its complexity. When a user boots up Slack, the application needs to figure out what data to fetch and where to fetch it from. Companies that use Slack heavily have thousands of messages in their history, and Slack needs to determine which of those should be pulled into the client. There are profile images, and logos, and custom emojis, all of which are used to define the user’s custom workspace experience. Anuj Nair joined Slack in late 2017. In the years since he has been with the company, Anuj helped rewrite the Slack frontend client, including work on the bootup experience, the caching infrastructure, and the role of service workers. Anuj joins the show to discuss his work on the Slack frontend architecture and the canonical view layer problems that Slack faces. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Low code tools can be used to build an increasing number of applications. Knowledge workers within a large corporation can use low code tools to augment their usage of spreadsheets. Entrepreneurs can use low code tools to start businesses even without knowing how to code. Modern low code tools have benefited from steady improvements in cloud infrastructure, front-end frameworks like ReactJS, and browser technology such as the V8 JavaScript engine. These building blocks led to the popular low code products such as Webflow, Bubble, Retool, and Airtable. The low code products are supported by a broad selection of domain-specific APIs such as Stripe, Twilio, and Zapier. Ben Tossell runs Makerpad, a site devoted to low-code and no-code applications. Makerpad describes how to use these tools to design sophisticated applications that don’t require you to write code. But they do require a different kind of software engineering. To create applications intelligently with low-code tools, you need to know how the tools fit together, and you need to be willing to persist through a process of iteration and debugging that is similar to traditional software engineering. Ben joins the show to talk about his experience building low-code tools, the use cases for these tools, and his predictions for how they will impact the future of software. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every company has a large number of routine data workflows. These data workflows involve spreadsheets, CSV files, and tedious manual work to be done by a knowledge worker. For example, data might need to be taken from Salesforce, filtered for new customers, and piped into Mailchimp. Or perhaps you need to sort all your customers to find only the ones who have spent more than $50. These data workflows might require some basic knowledge of SQL, or an understanding of how to make an API request. Not everyone knows how to execute these technical commands. A software company can be slowed down due to a shortage of technical analysts who have the necessary programming skills to build these data workflows. Parabola is a low-code tool for building data workflows. Parabola lets the user drag and drop different components together to build an application without using a programming language. Parabola lowers the technical barrier for knowledge workers who want to build these kinds of data workflows. Alex Yaseen is the CEO of Parabola, and he joins the show to talk about the ideas behind Parabola and his goals with the company. parabola.io parabola.io/careers Twitter: @alexyaseen Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cryptocurrencies today serve two purposes: store of value and speculation. The application infrastructure that has been built around cryptocurrency is mostly to support these use cases. At some point in the future, perhaps cryptocurrencies can be used as a global medium of exchange that is accepted at the grocery store. Perhaps we will use the blockchain for supply chain management, and as a universal ledger for real estate ownership. But today, cryptocurrencies are mostly used for speculative trading. Users buy and sell different cryptocurrencies and stablecoins, looking to make short-term profits. And the markets for trading cryptocurrencies have evolved to have a sophistication that looks like the centralized markets of derivatives and leverage-based day trading. The term “decentralized finance” refers to this phenomenon of cryptocurrency lending markets. Decentralized finance increases the volume of speculated capital by providing liquidity through smart contracts. This short-term liquidity is often collateralized by a volatile cryptocurrency such as Ethereum, creating an opportunity for a type of market participant called a “liquidator.” Tom Schmidt is an investor with Dragonfly Capital, a cryptoasset investment firm. Tom joins the show to describe the dynamics of decentralized finance. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At Airbnb, infrastructure management is standardized across the organization. Platform engineering teams build tools that allow the other teams throughout the organization to work more effectively. A platform engineering team handles problems such as continuous integration, observability, and service discovery. Other teams throughout a company use the tools that a platform engineering team builds. For example, there is a team at Airbnb that builds the search and discovery system that is used by customers who are looking for a place to stay. That team does not want to have to worry about how they are deploying, how their service is being logged, and how to scale up. All of that should be taken care of by the platform engineering team. At a large company like Airbnb, there is so much happening across the infrastructure. Services are being deployed, services are having outages, databases are being resharded. With all of this change occurring, it can be difficult for a team to pinpoint the cause of a service outage. Digging through logs and dashboards is often insufficient. Joey Parsons is the founder of Effx, a company that is building a platform for observing and managing the changes across the infrastructure. Effx is like a newsfeed for a service. An application instrumented with Effx gives the engineers a single endpoint that they can navigate to for understanding the history of their service. Joey joins the show to talk about his experience as an infrastructure engineer at Airbnb, and how that experience informs the work of his new company, Effx. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A gig economy application generates lots of notifications. There is SMS, mobile phone updates, emails, and native application updates. If you order a ride from Uber, you might receive a text message and a push notification at the same time. If an app overloads the user with notifications, the user might end up annoyed and delete the app from their phone. But perhaps all of these notifications are necessary. You would rather get three simultaneous notifications from your food delivery app than fail to get your food on time. If you are the mobile application developer building the food delivery app, what other choice do you have? At large companies such as Linkedin, there are entire teams devoted to figuring out how to optimize the notifications that they send you. It has a surprisingly large impact on the usability of a mobile application. Troy Goode is the founder of Courier, a company that provides notification optimization. This might sound like a small, trivial problem. But it actually has a large impact on the usage of apps. And it is not an easy engineering problem. Troy joins the show to talk about the problem that Courier solves and the backend infrastructure that powers it. Courier is built entirely on serverless APIs. This is a great case study in how to build a completely scalable infrastructure product based on serverless tools. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In a modern data platform, distributed streaming systems are used to read data coming off of an application in real-time. There are a wide variety of streaming systems, including Kafka Streams, Apache Samza, Apache Flink, Spark Streaming, and more. When Eric Anderson joined the show back in 2016, he was working at Google on Google Cloud Dataflow, a managed service for handling streaming data. Today, he works as an investor at Scale Venture Partners. In his current job, he analyzes companies built around data infrastructure, developer tooling, and other enterprise engineering domains. Eric also hosts the podcast Contributor, which explores open source maintainers and the stories of their projects. His podcast has featured the creators of projects such as Envoy, Alluxio, and Chef. In today’s episode, Eric returns to the show to discuss data infrastructure, investing, and the evolving world of open source. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Distributed stream processing frameworks are used to rapidly ingest and aggregate large volumes of incoming data. These frameworks often require the application developer to write imperative logic describing how that data should be processed. For example, a high volume of clickstream data that is getting buffered to Kafka needs to have a stream processing system evaluate that data to prepare it for a data warehouse, Spark, or some other queryable environment. In practice, many developers simply want to have the high volume of data become queryable in the fewest number of steps possible. Materialize is a streaming SQL materialized view engine that provides materialized views over streaming data. The materialized views are incrementally updated over time and reconciled with new data that may have come in out of order. Arjun Narayan and Frank McSherry are the co-founders of Materialize, a company whose technology is based on the Naiad paper, which was written at Microsoft Research. Arjun and Frank join the show to talk about modern streaming systems and their strategy for taking an academic paper and productizing it. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A cloud provider gives developers access to virtualized server infrastructure. When a developer rents this infrastructure via an API call, a virtual server is instantiated on physical machines. That virtual server needs to be made addressable through the allocation of an IP address to make it reachable from the open Internet. When the virtual server starts to receive too much traffic, that traffic needs to be load balanced with another virtual server. The backend networking code that runs a cloud provider needs to be fast, secure, and memory-efficient. Languages that fit that description include C++, Rust, and Go. Digital Ocean’s low-level networking code is mostly written in Go. Sneha Inguva is an engineer with Digital Ocean who has written and spoken about writing networking applications using Go. She joins the show to talk about her work at Digital Ocean, including the implementation of a DHCP server, a network server that assigns IP addresses and other parameters to devices that sit on that network. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A data pipeline is a series of steps that takes large data sets and creates usable results from them. At the beginning of a data pipeline, a data set might be pulled from a database, a distributed file system, or a Kafka topic. Throughout a data pipeline, different data sets are joined, filtered, and statistically analyzed. At the end of a data pipeline, data might be put into a data warehouse or Apache Spark for ad-hoc analysis and data science. At this point, the end-user of the data set expects that data to be clean and accurate. But how do we have any guarantees about the correctness? Abe Gong is the creator of Great Expectations, a system for data pipeline testing. In Great Expectations, the developer creates tests called “expectations”, which verify certain characteristics of the data set at different phases in a data pipeline. This helps ensure that the end result of a multi-stage data pipeline is correct. Abe joins the show to discuss the architecture of a data pipeline and the use cases of Great Expectations. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A data warehouse provides low latency access to large volumes of data. A data warehouse is a crucial piece of infrastructure for a large company, because it can be used to answer complex questions involving a large number of data points. But a data warehouse usually cannot hold all of a company’s data at any given time. Users need to move a subset of the data into the data warehouse by reading large files from a data lake on disk and putting that data into the data warehouse. The process of moving data from one place into another is broken down into three sequential steps, often called “ETL” (extract, transform, load) or “ELT” (extract, load, transform). In ETL, the data is extracted from a source such as a data lake, transformed into a schema that is customized for the data warehouse application, and then loaded into the data warehouse. In ELT, the last two steps are reversed, because modern systems can often leave the necessary schema transformation until after the data has been loaded into the data warehouse. Matthew Scullion is the CEO of Matillion, a company that specializes in building tools for data transformations. Matthew joins the show to talk about the problem of data transformation, and how that problem has evolved over the nine years since he started Matillion. If you enjoy the show, you can find all of our past episodes about data infrastructure by going to SoftwareDaily.com and searching for the technologies or companies mentioned. And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning applications are widely deployed across the software industry. Most of these applications used supervised learning, a process in which labeled data sets are used to find correlations between the labels and the trends in that underlying data. But supervised learning is only one application of machine learning. Another broad set of machine learning methods is described by the term “reinforcement learning.” Reinforcement learning involves an agent interacting with its environment. As the model interacts with the environment, it learns to make better decisions over time based on a reward function. Newer AI applications will need to operate in increasingly dynamic environments, and react to changes in those environments, which makes reinforcement learning a useful technique. Reinforcement learning has several attributes that make it a distinctly different engineering problem than supervised learning. Reinforcement learning relies on simulation and distributed training to rapidly examine how different model parameters could affect the performance of a model in different scenarios. Ray is an open source project for distributed applications. Although Ray was designed with reinforcement learning in mind, the potential use cases go beyond machine learning, and could be as influential and broadly applicable as distributed systems projects like Apache Spark or Kubernetes. Ray is a project from the Berkeley RISE Lab, the same place that gave rise to Spark, Mesos, and Alluxio. The RISE Lab is led by Ion Stoica, a professor of computer science at Berkeley. He is also the co-founder of Anyscale, a company started to commercialize Ray by offering tools and services for enterprises looking to adopt Ray. Ion Stoica returns to the show to discuss reinforcement learning, distributed computing, and the Ray project. If you enjoy the show, you can find all of our past episodes about machine learning, data, and the RISE Lab by going to SoftwareDaily.com and searching for the technologies or companies you are curious about . And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Distributed stream processing systems are used to read large volumes of data and perform operations across those data streams. These stream processing systems often build off of the MapReduce algorithm for collecting and aggregating large volumes of data, but instead of processing a calculation over a single large batch of data, they process data on an ongoing basis. There are so many different stream processing system for this same use case–Storm, Spark, Flink, Heron, and many others. Why is that? When there seems to be much more consolidation around the Hadoop MapReduce batch processing technology, why are there so many stream processing systems? One explanation is that aggregating the results of a continuous stream of data is a process that very much depends on time. At any given point in time, you can take a snapshot of the stream of data, and any calculation based on that data is going to be out of date by the time that your calculation is finished. There is a latency between when you start calculating something, and when you finish calculating it. There are other design decisions for a distributed stream processing system. What data do you keep in memory? What do you keep on disk? How often do you snapshot your data to disk? What is the method for fault tolerance? What are the APIs for consuming and processing this data? Maximilian Michels has worked on the Apache Flink and Apache BEAM stream processing systems, and currently works on data infrastructure at Lyft. Max joins the show to discuss the tradeoffs of different stream processing systems and his experiences in the world of data processing. You can find all of our past episodes about data infrastructure by going to SoftwareDaily.com and searching for the technologies or companies mentioned. And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Large companies generate large volumes of data. This data gets dumped into a data lake for long-term storage, then pulled into memory for processing and analysis. Once it is in memory, it is often read into a dashboard, which presents a human with a visualization of the data. The end-user who is consuming this data is often a data scientist who is looking at the data to find trends and design new machine learning models. Another kind of user is the operational analyst. An operational analyst is creating complex queries across this data to find latencies in the infrastructure, or perhaps slicing and dicing clickstream data that is coming from online advertisements, in order to figure out how to tweak those advertising algorithms and spend money more effectively. For an operational analyst, a key use case for a data warehouse is fast, interactive querying. The operational analyst needs to be able to query the data to quickly create a dashboard, make judgments based on that dashboard, and then change the query slightly to look at a slightly different dashboard. Druid is a high-performance database that is used for these kinds of queries. Druid is used for ad-hoc queries and operational analytics. Imply Data is a company that builds visualization, monitoring, and security around Druid. Jad Naous is vice president of R&D for Imply, and he joins the show to talk about the use case for Druid, the architecture, and the business model of Imply. If you enjoy the show, you can find all of our past episodes about data infrastructure by going to SoftwareDaily.com and searching for the technologies or companies mentioned. And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data infrastructure has been transformed over the last fifteen years. The open source Hadoop project led to the creation of multiple companies based around commercializing the MapReduce algorithm and Hadoop distributed file system. Cheap cloud storage popularized the usage of data lakes. Cheap cloud servers led to wide experimentation for data tools. Apache Spark emerged from academia, and Apache Kafka came out of the corporate challenges faced by LinkedIn. Over these 15 years, Ben Lorica has been following the world of data engineering as an engineer, a conference organizer, and a podcaster. When he was host of the O’Reilly Data Show, his material served as inspiration for some of the episodes of this podcast. Today he hosts The Data Exchange podcast and writes The Data Exchange newsletter. Ben joins the show to talk about modern data engineering, and his opinion on the past and future of data infrastructure. If you enjoy the show, you can find all of our past episodes about data infrastructure by going to SoftwareDaily.com and searching for the technologies or companies mentioned. And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A data platform contains all of the data that a company has accumulated over the years. Across a data platform, there is a multitude of data sources: databases, a data lake, data warehouses, a distributed queue like Kafka, and external data sources like Salesforce and Zendesk. A user of the data platform often has a question that requires multiple data sources to answer. How does this user join two data sources from a data lake? How does this user join data across a transactional database and a data lake? How does the user join data from two different data warehouse technologies? Presto is an open source tool originally developed at Facebook. Presto allows a user to query a data platform with a SQL statement. That query gets parsed and executed across the data platform to read from any heterogeneous data source. For some use cases, Presto is replacing the technology Hadoop MapReduce-based technology Hive. For other use cases, Presto is solving a problem in a completely novel way. Justin Borgman joins the show to discuss the motivation for Presto, the problems it solves, and the architecture of Presto. He also talks about the company he started, Starburst Data, which sells and supports technologies built around Presto. If you enjoy the show, you can find all of our past episodes about data infrastructure by going to SoftwareDaily.com and searching for the technologies or companies mentioned. And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Nubank is a popular bank that is based in Brazil. Nubank has more than 20 million customers, and has accumulated a high volume of data over the six years since it was started. Mobile computing and cloud computing have given rise to “challenger banks” that operate more like software companies. When a software company reaches the size that Nubank is at today, it needs a data platform. A data platform is a collection of different technologies that move data into different storage formats and applications, so that different members of an organization can access that data. New data often enters an organization through an OLTP database, which supports user transactions. That data is copied into a data lake, which provides cheap bulk storage. From the data lake, the data is moved into a data warehouse system for fast access. Along the way, tools like Kafka, Spark, and S3 are used to implement the needs of the data platform. Data platform architecture is not an exact science. Different companies build their data platform based on their own unique requirements. Previous shows have covered the data infrastructure companies like Lyft, Uber, and Facebook. Today’s show is another case study in data infrastructure, with a modern bank. In a previous episode, we covered the engineering of Nubank. Sujith Nair from Nubank joins today’s show to talk about the data infrastructure of the company. If you enjoy the show, you can find all of our past episodes about data infrastructure by going to SoftwareDaily.com and searching for the technologies or companies mentioned. And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Changelog is a podcast about the world of open source. As open source has become closely tied with the entire software development lifecycle, The Changelog has expanded its coverage to the broader software industry. Since starting the podcast ten years ago, Adam Stacoviak and Jerod Santo have become full-time podcasters, and they have started several other podcasts within the Changelog network, including Go Time, JS Party, and Practical AI. Throughout all of their shows, there is a consistent theme of technical, entertaining conversations about software. In the last decade, so much has changed within open source: GitHub became the de facto social network for open source; Kubernetes created a widely used platform for distributed systems; React has given frontend developers a component system to consolidate around. Adam and Jerod return to the show to discuss their perspective on the past and future of open source, and their learnings from interviewing influential software professionals for 10 years. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Animations can be used to create games, app tutorials, and user interface components. Animations can be seen in messaging apps, where animated reactions can convey rich feelings over a text interface. Loading screens can become less boring through animation, and voice assistant products can feel more alive through animation. But we still don’t see much animation in our everyday applications. This is partly because animation tooling is difficult to use. To make an animation, the typical workflow is to go into a tool like After Effects, render your animation, and then export that animation in a movie format. This format is not dynamic enough to be easily used on the wide variety of development platforms. The animation library Lottie did improve this tooling by creating a system for exporting animations to JSON and allowing them to easily scale up and down as vectors. But the animations still were simple and unidirectional. The developer did not have much freedom for how to move an animation in response to user input. Rive is a system for creating dynamic, movable animated objects. Rive allows for the creation of animated elements that respond to user input. Rive has a tool that runs in the browser and allows the user to define the animation. The animations in Rive use a bone system that allows animators and designers to define the points of the animated sprite that the developer can then manipulate with code. This improves the painful handoff process that exists between animators and developers, and gives the developer some programmatic control. Guido and Luigi Rosso are the founders of Rive and they join the show to talk about the frictions of animation tooling, and what they have built to improve Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Robotics has changed modern agriculture. Autonomous systems are powering the tractors, cotton pickers, and corn cutters that yield plants at industrial scale. John Deere is a company that has been making farm equipment for 183 years. Over that period, the planting and harvesting process has become increasingly mechanized, and John Deere has been at the forefront. Over the last few decades, software has played an increasingly important role at John Deere. Today, there is software inside the vehicles. These vehicles can operate autonomously, they collect large amounts of data, and they are supported by a large system of cloud services. The teams within John Deere who create the software have an elaborate testing workflow that allows them to deploy the software to the vehicles and drive them in the field. Ryan Bergman is a software engineer at John Deere and he joins the show to talk about the software engineering, management, and DevOps practices within the company. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Venture capital investing requires an understanding of market dynamics, technology, and finance. There is also an element of human nature. Consumer trends can make or break the viability of a new product. And early stage venture investing is always a bet on a small team or individual founder. Early stage investments are usually into companies that have not found perfect traction with their product. Judging the worth of an early stage investment means judging the likelihood that the founders can make their vision a reality. Venture Stories is a podcast that explores the wide spectrum of ideas that go into venture investing. Shows include two person interviews on economics, social networking, food technology, cryptocurrencies, and consumer psychology. Erik Torenberg is a co-founder and partner of Village Global, an early stage venture capital firm. He is also the host of Venture Stories. Erik joins the show to discuss investing, media, and the kinds of new technology companies that are being created today. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Stock trading takes place across a variety of software platforms. Etrade and Schwab have allowed individual traders to buy securities for decades. Robinhood built a business around a similar model, but also removed the commission. Wealthfront and Betterment provide “roboadvisor” services that abstract away the underlying securities and focus on managing a risk profile. Each of these services has a programmatic execution system for managing assets. In order for a developer to build a product like Robinhood or Wealthfront, that developer needs access to an API that can execute trades. Alpaca is an API for stock trading. Alpaca can be used to build financial products, apps, and algorithmic trading programs. Yoshi Yokokawa is the founder of Alpaca, and he joins the show to talk about why he built an API for trading and the potential applications of Alpaca. Yoshi’s background includes work in finance at Lehman Brothers, a period spent as an individual day trader, and the previous company he started selling custom trading algorithms to enterprises. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Large software companies have lots of users, and the activity from those users results in high volumes of traffic. These companies also have a large surface area across the enterprise. There are hundreds of services and databases that are fulfilling user requests. As these requests enter the infrastructure of the enterprise, the requests travel through the different services and result in database queries, payments, and other transactions. These transactions result in the generation of log messages. The log messages tell the story of what is happening across the entire company. Log messages can provide valuable data for security and site reliability engineering. But analyzing a high volume of log data requires a scalable system that can account for that high volume. Jack Naglieri is the CEO of Panther Security. He previously worked at Airbnb, where he helped develop a system called StreamAlert. At Airbnb, log messages are buffered into distributed queueing systems like Kafka or Kinesis, and they are written to bucket storage systems like S3. Those logs are processed by AWS Lambda functions that test the log messages for rules defined by a system operator. Jack left Airbnb and started Panther Security to generalize the tools he built within Airbnb and build a company around the same ideas. Jack joins the show to discuss modern logging infrastructure, his work at Airbnb, and his experience building Panther. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Distributed systems are required to run most modern enterprise software. Application services need multiple instances for scalability and failover. Large databases are sharded onto multiple nodes. Logging services, streaming frameworks, and continuous integration tools all require the orchestration of more than one server. Deploying a distributed system has historically been difficult because the nodes of the system must be managed by the underlying infrastructure. If I have a distributed database that I want to deploy, the complexity of that deployment is going to be different depending on whether I am running on AWS, or VMware, or my own bare metal server infrastructure. Heterogeneous server infrastructure makes it hard to sell distributed applications that get deployed to that infrastructure. A vendor that is selling a distributed database would need to figure out how to make their database work on the infrastructure of any given customer. Kubernetes has simplified the process of deploying a distributed application. Kubernetes is a container orchestration system that has steadily grown in popularity, to the point where the ecosystem is mature and the software is stable. Now that the software industry has a reliable, portable means of deploying a distributed application, the enterprise software market is becoming easier to enter for companies that sell a distributed application. Replicated is a company that builds products for software delivery. Replicated allows for the distribution and updating of applications that would have been hard to deploy in the past. Grant Miller and Marc Campbell are the CEO and CTO of Replicated, and they join the show to talk about the modern enterprise software market, and the process of delivering software to companies that might otherwise have trouble consuming it. Full disclosure: Replicated is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Chat systems have been a part of software development for decades. Older systems like Pidgin and Yammer were surpassed by newer systems like HipChat. And when Slack was created, it quickly became a part of most software companies. But Slack does not fulfill the needs of every company. Mattermost is an open-source chat system. Mattermost can be configured to work within enterprises that have strong constraints around compliance and data governance. Whereas Slack is a SaaS product that requires users to send their data to the cloud servers managed by Slack, Mattermost allows the enterprise to decide how data moves through services, and where the databases are hosted. Ian Tien is the CEO of Mattermost, and he joins the show to talk about why many companies need their chat system to be hosted in a private cloud or on-premises. In a previous episode with CTO Corey Hulen we discussed the engineering behind the company. In today’s episode, we explore the management and strategy of the business, as well as some additional engineering, since Ian Tien’s background is as a software engineer and computer scientist. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The word “DevOps” has a different definition depending on who you ask. For some people, it is about the process of managing and releasing code. It can involve container management and server orchestration. It can involve infrastructure-as-code, and safer configuration management. In addition to a set of technologies, DevOps can be seen as a management concept that describes agile practices, and breaking down communication barriers between different teams. One thing that most software companies have decided is that whatever DevOps is, we want it. We want to release more software, we want to do it faster, and we want to do it safer. We want streamlined communication between management and engineering. We want a full understanding of the “value chain” of software. Despite the elusiveness of a single description for what DevOps is, GitLab can credibly describe itself as a tool that satisfies the DevOps needs of most enterprises. GitLab started as an open source version control management system based on Git. It has expanded into products that include continuous integration, security, issue tracking, and monitoring. The trajectory of GitLab into such a large platform is something that nobody anticipated. The best explanation for how it happened is that it is the downstream result of an engineer within GitLab deciding that the code hosting product needed to have a continuous integration product bundled with it as an option for a tightly coupled, unified workflow. Today, there are many enterprises trying to make a big set of changes to their development practices. The world is consolidating around Git for version control and Kubernetes for container management. Almost every enterprise is figuring out a “cloud strategy”. Every team wants to have continuous integration, and they want it to have some security products paired with that release workflow in a popular, vaguely defined set of practices known as “DevSecOps”. With so many changes coming to enterprises, it turns out that many of these enterprises just want some sane defaults. When GitLab came to market with a bundled CI and code hosting product, the company discovered that the customers were very happy to have integrated tools that worked well out of the box. This was in stark contrast to the years of NxN tooling integration that an enterprise would have to make to stitch together their broad range of carefully selected tools. Sid Sibrandij is the CEO of GitLab, and he joins the show for a conversation about how GitLab arrived at its product development strategy. In a previous episode, Sid discussed some of the core features and history of GitLab. Today’s show expands on many of the subjects we explored previously. We also had a spirited discussion of the modern nature of work, and how GitLab’s unique culture and fully remote team have evolved as the company has scaled. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The ridesharing infrastructure of Lyft has a high volume of traffic that is mostly handled by servers on AWS. When Vicki Cheung joined Lyft in 2018, the company was managing containers with an internally built container scheduler. One of her primary goals at the company was to move Lyft to Kubernetes. In today’s episode, Vicki gives an overview of Lyft infrastructure and the core engineering problems within the company. One subject she touched on was the network communications between the user on a mobile phone and the cloud backend. This was a topic we explored in detail on a previous episode about Envoy Mobile with Matt Klein. Vicki also discussed the broader Kubernetes ecosystem, as well as her time at OpenAI, where she managed infrastructure deployments for scheduling large machine learning jobs. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

If the Internet was reimagined with the software and hardware infrastructure we have today, what would it look like? That is the question that DFINITY is working on answering. DFINITY’s goal is to build a decentralized, secure Internet computer. DFINITY takes concepts from the cryptocurrency world, but it is focused on computation, not financial products. DFINITY can be thought of as a decentralized cloud provider, with redundancy and scalability properties that are achieved by operating on data centers across the world. DFINITY wants to host web applications such as the ones that we use today on centralized servers. A developer who wants to run their application on DFINITY compiles their code to WebAssembly and deploys it to the DFINITY decentralized runtime. Transactions across DFINITY applications are processed through a collateralized proof-of-stake system to ensure reliable, decentralized computation. DFINITY is an ambitious project, and it would seem nearly impossible to bring to market if not for the quality of the team. DFINITY has hired Andreas Rossburg, a co-designer of WebAssembly, as well as talented engineers across security, web development, and backend infrastructure. Dominic Williams is the president and chief scientist of DFINITY, and he joins the show to talk about the vision for DFINITY and the roadmap to making it a reality. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Webflow is a visual programming tool used by designers, developers, and other technical users. Webflow is a leader in the “low code” or “no code” category of software tools that has become prominent in the last few years. Webflow has been years in the making. In a previous show with Webflow CEO Vlad Magdalin, he told the story of being heads down on Webflow, steadily working through the engineering problems that stood between him and his vision of a visual programming environment. In the early days, it was unclear who would even want to use Webflow. A critic of Webflow might have said that Webflow was too high level for developers, and too technical for designers. But Webflow caters to a large subset of developers and designers who want the kind of low code experience that Webflow provides. The product has also helped define a new category of knowledge worker: the “visual programmer.” Bryant Chou is a co-founder of Webflow and was the CTO in the early days. He joins the show to discuss the engineering problems that the company has had to work through, and his perspective on how Webflow fits into the software market going forward. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software has changed the way the world functions. The rapid pace of change has made it difficult to know how to navigate the new world. Knowledge workers who want to keep advancing in their careers develop a strategy of continuous learning in order to adapt to these changes. O’Reilly Media has existed for almost 40 years, providing resources for the technical consumer. As O’Reilly has expanded its product line from books to conferences to online learning, the business has grown slowly but steadily. That business trajectory stands in contrast to many of the software companies that are financially structured to either grow rapidly or die. Today, O’Reilly has a large impact on the software ecosystem. Software professionals congregate at O’Reilly conferences. Enterprises pay O’Reilly to educate their employees. And O’Reilly continues to grow into new product lines, recently acquiring the interactive learning platform Katacoda, which can be used to learn about Kubernetes and other popular technologies. In a previous episode, we discussed Tim O’Reilly’s book “What’s The Future”. In today’s show, Tim returns to the show to discuss his experience building O’Reilly, and how his business philosophy contrasts with much of the assumed wisdom of software company building. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

GraphQL has become a core piece of infrastructure for many software applications. GraphQL is used to make requests that are structured as GraphQL queries and responded to through a GraphQL server. The GraphQL server processes the query and fetches the response from the necessary databases, APIs, and backend services. Around 2016, when GraphQL was becoming popular, a company called Meteor was deciding what to do with its business. Meteor was started off of the popular framework MeteorJS, a system for building real-time JavaScript applications. MeteorJS was loved by many developers, but Meteor needed to decide if it was the most viable opportunity that it could be pursuing with its resources. From the vantage point within the Meteor company, there were some trends in the frontend ecosystem that were potentially disruptive to the viability of Meteor. There were also some large potential opportunities. The dramatic change to the frontend was largely coming as a downstream effect of Facebook’s open source technologies: React and GraphQL. Amidst these changes, Meteor shifted the company’s efforts entirely towards GraphQL and renamed the company Apollo. Geoff Schmidt is the CEO of Apollo, and he joins the show to talk about the GraphQL ecosystem, the business opportunities around GraphQL, and the process of pivoting from Meteor to Apollo. If you are planning a hackathon, check out FindCollabs Hackathons. Whether you are running an internal hackathon for your company, or you are running an open hackathon so that users can try out your product, FindCollabs Hackathons are a tool for people to build projects and collaborate with each other. You can create your own hackathon at FindCollabs.com. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The JavaScript ecosystem stretches across frontend, backend, and middleware. There are newer tools such as GraphQL, Gatsby, and WebAssembly. There are frameworks like React, Vue, and Angular. There is complex data handling with streams, caches, and TensorFlow.js. JavaScript is unlike any other ecosystem, because a single language can be used to construct every part of an application. Because JavaScript is used for such a broad spectrum of use cases, the amount of tooling available can be intimidating to someone new to the ecosystem. Kevin Ball is a host of JS Party, a podcast on The Changelog network. Kevin joins the show to give his perspective on the JavaScript ecosystem. We discussed ES Modules, the JAM Stack, and the growing number of tools, libraries, and workflows used by JavaScript developers. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud infrastructure is usually consumed in the form of virtual machines or containers. These VMs or containers are running on a physical host machine that is also running other VMs and containers. This is called multitenancy. Servers across cloud providers such as AWS have a high utilization because there are multiple virtual instances running on each physical server host. Cloud computing has led to a low cost of compute infrastructure. But in some cases, this low cost comes at the price of not being able to control the underlying hardware with as much precision as the user would want. Some users want specific types of hardware. Other users want to be using dedicated hardware that does not risk the “noisy neighbor” problem of sharing a physical server with some other application that is using most of the resources. Packet is a company that provides remote access to baremetal infrastructure. The user experience is similar to that of a cloud provider, but with more control over how a given physical host will be used. Zachary Smith is the CEO of Packet and Nathan Goulding is the chief architect. Zach and Nathan join the show to talk about the business and the engineering behind Packet, as well as the future goals for where they want to take the company. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Edge computing is the usage of servers that are geographically close to the client device. The first common use case for edge computing was CDNs: content-delivery networks. A content delivery network placed media files such as images and videos on multiple servers throughout the world. These are big files, and they take lots of bandwidth to transfer. By placing them at CDNs, the files would be closer to any user around the world. These early use cases for edge computing were purely about storing large files. But the vast majority of compute still took place at the central application servers. Over time, users have required faster and faster application experiences. Today, an increasing amount of compute has been moved to the edge, in addition to the existing storage applications. More user data is being cached at the edge to make for quicker transactional processing. Machine learning model training and hosting at the edge makes for a faster, more responsive machine learning feedback loop. Jaromir Coufal is an engineer with Red Hat. He joins the show to talk about modern applications of edge computing, and how the demand for edge computing is creating a market opportunity for companies that have lots of servers at the edge, such as telecoms. These telecoms can repurpose their widely distributed telecom infrastructure as edge servers that they can sell usage on. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every large company generates large amounts of data. Data engineering is the process of storing, transforming, and leveraging that data. Data infrastructure companies provide tools and platforms for performing data engineering. The last fifteen years has seen a rise in modern data management companies built in a time of decreasing storage costs, an increased volume of data, and the prevalence of cloud computing. Modern data companies include Hadoop vendors, cloud providers, and a wide variety of individual software companies offering products such as databases, ETL tools, and open source tooling. The go-to-market strategy for a data infrastructure company requires a deep understanding of the data engineering landscape. A company must build something useful, sell it to customers, and eventually build a replicable strategy. Sean Knapp is the CEO of Ascend, a company that builds Apache Spark-based data pipelines that connect APIs, data lakes, and data warehouses together to enable data applications. Sean joins the show to talk about the process of building a data infrastructure company, and his lessons building Ascend. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Slack is a messaging platform for organizations. Since its creation in 2013, Slack has quickly become a core piece of technology used by a wide variety of technology companies, groups, and small teams. The messages that are sent on Slack are generated at a very high volume, and are extremely sensitive. These messages must be stored on Slack’s servers in a way that does not risk a message from one company accidentally being accessible to another company. The messages must be highly available, and they also must be indexed for search. When Slack was scaling, the company started to encounter limitations in its data infrastructure that the company was unsure how to solve. During this time, Josh Wills was the director of data engineering at Slack, and he joins the show to retell the history of his time at Slack, and why the problem of searching messages was so hard. Josh also provides a great deal of industry context around how engineers from Facebook and Google differ from one another. When Slack was starting to become popular, the company quickly began to attract engineers from both of those companies. Facebook and Google have distinct solutions for how they have tackled the problems of data engineering. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

NoSQL databases provide an interface for storing and accessing data that allows the user to work with data in an “unstructured” fashion. SQL databases require the data in the database to be “normalized,” meaning that each object in the entire database has an entry (or a null value) for each field. One advantage of NoSQL is that the different objects are “denormalized,” meaning that different objects in the database can have unique fields. There is a widely held belief that NoSQL databases do not scale, or that there is some significant penalty that a developer will pay for using a NoSQL database as soon as their app becomes popular. The truth is much more subtle than that. NoSQL databases can perform as well as or better than SQL databases if the developers know the query patterns that their applications make. SQL databases will be a better choice in the condition where the database has a very wide spectrum of access patterns. But in many cases, an application has a narrow range of different requests for the database, and a NoSQL database can perform very well if the database is structured and optimized for these requests. Rick Houlihan is an executive with Amazon Web Services who works with database teams and engineers to optimize their products and database infrastructure. Rick joins the show to discuss the tenets of NoSQL and describe the fundamental contrast between NoSQL and SQL database limitations. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Amazon EC2 (Elastic Compute Cloud) is a virtualized server product that provides the user with scalable compute infrastructure. EC2 was created in 2006 as one of the first three AWS services along with S3 and Simple Queueing Service. Since then, EC2 has provided the core server infrastructure for many of the companies that have been built in the cloud. A large scale virtualization product requires its engineers to have a deep understanding of scheduling and multitenancy. In previous shows, we have touched on subjects such as hypervisors, the noisy neighbor problem, the cold start problem, and other aspects of multitenant infrastructure. To make EC2 successful, these issues must be continuously revisited and resolved at different areas of the stack. Dave Brown joined the EC2 team in 2007, and now leads the EC2 Compute, Networking, and Load Balancing teams as a Vice President. Dave joins the show to discuss the history of EC2 and the canonical problems of virtualized server infrastructure. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Amazon’s container offerings include ECS (Elastic Container Service), EKS (Elastic Kubernetes Service), and Fargate. Through these different offerings, Amazon provides a variety of ways that a user can manage Kubernetes clusters and standalone container instances. The choice of which containerization system to choose depends on the needs of the user, and the tradeoffs they want to make on control and portability. Amazon’s container products have been designed in the context of a shifting competitive landscape. Kubernetes presents a potential long-term threat to Amazon’s status as the most popular cloud provider. Properly responding to this threat has required Amazon to extend itself into the world of open source, contributing to Kubernetes and having more conversations with customers who want their products to have the high quality user experience of AWS along with the open characteristics of Kubernetes. Abby Fuller is a principal technologist with Amazon who works on containers and Linux. Abby joins the show to describe Amazon’s perspective on containers and Kubernetes. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When the Kubernetes project was started, Amazon Web Services was the dominant cloud provider. Most of the code that runs AWS is closed source, which prevents an open ecosystem from developing around AWS. Developers who deploy their application onto AWS are opting into a closed, controlled ecosystem, which has both costs and benefits. The software industry has a history of closed and open ecosystems existing at the same time. AWS represented a huge closed ecosystem. With the amount of money at stake in the cloud business, it was only a matter of time before a more open ecosystem emerged. Since the creation of Kubernetes, the world of cloud computing has evolved rapidly. Google and Microsoft have both invested heavily into Kubernetes, and Amazon itself has adapted to the newer competitive landscape with a Kubernetes offering of its own. Amazon has also made efforts to become more publicly involved in open source projects, including Kubernetes. Kelsey Hightower has been a part of the Kubernetes ecosystem since the project was started. He is one of the most recognizable faces in the world of Kubernetes, delivering keynotes, appearing on podcasts, and co-authoring the popular Kubernetes Up and Running. Kelsey joins the show to discuss the progress in the Kubernetes ecosystem, and the competitive dynamics between Kubernetes and AWS. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

freeCodeCamp was started five years ago with the goal of providing free coding education to anyone on the Internet. freeCodeCamp has become the best place to begin learning how to write software. There are many other places that a software engineer should visit on their educational journey, but freeCodeCamp is the best place to start, because it is free, and there are no advertisements. For most people learning to code, the price of that education is important, because they are learning to code to build a new career. It’s also important that a new programmer learns from an unbiased source of information, because an ad-supported environment will educate the new programmer towards products that they might not need. freeCodeCamp has not been easy to build. Building freeCodeCamp has required expertise in software engineering, business, media, and community development. The donation-based business model of freeCodeCamp doesn’t collect very much money. Why would somebody build a non-profit when they could spend their time building a highly profitable software company? Quincy Larson is the founder of freeCodeCamp, and he joins the show for a special episode about his backstory and the journey to building the best place on the Internet for a new programmer to begin. Support freeCodeCamp

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The software category known as “no-code” describes a set of tools that can be used to build software without writing large amounts of code in a programming language. No-code tools use visual interfaces such as spreadsheets and web based drag-and-drop systems. In previous shows, we have covered some of the prominent no-code products such as Airtable, Webflow, and Bubble. It is clear that no-code tools can be used to build core software infrastructure in a manner that is more abstract than the typical software engineering model of writing code. No-code tools do not solve everything. You can’t use a no-code tool to build a high performance distributed database, or a real-time multiplayer video game. But they are certainly useful for building internal tools and basic CRUD applications. We know that no-code tools can create value. But how do they fit into the overall workflow of a software company? How should teams be arranged now that knowledge workers can build certain kinds of software without writing code? And how should no-code systems interface with the monoliths, microservices, and APIs that we have building for years? Shawn Wang is an engineer with Netlify, a cloud provider that is focused on delivering high-quality development and deployment experience. Netlify is not a no-code platform, but Shawn has explored and written about the potential of no-code systems. Since he comes from a code-heavy background, he is well-positioned to give a realistic perspective on how no-code systems might evolve to play a role in the typical software development lifecycle.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Roblox is a gaming platform with a large ecosystem of players, creators, game designers, and entrepreneurs. The world of Roblox is a three-dimensional environment where characters and objects interact through a physics engine. Roblox is multiplayer, and users can interact with each other over the Internet. Roblox is not one single game—it is a system where anyone can design and monetize their own games within Roblox. Over the last 14 years, Roblox has grown to be massively popular. As the product has grown, the software has evolved to meet changes in consumer demands and engineering constraints. Client devices include mobile phones, desktop computers, and virtual reality. All of these clients must have a consistent experience in graphics and functionality. The backend platform has to support a high volume of concurrent players who are accessing a high volume of content. The networking needs to support multiple players operating in an environment that demands high bandwidth. Claus Moberg is a vice president of engineering at Roblox. He joins the show to discuss the engineering of Roblox and the future of gaming. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cruise is a company that is building a fully automated self-driving car service. The infrastructure of a self-driving car platform presents a large number of new engineering problems. Self-driving cars collect vast quantities of data as they are driving around the city. This data needs to be transferred from the cars onto cloud servers. The data needs to be used for training machine learning models. These models must be tested in a simulated environment, which provides more data to be integrated back into the self-driving system that is deployed to the cars. As the cars drive around the city, they can communicate with custom cloud services to get information about traffic, navigation, and weather. Cloud services are also used for internal tooling that can help with automotive diagnostics, configuration changes, deployments, and security policy management. The software platform used to manage infrastructure at Cruise is a combination of cloud products, open source tools, and custom built infrastructure that is mostly deployed to Kubernetes. Karl Isenberg is an engineer at Cruise, and he joins the show to talk about the engineering requirements of building a self-driving car service, and Cruise’s approach to platform engineering. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The software supply chain includes cloud infrastructure, on-prem proprietary solutions, APIs, programming languages, networking products, and open source software. Each of these software categories has its own security vulnerabilities, and each category has tools that can help protect a company from attackers that are trying to exploit known vulnerabilities. As open source software has grown in popularity, it has turned into an enormous potential attack surface that is difficult to protect. Snyk is a company that builds security tools for companies that are consuming open source. Guy Podjarny is the CEO of Snyk, and he joins the show to discuss the security vulnerabilities of open source projects, and how his business works. Guy was previously the CTO of Akamai, so he has significant experience in technical leadership. He also is the host of the podcast The Secure Developer, which I recommend for anyone who is interested in technical interviews about security topics. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

GitLab is a company that builds an open source platform for managing git repositories. GitLab was started in 2012, and has grown to have a large enterprise business with additional products such as continuous integration and security tooling. GitLab is also known for being a large, entirely remote workforce. GitLab does not have any offices, and the employees mostly communicate through Slack, email, and GitLab itself. Marin Jankovski was the first full-time engineer to join GitLab after the company was started. Marin joins the show to talk about the early days of GitLab, the evolution of the remote culture, and how product development works at GitLab today. He also talks about the experience of being a fast-growing company in the public spotlight of the software industry. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Automation has the potential to eliminate rote jobs such as call center workers and truck drivers. The downstream effects of automation also leads to new jobs, such as data labeling and robot operations. The net effect of modern automation technology is unclear, but it is likely to cause some disruption in the job market. Universal Basic Income (UBI) is an economic policy idea in which a government sends money to every person living in the country. The goal of UBI is to reduce the impact of dramatic changes to the economy that are resulting from accelerating technological change. Floyd Marinescu is the CEO of C4 Media, the company that produces the QCon conference series and the InfoQ website for software engineers. Floyd has worked in the software industry for decades and in recent years has become an advocate for basic income. He is a friend and supporter of Andrew Yang, a 2020 presidential candidate who is running on a platform centered around a basic income policy. Floyd joins the show for a discussion of the future, and the potential positive and negative consequences of implementing a basic income. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Logging provides raw data that can be abstracted into higher level information. Logs are generated at every layer of infrastructure: physical host, virtual machine, container, pod, and Kubernetes cluster. Logs are generated by network proxies, edge servers, and API requests. There is far too much logging information to be read by humans. Log messages need to be refined into statistical metrics that can be put into charts. A high volume of log messages can be used to detect anomalies across a system. If unusual behavior is present in a system, the relevant log messages can be identified and sent to a human operator for them to triage and respond to. Kalyan Ramanathan works at Sumo Logic, a platform for log management and continuous intelligence. Sumo Logic recently published the Continuous Intelligence Report, which is based on a study of over 2000 technology companies. This is a useful data set for anyone who is looking to understand adoption of cloud products and Kubernetes, and it can be found at softwareengineeringdaily.com/sumologic. Kalyan joins the show to discuss log management, continuous intelligence, and the data that Sumo Logic gathered in the Continuous Intelligence Report. Full disclosure: Sumo Logic is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Offices have historically been the place where most knowledge work takes place. An office is a central meeting point for everyone in an organization. Offices allow for high bandwidth, in-person communication. Employees have access to shared resources, such as food, tables, and quiet working space. Offices provide a means of encouraging a common culture within a given workplace. There are also significant downsides to offices–most notably the commute. Employees often spend between one and four hours per day driving to the office. Office work creates a huge restriction for parents with a young child to take care of. Many employees feel more productive when they are working at home, or working from a coffee shop. Remote work is becoming an increasingly popular mode of work for knowledge workers. Remote work has been made possible by increased bandwidth, more powerful computers, and new communications software such as Slack and Zoom. Remote work is a powerful trend that is reshaping how knowledge workers spend their time. It is also changing how companies are structured. A “remote first” culture impacts hiring, human resources, engineering, sales, work-life balance and every other aspect of operations. The downstream impacts of remote work will change the labor market even more thoroughly, and cause us to rethink contracting, equity structures, and the traditional five day workweek. Philip Thomas is the co-founder and CEO of Moonlight Work, a marketplace for software engineers who work on contract projects full-time or part-time. Philip is also the co-author of the Remote Work Encyclopedia, which is a collection of strategies and tactics for knowledge workers and companies that are looking to adapt successfully to the changes that remote work is bringing to the world. Philip joins the show to discuss remote work and his experience building Moonlight. Full disclosure: I am an investor in Moonlight Work. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning algorithms have existed for decades. But in the last ten years, several advancements in software and hardware have caused dramatic growth in the viability of applications based on machine learning. Smartphones generate large quantities of data about how humans move through the world. Software-as-a-service companies generate data about how these humans interact with businesses. Cheap cloud infrastructure allows for the storage of these high volumes of data. Machine learning frameworks such as Apache Spark, TensorFlow, and PyTorch allow developers to easily train statistical models. These models are deployed back to the smartphones and the software-as-a-service companies, which improves the ability for humans to move through the world and gain utility from their business transactions. And as the humans interact more with their computers, it generates more data, which is used to create better models, and higher consumer utility. The combination of smartphones, cloud computing, machine learning algorithms, and distributed computing frameworks is often referred to as “artificial intelligence.” Chris Benson is the host of the podcast Practical AI, and he joins the show to talk about the modern applications of artificial intelligence, and the stories he is covering on Practical AI. On his podcast, Chris talks about everything within the umbrella of AI, from high level stories to low level implementation details. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The container orchestration wars ended in 2016 with Kubernetes being the most popular open source tool for deploying and managing infrastructure. Since that time, most large enterprises have been implementing a “platform strategy” based around Kubernetes. A platform strategy is a plan for creating a consistent experience for software engineers working throughout an enterprise. At most companies, a software engineer should be thinking about business logic–whether that logic is related to banking, insurance, oil and gas, or e-commerce. Today, engineers at many enterprises need to think about continuous delivery, application deployment, security policy management, and other deeply technical problems that have nothing to do with the business that they are actually working at. Kubernetes is a foundational open source building block that allows enterprises to base the rest of their infrastructure decisions around. Kubernetes has made it much more viable for enterprises to pursue a platform strategy. With widespread adoption of Kubernetes, there is a business opportunity for companies that can offer other platform solutions that build on top of Kubernetes. A service mesh is one such tool. A service mesh provides networking and security features for all the services in an organization. The service mesh category is a large business opportunity because it sits on the critical path of every network request that goes through an enterprise. It is a potential insertion point for lots of other products including logging agents, distributed tracing, network packet scanning, security policy management, and A/B testing. The potential for business expansion is why so many businesses are entering the service mesh category today, from cloud providers to API gateways. Buoyant was one of the first companies to work on a service mesh tool, with the Linkerd open source project. William Morgan is the CEO of Buoyant, and he returns to the show to discuss the competitive dynamics of the service mesh market. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes has created a widespread system for deploying and managing infrastructure. As Kubernetes has been increasingly adopted, companies are thinking about how to leverage that common layer of infrastructure. With the common infrastructure abstraction of Kubernetes, it becomes easier to adopt other abstractions that are uniform across the entire company. This has created a market opportunity for products such as a service mesh. A service mesh consists of sidecar containers that get deployed alongside services in a distributed system. Each sidecar container is used as a proxy for all the communication that goes through the service it is deployed with. This consistent proxying layer provides each service with benefits such as security, routing, telemetry, and policy management. Istio is a service mesh that was created and open sourced by Google. Istio is built around the Envoy service proxy sidecar and a control plane that manages the Envoy sidecars. Since the launch of Istio, some of the Google employees who were working on Istio have started Tetrate, a company with the goal of commercializing Istio into a product that enterprises will pay for. The market demand for service mesh has been proven, but there are many competitors to Tetrate. Istio is open source and can be commercialized by other companies, as well as cloud providers such as Google and AWS. Linkerd is a service mesh built by the company Buoyant, which was the first company to focus exclusively on this space. There are other companies that are expanding existing products into service mesh: Kong, NGINX, and Hashicorp. Zack Butcher is a founding engineer with Tetrate, and he joins the show to discuss the market for service mesh and the plan for Tetrate to build a business around Istio. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A cloud provider gives a developer low-cost compute infrastructure on-demand. Cloud providers can be divided up into two categories: Layer 1 cloud providers and Layer 2 cloud providers. A Layer 1 cloud provider such as Amazon Web Services owns server hardware and sells compute infrastructure as a commodity. A Layer 2 cloud provider purchases compute infrastructure from a Layer 1 provider and builds a high quality developer experience on top of that compute infrastructure. Heroku was the first Layer 2 cloud provider. Heroku’s first business was to provide a high quality developer experience and low cost containerization infrastructure on top of Amazon’s EC2 virtual machine infrastructure. Heroku has added features for continuous integration, relational databases, caches, and queueing. Building a Layer 2 cloud provider is a very different challenge than building a Layer 1 cloud provider. A Layer 1 provider must focus on low level problems such as hardware infrastructure and virtualization. This does not leave much time for focusing on developer experience. A Layer 1 cloud provider must be able to serve every type of potential software customer. A Layer 2 provider can provide a streamlined experience. Mark Turner is an engineer at Heroku. He joins the show to discuss the architecture and engineering of a Layer 2 cloud provider. Heroku is built on top of Amazon Web Services, and the core compute infrastructure is built on top of a pool of EC2 virtual machines that are continually scheduled with applications that users create on Heroku. Full disclosure: Heroku is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Java has been popular since the 90s, when it started to be used as a programming language for enterprises. Today, Java is still widely deployed, but the infrastructure environment is dramatically different. Java is often deployed to containers in the cloud. If those containers can share resources, then those containers can share the same underlying Java infrastructure. Java 13 is the most recent public release of Java. The new features in Java 13 reflect the changing demands of modern application developers. Georges Saab is an engineer with Oracle who has been working on Java for more than a decade. He joins the show to discuss how Java development patterns are changing, and how the language is evolving to accommodate those changes, including discussion of garbage collection and dynamic application class data sharing. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Relational databases provide durable transactional systems for storing data. The relational model has existed for decades, but the requirements for a relational database have changed. Modern applications have requirements for high volumes of data that do not fit onto a single machine. When a database gets too big to fit on a single machine, that database needs to be sharded into smaller subsets of the data. These database shards are spread across multiple machines, and as the database grows, the database can be resharded to scale to even more machines. To ensure durability, a database needs to be replicated. The database needs to be able to survive any single machine losing power or getting destroyed. Sharding and replication allow a relational database to be scalable, durable, and highly available. There are many ways to build sharding and replication into a database. Karthik Ranganathan and Sidharth Choudhury are engineers with YugabyteDB, a distributed SQL database. In today’s episode, we discuss the modern requirements of a distributed SQL database, and compare the applications of distributed SQL to those of other systems such as Cassandra and Hadoop. We also talk through the competitive market of cloud-based distributed SQL providers such as Google Cloud Spanner and Amazon Aurora. YugabyteDB is an open source database that competes with these other relational databases. Full disclosure: YugabyteDB is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published July 14, 2016 EconTalk is a weekly economics podcast that has been going for a decade. On EconTalk, Russ Roberts brings on writers, intellectuals, and entrepreneurs for engaging conversations about the world as seen through the lens of economics. Russ Roberts is today’s guest, and it is a treat because I have been listening to EconTalk since 2006 and it was a central point of inspiration for what Software Engineering Daily has become. On this episode, we talk about how software impacts the world economically, from bitcoin’s promise of zero cost transactions to the opportunities and regulatory challenges of the software-enabled gig economy. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published May 24, 2018 When a user takes a ride on Uber, the app on the user’s phone is communicating with Uber’s backend infrastructure, which is writing to a database that maintains the state of that user’s activity. This database is known as a transactional database or “OLTP” (online transaction processing). Every active user and driver and UberEATS restaurant is writing data to the transactional data store. Periodically, that data is copied from the transactional data system to a different data storage system, where that data can be queried for large-scale data analysis. For example, if a data scientist at Uber wants to get the average amount of miles that a given user rode in February, that data scientist would issue a query to the analytical data cluster. Uber uses the Hadoop distributed file system (HDFS) to store analytical data. On this file system, Uber has a version history of all of the company’s useful historical data. Trip history, rider activity, driver activity–every data point that is in the transactional database–but in a file format that is easier to query for large scale processing. This file format is known as Parquet. Data scientists, machine learning engineers, and real-time application developers all depend on the massive quantities of data that are stored in these Parquet files on Uber’s HDFS cluster. To simplify the access of that data by many different clients, Uber uses Presto, an analytical query engine originally built at Facebook. Presto translates SQL queries into whatever query language is necessary to access the underlying storage medium–whether that storage system is an ElasticSearch cluster, a set of Parquet files, or a relational database. Presto is useful because it simplifies the relationship between data engineers and the application developers who are building on top of the data engineering infrastructure. In today’s show, Zhenxiao Luo joins to give an end-to-end description of Uber’s data infrastructure–from the ingest point of the OLTP database to the OLAP data storage system on HDFS, to the wide range of data systems and applications that run on top of that OLAP data. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published January 4, 2019 When a user makes a request to product like The New York Times, that request hits an API gateway. An API gateway is the entry point for an external request. An API gateway serves several purposes: authentication, security, routing, load balancing, and logging. API gateways have grown in popularity as applications have become more distributed, and companies offer a wider variety of services. If an API is public, and anyone can access it, you might need to apply rate limiting so that users cannot spam the API. If the API is private, the user needs to be authenticated before the request is fulfilled. Kong is a company that builds infrastructure for API management. The Kong API gateway is a widely used open source project, and Kong is a company built around supporting and building on top of the API gateway. Marco Palladino is the co-founder and CTO of Kong. He joins the show to tell the story of starting Kong eight years ago, and how the API gateway product evolved out of an API marketplace. Marco also discusses the architecture of Kong and his vision for how the product will develop in the future–including the Kong service mesh. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published June 7, 2018 Moore’s Law states that the number of transistors in a dense integrated circuit doubles about every two years. Moore’s Law is less like a “law” and more like an observation or a prediction. Moore’s Law is ending. We can no longer fit an increasing amount of transistors in the same amount of space with a highly predictable rate. Dennard scaling is also coming to an end. Dennard scaling is the observation that as transistors get smaller, the power density stays constant. These changes in hardware trends have downstream effects for software engineers. Most importantly–power consumption becomes much more important. As a software engineer, how does power consumption affect you? It means that inefficient software will either run more slowly or cost more money relative to our expectations in the past. Whereas software engineers writing code 15 years ago could comfortably project that their code would get significantly cheaper to run over time due to hardware advances, the story is more complicated today. Why is Moore’s Law ending? And what kinds of predictable advances in technology can we still expect? John Hennessy is the chairman of Alphabet. In 2017, he won a Turing award (along with David Patterson) for his work on the RISC (Reduced Instruction Set Compiler) architecture. From 2000 to 2016, he was the president of Stanford University. John joins the show to explore the future of computing. While we may not have the predictable benefits of Moore’s Law and Dennard scaling, we now have machine learning. It is hard to plot the advances of machine learning on any one chart (as we explored in a recent episode with OpenAI). But we can say empirically that machine learning is working quite well in production. If machine learning offers us such strong advances in computing, how can we change our hardware design process to make machine learning more efficient? As machine learning training workloads eat up more resources in a data center, engineers are developing domain specific chips which are optimized for those machine learning workloads. The Tensor Processing Unit (TPU) from Google is one such example. John mentioned that chips could become even more specialized within the domain of machine learning. You could imagine a chip that is specifically designed for a LSTM machine learning model. There are other domains where we could see specialized chips–drones, self-driving cars, wearable computers. In this episode, John describes his perspective on the future of computing, and offers some framework for how engineers can adapt to that future. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published November 4, 2016 Indie Hackers is a website that profiles independent developers who have made profitable software projects, usually without raising any money. These projects make anywhere from a few hundred dollars a month to more than $100,000 as in the case with park.io, one of the services profiled by Indie Hackers. Courtland Allen is the creator, engineer, and interviewer behind Indie Hackers. For each business that is profiled by Indie Hackers, Courtland conducts a short interview with the founder. Courtland joins the show to discuss the changing trends that are making it easier to bootstrap a software business if you are a capable developer–or even if you are a nontechnical person who understands how software works. Since Courtland and I are both in the business of interviewing engineers, we had a lot to talk about, and this is a fantastic episode. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

New software abstractions always take advantage of the abstractions that have been built before. Software libraries allow us to import code that sits on the same host as a new program. Open source software let us copy and paste existing code, or clone entire repositories. Cloud providers offer hosted tools and APIs that we can leverage to develop scalable, easy-to-use infrastructure. When existing pieces of software are built into new software, the existing software becomes a dependency. Managing those dependencies is an engineering problem. Mya Pitzeruse is the founder of deps.cloud, a project with the goal of improving dependency changes across a company’s ecosystem. In today’s show, we talk about the modern dependency issues of a large company, and her perspective on how to address them. Mya has developed the project in public on FindCollabs, and we also spend some time talking about building in the open. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Web applications are used on a wide variety of platforms. On each of these platforms the web app needs to load properly and allow the user to navigate the website and interact with all of the user flows, such as sign-up, login, and the various read and write operations that make up the functionality of any website. It is difficult to ensure web application functionality across all platforms, because there are so many platforms. Different operating systems, different underlying hardware, different browsers, and different device form factors all create potential sources of suboptimal website functionality and performance. Testing web applications often involves the work of a manual quality assurance (QA) employee. The QA can simulate the procedures that a normal user would go through. This QA process ensures that the website is operating as expected. But the manual workflow can slow down software development. Gabriel-James Safar is a software engineer and the founder of Madumbo, which was acquired by Datadog. Madumbo was founded with the goal of making web application testing simpler by identifying errors in pages and enabling users to create test suites from recordings of user activity. This process simplifies and accelerates the testing process. Gabriel-James joins the show to talk about his experience building Madumbo, and his perspective on the modern application testing process. Full disclosure: Datadog is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

HTTP is a protocol that allows browsers and web applications to communicate across the Internet. Everyone knows that HTTP is doing some important work, because “HTTP” is at the beginning of most URLs that you enter into your browser. You might be familiar with the request/response model, and HTTP request methods such as GET, PUT, and POST. But unless you have had a reason to learn more about the details of HTTP, you probably don’t know much more than that. Julia Evans is a software engineer and writer who creates Wizard Zines, a series of easy-to-read online magazines that explain technical software topics. Julia’s zines include “Linux Debugging Tools”, “Help! I Have A Manager!”, and recently “HTTP: Learn your browser’s language”. Her zines are a creative, innovative format for describing the world of software engineering while also exploring her own artistic pursuits in writing, design, and illustration. Julia was previously on the show to discuss Ruby profiling, and she returns to the show to discuss HTTP, as well as her creative process and goals with Wizard Zines. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Key Values is a platform where companies are profiled with descriptions of their company values. These profiles describe features such as work-life balance, company culture, daily routines, and strategy. Lynne Tye created Key Values with the goal of building a small business that would make money through connecting job seekers to companies with a culture that matched their own personal values system. Key Values has become highly successful, and Lynne is making enough money from the business to live comfortably. In a previous episode, Lynne and I discussed her founding story and the engineering of Key Values. Today’s episode picks up a few years later, with Lynne having found significant success with her own company. Lynne’s software business is an example of a growing trend: “Indie Hackers”. This trend was identified by Courtland Allen, founder of the Indie Hackers platform. Courtland is close friends with Lynne, and Lynne’s desire to start her own software company was influenced by her conversations with Courtland. At a certain point, Lynne was considering raising money and growing Key Values. She was accepted into Y-Combinator. But she decided to stick with the Indie Hackers route, and grow the business independently. Lynne joins the show to talk about the process of starting a software business, and the pivotal decisions she has made around financing, growth, and her own psychology. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The vision of code-free programming has existed for decades. Software engineers have always dreamed of empowering non-technical users with the same creative tools that programmers have access to. For many years, the underlying technology of the web was not powerful enough to make this dream a reality. Platforms such as WordPress, Squarespace, and Wix have allowed for some of the functionality of programming without writing code, but the scope of what those tools could accomplish was limited. Today, the web has caught up. Improvements in browser technology and client devices mean that the end user has access to more powerful technology. The API economy encapsulates large amounts of functionality into cheap, well-defined functionality. No-code platforms are finding success among developers and non-developers, after persisting for several years as their technology matured. Bubble is a code-free platform for building websites, startups, and internal tools. Emmanuel Straschnov and Joshua Haas are the founders of Bubble, and they join the show to tell the story of Bubble, and give their perspective on engineering, business, and the low-code movement. The story of Bubble is strikingly similar to that of no-code tools Airtable and Webflow, which we have covered in previous episodes. All of these products have taken years to get to maturity, with no shortcuts–only gritty, difficult engineering problems and performance improvements. Each of these no-code platforms has an inspiring story behind them, and persistent founders who eventually got their product to success. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

“Serverless” is an execution model where applications are scheduled and deployed to servers that are not directly managed by the application developer. In serverless execution, an application only loads and operates when a user actually needs to get a response from that application. This saves on resources, because many applications do not need to run at all times–they only need to be available for user requests. The serverless model was popularized by Amazon Web Services Lambda. When Lambda first launched in 2015, it was an experimental product. Today, it is a widely used product and the market has validated the desire for serverless execution. Other cloud providers have introduced different models of serverless functionality including Google Cloud Functions, Azure Functions, and Fastly edge computing. Zack Bloom is director of product for product strategy at Cloudflare, and he joins the show to discuss Cloudflare’s model for serverless execution. Zack also discusses Cloudflare’s growing product line, including the fast, privacy-protecting DNS resolver 1.1.1.1. Zack is a rare mix of engineering, business strategy, and product vision, which made for a great conversation. For more content, you can check out our episodes about serverless technology and episodes about Cloudflare. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data engineering is difficult. Companies want to be able to maximize the value they get from their large data sets, but there are so many steps required for deriving that value that most companies feel like they are always far behind the ideal state of where they could be. The cloud makes it cheap to save data. Tools like Spark and Snowflake give us usable APIs for simplifying the data processing. Workflow engines like Airflow help us visualize a complex execution path of a data pipeline. With all of this tooling, why is it so hard to make use of our data? Nick Schrock is the creator of Dagster, an open source system for building modern data applications. Nick is also the CEO of Elementl, a company that he is building around Dagster. Before creating Dagster, Nick worked at Facebook, where he co-created GraphQL. Nick returns to the show to discuss modern data engineering, and why it continues to be so difficult for engineers to be productive with their data. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Java programs run in a different environment than they did ten years ago. Modern infrastructure runs on containers sitting in a Kubernetes cluster. The optimal configuration for a Java program in that context is different than it was for an environment dominated by virtual machines and bare metal. When you are co-scheduling your services with each other, those services could be fighting for resources. You may want to optimize them with more ahead-of-time compilation. Quarkus is a system for accelerating Java performance through the use of GraalVM. In a previous show, we explored the basics of GraalVM. In today’s show, Guillaume Smet and Emmanuel Bernard join the show to describe an application of GraalVM: the acceleration of Java. Guillaume and Emmaneul are engineers at Red Hat, and are working on changes to the Java ecosystem that are informed by the cloud and the rise of Kubernetes. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Crash monitoring emerged as a software category over the last decade. Crash monitoring software allows developers to understand when their applications are crashing on client devices. For example, we have an app for Software Engineering Daily that people download on Android or iOS. Users download the app to their smartphone. When the user is playing an episode, and the app crashes, the details of the crash are sent to a server that collects all of these crash reports. Crash reports allow a company to understand where their application is breaking on client devices. This is important, since there are so many client surfaces to test, from iOS to Android to browsers. As a business, crash monitoring is a category that has some similarities to log management. There are lots of companies that offer crash monitoring. At first glance, it seems like a simple problem to solve. It seems like a market without winner-take-all or winner-take-most dynamics. But at scale, crash monitoring becomes a deeply complex engineering problem. From indexing to database choices to complex distributed systems tradeoffs, crash monitoring is not a simple business, and it promises to provide an extremely good business for the few companies who are able to out-execute the crowded market. James Smith is the CEO of Bugsnag, a company that makes crash monitoring and application stability tools. James returns to the show to discuss the growth-stage engineering challenges of error monitoring, and the business opportunities that come with them. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software bugs cause unexpected problems at every company. Some problems are small. A website goes down in the middle of the night, and the outage triggers a phone call to an engineer who has to wake up and fix the problem. Other problems can be significantly larger. When a major problem occurs, it can cause millions of dollars in losses and requires hours of work to fix. When software unexpectedly breaks, it is called an incident. To triage these incidents, an engineer uses a combination of tools, including Slack, GitHub, cloud providers, and continuous deployment systems. These different tools emit updates that can be received by an incident response platform, which allow the on-call engineer to have the information they need centralized to more easily work through the incident. On-call rotation means that different people will be responsible for dealing with different incidents that occur. When an incident happens, the current engineer who is on-call may not be aware that a similar incident happened last week. It might be easier for the new engineer to triage the issue if they have insights about how the incident was managed during the first time. Chris Riley is a DevOps advocate with Splunk. He joins the show to discuss the application of machine learning to incident response. We discuss the different data points that are created during an incident, and how that data can be used to build models for different types of incidents, which can generate information to help the engineer respond appropriately to an incident. Full disclosure: Splunk is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Frontend software development has become as complex as backend development. There was a time when frontend web development was simple. There was a small number of JavaScript frameworks and templating systems. Your CSS was simple configuration for the colors on your webpage. Today, there is a giant ecosystem of frontend tools, APIs, and middleware delivering data to the user. Gatsby is a framework based on React that allows developers to build performant web applications. Gatsby is not easy to explain. In some ways, it is like a compiler for your website. Gatsby pulls in the data that you need to build your website, including CMS data, APIs, and markdown, and then links that information into React components and CSS. This happens on the server, so your user gets served a website that does not require lots of round trips as your website renders. Kyle Mathews and Sam Bhagwat are the founders of Gatsby, the company that is based around GatsbyJS. They join the show to describe their vision for the framework, and their vision for the company. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Drones will deliver food to us. Drones will be able to extinguish fires. Drones will be used to relay Internet signals and make the world more connected. These all sound like great ideas, so why aren’t there more drones in the sky today? There are many answers to that question, some of which relate to engineering and some of which are about the regulatory barriers. Chris Anderson is the CEO of 3D Robotics, a drone company which he started seven years ago. Before 3DR, Chris worked for many years as a journalist writing about technology and science. He was the editor-in-chief at Wired for 11 years, a writer for The Economist for 7 years, and spent 3 years at both of the leading scientific journals, Nature and Science. Chris also wrote The Long Tail, the influential 2004 book which described a set of emergent Internet trends. Chris joins the show for a discussion about drones, journalism, and his perspective on modern technology. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A large social network needs to develop systems for ingesting, storing, and processing large volumes of data. Data engineering at scale requires multiple engineering teams that are responsible for different areas of the infrastructure. Data needs to be structured coherently in order to minimize the data cleaning process. Machine learning models need to be developed, deployed, and iterated on at scale. Areas of the company which produce data need to be decoupled from the areas of the company which consume data, so that engineers throughout the company can reliably build tools on top of these large data sets. In our previous episodes about LinkedIn, we covered two major components of LinkedIn’s data engineering systems: the Kafka infrastructure and the LinkedIn data platform used by engineers to productively build data applications. Kapil Surlaker is a senior director of engineering at LinkedIn, and he joins the show to discuss the bigger picture of LinkedIn’s data infrastructure. Kapil works with teams across LinkedIn to understand the requirements for the products and internal tools, and translate those requirements into team structures and software platforms that let LinkedIn use data more productively. We discuss a wide range of topics, including engineering management, the modern data platform, and LinkedIn’s adoption of public cloud. Full disclosure: LinkedIn is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every company has a software supply chain. A company builds its products from custom code, paid APIs, paid proprietary binaries, and open source software libraries. As the types of software available have increased, the management of the software supply chain has become complex. Large software companies have always needed to ensure the security of their software. With the growing variety of open source licenses, these companies also have to deal with an increased set of legal complexities. If an open source project is used in a way that violates an open source license, the company is subject to legal risk. FOSSA is a company that focuses on automating the management of open source compliance and security. Kevin Wang is the CEO of FOSSA, and he joins the show to discuss the modern issues of software licensing and his experience building a company. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning is widely understood by the software community. But it is still hard to build a company around machine learning, because there is not easy access to large, unique data sets. Scale is a platform for training and validating data that is used for machine learning. Most machine learning models are built with supervised learning. Labeled examples are analyzed to understand the mathematical correlations between those labels. The more labeled training examples there are, the more accurate the correlations will be. Today, we have high quality frameworks for writing the models. We have cheap cloud computing for training and deploying the models. The biggest factor that is preventing a wide variety of potential machine learning applications from existing is lack of access to large, labeled data sets. Scale gives developers an API for labeling images, sound, natural language, and video. Scale is used by self-driving car companies, Airbnb, OpenAI, retailers, and robotics companies. The product is used broadly and at high volume. Scale was started only three years ago, and recently raised $100m at a valuation above $1b, making it one of the fastest growing software companies in history. Alexandr Wang joins the show to discuss how Scale works, the future of machine learning, and the future of work. He also describes the complexities of building Scale, and how he manages his own psychological state. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Photo credit: Elisabeth Fall Ben Horowitz started Loudcloud with Marc Andreessen in 1999. He ran the company for eight years and chronicled his experience in his first book The Hard Thing About Hard Things. In his time running Loudcloud, the dotcom bubble burst, but Loudcloud needed cash so badly that he took the company public in 2001. Loudcloud went through layoffs, downsizing, and a difficult strategic maneuver in which Loudcloud sold its cloud provider business for cash, then used the core competency it had developed to create new software for building and running cloud services. This new software was the core product of the company Opsware, which was sold to HP in 2007 for $1.6bn. The Loudcloud story looks like a rational, straightforward execution in retrospect, but at many points in the timeline, Ben was unsure he was making the correct decision. As the subtitle of his first book states “there are no easy answers.” The Hard Thing About Hard Things tells the story of Loudcloud and Opsware in harrowing detail. Most founders of software companies will end up reading the book at some point when they are building their company, because there are so few books which capture the granular details of what it feels like to run a company. A CEO is completely alone in their understanding of the company. Nobody else has nearly as much information as the CEO–not the board, not the market, and not the employees. When you are a CEO, there is simply nobody to turn to who can give you the actionable advice that you wish you could have access to. And because there is nobody else, it means that the CEO’s own psychological state is extremely important. The Hard Thing About Hard Things provides a CEO with solace: while the CEO is alone within their company, they are not alone in the world. Every CEO has a set of issues which they have never faced before, and the CEO can learn to face those issues confidently and competently. Like any influential book, The Hard Thing About Hard Things presents the reader with useful answers, but also raises many questions. How can a normal person foster the mentality of a leader? How can a leader convince smart people to follow their direction? How can a seemingly crazy direction be framed as completely rational? The second book by Ben Horowitz is called What You Do Is Who You Are. This book surveys a set of case studies in leadership, including a Haitian slave revolt, the Mongol empire, and a dominant prison gang. By studying violent environments, Ben frames leadership in the context of the highest stakes. These stories are about life and death. When a leader’s performance is measured in blood, it frames the true nature of leadership in the starkest resolution. Ben uses each distilled example as a base case which inducts into broader applications: the cultures of Netflix, Facebook, Uber, and McDonalds are explored alongside editorials about Hillary Clinton and hip hop culture. Throughout all of these stories, the most important thread is continually reinforced: the leader creates the culture. The culture is the leader. What you do is who you are. Ben joins the show to discuss his writing, and how he has applied these beliefs to Andreessen Horowitz, the venture capital firm he co-founded and leads today. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook leadership has a significant amount of engineers in its ranks, and engineers understand how to create an environment that appeals to other engineers. Engineers do not like working on projects that they are not interested in, so Facebook optimizes for matching engineers to enjoyable work. Engineers do not like taking orders from managers, so Facebook optimizes for a collaborative relationship between engineering and management. Arturo Bejar was a director of engineering at Facebook for more than five years. Arturo managed a team that built site integrity tools, as well as the Product Infrastructure team, which created the groundbreaking React and GraphQL open source tools. Arturo has a deep understanding of what engineers want from a manager, and this is apparent in the results of the teams he has worked with. Arturo also cares deeply about the human side of technology. When he was at Facebook, he started the Compassion Project, which looked at ways that technology could help support people through difficult times, and alleviate suicide and bullying. Arturo joins the show for a conversation about engineering management, humanity, and the ethos of Facebook. Articles by Arturo Bejar Managing Your Manager Challenges and Opportunities in Communication Online: Summary and Full Version Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Chaos Monkeys is a book about Silicon Valley startups and Facebook. It is one of the most accurate books written about the modern technology industry, and captures both the negatives and the positives of software companies. Antonio Garcia Martinez is the author of Chaos Monkeys. He wrote the book after going through a gauntlet of prototypical Silicon Valley experiences. Antonio founded an adtech company called AdGrok. His company was funded by the startup accelerator Y-Combinator, and during his time at the company there were many dramatic events that Antonio tells in great detail. AdGrok was acquired by Twitter, and Antonio went to work at Facebook on its ads platform. Antonio is a fantastic writer. What makes Chaos Monkeys special is that it reads like a book written by an author who accidentally found himself in a technology career, rather than a technologist who opportunistically wrote a book. Note: This episode contains explicit language. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook engineering is unique. Software is built at Facebook in a way that is distinctly different than any other company. In our series of shows about Facebook engineering, we have mostly covered the positive side of Facebook’s practices. In today’s show, we explore the downsides. Facebook moves fast. Engineers within the company must move fast, which can reduce the time spent on formal process and documentation. A fast pace can lead to a chaotic environment. In this kind of environment, not every employee thrives, and product development can sometimes suffer. But there are clear benefits to moving fast, and many software organizations could benefit from moving faster. Pete Hunt was an engineer with Facebook who worked on Facebook Video, Instagram, and React. Nick Schrock worked on core infrastructure at Facebook and was the co-creator of GraphQL. Pete and Nick helped create the Facebook culture, and are aware of the best and worst aspects of it. They join the show to reflect on their time at Facebook, and the downsides of moving fast. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Peter Deng has worked on most of Facebook’s major products: Newsfeed, Instagram, Oculus, and Messenger. These different products have different requirements, but are all part of the same ethos of connecting people through social networks. Facebook is a consumer product company that is powered by a strong engineering workforce. The relationship between product managers and engineering are two parts of a three-legged relationship: product, engineering, and design. Every major product within Facebook is built with the teamwork of product, engineering, and design. After almost ten years at Facebook, Peter joined Uber as the Head of Rider. At Uber, Peter works on a very different platform: a real-world, two-sided marketplace. Every change to the Uber platform has an impact on the economic relationship between riders and drivers. This creates a set of product development constraints that contrast with the social network of Facebook. Peter joins the show to share how he thinks about product management, and how the core competencies of a business inform product strategy. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Jeff Rothschild was one of the earliest engineers to join Facebook. In the 1990s, Jeff had co-founded Veritas Software and helped it to its IPO in 2004. After Veritas, Jeff worked on several other of his own companies. He was working with Accel Partners on investments when he started to learn about Facebook. Accel was an investor in Facebook, having made a large investment in 2005. After the investment, Jeff was helping the company with hiring. As Facebook built out its senior engineering leadership, Jeff realized that he would enjoy working with the company in a more active role. For the next ten years, Jeff Rothschild worked as a VP of infrastructure software at Facebook. He helped scale the company’s infrastructure and architect the technical strategy. By 2005, Jeff’s entrepreneurial background included software companies ranging from storage management to gaming. His diverse skill set was useful for setting product direction as well as solving novel engineering problems. Jeff joins the show to discuss the early days of Facebook and his philosophy of entrepreneurship. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In 2009, the Berkeley AMPLab was a center of innovation. Three projects from AMPLab have turned into successful open source projects and companies: Spark, Mesos, and Alluxio. Haoyuan Li was the creator of Alluxio, and he returns to the show to discuss his journey taking Alluxio from a research project to a company that has customers including Alibaba, Baidu, Wells Fargo, and Samsung. Alluxio is a distributed, fault-tolerant, tiered storage system. Alluxio allows application developers to think in terms of the latency that they require from their infrastructure rather than the details of different storage systems. Haoyuan discusses the process of integrating with gigantic companies like cloud providers, telecoms, and huge ecommerce companies. Alluxio is also hosting an upcoming conference, the Data Orchestration Summit November 7th at the Computer History Museum in Mountain View California. November 7, 2019 is the first open source Data Orchestration Summit in the Computer History Museum in Mountain View, CA. The summit brings together data engineers, data platform engineers, and data scientists to share their challenges and learnings from building and using modern analytics, AI, and cloud technologies. Featuring tech talks covering use cases, demos, best practices, and tutorials by industry experts from EA, Walmart, DBS Bank, Netflix, AWS, Rakuten, Tencent, Google, Baidu, Alibaba and more, with a focus on how to build cloud native analytics & AI platforms. Use code “SDE” to get a 50% discount. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Redis is an in-memory database that persists to disk. Redis is commonly used as an object cache for web applications. Applications are composed of caches and databases. A cache typically stores the data in memory, and a database typically stores the data on disk. Memory has significantly faster access times, but is more expensive and is volatile, meaning that if the computer that is holding that piece of data in memory goes offline, the data will be lost. When a user makes a request to load their personal information, the server will try to load that data from a cache. If the cache does not contain the user’s information, the server will go to the database to find that information. Alvin Richards is chief product officer with Redis Labs, and he joins the show to discuss how Redis works. We explore different design patterns for making Redis high availability, or using it as a volatile cache, and we talk through the read and write path for Redis data. Full disclosure: Redis Labs is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

LinkedIn is a social network with petabytes of data. In order to store that data, LinkedIn distributes and replicates that data across a large cluster of machines running the Hadoop Distributed File System. In order to run calculations across its large data set, LinkedIn needs to split the computation up using MapReduce-style jobs. LinkedIn has been developing its data infrastructure since the early days of the Hadoop ecosystem. LinkedIn started using Hadoop in 2008, and in the last 11 years, the company has adopted streaming frameworks, distributed databases, and newer execution runtimes like Apache Spark. With the popularization of machine learning, there are more applications for data engineering than ever before. But the tooling around data engineering means that it is still hard for developers to find data sets, clean their data, and build reliable models. Carl Steinbach is an engineer at LinkedIn working on tools for data engineering. In today’s episode, Carl discusses the data platform inside LinkedIn, and the strategies that the company has developed around storing and computing large amounts of data. Full disclosure: LinkedIn is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cryptocurrencies are a fundamental computer science invention. Cryptocurrencies crashed in 2018 but the technology remains as promising as ever. Bitcoin is a decentralized currency, and a plausible end state that is implied by Bitcoin’s current trajectory is a permissionless, decentralized financial system. This idea of decentralized finance or “DeFi” begs numerous questions: who will build the companies that provide the infrastructure for decentralized finance? Who will be the lenders? Who will be the credit agencies? Who will be the escrow services? How big will the teams need to be? Will these systems be built on smart contracts, or can it be done with centralized cloud providers? Haseeb Qureshi is a managing partner with DragonFly Capital and a frequent guest on SE Daily. He returns to the show to discuss his thesis on what kinds of crypto companies make sense, and how he thinks about investments into crypto startups. Haseeb recently wrote an article about how to build a crypto startup. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Dark Lang is a programming language that is tightly integrated with the cloud. Dark takes an opinionated approach that most developers are going to want to run their applications in the cloud, and this perspective influences how Dark looks at deployments, IDEs, exception handling, and other aspects of software development. Paul Biggar is the founder of CircleCI, and ran the company for eight years before leaving to found Dark with Ellen Chisa. Ellen is a software engineer and the CEO of Dark. Paul and Ellen join the show to give their perspective on modern software engineering, and why it was time to build a new high level language that assumes the presence of a cloud. It is difficult to get programmers to adopt a new language. It is even harder to get those programmers to pay for products built around that language. But the timing could be perfect for Dark. Software development is undergoing tremendous change–and many of these changes work to Dark’s advantage, such as the growing adoption of feature flags, low code tools, and sophisticated continuous delivery workflows. Whether or not Dark is a success, it is a bold project, and the team is working on something they believe in. We also discussed AWS–and whether the largest cloud provider has an obligation to contribute back to the open source community. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Kafka was created at LinkedIn. Kafka was open sourced in 2011, when the company was eight years old. By that time, LinkedIn had developed a social network with millions of users. LinkedIn’s engineering team was building a range of externally facing products and internal tools, and many of these tools required a high-throughput system for publishing data and subscribing to topics. Kafka was born out of this need. Over time, Kafka’s importance within LinkedIn has only grown. Kafka plays a central role for services, log management, data engineering, and compliance. LinkedIn might be the biggest user of Apache Kafka in the entire software industry. Kafka has many use cases, and it is likely that they are almost all on display within LinkedIn. Nacho Solis is a senior software engineering manager at LinkedIn, where he helps teams build infrastructure for Kafka, as well as Kafka itself. Nacho joins the show to discuss the history of Kafka at LinkedIn, and the challenges of managing such a large deployment of Kafka. We also talk about streaming, data infrastructure, and more general problems in the world of engineering management. Full disclosure: LinkedIn is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern applications are distributed systems. These applications require an installation mechanism that can run and update the software across multiple nodes. When a SaaS company starts to work with large enterprise customers, that company needs to figure out a way to deliver their product to the enterprise. This requires the SaaS company to deploy the product to whatever infrastructure the enterprise is running. Some enterprises use on-prem infrastructure. Some use AWS. Some use a variety of cloud providers and on-premise servers. A SaaS company with limited resources must be able to have a standard deployment model that satisfies all of these different use cases. Ev Kontsevoy is the CEO of Gravitational, a company that builds software for application delivery. Ev’s company maintains Gravity, an open source tool for application imaging and delivery. Ev was also the founder of Mailgun, a popular email API service. Mailgun was acquired by Rackspace, and in his time running Mailgun, Ev became deeply aware of the problems faced by developers and operators who manage server infrastructure. Ev joins the show to discuss his experience building companies and the state of modern infrastructure. Full disclosure: Gravitational is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Databases go offline. Services fail to scale up. Deployment errors can cause an application backend to get DDoS’d. When an event happens that prevents your company from operating as expected, it is known as an incident. Software teams respond to an incident by issuing a fix. Sometimes that fix returns the software to its ideal state. Other times the software remains in a degraded state, and it takes more fixing to return the software to the place it should be. One way that a software team can learn from an incident is through incident reproduction. When an incident is turned into a reproducible system, it becomes a predictable training exercise rather than a surprising and painful outage. Tammy Butow is an engineer with Gremlin, a company that makes chaos engineering software. Chaos engineering is the process of creating controlled experiments that simulate outages. Tammy joins the show to discuss common incident types, and how those can be made reproducible for training exercises. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Diffbot is a knowledge graph that allows developers to interface with the unstructured web as if it was a structured database. In today’s show, Diffbot CEO Mike Tung returns for a second discussion about how he has built Diffbot and how Diffbot is used. The web has many different entities. Web pages, topics, people, stories, articles, companies, and much more. Humans use a search engine to find answers to their questions within web pages. Machines need to find answers to these kinds of questions as well, but a machine is not sophisticated enough to figure out answers from an unstructured web page. Diffbot brings structure to those webpages, and gives them an API interface for developers to build on top of. In order to create this system in a cost-efficient manner, Diffbot runs its own data centers, where web scraping, machine learning, and API infrastructure are all used to build the Diffbot application. Mike joins me for an interview about creating Diffbot, as well as his strategy for running the business. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Render is a cloud provider built on top of Amazon Web Services and Google Cloud. Render uses the compute abstractions provided by the major cloud providers to build a second layer cloud provider with the goal of providing a better user experience. Anurag Goel is the founder of Render, and he returns to the show to discuss how Render works, and why there is a need for a new cloud provider. Everyone knows that the market for cloud providers is gigantic, so why are so few companies pursuing it? From Anurag’s perspective, there is no good reason. AWS, Google, Azure, Digital Ocean, and Heroku have only explored a small percentage of the potential ways a cloud provider could be built. Anurag shares his strategy for building Render, and also talks through his belief around modern software engineering, including his belief that developers mostly choose their tools based on what they read from popular websites rather than what solves their problems–a phenomenon which he describes as “fashion driven development.” Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Webflow is a platform for building applications without programming. Software engineering has barely been around for 30 years. Over that period of time, there have been many attempts to create a platform that allows for the creation of software without writing a line of code. Most of these systems have not been able to fulfill that task. And this should come as no surprise. It is hard enough to build an application if you know how to program. Vlad Magdolin has been working on Webflow for more than seven years. He has persisted through multiple failed attempts at building Webflow, and pushed past continuous rejection from investors who did not see the viability of his vision. As Vlad patiently worked on Webflow with his two co-founders, the power of the web browser slowly improved. V8 became a powerful runtime that could deliver the performance necessary to build applications visually in the browser. The unmet goals of past WYSIWYG application platforms faded into irrelevance, as Webflow came into being and allowed for an entirely new type of software development, driven by a visual interface in the browser. Webflow is one of the coolest, most ambitious software platforms in existence. Vlad joins the show to discuss Webflow and the future of software development. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data visualization is the presentation of data in a way that emphasizes certain qualities about that data. Data visualization can be used to prove a specific point, or it can be used as a depiction of a data set to be explored. Data visualization is used in consumer software products as well as backend engineering systems such as logging data. As tools for data visualization have improved, data applications can consume more data at a faster pace. Browsers and mobile phones have improved, giving us the power to render high fidelity, complex visualizations in near-real time. Sherman Wood and Chad Lumley are engineers working on Jaspersoft, an embedded analytics tool from TIBCO. Embedded analytics is a type of software that allows for the creation of data visualizations inside of an application. Sherman and Chad join the show to discuss techniques for data visualization and how the field has evolved. Full disclosure: TIBCO is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The ability to create software is a superpower. But software education is not evenly distributed. Ryan Carson started Treehouse to provide a high quality education system for anyone to learn how to build software. On a previous episode, Ryan and I discussed the field of programming education. Ryan returns to the show for a conversation about building Treehouse, and the company’s expansion from an online programming school to a platform for technology apprenticeship. Thousands of people learn to program on Treehouse. Ryan’s goal is to connect those new programmers to companies where they can learn to apply those software skills to working at a technology company. Treehouse’s apprenticeship program is used by Airbnb, Adobe, Mailchimp, and others. By combining an online education platform with an apprenticeship program, Treehouse has created a business model that could allow it to grow significantly bigger even in an atmosphere where learning to program has been commodified to a large extent. The economic model of Treehouse contrasts with in-person bootcamps such as Hack Reactor and online income sharing platforms such as Lambda School. It was a great conversation about modern technology, the future of education, and the strategy of building a successful business. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Video surveillance impacts human lives every day. On most days, we do not feel the impact of video surveillance. But the effects of video surveillance have tremendous potential. It can be used to solve crimes and find missing children. It can be used to intimidate journalists and empower dictators. Like any piece of technology, video surveillance can be used for good or evil. Video recognition lets us make better use of video feeds. A stream of raw video doesn’t provide much utility if we can’t easily model its contents. Without video recognition, we must have a human sitting in front of the video to manually understand what is going on in that video. Veronica Yurchuk and Kosh Shysh are the founders of Traces.ai, a company building video recognition technology focused on safety, anonymity, and positive usage. They join the show to discuss the field of video analysis, and their vision for how video will shape our lives in the future. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data privacy policies have changed how software organizations need to operate. As consumer preferences have shifted in favor of strong privacy, software companies are having to examine their policies around data collection and retention. Many software companies were started in a time with different norms around data. Building a new application that is compliant with GDPR is hard. Updating an existing application to align with GDPR is even harder. Joshua Prismon is chief architect at FICO, a company that builds systems around credit scoring. Joshua joins the show to discuss how a large company like FICO has responded to changing consumer preferences through changes in its software architecture and engineering. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Indie Hackers is a platform for independent software businesses to discuss strategy and find inspiration. Courtland Allen founded Indie Hackers with the goal of sharing the stories of these businesses, and the company has become a thriving community of entrepreneurs, engineers, and creators. Business is a creative medium. The definition of a successful business is as subjective as the rules for what makes a successful work of art. A business owner can be miserable running a company that generates millions of dollars a year, and a new entrepreneur can feel ecstatic from making their first $5 sale. Indie Hackers is a platform that is impossible to define in relation to things you have seen before. It is a media company with a podcast that most SE Daily listeners will probably enjoy. It is a social platform for learning how modern software companies are built. And it is a place where makers post their own progress on their creative projects (I have posted mine here). Courtland was on the show three years ago to discuss the Indie Hackers movement in its nascent stages. Courtland returns to the show to discuss the thriving platform as it exists today, and a wide ranging conversation about software, game theory, and podcasting. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Scott Belsky founded Behance in 2006. Behance is a social platform where designers and creators share their work. Scott was motivated to start Behance due to his desire to combine his love for creativity with his desire to create a business. After 6 years of work, Behance was acquired by Adobe for more than $150 million. Today, Scott works as the Chief Product Officer at Adobe. Behance’s journey from idea to acquisition is told by Scott in his book The Messy Middle. His book chronicles the difficult, winding journey that an entrepreneur must take in order to succeed, and contains some harrowing stories. Scott has a gritty personality, which was required to endure the ups and downs of Behance. Scott joins the show to discuss the story of Behance, and the lessons of his life as an entrepreneur. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Large companies have multiple databases, multiple data formats, and multiple applications that need to use the data. Every data engineer needs to move data between these different components of a system. Moving data between different parts of a system is often called “ETL”, an acronym for “Extract, Transform, Load.” Data engineers spend much of their time writing code for ETL. This ETL code often moves data from a source such as a distributed file system into a data warehouse such as Amazon Redshift. ETL code also is used to take data out of platforms such as Google Analytics, and put that data into a system where it can be joined with internal data sets. George Fraser is the CEO of Fivetran, a company that builds data connectors to improve the data engineering process. George joins the show to discuss modern data warehousing, and the business of building Fivetran. Data connectors may sound like a simple or trivial product, but it turns out to be a gigantic opportunity. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The development of self-driving cars is one of the biggest technological changes that is under way. Across the world, thousands of engineers are working on developing self-driving cars. Although it still seems far away, self-driving cars are starting to feel like an inevitability. This is especially true if you spend much time in downtown San Francisco, where you will see a self-driving car being tested every day. Much of the time, that self-driving car will be operated by Cruise. Cruise is a company that is building a self-driving car service. The company has hundreds of engineers working across the stack, from computer vision algorithms to automotive hardware. Cruise’s engineering requires engineers who can work with cloud tools as well as low-latency devices. It also requires product developers and managers to lead these different teams. The field of self-driving is very new. There is not much literature available on how to build a self-driving car. There is even less literature on how to manage a team of engineers that are building, testing, and deploying software and hardware for real cars that are driving around the streets of San Francisco. Mo Elshenawy is VP of engineering at Cruise, and he joins the show to talk about the engineering that is required to develop fully self-driving car technology, as well as how to structure teams to align the roles of product design, software engineering, testing, machine learning, and hardware. Full disclosure: Cruise is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Investors often use the term “moat” to describe the durable competitive advantage of a company. When an investor puts money into a company, they are making that investment based on a valuation. That valuation is subjective–it is how much the investor thinks the company is worth. A valuation is determined by the present value of future cash flows. What are the future cash flows of a company? In order to figure that out, the investor needs to know how the business will look in the future. This is why moats are so important. If an investor looks at how much money a business made this year, it does not tell the investor very much information about how much money the business will make in the future. If the business has a durable competitive advantage, then that means that the cash flows of the company are also durable. It also means that any compounding of those cash flows will be durable growth. It is easy to understand durability for some businesses. Why do we keep using Google? Because there is no substitute search engine that is so integrated with our daily lives. Why do we keep using Facebook? Because there is no other social networking company that has all of our friends and family. But what about companies with substitutes? There are lots of cloud providers, log management companies, and analytics providers. These are crowded markets, and yet in each of the crowded markets there seems to be a dominant player who captures the most market share. Why is that? How do software companies in competitive markets develop a moat? Astasia Myers is a venture investor with Redpoint, a software investment firm that makes large bets on technology companies. Astasia joins the show to discuss how software companies form competitive advantages, as well as several specific markets such as log management and cloud cost optimization. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software engineering is a new field. There are theories about how we should be building our systems, but these theories might change over time. The same is true for engineering management. There are many successful examples of companies scaling with the management hierarchies pioneered by Microsoft and Google, but since everyone knows that those techniques work, they get continually copied. Of course, there is tremendous risk in pioneering a brand new management structure. There have been examples of successful software companies that have suffered tremendously due to a trial of an exciting new management structure. New software is much easier and safer to try than revolutionary new management structures. New software is easier to try in a company with a strong engineering team, because that team is equipped to assess the new software and figure out if it is actually solving a problem that the company has. Uma Chingunde is an engineering manager at Stripe on the compute team. Uma has worked in management for a decade, and has worked in virtualization in infrastructure for even longer than that. Uma joins the show to give her perspective on management of engineers as well as management of compute infrastructure. We discussed some timeless principles of engineering management, as well as contemporary ideas around virtualization and compute. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ethos Life Insurance is a software company that sells life insurance products. Software is reshaping established industries such as banking, insurance, and manufacturing. In these large, established industries, incumbents are adopting new technology as fast as they can, but the new technology needs to be integrated with the old technology. The slow rate of technology adoption by incumbents creates an opportunity for new companies to spring up who are building their entire company from scratch, with updated software. Insurance is a gigantic market that is dominated by companies which have been around for 50-100 years. The established players in the insurance industry are trusted brands, but many of them have not significantly updated their technology from legacy techniques of pricing risk. Vipul Sharma is the VP of engineering and Lingke Wang is the co-founder of Ethos Life Insurance, and they join the show to describe the insurance business, the technical problems, and the software stack of modern insurance company. Ethos has more than fifty employees and is growing rapidly, so it is a great case study in scaling a modern company in an established market. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Isolation is a fundamental concept in computer science. Software workloads are isolated from each other in order to keep resource access cleanly separated. When programs are properly isolated, it is easier for the programmer to reason about the memory safety of that program. When a program is not properly isolated, it can lead problems such as security flaws where one program can access the information that should be exclusive to a different program. Poor isolation can also lead to garbage collection problems, or running out of disk space. Isolation takes many forms, including individual processes, containers, and virtual machines. The techniques for isolation evolve over time. A more recent technology that can assist with isolation is WebAssembly, a newer execution system that can run a variety of languages that compile down into the WebAssembly binary format. For previous episodes about WebAssembly, you can listen to some of the shows in our archives. Tyler McMullen is the CTO at Fastly, a cloud provider that focuses on edge computing systems such as content delivery networking. Tyler has written and spoken about WebAssembly in detail. He joins the show to talk about computational isolation, and how WebAssembly presents new efficiencies for engineers looking to isolate their workloads. Full disclosure: Fastly, where Tyler works, is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud Foundry is a system for managing distributed applications. Cloud Foundry was released in 2011, and has been widely adopted by enterprises that need a platform for deploying and scaling the applications that run within their company. The ecosystem around Cloud Foundry includes systems for continuous delivery, pubsub messaging, and containerization. Abby Kearns is the executive director at Cloud Foundry Foundation, a nonprofit with the goal of raising awareness and adoption of the Cloud Foundry open source project. Abby joins the show to discuss her work with Cloud Foundry, and how the ecosystem has evolved with the maturity of distributed computing. We also talk about what enterprises need from an application runtime platform and how Cloud Foundry has adopted Kubernetes. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A new software product usually starts with a single database. That database manages the tables for user accounts and basic transactions. When a product becomes popular, the database grows in size. There are more transactions and more users. A company grows around that product, and the company starts to accumulate more data in different sources. Analytics systems, time series databases, and logging tools start to generate data. Moving this data around between systems starts to become complicated. Apache Kafka is often used as a system for moving data between these different systems, performing transformations, and generating aggregations and summaries of these large quantities of data. Robin Moffatt works at Confluent, and has written numerous articles about how to move data between systems and design effective workflows for data pipelines. Robin joins the show to talk about modern data platforms and databases, and the patterns for using Kafka to connect those systems to each other. If you are interested in learning more about how companies are using Kafka, the Kafka Summit in San Francisco is September 30th – October 1st. Companies like LinkedIn, Uber, and Netflix will be talking about how they use Kafka. Full disclosure: Confluent (the company where Tim works) is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A software company needs to get many things right in order to be successful. Having a useful product with solid engineering is only the beginning. ReadMe was started five years ago. The company solved a seemingly simple problem: documentation for software products. If you have worked as a software engineer, you have looked at documentation. You know that there is a wide range of quality among the documentation that exists for different software products. ReadMe solved the problem of documentation-as-a-service, and it solved the problem better than any other company in the market. But after building a great business around documentation, the direction in which to take the business was unclear. How do you expand a documentation software business? Should you start building API management systems? Should you build lead generation tools? Should you try to sell products to developer evangelists? Greg Koberger is the CEO of ReadMe, and he joins the show to talk about the business, the strategy, and the fundraising process for ReadMe. Greg was previously on the show very early on, and he stood out as a guest that was remarkably friendly and willing to talk about a wide range of topics. In the previous show with Greg, he was one year into his business. Today, it has been five years since ReadMe was started, and we catch up on how his business has evolved since our first interview. ReadMe had become a profitable business very early in its life. In Silicon Valley, a quickly profitable business is the exception. Rather than structuring the finances of ReadMe with the expectation of successive funding rounds every 18-24 months, Greg took a slower approach. His company grew at a rate that was affordable while maintaining that profitability. While ReadMe maintained profitability from its core product of documentation software, Greg patiently thought about what product to build next within the company. A software company will typically expand into adjacent markets, or offer products that their current users are willing to pay additional money for. Eventually, ReadMe found its second product: developer metrics. Greg and his team figured out that gathering metrics around how APIs are being consumed has synergies with the core ReadMe product. With a second product, ReadMe was now able to forecast significant growth. The company also gained an incentive to raise money. With additional money, ReadMe would be able to deepen the developer metrics product, hire a bigger team, and think much bigger than they would have been able to as a single-product company with a smaller total addressable market. ReadMe raised a series A from Accel, one of the most respected venture capital firms in the business, and the company is now entering a new period in its development. ReadMe is a fascinating case study in balancing ambition with financial discipline at a software company. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Open source software is very new. Open source has existed for less than 30-40 years, depending on who you ask. The idea of open source was popularized by Linux, and open source software started to get heavily commercialized in the 1990s. By the early 2000s, open source was used by nearly every large software company. In recent years, most of the new databases, web frameworks, user interface libraries, and other software primitives are being built in the open. Dirk Hohndel is VP and chief open source officer at VMware, where he works on strategies around Linux, containers, Kubernetes, and other open source software. Dirk joins the show to talk about his long history with open source and the current state of the open source ecosystem. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern databases consist of multiple servers that host the data in a distributed fashion. Using multiple servers allows a database to be resilient to the failure of any one database node, because copies of the data are shared to other servers. A multi-node setup also lets the database grow beyond the size of data that could be hosted on a single node. Although a distributed database gains in scalability and resiliency, a database that runs across multiple nodes has a variety of problems that are not faced by a database running on a single node. Every operation with a distributed database becomes more complex than the single-node database. For example, if you make a query to your distributed database, you might not be able to rely on the answer that you get from a single database node, because the other database nodes might have been involved in transactions that have not propagated to all of the nodes. Aly Cabral is a lead product manager at MongoDB and co-author of a paper on causal consistency in MongoDB. Aly joins the show to discuss the engineering of distributed databases and her experience architecting MongoDB. Full disclosure: MongoDB is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ever since Apache Kafka was open sourced from LinkedIn, it has been used to solve a wide variety of problems in distributed systems and data engineering. Kafka is a distributed messaging queue that is used by developers to publish messages and subscribe to topics with a certain message type. Kafka allows information to flow throughout a company such that multiple systems can consume the messages from a single sender. In previous shows, we have covered design patterns within Kafka, Kafka streams, event sourcing with Kafka, and many other subjects relating to the technology. Kafka is broadly useful, and new strategies for using Kafka continue to emerge as the open source project develops new functionality and becomes a platform for data applications. In today’s episode, Tim Berglund returns to Software Engineering Daily for a discussion of how applications are built today using Kafka–including systems that are undergoing a refactoring, data engineering applications, and systems with a large number of communicating services. If you are interested in learning more about how companies are using Kafka, the Kafka Summit in San Francisco is September 30th – October 1st. Companies like LinkedIn, Uber, and Netflix will be talking about how they use Kafka. Full disclosure: Confluent (the company where Tim works) is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A new employee at a software company needs access to a variety of tools. In order to get started working, the employee might need Slack, email, Google Docs, and Amazon Web Services, and all of these require an account with a username and password. Setting up all of these accounts can be time consuming, because the company needs to go into their admin portal and create the accounts. The accounts need to have the right security policies and configuration settings. And when the employee leaves the company, all of these accounts need to be shut down. Okta is a company that builds identity and access management software, such as an “SSO (single-sign on)” tool that allows users to log into all of these different types of accounts using only an Okta login. Okta was started in 2009 and has grown steadily since then, going public in 2017. Hector Aguilar is the president of technology at Okta and he joins the show to talk about the software stack of Okta and how the company has evolved over time as it has become a core infrastructure provider and hired a large engineering team. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Amazon Web Services first came out in 2006. It took several years before the software industry realized that cloud computing was a transformative piece of technology. Initially, the common perspective around cloud computing was that it was a useful tool for startups, but would not be a smart option for large, established businesses. Cloud computing was not considered economical nor secure. Today, that has changed. Every company that writes software is figuring out how to utilize the cloud. Software companies with on-prem servers are migrating old applications to the cloud, and most companies that have started in the last decade do not even have physical servers. Applications that are started on the cloud are referred to as “cloud-native.” The architecture of cloud-native applications is a newer topic of discussion, and some software patterns that became established in the pre-cloud era make less sense today. Cornelia Davis is VP of technology at Pivotal and the author of Cloud Native Patterns, a book about developing applications in the distributed, virtual world of the cloud. Cornelia was previously on the show to discuss Cloud Foundry. In today’s episode, our conversation centers on her book, and her perspective on the emerging patterns of cloud native software.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Python is one of the most popular programming languages in the software world. After working with Python and developing a love for the language, Michael Kennedy started to wonder why there was not a high quality podcast dedicated to covering the community and new technologies of the Python ecosystem. Michael started Talk Python To Me as a podcast with the goal of telling stories from the world of Python. Today, Talk Python To Me is the most popular podcast dedicated to Python and related technologies. Subjects on the podcast include web frameworks, compilers, career development, and cloud services. One of the more recent developments in the world of Python is the prevalence of data science, which Michael also covers in great detail. Michael has also spun up a second podcast called Python Bytes, which offers timely updates to the Python community. Michael joins the show to share his thoughts on several topics related to Python, including compilers, data science workflows, and web frameworks. Michael also gives his perspective on the world of software podcasting, which he has been doing for more than four years.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A new software application has simple requirements for a database. The database needs to be written to and read from. The database fulfills simple needs such as storing user information and providing the application frontend with the necessary data to render a simple webpage of financial transactions or blog posts. As an application becomes successful, the database grows in size. The complexity of queries increases, requiring more sophisticated logic in order to maintain performance. New databases need to be added to the overall system, as users begin to have demands for advanced features such as search, or analytics. Over time, the requirements for a database expand into a need for a data platform. A data platform might include multiple databases such as a NoSQL database, a relational database, a data warehouse, and a search index. The relationships between these different databases can vary in terms of consistency requirements, latency, and scalability. Andrew Davidson is the Director of Cloud Products at MongoDB. In a previous episode, Andrew discussed the tradeoffs of scaling databases while maintaining high performance indexing. Andrew returns to the show to discuss the emerging subject of “data platform.” As a growing number of companies have data requirements beyond that of a simple transactional database, Andrew’s work has increasingly involved figuring out the best ways for developers to adapt those transactional systems to providing a wider set of functionality. Full disclosure: MongoDB is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Spanner is a globally distributed, transactionally consistent database. Spanner initially emerged as a paper that came out of Google in 2012. Around this time, database scalability was difficult to solve, even for Google. The Spanner paper offered some breakthroughs in distributed systems which allowed Google to take some of the learnings from BigTable’s eventual consistency and construct a novel system that was transactionally consistent. Deepti Srivastava has worked on Spanner for more than six years. When she initially started working on Spanner, she was involved in the internally facing Spanner system that Google engineers used to spin up instances of the database. Today, she works as program manager for Cloud Spanner, a product within Google Cloud. Deepti joins the show to describe the breakthroughs of Spanner, and how her work has evolved since the product has gone from an internal system to an externally facing piece of cloud infrastructure.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Open source plays a key role in today’s world of technology businesses. Today, the impact of open source seems obvious. From Kubernetes to distributed databases to cloud providers, so much of our software is powered by open source. But it was not always this way. Bruce Perens was one of the earliest figures in the world of open source. He collaborated with Eric S. Raymond, the author of Cathedral and the Bazaar. He developed a differing belief set from Richard Stallman, who was another foundational advocate of open source. Bruce worked as an engineer at Pixar from 1987 to the year 2000. Bruce joins the show to give a history of open source and his experience in the software industry. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The browser has become the central application of the consumer operating system. Every piece of client software, from email to document management, has become usable through the browser. Even modern desktop software such as Slack is built using Electron, a tool for building client applications that essentially run inside of a browser without an address bar. One activity that still takes place largely outside of the browser is the process of writing and deploying code. A developer often uses an IDE such as Eclipse to write their code, then switches over to a terminal where they can build and deploy their code to a remote server running in the cloud. For a developer who has been writing code for a long time, this process feels completely intuitive. But for a new developer, it can be totally disorienting. New developers sometimes have trouble understanding the difference between a local and remote environment, or how to use repository management software like Git. This is in addition to all the other problems a new developer might be dealing with, such as language installation, syntax, and package management. Repl.it is a browser-based coding environment, compute engine, and collaborative workspace. Repl.it has found significant traction among new programmers who begin their programming journey within Repl.it and then stay in the environment, even as their applications become more richly featured and complicated. Repl.it is an amazing piece of software, and the story behind it is remarkable. Amjad Masad had the idea for Repl.it many years before he started the company for it, but he needed to first build up the money and confidence in order to go after the business with full force. Amjad joins the show to talk about his long journey towards building Repl.it, and to discuss the thriving Repl.it platform in its current form.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern software is built with cloud services, APIs, and other high level tools. Technical software development is moving beyond the realm of writing code. Individuals who do not have a background in computer science or programming can create increasingly complex tools. Higher level APIs include Twilio for managing phone communications, and Stripe for managing financial workflows. Platforms such as Shopify can be used as the core system for building an e-commerce business. Low-code tools such as Wix can be used for building web sites. Monday.com was started in 2012 with the goal of making software for managing business workflows. Business workflows link together different teams and processes within a company. Monday allows non-engineers to automate workflows that are traditionally automated by engineers. For example, Monday can be used to build workflows that link together SalesForce and Twilio, or to trigger actions in MailChimp based on how a user interacted with a Shopify web site. One of the biggest developing trends in technology is how non-engineers are becoming empowered to have more impact within an organization. The advances in these kinds of tools present us with an opportunity to rethink team structures, and invent entirely new roles for a software company. Eran Zinman is the CTO and co-founder of Monday.com and he joins the show to describe the vision for Monday and how the product fits into the changing enterprise software trends around APIs and “low code” tools. Monday has grown exponentially over the last 7 years, with the most recent growth curve looking almost vertical. Eran’s experience scaling the product and improving performance makes for excellent storytelling. Full disclosure: Monday is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cassandra was initially released in 2008 as a project out of Facebook. Cassandra offered an open source solution to database scalability issues that were being tackled internally by large companies like Amazon, Google and Facebook. 2008 was a golden age of new infrastructure, with systems such as Hadoop and Kafka gaining traction around the same time. Jonathan Ellis started working with Cassandra, and became intrigued by the system. With the help of investors, Jonathan began working on a company based around Cassandra called Datastax. Today, Datastax is a company with more than 450 employees and a large valuation. Jonathan joins the show to discuss his experience working with Cassandra, and his reflections on the becoming an entrepreneurial founder of a highly successful database company. There is a wealth of useful knowledge for both software engineers and entrepreneurs in this episode.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Airlines have always had an emphasis on new technology. Over the years, airlines have needed to develop more and more software. Digital transformation is causing every large company to adopt the tools and practices of software companies, and that includes Delta Airlines. Delta Airlines has existed for more than 90 years. Over that period of time, the company has developed new systems in every generation of software, from the days of mainframe computers to a modern Java-based backend. When the DevOps movement started to take shape, Delta Airlines started to take a more focused approach on continuous integration, version control, and an organizational structure that removed silos between teams. Jasmine James is a manager at Delta Airlines where she focuses on improving the software practices of the company. She joins the show to talk through the process of changing the developer culture within Delta, as well as what it is like to build software for an airline. Jasmine is speaking at GitLab Commit in Brooklyn on September 17, 2019, and GitLab is a sponsor of Software Engineering Daily, so if you are thinking about attending GitLab Commit, you can go to softwareengineeringdaily.com/commit and enter code COMMITSED to save 30% on your ticket, while also supporting Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ilya Sukhar founded Parse in 2011 to make cloud services for mobile developers. Parse was a newer kind of cloud provider called “backend-as-a-service”, built to simplify the complexities of Amazon Web Services and the complexities of the mobile ecosystem. During this time, Facebook was in the middle of its shift toward becoming a mobile application company. Ads on the smartphone were not yet a proven business model. Facebook was exploring other business lines, and decided to purchase Parse for $85 million, with the intention of building a cloud developer platform. Shortly after the acquisition, Facebook’s mobile ads business started to see considerable success. With the mobile ads business finding traction, Facebook shifted all available resources towards supporting that business model. In 2017, Parse was shut down. Ilya joins the show to give his experience starting Parse, selling the company to Facebook, and then seeing the company he had built get shut down as it became an unfortunate casualty of Facebook’s advertising success. We talked a lot about the experience of building a backend-as-a-service company, as well as what makes Facebook special as an organization, and how the success of Facebook’s mobile business happened to coincide with the Parse acquisition.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Parse was a backend-as-a-service company that started in 2011 with the initial focus of making the cloud easier to use for mobile developers. Parse had several novel engineering challenges. In 2011, it was not easy to build on top of AWS, nor was it easy to build within the young mobile ecosystem. Charity Majors was an early engineer at Parse, and stayed with the company through its acquisition by Facebook. Charity joins the show to discuss her experience at Parse and her reflections on being part of Facebook after the acquisition. Parse was shut down in 2017, which was a disappointing outcome for Charity. Charity currently works as the CTO of Honeycomb. In a previous episode, Charity gave some background on her career in software, as well as her perspective on modern problems in observability and DevOps. In this episode, Charity tells the story of joining Parse, working through novel engineering problems, and the feeling of being acquired by a company where she did not feel like a great cultural fit. It is a useful conversation for anyone interested in engineering culture, company philosophy, and the realities of acquisitions–which can be both painful and profitable for the acquired employees.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kent Beck is a legendary figure in the world of software engineering. Kent was an early advocate of Test-Driven Development (TDD), and popularized the idea of writing unit tests before writing code that would satisfy those unit tests. A unit test isolates and tests a small piece of functionality within a large piece of software. Practitioners of Test-Driven Development write tens or hundreds of tests in order to cover a large variety of cases that could potentially occur within their software. When Kent Beck joined Facebook in 2011, he was 50 years old and thought he had seen everything in the software industry. During Facebook Boot Camp, Kent started to realize that Facebook was very different than any other company he had seen. Facebook Boot Camp is the six-week onboarding process that every new hire learns about the software practices of the company. After graduating Facebook Boot Camp, Kent began to explore Facebook’s codebase and culture. He found himself rethinking many of the tenets of software engineering that he had previously thought were immutable. Kent joins the show to discuss his time at Facebook, and how the company’s approach to building and scaling products thoroughly reshaped his beliefs about software engineering.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When Chuck Rossi joined Facebook in 2008, he was one of the most experienced release engineers at the company. As he began to explore the engineering practices of the organization, he was surprised, confused, and impressed by the release engineering system that he encountered. Release engineering is the process by which software is released to users. As software is being developed, it moves through a series of testing environments. In these test environments, the software can be studied using simulated inputs that can help developers discover software bugs. Chuck had come to Facebook from Google. At Google, the crown jewel was Google web search which had a regimented release process. At Facebook, the crown jewel was facebook.com. Chuck found that the release process for facebook.com was much different than Google web search. Chuck joins the show to talk about release engineering at Facebook, and how the company constantly evolved its code deployment process. Chuck also describes Facebook’s pivot to mobile, and how the bottlenecks in the mobile app release process threatened Facebook’s ability to iterate and release quickly. This show provided some amazing perspective on continuous delivery, and will be useful to anyone who is working on figuring out their “DevOps” process. Chuck has a wealth of knowledge and context about the modern software industry.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook is a large multiuser application. Scaling Facebook was different than scaling a single-user application such as an ecommerce store or a search engine. A social network is faced with unique infrastructure scalability challenges, as well as subjective questions around user communications, privacy, and content. Pedram Keyani worked at Google before joining Facebook in 2007. In his years at Facebook, he worked on infrastructure, internal tools, and management. He became deeply familiar with the company culture and its operations. Pedram joins the show to talk about how Facebook has scaled and the lessons he took away from his time there. After his time at Facebook, Pedram joined Uber, where he worked as a director of engineering for four years. Uber is another multi-user application, with a very different set of constraints. At Uber, Pedram worked on several projects, including Uber’s push into China, which he describes as an intense, competitive experience. Pedram is able to contrast the culture and scaling processes of Uber, Facebook, and Google which made this a rare opportunity to see how three different high performing companies build software differently.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cryptocurrencies are decentralized monetary systems built on open source software. The open source software movement has evolved from the world of Linux, MySQL, and Apache to a thriving ecosystem of commercial enterprises built around open source software. This ecosystem includes projects such as Kubernetes, MongoDB, and ReactJS. It includes large organizations such as Amazon Web Services, Elastic, and Facebook. In a parallel software universe, the crypto ecosystem has built revolutionary new financial tools, creating billions of dollars of value, but not very many massive commercial companies. In the world of cryptocurrencies, many of the same rules apply to the classic open source world. But other rules do not. How do these two worlds differ from each other? How are they the same? And how might they end up colliding? Haseeb Qureshi, Joseph Jacks, and Alok Vasudev join the show for a spirited discussion of cryptocurrencies and open source. Haseeb is a cryptocurrency investor, JJ is the founder of OSS Capital, and Alok is an engineer and the founder of crypto venture capital firm Standard Crypto.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a new employee joins a software company, it is often unclear where that employee should begin. Do they have a mentor? What are they working on? What are the expectations for how fast that employee should be contributing? The early period of employment is often referred to as “onboarding.” During the onboarding period, an employee will learn the basics of the company they have just joined. These basics include the work procedures, meeting schedules, expectations, and technical tools that an employee needs to become productive. A poor onboarding process can slow an employee down, and even cause them to quit the company entirely. Conversely, a good onboarding process can accelerate an employee towards rapid contribution within the organization. A scalable onboarding process can be a difference in millions of dollars in productivity per year across the organization. Kristen Gallagher is the founder of Edify, a company that helps organizations define and implement their onboarding process. Kristen joins the show to talk about the ingredients of a successful onboarding process and what she is doing with her company Edify.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A time series database is optimized for the storage of high volumes of sequential data across time. Time series databases are often organized as columnar data stores that can write large volumes of data quickly. These systems can sometimes tolerate data loss, because the data they are gathering is used for monitoring and other applications that require aggregated data sets rather than highly important individual transactions. The demand for time series databases has grown over the last decade with the rise of mobile devices and the decreasing cost of cloud storage. There has been an increase in the number of systems that require monitoring, and some of those systems produce an incredibly large amount of data, requiring compression, downsampling, and garbage collection. Rob Skillington is an engineer at Uber, where he helped create M3DB, a time series database. In a previous show, Rob described the basics of M3DB and how it helps Uber with storing data from Prometheus, a monitoring system. In today’s show we discuss the field of time series databases, and Rob’s approach to building M3.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Insurance is an old business. Individuals and businesses have been buying insurance policies for decades. These insurance policies can cost hundreds, thousands, or tens of thousands of dollars per year. Software is remaking the insurance industry. One way new software can improve the insurance industry is through better brokerage technology. Insurance involves carriers and brokers, who work together on delivering insurance solutions to customers. The initiation and closing of an insurance transaction often involves lots of emails, PDF files, and antiquated software systems. Technology improvements will smooth out this process and reduce manual overhead. Another way insurance can improve thanks to technology is through smarter pricing. The price of an insurance policy is offered to a customer based on how risky the insurance policy is, how large the customer pool is, and how much the insurance company could lose in the event that it would have to pay out an insurance policy. These risk profiles are calculated based on historical data. Gordon Wintrob is the co-founder and CTO of Newfront Insurance. Gordon joins the show to discuss the insurance industry and how his company got started.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Open source software has evolved into a thriving, multifaceted ecosystem. Open source encompasses operating systems and databases. Open source embodies both altruism and self-interest. And open source enables thriving businesses from WordPress blogs to hundred billion dollar cloud providers. There is a large set of business models that can be built around a successful open source project. Some of these business models are more defensible than others. A company based around open source must make a deliberate set of tradeoffs around how their private company will relate to the public open source ecosystem. Today’s episode is a discussion of open source business models with Karthik Ranganathan, Heather Meeker, and Matt Asay. Karthik is the CTO of YugaByte, an open source distributed SQL database. Heather Meeker is an open source licensing specialist and a founding partner of OSS Capital. Matt Asay is a longtime executive who has worked in several open source based companies including MongoDB–he has also written at length about open source. Full disclosure: YugaByte, where Karthik works, is a sponsor of SE Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Andreas Antonopoulos is the author of several books about cryptocurrency engineering, including Mastering Bitcoin and Mastering Ethereum. In these books, Andreas lays out the systems of economics and computer science that underpin the two most mature decentralized monetary systems. When Andreas originally discovered the Bitcoin whitepaper, he had witnessed the repeated mismanagement of government-backed fiat currencies. Andreas has a Greek background, and the financial collapse of 2008 had led to an economic crisis in Greece. His firsthand observation of the weaknesses of centrally planned government currencies, together with a degree in computer science and distributed systems have made him a dedicated evangelist for Bitcoin. Andreas joins the show to discuss the Bitcoin ecosystem, and the relationship between decentralized cryptoeconomic systems and centralized corporations. Facebook has recently announced a cryptocurrency project called Libra, and Andreas suggests that Libra changes everything–not necessarily because Libra will make it to production, or because Libra itself will upend the world of finance–but because it allows us to further call into question the very nature of what makes a modern currency valuable and valid. After all, if a large government has the right to back a currency, why shouldn’t a large corporation have that same privilege? Andreas is also a co-host of one of my favorite Bitcoin podcasts, Let’s Talk Bitcoin.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software engineers often work as a contractor for some duration of their career. A contractor earns a fixed hourly salary for a defined period of weeks, months, or years. Contract work can be more flexible than full-time work, and often pays more than full-time software engineering, because contract jobs can end at any time, and they do not have the added employee benefits such as health insurance and stock options. Online contracting platforms such as Upwork and Fiverr have expanded the number of software contracting engagements that take place. Developers on Upwork and Fiverr have a wide range of skills and experience levels. The clients who come on Upwork and Fiverr looking for developers are sometimes unsophisticated at managing software projects, and in some cases they defraud the software developers who are making their living online, and trick these software developers into delivering free work. Moonlight is a contracting platform for software engineers. Today’s guests Emma Lawson and Philip Thomas are the founders of Moonlight, and they join the show to explain why they started the company, and the gaps that exist in the world of software contracting. Moonlight’s model is different than most other contracting platforms, in that Moonlight requires clients to pay a $300 subscription fee to recruit engineers on the platform. This $300 subscription price lowers the rate at which clients take advantage of software engineers. This dynamic causes the software engineers to take their work more seriously and act more professionally. Full disclosure: I am an investor in Moonlight. I have also been a paying client of the service, and our newest version of the Software Daily app was built by Mostafa Gazar, who is a talented Android engineer I met on Moonlight.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The service mesh abstraction allows for a consistent model for managing and monitoring the different components of a microservices architecture. In the service mesh pattern, each service is deployed with a sidecar container that contains a service proxy. These sidecars are collectively referred to as the “data plane.” Each sidecar provides the service that it is deployed next to with a set of features such as security policy, rate limiting, and monitoring instrumentation. The sidecars in the data plane communicate with a central module called a control plane. In the control plane, an engineer can operate across these individual services at scale, by pushing out updates to them. Kubernetes has made it easier to manage large fleets of microservices, and has led to wider adoption of service mesh. Istio is one of the most popular service mesh products. In today’s show, Varun Talwar returns to the show to describe the state of the Istio project and the process of deploying Istio to a cluster. Varun is the CEO of Tetrate, a company building an enterprise-ready service mesh. Prior to Tetrate, Varun was at Google, where he helped found the gRPC and Istio projects.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The PlayStation is a line of game consoles created by Sony. PlayStation devices include the PS2, PS3, PS4, and the PSP mobile system. Tony Godar worked as an engineer in the PlayStation ecosystem for 15 years, and he joins the show to give a retrospective on his time in the console industry. Developing hardware and software for game consoles differs significantly from the world of web development. Tony describes the culture of the game development world, and the challenges involved in the domains of software tooling, custom operating systems, and streaming media. In 2010, the PS3 was hacked by notorious tinkerer George Hotz, a previous guest on the show, an event which Tony also discusses. We also discuss the world of modern gaming and VR technology. Tony currently works as an engineer at MelodyVR, a company that makes virtual reality live music experiences.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Large software companies have become a target for criticism. Google, Facebook, Amazon and other prominent technology giants find themselves under a kind of scrutiny that is reminiscent of banks in 2008 and oil companies in the early 1900s. Across the planet, there is a growing sentiment that “big tech” has too much power, and that they are abusing that power in order to dominate markets, censor speech, and manipulate politics. Tyler Cowen is the author of Big Business: A Love Letter to an American Anti-Hero. He is also the host of the popular podcast Conversations with Tyler and an economist at George Mason University. In Big Business, Tyler explores the modern relationships between consumers and businesses, including the large technology companies. Tyler joins the show to discuss his new book. In previous episodes with Tyler, we also talked about his previous books as well as the effects of technology on American society and the philosophical approaches software engineers can bring to their careers.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In this episode Will Larson, author of An Elegant Puzzle: Systems of Engineering Management, speaks with Uma Chingunde of Stripe and Jeff Meyerson of Software Engineering Daily about engineering management. Will was also featured on SE Daily recently. An Elegant Puzzle is an excellent resource on management techniques and strategies for scaling software organizations. Will’s writing draws from his experiences working at Uber and Stripe.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The a16z Podcast is a show that is produced by Andreessen Horowitz, an investment fund based in Silicon Valley. The a16z Podcast covers topics including software engineering, biology, media, cryptocurrencies and entrepreneurship. A16z is one of the most popular podcasts about technology. Sonal Chokshi is the editor in chief at Andreessen Horowitz and the showrunner for the a16z podcast. For five years, she has been interviewing entrepreneurs, engineers, artists, and investors, exploring how software has increasingly impacted our lives and transformed society. The success of the a16z Podcast is largely a result of Sonal’s high editorial standards and her ability to ask the right questions and drive conversations in fruitful directions. Much of the content of Software Engineering Daily has been shaped by a16z, and I have listened to every single episode. Sonal Chokshi joins today’s show for a conversation about podcasting and technology. Sonal shares her beliefs for why the podcast medium has taken off, and describes how her background in education, ethnography, and technology has shaped the completely distinct voice and flavor of the a16z Podcast. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data-as-a-service businesses offer paid access to data sets. These data sets can be useful for building products or training machine learning models. There has been steady growth in the tools and practices around processing and storing data. But access to data sets remains a bottleneck for widespread development of machine learning applications in a large set of domains. SafeGraph is a company focused on the problem of data-as-a-service. Today, SafeGraph’s primary product is reliable, up-to-date location information on places. The data for these points-of-interest needs be acquired, verified, cleaned, made accessible through an API, and intelligently priced. In previous episodes with SafeGraph, we have explored the basic premise of data businesses and why they are important platforms for building futuristic data products that are impossible for entrepreneurs to build today. SafeGraph CEO Auren Hoffman returns to the show to discuss the data as a service business model. Software-as-a-service has existed as a category for more than a decade. Infrastructure-as-a-service has existed for just as long. Data-as-a-service is much more undeveloped. Auren recently published the “Data-As-A-Service Bible: Everything You Wanted To Know About Running A DaaS Business”. This was a very useful article, as it breaks down a category of software that is almost entirely unexplored.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A large sales organization has hundreds of sales people. Each of those sales people manages a set of accounts who they are trying to close sales deals on. Sales people are overseen by managers who ensure that the sales people are performing well. Directors and VPs ensure the scalability and health of the overall sales organization. The sales lifecycle mostly takes place within a piece of software called a CRM: customer relationship management. This tool documents the interactions between sales people and accounts. CRMs have been around for many years, and although CRM software is a useful repository of data, it does not fulfill all the needs of a salesperson. People.ai is a system of machine learning tools built around the sales tooling ecosystem. People.ai helps a sales organization avoid manual data entry, understand areas of potential improvement, and decide on who the highest value sales lead to pursue might be. Andrey Akselrod is the CTO At People.ai and he joins the show to discuss the potential applications of machine learning in the domain of sales, and the engineering work that his company has done.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

During 2015, Uber was going through rapid scalability. The internal engineering systems were constantly tested by the growing user base. Over the next two years, the number of internal services at Uber would grow from 500 to 2000, and standardizing the monitoring of all these different services became a priority. After working with a variety of available tools, Uber’s engineering team decided that something new needed to be built internally. Jaeger is an open source distributed tracing tool that provides observability features throughout Uber’s microservices architecture. Yuri Shkuro is an engineer at Uber, where he works on Jaeger and other infrastructure projects. He joins the show to discuss the history of engineering at Uber, the architecture of Jaeger, and the requirements for building and scaling a distributed tracing tool.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A knowledge base assembles information from a wide variety of sources into a central platform. The most popular knowledge base is Wikipedia, which covers a wide variety of concepts through a system that attempts to remain authoritative and impartial. Other open knowledge platforms include Stack Overflow, which focuses on programming concepts, and Quora, which adds a social element to the process of information accumulation. Golden.com is a knowledge base that indexes, categorizes, and surfaces information. Golden has information about software, genetics, world history, social media, sports, and all kinds of other subjects. The company monetizes with a paid knowledge base product for enterprises, which allows for easy easy querying of the open knowledge base and a private knowledge base for internal information. Jude Gomila is the founder of Golden, and joins the show to discuss the process of building a universal knowledge base, and the engineering problems that he is working on to improve Golden.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Coding bootcamps allow anyone to become a programmer at a faster pace than the traditional computer science education system. In the last five years, coding bootcamps have grown rapidly in popularity, with thousands of people gaining the necessary skills to work as a software engineer. Career Karma is a platform that allows individuals to find the best coding bootcamp. There are many coding bootcamps, and they are not all the same. Much like different schools have different cultures and focus on different disciplines, coding bootcamps vary widely in their teaching styles and acceptance path. Reuben Harris and Artur Meyster are co-founders of Career Karma, and they join the show to discuss the changing nature of software engineering education and the frictions that new programmers encounter as they navigate the world of coding bootcamps. They also describe their journey to entrepreneurship and their own personal experience with coding bootcamps.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A hackathon is an organized event where participants work together to build a product or tool. Hackathons are about creativity, learning, and exploration. A developer that is participating in a hackathon is often working on something that is outside of their normal day-to-day focus. Hackathons can provide significant value to the participants. Hackathons have led to friendships, new companies, and newly developed confidence that can be critical to a developer who feels uncertain in their ability to launch their own projects. Jonathan Gottfried is a co-founder of Major League Hacking, an official student hackathon league that powers invention competitions. Major League Hacking is a B corporation that is focused on improving the education and community of technology leaders, entrepreneurs, and young hackers. Jon joins the show to discuss hackathons–and how he has built and scaled an organization that is devoted to creating systematically successful hackathon experiences.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Videos play a key role in the continuing education of a software engineer. Video can capture many different types of content that is useful for engineers: conference talks, tutorial videos, and podcast-style interviews are all popular formats of online video. YouTube has become the predominant source for video content about software engineering. The open nature of YouTube’s format allows for a tremendous range of content. No matter what your preference is for how you like to learn and be entertained, YouTube has something for you. TechPrimers is a media channel that is dedicated to sharing technical knowledge in the form of videos, GitHub repositories, and a thriving community. TechPrimers has over 300 videos on YouTube, and more than 48,000 subscribers who regularly watch content about subjects like AWS, Spring, and Kubernetes. TechPrimers was founded by Ajay Kumar, a software engineer and vice president at JP Morgan. Ajay has been in the software industry for fifteen years, and has been working in enterprise banking systems for seven years. He has deep experience in modern technologies and engineering practices. Ajay joins the show to discuss the modern world of software engineering, and his experience building a media platform. He also talks about the technology culture of India. Ajay is based in Bengalaru, and it was exciting to learn how much our different societies have in common thanks to technology. Ajay also had me on his YouTube channel for an interview, which was a lot of fun.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

“Internet of Things” is a term used to describe the increasing connectivity and intelligence of physical objects within our lives. IoT has manifested within enterprises under the term “Industrial IoT,” as wireless connectivity and machine learning have started to improve devices such as centrifuges, conveyor belts, and factory robotics. In the consumer space, IoT has moved slower than many people expected, and it remains to be seen when we will have widespread computation within consumer devices such as microwaves, washing machines, and lightswitches. IoT computers have different constraints than general purpose computers. Security, reliability, battery life, power consumption, and cost structures are very different in IoT devices than in your laptop or smartphone. One technology that could solve some of the problems within IoT is WebAssembly, a newer binary instruction format for executable programs. Jonathan Beri is a software engineer and the organizer of the San Francisco WebAssembly Meetup. He has significant experience in the IoT industry, and joins the show to discuss the state of WebAssembly, the surrounding technologies, and their impact on IoT.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data engineering involves numerous tools–a data lake, databases, data warehouses, numerous APIs, streaming systems, and microservices. There is no shortage of ways to interact with data and manage data, but many companies are struggling to figure out design patterns and best practices for how to manage data and build data infrastructure. Zhamak Dehgani is a principal consultant and portfolio director with ThoughtWorks, where she works with enterprises to improve their software systems and workflows. She is the author of an article called How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh and she joins the show to discuss her perspective on data infrastructure, as well as the “data mesh” concept that she has coined. Data mesh represents the movement away from having a centralized data lake that all teams interact with, and towards having different data products and individual data management systems for individual domain teams.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software companies such as Slack, Zoom, and Uber have recently gone public. When a company goes public, they issue a document called an S-1. Within the S-1, there is a wealth of information about the company, providing a detailed story about the company’s business model, economics, and future prospects. The S-1 describes the operating model and the philosophy of a newly public company. Going public serves several purposes. Being public allows a company to gain access to the public capital markets. It allows previous investors to have a liquidity event, by selling the shares that they purchased from the company in private markets. Being public also puts some constraints and visibility on a company, which can be useful for a company that is trying to develop internal discipline. In the software industry, it is useful for most people to understand the dynamics of going public. A technology worker who is earning equity at a private company needs to understand the roadmap to their company going public, or potentially getting acquired. Anyone who invests in public technology stocks is evaluating the different available options for investment, and considering the best software companies to place a bet on. Tom Tunguz is a venture investor at Redpoint, and the author of a popular blog at tomasztonguz.com. In a recent series of posts, Tom has evaluated the S-1s and compared the growth dynamics between a variety of newer public software companies. Tom joins today’s show to discuss his writing, and offer reflections on what can be learned about company building from the recent series of IPOs and direct listings.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Envoy is an open source edge and service proxy that was originally developed at Lyft. Envoy is often deployed as a sidecar application that runs alongside a service and helps that service by providing features such as routing, rate limiting, telemetry, and security policy. Envoy has gained significant traction in the open source community, and has formed the backbone of popular service mesh projects such as Istio. Envoy has been mostly used as a backend technology, but the potential applications of Envoy include frontend client applications as well. The goal of Envoy is to make the network easier to work with–and the network includes client applications such as mobile apps running on a phone. Envoy Mobile is a network proxy for mobile applications. Envoy Mobile brings many of the benefits of Envoy to the mobile client ecosystem. It provides mobile developers with a library that can simplify or abstract away many of the modern advances that have been made in networking in recent years, such as HTTP2, gRPC, and QUIC. Matt Klein is the creator of Envoy, and he joins the show to discuss Envoy Mobile. Matt describes how the networking challenges of mobile applications are similar to those of backend systems and cloud infrastructure. We discuss the advances in networking technology that Envoy Mobile helps bring to the mobile ecosystem, and also touch on the scalability challenges that Matt is seeing at Lyft.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the developed world, it is easy to take for granted that we grew up with computers. Technology is so pervasive in the United States that we have debates about how early in child development a human should be given a smartphone. Across much of Africa, there is a shortage of access to computers. Children grow up without much exposure to computers at all. Smartphones are starting to proliferate the continent, but the bandwidth limitations prevent the sort of unrestricted mobile Internet usage that many of us have in the west. Tyler Cinnamon is a software engineer and the co-founder of TechLitAfrica, an organization dedicated to improving technology literacy and reducing poverty in Africa. TechLitAfrica takes old computers from the United States which are no longer in use and repurposes those computers with educational software and a downloaded subset of the Internet. Then, TechLitAfrica takes those computers to Africa and sets them up as computer labs. Tyler joins the show to talk through the technical and cultural challenges of building TechLitAfrica. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Hadoop ecosystem provided every company with the tools to store and query large amounts of data at a low cost. Since 2005, that ecosystem has expanded with more and more open source applications for data infrastructure. Data infrastructure includes databases, data lakes, distributed queues, data warehouses, query engines, web applications, on-prem software, closed source, open source, cloud platforms, and software-as-a-service. There is so much software, and it is difficult to keep track of everything in the data engineering ecosystem. Tobias Macey is the host of the Data Engineering Podcast, a show about technologies and practices in the world of data engineering. Tobias joins today’s episode as a guest to give his perspective on the evolving landscape of data engineering, and the trends he is seeing on the data engineering podcast.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

From Africa to India to Asia to South America–computer science and programming are rising in popularity in every emerging market. Each of these markets has regional needs for technology. Just like every culture develops its own food and television, every culture needs different types of applications to run their lives. In Vietnam, the day-to-day life of a citizen is different than it is in the United States. Yes, everyone needs Google and YouTube and Instagram. But the trends in messaging, food delivery, gig economy, and other B2C technology sectors are considerably different than the west. Charles Lee is the founder of CoderSchool a coding school in Vietnam. Before moving to Vietnam to start CoderSchool, he worked as a software engineer in San Francisco for several years. In today’s show, Charles describes the difference between the US and Vietnamese technology sectors, from consumer applications to business software to coding education.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In 2011, Facebook had begun to focus its efforts on mobile development. Mobile phones did not have access to reliable, high bandwidth connections, and the Facebook engineering team needed to find a solution to improve the request latency between mobile clients and the backend Facebook infrastructure. One source of latency was recursive data fetching. If a mobile application client made a request to the backend for newsfeed, the backend API would return the newsfeed, but some components of that feed would require additional requests to the backend. In practice, this might result in a newsfeed loading partially on a phone, but having a delayed loading time for the comments of a newsfeed item. GraphQL is a solution that came out of this problem of recursive data fetching. A GraphQL server provides middleware to aggregate all of the necessary information to serve a complete request. GraphQL connects to backend data sources and federates the frontend request across these different data sources. GraphQL was open sourced in 2015, and has found many use cases in addition to simplifying backend data fetching for mobile clients. Today, GraphQL is used by PayPal, Shopify, Twitter, and hundreds of other companies. Lee Byron is the co-creator of GraphQL and he joins the show to tell the story of GraphQL, and how it fit into Facebook’s shift to mobile.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook has released open source software projects that have changed the industry. The most impactful projects to date are the React frontend user interface tools: ReactJS and React Native. Before React became popular, there were multiple competing solutions for the dominant frontend JavaScript framework. React became the most prominent because of its invention of JSX, its one-way data flow, and the strength of its community. The React community is led by Facebook engineers. Facebook has a full-time team dedicated to improving React. Facebook also has the benefit of seeing the hardest problems in web development ahead of the rest of the industry, since Facebook is always pushing the leading edge of what software can do. As React evolves, Facebook itself is the first user of updates to the libraries. Engineers on the React team will proactively update the applications on other teams with new React code. In this way, Facebook’s React team is similar to a platform engineering team, except the open source community benefits from the improvements as well. Tom Occhino is an engineering director at Facebook. He joins the show to discuss his ten years of experience at the company and his role today as a senior leader in charge of open source developers.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook generates high volumes of data at a rapid pace. Dhruba Borthakur joined Facebook in 2008 to work on data infrastructure. His early projects at Facebook were around Hadoop, the distributed file system and MapReduce computation platform that laid the foundation for the “big data” movement. At the time, Facebook was generating as much data as any other startup, and the company needed to stay at the leading edge of scalability techniques for its Hadoop Distributed File System (HDFS) cluster. Traditionally, Hadoop managed its file system by synchronizing the coordination of the different data nodes with the help of a single master node. At Facebook, the scale of the data was such that the HDFS cluster had thousands of data nodes, which was too much volume for a single master node to handle. Dhruba helped implement redundancy at the master node to create a more resilient system. The early days of the big data movement was focused on batch processing. A company like Facebook would gather large amounts of data into databases and HDFS, and run offline analytics workloads to gather reports on an hourly, daily, or weekly basis. Over time, data infrastructure has moved closer to a “real time” processing model. Data infrastructure does not only support batch offline reporting–it also supports machine learning jobs that need to be run on a more frequent basis. These jobs have lower latency requirements, and have driven the adoption of in-memory stream processing systems like Spark and Flink. Dhruba joins the show to discuss his time at Facebook building data infrastructure. He takes us through the major projects he worked on, including the early Hadoop infrastructure, the refactoring of online user workloads to be more “pull” based than “push” based, and the creation of RocksDB, a storage engine he helped create at Facebook. Today, Dhruba is the CTO and co-founder of Rockset, a company that builds data infrastructure and database APIs on top of RocksDB. Rockset is building infrastructure for modern technology companies–many of which are facing problems that bear significant resemblance to the ones Facebook encountered as it scaled.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook moves fast because of vision, collaboration, and trust. The fast pace of development is enabled by constantly improving infrastructure and a sense of unity throughout the company. In Facebook’s early days, there was an emphasis on rapidly deploying new code to drive constant improvement and experimentation. Product quality was maintained by engineers closely checking each other’s code reviews, rather than writing detailed unit testing suites. Facebook engineers had a sense for how the product should operate, and they were able to evaluate whether a new feature was working properly by testing a live version of that feature. At Facebook, the vision of the company is clearly communicated to the employees. Every employee within Facebook can articulate the vision for the company, and will use similar language in describing that vision. Since the employees are aligned on strategy, they can also align in their implementation of product features. This reduces conflicts across roles and between teams. Facebook has also shown a willingness to trust its engineers. Trust was exemplified by Facebook’s tolerance for failures in the early days. When an engineer broke a build, or shipped a feature that failed to gain traction, that engineer was usually not punished. They may have even been rewarded, if the company could learn significantly from such an error. Raylene Yung was an engineer at Facebook from 2009 to 2015. As she moved from individual contributor to manager to engineering director, Raylene worked on products including News Feed, Timeline, Privacy, and Sharing. Raylene joins the show to give her reflections on the Facebook product and engineering environment. She explained how Facebook’s culture of collaboration, vision, and trust drive fast product development and minimizes conflict. Raylene left Facebook and joined Stripe, where she worked on payments systems and international expansion for almost four years.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook was built using PHP, a programming language that was used widely in the late 90s and early 2000s. PHP allows developers to get web applications built quickly and easily, although PHP has a reputation for being difficult to scale. In the early days of Facebook, the company was scaling rapidly on every dimension. New users were piling into Facebook. Existing users were increasing their interactions and developing new patterns of usage. The Facebook application was rolling out new features quickly, adding them into the Facebook PHP codebase. A common pattern for scaling a large software application is to use a microservices architecture, breaking up the monolithic application into small services which can scale independently. For many applications, this pattern works well. But for some applications, microservices makes less sense. Microsoft Excel is one example. In Excel, a user is making updates to a complex data model using formulas, functions, and other in-app tools that need to be fast, performant, and integrated. The user needs to have a sense that the Excel data model will update quickly in response to changes. A software team working on a spreadsheet product such as Excel might prefer to keep all the application logic in a monolithic application. A monolith can centralize logic and make it easier to reason about. A monolith can reduce the number of network hops, cutting down on distributed systems problems. Testing and deploying a monolithic application can be less complex than doing so in a distributed, microservices system. Facebook chose to scale its PHP monolith rather than breaking it up into distributed microservices. Scaling PHP allowed Facebook to continue moving fast without going through a painful refactoring that would have slowed down the entire company. The first effort to scale PHP involved transpiling the entire PHP application into C++. This C++ version of Facebook ran faster and with a lower memory footprint. But C++ required ahead-of-time compilation: the PHP codebase had to be converted to C++ in one synchronous step. The Hip Hop Virtual Machine (HHVM) was started to allow the Facebook PHP monolith to run on the V8 JavaScript virtual machine. The advantage of running PHP on V8 was that it allowed Facebook to transpile PHP to JavaScript, an interpreted language that fit better with Facebook’s dynamic application build path. HHVM also allows Facebook to take advantage of the rapidly improving JavaScript runtime environment. Keith Adams was an engineer at Facebook for six years, where he helped develop infrastructure to scale PHP effectively. Keith is now the chief architect at Slack, which is also a scaled PHP application. Keith returns to Software Engineering Daily to discuss why and how Facebook scaled PHP.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data engineering allows a company to take advantage of the large quantities of data that the company has generated. In many companies, new data has been produced rapidly for many years, but the company has not been able to take full advantage of it. Creating large data sets does not provide immediate value for a company. A company needs to perform data engineering and data science to take full advantage of it. When data gets generated, it is stored in a database, data lake, or API backend like Google Analytics. In order to manipulate that data, it is often pulled into a data warehouse. A data warehouse provides fast access time to large quantities of data. Pulling data from a source like a database or data lake into a data warehouse requires a process known as extract and load. Once the data is in the data warehouse, it may also undergo a transform, which enriches the data or puts it in a format that is easier to make use of. Once data is in a data warehouse, it can be used to build models, interactive dashboards, and Jupyter Notebooks. The data engineering lifecycle has many different components, which is why data engineering can often be intimidating to a company that is trying to make use of their data. Meltano is a project with the goal of providing a system of conventions for managing the data engineering lifecycle. Meltano was started by GitLab, and the Meltano project has some strategic similarities to GitLab. Danielle Morill is the general manager of Meltano at GitLab. She joins the show to discuss the world of data engineering, and the architecture of Meltano. We touch on the different components of a data engineering pipeline, and the most acute pain points for data engineers.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software engineers have a wide variety of media to choose from, including podcasts, blogs, YouTube videos, conferences. The amount of software engineering media that is available is growing and accelerating. Eight years ago, there were not as many options for information about software. Charles Max Wood founded Devchat.tv to create a network of podcasts and other content for software engineers. Today, his podcasts include the popular shows JavaScript Jabber, Ruby Rogues, and Adventures in Angular. Chuck joins the show for a conversation about software media. This was his second time on the show, with his first episode exploring his podcast JavaScript Jabber.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software applications running within a host operating system need to be isolated. Isolation prevents security vulnerabilities, such as one application accessing the memory of another. In modern cloud environments, a single physical host might be running multiple virtual machines on top of a hypervisor. Those virtual machines might be divided up into containers. The different virtual machines and containers might be operated by different users, or even different companies. gVisor is a container sandbox runtime open sourced by Google. gVisor runs containers in a new user-space kernel, and provides a container security system with low overhead. gVisor improves on the previous security properties of containers. Michael Pratt and Yoshi Tamura work on gVisor at Google, and they join the show to talk through the purpose of gVisor and the engineering around the project.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Creating a software company has never been easier. Software engineers are increasingly seeing entrepreneurship as a viable career path. But the path to being an independent software developer is not always clear. Most engineers spend some of their career working at a software company. Even an engineer who intends to build a company someday can thrive within the right environment. Lior Kanfi is a software engineer and the founder of Tikal, a company he started at the height of the dot com bubble in 1999. His initial vision for the company was to build a product around managing knowledge and people within a company. When the dot com bubble burst in the year 2000, it became much harder to run a product-focused business. Companies were not buying lots of experimental software, and investors were not aggressively funding new software companies. After the market collapsed, Lior shifted Tikal from a product-focused company into consulting, in order to have a more reliable income stream. Today, Tikal is a successful software consultancy based in Israel and the Bay Area. Lior describes the engineers within Tikal as “free radical” software developers: independent people who want to learn about new technologies and build experience interacting with clients. Lior joins the show to talk about his 20 year journey building Tikal and the differences between engineers in the Bay Area and those in Israel. Lior also hosted the Full Stack Tech Radar Day in Tel Aviv, which was a great conference I attended.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Edge computing allows for faster data access and computation. When your client application makes a request, that request might be routed to the edge. Edge servers are more numerous and more widely distributed than normal data centers, but an edge server might not have all of the data or the complete application logic for the backend to serve your request. Edge servers have historically been used for content delivery networks (CDN). CDNs are useful for hosting and serving media files that might otherwise be slow to access over a network. More recently, applications are also using edge servers for computation, as well as storage of resources which are smaller than the movies, music, and images that have traditionally been stored on an edge server. Steve Klabnik is an engineer with Cloudflare, and he returns to the show to discuss storage at the edge. In Steve’s previous appearances we have explored Rust and WebAssembly, and we also touch on those topics in today’s episode.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Today’s episode is a keynote I gave at Full Stack Tech Radar Day in Tel Aviv. The talk is called “You Are Not a Commodity”. This talk is also available as a YouTube video. The slides can be accessed here. The world of commodity engineering is coming to an end. Developers are becoming more productive, more flexible, and more entrepreneurial. How does this impact you as an engineer? And how does it impact large companies who need a fungible set of engineers to maintain their software?

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

ListenNotes allows users to find podcasts by categories, popularity, and search queries. ListenNotes is not a podcast client that you download onto your phone, but it allows users to build playlists. These playlists are automatically published to an RSS feed, so that users can curate a playlist on desktop and subscribe to that playlist from any client. Podcasts are growing in popularity. There are more podcasts than ever before, and a podcast search engine needs to refresh its index regularly enough to capture the updates to the podcast universe. A podcast search engine needs to decide what to index. There are many potential fields to choose from within a podcast: the podcast title, the description, the audio transcription, and other metadata–should all of these factor into the results of a search engine? Wenbin Fang is the founder of ListenNotes, and the engineer who has built most of the application. He joins the show to talk about the world of podcasting and his work on ListenNotes, including the business model. Today, ListenNotes makes its money from advertisements and API requests–there is a growing market for applications that want an API for querying the ListenNotes backend.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Open source software allows developers to take code from the Internet and modify it for their own use. Open source has allowed innovation to occur on a massive scale. Today, open source software powers our consumer client applications and our backend cloud server infrastructure. Linux powers single node operating systems and Kubernetes is the foundation for new distributed systems. Hadoop created an open source distributed file system, Spark gave us a computational runtime on top of it, and Kafka created a middleware platform for shuttling data from one place to another. There are numerous other examples of how open source has changed the world of software development. Open source has also reshaped the business landscape of infrastructure software companies. A common business structure for a modern infrastructure company is the “open core” model. An open core company maintains an open source project that is free to use, but also sells a product or service around that product. Companies with an open core model include Red Hat, HashiCorp, and GitLab. Many companies are building a thriving business with the open core business model. But these companies do not directly control the most important part of the infrastructure supply chain: the cloud provider. Cloud providers have a fundamental tension with open core companies because the cloud providers offer services that compete with the open core companies. In addition to the issue of cloud providers competing directly with the open core companies, some people have questioned whether Amazon Web Services is capturing an unfair portion of the value that is being created by open source. Amazon Web Services is the biggest cloud provider, and it has built a large catalog of services that are built off of open source software. But AWS has not historically contributed heavily to open source relative to the value it has captured. One example of an open core company which has lost market share to an AWS cloud-hosted offering is Elastic, the open core company which maintains the ElasticSearch open source project. Amazon ElasticSearch Service is a closed-source hosted offering built on top of the ElasticSearch. Elastic (the company) has increasingly intermingled proprietary software with their open source repository, making it less clear how that open source repository can be used by companies that want to deploy it for their commercial use. Open core companies such as MongoDB, Redis Labs, and Cockroach Labs have responded to the competitive pressures of AWS by changing their licenses and making it more expensive for cloud providers to offer a cloud-hosted offering of their open source project. The dynamics between cloud providers and open core companies will continue to evolve in the coming years. The norms around open source are up for debate. Joseph Jacks is the founder of OSS Capital, a venture firm focused on investments in commercial open source software companies. He returns to the show to discuss the changing landscape of open core companies, and the benefits of permissionless innovation.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google Earth allows users to explore the imagery of the real world. Imagery for Google Earth is taken from satellites, cars equipped with cameras, and other sources. Google Earth renders a data intensive 3-D model of the world on a client application such as a desktop browser or virtual reality system. WebAssembly is a runtime for executing code other than JavaScript in a browser-based environment. WebAssembly is useful for data-intensive workloads, and developers can use programming languages such as Rust or C++ in the browser by compiling to WebAssembly. Jordon Mears works on Google Earth, and he joins the show to talk about the engineering behind Google Earth and how WebAssembly is being used to improve efficiency. He also discusses the state of tooling around WebAssembly today.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

FoundationDB is a multi-model distributed key-value store. It is fully ACID compliant and horizontally scalable. FoundationDB is not usually used directly by an application developer–FoundationDB is a foundational building block for higher level distributed systems such as the metadata store for data warehousing tool Snowflake. Ryan Worl is a software engineer who specializes in FoundationDB. He joins the show to discuss the architecture of FoundationDB, including the roles of different server components and the read and write path of FoundationDB. We also talk about applications of FoundationDB, and how it compares to storage engines such as RocksDB and databases such as CockroachDB and Spanner.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

FindCollabs is a platform for finding collaborators and building projects. Three months ago we had our first hackathon, with lots of projects being created and collaborated on. In an earlier episode, we showcased the first place winner ARhythm. Today’s show features two more interviews with winners from the first FindCollabs hackathon. Kitspace is an open source hardware registry, and Rivaly is an app for informal ranked leagues, such as ping pong games at work, or board game clubs among your friends. The second FindCollabs hackathon is going on today. Check it out by going to findcollabs.com/open.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A new database company needs to solve numerous problems in order to succeed. There are already lots of existing database companies, so a new company needs to find a way to strongly differentiate itself. Databases are core infrastructure, so a new database company must earn trust with its customers. A database is a complicated distributed system, so a database company must have strong engineering to get its product to market. Citus Data was founded in 2011. Around that time, companies were looking for solutions to both their online transactional workloads and their large-scale offline analytics workloads. Citus Data built its company around Postgres, a popular database that has been around since 1996. Products from Citus Data include a Postgres scalability extension as well as a cloud-hosted offering for spinning up Postgres instances. Citus Data was acquired by Microsoft earlier this year. In today’s episode, Umur Cubukcu joins the show to talk about the story of the company from its early days to its eventual acquisition. Umur describes the landscape of data systems in 2011 when the company started, and explains how that evolved to the current ecosystem. Umur also talks about how to make an acquisition successful and gives some perspective on the future of data platforms.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Programmers are in high demand, and software engineering is a career path that is fun, creative, and lucrative. There are many people who want to transition into a career in software and are looking for the right path toward writing code. The traditional college computer science curriculum teaches some software engineering skills, but the time and financial cost of attending a university is prohibitive for many people who want to learn to code. Over the past decade, there have been several new models for software education. Online video platforms such as Udacity and Coursera put computer science courses online to be watched at the viewer’s convenience. Online schools such as Free Code Camp allow someone to learn how to program without any experience, and with no financial payment. Boot camps with income-sharing agreements such as App Academy create an in-person education environment that mimics a university, but with better cost structure and incentive alignment. Lambda School is an education system that takes elements of other software education models and combines them with newer SaaS technologies such as Slack and Zoom videoconferencing. Lambda School is an online software engineering curriculum with an income sharing agreement. “Income sharing agreement” means that the student does not pay for their education until they get a job. With this model, the student can pay back Lambda School after their software engineering education gets them a high-paying software engineering job. Andrew Madsen works at Lambda School, and he joins the show to describe the path that a student takes through Lambda School, the school’s curriculum for software education, and how Lambda School differs from the other options for coding education.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A grocery store contains fruit, vegetables, meat, bread, and other items that can expire. In order to keep these items in stock, the store must be aware of how much food has been sold and what has gone bad. When a food item is low in stock, the store needs to order more of that food from a central distribution system. Managing food inventory is not simple. Some kinds of meat might expire faster than others. Avocados do not become ripe at the same rate as apples. In order to keep the shelves stocked, there are manual workflows for checking the inventory and ordering new inventory. Afresh is a company that builds software for grocery stores. Afresh works with grocery chains that have a central distribution center. These grocery stores already have some software. At the back of the store, inventory management systems maintain records of the items that the store has on the shelves. At the front of the store, checkout systems detect what has been sold and help to update inventory. When the inventory is running low, the store can order more inventory from the central distribution center, so that trucks can deliver more inventory. Afresh improves the operational intelligence of these stores by detecting spoilage among items that are prone to expiration, such as fruit. Volodomyr Kuleshov is the CTO and co-founder of Afresh and he joins the show to discuss the technical challenges of a grocery store, and the software that Afresh is building to make groceries more intelligent.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Airflow is a system for scheduling and monitoring workflows for data engineering. Airflow can be used to schedule ETL jobs, machine learning work, and script execution. Airflow also gives a developer a high level view into the graph of dependencies for their data pipelines. Chaim Turkel is a backend data architect at Tikal. He joins the show to discuss a case study of using Airflow to rearchitect the data engineering workflow of a complex financial application. We discussed the problems that Airflow solves and the process of porting existing workflows to Airflow.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software projects are organized and planned using project management software. Examples of project management software include JIRA, Trello, and Asana. There are hundreds of tools for managing a software project because there are infinite ways that a project could be managed. Google Docs changed project management by allowing documents to be easier to share and collaborate on. Newer SaaS tools such as Slack, Dropbox, and Notion have taken the design lessons from social networking apps to make enterprise software more engaging. As the tools improve, our project management strategies change, and new software tools emerge to fit those new management strategies. Kurt Schrader is the CEO of Clubhouse, a project management tool for software engineers. Kurt joins the show to talk about the history and future of project management tools. He also discusses the engineering challenges of improving performance on a complicated webapp. Project management tools often have to load hundreds of small objects on a page, which required performance optimizations in the Clubhouse frontend JavaScript library.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Niantic is the company behind Pokemon Go, an augmented reality game where users walk around in the real world and catch Pokemon which appear on their screen. The idea for augmented reality has existed for a long time. But the technology to bring augmented reality to the mass market has appeared only recently. Improved mobile technology makes it possible for a smartphone to display rendered 3-D images over a video stream without running out of battery. Ingress was the first game to come out of Niantic, followed by Pokemon Go, but there are other games on the way. Niantic is also working on the Niantic Real World platform, a “planet-scale” AR platform that will allow independent developers to build multiplayer augmented reality experiences that are as dynamic and entertaining as Pokemon Go. Paul Franceus is an engineer at Niantic, and he joins the show to describe his experience building and launching Pokemon Go, as well as abstracting the technology from Pokemon Go and opening up the Niantic Real World platform to developers. Niantic developer contest finalists

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

WebAssembly allows for web-based execution of languages other than JavaScript. Programs written in Rust or C++ can be compiled down to WebAssembly and shipped over the browser for on-the-fly execution in a safe, memory controlled environment. WebAssembly has been in development for more than two years, and is still an immature ecosystem because building the necessary tooling for WebAssembly is hard. Much of the web has been built around JavaScript and the V8 JavaScript engine, which has been tuned to optimize an interpreted language (JavaScript). WebAssembly modules are often written in C++ or Rust, which are compiled languages. There are engineering challenges at the edge between the interpreted JavaScript runtime and the precompiled WebAssembly modules. Till Schneidereit is a senior research engineering manager at Mozilla. He joins the show to discuss the compilation path of WebAssembly and the state of the ecosystem.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Sheng Liang was the lead developer on the original Java Virtual Machine. Today he works as the CEO of Rancher Labs, a company building a platform on top of Kubernetes. Sheng joins the show to discuss his experiences in the technology industry. The container orchestration wars had many victims. The competing standards for how an enterprise should manage its numerous containers caused several companies to go down a path where they were building infrastructure which eventually had to be replaced. As Sheng discusses in today’s episode, the container orchestration wars almost killed his company. Rancher was originally built on top of a different container orchestrator, and the migration to Kubernetes required a massive rewrite of the Rancher platform. The container orchestration wars were not the first technology battle that Sheng has seen in his career–and it won’t be his last. In today’s show, we discuss the nature of technology wars. Are they necessary? How can a software company minimize the damage caused by a war between competing standards? Sheng was an excellent guest and we didn’t cover nearly as many subjects as I wanted to, so we will have to do another show in the future!

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes has made distributed systems easier to deploy and manage. As Kubernetes has become reliable, engineers have started to look for higher level abstractions we can define on top of Kubernetes. An operator is a method of packaging, deploying, and managing a Kubernetes application. Operators are useful for spinning up distributed systems such as Kafka, Redis, or MongoDB. These data systems are complicated, stateful applications with lots of failure domains. The operator framework enables a developer to deploy one of these complicated applications with less fear of the system crashing, or entering an erroneous state. Rob Szumski is an engineer at Red Hat. He joins the show to discuss Kubernetes, the operator pattern, and his time at CoreOS, which was acquired by Red Hat.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud computing was popularized in 2006 with the launch of Amazon Web Services. AWS allowed developers to use remote server infrastructure with a simple set of APIs. But even with AWS, it was still not simple to deploy and manage a web application. In 2007, Heroku launched a platform built on top of AWS. Heroku focused on the developer experience by optimizing for users who were deploying Ruby on Rails applications. Since then, Heroku has expanded into other forms of managed infrastructure, including other application frameworks like NodeJS, and databases like Postgres. Heroku was the first popular “layer 2” cloud provider. Twelve years later, it is probably still the most popular. But there have been many other cloud providers built on top of AWS, including Netlify, Zeit, Spotinst, and Firebase. Layer 1 cloud providers are Google Cloud, AWS, Azure, Digital Ocean, and other raw infrastructure providers. These companies provide a great service in their low cost, commodity infrastructure. But the layer 1 providers are not optimizing for developer experience. They need to cater to a broad set of developers, some of whom want to work at a low level. A layer 2 cloud provider can build an opinionated solution that serves a subset of the overall cloud market particularly well. Render is a layer 2 cloud provider that optimizes for specific developer workflows, such as deploying a NodeJS web server, a static site, or a Docker container. Anurag Goel is the founder of Render, and he joins the show to discuss the strategy and the economics of Render. Anurag was also one of the early employees at Stripe, and he discusses his experience and learnings from working at the company.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software engineering is an art and a science. To manage engineers is to manage artists and scientists. Software companies build practical tools like payment systems, messaging products, and search engines. Software tools are the underpinnings of our modern lives. You might expect this core infrastructure which modern humans rely on to have been constructed with pure formulaic rigor. But the best software tools are not built within a totally defined process. Software is built through messy iteration. When a piece of software looks pristine, that is often a function of how many mistakes have been made, and then subsequently corrected for. There is no fixed process for how to build good software. As our tools get better, we have to update our software engineering practices to utilize those new tools. We have to rethink the style that we are working in. We have to discard old tools and procedures in order to pick up the new ones, and have higher leverage. As an organization scales, the structure of the organization needs to be modified. Team members need to be reallocated. Checks and balances need to be put in place. Rules and cultural practices need to codified, because a larger organization cannot have ties broken by an individual. Software is built by humans, and every management decision must be considered in the light of human psychology. When we change a line of code, the code does not get emotional about being altered. But the same cannot be said of humans. Even a minor conversation between an engineering manager and a direct report can have lasting implications. Will Larson is the author of An Elegant Puzzle: Systems of Engineering Management. He works on Foundation Engineering at Stripe, and has worked in engineering management at Uber, Digg, and other software companies. Elegant Puzzle provides strategies, tactics, and ruminations on software development. Will joins the show to explore the multifaceted subject of engineering management.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning allows software to improve as that software consumes more data. Machine learning is a tool that every software engineer wants to be able to use. Because machine learning is so broadly applicable, software companies want to make the tools more accessible to the developers across the organization. There are many steps that an engineer must go through to use machine learning, and each additional step inhibits the chances that the engineer will actually get their model into production. An engineer who wants to build machine learning into their application needs access to data sets. They need to join those data sets, and load them into a machine (or multiple machines) where their model can be trained. Once the model is trained, the model needs to test on additional data to ensure quality. If the initial model quality is insufficient, the engineer might need to tweak the training parameters. Once a model is accurate enough, the engineer needs to deploy that model. After deployment, the model might need to be updated with new data later on. If the model is processing sensitive or financially relevant data, a provenance process might be necessary to allow for an audit trail of decisions that have been made by the model. Rob Story and Kelley Rivoire are engineers working on machine learning infrastructure at Stripe. After recognizing the difficulties that engineers faced in creating and deploying machine learning models, Stripe engineers built out Railyard, an API for machine learning workloads within the company. Rob and Kelley join the show to discuss data engineering and machine learning at Stripe, and their work on Railyard.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The software market changes every year. As individuals and enterprises become more willing to buy software, there are new markets for entrepreneurs to sell software to. Good software has high margins and high retention, so even a niche software business can prove profitable. As software spreads across the world, developing countries are showing a willingness to buy the same software tools as the developed countries, making the niche software businesses even more successful than expected. It’s a great time to start a software business. But the fundamental challenges faced by an entrepreneur have not gone away. An entrepreneur must find the necessary capital, build an initial product, iterate on that product, market to their audience, sell to their customers, and retain their users. Hiten Shah is an entrepreneur and investor who has started several businesses, both successful and unsuccessful. He is also the host of the Startup Chat podcast, a show about strategies and tactics for software businesses. Hiten joins the show to discuss his experience in the software industry, including lessons on financing, product development, and the future of software products.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google Cloud was started with a vision of providing Google infrastructure to the masses. In 2008, it was not obvious that Google should become a cloud provider. Amazon Web Services was finding success among startups who needed on-demand infrastructure, but the traditional enterprise market was not yet ready to buy cloud resources. Googlers liked the idea of becoming a cloud provider. But was it the right time to enter the market? Google’s advertising business was a large and growing cash cow. Executives within Google were not sure how much capital and effort should be allocated into an infrastructure business. When Google decided to go into the cloud business, Joe Beda was one of the engineers who helped lead the effort, and joins the show as today’s guest. Google’s internal server infrastructure is managed by Borg, a system for allocating resources to applications. Google Cloud runs on Borg, and there were a number of early engineering challenges to building the necessary functionality into Borg for running a cloud provider on top of it. One example of a technical challenge that Google faced was the refactoring of Borg to run Google Cloud workloads. The requirements for public infrastructure are different than those of internal Googlers. Inside of Google, developers deploy their applications to containers running on bare metal. Outside of Google, developers want to create virtual machines. Borg needed to be refactored in order to instantiate VMs. Google solved this technical problem, as well as many other challenges, and Google Cloud slowly gained momentum in the market. But AWS remained the default choice for profitable enterprise workloads. It wasn’t until the container orchestration wars that Google found an opportunity to jump on a market segment that offered strong differentiation. By open sourcing Kubernetes and presenting a clear vision for where the project was going, Google shifted the battlefield of the public cloud toward a competitive landscape where it has many advantages. Kubernetes also provided many other technology companies with an opportunity to get into the cloud market, creating a collaborative, multi-company ecosystem that has accelerated the pace of software faster than anyone expected. Joe Beda has been instrumental in the evolution of the cloud native ecosystem. In today’s episode, Joe gives his memories on Google Cloud, Kubernetes, and his Kubernetes company Heptio, which he sold to VMware.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Containers are made to fail gracefully. When your container shuts down due to a hardware or software failure, your distributed application should be able to tolerate that failure. One simple way to be able to tolerate such a failure is to make all of your application logic “stateless.” If your application does not maintain state, then shutting it down in the middle of a computation is not a problem–you can just restart the application, restart your computation, and get the same result. But applications need to maintain state. We need to use databases and in-memory systems to manage long-lived user sessions and other interactions. A database is not just an on-disk abstraction–a database requires an application server to be accepting network traffic. We can run those database applications within containers. There is a fundamental tension between stateful applications and the idea that containers are meant to tolerate failure gracefully. Saad Ali is an engineer at Google, and he returns to the show to discuss Kubernetes storage and state management. He gave a keynote at KubeCon EU, which I spoke to him about. New Software Daily app for iOS. You can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Amazon Web Services is the leading cloud provider by a large margin. Amazon established its lead by being first to market in 2006, with Google and Microsoft taking several years to catch up to the huge business opportunity of the cloud. Since 2008, Google Cloud has been working on cloud products for developers. It started with App Engine, which is widely used internally at Google, but has not had overwhelming public adoption. Over the last eleven years, Google has refined its understanding of how customers want to buy public cloud resources. Google Cloud products like Cloud Storage, Persistent Disk, and BigTable have given Google parity with many of the AWS public cloud offerings. Although Google has caught up to AWS in terms of products, the enterprise market has continued to choose AWS as its default. AWS is widely perceived as having more experience in running enterprise workloads, and a better responsiveness to customers. In order to keep Amazon from running away with the cloud market entirely, Google needed to shift the competitive landscape to different territory. Kubernetes provided the paradigm shift that Google needed. The market for cloud providers has changed completely due to Kubernetes. When Google open sourced Kubernetes, it created a common codebase for software companies to build software for managing distributed systems. In the span of five years, Kubernetes has turned the world of cloud products into a world resembling the open source Linux ecosystem. This is a remarkable shift, and every infrastructure software vendor is still figuring out its strategy for adapting. Adam Glick is the head of modern infrastructure and serverless marketing at Google. With Craig Box, he hosts the Kubernetes Podcast from Google, an excellent show about recent developments and evergreen concepts within the world of Kubernetes. Prior to Google, Adam worked at AWS for 3 years and Microsoft for twelve years. He has seen each of the major cloud providers up close and has a deep awareness for how each company thinks. We had a great conversation about the cloud native landscape, podcasting, and developer marketing. New Software Daily app for iOS. You can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Containers offer a lightweight abstraction for running a server. Cloud providers are able to manage billions of containers from different users, allowing for economies of scale so that each user can pay less. Today, there is a variety of ways that users can deploy containers on a cloud provider. These containers can run in managed Kubernetes clusters, in functions-as-a-service, or in long-lived standalone container instances. User preferences are getting more sophisticated, with some users showing an interest in Knative, an open source serverless system originally created at Google. Whichever container deployment system you choose, your application and its multiple servers need a way to route traffic, measure telemetry, and configure security policy. A service mesh abstraction can help serve these use cases. Lachlan Evenson has worked in containers and Kubernetes since before the container orchestration wars. He was an engineer at Deis, a company which built an open source platform-as-a-service running on top of containers and Kubernetes. Deis was acquired by Microsoft, where Lachlan now works as principal program manager of Container Compute. Lachlan joins the show to discuss containers, Kubernetes, and the Service Mesh Interface, an interoperable service mesh layer that Microsoft launched with Buoyant.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Each cloud provider offers a different set of services which are not always compatible with each other. What are the challenges of building an application that interoperates with multiple different clouds? The first issue is API compatibility. Most cloud providers have a managed SQL offering, a bucket storage system, and server abstractions like virtual machines and containers. But these tools might have different APIs on each cloud. The code that you wrote to save your application data to Amazon S3 might have to be rewritten if you decide to switch your bucket storage to a different provider. Another issue that interferes with cloud interoperability is the degree of integration on a particular cloud. If I build my application for AWS, I might be heavily integrated with Amazon’s Identity and Access Management policy system and AWS logging. Each cloud provider makes it particularly easy to connect to their integrated solutions. There is also the problem of services on one cloud that simply do not map to a service on any other cloud. Google Cloud Bigtable does not have an equivalent service on Amazon. Microsoft CosmosDB does not have an equivalent service on Digital Ocean. As developers, this irritates us. We want to be able to deploy our application to any cloud. We want to be able to move applications easily from one cloud to another. And we want to mix tools from different clouds together as easily as we import a library. Bassam Tabbara is the CEO of Upbound, a company focused on making multicloud applications easier to deploy and operate. I spoke to Bassam at KubeCon EU 2019, and he described the problems of multicloud deployments and the opportunities for the cloud native ecosystem to become more cross-compatible.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes has evolved from a nascent project within Google to a thriving ecosystem of cloud providers, open source projects, and engineers. Tim Hockin is a principal software engineer who has been with Google for 15 years. Tim joins the show to talk about the early days of the Kubernetes projects, and the engineering efforts that are under way five years into the project. At KubeCon EU 2019, two of the prevalent subjects of discussion were service mesh and serverless–particularly the Knative project. Tim gave his perspective for how projects that are adjacent to Kubernetes are developed within the community.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google’s cloud business was long regarded as a place where startups could build a business, but not established enterprises. For serious workloads, enterprises chose Amazon almost unanimously. This phenomenon of Amazon as the default was described by a phrase that harkened back to the days of IBM’s dominance: “nobody ever got fired for choosing AWS.” With the rise of Kubernetes, Google has established a reputation as a reliable provider of container orchestration. As enterprises look to roll out Kubernetes workloads, Google is convincing many of them to work with Google Kubernetes Engine. Kubernetes is one part of the changes which fall under the description of “digital transformation”. As these enterprises build out their digital transformation strategy, they are also looking for a strategy around multicloud, on-prem, policy management, and consulting partnerships. Anthos is a platform where enterprises manage the resources and configuration of their cloud deployments, as well as a system to partner with services integrators and independent software vendors. Aparna Sinha works on Anthos, Google Kubernetes Engine, and other projects at Google. In today’s show we discuss the process of digital transformation, as well as the tactics for how a digital transformation is actually implemented. We also talked about GKE on-prem, and the kinds of tooling needed by on-prem application deployments.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The CNCF was formed under the looming shadow of AWS. The CNCF was seeded with the donation of Kubernetes by Google. Much like the Linux community was positioned as a rebellious movement in reaction to Microsoft’s dominance, the Kubernetes community represents a fervent desire to open up the market to cloud providers beyond the tight-lipped, proprietary dominion of Amazon. With such a deep spirit of insubordination, it is no surprise that the community has rejected Istio like a set of loosely coupled organs rejecting a foreign skin attempting to layer itself across them. Even though the CNCF was founded by Google, the community was formed in spite of big centralized clouds, not as a marketing vessel for their products which may or may not be open source. Microsoft seems to understand this fact better than Google, at least in the domain of service mesh. The day after this interview with William, Microsoft announced the Service Mesh Interface (SMI), a project it partnered with Buoyant and other companies on to create a minimal spec for what a service mesh should offer to a Kubernetes deployment. The SMI presents a safe buy-in point for enterprises who want a service mesh, but do not want to get caught in the evangelistic crossfire of Istio and Linkerd. It is in this environment that we begin our next series of shows on the current cloud native ecosystem. Thanks to the Cloud Native Computing Foundation for putting together an amazing podcasting zone at KubeCon, and allowing me to conduct these interviews. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud computing, open source, and mobile computing are trends that affect every organization. When a large organization adapts to these trends, it is commonly referred to as a “digital transformation”. Digital transformation causes many companies to reframe their business as a software company. A candy manufacturer now must think of itself as a software company that makes candy. An insurance company must now think of itself as a software company that issues insurance. Capital One is a bank that was started in 1988. Capital One has always had an emphasis on software, but the company’s digital transformation has affected it as much any other company. The company is migrating to the cloud, building out microservices, rolling out continuous delivery pipelines, and shifting the internal culture to be more adept at using software. Hillary McTigue is a senior director of data engineering at Capital One. She joins the show to discuss her experience implementing a digital transformation within a large company. Subjects we discuss include culture, management strategy, and the sequencing of different phases of a digital transformation. Full disclosure: Capital One is a sponsor of Software Engineering Daily. We are hiring two interns for software engineering and business development! If you are interested in either position, send an email with your resume to jeff@softwareengineeringdaily.com with “Internship” in the subject line.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Netflix started with a DVD-by-mail product. The software infrastructure and operations practices needed for the DVD business were very different from those needed by a streaming video company. Since the early days of Netflix, CEO Reed Hastings knew that the company would evolve to becoming a streaming video platform. But he did not know when the technology would be advanced enough to support video streaming, and he did not know how users would consume it. Greg Burrell has worked at Netflix for 14 years. Greg was one of the first engineers to start working on video streaming, which Netflix first attempted to implement with a set top box that downloaded movies and played them on your television. After evolving this strategy, Netflix arrived at the current model of video streaming through apps on browsers and mobile devices. As the company pivoted from DVD-by-mail to video streaming, Netflix encountered multiple challenges across engineering, operations, and communications across the company. At the time, there was no “DevOps” movement. There were not established continuous delivery practices. The available cloud technologies were immature and low level. Greg joins the show to describe the evolutionary arc of Netflix’s engineering process. Greg also presents a model for software development that he describes as “Full Cycle Development”. At Netflix, engineering teams of full cycle developers work without dedicated operations or testing teams. It is a sophisticated approach to engineering management. I spoke to Greg at the Fullstack Tech Radar Day, a software conference in Tel-Aviv put on by Tikal, an engineering community based out of Israel and San Francisco. This was a great conference, and we’ll be airing some additional content from it in the coming weeks. We are hiring two interns for software engineering and business development! If you are interested in either position, send an email with your resume to jeff@softwareengineeringdaily.com with “Internship” in the subject line.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Augmented reality applications can be used on smartphones and dedicated AR headsets. On smartphones, ARCore (Google) and ARKit (Apple) allow developers to build for the camera on a user’s smartphone. AR headsets such as Microsoft HoloLens and Magic Leap allow for a futuristic augmented reality headset experience. The most prominent use of augmented reality today is gaming, with a notable example being Niantic’s Pokemon Go. Tony Godar is a software engineer who works on augmented and virtual reality applications. He joins the show to talk about his day job working on virtual reality experiences, and an AR game he built called ARhythm. Tony was the winner of the FindCollabs Hackathon and we also discussed his experience working on the project through FindCollabs. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

CamelCamelCamel is a tool for tracking prices on items on Amazon.com. The company was launched eleven years ago and is built off of the Amazon Product Advertising API. Daniel Green is one of the founders of CamelCamelCamel, and he joins the show to describe his experience building the product. Amazon and CamelCamelCamel (C3) have a complex relationship. C3 makes its money from referral listings. When a user tracks the price of an Amazon item using C3, that user will probably eventually click on the referral listing on C3. If the user purchases the item on Amazon, C3 gets a percentage of the purchase from Amazon. C3 has millions of product listings where they are tracking the price of items on Amazon. They have created a directory with a large subset of Amazon’s items by leveraging an API that was originally meant for advertising. Whether or not this is a proper use of the API, C3 arguably leads to more purchasing volume on Amazon–which is ostensibly why Amazon lets the company continue to operate. In today’s show, Daniel gives a history of CamelCamelCamel, including his own background as an engineer working in finance and how he wound up working on C3 for the past decade. We also talk through how C3 is architected. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Gaming is becoming mainstream. Popular multiplayer games such as Fortnite and Minecraft present players with a massive virtual world to explore, build, and compete within. Turn-based games such as Hearthstone and Magic are breeding a new generation of board game and card game aficionados. Social media networks like Twitch and YouTube have turned gaming into a voyeuristic sport that is outcompeting many physical sports games for attention. Guilded is a platform for managing gaming teams. On Guilded, there are teams for games like League of Legends, Fortnite, and World of Warcraft. These teams use Guilded to manage calendars, Discord bots, forum software, documents, statistics, and recruiting. This might sound confusing–why does a gaming team need document management, calendars, and analytics? Are we talking about a video game team or a software company? To understand Guilded, you need to understand the rapidly changing modern gaming ecosystem. Eli Brown is a founder of Guilded.gg. He joins the show to talk about the world of gaming, its intersection with social media, and the fascinating engineering problems involved in building a platform for gaming teams. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google’s codebase is managed in a single monolithic repository. An engineer at Google can explore almost any area of the codebase within the entire company. In order to enable this, Google has built tooling to support the monolithic repo, including a virtual file system and a set of build tools. A monolithic repository is not to be confused with a monolithic deployment. Google’s infrastructure consists of thousands of small services interacting over a network, and scaling individually. But all of the code for each of these different independent modules is in the same version control system. Ciera Jaspan is a staff software engineer at Google working on developer infrastructure. She worked on an internal research project within Google to find out how engineers felt about the monolithic repository system and how it compared to a large number of small repositories. Ciera joins the show to discuss repository management, internal tooling, and Google’s approach to researching developer productivity within the company. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Alex Balazs is the Intuit Chief Architect and has been working at the company for almost twenty years. Intuit’s products include QuickBooks, TurboTax, and Mint. These applications are used to file taxes, manage business invoices, conduct personal accounting, and other critical aspects of a user’s financial life. Because the applications are managing money for users, there is not much room for error. When Intuit was started, the company made desktop software. In his time at Intuit, Alex played a key role in rearchitecting the monolithic desktop applications to be resilient, reliable web applications. Intuit originally managed this software on their own servers. Since then, Intuit has migrated to the cloud using AWS. Alex joins the show to discuss his experience scaling Intuit, his strategy for cloud migration, and his evaluation criteria for questions of build versus buy. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Africa is rapidly adopting the same software and hardware technologies that have transformed the western world over the last few decades. But access to computers and technology education is still uneven. Where there is access to computers, smartphone adoption often comes before access to laptops or desktop computers. Nelly Cheboi is the founder of TechLit Africa, an organization that works to connect schools and families in Africa with computers and software. Nelly studied computer science, and worked as a software engineer before leaving her career to focus full-time on building a scalable model to take refurbished computers and give them to Africans who can make good use of them. TechLit Africa is also building a software stack to equip schools in Africa without an Internet connection with an internal subnet including Wikipedia and other educational resources, so that people in the school can get an Internet-like experience despite a lack of access to the full Internet. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook’s strategy is shaped by long term goals, short term requirements, and the available resources of the company. Long term goals are necessary for thinking through big decisions such as acquisitions, hardware product investments, and open source software ecosystems. To implement long term goals, Facebook needs to communicate the vision of the company and foster an internal culture that supports that vision. Short term requirements can affect how the company is thinking on a more immediate time horizon. When Facebook realized the importance of mobile computing, the mentality in the company quickly shifted from looking at mobile as a tax on engineering resources to a long-term source of business value. When Google started to work on Google+, Facebook engineers focused their resources on the potential competitive threat. Facebook’s strategy is implemented by the engineers, product managers, and other employees of the company. Facebook is unique in its ability to allow those employees to self-assemble into work that is meaningful to the individuals as well as to the company. As the long term goals and short term requirements of Facebook change over time, company resources are shifted to focus the company on the correct set of priorities. Some of those priorities might be speculative investments in new technologies. Other priorities might include doubling down on areas of the company that are showing promise. Mike Vernal worked as a VP of product and engineering at Facebook for 8 years. He left the company in 2016 and joined Sequoia Capital, where he now works as a partner. In his time at Facebook, he helped architect and implement strategies relating to product direction and engineering. Mike joins the show for a discussion about his time at Facebook and the strategic lessons that he learned from his time at the company. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

React is a set of open source tools for building user interfaces. React was open sourced by Facebook, and includes libraries for creating interfaces on the web (ReactJS) and on mobile devices (React Native). React was released during a time when there was not a dominant frontend JavaScript library. Backbone, Angular, and other JavaScript frameworks were all popular, but there was not any consolidation across the frontend web development community. Before React came out, frontend developers were fractured into different communities for the different JavaScript frameworks. After Facebook open sourced React, web developers began to gravitate towards the framework for its one-way data flow and its unconventional style of putting JavaScript and HTML together in a format called JSX. As React has grown in popularity, the React ecosystem has developed network effects. In many cases, the easiest way to build a web application frontend is to compose together open source React components. After seeing the initial traction, Facebook invested heavily into React, creating entire teams within the company whose goal was to improve React. Dan Abramov works on the React team at Facebook and joins the show to talk about how the React project is managed and his vision for the project. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook leadership was able to recognize the importance of mobile computing in time to develop high quality mobile applications, but there were numerous challenges. The Facebook desktop web app had been difficult enough to build due to the unprecedented data requirements and amount of interactivity. Mobile introduced the additional hurdles of limited bandwidth and distinct native operating systems in Android and iPhone. Facebook’s early efforts to build a mobile application involved a cross-platform HTML5 solution. HTML5 had insufficient performance for Facebook’s needs, and the company needed to develop native apps in order to deliver the desired experience. Facebook’s ability to pivot to mobile is comparable to the classic story of Intel pivoting from a memory company to a microprocessor company. To succeed at mobile application development, Facebook had to shift its focus dramatically, reallocating engineering resources and acqui-hiring small mobile companies in order to build up the domain expertise for mobile. As a side effect of this transition to mobile, Facebook developed an understanding of how dramatically software engineering was changed by the introduction of smartphones and the high bandwidth requirements of social networking. The challenges of this new paradigm led to the development of open source tools such as GraphQL and React Native, which have allowed countless projects to build applications more easily. Jocelyn Goldfein was an engineering director at Facebook for four years, from 2010 to 2014. She currently works as an investor at Zetta Venture Partners. In her time at Facebook, Jocelyn saw the shift to mobile firsthand. In today’s episode, she describes how Facebook management works, and gives her perspective on the distinguishing characteristics of the engineering organization as a whole. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook is a case study in the ability for developers to self-organize into groups who are working on projects that are meaningful to the company and personally satisfying to the individual engineers. Many engineers in the software industry work under a less capable manager who has complete control over their creativity. This leads to employee churn, dissatisfaction, and burnout. Facebook’s ability to move fast is predicated on its ability to match engineers with problems that are interesting to those particular individuals. Whether you want to work on newsfeed or developer productivity tools or machine learning research, there is a path within Facebook to finding a problem that is both important and fun. Facebook’s unique set of engineering challenges required the company to develop a unique set of internal tools. Because Facebook had data and throughput requirements which were unprecedented, the available tools and best practices at the time did not satisfy Facebook’s requirements. Over the years, Facebook has developed its own databases, caching strategies, and JavaScript frameworks. Nick Schrock worked at Facebook for eight years. He is best known as a co-creator of GraphQL, a tool for efficiently fetching data through a federated request language. GraphQL was the result of years of evolution of internal tooling within Facebook. Nick has discussed the creation of GraphQL in other podcasts, and we will have a more dedicated episode around a retrospective of GraphQL in the near future. Today’s episode is about the process by which developers at Facebook self-organized, and Nick’s ideas around how to identify a need for an internal tool. Since leaving Facebook, Nick has parlayed his experience in developer tools into Dagster, a programming model for data applications. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook engineering is commonly described by two words: move fast. Building products quickly has been a differentiating characteristic of the company since its inception. From the longtime engineers to the summer interns, Facebook instills a sense of immediacy and opportunity in all of its employees. The goal of Facebook is to make the world more open and transparent, with the intention of creating greater understanding and connection through Internet services. More than any other company in history, Facebook has enabled people to communicate with each other via simple user interfaces and real, authenticated human identity. Facebook must move fast, because the vision for Facebook is without precedent. It may feel like the Facebook mission is already finished, because you can already use Facebook to connect with anyone across the world with an Internet connection. But once you are connected to somebody on Facebook, there are only a small number of interactions you can take: sending a message, sharing a photo, broadcasting a video stream. There are so many more parts of our lives waiting to be digitized, and many of these require a real identity system to work properly. More than any other company, Facebook is positioned to expand our system of real-world human trust onto the Internet. The depth and breadth of the engineering problems required to accomplish this demands that Facebook move fast. To move slower would cause all of us to pay the opportunity cost of having to wait longer to interconnect our global society. Pete Hunt worked as an engineer at Facebook for three and a half years. At Facebook, he helped build React, a set of technologies that have significantly improved frontend application interface development. After the Instagram acquisition, Pete was the first engineer from Facebook to join the Instagram team to help bring the two companies together. Pete left Facebook in 2014 to start Smyte, a company that made trust and safety tools for marketplaces and social networks. Smyte was acquired by Twitter, where Pete now works on engineering problems relating to trust, safety, health, and infrastructure. Pete joins the show for the first of several episodes with Facebook engineers. In these episodes, we will explore the engineering practices of Facebook–from scaling Facebook’s PHP monolith to open sourcing React and GraphQL. Other topics will include management, onboarding, and product strategy. Our goal is to present a holistic picture of how Facebook engineering works, so that other organizations can learn to adopt practices that will allow them to move faster. We hope you enjoy this series on Facebook engineering. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software engineering is harder than it should be. There are many people who have an app idea that they are not sure how to build. Some of these people are highly technical professionals like real estate agents, scientists, and accountants. These professionals learn to use spreadsheets in their day-to-day work. Spreadsheets are also used widely by young people such as students. Spreadsheet users vary in terms of how familiar they are with the programmability of a spreadsheet, but there are certainly more people who have built complex spreadsheets than there are people who have built complex web apps. Airtable is a tool for making application development easier and more accessible. The Airtable interface is similar to a spreadsheet and can be used for most spreadsheet applications. It can also serve as a rich backend database system to improve the productivity of software developers who are fully capable of building web applications. There are high-level programmable components called Blocks and integrations with developer APIs like Twilio and Stripe. Airtable has a permissions and collaboration system that allows interaction between engineers who might be using Airtable as a programmatic transactional database and operations members who might need to read or edit specific parts of the data on an ad hoc basis. Howie Liu is the CEO of Airtable and he joins the show to talk about his vision for the product and the engineering problems he is working on to realize that vision. Airtable has not been trivial to build, and has required its own custom database backend and its own JavaScript rendering system. Special thanks to Gareth Pronovost who is a full-time Airtable expert that I found on YouTube and who was generous enough to take some time to have a call with me and describe his experience using Airtable. The fact that there is a full profession around creating Airtable applications speaks to how unique this platform is. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Relational data systems have evolved from single node instances to complex distributed systems. Almost any database can be accessed through a SQL statement, but the guarantees of these databases can vary in terms of consistency, availability, latency, durability, and financial cost. Relational database systems that explore these different sets of tradeoffs are sometimes categorized as “NewSQL”. There are also a wide variety of data systems that are not categorized as databases. Kafka is a distributed queue. HDFS is a distributed file system. Spark provides a distributed in-memory working set to process data. Cloud providers offer hosted bucket storage for your data lake and fast processing in the form of a data warehouse. Sunil Kamath is a principal PM with Microsoft. Sunil has worked on database systems for two decades, and he joins the show to give his perspective on the current data world and his predictions for how data platforms will become easier to use. Sunil is optimistic about the use of virtual data for unifying the access of data for a variety of operational use cases. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

WebAssembly is a binary instruction format for applications to run in a memory-constrained, stack-based virtual machine. The WebAssembly ecosystem consists of tools and projects that allow programs in a variety of languages to compile into WebAssembly and run in a safe, fast, sandboxed runtime environment. WebAssembly is a transformative technology for the Internet. Most users will experience it as a set of gradual, incremental improvements to their online experiences. Pages will load faster and become more dynamic. Applications will become more secure. Infrastructure will become cheaper, and those cost savings will eventually reach the consumer. For developers, WebAssembly opens a world of possibility. In today’s operating systems, the user can feel a big difference between applications that need a large client-side runtime (such as video editing tools, or render-heavy games such as Half Life) and applications that are more lightweight and can run entirely on the web (such as Twitter). Tyler McMullen is the CTO at Fastly. He joins the show to talk about the compilation path, the runtime, and the opportunities of WebAssembly. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern server infrastructure usually runs in a virtualized environment. Virtual servers can exist inside of a container or inside of a virtual machine. Containers can also run on virtual machines. Kubernetes has allowed developers to manage their multiple containers, whether those containers are running in VMs or on bare metal (servers without VMs). As organizations expand their Kubernetes deployments, the overhead of those deployments is becoming a relevant concern. So-called “Kubesprawl” can occur within organizations due to a lack of best practices on when new clusters should be spun up or spun down, and when clusters should be shared by teams or shared by services. Paul Czarkowski is a principal technologist with Pivotal. He joins the show to discuss virtualization, Kubernetes, and the state of the cloud native ecosystem. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Relational databases such as Postgres are often used for critical workloads, such as user account data. To run a relational database service in the cloud requires a cloud provider to set up a highly durable, highly available system. Jon Daniel is an infrastructure engineer at Heroku. Jon joins the show to describe the engineering and operations required to build a managed relational database service. Full disclosure: Heroku is a sponsor of Software Engineering Daily. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Satellite images contain vast quantities of data. By analyzing the contents of satellite images over time, we can identify trends in weather, soil, and agriculture. If we combine that data with ground-level sensors, we can gather a clearer understanding of how chemicals in the air or in the dirt map to how things look from above via satellite. Descartes Labs is a company that gathers high dimensional data about our planet and turns it into machine learning models to be used by customers. In order to do this, the company has built out a data pipeline involving queueing systems, machine learning frameworks, and internal tools that are used to aggregate, clean, model, and measure data. Tim Kelton is a co-founder of Descartes Labs and he joins the show to discuss the high volume of data and the distributed systems that make up the Descartes Labs infrastructure. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The modern software supply chain contains many different points of distribution: JavaScript frameworks, npm modules, Docker containers, open source repositories, cloud providers, on-prem firmware, IoT, networking proxies, and so much more. With so much attack surface, securing a large enterprise is an uphill battle. Jeff Williams is the CTO at Contrast Security, a company that makes infrastructure monitoring tools. Contrast Security works by intercepting network traffic at a low level and assessing whether that traffic maps to a common threat model. Jeff joins the show to talk about different approaches to monitoring and securing large infrastructure deployments. Contrast Community Edition RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Growing a software business requires an understanding of engineering, sales, and marketing. As we learn software engineering, we also pick up some knowledge about how a business should operate. We know that there are customers, and that our product needs to be scalable to serve more customers. We know that some features are more important than others, and so we focus on building the features that matter the most. But unless we make a deliberate focus, engineers do not learn how to sell and market a software product. Learning how to sell and market software is an important skill to develop. It allows a software engineer to be self-sufficient. If you already know how to write software, sales and marketing are actually the only other pieces you need to be an “entrepreneur”. And the basics of sales and marketing are often easier and more fun to learn than the first painful days of learning basic programming. Greg Kogan is an engineer who has shifted his focus to working as a consultant for companies that are trying to go to market with a technical product. Greg has helped grow companies such as Netlify, Scalyr, and Domino Data Lab. Much of his work is around products targeted toward developers. Greg joins the show to describe his methodical approach to selling and marketing software. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A Kubernetes instance occupies a wide footprint of multiple servers, creating an appealing target to an attacker, due to its access to a large pool of compute resources. A common attack against an exposed Kubernetes cluster is to take it over for the purposes of mining cryptocurrency. Thus it is important to keep a cluster secure. The importance of security is magnified for a cloud provider. A cloud provider runs a managed Kubernetes service, which might be running thousands of Kubernetes clusters. If the cloud provider’s chosen distribution of Kubernetes contains a vulnerability, or if the Kubernetes instances are misconfigured, all of these clusters could be exposed to the same vulnerability. Maya Kaczorowski works on the security of Google’s managed Kubernetes service GKE. In today’s show we discuss the attack surface of a managed Kubernetes service. Maya was previously on the show to talk about container security. This episode is a good companion to that one, as well as a previous show with Liz Rice about container security. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Lyft generates petabytes of data. Driver and rider behavior, pricing information, the movement of cars through space; all of this data is received by Lyft’s backend services, buffered into Kafka queues, and processed by various stream processing systems. Lyft moves the high volumes of data into a data lake for different users throughout the company to use offline. Machine learning jobs, batch jobs, streaming jobs and materialized databases can be created on top of that data lake. Druid and Superset are used for operational analytics and dashboarding. Li Gao is a data engineer at Lyft. He joins the show to explore the different aspects of Lyft’s data platform. We also talk about the tradeoffs of streaming frameworks, and how to manage machine learning infrastructure. This episode is a great companion to our show about Uber’s data platform, and illustrates some fundamental differences in how the two ridesharing companies operate.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

To the extent that I am a software engineering journalist, I feel inclined to scrutinize all of the cloud providers. But to the extent that I am an engineer and a business person, I feel only admiration and love for the cloud providers. Cloud computing has brought the cost of starting an Internet business down to zero. Cloud computing has opened up my eyes to a world of creative possibilities that knows no boundaries, and for that I will always be a fan of all of the rivaling cloud companies because they all have played a role in creating the current software landscape. Eric Brewer is a Google Fellow and VP Infrastructure. He is well-known for his work on the CAP theorem, a distributed systems concept that formalized the tradeoffs between consistency, availability, and partition tolerance in a distributed system. At Google, Eric is as much a strategist and product creator as he is a theoretician. He has worked on database systems such as Spanner, machine learning systems such as TensorFlow, and container orchestration systems such as Kubernetes and GKE. Eric joins the show to talk about Google’s philosophy as a cloud provider, and how his understanding of distributed systems has evolved since joining the company. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Intricately is a company that maps the breadth and depth of cloud infrastructure usage. Using a combination of clever algorithms, data engineering, and web crawlers, Intricately derives information about how different companies spend money on infrastructure. Fima Leshinsky is the CEO and co-founder at Intricately. In his previous job at Akamai, he began to study how a cloud provider such as Akamai could figure out how much its competitors were charging certain customers. Since CDN infrastructure is a commodity with reasonably low switching cost, a provider that can undercut its competitors significantly can have an edge in the marketplace. From his work at Akamai, Fima felt there was a market opportunity to provide this kind of service to the broader market of cloud providers. There are more cloud providers than ever before, and the kind of data that Intricately aggregates is highly useful to this competitive marketplace. Fima joins the show to talk about the modern landscape of cloud providers, and how to build a system that maps the Internet.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat. The Linux operating system includes user space and kernel space. In user space, the user can create and interact with a variety of applications directly. In kernel space, the Linux kernel provides a stable environment in which device drivers interact with hardware and manage low level resources. A Linux container is a virtualized environment that runs within user space. To perform an operation, a process in a container in user space makes a syscall (system call) into kernel space. This allows the container to have access to resources like memory and disk. Kernel space must be kept secure to ensure operating system integrity–but Linux includes hundreds of syscalls. Each syscall represents an interface between the user space and kernel space. Security vulnerabilities can emerge from this wide attack surface of different syscalls, and most applications only need a small number of syscalls to perform their required functionality. gVisor is a project to restrict the number of syscalls that the kernel and user space need to communicate. gVisor is a runtime layer between the user space container and the kernel space. gVisor reduces the number of syscalls that can be made into kernel space. The security properties of gVisor make it an exciting project today–but it is the portability features of gVisor that hint at a huge future opportunity. By inserting an interpreter interface between containers and the Linux kernel, gVisor presents the container world with the opportunity to run on operating systems other than Linux. There are many reasons why it might be appealing to run containers on an operating system other than Linux. Linux was built many years ago, before the explosion of small devices, smart phones, IoT hubs, voice assistants and smart cars. To be more speculative, Google is working on a secretive new operating system called Fuscia. gVisor could be a layer that allows workloads to be ported from Linux servers to Fuscia servers. Yoshi Tamura is a product manager at Google with a background in containers and virtualization. He joins the show to talk about gVisor and the different kinds of virtualization.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs; we will be announcing our next hackathon in a few weeks, so stay tuned Products are an art form. As with any art, the world of products includes creators, patrons, fans, business people, and investors. Product Hunt is a place where those different people connect to build and discuss products. Products are different from other art forms in that they are measured not only through the lens of design and beauty–but also through utility. From software to books to couches to toiletry–we all have products that have improved our lives so much that we feel a deep sense of connection and hope for that product and the people behind it. Ryan Hoover is the founder of Product Hunt, a product I have found tremendous value and satisfaction from over the years. He is also a host of Product Hunt Radio, a weekly podcast with the people creating and exploring the future. Ryan joins the show to discuss products, the process of creating something useful, and his investing strategy. Ryan runs the Weekend Fund, an early stage investment fund.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs; we will be announcing our next hackathon in a few weeks, so stay tuned Open source policy has become a business issue as well as a political one. Businesses like Elastic, MongoDB (the company), and Redis Labs have started to view the open source licenses of the projects they work on as a means for business defensibility against cloud providers offering similar services. It remains to be seen how viable this strategy will be for the commercial open source vendors. Companies that do not directly sell commercial open source are also grappling with questions around open source licensing. Facebook has become a force in the open source world through projects like React and GraphQL. Facebook leads these projects, but Facebook is not monetizing them other than to the extent that they use the projects to build Facebook.com. Facebook’s incentives are aligned with the rest of the industry on the quality of the GraphQL and React projects. Proper licensing can help Facebook keep those incentives in alignment. Joel Marcey, Michael Cheng, and Kathy Kam from Facebook join me for a discussion of the state of open source licensing, and how that impacts Facebook.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs; we will be announcing our next hackathon in a few weeks, so stay tuned Drishti is a company focused on improving manufacturing workflows using computer vision. A manufacturing environment consists of assembly lines. A line is composed of sequential stations along that manufacturing line. At each station on the assembly line, a worker performs an operation on the item that is being manufactured. This type of workflow is used for the manufacturing of cars, laptops, stereo equipment, and many other technology products. With Drishti, the manufacturing process is augmented by adding a camera at each station. Camera footage is used to train a machine learning model for each station on the assembly line. That machine learning model is used to ensure the accuracy and performance of each task that is being conducted on the assembly line. Krish Chaudhury is the CTO at Drishti. From 2005 to 2015 he led image processing and computer vision projects at Google before joining Flipkart, where he worked on image science and deep learning for another four years. Krish had spent more than twenty years working on image and vision related problems when he co-founded Drishti. In today’s episode, we discuss the science and application of computer vision, as well as the future of manufacturing technology and the business strategy of Drishti.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs; we will be announcing our next hackathon in a few weeks, so stay tuned Lyft is a ridesharing company with petabytes of data. Within Lyft, many different employees can use those data sets to build useful applications. A business analyst creates a dashboard to see how driver satisfaction is changing over time. An economist studies the pricing data to ensure that Lyft’s prices are competitive. A data scientist creates a report of how the speed of a ride correlates with 5 star ratings. A machine learning engineer trains a model to detect fraud on the platform. All of these use cases make sense–and in each of them, the employee at Lyft needs to find the necessary data sets within the company to build their application. Amundsen is a tool for finding and discovering data sets within the company. Tao Feng and Mark Grover are engineers at Lyft and join the show to talk about the problem of data discovery and the tools they have built at Lyft.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs; we will be announcing our next hackathon in a few weeks, so stay tuned Until Google DeepMind came into the field, protein structure prediction was dominated by academics. Protein structure prediction is the process of predicting how a protein will fold by looking at genetic code. Protein structure prediction is a perfect field to approach through the application of deep learning, because the inputs are highly dimensional and there is a plentiful array of different sets of labeled data. Protein structure deep learning is a field in which many different approaches are taken, often involving supervised learning and reinforcement learning. Mohammed Al Quraishi is a systems biologist at Harvard. His background spans computer engineering, statistics, and genetics. In his work, Mohammed explores the interplay between biology and computer systems. One area of Mohammed’s focus is protein structure prediction. In a blog post last year, Mohammed gave a brief history of protein structure prediction and described the significance of DeepMind entering the field. DeepMind’s AlphaFold technology surpassed all other competitors in the most recent CASP protein structure competition. Mohammed joins the show to discuss biology, academia, deep learning, and DeepMind.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Podsheets is a set of open source tools for podcast hosting, publishing, ad management, community engagement, and more. Podsheets is influenced by our experience managing Software Engineering Daily, a full-time podcast business. Software Engineering Daily is a podcast that airs 5 times per week. With 4 ads per show and 50 business weeks per year, we

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Haseeb Qureshi is an entrepreneur and investor. As a teenager, Haseeb played poker professionally through the online poker bubble. His path from poker to software entrepreneurship has been explored in previous episodes. In 2007, Haseeb and I met at an online poker table. As we battled each other for thousands of dollars, Haseeb and I realized we shared an affinity for obnoxious screen names, obnoxious online avatars, and the city of Austin, Texas. We were both living in the city, and met each other in the real world. In our earliest days, Haseeb and I were not friends. It was a strange time–we were disembodied minds, drifting on the Internet, attached mostly to the fluctuating balances of our Full Tilt Poker and Pokerstars accounts. This was not a time for friendship–it was a time for ruthless, modern competition. Haseeb grew tired of poker. He wrote a book about the game to memorialize his thoughts, then abandoned it. He studied philosophy and literature, searching for something new in the historical musings of humanity. He traveled Europe, working as a farmer to reconnect with the physical world. He discovered the Effective Altruism movement. Finding no solace in his poker spoils, Haseeb gave away most of his money and started from scratch. As he rebuilt himself, he found software engineering and charted a path to San Francisco, where we reconnected. In this episode, Haseeb joins me for a discussion of software, philosophy, poker, and the nature of bubbles. Indeed, Haseeb and I have now lived through four major bubbles: dot coms, poker, the 2008 financial crisis, and the crypto bubble. Throughout these bubbles, the mediums change but never does the message: human beings are deeply irrational, tribalistic, and emotional. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RECENT UPDATES: FindCollabs $5000 Hackathon Ends Saturday April 15th, 2019 New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more Consul is a tool from HashiCorp that allows users to store and retrieve information from a highly available key/value data store. Consul is used for storage of critical cluster information, such as service IP locations and configuration data. A service interacts with Consul via a daemon process on the node of that service. The daemon process periodically shares information with the Consul server over a gossip UDP protocol and can share data on a more immediate basis using TCP. Consul’s functionality has increased recently to add secure service connectivity. Consul Connect allows services to establish mutual TLS encryption with each other. The addition of mutual TLS to the Consul feature set is closely incidental with Consul gaining a title of “service mesh.” Service mesh is an increasingly popular pattern that can encompass a variety of features: load balancing, security policy management, service discovery, and routing. Tools which offer self-described “service mesh” functionality include Linkerd, Kong, AWS App Mesh, Solo.io Gloo, and Google’s Istio open source project. Paul Banks is the engineering lead of Consul at HashiCorp. He joins the show to talk about the service mesh category and the past, present, and future of Consul.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RECENT UPDATES: FindCollabs $5000 Hackathon Ends Saturday April 15th, 2019 New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more Data sets can be modeled in a row-wise, relational format. When two data sets share a common field, those data sets can be combined in a procedure called a join. A join combines the data of two data sets into one data set that is often bigger than the initial two data sets independently occupied. In fact, this new data set is often so much bigger that it creates problems for the machine learning engineers. Arun Kumar is an assistant professor at UC San Diego. He joins the show to discuss the modern lifecycle of machine learning models, and the gaps in the tooling. Arun’s research into improving processing of joined data sets has been adopted by companies such as Google. Some of that research has been adapted into open source machine learning tools that improve the performance of machine learning jobs with minimal code required.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RECENT UPDATES: FindCollabs $5000 Hackathon Ends Saturday April 15th, 2019 New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more Distributed stream processing allows developers to build applications on top of large sets of data that are being rapidly created. Stream processing is often described as an alternative to batch processing. In batch processing, a single large computation is performed over a large, static data set. In stream processing, a computation is performed repeatedly and continuously over a data set that is being appended to. A stream is often stored in a distributed queue such as Kafka, Kinesis, Pulsar, or Google PubSub. A stream is often processed with a stream processing tool such as Spark, Flink, Storm, or Google Cloud Dataflow. Holden Karau is an engineer who works on open source projects at Google. She returns to the show to describe the state of stream processing and discuss modern best practices.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RECENT UPDATES: FindCollabs $5000 Hackathon Ends Saturday April 15th, 2019 New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more A software application requires compute and storage. Both compute and storage have been abstracted into cloud tools that can be used by developers to build highly available distributed systems. In our previous episode, we explored the compute side. In today’s episode we discuss storage. Application developers store data in a variety of abstractions. In-memory caches allow for fast lookups. Relational databases allow for efficient retrieval of well-structured tables. NoSQL databases allow for retrieval of documents that may have a less defined schema. File storage systems allow the access pattern of nested file systems, like on your laptop. Distributed object storage systems allow for highly durable storage of any data type. Amazon S3 is a distributed object storage system with a wide spectrum of use cases. S3 is used for media file storage, archiving of log files, and data lake applications. S3 functionality has increased over the years, developing different tiers of data retrieval latency and cost structure. AWS S3 Glacier allows for long-term storage of data at a large cost reduction, in exchange for increased latency of data access. Kevin Miller is the general manager of Amazon Glacier at Amazon Web Services. He joins the show to talk about the history of storage, the different options for storage in the cloud, and the design of S3 Glacier.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming event: FindCollabs Hackathon at App Academy on April 6, 2019 On Amazon Web Services, there are many ways to run an application on a single node. The first compute option on AWS was the EC2 virtual server instance. But EC2 is a large abstraction compared to what many people need for their nodes–which is a container with a smaller set of resources to work with. Containers can be run within a managed cluster like ECS or EKS, or run on their own as AWS Fargate instances, or simply as Docker containers running without a container orchestration tool. Beyond the option of explicit container instances, users can run their application as a “serverless” function-as-a-service such as AWS Lambda. Functions-as-a-service abstract away the container and let the developer operate at a higher level, while also providing some cost savings. Developers use these different compute options for different reasons. Deepak Singh is the director of compute services at Amazon Web Services, and he joins the show to discuss the use cases and tradeoffs of these options. Deepak also discusses how these tools are useful internally to AWS. ECS and Lambda are high-level APIs that are used to build even higher level services such as AWS Batch, which is a service for performing batch processing over large data sets.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Ben Lorica is the chief data scientist at O’Reilly Media and the program director of the Strata Data Conference. In his work, Ben spends time with people across the software industry, giving him broad perspective. In the early days of the data engineering ecosystem, the Hadoop vendor wars were starting between Cloudera and Hortonworks. Strata was a neutral ground for practitioners and open source contributors to meet and share ideas about the Hadoop ecosystem. Since then, the conference has grown to encompass topics such as data science, distributed databases, streaming frameworks, and machine learning. There are many open questions in the data world right now. What is the best path that an enterprise can take to build out a data platform? How should a software team be arranged to efficiently build machine learning models? Which distributed streaming frameworks should I use for what purpose? Ben joins the show to discuss modern data engineering, data science, and infrastructure.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 A currency can fulfill numerous financial use cases. One use case is store of value: currency holders can reliably expect their currency to maintain some value, though that value may fluctuate over time. Another use case is speculation: currency holders are owning currency in the hope that the market price of the currency will increase over time. Bitcoin is a useful store of value and an instrument for speculation. However, Bitcoin still does not fulfill the financial use case that most people need from a currency: price stability. The price of Bitcoin fluctuates rapidly, making it difficult to use Bitcoin for small purchases such as coffee. Imagine you want to buy a cup of coffee with Bitcoin. The coffee shop owner needs to offer the option to sell you that cup of coffee using Bitcoin as the medium of exchange. This owner must denominate the price of that coffee as some number of Bitcoin. Since the price of Bitcoin fluctuates so rapidly, the coffee shop owner needs to adjust the price of that cup of coffee constantly in order to make sure that the coffee is cheap enough for the consumer to want to buy it, but expensive enough to make a profit. It is hard to assign prices to market goods in terms of Bitcoin because the currency is in constant flux. Even though many of us would like to use Bitcoin in our everyday lives, most marketplaces are denominated in US dollars or other currencies because a marketplace needs a stable currency in order to operate. Rune Christensen is the CEO of MakerDAO, a system that provides a price-stable cryptocurrency. MakerDAO is an elegant set of currencies, collateralized debt, smart contracts, and other incentive tools that result in the creation of several transparent, decentralized financial instruments. Rune joins the show to talk about the importance of stablecoins and how MakerDAO has engineered a decentralized currency that has maintained stability even through tumultuous market conditions.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Chris Yeh is an entrepreneur, investor, and author. He co-wrote Blitzscaling with LinkedIn founder Reid Hoffman. Blitzscaling is a strategy for growing a company that has found product market fit. Blitzscaling prioritizes speed over efficiency, arguing that fast growth is necessary to achieve “first scaler advantage.” When a company is the first to scale successfully within a large market, that company gains access to a wealth of market opportunities that are not available to companies which are not at scale. Examples of successful Blitzscalers include Airbnb, LinkedIn, Amazon, and Facebook. In the hypergrowth phases of these companies, there were deliberate strategic tradeoffs that caused the company to suffer in the short term in exchange for the chance at market dominance in the long term. Blitzscaling is a broad strategic concept which manifests differently in different companies. When Airbnb was in its early stages of growth in 2011, the company was faced with the existential threat of a European competitor called Wimdu. Wimdu offered to sell to Airbnb, but this would have required the merger of two companies with distinctly different cultures. Instead, Airbnb chose to raise more money and rapidly expand into Europe. In contrast, Google’s rapid path to becoming a dominant information service involved acquisitions that we now see as key Google products, including Android, Google Maps, and Google Earth. Through numerous examples in recent business history, Blitzscaling explores the fundamental tradeoff between speed and efficiency, usually biasing speed as the preferable element. But Blitzscaling does not work for every company. In the food delivery sector, many companies who tried to blitzscale ended up going out of business because they had lowered their prices too much in order to try to earn customer loyalty. By lowering their prices too much, food delivery startups built businesses with fundamentally bad unit economics and a fickle customer base. In other cases, aggressive blitzscaling can work for a short period of time, but can cause a company’s culture to suffer in ways that are very hard to repair. Blitzscaling can also cause problems in a core software product. Growing too quickly can cause a product to have a bloated user interface. If the backend infrastructure layer expands too quickly, sensitive data could be left exposed due to a lack of proper software security policies. Chris Yeh joins the show to talk about the strategy of Blitzscaling and his wide-ranging career. Chris studied creative writing and product design at Stanford before joining DE Shaw, the famous quantitative hedge fund. Later, he became an investor and worked in several leadership roles in software companies. His wide range of experiences make Chris an excellent author and conversationalist. We explored the ideas of both Blitzscaling and his previous book The Alliance, which lays out a modern vision for the dynamic between employers and employees. We also talked about investing, Dungeons and Dragons, and podcasting.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Uber’s infrastructure supports millions of riders and billions of dollars in transactions. Uber has high throughput and high availability requirements, because users depend on the service for their day-to-day transportation. When Uber was going through hypergrowth in 2015, the number of services was growing rapidly, as was the load across those services. Using a cloud provider was a risky option, because the costs could potentially grow out of control. Uber made a decision early on to invest in physical hardware in order to keep costs at a reasonable level. In the last 3 years, Uber’s infrastructure has stabilized. The platform engineering team has built systems for monitoring, deployment, and service proxying. Developing and maintaining microservices within Uber has become easier. Prashant Varanasi and Akshay Shah are engineers who have been with Uber for more than three years. They work on Uber’s platform engineering team, and their current focus is on the service proxy layer, a sidecar that runs alongside Uber services providing features such as load balancing, service discovery, and rate limiting. Prashant and Akshay join the show to talk about Uber infrastructure, microservices, and the architecture of a service proxy. We also talk in detail about the benefits of using Go for critical systems infrastructure, and some techniques for profiling and debugging in Go.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Google has been building large-scale scheduling systems for more than fifteen years. Google Borg was started around 2003, giving engineers at Google a unified platform to issue long-lived service workloads as well as short-lived batch workloads onto a pool of servers. Since the early days of Borg, the scheduler systems built by Google have matured through several iterations. Omega was an effort to improve the internal Borg system, and Kubernetes is an open source container orchestrator built with the learnings of Borg and Omega. A scheduling system needs to be able to accept a wide variety of workload types and find compute resources within a cluster to schedule those workloads onto. There is a wide variety of potential workloads that could be scheduled–batch jobs, stateful services, stateless services, and daemon services. Different workloads can have different priority levels. A high priority workload should be able to find compute resources quickly, and a low priority workload can wait longer to find resources. Brian Grant is a principal engineer at Google. He joins the show to talk about his experience building workload schedulers and designing APIs for engineers to interface with those schedulers.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Google’s Borg system is a cluster manager that powers the applications running across Google’s massive infrastructure. Borg provided inspiration for open source tools like Apache Mesos and Kubernetes. Over the last decade, some of the largest new technology companies have built their own systems that fulfill the roles of cluster management and resource scheduling. Netflix, Twitter, and Facebook have all spoken about their internal projects to make distributed systems resource allocation more economical. These companies find themselves continually reinventing scheduling and orchestration, with inspiration from Google Borg and their own internal experiences running large numbers of containers and virtual machines. Uber’s engineering team has built a cluster scheduler called Peloton. Peloton is based on Apache Mesos, and is architected to handle a wide range of workloads: data science jobs like Hadoop MapReduce; long running services such as a ridesharing marketplace service; monitoring daemons such as Uber’s M3 collector; and database services such as MySQL. Min Cai and Mayank Bansal are engineers at Uber who work on Peloton. When they set out to create Peloton, they looked at the existing schedulers in the ecosystem, including Kubernetes, Mesos, Hadoop’s YARN system, and Borg itself. Both Min and Mayank join the show today to give a brief history of distributed systems schedulers and discuss their work on Peloton. They have been working in the world of distributed systems schedulers for many years–including experiences building core Hadoop infrastructure and virtual machine schedulers at VMware.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Log management requires the processing and indexing of high volumes of semi-structured data. A log management service takes log data and puts it in a cloud-hosted application so that application operators can access those logs to troubleshoot issues. A large tech company will produce terabytes of logs. Those logs are produced on the host where a service is running. A logging agent on that host will transfer the logs to the log management service in the cloud. Once the logs are in the cloud, they are parsed, indexed, and stored in a way that is easy to query. In 2014, Renaud Boutet co-founded Logmatic, a log management service that eventually became a leading provider. Logmatic was acquired by Datadog, and Renaud now works as a vice president at Datadog. In today’s episode, Renaud joins the show to talk about the architecture of a log management service. We talk about storage tiers, scalability requirements, failover strategies, and logging for serverless functions. Full disclosure: Datadog is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Steve Herrod was the CTO at VMware and now works as a managing director at General Catalyst, where he focuses on investments relating to security. Large enterprises are difficult to secure. An enterprise has sprawling infrastructure, with both on-prem and cloud infrastructure. Identity management systems, vulnerability scanning, secure network infrastructure, and policy management tools are just a few example areas where enterprises spend billions of dollars on security software. Threats often make their way into an enterprise by way of social engineering. This can result in phishing attacks, corporate espionage, and ransomware. Protecting against social engineering is very difficult, as there are so many channels to communicate through–Facebook Messenger, Linkedin, email, and ad networks can all be used to perform social engineering attacks. Enterprise security software is a very different business from other types of software companies. Unlike developer tools or cloud infrastructure, security software is usually not self-serve. Security solutions usually require a longer sales and integration process with a customer. Steve Herrod joins the show to talk about the enterprise security world, the go-to-market strategy for successful security companies, and his perspective on what makes for a viable venture capital investment.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Coding in the browser has been attempted several times in the last decade. Building a development environment in the browser has numerous technical challenges. How does the code execute safely? How do you fit all of the requirements of a development environment into a browser window? How do you get users to switch from their normal IDE (interactive development environment)? CodeSandbox is an online code editor created by Ives van Hoorne and Bas Buursma. CodeSandbox allows users to program and run applications in the browser. It is a full developer platform that allows users the ability to install npm modules, run their code, and share their applications with other users. The engineering problems within CodeSandbox are not easy–building a web-based IDE is complicated. But CodeSandbox is also an exciting project because it lowers the barrier to entry for many newer programmers. The development experience for a new programmer is still a difficult onramp. If you are an experienced developer, you have a workflow that you are comfortable with. It might involve vim, or emacs, or JetBrains IDEs, or Eclipse. But newer developers can find these environments confusing and hard to get started with. The development environments of today are integrated with build tools, Github repositories, and deployment platforms. This can be overwhelming for a newer developer. CodeSandbox is a very visual tool, which makes it especially useful for new developers who learn through seeing examples running live in the browser. CodeSandbox is also used by web developers who want a modern, shareable form of developing software. Ives and Bas join the show to talk about the motivation for CodeSandbox and the engineering challenges they have solved.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Data engineering touches every area of an organization. Engineers need a data platform to build search indexes and microservices. Data scientists need data pipelines to build machine learning models. Business analysts need flexible dashboards to understand the trends and customer use for a product. Max Beauchemin is a data engineer who has worked at Airbnb, Lyft, and Facebook. He’s the creator of two successful open source projects: Apache Airflow and Apache Superset. In a previous show, Max discussed data engineering at Airbnb, and the usage of Airflow. In today’s show, Max discusses the engineering of Apache Superset. Superset is an open source business intelligence web application. Superset allows users to create visualizations, slice and dice their data, and query it. Superset integrates with Druid, a database that supports exploratory, OLAP-style workloads. One reason Superset is distinctive is that it is a full open source application. Many open source projects are tools like databases, command line tools, and web frameworks. Superset is an open source application that can be used by individuals who are not developers–so the audience is wider than the typical open source tool built for engineers. Max joins the show to talk about his experience as a data engineer at Airbnb and Lyft, and the open source projects he has started.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Twitter’s early engineers faced scalability problems that caused infrastructure failures on a regular basis. The infamous “fail whale” could happen as a result of problems in the application servers, the network, or the database layer. When Twitter was scaling in its early days, the cloud providers were still immature. Engineers did not have access to the autoscaling cloud infrastructure that is available today. The early Twitter architecture was a combination of open source tools and internally created infrastructure custom built for Twitter’s workloads. Evan Weaver was an early engineer at Twitter, and he saw the deficiencies of the data tools that the company had access to. Twitter engineers wanted access to a truly reusable data platform that would fit Twitter’s requirements: high availability, globally replicated, and transactionally consistent. By 2012, Evan had left Twitter and started consulting for other technology companies. He found that databases across the industry were lacking the same properties that Twitter wanted, and the ideas for FaunaDB began to percolate. Around this time, there were two relevant papers about distributed databases that had come out: the Spanner paper from Google and the Calvin paper, a distributed systems paper from Yale. With inspiration from the literature, his time at Twitter, and his knowledge from consulting, Evan started FaunaDB. Seven years later, FaunaDB is a fully fledged open source project as well as a database company with a cloud service offering. Fauna is an OLTP database used by companies like Nvidia, Nextdoor, and Capital One. Evan joins the show to talk about his time spent scaling Twitter and the architecture of FaunaDB.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Bol.com is the biggest e-commerce company in the Netherlands and Belgium. For 20 years, Bol has been developing its software architecture, which includes a variety of services and databases, and a mix of physical and cloud infrastructure. For an ecommerce company, the search engine is critical for allowing customers to find the products they are looking for. But search also has many applications for internal systems. A search engine is a database with a query engine, and internal application developers want to build on top of that database. Volkan Yazici is an engineer at Bol.com specializing in search and the author of the blog post Using ElasticSearch as the Primary Data Store. In his post, Volkan describes the process of scaling ElasticSearch to fit the use cases of both internal and external users at a large ecommerce company. Volkan joins the show to discuss how search infrastructure at scale can require a carefully architected data pipeline in order to propagate changes to a large data set to a search index.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Modern web development tools have given frontend developers more power. On the frontend, JavaScript frameworks like React and Vue have become easier to work with. For deployment, tools like Netlify and Zeit give developers a workflow that is tightly integrated with GitHub. At the database layer, autoscaling document storage systems like Firebase and hosted Mongo solutions make it easier to work with objects. There are also a multitude of APIs that give developers rich business functionality out of the box, making it easy to build applications around SMS, payments, and computer vision. If you are building a new application today, you have the option to build it around a completely “serverless” architecture. As the backend and frontend have changed, the middleware to communicate between those layers has also evolved. GraphQL is a modern way of fetching data from disparate data sources. In previous episodes, we have talked about how GraphQL works, and some common patterns for using GraphQL in mature applications. In today’s episode, Tanmai Gopal joins the show to describe how to use GraphQL in newer applications. Tanmai is the CEO of Hasura, a company building tools around GraphQL. He discusses the advantages of using serverless functions together with GraphQL, and how to architect an event-based serverless application.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the world of commercial open source, there is plenty of room for both point solution providers and cloud providers. But they are competing for the same customers, and the competitive battlefield is expanding to the nuanced world of software licensing. By changing their licenses, open source projects like Kafka, MongoDB, and Redis can prohibit AWS from certain usage patterns. This might offer some protection for companies based around the point solutions–companies like Confluent and RedisLabs. Beyond the fracas of the battle between cloud providers and point solutions, there are newer open source companies with models that do not fit tightly into any historical business models. HashiCorp makes a suite of differentiated open source tools that have not been seriously contested or offered as a service by cloud providers. GitLab makes an open source platform that is built with monitoring, logging, CI, and code hosting out of the box. As the world of open source business models expands, more companies will find opportunity in open sourcing the code that runs their products. In many cases, they will find that it strengthens their advantage rather than weakens it. The defensibility of many businesses relies more on data and network effects than the contents of the codebase. We may see the default question gradually shift from “why should I open source my codebase?” to “why shouldn’t I open source my codebase?” Mike Volpi is a partner at Index Ventures and has invested in many open source businesses over the last decade. He is on the board of Confluent, Cockroach Labs, Kong, and Elastic. Mike joins the show to share his perspective on open source business models of the past, present, and future. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

This is a post written and narrated by Haseeb Qureshi, a cryptocurrency investor and entrepreneur. Haseeb is speaking at an upcoming Software Engineering Daily Meetup. The ICO bubble had no single cause. Mono-causal explanations always fall short in explaining complex phenomena. But its effects are easier to pinpoint. There are now many world class teams well-capitalized to build, scale, and evolve blockchain technology, and tens of millions of people in the world who now understand decentralization, proof of work, and private keys. Looking back, it’s really quite amazing! It comes at a high cost, but Perez hints: it’s likely that bubbles like these are the only way to overcome technological inertia. At the same time, most people had their first interaction with crypto during its orgiastic adolescence. It’s not a great look. But this has been true for every technological revolution of the last 250 years. In that regard, crypto is in good company. I was too young to appreciate the dot com bubble when it happened. It’s strange to say, but I’m glad to have witnessed a speculative bubble from up close. I’ve now got war stories to share with future generations. It was a wild time, when anyone in the world could launch a coin and raise tens of millions of dollars to build a network that no one could control. I don’t think we’ll see anything like that again for a long time. So what happens now? If you believe that crypto has the stuff of a technological revolution, then as Perez puts it, the collapse will pave the way for a more fruitful deployment phase. At the end of the day, I’m an optimist about technology. So it won’t surprise you that I think this deployment phase is coming. But it will be slow, unglamorous, and probably won’t make for nearly as entertaining of headlines. Oh well. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

GitLab is an open source platform for software development. GitLab started with the ability to manage git repositories and now has functionality for collaboration, issue tracking, continuous integration, logging, and tracing. GitLab’s core business is selling to enterprises who want a self-hosted git installation, such as banks or other companies who prefer not to use a git service in the cloud. The vision for GitLab is to provide a platform for managing the full software development lifecycle, from code hosting to deployment–as well as tools for observability and project management. Sid Sijbrandij is the CEO of GitLab and he joins the show to talk about the product, the business, and the company’s vision for the future. GitLab’s strategy is to offer a set of tools that work for developers out of the box, cutting down on time spent integrating each individual vendor.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

An operating system kernel manages the system resources that are needed to run applications. The Linux kernel runs most of the smart devices that we interact with, and is the largest open source project in history. Shuah Khan has worked on operating systems for two decades, including 13 years at HP and 5 years at Samsung. She has worked on proprietary operating systems and a variety of Linux operating system environments, including mobile devices. Shuah joins the show to discuss her work within Linux and her experience contributing to open source. Shuah has made significant contributions to kselftest, a set of tests for Linux. Testing the Linux kernel is complicated. Because there is so much depth to the codebase, and such a variety of ways that Linux can be used, there is also a variety of ways that the operating system gets tested. There is smoke testing, performance testing, and regression testing. There are trees of tests, and as a developer you may only want to run a subset of the tests in that tree. The conversation with Shuah ranged from the low level practices of testing the kernel to a high level discussion of how the Linux kernel can reveal dynamics of human nature.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Malware is malicious software that makes money for the creator of that software. Malware can appear onto a user’s computer if that user visits a malicious website or installs malicious software by accident. There are many types of malware. Spyware sits on your machine and logs your data in order to sell it. Ransomware can lock your computer and demand that you pay money to unlock it. Adware serves you popup ads that you don’t want to see. Cryptojacking is a newer form of malware. Cryptojacking software uses your computer to mine Bitcoin and other cryptocurrencies. Cryptojacking can occur when you visit a website that is running JavaScript that is executing along with the rest of the webpage. When you visit a website with a cryptojacker, your computer will become slower, because your CPU is being taken over to mine cryptocurrency. Cryptojacking can occur anywhere that code runs–and there is a lot of code running on cloud providers. Cloud providers themselves are very secure. But a cloud provider cannot force its customers to be secure. Users who host an insecure application on a cloud provider may get infected with a cryptojacker. If I host a large, complex website on a cloud provider, and I’m serving millions of users, I’m already paying a lot in cloud costs. But when my application gets infected with a cryptojacker, my costs could shoot up. And if I don’t know why my costs are increasing, I might leave the cloud provider. Estaban Vargas is the co-founder of SafeTalpa, a company that provides defense against cryptojackers. Estaban joins the show to explain how cryptojackers work and why cloud providers have trouble defending against them.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Advertising fraud occurs when a brand pays for an advertisement online and that advertisement is shown to an automated bot account that has been created to view ads. Advertising fraud is rampant on the Internet. It’s not possible to know how much money is lost to ad fraud, but the costs are in the billions of dollars. Praneet Sharma and Shailin Dhar are the founders of Method Media Intelligence, a company that builds solutions around improving advertising quality. In previous shows, Praneet and Shailin have described the online advertising ecosystem in detail. They have told stories of bot farms, replay attacks, and adtech companies. In today’s episode, Praneet and Shailin return to the show to discuss how advertising fraud is getting worse–not better. Praneet and Shailin worked with BuzzFeed reporter Craig Silverman, who was a previous guest on the show to talk about his remarkable findings about mobile advertising fraud, which accounts for hundreds of millions of dollars in theft every year.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The demand for electricity is based on the consumption of the electrical grid at a given time. The supply of electricity is based on how much energy is being produced or stored on the grid at a given time. Because these sources of supply and demand fluctuate rapidly but predictably, energy markets present profit opportunities for traders. Minh Dang and Corey Noone are engineers with Advanced Microgrid Solutions, a company that builds software to help traders capture better opportunities in the energy markets. Minh and Corey join the show to talk about how their company builds and deploys machine learning models for market prediction. We discussed data infrastructure, machine learning model deployments, and the dynamics of the energy markets.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud computing started to become popular in 2006 with the release of Amazon EC2, a system for deploying applications to virtual machines sitting on remote data center infrastructure . With cloud computing, application developers no longer needed to purchase expensive server hardware. Creating an application for the Internet became easier, cheaper, and simpler. As the cloud has become popular, new ways of deploying applications have emerged. A developer with a web app has so many different options. You can host your app on an Amazon EC2 server, which will require you to manage cloud infrastructure in case your server crashes. You can deploy your app to Heroku, which gives your cloud deployment better uptime guarantees for a higher price than Amazon EC2. Or you can use Linode, or Microsoft Azure, or Google Cloud. There is such a large market for cloud computing that the world of cloud providers serves more niches every year. In past episodes we have explored a variety of different cloud providers, and the markets they target. Pivotal Cloud Foundry is for managing complex distributed systems applications, typically with large teams. Firebase is a cloud provider that simplifies the developer experience for applications with small teams. Spotinst is a cloud provider that emphasizes low cost. Zeit is a cloud provider that is built to manage applications through serverless “functions-as-a-service” like AWS Lambda. In today’s episode, Mathias Biilman Christensen, CEO of Netlify, joins the show. Netlify is a cloud provider that was built for modern web projects. Netlify represents the convergence of several trends in software development converging: static site deployment, serverless functions, a desire to have a “no-ops” deployment with minimal management, and the rise of newer tools like GraphQL and Gatsby. Mathias explores these trends in detail, and explores the technical challenges of building Netlify. He was a great guest, capable of talking about difficult backend problems that require writing C++, as well as the frontend world of JavaScript frameworks. One announcement before we begin: we are having a $5000 hackathon. The $5000 hackathon is for a new product we’ve been working on: FindCollabs. FindCollabs is a platform for finding collaborators and building projects. Whether you are an engineer, a musician, a designer, a videographer, or an artist, FindCollabs lets you find people and collaborate. To try out FindCollabs, just go to FindCollabs.com, you can make a project or you can join someone else’s project. And it’s very easy to make these projects–you don’t need to have anything built yet–you need to have a vision for what you want to build. And to find out about the hackathon, go to findcollabs.com/hackathon. We are giving away $5000 in cash to the coolest projects that get built before Sunday April 14th. So I recommend getting started early, finding some people to collaborate with, and building some cool stuff!

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Monitoring tools are used by every area of an organization. Business development teams use monitoring to understand the metrics for product performance. Finance teams need to understand how the costs of cloud computing resources are changing. Site reliability engineers use monitoring dashboards that applications are up and running without problematic latency. Product managers evaluate the results of AB tests based off of the monitoring data of how users are reacting to new features. A monitoring system needs to be able to handle large volumes of data that are being generated at a high velocity. The data needs to be queryable in an aggregated format, which might require an ETL system for getting data into columnar format. Alexander Pucher is an engineer at LinkedIn, where he works on a monitoring platform called ThirdEye. ThirdEye is built on top of Apache Pinot, a distributed columnar storage engine that ingests data and serves analytical queries at low latency. Pinot uses RocksDB, and is comparable to Apache Druid. Alexander joins the show to discuss ThirdEye, and explain why Pinot is a useful building block for monitoring infrastructure.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

WebAssembly is a runtime that lets languages beyond JavaScript to execute in frontend web applications. WebAssembly is novel because most modern frontend applications are written entirely in JavaScript. WebAssembly lets us use languages like Rust and C++ after they have been compiled down to a web assembly binary module. Language interoperability is only one part of why WebAssembly is exciting. The execution environment for WebAssembly modules has benefits for security and software distribution and consumption as well. In previous shows, we’ve given an overview of WebAssembly and explored its future applications as well as its relationship to the Rust programming language. In today’s episode, we explore the packaging and execution path of a WebAssembly module, and some other applications of the technology. Syrus Akbary is the CEO and founder of Wasmer, a company focused on creating universal binaries powered by WebAssembly. Wasmer provides a way to execute WebAssembly files universally. He joins the show to talk about the state of WebAssembly, and what his company is building.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cryptocurrencies enable a large number of applications. Trustless reputation systems, decentralized identity tools, micropayments, non-fungible Internet items, borderless currencies, just to name a few. But cryptocurrencies have not yet impacted daily life, for most of us. Why is that? One reason is that it is still very hard for developers to build within the cryptocurrency ecosystem. The programming languages, such as Solidity, are not widely used by software engineers. Building and deploying smart contracts is not as easy as deploying a simple Ruby on Rails webapp. The open source tooling is immature, as are the paid developer tools. Sean Li is the CEO of Fortmatic, a company that is building tools to improve the Ethereum developer experience. Fortmatic simplifies wallet creation, user identity management, security, and money transfer for Ethereum developers. Before starting Fortmatic, Sean was the founder of Kitematic, a company that made the developer experience of Docker easier. Kitematic was acquired by Docker. Sean is one of the few people with significant experience in both the enterprise container ecosystem and the cryptocurrency ecosystem. Sean joins the show to discuss his time in the Docker ecosystem, his new company Fortmatic, and his perspective on how to build tools for developers. Someday there will be hundreds of thousands of developers building applications around cryptocurrencies, just like people use cloud computing today. The road to getting there is unclear, and Sean provides useful insights and predictions for the future.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Computational integrity is a property that is required for financial transactions on the Internet. Computational integrity means that the output of a certain computation is correct. If I deposit money into my bank, my bank sends me a number that represents the new balance in my account. I assume that the number they have sent me is correct. The bank could be lying to me–maybe this bank is not actually trustworthy. But I use a bank with a good reputation. If the bank stole money from its users, it would quickly go out of business. Therefore, I feel safe by trusting a bank with my money, because the bank needs to maintain its reputation. The problem with reputation-based systems is that they are opaque. It’s not easy for us to audit the bank and prove the bank actually has the money that it claims to have. Most of the time, the reputation-based systems work fine. But occasionally, we have catastrophic events–think of the 2008 financial crisis, or the Bernie Madoff financial scandal. These circumstances would have been avoided if the financial institutions could have been continuously audited for their solvency. With blockchains and cryptocurrencies, we now have tools that allow us to maintain computational integrity without the opaque systems of reputation. We no longer have to trust a central authority–we can verify computational integrity with math. Eli Ben-Sasson is a co-founder and chief scientist at StarkWare Industries, a company that is bringing zero-trust technology to market. Implementations of zero-trust technology include zk-STARKs, zk-SNARKs, and bulletproofs. StarkWare is focused on the application of zk-STARKs, which can be used to improve scalability and privacy. Eli joins the show to discuss the topic of computational integrity, and how STARKs can be used to provide scalable, secure infrastructure to blockchain applications.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Collaboration on the Internet creates innovation. New inventions, new art, and new products–built by people working together on the Internet. FindCollabs is a product we have been working on to enable people to find and collaborate with each other. If you want to try it out, you can go to FindCollabs.com. FindCollabs is for finding people to create your projects with, and getting those projects built. Whether you are a programmer, a writer, a musician, a game designer, an actor, a videographer, or a project manager–you can find people to collaborate with. I love to work on so many different types of projects, and I love to collaborate with other people. For me, the best way that I learn new skills is by building things. I like to write music, create software, make podcasts, and build businesses. I like to create all the time. For big projects, it’s easier to build your project with a team. Finding team members can be very hard. I value other people who are creative and reliable. FindCollabs lets you find and invite people to your projects, so that you can put together a team to build your project. FindCollabs also has a reputation system. When you work on a project, your collaborators rate you. As you make contributions to projects, you show the FindCollabs community that you are reliable and productive–and other people will want to work with you because of that. The only way to build huge, ambitious projects as a team is for people to trust each other. If you are unreliable, people will not want to work with you. When you join a project on FindCollabs, you are committing to doing work that will add value to that project. If you like to build projects and be creative, you might like FindCollabs. To get started, you can go to FindCollabs.com, log in, and post a project. Or join someone else’s project. If you are ever confused about anything, you can always send me an email. We are sponsoring a series of hackathons on FindCollabs. These hackathons are for anyone with a creative project–whether you want to make a music video, a virtual reality game, an acoustic guitar song, a cryptocurrency whitepaper, a mobile app, a commercial–anything creative. Our first hackathon starts today, March 3rd 2019, and ends at 11:59 PM PST on Saturday March 16th. On March 17 2019 we will announce the winners of the first hackathon, and send them emails. We will also announce the details of the second hackathon. Prizes In the first hackathon, the prizes are not very big. But they will get bigger over time. If you like the idea of FindCollabs, it might benefit you to get involved now, so that you can build your reputation and find better people to work with in the future. 1st place: $500 divided evenly among the winning team; SE Daily hoodies for each member of the team 2nd place: SE Daily hoodies for each member of the team Most valuable feedback on the product: SE Daily Towel Most helpful community member award: SE Daily Old School Bucket Hat The FindCollabs hackathons will be judged by a panel of investors, entrepreneurs, podcasters, artists, and technologists. We will announce the judges of the first hackathon in the next few days. These judges will be voting based on which projects they like the most. Every project on the FindCollabs site before 11:59 PM PST on Saturday March 16th will be entered to win the contest. To find our detailed terms and conditions, go to findcollabs.com/terms. Thanks for taking the time to read through this post. If you get a chance, check out FindCollabs and feel free to send me feedback. I’d love to know what you think, and any suggestions you have.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Internet has transformed humanity. The Internet is the result of a long series of innovations from military, academia, business, and the open source community. In his book, How The Internet Happened: From Netscape to the iPhone, Brian McCullough tells the story of the last 25 years of Internet development through the lens of companies like ebay, Amazon, Google, and Apple. Whereas other books have focused on the trajectory of these individual companies, Brian explains how innovations in one company often lead to success in another. Without the lessons from Napster, we might not have Spotify. Without the trust model pioneered by ebay, we would not have marketplaces like Airbnb. Brian is also the host of The Internet History Podcast and the Techmeme Ride Home podcast. In The Internet History Podcast, Brian interviews entrepreneurs and engineers who were firsthand witnesses to the developments that led to our modern Internet, including early employees at Amazon, Tesla, and TheGlobe.com. In his other podcast, the Techmeme Ride Home, Brian gives a daily overview of the day’s Internet news. Through his podcasts about the Internet’s past and present, Brian has also accumulated an intuition about the future. He joins the show to discuss his book, the art of podcasting, and the historical lessons of technology.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Advertising fraud steals billions of dollars every year. BuzzFeed reporter Craig Silverman reports on advertising fraud and its impact on the Internet. In one investigation, Craig uncovered a mobile advertising fraud scheme in which four people stole millions of dollars (perhaps as much as $75 million or even $750 million) by serving advertisements to automated users on mobile apps. The scheme worked as follows: This scheme was easy to pull off. It did not require much sophistication in terms of engineering or business skills. If a group of four people can generate tens of millions of dollars, how much ill-gotten capital is being generated by large corporations that are deeply involved in the advertising market? Craig’s article went viral, and he has followed it up with other pieces about ad networks, fraud investigations by Google, and the potential for mobile apps to be used for large scale surveillance of Americans by the Chinese. Craig is the most dedicated reporter covering advertising fraud today. His work is invaluable because he is asking difficult questions about the economics of our Internet. As we discuss in the episode, there is currently no effective automated means of detecting a bot from a human on the internet. We have also discussed this in detail on previous episodes about ad fraud, the advertising industry, advertising analytics, and the techniques of ad fraud. Ad fraud is not the fault of any one party. It is an emergent result of the way that our Internet is set up. It is as hard to imagine a world without advertising fraud as it is to imagine a world without email spam.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Podcast listeners usually find out about a new podcast in one of two ways: either a friend recommends that podcast or the Apple podcast charts rank that new podcast highly. The Apple podcast charts are created using an algorithm that is not public. Many people believe that the chart ranking of a podcast is based on the number of podcast subscribers, the number of podcast downloads, and the reviews that are written about the podcasts on iTunes. Jack Rhysider is the host of Darknet Diaries, a podcast about the dark and strange elements of the Internet. Darknet Diaries is told in a high quality, narrative audio format. Jack is a security engineer with a deep understanding of technology, and has been blogging for a long time. As Jack has built a following with his podcast, he has spent more time looking at the iTunes podcast charts. He has seen the rank of Darknet Diaries increase–but he has seen the rank of other podcasts increase much faster. Some of these podcasts have low quality content. The audio quality is poor, the host is unprepared–these are the kinds of podcasts you would listen to once, and never subscribe to. And yet, numerous podcasts with low quality were somehow able to game the rankings and make it to the top of the charts. In episode 27 of Darknet Diaries, Jack investigated the phenomenon of fraudulent podcast chart manipulation. It was one of my favorite podcast episodes ever (and this is coming from someone who has listened to a lot of podcasts). The investigation went to several unexpected places, but Jack did solve the riddle of how low quality podcasts climb the iTunes charts. Jack joins the show to talk about fraudulent–and the broader implications of the fake Internet. Today’s episode is a simple example of how easily Internet platforms can be gamed–for a deeper dive into the fake Internet, listen to our past episodes on advertising fraud, or tomorrow’s episode with ad fraud investigative journalist Craig Silverman, which I am very excited about.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Many devices in our world are not “smart.” Air conditioners, electric guitars, power outlets, and factory conveyor belts, just to name a few. There are exciting software applications that we could build around these devices, but we need to be able to interface with them programmatically. We need to be able to know the state of these devices. We need to be able to save that state, and then we can use that state data to perform actions that change the state of those devices. To make these devices smart, we can use a microcontroller, a small device with a constrained amount of CPU, memory, and I/O. Device data can be sent to the cloud or processed locally, and that data can be used perform predictive maintenance, or create machine learning models, or create simple dashboards so human operators can understand the state of their hardware. Dirk Didascalou is the VP of Internet of Things with Amazon Web Services. Dirk joins today’s show to discuss the strategy and philosophy of the AWS Internet of Things set of tools. We talk about a wide-ranging set of topics–including IoT security, edge deployments, and machine learning.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Edge computing refers to computation involving drones, connected cars, smart factories, or IoT sensors. Any software deployment that is not a large centralized server installation could qualify as an edge device–even a smartphone. Today, much of our heavy computation takes place in the cloud–a set of remote data centers some distance away from our client devices. For many use cases, this works fine. But there are a growing number of use cases with lower latency and higher bandwidth requirements at the edge. A simple example is video. Let’s say you want to record a video stream, and detect people in that video stream in real time. Based on who those people in the video stream are, you want to do different things–maybe you want to send them a text message, or report to the police that a dangerous person has entered the premises. This video stream could be captured by a drone, or by a smart car, or by a video camera mounted somewhere. Where is the video stream getting stored? Where is the machine learning model running? How do you deploy new machine learning models to the operating system with the machine learning model? This is a simple example, and there are many open questions as to how to best solve such a problem. With the increased resource constraints at the edge, there is a need for new hardware and software to power these edge applications. This led to the creation of LF Edge, a new open source group under the Linux Foundation. The goal of LF Edge is to build an open source framework for the edge. Arpit Joshipura is the general manager of networking, orchestration, edge computing, and IoT with the Linux Foundation. He joins the show to describe the state of edge computation, and the mission of LF Edge. This episode was exciting for several reasons. After seeing the rise of Kubernetes for container orchestration, we know that a popular open source technology that solves a widespread problem can have dramatic influence on the software world. And when multiple large companies get involved in that open source project, it can gain traction quite quickly. Edge computing has a large set of unanswered questions, but telecom providers like AT&T and large infrastructure companies like Dell EMC are getting heavily involved with the Linux Foundation Edge group. This represents a significant expansion of the open source model, and a suggestion of further investment into open source projects in the near future.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

React Native allows developers to build native applications for iOS and Android using components written in the React JavaScript framework. These ReactJS components render to native application code by going over a JavaScript bridge, a message bus that communicates between JavaScript code and native iOS or Android runtimes. For most mobile application use cases, React Native works well. But in some cases, the platform suffers from performance issues due to the functionality of the JavaScript bridge. For example, mobile games with high demands on graphics, networking, and fast real-time updates to the UI can stutter when using React Native. To address the performance issues of React Native, the core team working on React Native at Facebook is rearchitecting the React Native runtime within a project called Fabric. Fabric consists of changes to the threading model, the data handling system, and the JavaScript bridge. Chris Severns and Lee Johnson work at G2i, a group of React and React Native specialists. Chris and Lee join the show to discuss the rearchitecture, including the engineering history of React, the technical debt within the React project, and the vision that the React team has for the future. We also discuss Google’s Flutter project, a cross-platform native framework with a different architectural model than React Native.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the early days of YouTube, there were scalability problems with the MySQL database that hosted the data model for all of YouTube’s videos. The state of the art solution to scaling MySQL at the time was known as “application-level sharding.” To scale a database using application-level sharding, you break up the database into shards–disjoint regions of data. When you want to query the database, you need know which shard to query. In your application code, you have to issue the query to a specific shard. The solution of application-level sharding does scale your database. But the downside is that every application that interfaces with the database now has to include code that is aware of the sharding schema. If you are an application engineer, you don’t want to have to worry about the way that the database is sharded, because it adds significant complexity to your code. The engineers at YouTube decided to fix this problem with a project called Vitess. Vitess abstracts away the details of sharding by orchestrating reads and writes across the distributed database. In a previous episode, we covered the architecture, read and write path, and the story of Vitess in detail. In today’s episode, Jiten Vaidya and Dan Kozlowski of PlanetScale Data join the show to give their perspective on MySQL scalability, and their work taking Vitess to market as a solution to scaling relational databases.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Zoox is a full-stack self-driving car company. Zoox engineers work on everything a self-driving car company needs, from the physical car itself to the algorithms running on the car to the ride hailing system which the company plans to use to drive around riders. Since starting in 2014, Zoox has grown to over 500 employees. Ethan Dreyfuss is a software infrastructure engineer at Zoox. He joins the show to discuss scaling an engineering team for self-driving. Machine learning was a big part of our conversation, because there are so many different approaches that an engineering team can take when it comes to machine learning for cars. Can you take computer vision algorithms from academic papers and apply them to cars? Can you use the computer vision APIs from the cloud providers for anything useful? What about physical world mapping companies like Mapillary? How do you do data labeling, and data management? And how do you manage the interactions across the stack, from mechanical engineering to user interface design? We touched on some of these areas, but barely scratched the surface of the self-driving car domain.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

DoorDash is a food delivery company where users find restaurants to order from. When a user opens the DoorDash app, the user can search for types of food or specific restaurants from the search bar or they can scroll through the feed section and look at recommendations that the app gives them within their local geographic area. Recommendations is a classic computer science problem. Much like sorting, or mapping, or scheduling, we will probably never “solve” recommendations. We will adapt our recommendation systems based off of discoveries in computer science and software engineering. One pattern that has been utilized recently by software engineers in many different areas is the “word2vec”-style strategy of embedding entities in a vector space and then finding relationships between them. If you have never heard of the word2vec algorithm, you can listen to the episode we did with computer scientist and venture capitalist Adrian Colyer or listen to this episode in which we will describe the algorithm with a few brief examples. Store2vec is a strategy used by DoorDash to model restaurants in vector space and find relationships between them in order to generate recommendations. Mitchell Koch is a senior data scientist with DoorDash, and he joins the show to discuss the application of store2vec, and the more general strategy of word2vec-like systems. This episode is also a great companion to our episode about data infrastructure at DoorDash.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The nature of software projects is changing. Projects are using a wider variety of cloud providers and SaaS tools. Projects are being broken up into more git repositories, and the code in those repositories are being deployed into small microservices. With the increased number of tools, repositories, and deployment targets, it can become difficult to manage software policy. “Policy” defines how different parts of an application can behave. Which parts of your application can access an Amazon S3 bucket? Which parts of your application can communicate with the authentication microservice? Which developers are allowed to push a new build to production? Shimon Tolts is the CTO and co-founder of Datree, a platform for policy enforcement and code compliance. He joins the show to talk about continuous delivery, configuration management, and policy enforcement. He also explains the motivation for his company Datree, which performs analysis across a user’s GitHub repo to map the committers, code components, and repositories.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ethereum allows developers to run decentralized applications. But the tooling for building and managing those decentralized applications is immature. Experienced software engineers have difficulty getting started with writing Ethereum applications because the stack of tools is so unfamiliar and different than traditional software tools. Whether or not Ethereum itself succeeds, developers in the future will probably be building some decentralized apps. We will be treating money as a first-class citizen and architecting software that transfers financial value as easily as we transmit JavaScript today. “Web3” will be a world in which many more software applications will be possible. As we move towards Web3, many new tools will be built. Web2 was the result of Ruby on Rails, Amazon Web Services, the iPhone, and other software tools that made it easier to deploy web servers and consume Internet services. In the world of Web2, we saw the birth of Airbnb, Uber, and Netflix. In the world of Web3, we will see new types of gig economy apps, sharing economy platforms, and social networks. These new applications will arrive gradually as the tooling improves, and makes it easier for developers to hack together businesses and side projects built on cryptocurrencies. Brian Soule is the founder or Ethsimple, a company that makes tools for Ethereum developers. Brian joins the show to talk about the state of cryptocurrencies, the tooling that developers have access to, and his company Ethsimple. We cover high-level ideas, such as Bitcoin maximalism and also talk about some more technical areas of the Ethereum ecosystem, such as the Ethereum Name Service.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A Kubernetes cluster presents multiple potential attack surfaces: the cluster itself, a node running on the cluster, a pod running in the node, a container running in a pod. If you are managing your own Kubernetes cluster, you need to be aware of the security settings on your etcd, your API server, and your container build pipeline. Many of the security risks of a Kubernetes cluster can be avoided by using the default settings of Kubernetes, or by using a managed Kubernetes service from a cloud provider or an infrastructure company. But it is useful to know about the fundamentals of operating a secure cluster, so that you can hopefully avoid falling victim to the most common vulnerabilities. Liz Rice wrote the book Kubernetes Security with co-author Michael Hausenblas. Liz works at Aqua Security, a company that develops security tools for containerized applications. In today’s show, Liz gives an overview of the security risks of a Kubernetes cluster, and provides some best practices including secret management, penetration testing, and container lifecycle management.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud computing has been popular for less than twenty years. Large software companies have existed for much longer. If your company was started before the cloud became popular, you probably have a large, data center on your companies premises. The shorthand term for this software environment is “on-prem”. Deploying software to your own on-prem servers can be significantly different than deploying to remote servers in the cloud. In the cloud, servers and resources are more standardized. It is often easier to find documentation and best practices for how to use cloud services. Many of the software vendors who got started in the last decade created their software in the cloud. For example, Readme.io makes it easy for companies to create hosted documentation. Their early customers were startups and other cloud-native companies. All of those companies were happy to consume the software in the cloud. As time went on, Readme found that other customers wanted to use the Readme product as a self-hosted, on-prem service. Readme needed to figure out how to deploy their software easily to the “on-prem” environment. It turns out that this is a common problem. Software vendors who want to sell to on-prem enterprises must have a defined strategy for making those deployments to on-prem infrastructure–and those deployments are not always easy to configure. Replicated is a company that allows cloud-based software to easily deploy to on-prem infrastructure. Grant Miller is the founder of Replicated and he joins the show to discuss on-prem, cloud, and the changing adoption patterns of enterprise software companies.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Uber manages the car rides for millions of people. The Uber system must remain operational 24/7, and the app involves financial transactions and the safety of passengers. Uber infrastructure runs across thousands of server instances and produce terabytes of monitoring data. The monitoring data is used to understand the health of the software systems as well as relevant business metrics, such as driver efficiency, daily revenues, and user satisfaction. Uber adopted the Prometheus monitoring system to manage their monitoring data. Prometheus regularly scrapes metrics across infrastructure to gather time series data about the state of everything across Uber. As the usage of Prometheus has grown within the company, Uber has had to figure out how to scale their monitoring platform. M3 is a monitoring system built at Uber to scale Prometheus and provide a platform that can effectively scale the data storage as well as the query serving. Rob Skillington is a staff software engineer at Uber, and he joins the show to talk about monitoring at Uber–from the requirements of the system to the implementation of M3. At Uber, M3 powers dashboards, ad-hoc queries, and alerting. M3 was open sourced to give other users access to a scalable Prometheus solution. In a previous episode with Brian Boreham, we discussed one strategy for scaling Prometheus. Today’s episode covers another scalability solution, with M3.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data infrastructure is advancing beyond the days of Hadoop MapReduce, single-node databases, and nightly reporting. Companies are adopting modern data warehouses, streaming data systems, and cloud-specific data tools like BigQuery. Every company with a large amount of data wants to aggregate that data into a data lake and make the data available to developers. All of this data can be used to power machine learning models which can potentially improve every area within a company where they have historical data. “Data pipeline” is a term used to describe the process of preparing data, building machine learning models, deploying those models, and tracking the results of those models. Pachyderm is a company and open source project that is focused on deployment, management, and scalability of data pipelines. Pachyderm allows developers to version data, track the state of data sets, backtest machine learning models, and collaborate on data. It also tackles the very hard problem of machine learning auditability. Joe Doliner is the CEO of Pachyderm and joins the show to discuss his experience building Pachyderm over the last five years. Data infrastructure has changed a lot in five years, and the world has moved in a direction that has benefitted Pachyderm, with more infrastructure moving to containers and more data teams advancing beyond a world of just Hadoop MapReduce. In today’s show, Joe talks about modern infrastructure, data provenance, and the long-term vision of Pachyderm.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Infrastructure software is having a renaissance. Cloud providers offer a wide range of deployment tools, including virtual machines, managed Kubernetes clusters, standalone container instances, and serverless functions. Kubernetes has standardized the container orchestration layer and created a thriving community. The Kubernetes community gives the cloud providers a neutral ground to collaborate on projects that benefit everyone. The two forces of cloud providers and Kubernetes have led to massive improvements in software quality and development practices over the last few years. But one downside of the current ecosystem is that many more developers learn how to operate a Kubernetes cluster than perhaps is necessary. “Serverless” tools are at a higher level than Kubernetes, and can improve developer productivity–but a risk of using a serverless tool is the potential for lock-in, and a lack of portability. Knative is an open-source serverless platform from Google built on top of Kubernetes. Ville Aikas is a senior staff engineer at Google who has worked at the company for eleven years. With his experience, Ville brings a rare perspective to the subjects of Kubernetes, serverless, and the infrastructure lessons of Google. Ville joins the show to discuss Knative, the motivation for building it, and the future of “serverless” infrastructure.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Virtualization software allows companies to get better utilization from their physical servers. A single physical host can manage multiple virtual machines using a hypervisor. VMware brought virtualization software to market, creating popular tools for allowing enterprises to deploy virtual machines throughout their organization. Containers provide another improvement to server utilization. A virtual machine can be broken up into containers, allowing multiple services to run within a single VM. Containers proliferated after the popularization of Docker, and the Kubernetes open source container orchestration system grew to be the most common way of managing the large numbers of containers that were running throughout an organization. As Kubernetes has risen to prominence, software infrastructure companies have developed Kubernetes services to allow enterprises to use Kubernetes more easily. VMware’s PKS is one example of a managed Kubernetes service. Brad Meiseles is a senior director of engineering at VMware with more than nine years of experience with the company. He joins the show to discuss virtualization, Kubernetes, containers, and the strategy of a large infrastructure provider like VMware.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Real estate is an asset that is not straightforward to invest in. Real estate can generate excellent returns for investors, but can require much more time and expertise than stocks. Cadre is a company that allows users to invest in real estate more easily and intelligently. Cadre provides users with lots of data about potential investments and enables investments in those opportunities within the platform. Leonid Movsesyan is the head of engineering at Cadre and joins the show to talk about the problems being solved by the company in areas of product development, infrastructure engineering, hiring, and data science. To build a platform for evaluating real estate investments, Cadre ingests and merges lots of data sets–some public and some private. This gives investors a detailed picture of the value of investments. Fintech Daily is a new podcast from Software Engineering Daily covering payments, cryptocurrencies, trading, and the intersection of finance and technology. We are looking for volunteer hosts for Fintech Daily, and if you are interested in working with us to conduct interviews, send an email to host@fintechdaily.co. You can find the podcast on iTunes, Google, and everywhere else, and if you are interested in hosting, don’t hesitate to reach out.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RocksDB is a storage engine based on the log structured merge tree data structure. RocksDB was developed at Facebook to provide a tool for embedded databases. The code for RocksDB is a fork of LevelDB, an embedded database built by Google for the Chrome browser. Every database has a storage engine. The storage engine is the low level data structure that manages the data in the database. RocksDB is widely used in database applications where a log structured merge tree is preferable to a b-tree. These tend to be write-heavy workloads. In past shows, we have explored applications of RocksDB in our coverage of databases like TiDB, data intensive applications like Smyte, and data platforms like Rockset. In today’s episode, Dhruba Borthakur and Igor Canadi join for a deep dive into how RocksDB works. Dhruba was the original creator of RocksDB, and Igor is a former Facebook engineer who worked on RocksDB in its early days. Both Dhruba and Igor work at Rockset. We talk about the log structured merge tree, discuss why an LSM has higher write throughput than storage engines based on a b-tree, and evaluate some of the use cases for RocksDB.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Hashicorp was founded seven years ago with the goal of building infrastructure tools for automating cloud workflows such as provisioning, secret management, and service discovery. Hashicorp’s thesis was that operating cloud infrastructure was too hard: there was a need for new tools to serve application developers. Hashicorp founders Mitchell Hashimoto and Armon Dadgar began releasing open source tools to fulfill their vision of better automation. Terraform, Vagrant, Consul, and other tools created by Hashicorp gained popularity, and Hashicorp began iterating on their business model. Today, Hashicorp makes money by offering enterprise features and support to enterprises such as Pinterest, Adobe, and Cruise Automation. Over the last seven years, enterprise software infrastructure has changed rapidly. First, enterprises moved from script-based infrastructure automation to container orchestration frameworks. Then, the container orchestration world consolidated around Kubernetes. Today, large enterprises are rapidly adopting Kubernetes with a mix of public cloud and on-prem vendors. At the same time, these enterprises are also becoming more willing to consume proprietary tools from the public cloud providers. Hashicorp has benefitted from all of this change. Their different tools fit into a variety of workflows, and are not closely coupled with any particular cloud provider or platform solution. Armon and Mitchell join today’s show to discuss the business model and the product philosophy of Hashicorp. We also touch on service mesh, zero trust networking, and their lessons from the container orchestration wars.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Tyler Cowen’s book Stubborn Attachments outlines a framework that individuals can use to make decisions grounded in economic philosophy. In his previous books, Tyler examined recent economic history. Stubborn Attachments gives his perspective for navigating the future. Tyler is a professor of economics at George Mason University. He is also the host of Conversations with Tyler, a podcast that includes guests such as Ethereum creator Vitalik Buterin, Stripe co-founder Patrick Collison, and Coinbase CTO Balaji Srinivasan. Tyler blogs frequently at Marginal Revolution. Tyler’s previous appearance on Software Engineering Daily centered around his earlier books, including The Complacent Class. In this episode, Tyler describes the philosophy outlined in Stubborn Attachments, then we discuss how his philosophy relates to software engineering, podcasting, and economics. To find all 900 of our old episodes, including past episodes with writers, entrepreneurs, and venture capitalists, check out the Software Engineering Daily app in the iOS and Android app stores. Whether or not you are a software engineer, we have lots of content about technology, business, and culture. In our app, you can also become a paid subscriber and get ad-free episodes–and you can have conversations with other members of the Software Engineering Daily community.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Artificial intelligence is reshaping every aspect of our lives, from transportation to agriculture to dating. Someday, we may even create a superintelligence–a computer system that is demonstrably smarter than humans. But there is widespread disagreement on how soon we could build a superintelligence. There is not even a broad consensus on how we can define the term “intelligence”. Information technology is improving so rapidly we are losing the ability to forecast the near future. Even the most well-informed politicians and business people are constantly surprised by technological changes, and the downstream impact on society. Today, the most accurate guidance on the pace of technology comes from the scientists and the engineers who are building the tools of our future. Martin Ford is a computer engineer and the author of Architects of Intelligence, a new book of interviews with the top researchers in artificial intelligence. His interviewees include Jeff Dean, Andrew Ng, Demis Hassabis, Ian Goodfellow, and Ray Kurzweil. Architects of Intelligence is a privileged look at how AI is developing. Martin Ford surveys these different AI experts with similar questions. How will China’s adoption of AI differ from that of the US? What is the difference between the human brain and that of a computer? What are the low-hanging fruit applications of AI that we have yet to build? Martin joins the show to talk about his new book. In our conversation, Martin synthesizes ideas from these different researchers, and describes the key areas of disagreement from across the field. To find all 900 of our old episodes, including past episodes with authors and artificial intelligence researchers, check out the Software Engineering Daily app in the iOS and Android app stores. Whether or not you are a software engineer, we have lots of content about technology, business, and culture. In our app, you can also become a paid subscriber and get ad-free episodes–and you can have conversations with other members of the Software Engineering Daily community.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Mars is a cold, inhospitable planet far from earth. It presents one of the most complex challenges faced by engineers: how can we create a new world? To create a new world, first we have to get there. We can build new rockets with improved propulsion systems. We can build ships that allow us to survive the long, grueling trip from Earth to Mars. We can build robots that will help us construct our new home. And this is just the beginning. Mars could be warmed, and could develop a hydrologic cycle like the system of clouds and oceans on earth. Mars could be a place for new ideas and new cultures, unfettered by the conventions of Earth. Mike Solana is the host of Anatomy of Next, a podcast about technologies and philosophies of the future. He’s also a vice president at Founder’s Fund. In a previous episode, Mike joined the show to talk about artificial intelligence, genetics, and robotics. Today, we discuss Mars. The latest season of Anatomy of Next explores the science that is bringing us closer to exploring other planets. On his podcast, Mike speaks with engineers, researchers, and entrepreneurs about the state of the art of space technology–as well as the challenges that remain unsolved. Mike returns to the show to discuss this dream of a new world. Why should we go to Mars? And why should the software engineers listening to this podcast even care? To find all 900 of our old episodes, including past episodes with venture capitalists, futurists, and philosophers, check out the Software Engineering Daily app in the iOS and Android app stores. Whether or not you are a software engineer, we have lots of content about technology, business, and culture. In our app, you can also become a paid subscriber and get ad-free episodes–and you can have conversations with other members of the Software Engineering Daily community.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Social media has transformed our lives. It has also transformed how wars are fought. P.W. Singer’s new book “Likewar: The Weaponization of Social Media” describes the far-reaching impact of social media on the tactics and strategies used by military, business, and everyday citizens. We have all read about stories such as Russian bots and Cambridge Analytica, but Likewar covers many more cases that are surprising and mildly frightening. From the Gaza Strip to the streets of Chicago to Taylor Swift’s Instagram feed, Likewar describes just how pervasive the effect of social media has been on warfare. Likewar also provides historical context. For software engineers, the repurposing of social media as a weapon is disconcerting. Many of us are working on products with a social networking component. Does this make us complicit in building weapons? We can find some reassurance in the fact that this has happened before: from the newspaper to the television, every new invention has been used repurposed for war. In a war, a new piece of technology always presents a new vector to gain an advantage in a conflict. Because the stakes are so high in a war, there is a large incentive to find creative ways to use technology to undermine your adversaries and to help your allies. P.W. Singer has written about robotics, cybersecurity, and modern warfare for a decade. In a previous episode, we discussed subjects like Stuxnet, drones, and social media manipulation. In today’s show, P.W. returns to talk about his book Likewar: The Weaponization of Social Media.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Infrastructure software can be a great business. An infrastructure software company sells core technology to a large enterprise such as a bank or insurance company. This software has near zero marginal cost and generates a large annuity for the infrastructure software company. Once a bank has purchased your infrastructure software, the bank is likely to renew every year and never remove the software. Selling infrastructure software is like selling concrete or steel, except the software is cheaper to produce, easier to distribute, and generates an annuity rather than being a one-time sale. The fundamental economics of enterprise infrastructure software are extremely appealing, and every year more businesses enter the space–but few businesses ever leave. If you are starting an infrastructure software company, you can expect a complex battle for market share. There is no easy trick to get it into the hands of your target customer. Martin Casado studied computer science at Stanford before founding Nicira, a company that pioneered software-defined networking and virtualization technology. In 2012, Nicira sold to VMware for $1.26 billion. Martin now works as a general partner at Andreessen Horowitz. Martin writes about the modern strategies of building a successful infrastructure software company. He describes two methods of selling into an enterprise: bottoms-up and top-down. In a bottoms-up model, engineers within an enterprise start using your product to solve a well-defined problem, such as API management. As more and more employees within the organization start to use your product, you can begin to engage the enterprise about becoming a paying customer for your product. Since the enterprise is already using your product, the sales conversation is much easier. In the top-down model, you engage the CIO, CEO, or CTO directly and try to convince them that your product is worth paying for. When the senior leadership of a bank buys into your product idea, you can count on that senior leadership to convince their developers to use your product within the bank. It is a rare occurrence that your infrastructure software company will be able to fit cleanly into either of these models–bottoms-up or top-down. More often, there will be some bottoms-up usage, and some top-down buy-in for your product. But you will have to evangelize the product on all fronts. You will have to convince both the engineers and the senior leadership. Your product probably won’t speak for itself. You will have to develop expertise in sales, marketing, and consultancy. And in many cases, you might end up in an unending chasm. The unending chasm describes a mode in which an infrastructure company must function as both a product company and a consultancy. Your consultancy is necessary to integrate your product into the enterprise, and ensure that your software actually gets used. But it reduces the appealing economics of a pure software company. The unending chasm does not prevent you from being successful. Companies who have had very successful IPOs remain in the unending chasm. But it’s useful to know whether you are heading for an unending chasm–or if you are already in one. Martin Casado joins the show today for a discussion of product development, software engineering, and go-to-market strategy. To find all 900 of our episodes, including past episodes with a16z partners, check out the Software Engineering Daily app in the iOS and Android app stores. Whether or not you are a software engineer, we have lots of content about technology, business, and culture. In our app, you can also become a paid subscriber and get ad-free episodes–and you can have conversations with other members of the Software Engineering Daily community. Image Source

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When TensorFlow came out of Google, the machine learning community converged around it. TensorFlow is a framework for building machine learning models, but the lifecycle of a machine learning model has a scope that is bigger than just creating a model. Machine learning developers also need to have a testing and deployment process for continuous delivery of models. The continuous delivery process for machine learning models is like the continuous delivery process for microservices, but can be more complicated. A developer testing a model on their local machine is working with a smaller data set than what they will have access to when it is deployed. A machine learning engineer needs to be conscious of versioning and auditability. Kubeflow is a machine learning toolkit for Kubernetes based on Google’s internal machine learning pipelines. Google open sourced Kubernetes and TensorFlow, and the projects have users AWS and Microsoft. David Aronchick is the head of open source machine learning strategy at Microsoft, and he joins the show to talk about the problems that Kubeflow solves for developers, and the evolving strategies for cloud providers. David was previously on the show when he worked at Google, and in this episode he provides some useful discussion about how open source software presents a great opportunity for the cloud providers to collaborate with each other in a positive sum relationship.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a user interacts with an application to order a ride with a ridesharing app, the data for that user interaction is written to a “transactional” database. A transactional database is a database where specific rows need to be written to and read from quickly and consistently. Speed and consistency are important for applications like a user ordering a car, and riding around in that car, because the user’s client is frequently communicating with the database to update their session. Other applications of a transactional database would include a database that backs a messaging system, a banking application, or document editing software. The data from a transactional database is often reused in “analytic” databases. An analytic database can be used for performing large scale analysis, aggregations, averages, and other data science queries. The requirements for an analytic database are different from a transactional database because the data is not being used for an active user session. To fill the data in an analytic database, the transactional data gets copied from the transactional database in a process called ETL. The separation of the transaction data store from the analytic data store causes problems for data engineering. To address these problems, some newer databases combine transactional and analytic functionality in the same database. These databases are often called “NewSQL”. TiDB is an open source database built on RocksDB and Kubernetes. TiDB is widely used in China by high volume applications such as bike sharing and massively multiplayer online games. Kevin Xu works at PingCAP, a company built around TiDB. He joins the show to talk about modern databases, distributed systems, and the architecture for TiDB.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

React, Vue, and Angular are the most popular frontend JavaScript frameworks. Each of these frameworks lets frontend developers build components. A component is a high level visual abstraction that is used to compose a user interface. Frontend development has moved towards component-driven-development. At a typical technology company, a designer will put together a design file of different user interface elements, and the frontend engineer will take those UI elements and program code that can render those designs as components. As organizations have started to reuse their components and share them across the organization, the efficiency of design and frontend engineering is improving. User interface is gaining more of an emphasis with organizations and new tools are allowing frontend engineers and designers to work together more productively. One of these tools is Storybook, a system for sharing components and the code that renders those components. Zoltan Olah joins the show to talk about Storybook, and his company Chroma. Chroma is building tools to allow design-driven teams to work more effectively. We talked about how the relationship of designers and frontend engineers has some resemblance to the relationship between “dev” and “ops” before the DevOps movement. There are some frictions in the process of moving between design and engineering implementation, and in talking to Zoltan, I got an understanding for how much the UI layer could improve through better tooling.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Netflix has thousands of service instances communicating with each other. When a Netflix client on a smartphone makes a request for a movie, that request hits Netflix’s backend, where the request is fulfilled by a chain of requests through different services. Services and clients communicate using several different interaction patterns. A service might send a single request and expect a single response. Or it might fire and forget, not expecting a response. A service also might send a single request and expect a stream of messages to be sent back over the network. In a highly interactive application like Netflix, there is a frequent use of “streams” of data. RSocket is a protocol that makes reactive streams easier to work with. Ryland Degnan is the CTO of Netifi, and he joins the show to discuss reactive streams and service-to-service networking. Ryland worked at Netflix on the Edge Platform team for four years, and he shares his experience working at Netflix, the challenges of networking at scale, and the company he is building around RSocket.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Prometheus is an open source monitoring system and time series database. Prometheus includes a multi-dimensional data model, a query language called PromQL, and a pull model for gathering metrics from your different services. Prometheus is widely used by large distributed systems deployments such as Kubernetes and Cloud Foundry. Prometheus gathers metrics from your services by periodically scraping those services. Those metrics get gathered, compressed, and stored onto disk for querying. But Prometheus is designed to store all of its records on one host in one set of files–which limits the scalability and availability of those metrics. Cortex is an open source project built to scale Prometheus. Cortex effectively shards Prometheus by parallelizing the “ingestion” and storage of Prometheus metrics. Cortex can take metrics from multiple Prometheus instances and store them across a distributed NoSQL database like DynamoDB, BigTable, or Cassandra. Bryan Boreham is an engineer at Weaveworks, where he works on deployment, observability, and monitoring tools for containers and microservices. He wrote much of the code for Cortex, and we met up at KubeCon North America to talk about the motivation for creating Cortex, the broader landscape of Kubernetes monitoring, and other approaches to scaling Prometheus.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a developer provisions a cloud server, that server is called an “instance”. These instances can be used for running whatever workload a developer has, whether it is a web application, a database, or a set of containers. The cloud is cheap to get started on. New applications with few users can often be hosted on infrastructure that is less than $10 per month. But as an application grows in popularity, there is more demand for CPUs and storage. A company will start to buy more and more servers to scale up to the requirements of their growing user base. The costs of running infrastructure in the cloud will increase, and the company will start to look for ways to save money. One common method of saving money is to buy “spot instances”. A spot instance is an instance that is cheaper than “reserved instances” or “on-demand” instances. The reason that there are different instance types is because a giant cloud provider has a highly variable amount of work that is being demanded from that cloud provider. If you are in charge of AWS, you have to make sure that at any given time, you can give server resources to anyone that asks for it. Your data centers need to have physical machines that are ready to go at any time. This means that much of the time, you have server resources that are going unused. If you are a cloud provider, how can you get people to use your compute resources? You can make them cheaper. So a user can come along and buy your compute at the discounted “spot” price. But this presents a problem for the cloud provider. If you start to give away your compute at cheaper prices, and then the overall demand for your cloud resources go up once again, you are going to miss out on profits. As the cloud provider, you need to kick people off of your spot instances, so that you can take those same instances and sell them to people at the higher market prices. And this presents a problem for the user. If you buy a cheap spot instance, that instance is only available until the cloud provider decides to kick you off. You have a tradeoff between cost and availability of your instances. Because of this, spot instances are typically used only for workloads that are not mission critical–workloads that can afford to fail. Spotinst is a company that allows developers to deploy their workloads reliably onto spot instances. Spotinst works by detecting when a spot instance is going to be reclaimed by a cloud provider and re-scheduling the workload from that cloud provider onto a new spot instance. Amiram Shachar is the CEO of Spotinst. He joins the show to talk about the different types of instances across cloud providers, the engineering behind Spotinst, and how the usage of containers and the rise of Kubernetes is changing the business landscape of the cloud.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

If a business has been operating successfully for a few years, that business has accumulated a high volume of data. That data exists in spreadsheets, CSV files, log files, and balance sheets. Data might be spread across local files on a user’s laptop, databases in the cloud, or storage systems in an on-premise data center. Older businesses have more data, in more places, in more formats. Legacy systems and old batch processing jobs that have been running for years are taking data from one place and porting it to another. Every mature company needs to access and analyze the data in all of these different places–whether they are a publication with millions of readers like The Economist or a fast growing infrastructure provider like Twilio. “Business intelligence” is a term often used to describe tools for analyzing data in the form of charts, graphs, and reports. Business intelligence applications are crucial to the success of a business because they are used by everyone in an organization–whether you are a business analyst forecasting sales for the next quarter, an engineer who is determining how many servers to provision, or a CEO trying to decide what the best area of your business to focus on is. There have been several generations of business intelligence tools. Each generation of business intelligence is built for the trends and infrastructure of that generation. Looker is a more recent business intelligence tool that was built in light of several trends in software: the growth in volume of data; the growth in the number of systems that users need; the changing types of users that need to access data; and the need to share business intelligence across social workplace tools like Slack, Asana, and email. Daniel Mintz joins the show to describe his experience using business intelligence tooling and his work at Looker, as well as the landscape of business intelligence, ETL, and data engineering.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Robots are making their way into every area of our lives. Security robots roll around industrial parks at night, monitoring the area for intruders. Amazon robots tirelessly move packages around in warehouses, reducing the time and cost of logistics. Self-driving cars have become a ubiquitous presence in cities like San Francisco. For a hacker in a dorm room, or a researcher in a small lab, how do you get started with robotics? There are drones and other small options like AWS DeepRacer–but what is the equivalent of the Raspberry Pi for large, human-sized robots? Zach Allen is the founder of Slate Robotics, a company that makes large, human-sized robots that are at a low enough cost to be accessible to tinkerers, researchers, and prototype builders. Zach joins the show to talk about the state of robotics and why he started a robot company. What Zach is doing is quite hard–he is a solo founder who has bootstrapped a robotics company from scratch. He is set up in a strip mall in Missouri, where he has set up a row of 3-D printers to create the parts for his robots. He programs and assembles these robots himself. Whether you are interested in robots are thinking about starting a hardware company, this episode could be useful to you.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Netflix has petabytes of data and thousands of workloads running across that data every day. These workloads generate movie recommendations for users, create dashboards for data analysts to study, and reshape data in ETL jobs, to make it more accessible across the organization. Over the last ten years, data engineering has become a key component of what makes Netflix successful. There are many different engineering roles who interact with the data infrastructure–including data analyst, machine learning scientist, analytics engineer, and software engineer. Data engineering at Netflix has come a long way from the days of Hadoop MapReduce jobs running nightly, and generating reports of the most popular movies. As data engineering and data science has grown, the tooling has expanded. The people in different data roles at Netflix might use Apache Spark, Presto, Python, Scala, SQL, and many other applications to study data–but in recent years, there is one tool that has stood out for its ability to be distinctly useful: Jupyter Notebooks. A Jupyter Notebook lets users create and share documents that contain live code, visualizations, documentation, and many other types of components. In some ways, it is like a shareable IDE, that allows other people to see how you are working with your code and why you are making certain decisions. It is also a tool for building interactive, user-friendly applications–you can embed videos and images in a Jupyter notebook. A Jupyter Notebook stores both the code and the results together in one place. By combining code with results in one document, you can have context around why a certain result came out the way it did. Matthew Seal is a senior software engineer at Netflix, where he builds infrastructure and internal tools around Jupyter Notebooks. He joins the show to explain what problems Jupyter Notebooks solve for Netflix, and why they have quickly grown in popularity within the company. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Chinese Internet companies operate at a massive scale. WeChat has over a billion users and is widely used as the primary means of payment by urban Chinese consumers. Alibaba ships 12 million packages per day, which is four times the amount of Amazon. JD.com, a Chinese ecommerce company, has perhaps the largest production Kubernetes installation in the world. China’s rapid adoption of Internet services, combined with a large population and a growing middle class has led to the creation of Internet giants on par with the social networks, ecommerce sites, and ridesharing startups of the United States. Last November, I attended the first KubeCon China and saw firsthand how the Chinese Internet companies are using open source software to scale their infrastructure. Despite the differences between the US and China, the culture of technologists at KubeCon felt familiar. In some ways, it was just like any other Kubernetes conference that I have attended: large numbers of engineers trying to find the cutting edge of technology, and learning how to solve the problems they are facing back at the office. There were presentations on scaling databases and service meshes and machine learning on Kubernetes. Outside of these presentation halls, there were tables where you could pick up a translation device so that Chinese-only and English-only presentations could be understood by the other nationality. Dan Kohn joins the show to talk about Chinese Internet companies and how they are adopting Kubernetes. Dan is the executive director of the Cloud Native Computing Foundation, an organization within the Linux Foundation that organizes KubeCon. Before joining the CNCF, Dan worked as an entrepreneur, engineer, and executive at several technology companies.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Amazon Web Services changed how software engineers work. Before AWS, it was common for startups to purchase their own physical servers. AWS made server resources as accessible as an API request, and has gone on to create higher-level abstractions for building applications. For the first few years of AWS, the abstractions were familiar. S3 provided distributed, reliable object storage. Elastic MapReduce provided a managed Hadoop system. Kinesis provided a scalable queue. Amazon provided developers with managed alternatives to complicated open source software. More recently, AWS has started to release products that are unlike anything else. A perfect example is AWS Lambda, the first function-as-a-service platform. Other newer AWS products include Ground Station, a service for processing satellite data and AWS DeepRacer, a miniature race car for developers to build and test machine learning algorithms on. As AWS has grown into new categories, the blog announcements of new services and features have started coming so frequently that it is hard to keep track of it all. Corey Quinn is the author of “Last Week in AWS”, a popular newsletter about what is changing across Amazon Web Services. Corey joins the show to give his perspective on the growing, shifting behemoth that is Amazon Web Services–as well as the other major cloud providers that have risen to prominence. He’s also the host of the Screaming in the Cloud podcast, which you should check out if you like this episode.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Serverless computing is a technique for deploying applications without an addressable server. A serverless application is running on servers, but the developer does not have access to the server in the traditional sense. The developer is not dealing with IP addresses and configuring instances of their different services to be able to scale. Just as higher level languages like C abstracted away the necessity of a developer to work with assembly code, serverless computing gives a developer more leverage by letting them focus on business logic while a serverless platform takes care of deployment, uptime, autoscaling, and other aspects of cloud computing that are fundamental to every application. “Serverless” can several different things: backend-as-a-service products like Firebase, functions-as-a-service like AWS Lambda, and high-level APIs such as Twilio. Zeit is a deployment platform built for serverless development. In Zeit, users model a GitHub repository in terms of the functions within their application. Zeit deploys the code from those functions onto functions-as-a-service and allows you to run your code across all the major cloud providers. Guillermo Rauch is the founder of Zeit, and he joins the show to discuss his vision for the company and the platform as it looks today. Guillermo was previously on the show to discuss Socket.io, which he created.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Functions-as-a-service allow developers to run their code in a “serverless” environment. A developer can provide a function to a cloud provider and the code for that function will be scheduled onto a container and executed whenever an event triggers that function. An “event” can mean many different things. It is a signal that something has changed within your application. When you save a file to an Amazon S3 bucket, that creates an event. When a user signs up for your app, that can create an event. Functions-as-a-service are allowing people to build applications completely out of managed cloud infrastructure. Apps can be fully “serverless”, with managed databases, queueing systems, and APIs tied together by event-triggered functions. Today, there is not a consistent format for events across different applications and cloud providers. This makes it more difficult to stitch together events across these different environments. Ideally, events would be lightweight, easy to deserialize, and easy to interoperate with. The Cloud Events specification is a project within the Cloud Native Computing Foundation with the goal of creating a standard format for events. Doug Davis is the CTO for developer advocacy of containers at Microsoft. He joins the show to discuss how events and event-based programming works, and the need for a common format across cloud events.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Most applications today are either deployed to on-premise environments or deployed to a single cloud provider. Developers who are deploying on-prem struggle to set up complicated open source tools like Kafka and Hadoop. Developers who are deploying to a cloud provider tend to stay within that specific cloud provider, because moving between different clouds and integrating services across clouds adds complexity. Ben Hindman started the Apache Mesos project when he was working in the Berkeley AMPLab. Mesos is a scheduler for resources in a distributed system, allowing compute and storage to be scheduled onto jobs that can use those resources. In his time at the AMPLab, Ben collaborated with Matei Zaharia, creator of Apache Spark. Ben founded Mesosphere based off of his work on Apache Mesos, and since 2013 he has been building a company to bring it to market. In the meantime, several market forces have influenced the enterprise market. Enterprise businesses built on virtual machines and on-prem hardware are trying to migrate to containers, Kubernetes, and Spark. Cloud providers like Google and Microsoft have risen to prominence in addition to Amazon’s continued growth, and enterprises are increasingly willing to adopt multiple clouds. I spoke with Ben Hindman at Kubecon North America. Today, the company that he co-founded works to provide tools for managing these changes in infrastructure. In our conversation, we talked about the necessary mindset shifts for taking a research project and turning it into a highly successful product. We also talked about the newer trends in infrastructure–why enterprises will want multicloud deployments and how serverless APIs and backends will make the lives of developers much easier.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In a cloud infrastructure environment, failures happen regularly. The servers can fail, the network can fail, and software bugs can crash your software unexpectedly. The amount of failures that can occur in cloud infrastructure is one reason why storage is often separated from application logic. A developer can launch multiple instances of their application, with each instance providing a “stateless” environment for serving API requests. When the application needs to save state, it can make a call out to a managed cloud infrastructure product. Managed cloud databases provide a reliable place to manage application state. Managed object storage systems like Amazon S3 provide a reliable place to store files. The pattern of relying on remote cloud services does not work so well for on-prem and hybrid cloud environments. In these environments, companies are managing their own data centers and their own storage devices. As companies with on-prem infrastructure adopt Kubernetes, there is a need for ways to manage on-prem storage through Kubernetes. Saad Ali is a senior engineer at Google, where he works on Kubernetes. He is also a part of the Kubernetes Storage Special Interest Group. Saad joins the show talk about how Kubernetes interacts with storage, and how to manage stateful workloads on Kubernetes. We discuss the basics of Kubernetes storage, including persistent volumes and the container storage interface.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a user makes a request to product like The New York Times, that request hits an API gateway. An API gateway is the entry point for an external request. An API gateway serves several purposes: authentication, security, routing, load balancing, and logging. API gateways have grown in popularity as applications have become more distributed, and companies offer a wider variety of services. If an API is public, and anyone can access it, you might need to apply rate limiting so that users cannot spam the API. If the API is private, the user needs to be authenticated before the request is fulfilled. Kong is a company that builds infrastructure for API management. The Kong API gateway is a widely used open source project, and Kong is a company built around supporting and building on top of the API gateway. Marco Palladino is the co-founder and CTO of Kong. He joins the show to tell the story of starting Kong eight years ago, and how the API gateway product evolved out of an API marketplace. Marco also discusses the architecture of Kong and his vision for how the product will develop in the future–including the Kong service mesh.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Augmented reality glasses will let us walk through a world where the digital blends together with the physical. 3-D objects will be rendered and superimposed onto our field of vision, creating an environment for people to build applications we can hardly dream of today. These augmented reality glasses are probably three to five years away from being ready for consumer use. But developers are already building augmented reality applications for smartphones using Apple ARKit and Android ARCore. These augmented reality toolkits use powerful smartphone processors and computer vision to give developers simple primitives for placing and manipulating 3-D objects. Most of these AR applications are made for a single phone, and AR is useful for a single phone–for example, you could hold your phone up in front of an empty room, and see on your phone how it would look if you had an IKEA couch sitting in the middle of that room. But shared augmented reality experiences are much more exciting. Shared augmented reality can allow us to play a game of virtual basketball, both controlling the game that is synchronized between us. Shared AR would let me go to a restaurant, and create a virtual billboard in front of the restaurant that only you could see when you walked up to the restaurant and held your phone in front of you. Ubiquity6 is a company with the goal of enabling shared AR experiences. Ankit Kumar is the co-founder and CTO of Ubiquity6, and he joins the show to explain why building shared AR is a challenging technical problem. It requires building a digital model of the real world, and mapping that model to coordinates in space, so that users can reliably persist augmented reality objects that each other can see. We discuss computer vision, digital mapping, the increasing power of phone processors, and the potential of shared AR.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud providers created the ability for developers to easily deploy their applications to servers on data centers. In the early days of the cloud, most of the code that a developer wrote for their application could run on any cloud provider, whether it was Amazon, Google, or Microsoft. These cloud providers were giving developers the same Linux server that they would expect from an on-premise deployment. Early cloud applications such as Netflix, Airbnb, and Uber took advantage of this cloud infrastructure to quickly scale their businesses. In the process, these companies had to figure out how to manage open source distributed systems tools such as Hadoop and Kafka. Cloud servers were easy to create, but orchestrating them together to build distributed systems was still very hard. As the cloud providers matured, they developed higher level systems that solved many of the painful infrastructure problems. Managed databases, autoscaling queueing systems, machine learning APIs, and hundreds of other tools. Examples include Amazon Kinesis and Google BigQuery. These tools are invaluable because they allow a developer to quickly build applications on top of durable, resilient cloud infrastructure. With all of these managed services, developers are spending less time on infrastructure and more time on business logic. But managed services also lead to a new infrastructure problem—how do you manage resources across multiple clouds? A bucket storage system like Amazon S3 has different APIs than Google Cloud Storage. Google Cloud PubSub has different APIs than Amazon Kinesis. Since different clouds have different APIs, developers have trouble connecting cloud resources together, and it has become difficult to migrate your entire application from one cloud provider to another. Crossplane is an open source control plane for managing resources across multiple clouds. Crossplane’s goal is to provide a single API surface for interfacing with all the parts of your application, regardless of what cloud they are on. Crossplane is a project that was started by Upbound, a company with the goal of making multicloud software development easier. Bassam Tabbara is the CEO of Upbound, and he joins the show to talk about multi cloud deployments, Kubernetes federation, and his strategy for building a multi cloud API.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally posted on 13 September 2017. Machines understand the world through mathematical representations. In order to train a machine learning model, we need to describe everything in terms of numbers. Images, words, and sounds are too abstract for a computer. But a series of numbers is a representation that we can all agree on, whether we are a computer or a human. In recent shows, we have explored how to train machine learning models to understand images and video. Today, we explore words. You might be thinking–”isn’t a word easy to understand? Can’t you just take the dictionary definition?” A dictionary definition does not capture the richness of a word. Dictionaries do not give you a way to measure similarity between one word and all other words in a given language. Word2vec is a system for defining words in terms of the words that appear close to that word. For example, the sentence “Howard is sitting in a Starbucks cafe drinking a cup of coffee” gives an obvious indication that the words “cafe,” “cup,” and “coffee” are all related. With enough sentences like that, we can start to understand the entire language. Adrian Colyer is a venture capitalist with Accel, and blogs about technical topics such as word2vec. We talked about word2vec specifically, and the deep learning space more generally. We also explored how the rapidly improving tools around deep learning are changing the venture investment landscape. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally posted on 28 July 2017. Self-driving cars are here. Fully autonomous systems like Waymo are being piloted in less complex circumstances. Human-in-the-loop systems like Tesla Autopilot navigate drivers when it is safe to do so, and lets the human take control in ambiguous circumstances. Computers are great at memorization, but not yet great at reasoning. We cannot enumerate to a computer every single circumstance that a car might find itself in. The computer needs to perceive its surroundings, plan how to take action, execute control over the situation, and respond to changing circumstances inside and outside of the car. Lex Fridman has worked on autonomous vehicles with companies like Google and Tesla. He recently taught a class on deep learning for semi-autonomous vehicles at MIT, which is freely available online. There was so much ground to cover in this conversation. Most of the conversation was higher level. How do you even approach the problem? What is the hardware and software architecture of a car? I enjoyed talking to Lex, and if you want to hear more from him check out his podcast Take It Uneasy, which is about jiu jitsu, judo, wrestling, and learning. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally posted on 1 May 2018. Technology is pushing us rapidly toward a future that is impossible to forecast. We try to imagine what that future might look like, and we can’t help having our predictions shaped by the media we have consumed. 1984, Terminator, Gattaca, Ex Machina, Black Mirror–all of these stories present a dystopian future. But if you look around the world, the most successful technologists are mostly guided by a sense of optimism. Technologists themselves are mostly idealistic–they see the future through a utopian lens. Popular media largely tells a different story: that we are headed for a dystopian world. Why is there such a gulf in the level of idealism between technologists and the media? Mike Solana found himself asking that question on a regular basis during his work at Founder’s Fund, where he is a vice president. Founder’s Fund has a bias toward funding difficult, cutting-edge technology like gene editing, robotics, and nuclear energy. This technology that Mike was seeing made him excited about the future–which led to his creation of the podcast “Anatomy of Next.” “Anatomy of Next” has explored biology, robotics, nuclear energy, superintelligence, and the nature of reality. Soon the podcast will be exploring how our civilization will explore and settle the solar system–specifically Mars. I’ve listened through the entire first season of the show twice and enjoyed it so much because Mike explores questions that are on the border of philosophy and technology–questions about the nature of reality, and what makes us human–and nobody can give perfect answers to these questions. But Mike interviews top experts on the show, which provides us with a framework. Guests on “Anatomy of Next” include Nick Bostrom (the author of Superintelligence), George Church (a pioneer in gene editing), and Palmer Luckey (the founder of VR company Oculus). Mike joins the show to talk about why he started “Anatomy of Next,” and his own perspective on the future.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally posted on 16 June 2017. John Looney spent more than 10 years at Google. He started with infrastructure, and was part of the team that migrated Google File System to Colossus, the successor to GFS. Imagine migrating every piece of data on Google from one distributed file system to another. In this episode, John sheds light on the engineering culture that has made Google so successful. He has very entertaining stories about clusterops and site-reliability engineering. Google’s success in engineering is due to extremely high standards, and a culture of intellectual honesty. With the volume of data and throughput that Google responds to, 1-in-a-million events are likely to occur. There isn’t room for sloppy practices. John now works at Intercom, where he is adjusting to the modern world of Google infrastructure for everyone. This conversation made me feel quite grateful to be an engineer in a time where everything is so much cheaper, so much easier, and so much more performant than it was in the days when Google first built everything from scratch. I had a great time talking to John, and hope he comes back on the show again in the future because it felt like we were just scratching the surface of his experience. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally posted on 14 February 2017. Most tech companies are moving toward a highly distributed microservices architecture. In this architecture, services are decoupled from each other and communicate with a common service language, often JSON over HTTP. This provides some standardization, but these companies are finding that more standardization would come in handy. At the ridesharing company Lyft, every internal service runs a tool called Envoy. Envoy is a service proxy. Whenever a service sends or receives a request, that request goes through Envoy before meeting its destination. Matt Klein started Envoy, and he joins the show to explain why it is useful to have this layer of standardization between services. He also gives some historical context for why Envoy was so helpful to Lyft. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At Facebook, Venkat Venkataramani saw how large volumes of data were changing software infrastructure. Applications such as logging servers and advertising were creating fast moving, semi-structured data. The user base was growing, the traffic was growing, and the volume of data was growing. And the popular methods for managing this data were insufficient for the applications that developers wanted to build on top. In previous episodes about data platforms, we have covered similar difficulties as experienced by Uber and Doordash. Incoming data is often in JSON, which is hard to query. The data is transformed to a file format like Parquet, which requires an ETL job. Once it is in a Parquet file on disk in a data lake, the access time is slow. To query the data efficiently, it must be loaded into a data warehouse, which loads the data into memory, often in a columnar format that is easy to aggregate. Imagine being a developer at Facebook, Uber, or Doordash, and trying to build a simple dashboard, or a machine learning application on top of this data platform. Where do you find the right data? How do you know it is up to date? And what if you don’t know the shape of your queries ahead of time, and you haven’t defined indexes over your data? The access speed will be too slow to do exploratory analysis. There are so many steps in this process, and each of these steps creates friction for application developers that want to build on top of “big data”. Since even Facebook was having trouble managing this problem of the data platform, Venkat figured there was an opportunity to build a company around solving the data platform for other software companies. Venkat is the CEO of Rockset, a data system that is built to make it easy for developers to build data-driven apps. In Rockset, data can be ingested from data streams, data lakes, and databases. Rockset creates multiple indexes and schemas across the data. Because there are multiple models for querying, Rockset can analyze an incoming query and create an intelligent query plan for serving it. Venkat joins the show to discuss his time working on data at Facebook, the untapped opportunities of using that data, and the architecture of Rockset.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ten years ago, there was a distinction between “backend” and “frontend” developers. A backend developer would be managing the business logic and database transactions using Ruby on Rails or Java. A frontend developer would be responsible for implementing designs and arranging buttons using raw HTML and JavaScript. Today, developers can build entire applications in JavaScript. Developers who spent their early career developing frontend JavaScript skills are finding themselves with a surprising amount of power. With NodeJS providing a backend framework and React, Vue, or Angular on the frontend, a single JavaScript developer can write all the code for a whole application—hence the rise of the “full stack developer”. At the same time, the cloud infrastructure is becoming easier to use. Backend-as-a-service simplifies the frustrations of deploying your application, and standing up a database. GraphQL improves the relationship between the frontend and the backend. And futuristic technologies like WebAssembly and web virtual reality are promising to make a JavaScript engineer’s life even more interesting. Adam Conrad is an engineer and a writer for Software Engineering Daily. In recent articles, he has documented the changing nature of the frontend, including JavaScript engines, virtual reality, and how mature corporations are using React and GraphQL. He joins the show to share his perspective on what is changing in the frontend—and how full stack JavaScript engineers can position themselves for future success in a quickly changing market.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software products are distributed across more and more servers as they grow. With the proliferation of cloud providers like AWS, these large infrastructure deployments have become much easier to create. With the maturity of Kubernetes, these distributed applications are more reliable. Developers and operators can use a service mesh to manage the interactions between services across this distributed application. A service mesh is a layer across a distributed microservices application that consists of service proxy sidecars running alongside each service in a cluster, along with a central control plane for communicating with those sidecar proxies. A service mesh has many uses. Every request and response within the application gets routed through the service proxy, which can improve observability, traffic control to different instances, and circuit breaking in case of an instance failure. The central control plane can be used manage network policy throughout the whole system. We have done shows about each of the different components of a service mesh system, including different types of service proxies, as well as the service meshes built on top of these proxies. Linkerd, which is made by the startup Buoyant, was the first service mesh product to come to market, and it has the most production use, with customers like Expedia and Monzo bank. Istio is a more recent service mesh which uses the Envoy service proxy. Istio came out of Google and is also supported by IBM—setting up a classic competition between a startup and the large incumbents. William Morgan is the CEO of Buoyant, and he joins the show to talk about the use cases and adoption of service mesh. He also talks about the business landscape of the service mesh category, and how to compete with giant cloud providers.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Market strategy defines how a company is positioning itself to be successful. This strategy encompasses engineering, sales, marketing, recruiting, and everything else within a company. Herb Cunitz has led teams at Hortonworks, VMware, SpringSource, and several other companies over his 30 year career in software. After working as president of Hortonworks, Herb started AccelG2M. AccelG2M works with software companies to define their go-to-market strategy. Software companies require a great deal of long-term strategic thinking. Engineering, sales, marketing, and leadership must work together to build a plan that will allow the company to reach an exit: either an acquisition or an IPO. Executives at a software company must create a clear strategy and communicate it to the employees throughout the organization. The strategy must be implemented, meeting deadlines and hitting milestones. New team members must be recruited, and unsuccessful workers must be let go. In today’s show, Herb provides some invaluable strategic wisdom for anyone working in software–whether you are an engineer, salesperson, or investor.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software companies today rely on group chat applications. The world of startups and small businesses is dominated by Slack. But for some large enterprises, regulatory constraints prevent them from using Slack. Slack is a web application that is hosted in the cloud, and regulated industries such as banking often need to run their applications on their own on-prem infrastructure. Mattermost is an open source alternative to Slack that can be self-hosted. This means that all of the networking complexities and scalability challenges that are controlled in the cloud by Slack need to be handled by open source code rather than managed services running in the cloud. Because it is open source, Mattermost can also be redesigned and customized. Uber designed their own custom version of Mattermost called uChat. Corey Hulen is a co-founder and the CTO of Mattermost. He joins the show to discuss the motivation for building Mattermost and the engineering challenges of building an open source chat system. For more episodes about building chat systems, we’ve done several shows about Slack, covering the engineering, security, and chat system.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Wes Bos has created popular courses on React, GraphQL, and JavaScript. With hundreds of thousands of students, Wes has earned a cult following for his fun, practical lessons on web development. The courses produced by Wes teach developers how to build useful applications such as a complete e-commerce store. Wes has built a career around studying and evangelizing JavaScript. His approach to education centers around practice, repetition, and hacking on fun projects. He also co-hosts a podcast called SyntaxFM, and is a frequent Twitter user. Wes is a rare mix of developer, teacher, businessman, and designer. Throughout his work, there is an artist’s sense of attention to detail, and a modern entrepreneur’s sense of pricing and marketing. His sites, such as JavaScript30 and React For Beginners have the deliberate style of someone who has been building websites for a very, very long time.In today’s episode, Wes Bos joins the show to give his perspective on JavaScript, entrepreneurship, and podcasting. To learn more about Wes’s business and his background, check out the IndieHackers podcast with him.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A bank account is a platform for apps to be built on top of. If that sounds like a weird idea, think about the features of a bank account. Most users only have a single bank account, making it a tool for identity and authentication. The series of transactions in a bank account provides a data set that can be used for analyzing payment history and issuing loans, or insurance. But there are difficulties to building a platform on top of banking. There are thousands of different banks. If you want to build an application that integrates with a user’s bank, you need to be able to integrate with any bank that the user might use–whether it’s Bank of America, Wells Fargo, or Chase. Plaid is a company that builds APIs for users to connect to banks. Applications such as Venmo, Betterment, and Coinbase use Plaid to connect with the bank accounts of their users. Jean-Denis Greze joins the show to explain how applications use Plaid, and how Plaid has scaled its infrastructure to handle a high volume of requests. He also discusses the potential of banking as a platform, and the strategy for expanding the APIs that Plaid can offer to developers. Fintech Daily is a new podcast from Software Engineering Daily covering payments, cryptocurrencies, trading, and the intersection of finance and technology. We are looking for volunteer hosts for Fintech Daily, and if you are interested in working with us to conduct interviews, send an email to host@fintechdaily.co. You can find the podcast on iTunes, Google, and everywhere else, and if you are interested in hosting, don’t hesitate to reach out.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a startup finds product market fit, the adoption of that product can grow rapidly, turning a startup into a high growth company. All of a sudden, a startup that was struggling to find its first customer is bombarded with new challenges. The startup has to hire tens of new employees. This requires raising capital, so the startup has to meet with investors and lawyers. A rapid influx of new customers puts a strain on the engineering and customer service elements of the company. There is too much to do, and there is only so much time in a day. The CEO of the high-growth company is up late into the night, answering emails and losing sleep. But these are good problems to have, and the company is in a state of exuberance. The CEO must balance psychological health with the stressful task of scaling a company. Elad Gil is an entrepreneur and author of “High Growth Handbook”, a book of lessons and guidelines about how to navigate a startup that has found product market fit, and is beginning to scale. High Growth Handbook includes interviews with experienced entrepreneurs such as Marc Andreessen and Patrick Collison, whom Elad met with as he wrote the book. Elad joins the show to discuss his book, and his own personal lessons of working with companies such as Twitter, Google, Stripe, and Coinbase. Elad has worked at several high growth companies and invested in others, and he has gathered a lot of wisdom from these different experiences.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Releasing software has inherent risk. If your users don’t like your new feature, they might stop using your product immediately. If a software bug makes it into production, it can crash your entire application. Releasing software gradually has many benefits. A slow rollout to an increasing population of users allows you to test your software in multiple real-world environments before it goes live to everyone. A system of AB testing different versions of your software lets you see how different flavors of your software perform against similar audiences. Edith Harbaugh is the CEO of LaunchDarkly, a system for feature management. LaunchDarkly allows developers to deploy new software releases in a controlled fashion. Edith joins the show to discuss how to implement feature flagging, and why an intelligent release process can lead to a more scientific, predictable environment for software development. Edith is also the host of To Be Continuous, a podcast about continuous delivery, software engineering, and DevOps.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Berkeley AMPLab was a research lab where Apache Spark and Apache Mesos were both created. In the last five years, the Mesos and Spark projects have changed the way infrastructure is managed and improved the tools for data science. Because of its proximity to Silicon Valley, Berkeley has become a university where fundamental research is blended with a sense of industry applications. Students and professors move between business and academia, finding problems in industry and bringing them into the lab where they can be studied without the day-to-day pressures of a corporation. This makes Berkeley the perfect place for research around “serverless”. Serverless computing abstracts away the notion of a server, allowing developers to work at a higher level and be less concerned about the problems inherent in servers–such as failing instances and unpredictable network connections. With serverless functions-as-a-service, the cloud provider makes guarantees around the execution of serverless code–such as with AWS Lambda. With serverless backend services, the cloud provider makes guarantees around the reliability of a database or queueing system. The cloud provider is operating servers to power this functionality. But the user is not exposed to those servers. Today’s show centers around the serverless functions-as-a-service. This is a new paradigm of computing, and there are many open questions. How can the servers for our functions be quickly provisioned? How can we parallelize batch jobs into functions as a service? How can large numbers of serverless functions communicate with each other reliably to coordinate? In production applications, functions-as-a-service are mostly used for “event-driven” applications. But the potential for functions-as-a-service is much larger. Ion Stoica is a professor of computer science at Berkeley, where he leads the RISELab. He is the co-founder of Conviva Networks and Databricks. Databricks is the company that was born as a result of the research on Apache Spark. Ion now serves as executive chairman of Databricks. Ion joins the show to describe why serverless computing is exciting, the open research problems, and the solutions that researchers at the RISELab are exploring.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Robotics, genomics, and backend infrastructure: as an in vestor, it can be difficult to assess the viability of a startup that is on the cutting edge in any of these areas. A robotics startup requires a team with an integrated understanding of hardware and software. A genomics company will not only have to develop a successful healthcare product, but will have to bring it to market through regulation. And in the world of backend infrastructure, building a business that will be differentiated from giant cloud providers gets harder every day. Amplify Partners is a venture capital fund with an emphasis on technical investments. Their portfolio includes infrastructure companies like Datadog and Gremlin, as well as pharmaceutical and hardware companies. Sunil Dhaliwal is the founder of Amplify Partners, and joins the show to discuss the thesis of Amplify. The investments that Amplify makes are in technical companies–which makes these financing decisions complex enough to require detailed, individualized research. But there are commonalities among the founding teams. Sunil lays out a useful rubric for anyone who is looking to learn about venture capital investing.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Building software was simplified by cloud providers. With the cloud, it became much easier to deploy a server, spin up a database, and scale an application. Cloud providers like AWS gave developers access to these infrastructure primitives like storage and compute. On top of those primitives, numerous API companies have been built. An API company offers a more specific set of services. Twilio offers SMS text messaging API services. Stripe offers payment API services. These APIs give developers another level of tooling to build software out of. Developers can now think of entire applications in terms of APIs, and the number of APIs is growing rapidly. From business services such as booking a flight to machine learning models like image classification, the “API economy” has given developers a huge catalog of tools. Since developers have this additional leverage, software can be built with smaller teams. The codebase can also be smaller. But one area where the complexity is growing is the number of APIs that need to be managed. For each API, there is a different system for integrating the API into your application. Different API providers have different levels of reliability. Another area of difficulty is the discoverability of APIs. If I don’t know about a flight search API, I am never going to think of what applications I could build on top of that. There are APIs for generating memes, and APIs for easily querying what music is trending across the world. RapidAPI is a marketplace for APIs. It includes search and discovery features for the wide variety of different APIs that can be found across the internet. RapidAPI is also a system for integrating with multiple APIs through it’s API management system. Iddo Gino is the CEO and founder of RapidAPI, and he joins the show to discuss the motivation for creating an API marketplace, as well as the engineering behind RapidAPI.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Bitcoin main chain is a large distributed ledger of transactions. Bitcoin is useful for maintaining a trusted record of payments, but is not practical for small day-to-day payments. Bitcoin payment channels allow users to issue small payments to each other without paying the high transaction cost and latency of going through the main chain. When payment channels are connected to each other, a “lightning network” is formed. Lightning network is often referred to as a “second layer” scalability solution. Alex Bosworth is a lightning infrastructure lead at Lightning Labs, a company that builds infrastructure for scaling blockchains. In today’s show, Alex explains how Bitcoin payment channels work, and provides some context on how developed the modern infrastructure is in terms of practical use cases for Bitcoin.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Demand for live streaming video over the internet is increasing. After the emergence of early live streaming platforms, like Twitch and Facebook Live, more forms of video have become accessible over live streams, such as sports. Live streaming is a harder engineering problem than delivering a static video file because the information distributed on a live stream is constantly changing. DZN, spelled D-Z-N, is a live streaming service for watching fight events, such as boxing. The workloads for live streaming can be highly bursty. When a fight is scheduled to happen, the vast majority of traffic will hop on to watch the fight 20 seconds before the fight starts. A huge number of users logs into DZN and starts watching all at the same time. This quick spike in traffic means that DZN has to have servers spun up and be ready in advance. Luca Mezzarlira and Yan Cui are engineers at DZN. Yan was previously on the show in a few amazing episodes to talk about serverless infrastructure and the complexities of real-time video game software development. Those episode links are in the show notes. I highly recommend checking them out. Today’s show is a discussion of architecting a system to handle a high bandwidth customer use case. I hope you like it.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Not every company wants to move to the public cloud. Some companies have already built data centers, and can continue to operate their business with their own servers. Some companies have compliance issues with the public cloud, and want to operate their own servers to avoid legal risk. Operating a data center is not easy. Operating systems need to be updated and security vulnerabilities need to be patched. Servers fail, and their workloads need to be automatically scheduled onto other servers to avoid downtime. In contrast to classic on-prem data center management, the cloud provides many benefits: automatic updates, an infinite pool of resources, fully programmable infrastructure as code. In the cloud, developers can provision infrastructure with an API request. Continuous delivery pipelines can be spun up at the click of a button. This tooling makes it dramatically easier for developers to move quickly, and for a business to move faster. Companies that operate their own data center want to be able to have these benefits of the cloud while still controlling their own infrastructure. Today’s guest Bob Fraser works at HPE on OneView, a tool for managing on-prem infrastructure like a cloud. Bob describes the difficulties of managing legacy on-prem infrastructure, and the advantage of building a management layer on top of data center infrastructure to make it more programmable. We’ve done lots of shows recently about Kubernetes in the context of cloud computing. Today’s show outlines how modern on-prem infrastructure can be managed like a cloud. Full disclosure: HPE is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Podcasts have surged in popularity, but the podcast ecosystem remains difficult to work with. Podcast listeners have difficulty finding episodes. Podcast creators have difficulty finding out how to get started. The advertising marketplaces for podcasts are immature, and it can be difficult to build a business as a podcaster. Podcasting is unlike almost any other media format that we consume on the Internet. There is not an algorithmic feed of podcasts–we subscribe to podcasts we like and we see everything that gets published. Podcasting originated with Apple, who has not shown much interest in the podcast medium. Anchor is a platform that makes it easy for users to publish podcasts. Today, a large percentage of the new podcasts created on the Internet are started on Anchor. Nir Zicherman is the CTO at Anchor, and he joins the show to discuss the strange world of podcast technology, and how Anchor is building a business.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud computing changed the economics of running a software company. Before the cloud, a software company had to purchase physical machines which often required thousands of dollars paid up front. The cloud allowed developers to deploy their applications for free, to operate a business for cheap, and to scale without hiring a dedicated team to manage the servers. Building in the cloud is cheap, but scaling in the cloud can get expensive. A growing company can often save money by changing which cloud instances and services they use. Reducing the number of server instances, changing the size of compute instances, and changing rules around auto scaling. By using monitoring, dashboards, and regular analysis of where money is spent, a business can find thousands of dollars of wasted spend per month. There are also broad strategic decisions around cost. One area to study is the use of “managed” services like Amazon DynamoDB, Google BigQuery, and Amazon Lambda. These services are proprietary, and can lead to lock-in. Sometimes they can be quite expensive. But they can save developers hours of time because they are easy to use, and provide high uptime guarantees. Ran Rothschild works at DoIT International, a company that helps businesses figure out how to save money on their cloud infrastructure. He joins the show to discuss the places where the most money is wasted and how startups can manage their infrastructure in a cost-effective manner. He also tells some stories about significant overspend. Full disclosure: DoIT International is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Slack is a real-time messaging system for work communication. On Slack, chat rooms as big as 100,000 people have productive conversations. This might sound like the same problem solved by social networks like Facebook, where billions of users communicate over a newsfeed. But the engineering constraints of a messaging system are different than that of a social network. On a newsfeed, the order in which events appear is not chronological. Events can be out of order. You can miss events. When a user posts a message to a social network, there are not strict guarantees around when other people will see that message. On Slack, messages have strong guarantees around arrival. When I send a message, everyone else who is in the room and connected should quickly receive that message as well. The messages need to be ordered and delivered exactly once. All messages on Slack are persisted. We have covered the architecture and security model of Slack in previous shows. In today’s show, Keith Adams returns to discuss how messages are processed and broadcast in Slack. The problem of Slack’s messaging system is similar to the distributed systems problem of “atomic broadcast”, in which a single process broadcasts a message which needs to be received by all other processes correctly–or else received by none of them. In Keith’s last show, he talked through the benefits of building a large system on PHP. He worked on infrastructure at Facebook, which was also a PHP application. It’s worth noting that both Slack and Facebook have scaled a monolithic architecture.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Many people have saved some money which they want to invest for the future. Some people are happy investing their money in a roboadviser, which programmatically puts money into long-term investments. Other people want a more personal approach involving a certified financial planner (CFP). A CFP is a human who allocates capital for an individual based off of that individual’s preferences. A CFP spends time and effort researching the options for a client. If the client only has a small amount of money (say, $15,000), it is not worth it for the CFP to spend much time on that account. As a result, there is a type of client who has saved a little bit of money, but has not saved enough to be an important client for a CFP. Facet Wealth is a software company that makes software for CFPs to work more effectively with their client accounts. Facet Wealth has in-house CFPs who work with this software to manage client accounts. In addition, Facet Wealth buys client accounts from independent CFPs who have small accounts which they do not have time to manage. This is an innovative way to aggregate users onto the platform. Gorkem Sevinc is CTO at Facet Wealth. He joins the show to describe the business and the software architecture of the company. We touched on many different areas–from human-computer interaction to the future of investing. We recently launched a new podcast: Fintech Daily! Fintech Daily is about payments, cryptocurrencies, trading, and the intersection between finance and technology. You can find it on fintechdaily.co or Apple and Google podcasts. We are looking for other hosts who want to participate. If you are interested in becoming a host, send us an email: host@fintechdaily.co

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Parity is a company that builds blockchain infrastructure. Parity has built several open source projects and works with enterprises to put blockchain technology in production. Gavin Wood is the founder of Parity, and he joins the show to talk about the state of blockchain technology and what his company is currently focused on. Four years ago, Gavin helped start the Ethereum project, so he has lots of context on decentralized technology. Gavin envisions a world with many different blockchains for many different use cases. These blockchains will interact with each other to enable trusted relationships between parties. One project that Parity has created is Substrate, a technology that allows developers to quickly stand up a blockchain with the right privacy level. Another project is Polkadot, which allows blockchains to connect and interoperate with each other. Gavin and I discussed why the world needs a variety of blockchains–and whether all of these different blockchains should need their own cryptocurrency. Gavin described the use case of blockchains for mediating supply chain trust. We also talked about the technologies used to build these projects, including WebAssembly and Rust.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally posted on June 23, 2016. Pieter Hintjens grew up writing software by himself. The act of writing code brought him great pleasure, but the isolated creative process disconnected him from the rest of the world. As his life progressed he became involved in open source communities, and he discovered a passion for human interaction. Open source software succeeds or fails on the strength of the community. One story of success is ZeroMQ, a popular open source distributed messaging system that was started by Pieter Hintjens. In this episode, Pieter gives his thoughts on human nature, distributed systems, and death. “A Protocol For Dying” is a blog post Pieter wrote recently, where he discussed his terminal diagnosis of cancer, and how it has reframed his perspective on life. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally posted on May 12, 2016. When you are deciding who to marry, you are using an algorithm. The same is true when you are looking for a parking space, playing a game of poker, or deciding whether or not to organize your closet. Algorithms To Live By is a book about the computer science of human decisions. It offers strategies for how to think through everyday life like a computer scientist. Brian Christian has a background in computer science and philosophy, and is an author of Algorithms to Live By. He joins the show to explain how the same algorithms and data structures we use for our computer programs can be applied to the real world. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally posted on May 12, 2015. Humans have now been defeated by computers at heads up no-limit holdem poker. Some people thought this wouldn’t be possible. Sure, we can teach a computer to beat a human at Go or Chess. Those games have a smaller decision space. There is no hidden information. There is no bluffing. Poker must be different! It is too human to be automated. The game space of poker is different than that of Go. It has 10^160 different situations–which is more than the number of atoms in the universe. And the game space keeps getting bigger as the stack sizes of the two competitors gets bigger. But it is still possible for a computer to beat a human at calculating game theory optimal decisions–if you approach the problem correctly. Libratus was developed by CMU professor Tuomas Sandholm, along with my guest today Noam Brown. The Libratus team taught their AI the rules of poker, they gave it a reward function (to win as much money as possible), and they told it to optimize that reward function. Then they had Libratus train itself with simulations. After enough training, Libratus was ready to crush human competitors, which it did in hilarious, entertaining fashion. There is a video from Engadget on YouTube about the AI competing against professional humans. In this episode, Noam Brown explains how they built Libratus, what it means for poker players, and what the implications are for humanity–if we can automate poker, what can’t we automate? Stay tuned at the end of this episode for the Indeed Prime tip on hiring developers. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Featured Image Photo Credits Originally posted on July 11, 2016. Negotiation is an important skill for software engineers. The salary you negotiate at the beginning of your job could be a difference of tens of thousands of dollars over the course of an engineer’s career, but intimidating recruiters and exploding offers scare many engineers from negotiating at all. Today, Haseeb Qureshi returns to the show to discuss his epic story of salary negotiation. On a previous episode, Haseeb discussed leaving his career as a poker player to join a coding boot camp and start down the path of a software engineer. In this episode and his recent blog post, Haseeb describes his approach to the job search and salary negotiation process, which eventually landed him at Airbnb with a $250,000 annual salary–after about a year of learning to code. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published on July 6, 2016. Scheduling is the method by which work is assigned to resources to complete that work. At the operating system level, this can mean scheduling of threads and processes. At the data center level, this can mean scheduling Hadoop jobs or other workflows that require the orchestration of a network of computers. Adrian Cockcroft worked on scheduling at Sun Microsystems, eBay, and Netflix. In each of these environments, the nature of what was being scheduled was different, but the goals of the scheduling algorithms were analogous–throughput, response time, and cache affinity are relevant in different ways at each layer of the stack. Adrian is well-known for helping bring Netflix onto Amazon Web Services, and I recommend watching the numerous YouTube videos of Adrian talking about that transformation. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The volume of data in the world is always increasing. The costs of storing that data is always decreasing. And the means for processing that data is always evolving. Sensors, cameras, and other small computers gather large quantities of data from the physical world around us. User analytics tools gather information about how we are interacting with the Internet. Logging servers collect terabytes of records about how our systems are performing. From the popularity of MapReduce, to the rise of open source distributed processing frameworks like Spark and Flink, to the wide variety of cloud services like BigQuery: there is an endless set of choices for how to analyze gigantic sets of data. Machine learning training and inference is another dimension of the modern data engineering stack. Whereas tools like Spark and BigQuery are great for performing ad-hoc queries, systems like TensorFlow are optimized for the model training and deployment process. Stitching together these tools allows a developer to compose workflows for how data pipelines progress through a data engineering system. One popular tool for this is Apache Airflow, which was created in 2014 and is widely used at companies like Airbnb. Over the next few years, we will see a proliferation of new tools in the world of data engineering–and for good reason. There is a wealth of opportunity for companies to leverage their data to make better decisions, and potentially to clean and offer their internal data as APIs and pre-trained machine learning models. Today, there is a vast number of enterprises who are modernizing their software development process with Kubernetes, cloud providers, and continuous delivery. Eventually, these enterprises will improve their complex software architecture, and will move from a defensive position to an offensive one. These enterprises will shift their modernization efforts from “DevOps” to “DataOps”, and thousands of software vendors will be ready to sell them new software for modernizing their data platform. There is not a consensus for the best way to build and run a “data platform”. Nearly every company we have talked to on the show has a different definition and a different architecture for their “data platform”: Doordash, Dremio, Prisma, Uber, MapR, Snowflake, Confluent, Databricks… We don’t expect to have a concise answer for how to run a data platform any time soon–but on the bright side, data infrastructure seems to be improving. Companies are increasingly able to ask questions about their data and get quick answers, in contrast to the data breadlines that were so prevalent five years ago. Today we cover yet another approach to large scale data processing. Reflow is a system for incremental data processing in the cloud. Reflow includes a functional, domain specific language for writing workflow programs, a runtime for evaluating those programs incrementally, and a scheduler for dynamically provisioning resources for those workflows. Reflow was created for large bioinformatics workloads, but should be broadly applicable to scientific and engineering computing workloads. Reflow evaluates programs incrementally. Whenever the input data or the program changes, only the outputs that depend on the changes are recomputed. This minimizes the amount of recomputation that needs to be performed across a computational graph. Marius Eriksen is the creator of Reflow and an engineer at GRAIL. He joins the show to discuss the motivation for a new data processing system–which involves explaining why workloads in bioinformatics are different than in some other domains.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The software release process is a barrier between written code and a live production environment that affects users. A software release process can involve a variety of different practices. Code might be tested for bugs using automation and manual testing. Static analysis tools can look at the code for potential memory leaks. A software release might go out to a small percentage of the total user base before it gets deployed to the entire audience. At some organizations, a software release can be slow and painful. The release might be bottlenecked by a manual approval step, which slows down developers from quickly deploying their own changes. If a consistent version history of software is not maintained, a release can be hard to roll back in the event of an error. In the case of a large, monolithic architecture, a release can be scary, because it can be hard to understand how the monolithic codebase functions. This set of challenges within the release process lowers the quality of software, and can make it frustrating to build software. The release process is just one area of software development that many organizations have a desire to smooth out. Over the past ten years, a set of technologies and philosophies have provided improvements to the software development process. DevOps, continuous delivery, microservices, cloud providers, and serverless tools all make it easier for a company to focus on its core competency and release software faster. Baruch Sadogursky is an author of Liquid Software, a book about continuous updates and DevOps. Liquid Software describes an idealized vision of what today’s architecture could aspire to. The focus of the book is continuous updates, which allow for rapidly improving, evolving software quality. Baruch joins the show to discuss how software has changed in the last twenty years, and how the future of software development could look. Full disclosure: Baruch works at JFrog, which is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern software consists of sprawling international networks of servers. Users contact these servers to access applications. Microservices talk to each other to fulfill complicated requests. Databases and machine learning frameworks crunch terabytes of information to provide complicated answers. Across this infrastructure, there is a lot of different activities–and a lot of vulnerabilities. Without a reliable model for security and trust, software can be easily compromised. In the past, systems were often protected by a “firewall”, which is a security system around the perimeter of the network. A problem with this model is that if the attacker is able to penetrate the firewall, they can compromise anywhere in the network. Firewalls can be penetrated, so a much better security model is to assume that your network has already been compromised, and to require every internal system to identify and authenticate with each other. “Zero-trust security” is a security model that requires internal systems to communicate with each other as if they were potentially compromised. Evan Gilman is the author of Zero Trust Networks: Building Secure Systems in Untrusted Networks. He also works on SPIFFE, a system for managing identity and trust within a zero-trust network. In a previous episode about Google BeyondCorp, Max Saltonstall talked about zero-trust networking in the context of user and device authentication. In today’s episode, Evan discusses another side of zero-trust networking: workload identity and authentication. Just as Google BeyondCorp outlines an architecture for allowing devices to communicate with the network, the SPIFFE project outlines a system for workloads to identify and authenticate themselves. Workloads can range from MapReduce jobs to microservices to frontend application servers.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Serverless computing abstracts away the idea of a server node. Serverless lets programmers treat compute resources as high-level, reliable APIs, rather than unreliable, low-level compute nodes that might fail. Serverless dramatically improves the efficiency of programmers. Instead of thinking of a database as a set of servers that need to be sharded and replicated, the programmer can think of a database as a place to read and write data. Instead of modeling an application as a large monolith running on an application server in a container or a VM, the programmer can think of their application as a decoupled set of functions. Serverless computing is a natural evolution of software engineering in a world with cloud providers. The first version of FaaS came out of AWS with their Lambda service, which allows users to run functions in the cloud. Those functions are scheduled onto a physical server somewhere in an Amazon data center. They are executed, and they return the result. With AWS Lambda, programmers got a new abstraction to model their applications with. But it requires the use of a closed-source API. Lambda is not open source, and this makes some developers reluctant to integrate with it too tightly. Fission is an open source framework for serverless functions built on Kubernetes. Fission allows developers to deploy functions-as-a-service without being locked in to any specific cloud provider. Soam Vasani is the creator of Fission and an engineer at Platform9. In a previous episode, Soam talked about the architecture for Fission and the design choices for solving the cold start and scheduling problems. Soam joins the show today to discuss how serverless applications have evolved since last spoke more than a year ago. He also talks about how Fission itself has evolved, and the features that an open source serverless platform needs to have in order to compete with a fully developed cloud provider. Full disclosure: Platform9 is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Policies define which users and applications can access and modify resources in a computer system. In a file system, a user might have permission to read or write to a file. In a cloud infrastructure deployment, a user might have the rights to deploy a new server. One microservice may or may not have the necessary permissions to talk to another microservice. All of these are use cases where a “policy” defines the behavior within a computer system. Policies in a company can be managed in a range of ways: configuration files, dashboards, and centralized permissions databases. A policy engine is a system for managing and automating the policy creation and deployment within an organization. Microservices need to verify each request that comes in to ensure that the request has the correct permissions. To check those permissions, a microservice can contact the policy engine. The policy engine has all the information from the whole organization about who is allowed to do what. However, talking to the policy engine over the network can be a slow process. Open Policy Agent is a deployable agent that can run as a sidecar next to a service, and check policies by looking inside of a cache. Torin Sandall is a core committer to the Open Policy Agent project, and he joins the show to talk about policy management, the Open Policy Agent, and the Kubernetes ecosystem (and surprisingly, WebAssembly).

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

TLA+ is a formal specification language. TLA+ is used to design, model, and verify concurrent systems. TLA+ allows a user to describe a system formally with simple, precise mathematics. TLA+ was designed by Leslie Lamport, a computer scientist and Turing Award winner. Leslie joins the show to talk about the purpose of TLA+. Since its creation in 1999, TLA+ has been used to discover bugs in systems such as Amazon S3, DynamoDB, Xbox, and Cosmos DB. “TLA” stands for “temporal logic of actions”, a logical system that can be used to describe the behaviours of concurrent systems. This podcast is meant as a brief introduction of TLA+. To go deeper, check out the TLA+ website and the TLA+ video course (note: these videos are highly entertaining because of Leslie’s dry, unpredictable sense of humor).

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Mapillary is a company that processes high volumes of images to develop a labeled 3-D model of the physical world. Mapillary’s APIs allow developers to build applications that are aware of stop signs, buildings, streets, trees, and other physical objects in real-world space. The potential use cases for Mapillary are numerous, ranging from self-driving cars to augmented reality. We can now build a 3-D model of the real world. It’s not a perfect representation of reality, but it is much better than we had just a few years ago. What has changed? How have the tools advanced such that we are able to build an API for accessing accurate information about the physical world around us? Mapillary is possible because of a combination of modern developments. High quality smartphone cameras enable users to crowdsource images of the world around them. Cloud computing allows for cheap workload processing. Newer computer vision techniques allow 2-D images to be stitched together in a 3-D representation. Deep learning architectures improve the classification and segmentation of objects in an image. Peter Kontschieder is the head of research at Mapillary, and he joins the show to talk about the technologies and research that has enabled Mapillary to build a futuristic business–an API for accessing information about the physical world. Software Engineering Daily is looking for sponsors. If you are interested in reaching over 50,000 developers, you can go to softwareengineeringdaily.com/sponsor to find out more, and you can send us a message. We’d love to hear from you. And if you are an engineer working at a company that is marketing to developers, or hiring developers, if you tell your marketing department or your recruiting department about softwareengineeringdaily.com/sponsor, that is one way to help us out.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Open source projects benefit from the network effects of a large audience of developers. A popular open source project will be contributed to and used by thousands of developers, who are continuously testing, deploying, and improving the software. The open source movement has created massive communities and a thriving, collaborative economy. Infrastructure software companies are increasingly built within an open source business model. Databases, queueing systems, orchestrators, operating systems, and search engines have been started as freely available open source projects, and leveraged into billion dollar businesses. In previous shows we have talked about business strategy, go-to-market tactics, and licensing of infrastructure software. There remains plenty of room for more open source infrastructure companies. We still need better databases and distributed systems management. But over time, open source will move up the stack. From Netflix to Uber to social networks to payments systems–all software verticals will become open source because the benefits of making your software open source outweigh the costs. For many software business models, the competitive advantage is not found in their source code–it’s in their data, their network effects, their sales strategy, and their brand. Therefore, it makes sense that someday the source code will be freely available, democratizing the infrastructure concerns and letting these software businesses move up the value chain and become less operationally intensive at the bottom of the stack. Rather than asking “why should we open source our code”, these companies will be asking “why shouldn’t we open source our code?” Joseph Jacks is the founder of OSS Capital, a venture capital firm that invests exclusively in commercial OSS startup companies. Joe believes that over time, open source eats everything. In today’s show, we talk about the future of open source businesses, the impact of licensing, cloud providers, and cryptocurrencies.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Open source software powers everything we do on the Internet. Google runs on Linux servers. Content sites are served by WordPress. Our data is queued in Kafka clusters and stored in MongoDB instances. The success of an open source project often leads to the creator of that open source software becoming wealthy. An open source project can be monetized through enterprise add-ons, or consultation, or simplified hosting. The creators of open source software know their domains so well, that they are usually well-suited to operate this kind of open source business. Open source business model success stories include Elastic (ElasticSearch), Cloudera (Hadoop), and Red Hat. The rise in usage of cloud providers has changed the viability of some open source business models. AWS can monetize almost any open source project more profitably than the creators. This is because AWS has established distribution channels. If I already run my application on AWS, and I am looking for someone to provide me with a hosted version of a database, I will probably choose the hosted database that AWS provides. The Commons Clause is a license that open source projects can use to protect their code from being profited from. Redis, an open source in-memory object storage system, recently licensed their code with the Commons Clause with the goal of improving the business of Redis Labs, a company built by the creators of the Redis project. Kevin Wang joins the show to discuss everything open source–from business models to security vulnerabilities to licensing. Kevin is the CEO at FOSSA, a system for managing open source licensing and security. Kevin was involved in the creation of the Commons Clause and has written about it in detail.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Matt Klein has worked for three rapidly growing Internet companies. At AWS, he worked on EC2, the compute-as-a-service product that powers a large percentage of the Internet. At Twitter, he helped scale the infrastructure in the chaotic days before Twitter’s IPO. Today he works at Lyft, building systems to allow for ride sharing infrastructure to work more safely and reliably. Hypergrowth Internet companies are faced with quickly growing demands on their software. The demands on the software expose problems with the core infrastructure. Simultaneously, the company tries to ramp up its hiring process. More engineers get hired, and the institutional knowledge within the company starts to weaken. Documentation gets out of date. Senior engineers burn out and leave the company. When a company starts growing quickly, communications can break down. A hypergrowth company can suffer from a lack of “human scalability”. Matt Klein has observed these challenges at AWS, Twitter, and Lyft. In his article “The Human Scalability of ‘DevOps’”, he explains why these problems manifest and what can be done to alleviate them. In a previous show, Matt discussed the engineering challenges at Lyft that led him to create Envoy, a service proxy. This episode covers some broad technical topics–DevOps, site reliability engineering, platform engineering–but the episode is mostly about how a hypergrowth company can manage culture, hiring, and engineering organization. Matt is a very fun guest to have because he questions some of the strange practices that have been widely adopted by successful companies. Internet companies are a very new phenomenon, and the management tactics that they have adopted are not well proven–so it is great to have someone like Matt provide a fresh perspective on ways that companies can scale their technology and their organization more effectively.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Online labor marketplaces are widely used for one-to-one transactions. On Uber, a rider hires a driver for transportation. On TaskRabbit, a homeowner hires a cleaner to come clean their kitchen. These types of marketplaces are not as widely used for one-to-many transactions, but they can be just as useful. A warehouse owner would want to hire a group of workers to help with holiday shipments. A conference organizer would want to hire a group of event staffers to help run the conference. Wonolo is an on-demand staffing platform. Businesses post jobs and workers apply for those jobs. The types of work include event staffing, warehouse operations, merchandising, and other general labor tasks. In past shows, we have covered on-demand work platforms such as Fiverr, Thumbtack, Uber, and Instacart. Wonolo presents another variation in the business model and software architecture of the gig economy. Jeremy Burton is the CTO and chief data scientist at Wonolo. He joins the show to talk about building and scaling Wonolo, and some of the key strategic decisions that Wonolo has made along the way. As with any successful marketplace business, Wonolo has solved the chicken and egg problem of how to get supply and demand on the platform simultaneously. The company has grown deliberately, setting up operations in one city at a time to make sure that they can provide a good experience in both sides of the market in each of the new geographies. Jeremy and I also talked about the broader effects that the gig economy could potentially have on the labor market. Gig economy platforms use a 5-star rating system and written reviews to judge workers, instead of a resume system. The gig economy allows for rapid job liquidity, and the potential for workers to steadily “level up” more quickly than they might be able to in a typical corporate job. These aspects of the gig economy are rarely discussed, so it was enlightening to hear Jeremy’s views on them.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google Search allows humans to find and access information across the web. A human enters an unstructured query into the search box, the search engine provides several links as a result, and the human clicks on one of those links. That link brings up a web page, which is a set of unstructured data. Humans can read and understand news articles, videos, and Wikipedia pages. Google Search solves the problem of organizing and distributing all of the unstructured data across the web, for humans to consume. Diffbot is a company with a goal of solving a related, but distinctly different problem: how to derive structure from the unstructured web, understand relationships within that structure, and allow machines to utilize those relationships through APIs. Mike Tung is the founder of Diffbot. He joins the show to talk about the last decade that he has spent building artificial intelligence applications, from his research at Stanford to a mature, widely used product in Diffbot. I have built a few applications with Diffbot, and I encourage anyone who is a tinkerer or prototype builder to play around with it. It’s an API for accessing web pages as structured data. Diffbot crawls the entire web, parsing websites, using NLP and NLU to comprehend those pages, and using probabilistic estimations to draw relationships between entities. It’s an ambitious product, and Mike has been working on it for a long time. I enjoyed our conversation. We recently launched a new podcast: Fintech Daily! Fintech Daily is about payments, cryptocurrencies, trading, and the intersection between finance and technology. You can find it on fintechdaily.co or Apple and Google podcasts. We are looking for other hosts who want to participate. If you are interested in becoming a host, send us an email: host@fintechdaily.co

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

David Cancel has started five companies, most recently Drift. Drift is a conversational marketing and sales platform. David has a depth of engineering skills and a breadth of business experience that make him an amazing source of knowledge. In today’s episode, David discusses topics ranging from the technical details of making a machine learning-driven sales platform to the battle scars from his early career, when he spent a lot of time building products that people did not want. He has found success by focusing on building software that the market has shown a desire for. Chatbots were a popular, trendy subject a few years ago. The success of chatbots manifested in them fading into the background, and becoming a subtle, increasing part of our everyday interactions. Not every online interaction can be replaced by a chatbot, but many online interactions can be made more efficient by using chatbots. Chatbots can serve well-defined information, like product features, or the hours of operation of a business. When a chatbot gets a question that it cannot answer, the bot can route the conversation to a human. When a customer lands on a web page of a company using Drift, they see a chat box appear in the corner of the screen. The customer is able to communicate through that chat box with a bot that represents the company. The customer can learn about the product, schedule a call with a salesperson, and get other useful utilities from the Drift sales bot. The Drift chatbot messaging system is handled by Elixir/Erlang. Erlang is widely known as the messaging language that was used to scale WhatsApp while maintaining high availability. On the backend, Java services take the interactions from the Driftbot and pull it into a CRM, which allows sales and marketing people to manage information about the customers that are interacting with the chatbot. David gives lots more detail around the engineering stack, the deployment model, and his thoughts on the business and modern engineering. We recently launched a new podcast: Fintech Daily! Fintech Daily is about payments, cryptocurrencies, trading, and the intersection between finance and technology. You can find it on fintechdaily.co or Apple and Google podcasts. We are looking for other hosts who want to participate. If you are interested in becoming a host, send us an email: host@fintechdaily.co

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Engineers who start companies often find themselves building something they have no experience with: a hiring process. Hiring engineers today is not as systematic as building software. We don’t have lots of data that tells us what makes for an effective programming interview question. The smartest tech companies in the world are still making hiring mistakes–often through the “false negative” of rejecting candidates who did not do well in their interview process or through the “false positive” of hiring candidates who did well in the interview, but were not a good fit for the job. If you are a hiring manager or a company founder, you will eventually have to build a hiring process. If you don’t treat that hiring process scientifically, you will likely make some mistakes. Ammon Bartram has conducted more than 1000 interviews with engineers, accumulating a vast amount of data. This data was gathered deliberately and scientifically, through closely tracked interview questions and a consistent end-to-end process for the job candidate. Ammon joins the show to talk about the data set he has accumulated, the conclusions from all of these interviews, and how engineering organizations can use this data to develop a smart, data-driven hiring process. Ammon is co-founder of Triplebyte, a company that helps match engineers and tech companies. Triplebyte also publishes lots of research and blog articles about conducting good interviews, developer salaries, and bootcamps vs. computer science degrees. Full disclosure: Triplebyte is a sponsor of Software Engineering Daily. (However, Ammon has been a guest several times before on the show, since before Triplebyte was a sponsor, and I always enjoy getting to talk to him.)

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

I like to write music, and a year ago I started working on an album called “Gig Economy”. The plan for the album was to hire musicians from gig economy platforms like Fiverr and Upwork to perform on songs that I produced. The album is finished and I’m happy with the result, so I’m sharing it on the podcast today as an extra episode released on a Sunday. We’ll be back tomorrow with content about software engineering. Writing this album made me think about the future of work. I love music, but I am not a professional musician. I take the craft seriously, but perhaps not as seriously as someone who has made music their full-time career. I’ve never taken the time to network with musicians and develop collaborative relationships. The gig economy allowed me to pay creative people to collaborate with me. The transactional nature of my relationship with the collaborators meant that both of us had the incentive structure to work effectively. This improved quality and kept the pace of the project moving quickly. Today, the gig economy is widely regarded as a mode of work that removes individuality. Critics of the gig economy say that it turns workers into a commodity. If you use Uber, you aren’t thinking about the human who is driving you. As long as your driver has a high rating, you are satisfied. My experience producing this album with gig economy musicians showed a different side of our new employment systems. I found artists who I deeply enjoyed working with. There was no ambiguity about our relationship. Nobody was flaky. I paid these collaborators well, and in return they helped me fulfill an artistic vision. Without gig economy platforms, I could not have written this album. The gig economy is a playground for creativity. If you want to get paid to work as an artist, you can do so as long as you have a laptop. If you are a corporate worker making a good salary, but you spend your weekend producing art, you can pay artists to help you complete your vision. The greatest works of art are often the result of a talented workforce directed by an established leader. The gig economy lets you become a leader and recruit a team of creative, proven artists in a single day. I hope you enjoy “Gig Economy”. If you would rather listen on Spotify or YouTube, links are below. If you like the album, please share it on Twitter or Facebook. If you listened all the way through and have feedback for me, I’d love to know your thoughts. You can send me a tweet @the_prion or an email: jeff@softwareengineeringdaily.com

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

BloxRoute Labs is a company that is developing a blockchain distribution network (BDN). Much like a CDN pushes media files out to the edges of the web to make them faster to access, a BDN pushes out information to miners in the network. Of course, this means that the BDN could potentially be centralized infrastructure. In order to make the BDN effectively decentralized and trustworthy, BloxRoute claims to have a provably trustable network protocol, to go with its token-based incentive system that keeps its goals aligned with that of the larger blockchain world. Aleksander Kuzmanovic is the founder of BloxRoute Labs, as well as a professor of computer science at Northwestern University. His co-founders of BloxRoute include former guests of the show Emin Gun Sirer and Soumya Basu. The BloxRoute founders have a strong theoretical background and a great reputation in the cryptocurrency community, unlike the vast majority of founders who have issued a token. As we have heard in previous episodes, most of the founders of companies that issue tokens cannot give a good explanation for why their protocol needs a token. Tokens are a great idea that have mostly been applied as a mechanism to get rich quickly. However, BloxRoute has a credible explanation for their token, and I asked some very pointed questions to Aleksander in today’s episode to try to vet the project for legitimacy, and his reasoning made sense. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A smart security camera takes in a high volume of video images and processes those images using a set of machine learning models. Those models can be used to identify interesting snippets of movement throughout the day, and decide which of those snippets to keep. Some of the video snippets might contain movement of birds–but other video snippets might contain footage of intruders. As the video stream is processed on the smart security camera, and machine learning models are used to classify the entities in the video stream, some of the data gets thrown out as useless. Some of the data gets sent to the cloud for additional processing. Some of the data might trigger an alert that there is an intruder on the premises. Each piece of video data is an “event”. These events are processed and acted upon. Modern applications are highly interactive, and have lots of “events”. Other examples of event data streams are website traffic data, self-driving car data, time-series logging data, and video game session data. Building applications that respond to these high volumes of events requires us to program triggers to react to data streams, actions to take in response to the data streams, and workflows to orchestrate what the overall picture of our application is doing as the application is consuming this large data stream. Flogo is an event-driven ecosystem for building applications around streams of events. Leon Stigter and Matt Ellis work on Flogo at TIBCO, and they join the show to discuss event-driven application development and their work on Flogo. They also talk about the constraints of machine learning applications at the edge, and how event processing systems like Flogo can be used to handle large data streams on edge devices.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Engineers can build applications faster by using tools that abstract away infrastructure. Major cloud providers offer this tooling in the form of functions-as-a-service, as well as managed services such as Google BigQuery or Azure Container Instances. The term “serverless” refers to these functions-as-a-service and the managed services–because when you use these tools, you are not making calls to specific servers–you are making calls to APIs that abstract away the servers from you, while guaranteeing uptime and reliability. In previous shows we have covered Heroku, Firebase, serverless functions, serverless event driven application development, and a few startups that are built almost entirely on serverless infrastructure. “Serverless” is a way of describing backend services that are represented by an API. But what about the rest of the application stack that you use to build on top of serverless? You still need to use JavaScript to define the custom code of your application. You still need to use HTML markup to describe the look and feel of your application. The “JAM Stack” is a way of building applications consisting of JavaScript, APIs, and markup. Phil Hawksworth is the head of developer relations at Netlify, and he joins the podcast to explain how these JAM Stack applications are developed and deployed, and how developers can use the JAM stack to rapidly build new systems. We recently launched a new podcast: Fintech Daily! Fintech Daily is about payments, cryptocurrencies, trading, and the intersection between finance and technology. You can find it on fintechdaily.co or Apple and Google podcasts. We are looking for other hosts who want to participate. If you are interested in becoming a host, send us an email: host@fintechdaily.co

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

DevSecOps emphasizes moving security out of a siloed audit process and distributing security practices throughout the software supply chain. In the past, software development usually followed a waterfall development process. Each step in building software was serialized, one after another. First, software was planned. Then it was built. Then it was tested. Finally, the software received a security audit at the end. If a security vulnerability was not discovered during that audit, it was likely that the software would be released with the vulnerability. With continuous delivery, we can be continuously checking for security. Every new release can be tested against a battery of automated security tests. The open source libraries we use can be scanned to make sure they are up-to-date with patched versions. Static analysis can discover memory leaks and buffer overrun vulnerabilities. Edward Thomson is the principal program manager for Azure DevOps at Microsoft. He joins the show to talk about how an organization can adopt DevSecOps and introduce security practices into continuous delivery pipelines. We also talk more philosophically about security–defining the most common security risks of a software company today, from “shadow IT infrastructure” to phishing. Full disclosure: Microsoft is a sponsor of Software Engineering Daily. We recently launched a new podcast: Fintech Daily! Fintech Daily is about payments, cryptocurrencies, trading, and the intersection between finance and technology. You can find it on fintechdaily.co or Apple and Google podcasts. We are looking for other hosts who want to participate. If you are interested in becoming a host, send us an email: host@fintechdaily.co

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google Search is a highly interactive JavaScript application. As you enter a query, results are being automatically suggested to you before you even finish typing. When you press enter, some of your search results may be widgets that represent the weather, the price of a stock, a recipe for green bean soup, or a language translation for a phrase. These complex frontend components are loading dynamically–the Google Search application cannot prefetch every single possible widget that you might ask for–but the results do load very quickly. Google has many other examples of advanced JavaScript engineering. The company is mostly known for backend engineering inventions like MapReduce, TensorFlow, Dremel, and Spanner. To turn these backend tools into user facing products, Google develops its own JavaScript frameworks and infrastructure to deliver information from the backend to the frontend. “Backend” and “frontend” are not precise terms. At Google, there are so many layers of infrastructure between a user and the data center. If you are an engineer working on a service at Google, you probably have several “frontends” and “backends” on either side of you. Malte Ubl is a senior staff engineer at Google. He’s heavily involved in Google’s JavaScript infrastructure, and has written about managing large JavaScript applications in detail. He also works on AMP, an open-source project for delivering web pages in a fast, performant format. He joins the show to describe Google’s history with JavaScript frameworks, the process of building frontends and middleware to deliver JavaScript applications, and the engineering behind AMP. There are criticisms of AMP, but some of them misunderstand how the AMP technology actually works. AMP allows pages to be cached, prefetched, and served to a user more quickly. AMP does not necessarily centralize pages around being served from Google Search. A good example of AMP speeding up pages outside of Google is reddit. We recently launched a new podcast: Fintech Daily! Fintech Daily is about payments, cryptocurrencies, trading, and the intersection between finance and technology. You can find it on fintechdaily.co or Apple and Google podcasts. We are looking for other hosts who want to participate. If you are interested in becoming a host, send us an email: host@fintechdaily.co

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Deepak Singh is the director of compute services at AWS, where he works on cloud products relating to containers, Linux, and High Performance Computing. In today’s show, Deepak describes how the market for containers and serverless has evolved, and how Amazon thinks about product strategy. Back in 2014, Docker containers were becoming a popular way to deploy and manage application infrastructure. Containers allowed people to take advantage of their servers in a more economical way. Containers let developers move faster by quickly setting up and tearing down small composable units of software. As these containers grew in number within software companies, companies started figuring out that they needed tooling to manage and orchestrate all these containers. Infrastructure software companies realized that there would be a big business in providing orchestration software to developers who needed to manage these high volumes of containers. This led to the “container orchestration wars”, in which a variety of companies such as Red Hat, CoreOS, Docker, and Mesosphere all began to offer platforms for managing containerized applications. During the container orchestration wars, many large enterprises such as banks and telcos resisted picking any specific container orchestration system because there was no clear winner. Enterprises were hesitant to place a large bet on an infrastructure orchestration tool that might go out of fashion. Amazon had a large number of customers that wanted to orchestrate their containers, but it was unclear how the market for open source container orchestration was going to unfold. Around this time, Amazon created ECS, a closed-source container orchestration system. In the following years, Kubernetes was released and became the most popular container orchestrator. Amazon released EKS, a managed Kubernetes service. They also released AWS Lambda for running serverless functions and AWS Fargate for spinning up long-lived container instances. Deepak and I discuss the history of containers at Amazon, but we also discuss how developer preferences are changing towards managed services, and how AWS is able to continually build off of its own tools to build higher and higher level services for developers.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Linux has created much more value for Google than it has for Linus Torvalds. Ruby on Rails has created more value for Airbnb than it has for David Heinemeier Hansson. Successful open source projects create more value than their creators’ capture–and that’s one reason why collections of people on the Internet are often inspired to work together on open source. When an engineer creates an open source project, and that open source project finds a large audience, that engineer can often build a successful business. SpringSource, Cloudera, and Elastic are examples of massively successful enterprises that were founded by the creators of open source software. But in other cases, the value of an open source project gets largely captured by cloud providers that create a closed source version of the open source project and offer it as a service. Shaun Connolly has worked in senior strategic roles at software companies such as SpringSource, VMWare, and Hortonworks. Throughout his decades of experience, much of his time has been spent figuring out how to monetize open source projects intelligently. Shaun joins the show to talk about his past experiences building enterprises, as well as modern issues–such as how to compete with major cloud providers. We also discuss the Commons Clause license, a new software license that open source projects can use to try to protect their value from being entirely captured by a cloud provider. Software Engineering Daily is looking for sponsors. If you are interested in reaching over 50,000 developers, you can go to softwareengineeringdaily.com/sponsor to find out more, and you can send us a message. We’d love to hear from you. And if you are an engineer working at a company that is marketing to developers, or hiring developers, if you tell your marketing department or your recruiting department about softwareengineeringdaily.com/sponsor, that is one way to help us out.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Search is part of almost every application. Users search for movies to watch. Engineers search through terabytes of log messages to find exceptions. Drivers search through maps to find a destination. Search remains an unsolved problem, with lots of room for optimization. Many search applications have been built Elasticsearch, an open source distributed search engine. Elasticsearch is the code that powers some search-as-a-service products offered by major cloud providers. After eight years of open source development, Elasticsearch is excellent at core search functionalities, such as indexing data, sharding, and serving queries. With improved access to machine learning tools, search applications can advance in new and interesting ways. For example, an incoming search query can be sent to an API for natural language processing before being served by the search engine. A natural language processing API can derive additional meaning from the query, adding metadata to a search query. Machine learning can also be applied to better understand how people are searching across your search index, and to optimize the search index to incorporate those user preferences. Liam Cavanagh is the principal program manager on Azure Search. He joins the show to talk about the architecture of a search index, how search queries are served by an index, and how machine learning APIs can be used to improve queries.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Virtual machines are operating system instances that run alongside each other on the same physical host. The virtual machines running on a physical host are managed by a hypervisor running on the physical host. A cluster of two physical servers could have four virtual machines running across those two physical instances. Those four virtual machines can communicate over a virtual switch. A network switch allows packets of bytes to be routed between machines. With a physical network switch, a dedicated physical device sits in the computer network to do this routing. A virtual network switch provides this packet routing without needing a dedicated physical hardware device for routing. Open vSwitch is a distributed virtual multilayer switch. Open vSwitch provides network switching for hardware virtualization environments. Ben Pfaff is a core contributor to Open vSwitch, and he joins the show to talk about operating system virtualization. Ben was an early employee at Nicira, a company that made significant developments in software-defined networking before being acquired by VMware in 2012.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A data warehouse provides fast access to large data sets for analytics, data science, and dashboards. A data warehouse differs from a transactional database, because you often do not need to update specific records. Because of the read-only nature of the access patterns, and the high volumes of data being queried, the design of a data warehouse is very different than a transactional database. With a transactional database (such as MySQL or MongoDB), it is important to have consistency guarantees. For example, consider a transactional database that serves as the backend for banking applications. If multiple frontend servers are hitting that transactional database to withdraw money, you need the records to be quickly updated. You need to avoid race conditions, so that two servers cannot withdraw the entire bank account balance simultaneously from different locations. In contrast to transactional databases, a data warehouse is often used to process a query that encompasses a big data set. For example, Netflix might want to answer the question: “how many users that watched House of Cards also watched Black Mirror?” Netflix has a lot of users, so they will want to be accessing those user records in a way that lets them scan through the records quickly. Christian Kleinerman is the VP of product at Snowflake Computing. Snowflake’s main product is a cloud data warehouse. In today’s show, we talk about the difference between a data warehouse, a data lake, and a transactional database, and the process of moving data sets between them, often known as ETL. This show continues our series on data engineering and data platforms. As companies accumulate more and more data, the complexity of managing that data and taking full advantage of it is escalating. Christian gives his perspective on these changing trends, and describes the plans for Snowflake to evolve as a business.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Windows operating system is one of the most widely used pieces of software in history. Windows was started before there was any alternative to a monolithic codebase, because Microsoft was building software before the Internet was widely used by consumers. Networked computers gave rise to web applications, and software engineers began to rethink how to build everything. Software development got reimagined with agile. Monolithic codebases got broken up into service-oriented architecture. Instead of going to a store to buy a box with software in it, users downloaded software from the Internet, and that software was regularly updated. Software that is regularly updated needs to be regularly tested. Instead of a single round of testing for every round of boxed software that was shipped to a store, continuous testing and delivery gradually became the norm. The process of releasing and operating software became its own set of engineering challenges–which was tackled by the “operations” or “sys admin” team at a software company. Now there were two different sets of engineers–those who were developing the software and those who were operating the software. The incentives of these two types of engineers were not completely aligned. The software developers wanted to build software quickly and release new features. The operators wanted things to release software slowly, because if something broke then the operators were the first line of defense for fixing it. These problems between development and operations gave rise to the “DevOps” movement, in which developers and operations started working more closely together and sharing responsibilities. Incentives became aligned, and new types of software was created to facilitate more harmonious relationships between developers and operations–for example, continuous delivery pipelines. Today, most enterprises are still undergoing a transformation from monolithic software release cycles to continuous delivery. This is often referred to as a “DevOps Transformation”. A DevOps Transformation requires the entire organization to reorient itself around faster software release cycles. This can be a painful process, and we have covered it in many past shows. Hearing case studies from enterprises can be helpful for figuring out how to reorient your own enterprise. Microsoft is a useful case study in shifting towards DevOps. Windows is perhaps the biggest monolithic codebase in history. The fact that Microsoft could rearchitect Windows to be easier to work with should provide some reassurance to other enterprises who are currently undergoing their own migrations. Martin Woodward has been at Microsoft for 13 years and he joins the show to talk about how software delivery within the company has evolved. We discussed the move from boxed software delivery to delivery via the cloud, and focused on a few specific, longstanding products such as Windows. Martin has been part of the effort to build Azure DevOps, which is a product that offers similar tools to the ones Microsoft built internally for DevOps as a service. We also talk about the specific difficulties that enterprises often have when moving toward DevOps. Full disclosure: Microsoft is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google Brain is an engineering team focused on deep learning research and applications. One growing area of interest within Google Brain is that of generative models. A generative model uses neural networks and a large data set to create new data similar to the ones that the network has seen before. One approach to making use of generative models is GANs: generative adversarial networks. GANs can use a generative model (which creates new examples) together with a discriminator model (which can classify examples). As an example, let’s take the task of generating new pictures of cats. We want an artificial cat picture generator. First, we train a discriminator by feeding it billions of example pictures of cats. We now have a model that can tell what a cat is. Next, we make a model that generates completely random images. We feed those randomly generated images to the discriminator. The discriminator outputs a “loss” for these random images. Loss is a metric we can use to represent how far off a given image is from being something that the discriminator would recognize as a cat. Finally, you can feed this “loss” back into the generative model, so that the generative model will adjust its weights in a way that will reduce loss. Over time, the generator gets better and better at reducing loss, until the discriminator starts believing that some of these semi-random images are actually cats. Generative model systems have produced useful applications, such as object detection, image editing, and text-to-image generation. Today’s guest Doug Eck works on the Magenta team at Google Brain. Magenta uses applications of deep learning to produce tools and experiments around music, art, and creativity. In a previous show, Doug described his vision for humans and computers to work together on creative tasks such as music. Today, we dive into some of the core machine learning building blocks that make machine creativity possible.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Mapillary is a platform for gathering photos taken by smartphones and using that data to build a 3D model of the world. Mapillary’s model of the world includes labeled objects such as traffic signs, trees, humans, and buildings. This 3D model can be explored much like you can explore Google Street view. The data set that underlies Mapillary is crowdsourced from volunteer users who are taking pictures from different vantage points. These smartphone photos are uploaded to Mapillary, queued, and processed to constantly update and refine the Mapillary model. Mapillary processes high volumes of photos from around the world. The images in these photos need to be correctly fit into Mapillary’s model of the world like a puzzle piece sliding into place. The image needs to be segmented into the different entities within, and those entities need to be put through object recognition algorithm. When two pictures have a conflict, that conflict needs to be resolved. Mapillary is full of interesting engineering problems. The high volume of images and the level of processing has created the need for a unique sequence of indexing, queueing, and distributed processing using Apache Storm. In addition to processing all of this data and building a 3-D model, Mapillary serves an API for querying geolocations about traffic signs, road conditions, and bus stops. Peter Neubauer is the co-founder of Mapillary, and is also a co-founder of Neo Technology, the company behind Neo4j. Peter is a world-class engineer and he joins the show to give a detailed overview of the technology behind Mapillary, from ingressing the photos to running data engineering jobs to serving the API.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When Aran Khanna was a college student, he accepted an internship to work at Facebook. Even before his internship started, he started playing around with Facebook’s APIs and applications. Aran built a Chrome extension called Marauder’s Map, which used Facebook Messenger’s web APIs to track where people lived, what their schedule was, and other highly sensitive information. These were not public features of Messenger, but Aran was able to reverse engineer the APIs. As a result, of making Marauder’s Map, Aran’s invitation to work at Facebook was retracted. Aran remained curious about the norms of publicly available social network data, and the second order data sets that could be built on top. Out of this curiosity, Aran created a tool called Money Trail, which used public Venmo data to model a graph of how users were paying each other. Aran showed for a second time that data that seems innocent to share can be repurposed to identify, classify, and incriminate users. Developers of these online applications face tradeoffs between privacy, convenience, and security. By interacting with these applications, we generate data that suggests how we think, what we like to do, and who we are affiliating with. Google and Facebook probably understand you better than you understand yourself. Aran Khanna previously was on the show to talk about machine learning at the edge. At the time he worked at Amazon Web Services. He now works as a digital privacy researcher. His background in machine learning makes him well-equipped to talk through the subtleties of modern digital privacy. In this show, Aran returns to talk through the finer points of privacy, data, and artificial intelligence.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Airbnb began in 2008 as a monolithic Rails application serving the simple purpose of listing homes for rental. Over time, the number of listings increased dramatically, as did the number of people who were renting. With that scale, the Rails app had to be broken into different services, and entire teams were built out to focus on challenges such as pricing, application infrastructure, and search. Surabhi Gupta joined in 2013 to work on the search team, and has worked on different teams at Airbnb over time. Today she is a director of engineering leading the Homes business for Airbnb, which includes Growth, Search, Hosts, Pricing, and Business Travel. Surabhi has helped scale Airbnb through a hypergrowth period, and joins the show to share those experiences. One distinct area that we spent time on was Airbnb’s search engine. Surabhi formerly worked at Google, and she described how the engineering problem of a search engine for homes differs from a general purpose search engine like Google.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

We previously released this episode with the wrong audio file and are re-releasing it on a weekend. TIBCO was started in the 90’s with a popular message bus product that was widely used by finance companies, logistics providers, and other systems with high throughput. As TIBCO grew in popularity, the company expanded into other areas through products it developed in-house as well as through acquisitions. One acquisition was Jaspersoft, a business intelligence data platform. When TIBCO acquired Jaspersoft in 2014, the architecture was a monolithic Java application. Around this time, customer use cases were shifting from centralized reporting to real-time, embedded visualizations. The use case of the Jaspersoft software was becoming less centralized and less monolithic and the software architecture needed to change in order to reflect that. Jan Schiffman is a VP of engineering at TIBCO and Sherman Wood is a director at TIBCO. They join the show to discuss the process of migrating a large Java monolith to a composable set of services. Breaking up a monolith is not an easy process–nor is it something that every company should do just because they have a monolith. In some cases, a monolith is just fine. Jan and Sherman explain why the business use case for why the Jaspersoft monolith needed to be refactored, and their approach to the refactoring. We also talk through the modern use cases of embedded analytics and the interaction between business analysts and data engineers. At a higher level, we discuss the lessons they have learned from managing a large, complex refactoring. Full disclosure: TIBCO is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Log messages are fast, high volume, unstructured data. Logs are often the source of metrics, alerts, and dashboards, so these critical systems are downstream from a log management system. A log management system needs to be highly available, so that a failure in one part of your system will not be correlated with failure of the log management system. Users of a log management system are often building tools based off of the query engine of that log management system. For example, I might build a dashboard that gives me a line graph representing the number of times a certain log message is alerting me due to a memory warning. I write a query to return the instances of these memory warnings, and my line graph is a visual representation that query. A log management system needs to be able to quickly serve users that are querying their logs–whether for dashboards or for ad-hoc queries. When logs are ingested by a log management system, the logs get parsed in a way that can bring some structure to the blob of text that is a raw log message. Some log management systems will then add the log message to an index. An index can allow for very fast lookups of particular types of queries. But an index also has certain constraints–such as processing regular expression queries. Steve Newman is the CEO and founder of Scalyr, a log management system that uses a column-oriented data storage system instead of the more conventional index-based log management systems. Today’s episode is a great case study in distributed systems tradeoffs. Steve talks in great detail about how Scalyr maintains high uptime, and its system for ingesting logs and serving queries.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a database gets large, it can start to perform poorly. This can manifest in slow query speed. You can speed up a query by defining an index, which is a data structure that allows for faster access to the data that is being indexed. As a consequence, whenever you update the database, you will now need to update the index with that new piece of data. The more you index your data, the faster the access time. In order to have more indexes you must pay a write penalty in order to maintain consistency around that data, since the indexes need to be updated with each new entry. This illustrates one simple tradeoff that a developer can make within a database deployment. Why are there so many different databases in the world? Why do we need SQL databases like Postgres, document databases like MongoDB, key/value systems like Cassandra, and search systems like Elasticsearch? Because each of these each system optimizes for different sets of tradeoffs. Tradeoffs can affect the speed of a read, the speed of a write, the user experience, the consistency of data, and the cost of running the database. Andrew Davidson is the lead product manager of MongoDB Atlas. Andrew joins the show to talk about how database performance can degrade when a database gets large, and how to measure and optimize performance of a critical database. Andrew explores the range of distributed systems cases–from a single node database to a multi-geographic distribution of nodes around the world, and describes how the configuration of a database in the cloud can help or hurt the application that the database is serving. Full disclosure: MongoDB is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Linkedin is an organization with thousands of employees. An enterprise of that size starts to develop problems with data collaboration. Data collaboration is the process of sharing and analyzing data with multiple users, such as data scientists, business analysts, and engineers. How do data scientists know what questions to ask? How do business analysts know the right way to query a database? How does a data engineer even find where the right database is within the company infrastructure? And how can these different users share information with each other so that redundant work is avoided? When Adam Weinstein was at Linkedin, he saw these problems firsthand. The process of accessing and utilizing data felt slow and broken. Engineers were searching through a company wiki to find out how to leverage data, and the wiki was often out of date. When an engineer would leave the company, there was not a durable, institutional memory of how that engineer worked with data. Adam used this experience as inspiration for Cursor, a tool for data collaboration. Cursor allows different users in the data pipeline to share data sets, queries, access patterns, and comments about data within a company. Cursor is used by Linkedin, Slack, Apple, and other companies. Adam is the CEO of Cursor, and he joins the show for an interview about the problems and opportunities of data collaboration.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kotlin is a statically typed programming language that started as a JVM language. It gained popularity because it reduces the amount of boilerplate code required for a typical Java project. Many of the early adopters of Kotlin were building Android apps or Java applications, but it has grown to a variety of use cases including at companies like Uber, Pinterest, and Atlassian. Andrey Breslav is the lead language designer of Kotlin at JetBrains. He joins the show to describe the original goals of Kotlin, the compilation path of the language, and how it has moved beyond its days of only running on the JVM.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Open source software is key to our software infrastructure. Closed source enterprises rely on open source software, but the development processes for closed source and open source software are often different in their approach to continuous integration and delivery. Oren Novotny is a chief architect of DevOps and modern software at BlueMetal Architects where he works with a variety of clients to build products and internal applications. Oren spends lots of time developing open source software for his job as well as during his spare time. He’s been in the software industry for more than 15 years, and has a wide breadth of insights from different businesses in how they apply software. We started the conversation talking about electronic trading companies, which in some ways operate like large enterprises and in other ways operate like startups. Oren described working in the financial industry through the 2008 crisis, then switching industries to work at Microsoft, before coming to BlueMetal Architects. We then discussed the process of setting up continuous integration for an open source project–including the difficulties and the large benefits for adding continuous integration to an open source project.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

GraphQL allows developers to communicate with all of their different data backends through a consistent query interface. A GraphQL query can be translated into queries to MySQL, MongoDB, ElasticSearch, or whatever kind of API or backend is needed to fulfill the GraphQL query. GraphQL users need to set up a GraphQL server to fulfill this query federation. Prisma is a tool for automatically generating a GraphQL API and serving GraphQL queries. The developer defines a data model and deploys with Prisma. Prisma generates the necessary GraphQL infrastructure to serve queries from the developer’s database. This can allow the developer to get up and running faster than they would setting up GraphQL infrastructure and defining the middleware query layer by hand. Prisma is an open source project, but it is also a company. The opportunities to build a business around a GraphQL infrastructure layer are numerous. In recent episodes, we have explored the complexities of the “data platform.” From newer companies like Uber to older companies like Procter and Gamble, engineers are struggling to find and access their data sources. Data engineers and data scientists spend months configuring their infrastructure to connect to BI tools and run distributed queries. GraphQL could simplify data platforms by providing a unified, standardized layer. At this layer, you could also offer caching, virtual data sets, and crowdsourced queries from across the company. Soren Bramer Schmidt is the CTO and co-founder of Prisma, and he joins the show to discuss why GraphQL has become so popular, how Prisma works, and the opportunities to build developer tooling around GraphQL.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Internet of Things is a concept that describes lots of devices that you interact with regularly being connected to the Internet and networked together. Technologists have been dreaming of the world of IoT for many years, where our connected refrigerator can detect that we are out of food, and automatically order more food. Or our connected bathroom can scan us for diseases and recommend treatment. The bright future of IoT is slowly coming together. Hardware prototyping is getting cheaper. Voice interfaces and machine learning are creating new mediums for communicating with devices. Platforms like Kickstarter are allowing developers to validate the market for their products and raise the necessary capital to build their product. Android Things is a developer platform for IoT applications based on the Android Operating system. Android Things consists of hardware devices and software tools that reduce common IoT problems such as software updates and security patches. Wayne Piekarski is a staff developer advocate at Google, and he joins the show to talk about the state of IoT and why Google built Android Things.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

JavaScript performance has improved over time due to advances in JavaScript engines such as Google’s V8. A JavaScript engine performs compiler optimization, garbage collection, hot code management, caching, and other runtime aspects that keep a JavaScript program running efficiently. JavaScript runs in browsers and servers. The resources that are available to a JavaScript engine vary widely across different machines. JavaScript code is parsed into an abstract tree before being handed off to the compiler toolchain, in which one or more optimizing compilers produce efficient low-level code. In recent shows about WebAssembly, we have covered compiler pipelines. In an episode about GraalVM, we explored the impact that “code shape” has on the efficiency of JavaScript execution. Mathias Bynens is a developer advocate at Google working on the V8 JavaScript engine team. In today’s show we explore how a JavaScript engine works, and how compiler toolchains can adapt the hot code paths depending on what code needs to be optimized for.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Unity is a game engine for building 2-D and 3-D experiences, augmented reality, movies, and other applications. Unity is cross-platform, so that applications can be written once and deployed to iOS, Android, web, and other surfaces. Unity has been around for 13 years, and has grown in popularity with the rise in gaming and game development. Brett Bibby is VP of engineering at Unity, and he joins the show to describe how Unity applications are built. Since Unity SDKs allow Unity code to run across all the different platforms, this requires writing and maintaining native code libraries for each of these devices. When asm.js came out, Unity developers were able to deploy 3-D games to the web–these were some of the first examples of asm.js being used. Asm.js is a small, performant subset of JavaScript that other languages could compile down into. So in this case, Unity programs in C# were running in the browser after being compiled down into asm.js. Since then, WebAssembly has improved the tooling further, allowing a high-performance compilation path for non-JavaScript programs. After exploring the basics of Unity, Brett described how Unity works with WebAssembly, and the potential for creative applications of Unity both on and off the web.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Front is a shared inbox application that has seen rapid adoption within companies. Front allows multiple members of a company to collaborate together on a conversation–whether that conversation is in email, Twitter, or Facebook Messenger. This is useful when a customer email needs to be shared between the sales and engineering teams, or when a single email address is shared between different members of the same team, such as “contact@softwareengineeringdaily.com”. This might sound like a niche problem, but it is actually a problem faced somewhere within every single company. Because the problem of shared inbox is so prevalent, the company has grown its user base quickly, scaling the team as well as the infrastructure. The sensitivity of the data (emails) that Front is handling means that security is paramount. And as users of Front rely on it more and more as a central point of communication, uptime and consistency needs to be maintained. Laurent Perrin is the CTO at Front, and he joins the show to describe the software architecture and product strategy for Front. It was a fascinating show, and we covered the full stack. On the backend, Front pulls emails into S3 buckets and maintains the schema of the inbox in a SQL database. The desktop Front client is written in Electron, which is a way to write desktop applications in HTML5, JavaScript, and CSS. We also talked about the system for keeping the communications “real-time”–it’s important that users are aware of what each other is doing, since you don’t want to be preparing a response to an email at the same time I am.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Background checks are a routine part of the hiring process. After a potential employee has made it through job interviews, a background check is administered to look through the applicant’s work history, criminal record, and other available data. Conducting a conventional background check can require manual work–including phone calls for reference checks and going to a courthouse to look up physical records of a person’s criminal history. The on-demand economy has rapidly increased the volume of workers who are getting hired–and all of them need background checks. Lyft drivers, DoorDash food delivery people, Instacart shoppers–these on-demand workers are being trusted with our lives. We get into their cars, let them into our houses, and eat the food that they hand to us. We want some guarantees about their reputation. Checkr is a background check platform that allows companies to request background check services via API request. Checkr was started 4 years ago, and has benefitted from the growth of gig economy services like ridesharing and food delivery. Since the background check API product has found success, Checkr has raised additional capital and invested in other new products: a next-generation background check product based on machine learning, and a mobile app that allows people to instantly background check themselves and find jobs that align with the results of that background screening. Tomas Barreto is the VP of product and engineering at Checkr and he joins the show to describe how the core Checkr API product works, and the challenges of automating the background check process. We also explored the product development roadmap for Checkr, and the product opportunities that come from building within a specialized vertical such as background checks.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google has two consumer operating systems: Android and Chrome. The Android operating system has been widely deployed on mobile devices. Chrome is an operating system for laptops and tablets, originally based around the Chrome browser. For several years, these two ecosystems were mostly separate–you could not run Android apps on a Chrome operating system. Shahid Hussain and Stefan Kuhne are engineers at Google who worked on support for Android apps on ChromeOS. The implementation of Android on Chrome involves running the Android OS in a Linux container on the host Chrome operating system. In today’s episode, Shahid and Stefan compare the Android and Chrome operating system platforms. They explain why Google has two different consumer operating systems, and the advantages of allowing Android apps to deploy to Chrome. Shahid and Stefan also talk about the challenges of porting mobile applications to ChromeOS. Android apps are made to run on small screens and tablets. In order to make them run on ChromeOS, the applications need to support running on a desktop or laptop.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes is an open source container management system. Kubernetes is sometimes described as “the Linux of distributed systems” and this description makes sense: the large numbers of users and contributors in the Kubernetes community is comparable to the volume of Linux adopters in its early days. There are many different distributions of Linux: Ubuntu, Red Hat, Chromium OS. These different operating system distributions were created to fulfill different needs. Linux is used for Raspberry Pis, Android phones, and enterprise workstations. These different use cases require different configurations of an operating system. Similarly, there are different distributions of Kubernetes because there are different types of distributed systems. The internal infrastructure of a cloud provider might use one type of Kubernetes to serve users running application containers. A network of smart security cameras might be networked together with a different distribution of Kubernetes. Brian Gracely and Michael Hausenblas join the show today to discuss Kubernetes distributions. Brian and Michael work at Red Hat, which helps maintain the Origin Community Distribution of Kubernetes, which Red Hat OpenShift runs on. OpenShift is a platform as a service that enterprises use to deploy and manage their applications. Full disclosure: Red Hat is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Continuous integration and delivery allows teams to move faster by allowing developers to ship code independently of each other. A multi-stage CD pipeline might consist of development, staging, testing, and production. At each of these stages, a new piece of code undergoes additional tests, so that when the code finally makes it to production, the developers can be certain it won’t break the rest of the project. In a company, the different engineers working on a software project are given the permissions to ship code through a continuous delivery pipeline. Employees at a company have a strong incentive not to push buggy code to production. But what about open source contributors? What does the ideal continuous delivery workflow look like for an open source project? Abel Wang works on Azure Pipelines, a continuous integration and delivery tool from Microsoft. Azure Pipelines is designed to work with open source projects as well as companies. Abel joins the show to talk about using continuous integration and delivery within open source, and the process of designing a CI/CD tool that can work in any language and environment. Full disclosure: Microsoft is a sponsor of SE Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The DEV Community is a platform where developers share ideas, programming advice, and tools. Ben Halpern started it after running an extremely successful Twitter account creating humorous tweets for developers. One way to describe DEV Community is as a cross between Medium, Stack Overflow, and Reddit–but it has its own personality, so I recommend checking it out. The DEV Community was open sourced, and we discussed the challenges and the opportunities of having an open source social network. We also talked about his plans for the future, and where he is taking DEV Community. Ben is an entrepreneur who tries lots of different creative projects, so his perspective has always resonated with me. Ben has been on the show a few times before, when we talked about the state of developer media, side projects, and the identity of the software engineer. DEV Community was originally called Practical Dev, which I mistakenly referred to it as in the earlier parts of the show.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern applications produce large numbers of events. These events can be users clicking, IoT sensors accumulating data, or log messages. The cost of cloud storage and compute continues to drop, so engineers can afford to build applications around these high volumes of events, and a variety of tools have been developed to process them. Apache Kafka is widely used to store and queue these streams of data, and Apache Spark and Apache Flink are stream processing systems that are used to perform general purpose computations across this event stream data. Kafka, Spark, and Flink are great general purpose tools, but there is also room for a more narrow set of distributed systems tools to support high volume event data. Apache Druid is an open source database built for high performance, read only analytic workloads. Druid has a useful combination of features for event data workloads, including a column-oriented storage system, automatic search indexing, and a horizontally scalable architecture. Druid’s feature set allows for new types of analytics applications to be built on top of it, including search applications, dashboards, and ad-hoc analytics. Fangjin Yang is a core contributor to Druid and the CEO of Imply.io, a company that makes a storage, querying, and visualization tool build on top of Druid. He joins the show to talk about the architecture of Druid and his company Imply.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A company runs a variety of distributed systems applications such as Hadoop for batch processing jobs, Spark for data science, and Kubernetes for container management. These distributed systems tools can run on-prem, in a cloud provider, or in a hybrid system that uses on-prem and cloud infrastructure. Some enterprises use VMs, some use bare metal, some use both. Mesosphere is a company that was started to abstract the complexity of resource management away from the application developer. Instead of a developer managing virtual machines, provisioning cloud infrastructure, or wiring all that infrastructure together to run distributed applications, the developer spins up distributed applications like Kubernetes, Spark, or Jenkins on top of Mesosphere, and Mesosphere provisions the machines on the underlying infrastructure. Using Kubernetes on top of Mesos allows you to separate resource provisioning from the actual container orchestration. In a previous episode, we explored how Netflix uses Mesos with a container orchestrator on top to simplify the resource management of microservice application containers as well as data science workloads. Chris Gaun is a product manager at Mesosphere who helped build Kubernetes-as-a-service. In today’s show, he describes why it is useful to have separate layers for resource provisioning and container orchestration. He also talks about the difficulties of manually installing Kubernetes, and why Mesosphere built a Kubernetes-as-a-service product. Full disclosure: Mesosphere is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Netflix users stream terabytes of data from the cloud to their devices every day. During a high bandwidth, long-lived connection, a lot can go wrong. Networks can drop packets, machines can run out of memory, and the Netflix app on a user’s device can have a bug. All of these events can result in a bad user experience. Other errors can occur that do not disrupt the user experience. Netflix runs thousands of machine learning jobs, logging servers, and other pieces of internal infrastructure. Customer service dashboards, CI/CD pipelines, and AB testing frameworks are all software built by Netflix–and when an error occurs in any of these places, engineers need to be able to diagnose and debug that error. Observability is the practice of using logs, monitoring, metrics, and distributed tracing to understand how a system is working. Kevin Lew is a senior software engineer at Netflix with the Edge Insights team. He joins the show to talk about adding observability across the microservices deployed at Netflix. We also talk about how to manage high volumes of logging data effectively using stream processing.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Stock traders have access to high volumes of information to help them make decisions on whether to buy an asset. A trader who is considering buying a share of Google stock can find charts, reports, and statistical tools to help with their decision. There are a variety of machine learning products to help a technical investor create models of how a stock price might change in the future. Real estate investors do not have access to the same data and tooling. Most people who invest in apartment buildings are using a combination of experience, news, and basic reports. Real estate data is very different from stock data. Real estate assets are not fungible–each one is arguably unique from all others, whereas one share of Google stock is the same as another share. But there are commonalities between real estate assets. Just like collaborative filtering can be applied to find a new movie that is similar to the ones you have watched on Netflix, comparable analysis can be used to find an apartment building that is very similar to another apartment building which recently appreciated in asset value. Skyline.ai is a company that is building tools and machine learning models for real estate investors. Or Hiltch is the CTO at Skyline.ai and he joins the show to explain how to apply machine learning to real estate investing. He also describes the mostly serverless architecture of the company. This is one of the first companies we have talked to that is so heavily on managed services and functions-as-a-service.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Engineering organizations can operate more efficiently by working with a continuous integration and continuous deployment workflow. Continuous integration is the process of automatically building and deploying code that gets pushed to a remote repository. Continuous deployment is the process of moving that code through a pipeline of environments, from dev to test to production. At each stage, the engineers feel increasingly safe that the code will not break the user experience. When a company adopts Kubernetes, the workflow for deploying software within that company might need to be refactored. If the company starts to deploy containers in production, and managing those containers using Kubernetes, the company will also want to have a testing pipeline that emulates the production environment using containers and Kubernetes. Sheroy Marker is the head of technology at ThoughtWorks products, where he works on GoCD, a continuous delivery tool. Sheroy joins the show to talk about how Kubernetes affects continuous delivery workflows, and the process of building out Kubernetes integrations for GoCD. We also discussed the landscape of continuous delivery tools–why there are so many continuous delivery tools, and the question of how to choose a continuous delivery product if you are implementing CD. Continuous delivery tooling is in some ways like the space of monitoring, logging, and analytics–there are lots of successful products in the market. Full disclosure: ThoughtWorks and GoCD are sponsors of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Engineers need to have an awareness of the business model that allows their company to succeed. When a software company is going to market, the engineers need to work closely with the sales and marketing team to formulate a strategy for building and selling that product. This is especially true in highly technical products, such as database- or platform-as-a-service companies. An engineer at a Hadoop-as-a-service product needs to work with the sales and marketing team to explain why a customer might want a data platform. An engineer at a SaaS company needs to understand how the cost to provide a service might scale, so that the sales team can decide on appropriate pricing. Mitch Ferguson has been developing businesses at software companies since the 90s. He helped build out SpringSource and arrange the acquisition of SpringSource by VMWare–an acquisition that later enabled the creation of Pivotal Software. He then joined Hortonworks as an early member of the team bringing their Hadoop platform to market. Today Mitch works as a co-founder of Accel G2M, an organization that helps bring technology companies to market–building out their sales, marketing, product, and organizational strategies. Accel G2M

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

For most of history, a typical musician would learn to play one specific instrument. As synthesizers became available to the public, it became commonplace for a musician to create their own instruments using hardware and software. By the early 2000s, digital audio workstation software allowed a musician with a laptop to have access to the tools of a record producer. These tools changed how music is made, and gave rise to new genres. Creating electronic music on the computer is a practice much like software engineering. Iteration, modularity, and software architecture skills are required to build a song intelligently. Music engineering also requires working at numerous levels of abstraction: the synthesizer level, the song arrangement level, the mixer level, and the design of melodies. Dom Kane is a musician and sound engineer who writes music for mau5trap, a label started by deadmau5. He has built software synthesizers, worked with numerous artists as a producer, and written music for film and TV. He joins the show to talk about working as a professional electronic musician. We also talk about the overlap between engineering and the different facets of crafting modern music on the computer.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Robinhood is a platform for buying and selling stocks, cryptocurrencies, and other assets. Since its founding in 2013, Robinhood has grown to have more than 3 million user accounts, which is approximately the same as the popular online broker E-Trade. With the surge in user growth and transaction volume, the demands on the software infrastructure have increased significantly. When a user buys a stock on Robinhood, that transaction gets written to Kafka and Postgres. Multiple services get notified of the new entry on the Kafka topic, and those services process that new event using Kafka Streams. Kafka Streams are a way of reading streams of data out of Kafka with exactly-once semantics. Developers at Robinhood use a variety of languages to build services on top of these Kafka streams–including Python. Commonly used systems for building stream processing tasks on top of a Kafka topic include Apache Flink and Apache Spark. Spark and Flink let you work with large data sets while maintaining high speed and fault-tolerance. These tools are written in Java. If you want to write a Python program that interfaces with Apache Spark, you have to pay an expensive serialization/deserialization cost as you move that object between Python and Spark. Ask Solem is an engineer with Robinhood, and the author of Faust, a stream processing library that ports the ideas of Kafka Streams to Python. Faust provides stream processing and event processing in a manner that is similar to Kafka Streams, Apache Spark, and Apache Flink. He is also the author of the popular Celery asynchronous task queue. Ask joins the show to provide his perspective on large scale, distributed stream processing, and why he created Faust.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

TIBCO was started in the 90’s with a popular message bus product that was widely used by finance companies, logistics providers, and other systems with high throughput. As TIBCO grew in popularity, the company expanded into other areas through products it developed in-house as well as through acquisitions. One acquisition was Jaspersoft, a business intelligence data platform. When TIBCO acquired Jaspersoft in 2014, the architecture was a monolithic Java application. Around this time, customer use cases were shifting from centralized reporting to real-time, embedded visualizations. The use case of the Jaspersoft software was becoming less centralized and less monolithic and the software architecture needed to change in order to reflect that. Jan Schiffman is a VP of engineering at TIBCO and Sherman Wood is a director at TIBCO. They join the show to discuss the process of migrating a large Java monolith to a composable set of services. Breaking up a monolith is not an easy process–nor is it something that every company should do just because they have a monolith. In some cases, a monolith is just fine. Jan and Sherman explain why the business use case for why the Jaspersoft monolith needed to be refactored, and their approach to the refactoring. We also talk through the modern use cases of embedded analytics and the interaction between business analysts and data engineers. At a higher level, we discuss the lessons they have learned from managing a large, complex refactoring. Full disclosure: TIBCO is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Self-driving transportation will be widely deployed at some point in the future. How far off is that future? There are widely varying estimations: maybe you will summon a self-driving Uber in a New York within 5 years, or maybe it will take 20 years to work out all of the challenges in legal and engineering. Between now and the self-driving future, there will be a long span of time where cars are semi-autonomous. Maybe your car is allowed to drive itself in certain areas of the city. Maybe your car can theoretically drive itself in 99% of conditions, but the law requires you to be behind the wheel until the algorithms get just a little bit better. While we wait for self-driving to be widely deployed to consumers, a lot could change in the market. We know about Uber, Lyft, Waymo, Tesla and Cruise. But what about the classic car companies like Ford, Mercedes Benz, and Volkswagen? These companies are great at making cars, and they have hired teams of engineers working on self-driving. But self-driving functionality is not the only piece of software you need to compete as a transportation company. You also need to build a marketplace for your autonomous vehicles, because in the future, far fewer people will want to own a car. Customers will want to use transportation as a service. RideOS is a company that is building fleet management and navigation software. If you run a company that is building autonomous cars, you need to solve the problem of making an autonomous, safe robot that can drive you around. Building an autonomous car is hard, but to go to market as a next-generation transportation company, you also need fleet management software, so you can deploy your cars in an Uber-like transportation system. And you need navigation software so that your cars know how to drive around. RideOS lets a car company like Ford focus on building cars by providing a set of SDKs and cloud services for managing and navigating fleets of cars. Rohan Paranjpe joins today’s show to talk about the world of self-driving cars. Rohan worked at Tesla and Uber before joining RideOS, so he has a well-informed perspective on a few directions the self-driving car market might go in. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every company with a large set of customers has a large set of data–whether that company is 5 years old or 50 years old. That data is valuable whether you are an insurance company, a soft drink manufacturer, or a ridesharing company. All of these large companies know that their data is valuable, but some of them are not sure how to standardize the access patterns of that data, or build a culture around data. The larger the company is, the more the data is spread throughout the company, and the more heterogeneous the data sources are. Older companies often have older pieces of data infrastructure, and it might not be well documented. It is hard to make data driven decisions when an organization cannot effectively query their own data. For example, consider a simple question about marketing. An insurance company wants to know how their spending on TV advertising correlates with sales in California over the last 25 years. The VP of marketing sends an email to a business analyst, asking for a historical report of this marketing data. The business analyst knows how to present the data with a business intelligence tool, but the analyst needs to ask the data scientist for how to make that query. The data scientist needs to ask the data engineer where to find those records in a large Hadoop distributed file system cluster. And the data engineer joined the company last week and has no idea where anything is. These are the problems of DataOps. Similarly to DevOps, DataOps is the recognition that a set of problems have crept into organizations over time and slowed down productivity. The story of the DevOps movement is that old infrastructure, lack of testing, and complicated monolithic backends slowed down everyone in an old, big enterprise. The slow pace of change destroys morale and erodes trust. The DevOps movement is about revamping organizations through tooling and organizational behavior. We have covered this in lots of episodes, such as in a great episode with Gene Kim who wrote “The Phoenix Project.” When an organization wants to reinvent itself with DevOps, it often begins with testing and continuous delivery. DataOps encourages data driven organizations to begin with a similar practice of testing their data pipelines to build trust and evolve best practices. There are other similarities between DataOps and DevOps, such as continuous delivery and the breaking down of siloes between different organizational roles. Chris Bergh joins the show to talk about the data problems encountered by large companies, the practices of DataOps, and his company Data Kitchen, which builds tools to help companies move towards more productive data practices.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The main user interfaces today are the smartphone, the laptop, and the desktop computer. Some people today interact with voice interfaces, augmented reality, virtual reality, and automotive computer screens like the Tesla. In the future, these other interfaces will become more common. Developers will want to be able to expose their applications to these new interfaces. For example, let’s say I am a developer who builds a podcast playing app. I have a website and a mobile app, but what if I want to expose that app to a voice interface? Or, what if I want to expose a specific piece of functionality from that app, to make shortcuts easier? Android Slices are user interface components that expose pieces of application functionality to Google Search, Google Assistant, and other applications. Jason Monk is a software engineer who works on Android Slices at Google. Jason joins the show to discuss how mobile user interfaces are changing, the motivation behind Android Slices, and the engineering behind this newer building block for Android developers.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Back in 2014, platform-as-a-service was becoming an increasingly popular idea. The idea of PaaS was to sit on top of infrastructure-as-a-service providers like Azure, AWS, or Google Cloud, and simplify some of the complexity of these infrastructure providers. Heroku had built a successful businesses from the idea of platform-as-a-service, and there was a widely held desire in the developer community to have an “open source Heroku.” One project that was working towards the idea of an open source platform-as-a-service was Deis. Deis made it easier for people to deploy and manage their applications, and it simplified some of the hard parts of container management. When Kubernetes came out, Deis got refactored to use Kubernetes under the hood for container orchestration. Deis was one of the first projects to use Kubernetes as a tool to build a platform-as-a-service, and the team that was working on Deis got very early exposure to the process of building a platform on top of Kubernetes. Michelle Noorali was one of the engineers on the Deis team. When Deis got acquired by Microsoft, Michelle was working on Helm, a package manager for distributed systems. Helm allows developers to deploy distributed applications on top of Kubernetes more easily. A few examples of distributed applications that can be deployed using Helm are Kafka, Prometheus, and IPFS. One reason Helm is so useful is that distributed systems are notoriously hard to configure and run. Since joining Microsoft, Michelle has continued to work on Helm. She is also a member of the Kubernetes Steering Committee and the board of the CNCF. Michelle joins the show to talk about her early experiences building PaaS and her perspective on the Kubernetes ecosystem. Full disclosure: Microsoft is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Stitch Fix is a company that recommends packages of clothing based on a set of preferences that the user defines and updates over time. Stitch Fix’s software platform includes the website, data engineering infrastructure, and warehouse software. Stitch Fix has over 5000 employees, including a large team of engineers. Cathy Polinsky is the CTO of Stitch Fix. In today’s show Cathy describes how the infrastructure has changed as the company has grown–including the process of moving the platform from Heroku to AWS, and the experience of scaling and refactoring a large monolithic database. Cathy also talked about the management structure, the hiring process, and engineering compensation at Stitch Fix.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Food gets thrown away from restaurants, homes, catering companies, and any other place with a kitchen. Most of this food gets thrown away when it is still edible, and could provide nutrition to someone who is hungry. Just like Airbnb makes use of excess living capacity, OLIO was started to connect excess food with people who want to eat that food. There are numerous challenges with this idea. How do you control quality and ensure the food is safe? How do you make money as a business? How do you solve the chicken and egg problem, and make sure that you get hungry users and people with food to give away at the same time? Lloyd Watkin is a software engineer at OLIO, and he joins today’s episode to describe how the platform works, how it is built, and how the company plans to scale their large base of volunteers. It’s a fascinating set of operational and engineering issues.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Building software today is much faster than it was just a few years ago. The tools are higher level, and abstract away tasks that would have required months of development. Much of a developer’s time used to be spent optimizing databases, load balancers, and queueing systems in order to be able to handle the load created by thousands of users. Today, scalability is built into much of our infrastructure by default. We have had several years of infrastructure with automatic scalability, and some of the more recent advances in developer tooling are about convenience, and faster development time. Developers are spending less time dealing with the ambiguous idea of a “server” and more time interacting with well-defined APIs and data sources. A few examples are AppSync from Amazon Web Services and Firebase from Google. These tools are like databases with rich interactive functionality. Instead of having to create a server to listen to a database for changes and push notifications to users in response to those changes, AppSync and Firebase can be programmed to have this kind of functionality built in. There are many other examples of high level APIs, rich backends, and developer productivity tools that lead to shorter development time. What does this mean for developers? It means we can build much faster. We can prototype quickly for low amounts of money–without sacrificing quality. We can spend more time focusing on design, user experience, and business models and less time focusing on keeping the application up and running. Nader Dabit is a developer advocate at Amazon Web Services, and he returns to the show to discuss modern tooling, and how that tooling changes the potential for high output and fast iteration among developers. It is a strategic, philosophical discussion of how to build modern software.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

WebAssembly allows developers to run any language in a sandboxed, memory controlled module that can be called via well-defined semantics. As we have discussed in recent episodes with Lin Clark and Steve Klabnik from Mozilla, WebAssembly is changing application architectures both in and outside the browser. WebAssembly is being adopted by all of the major browser vendors, including Google. Today’s guests are Thomas Nattestad and Ben Smith from Google. Thomas is the PM for V8, WebAssembly, Storage, and Games on the web and Ben is a software engineer on the Chrome team. Ben and Thomas talk about the state of WebAssembly, what the different browser manufacturers are doing, and some cool uses for WebAssembly–from games to CDNs to cryptocurrency infrastructure. As in the previous episodes, there was discussion of WebAssembly’s security and memory benefits from being a bounded context.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

WebAssembly is a low-level compilation target for any programming language that can be interpreted into WebAssembly. Alternatively, WebAssembly is a way to run languages other than JavaScript in the browser. Or, yet another way of describing WebAssembly is a virtual machine for executing code in a low level, well-defined sandbox. WebAssembly is reshaping what is possible to do in the web browser. A developer can write a program in Rust or C++, compile it down into a WebAssembly module, and call out to that module via JavaScript. This is very useful for running memory-sensitive workloads in the browser—such as 3-D games. But WebAssembly is also useful for running modules outside of the browser. Why is that important? If you can already run C++ or Rust code outside of the browser by executing the program from the command line, why would you want to put the code into a WebAssembly module before executing it? One answer is security. WebAssembly modules have well-defined semantics for what memory they can access. WebAssembly could provide more reliable sandboxing for untrusted code. Steve Klabnik is a software engineer at Mozilla, and he joins the show today to play the role of a WebAssembly futurist. We revisit the basics of WebAssembly and the current state of the technology. Steve talks Steve also describes the lessons of past web technologies such as Flash—and what they did right and wrong. We also explore the current and future applications of WebAssembly, which we will talk about in more detail in tomorrow’s episode.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

DoorDash is a logistics company that connects customers, restaurants, and drivers that can move food to its destination. When a customer orders from a restaurant, DoorDash needs to identify the ideal driver for picking up the order from the restaurant and dropping it off with the customer. This process of matching an order to a driver takes in many different factors. Let’s say I order spaghetti from an Italian restaurant. How long does the spaghetti take to prepare? How much traffic is there in different areas of the city? Who are the different drivers who could potentially pick the spaghetti up? Are there other orders near the Italian restaurant, that we could co-schedule the spaghetti delivery with? In order to perform this matching of drivers and orders, DoorDash builds machine learning models that take into account historical data. In today’s episode, Raghav Ramesh explains how DoorDash’s data platform works, and how that data is used to build machine learning models. We also explore the machine learning model release process—which involves backtesting, shadowing, and gradual rollout.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cryptocurrency security is a concern to anyone who has a significant amount of money in the form of Bitcoin, Ethereum, or other crypto assets. Most Bitcoin is held in either a Bitcoin wallet or a Bitcoin bank. Your Bitcoin holdings are recorded on a public ledger. You access these holdings by authenticating with your private key. A Bitcoin wallet could be described more accurately as a Bitcoin keyring. Securing your Bitcoin wallet is about securing that private key. Just as there are many different ways to secure any individual piece of text, there are many ways to secure a Bitcoin private key. A Bitcoin “bank” is a term that can be used to describe institutions such as Coinbase. Coinbase takes the technology of the Bitcoin wallet and wraps it in additional layers of security, identity, and failover that we associate with banks and large technology companies. By using a Bitcoin bank, you sacrifice the autonomy of managing your own private key. On the bright side, you don’t have to manage your own private key. If you lose your Coinbase password, there are plenty of ways to recover it. A Bitcoin bank gives you the downsides and the upsides of working with a centralized service provider. Jameson Lopp is a cypherpunk and cryptocurrency engineer at Casa. Casa is a company that is building long-term cryptocurrency storage and secure key infrastructure. In this episode, we explore how Bitcoin wallets work, how to secure them, the common threats, scams and hacking attempts of Bitcoin, and what he is working on at Casa.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At Google, the job of a site reliability engineer involves building tools to automate infrastructure operations. If a server crashes, there is automation in place to create a new server. If a service starts to receive a high load of traffic, there is automation in place to scale up the instances of that service. In order to create an automated response to an infrastructure problem, a site reliability engineer needs insights into that infrastructure. Every service needs tools around monitoring, alerting, debugging, and distributed tracing. One benefit of working at a large company like Google is that an engineer building a new product gets this kind of tooling by default. If I am hacking on a project at home, I have to set up all kinds of tools to help me diagnose and resolve problems. Setting up this tooling takes time, and requires expertise. Stackdriver is a set of tools and instrumentation that allows developers to monitor, debug, and inspect infrastructure. Stackdriver is based on the internal observability tools built for Google. Mark Carter is a group product manager at Google, and he joins the show to discuss site reliability engineering and the creation of Stackdriver.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Continuous delivery is a way of releasing software without requiring software engineers to synchronize during a release. Over the last decade, continuous delivery workflows have evolved as the tools have changed. Jenkins was one of the first continuous delivery tools and is still in heavy use today. Netflix’s open sourced Spinnaker has also been widely adopted. As Kubernetes has grown in popularity, some engineers have developed a workflow around Kubernetes and Git known as GitOps. GitOps treats Git as the source of truth for deployments. Under GitOps, when a divergence occurs between your git repository’s configuration files and the state of your production infrastructure, your infrastructure should automatically adjust its state to align with the state defined in git. Alexis Richardson is the CEO of Weaveworks, a company that has built tooling around GitOps. He joins the show to describe how GitOps works, and explain how it compares to other methods for continuous delivery.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Klarna is a payments company headquartered in Sweden. Since being established in 2005 it has grown to handling $21 billion in online sales in 2017. Roughly 40% of all e-commerce sales in Sweden go through Klarna. Klarna’s original differentiator was that it allowed users to checkout of e-commerce stores without entering in credit card information. Instead, the user enters an email address and registers with Klarna. This allows Klarna to assume the risk of the transaction, in place of the credit card company. Klarna’s clever payment method became very popular, and 13 years later Klarna is a bank with a variety of financial services and payment methods. Marcus Granstrom is a director of engineering at Klarna. His work ranges from product development to systems architecture to management. His cross functional role has some similarity to Raylene Yung from Stripe, who is also an engineering director at a payments company, and was on the show yesterday. Marcus walked me through the life of a payment hitting Klarna’s servers, and this served as a nice starting point for a conversation about Klarna’s infrastructure, their product, and their engineering practices.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Stripe is a payments API that allows merchants to transact online. Since the creation of the payments API, Stripe has expanded into adjacent services such as fraud detection, business management, and billing. These other verticals leverage the existing customer base and infrastructure that Stripe has developed from the success of their payments business. Raylene Yung is the head of payments at Stripe. She joins the show to talk about her work, which includes elements of engineering, product development, design, and management. All of these dimensions of her job came up in our conversation, which made for a wide ranging conversation. This interview comes in the context of Stripe’s rapid growth. The organization is changing, and Raylene explored the questions that Stripe is asking itself internally about org structure. Namely: what is the tradeoff between a defined, hierarchical structure of direct reports versus a decentralized, flat org structure? Is there any advantage to making roles highly defined (such as “senior infrastructure software engineer”)? Or is it better to let people have fluid roles, and self-assemble? Raylene was willing to explore these questions–and I found her answers highly useful and thought provoking.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the smartphone market there are two dominant operating systems: one closed source (iPhone) and one open source (Android). The market for self-driving cars could play out the same way, with a company like Tesla becoming the closed source iPhone of cars, and a company like Comma.ai developing the open source Android of self-driving cars. George Hotz is the CEO of Comma.ai. Comma makes hardware devices that allow users with “normal” cars to be augmented with advanced cruise control and lane assist features. This means you can take your own car–for example, a Toyota Prius–and outfit your car to have something similar to the Tesla Autopilot. Comma’s hardware devices cost under $1000 to order online. George joins the show to explain how the Comma hardware and software stack works in detail–from the low level interface with a car’s CAN bus to the high level machine learning infrastructure. Users who purchase the Comma.ai hardware drive around with a camera facing the front of their windshield. This video is used to orient the state of the car in space. The video from that camera also gets saved and uploaded to Comma’s servers. Comma can use this video together with labeled events from the user’s driving experience to crowdsource their model for self-driving. For example, if a user is driving down a long stretch of highway, and they turn on the Comma.ai driving assistance, the car will start driving itself and the video capture will begin. If the car begins to swerve into another lane, the user will take over for the car and the Comma system will disengage. This “disengagement” event gets labeled as such, and when that data makes it back to Comma’s servers, Comma can use the data to update their models. George is very good at explaining complex engineering topics, and is also quite entertaining and open to discussing the technology as well as other competitors in the autonomous car space. I have not been able to get many other people on the show to talk about autonomous cars, so this was quite refreshing! I hope to do more in the future.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Chad Fowler was the CTO of Wunderlist prior to its acquisition by Microsoft. Since the acquisition, Chad has become the general manager of developer advocacy at Microsoft. He also works as a venture capitalist at BlueYard Capital, an early stage investment firm. I’ve had a lot of fun talking to Chad, because he can move seamlessly between talking about disparate subjects like cloud computing, investing, cryptocurrencies, and music composition. And he has novel things to say about all of them! When Chad joined Wunderlist, he helped start a large refactoring of the software architecture. He then helped the company navigate to the successful Microsoft acquisition. We started off the conversation with the story of this rearchitecture, and how he sees the current opportunities in front of Microsoft. Chad gives his perspective on Kubernetes, functions-as-a-service, and how developer tooling might evolve in the near future. After talking about near-term developer tooling, we talked about the distant future: bug bounty marketplaces on the blockchain; using Github repositories to train machine learning models about how to write code; the comparison between music collaboration and software collaboration. This was a wide array of topics, but Chad was equipped to discuss all of them–since he works at Microsoft, makes large investments in the future, and studied music when he was in school.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Music collaboration has historically been accomplished by musicians gathering in bands. A band is usually an in-person, physical manifestation: a drummer, a guitarist, a piano player. Or, on a large scale, a symphony of classical instruments led by a conductor. Today, the most flexible instrument that anyone can play is arguably the computer, because a computer can simulate or replay any of the sounds made by any other instrument. Another advantage of the computer is that it removes physicality as a constraint on the musician. A computer musician does not have to train their muscles to play piano, or guitar, or drums. The computer musician can imagine a sound and bring it to life inside a digital audio workstation (a program for composing and arranging music). The rise of the computer musician has coincided with a change in the way popular music is created. Instead of bands needing to work together to create a piece of music, a single producer can simulate all of the members of the band by programming piano, drums, and everything else. The rise of the solo producer has given birth to new kinds of music–but solo music production inherently limits the range of musical ideas that can be explored. The most important works of art have input from multiple people. And even the most successful solo producers love to work with other artists who have a complementary skill–such as vocals. For the last twenty years, the model of solo producer working with pop vocalist has largely dominated the charts. Musical collaboration has stuck to a model that mimics its pre-Internet form, with very small groups of 1-5 people making the core of a song. The main tools that people use to collaborate are email and Dropbox. Splice is a tool for music collaboration. Splice combines version control, revision history, social networking, sample discovery, synthesizer rental, and other features. Splice is changing the way that music is created, with a large percentage of top producers adopting it. The impact Splice has on music will be on par with what Github has done for software engineering. Matt Aimonetti is the CTO and co-founder of Splice, and he joins the show to talk about the founding story, the product development, and the engineering of Splice.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Java programs compile into Java bytecode. Java bytecode executes in the Java Virtual Machine, a runtime environment that compiles that bytecode further into machine code, and optimizes the runtime by identifying “hot” code paths and keeping those hot code paths executing quickly. The Java Virtual Machine is a popular platform for building languages on top of. Languages like Scala and Clojure compile down to Java bytecode, and can take advantage of the garbage collection system and the code path optimizations of the JVM. But when Scala and Clojure compile into Java bytecode, the code “shape”–the way that the programs are laid out in memory–is not the same as when Java programs compile into Java bytecode. Executing bytecode that comes from Scala will have certain performance penalties relative to a functionally identical program written in Java. GraalVM is a system for interpreting languages into Java bytecode that can run efficiently on the JVM. Any language can be interpreted into an abstract syntax tree that the GraalVM can execute using the JVM. Languages that can run on GraalVM include JavaScript, R, Ruby, and Python. Thomas Wuerthinger is a senior research director at Oracle and the project lead for GraalVM. He joins the show to explain the motivation for GraalVM, the architecture of the project, and the future of language interoperability. It was an exciting discussion and I learned a lot about the Java ecosystem.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A token is a unit of virtual currency. Most tokens are built on a blockchain-based cryptocurrency platform, such as Ethereum. Building on top of a platform like Ethereum allows these tokens to form their own financial ecosystem while leveraging the scale of an existing currency. Tokens became highly popular in early 2018, with the boom in ICOs–initial coin offerings. Many of these coins offer a value proposition of a “utility token.” The idea of a utility token is that the token is necessary to transact in a particular ecosystem. If Amazon were to require you to convert US dollars to Amazon coins in order to buy items on Amazon, the Amazon coin would be a “utility token.” There are many different kinds of utility token schemes, and time will tell if this model makes sense for the cryptocurrency investment landscape. Another type of token is the “security token,” in which a token represents a share in an organization. This token type is more like a stock, or bond, or certificate of ownership of a financial instrument. These types of tokens also have their share of criticism. If I start a company, most of my assets are not represented on a blockchain–the assets are things like hiring contracts, intellectual property, real estate, etc. The legal ownership of these assets is settled by a complicated legal system which has no notion of a blockchain. It’s unclear how the claims of a security token today would be enforced–or why a security token is presently a better option for raising capital than traditional equity or debt instruments. Felipe Pereira is the author of “On the immaturity of tokenized value capture mechanisms,” a Medium article in which he documents different types of token systems, including several flavors of utility tokens and security tokens. He’s also the co-founder at a company called Paratii. He joins the show to discuss the present viability of token-based systems–and what blockchains have actually proven to be useful for today.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Medical breakthroughs require medical research. Medical research requires patient testing and data collection. The most common form of capturing patient data is through surveys–and most of those surveys today are done on paper. Surveying patients to understand the side effects or benefits of trial drugs or treatments, and getting accurate results out of these are critical aspects of medical research. Traditionally, these surveys are filled and read manually, and entered into a database by a human operator. In these steps, there is too much room for human error, from unreadable handwritings to typos being entered into the databas Electronic Data Capture platforms were created out of this need for easy and accurate data collection for researches. By enabling online survey creation and result collection, EDC platforms improved medical research immensely. However, these platforms are complex to design. Where patient medical data is concerned, privacy and security are of extremely high importance. Compliance with laws that protect anonymity and privacy of the patients is necessary. On top of these, the platform must be easy-to-use, and reliable. Castor EDC is a company specializing on EDC for medical research, founded in the Netherlands and active in many countries around the globe. Our guest today is Derk Arts, the founder and CEO of Castor EDC. In this episode we discuss Electronic Data Capture platforms, how Castor EDC overcame the engineering and design problems, how they comply with the laws, and their business model.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apple operating systems are closed source. This closed source nature gives Apple an extremely successful business model–and a very different software developer ecosystem than Linux-based systems. Since Linux is open source, the information on how to manipulate the system at a low level is very public. The lack of information about low-level programming in Apple operating systems has led to a large community of “jailbreaking”–where people try to reverse engineer how the closed source systems function. In today’s episode, Max Bazaliy joins the show to describe how he reverse engineered an Apple Watch. It’s a complex security challenge to jailbreak an Apple Watch, as he describes in detail. Max is a security researcher at Lookout, a mobile security company.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

“Edge computing” is a term used to define computation that takes place in an environment outside of a data center. Edge computing is a broad term. Your smartphone is an edge device. A self-driving car is an edge device. A security camera with a computer chip is an edge device. These “edge devices” have existed for a long time now, but the term “edge computing” has only started being used more recently. Why is that? It is mostly because the volume of data produced by edge devices, and the type of computation that we want from edge devices is changing. We want to develop large sensor networks to enable smart factories, and smart agriculture fields. We want our smartphones to have machine learning models that get updated as frequently as possible. We want to use self-driving cars, and drones, and smart refrigerators to develop elaborate mesh networks–and perhaps even have micropayments between machines, so that computation can be offloaded from edge devices to a nearby mesh network for a small price. Kubernetes is a tool for orchestrating distributed, containerized computation. Just as Kubernetes is being widely used for data center infrastructure, it can also be used to orchestrate computation among nodes on-premise at a factory, or in a smart agriculture environment. In today’s episode, Venkat Yalla from Microsoft joins the show to talk about Kubernetes at the edge, and how Internet of things applications can use Kubernetes for their deployments today–and what the future might hold. Full disclosure: Microsoft is a sponsor of SE Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

React Native allows developers to reuse frontend code between mobile platforms. A user interface component written in React Native can be used in both iOS and Android codebases. Since React Native allows for code reuse, this can save time for developers, in contrast to a model where completely separate teams have to create frontend logic for iOS and Android. React Native was created at Facebook. Facebook itself uses React Native for mobile development, and contributes heavily to the open source React Native repository. In 2016, Airbnb started using React Native in a significant portion of their mobile codebase. Over the next two years, Airbnb saw the advantages and the disadvantages of adopting the cross platform, JavaScript based system. After those two years, the engineering management at Airbnb came to the conclusion to stop using React Native. Gabriel Peal is an engineer at Airbnb who was part of the decision to move off of React Native. Gabriel wrote a blog post giving the backstory for React Native at Airbnb, and he joins the show to give more detail on the decision.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Blogging is more than 20 years old. Over that period of time, numerous publishing platforms have been created. Squarespace, Blogger, Medium, and Twitter are popular closed source platforms. WordPress has been the most popular open source blogging platform–and much of the Internet (including Software Engineering Daily) runs on WordPress. WordPress is a powerful platform. News companies, ecommerce websites, and many other kinds of businesses use WordPress as their central publishing tool. But WordPress has been around for 15 years–and there are some potential conflicts of interest between WordPress the open source project and WordPress.com (a company started to host WordPress websites). John O’Nolan was working as a WordPress developer when he decided to start a new publishing platform called Ghost. Five years later, the Ghost project is a success–with a thriving open source community, a profitable SaaS business, and companies like Digital Ocean and Mozilla using Ghost to host their blogs. John and I discussed his background with WordPress, what he wanted to do differently with Ghost, and the software architecture of Ghost. We also touched on the Ghost SaaS business and the management of the open source project.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Howard Marks ran two video game companies in the 90’s: Activision and Acclaim. While running these companies, he developed a love for entrepreneurship that he maintains today. Howard is the CEO of StartEngine, a company that functions as an accelerator, a crowdfunding platform, and ICO launcher. Howard joins the show to talk about his background as an entrepreneur, as well as some modern alternative funding mechanisms that he’s working on at StartEngine. Hearing Howard’s thoughts on building a video game company in the 90’s was particularly new information to me–it’s an era of software development that we have not covered much at all. As a side note–some listeners have asked recently why we cover subjects such as ICOs, when there have been so many dubious companies that have launched ICOs. There are two reasons why we cover this area. Firstly, cryptocurrencies are a breakthrough computer science construct. It’s important for us to try to understand their implications. The other reason why we cover ICOs is that some technology companies require high upfront capital costs. The amount of capital you have available affects the speed at which your engineering team can move. New funding mechanisms could mean more capital for certain types of software companies–and this could be a good thing and a bad thing, depending on the company.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Video streaming platforms like Netflix offer a convenient way to watch video content. We are now able to watch our favorite TV shows, movies, or content creators on a range of devices. However, buffering while watching videos can be a painful experience on mobile phones and tablets that use 4G or LTE. As streaming becomes available to a wider range of devices with varying bandwidth restrictions, different encodings of the video need to be created for different devices, and different bandwidth situations. To get the best quality viewing possible with the bandwidth available to connections, there needs to be a balance between the resolution of the video, and the bitrate, which defines the data that the video consumes. Mux is a company that builds video hosting and analytics. Ben Dodson is a data scientist at Mux, who built a system for optimizing the bitrate of videos through machine learning. In this episode we discuss video encoding and how Mux solved the problem of serving the highest quality video with the ideal bitrate.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Enterprises want to update their technology faster. One way an enterprise can accelerate the adoption of new tools is to move more aggressively towards the cloud. By giving internal developers access to the cloud, it becomes easier to provision new servers–allowing for rapid experimentation, test environments, and scalability. In previous shows we have explored how large enterprises successfully learn to move their technology faster. Much of this process is rooted in being able to experiment quickly–which requires well-defined testing procedures, and the ability to quickly provision and destroy infrastructure. Many enterprises have large on-premise infrastructure deployments. An enterprise’s movement towards the cloud can be made complex by this existing set of servers. In today’s show, Aparna Sinha discusses how Kubernetes is useful for enterprises–and how it can improve development speed, experimentation, and observability. Aparna is the leader of the product team for Kubernetes and Container Engine at Google. Much of her job is centered around understanding what would be useful to enterprises who are choosing a cloud provider. The open source version of Kubernetes is useful on its own, but most enterprises choose a managed provider of Kubernetes–such as Google Kubernetes Engine–to help with support and onboarding . Full disclosure: Google is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

JavaScript has been the exclusive language of the web browser for the last 20 years. Whether you use Chrome, Firefox, Internet Explorer, or Safari, your browser interprets and executes code in a virtual machine–and that virtual machine only runs JavaScript. Unfortunately, JavaScript is not ideal for every task we want to perform in the browser. Think about the use cases where you need to use software outside of the browser: video editing, music production, 3D art, video games. These applications require a high degree of performance that is hard to get from raw JavaScript. WebAssembly was created to get better performance on the web. WebAssembly allows code from other languages to be compiled and run in the browser. With WebAssembly, languages such as C, C++, and Rust can be used to achieve major performance gains. WebAssembly is still under development, and eventually more programming languages will be accessible as well. Lin Clark is an engineer on the Mozilla Developer Relations team, and has been working closely on the WebAssembly project. She is the author of a detailed series of illustrated blog posts that explain how WebAssembly works. In this episode, we discuss how WebAssembly came to be, its advantages over a web driven purely by Javascript, what is possible with WebAssembly, and its engineering implementation.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

“Bots” are becoming increasingly relevant to our everyday interactions with technology. A bot sometimes mediates the interactions of two people. Examples of bots include automated reply systems, intelligent chat bots, classification systems, and prediction machines. These systems are often powered by machine learning systems that are black boxes to the user. Today’s guest Rob May argues that these systems should be auditable and accountable, and that using a blockchain-based identity system for bots is a viable solution to the machine learning auditability problem. Rob is the CEO of Talla, a knowledge base provider for business teams. The Botchain project was spun out of Talla as a solution to the problem of bot identity. In this episode, we talk about Botchain and the application of blockchain to bot identity, the current state of ICOs, and the viability of utility token ecosystems. Botchain has its own cryptotoken called “Botcoin.”

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Banking has been a part of the economy for 600 years. Banking has always been evolving. The most recent evolution: the financial industry has been going digital. Newer “fintech” companies have created innovative ways of doing everything related to money–from friendly payments to budgeting; from business transactions to insurance. However, the traditional banks themselves have been relatively slow at adjusting to these digital changes, creating the opportunity for native digital banks–often referred to as “challenger banks.” N26 is a digital first bank established in Berlin and is active in 17 European countries with a million users. Pat Kua is the CTO at N26. Digital banks are hosted on the cloud, without a physical branch the user must go to to perform an operation. The user accesses their account through a mobile application, and can complete everything online from opening an account to performing transactions. This system has numerous advantages, like simplicity for the user, and higher scalability in terms of users from the bank’s perspective. In this episode, we discuss the advantages of digital banks, the fintech industry and N26 as a itself. We also explore product development and how Pat manages his time as CTO–which is a useful discussion for anyone who is learning to be a technical leader or manager.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Git is a distributed file system for version control. Git is extremely reliable, fast, and secure, owing to the fact that it is one of the oldest pieces of open source software. But even battle-tested software can have vulnerabilities. In this episode, we explore a subtle git vulnerability that could have potentially led to git users executing malicious scripts when they intended to simply pull a repository. Today’s guest Edward Thomson is a program manager at Microsoft, and a maintainer of libgit2, a C implementation of git. He also writes about git and hosts the podcast All Things Git. He is passionate about git development, which gave me a deeper perspective on something that I just consider a tool. But the only reason that tool is so good–the only reason it fades into the background–is because there are people that are passionate enough to work on it on a regular basis. We also spent some time talking about the vulnerabilities that can spread through shared code environments–particularly in the realm of git, npm, and PHP. And we touched on how deployment workflows around git and Kubernetes are changing. Full disclosure: Microsoft, where Edward works, is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

If you operate a restaurant, you want to know how many people are inside your restaurant at any given time. You also want to be able to know your occupancy if you operate a movie theater, coffee shop, or apparel store. Knowing how many people are in your building can answer several business-related questions. Do you need to unlock an additional entrance? Should you open another store? Do you really need a building this big? This might sound like a simple question, but how do you solve the problem of counting people inside of a building? A naive approach to counting people is to use video cameras and count the number of people entering and exiting the building. Machine learning algorithms are good at classifying humans. But the downside of this is that you have to put cameras anywhere you want a people-counter. There are many situations where you would want to count the number of people where a camera is not acceptable. What if you wanted to count people in a privacy preserving way? What if you wanted to obscure any identifiable traits of a person that you were counting? Density is a device for counting people. It sits above a doorway and counts the people who are entering or exiting the building. Andrew Farah is the CEO at Density, and in today’s episode he explains why the problem of counting people is harder than it sounds, and how the Density people counter functions.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning models allow our applications to perform highly accurate inferences. A model can be used to classify a picture as a cat, or to predict what movie I might want to watch. But before a machine learning model can be used to make these inferences, the model must be trained and deployed. In the training process, a machine learning model consumes a data set and learns from it. The training process can consume significant resources. After the training process is over, you have a trained model that you need to get into production. This is known as the “deployment” step. Deployment can be a hard problem. You are taking a program from a training environment to a production environment. A lot can change between these two environments. In production, your model is running on a different machine–which can lead to compatibility issues. If your model serves a high volume of requests, it might need to scale up. In production, you also need caching, and monitoring, and logging. Large companies like Netflix, Uber, and Facebook have built their own internal systems to control the pipeline of getting a model from training into production. Companies who are newer to machine learning can struggle with this deployment process, and these companies usually don’t have the resources to build their own machine learning platform like Netflix. Diego Oppenheiner is the CEO of Algorithmia, a company that has built a system for automating machine learning deployments. This is the second cool product that Algorithmia has built, the first being the algorithm marketplace that we covered in an episode a few years ago. In today’s show, Diego describes the challenges of deploying a machine learning model into production, and how that product was a natural complement to the algorithms marketplace. Full disclosure: Algorithmia is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern programming requires lots of integration between APIs. Some of these integrations are trivial–such as using Twilio or Stripe. But there are many more complex integrations. For example, when a large company acquires a smaller company, the acquiring company might want to integrate with that smaller company to leverage the synergies between the two companies. How do you build clean communication patterns between the services of one company and another? Two teams within a single enterprise can also have integration issues. One team might have a different data model than the other team. One team might be using JSON and the other using XML. In these cases, integrations between APIs can take considerable time. Ballerina is a programming language that is designed for writing integrations. Ballerina is made for building services that allow two APIs to communicate easily–in contrast to other patterns of API integration such as those involving an enterprise service bus. Tyler Jewell is the CEO of WSO2, a company that specializes in integrations. WSO2 created the Ballerina language and is investing heavily into it (with ~80 people working on Ballerina today). In this episode we explored integrations–and why this problem required creating a new programming language. BallerinaCon is July 18th in San Francisco and our listeners can attend for free–use code BalCon-SEDaily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Flutter allows developers to build cross-platform mobile apps. In our previous show about Flutter, Eric Seidel from Google described the goals of Flutter, why he founded the project, and how Flutter is built. In today’s show, Randal Schwartz talks about Flutter in more detail–including the developer experience of building Flutter apps and why he finds Flutter so exciting. Randal is a longtime software developer who has focused mostly on web applications. He’s also the host of the popular podcast FLOSS Weekly, a show about open source software. Like many developers, Randal has stayed away from mobile development in the past. If you write a mobile application, you have historically had to build iOS and Android apps separately. That is a large up-front cost, and Flutter reduces the cost by allowing users to develop mobile apps for iOS and Android with a single codebase. Randal has been podcasting about open source software for 12 years, so he brings historical depth to this conversation. He has also been working with Dart for several years–Dart is a language developed by Google that is used in Flutter.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Nubank was started in 2013 with a credit card that was controlled through a mobile app. At the time, it was the first service in Brazil that allowed customers to do banking without going to a physical bank branch. Since then, Nubank has expanded into additional financial services and currently has 850 employees working in Brazil. Edward Wible is a co-founder and CTO of Nubank and in this episode he discusses his work growing Nubank from a small team of less than 10 people into a company with almost 1000 people. We have covered two other banks in the past few weeks: Monzo and N26. In terms of software engineering and product management, Nubank is similar to Monzo and N26. One characteristic that stood out was Nubank’s use of Clojure, a functional programming language built on the JVM. A question that came up during this show: what is the line between “fintech company” and “bank”? We hope explore this more in future shows about the intersection of money and software. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Flutter is a project from Google that is rebuilding user interface engineering from the ground up. Today, most engineering teams have dedicated engineering resources for web, iOS, and Android. These different platforms have their own design constraints, their own toolset, and their own programming languages. But each platform is merely building a user interface. Why should development across these three user surfaces be so different? This was the question that Eric Seidel was asking himself three years ago, when he co-founded the Flutter project. The Flutter project had a few rough starts, as the team tried to figure out exactly what layer of abstraction they were trying to provide. Around that time, ReactJS and React Native were growing in popularity. Seeing the React projects provided some data points, and some inspiration. But Flutter takes a lower level approach to cross platform app development, by presenting a rendering layer and a runtime API that are interfacing with the hardware in the same way that OpenGL does. In today’s episode, Eric joins the show to explain how the Flutter project came to life, and his lessons from starting an ambitious project that took several years to pick up steam. I enjoyed this episode because Flutter could have massive improvements for how quickly we can build apps–and also because Eric is a serious engineer and there are so many insights in this episode about computer science, software engineering, and project management. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Tim O’Reilly’s book What’s the Future? is an overview of business, technology, and society. As the founder of O’Reilly Media, Tim has been steeped in technology trends for the last 40 years. From his vantage point running conferences and publishing technical content, Tim has been able to make informed predictions about what is coming next. In today’s conversation, Tim gave his perspective on how artificial intelligence will impact our world in the coming decades. More importantly–Tim emphasizes the role of human agency. The future is not something that merely happens to us as we sit back and eat popcorn. Today, we make decisions, and those decisions that we make could help lead us to technology utopia or towards the fall of our great technological empire. On the subject of business, Tim gave a radically different perspective than most of the entrepreneurs that come on Software Engineering Daily. In our conversation, he raised the question of why entrepreneurs raise massive amounts of money, get on the treadmill of startup hype, and build a company around negative cash flows. For that model, the only possible outcomes are going public, being acquired, or flaming out completely. O’Reilly Media has been cash flow positive since the beginning, and the company has steadily compounded, growing successively bigger businesses from publishing to conferences to online learning. This episode gave me a lot to think about–just as the O’Reilly Conferences have throughout the years. O’Reilly Media has graciously partnered with SE Daily since we were very small, so I have great admiration for the company and Tim himself. It was a pleasure to get the chance to meet him in person. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a patient comes into the hospital with stroke symptoms, the hospital will give that patient a CAT scan, a 3-dimensional imaging of the patient’s brain. The CAT scan needs to be examined by a radiologist, and the radiologist will decide whether to refer the patient to an interventionist–a surgeon who can perform an operation to lower the risk of long-term damage to the patient’s brain function. After getting the CAT scan, the patient might wait for hours before a radiologist has a chance to look at the scan. In that period of time, the patient’s brain function might be rapidly degrading. To speed up this workflow, a company called Viz.ai built a machine learning model that can recognize whether a patient is at high risk of stroke consequences or not. Many people have predicted that radiologists will be automated away by machine learning in the coming years. This episode presents a much more realistic perspective: first of all, we don’t have nearly enough radiologists, so if we can create automated radiologists that would be a very good thing; second of all, even in this workflow with a cutting-edge machine learning radiologist, you still need the human radiologist in the loop. David Golan is the CTO at Viz.ai, and in today’s show he explains why he is working on a system for automated stroke identification, and the engineering challenges in building that system. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Computer systems consume memory, CPU, battery, data, and network bandwidth as inputs. These systems provide value for the end user by delivering information, virtual objects, and physical products as outputs. Another fundamental resource that is becoming easier to consume as input is money. There are also new outputs–financial constructs that are made possible by cloud computing, machine learning, and cryptocurrencies. This is why so much opportunity exists in fintech. Money has always been a flexible tool for brokerage between humans. But as recently as the early 2000s, the interfaces between money and computers have been clunky and inflexible. Engineers that wanted to build financial systems around money had to work directly with banks and credit card processors. More recently, there has been an explosion in new APIs and completely new financial primitives like cryptocurrencies. In the year 2000, a well-funded team had to struggle to put together a basic ecommerce company. Today, a blue ocean of opportunity has opened up for entrepreneurs building businesses around lending, insurance, underwriting, banking, and every other microcosm of the financial system. Michael Walsh is a general partner and co-founder of Green Visor Capital. In today’s episode, he described his perspective on the modern fintech environment, and what the future holds. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Napster, Kazaa, and Bittorrent are peer-to-peer file sharing systems. In these P2P systems, nodes need to find each other. Users need to be able to search for files that exist across the system. P2P systems are decentralized, so these routing problems must be solved without a centralized service in the middle. Without a centralized service that has all the information in one place, how can you solve these problems of node discovery and file lookup? This is the central question that Petar Maymounkov sought to answer with Kademlia. Kademlia is a peer-to-peer distributed hash table. Kademlia implements the “put” and “get” operations of an efficiently scalable hash table without using any centralized service. Each node in the system maintains its own routing table. When a user queries the system (a “get” operation), that query is serviced by the nodes coordinating with each other to intelligently route the user to their target location. When a file is stored (a “put” operation), that update to the file system can propagate through the network in a decentralized, uncoordinated way. Petar joins the show to give a brief history of P2P networks, why he created Kademlia, and what he is working on today. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every company has the idea of the “nightly report.” A business analyst comes into the office, sits down in front their inbox, and looks at yesterday’s data. Did sales go up? Did the marketing campaigns bring in the expected number of customers? Was there an increase in helpdesk tickets? The statistics that these reports deliver to human analysts can change the direction of the business. Everyone within a company could use a regular report that documents how the business is changing over time. Outlier.ai is a company that processes the data sets within a business and generates automated reports that are relevant to different people within the organization. If you are an email marketing analyst, your data from MailChimp campaigns will be analyzed. If you manage a customer success team, your Zendesk tickets will be analyzed. If you are a technical support analyst, the crash reports and error messages from your users will be analyzed. In all of these cases, the data gets processed automatically, and a story is sent to you, so that you can have the information in your inbox waiting for you, instead of having to go ask a data scientist to generate it for you. Mike Kim is the CTO of Outlier.ai, and in this show he describes the engineering challenges of integrating with all the different data sets of an organization–and why there is so much value in the idea of the automated “report” or “story” for analysts. In past shows, we have explored how data engineering has progressed over the last twenty years–from database administration to Hadoop cluster management to the emergence of “data breadlines” where analysts wait for a data scientist to process the job they asked for. Outlier represents a step towards a world where the data science reports are delivered to us before we even ask, rather than us having to query the system. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Chromium is an open source browser that shares code with the Chrome browser from Google. A browser is a large piece of software, with engineering challenges around threading, rendering, resource management, and networking. To add to the complexity, Chrome runs on iOS, Android, MacOSX, Windows, and other platforms. Chrome OS is an operating system based on Chrome. There is also Chromium OS, the open source version of Chrome OS. The Chrome/Chromium operating systems are based off of Linux. Through this entire episode, the line between browser and operating system is blurry. There is so much resource management involved in the Chrome browser that it has its own task manager. For many people (including myself) the browser is the main application you are interfacing with throughout the day. It handles all of your business applications. Even many of your desktop apps, such as Slack, are running on Electron, which is a framework for building cross-platform apps that uses Chromium. David Bokan is an engineer on the Chromium team at Google, and he joins the show to describe the engineering of Chrome and the development and release process. David also gives his thoughts on future developments for browsers, apps, and the Internet. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Shopify runs more than 500,000 small business websites. When Shopify was figuring out how to scale, the engineering teams did not have a standard workflow for how to deploy and manage services. Some teams used AWS, some teams used Heroku, some teams used other infrastructure providers. To manage all those stores effectively, Shopify has built its own platform-as-a-service on top of Kubernetes called Cloudbuddies. Cloudbuddies was inspired by Heroku, and it allows engineers at Shopify to deploy services in an opinionated way that is perfect for Shopify. Niko Kurtti is a production engineer at Shopify, and he joins the show to describe Shopify’s infrastructure–how they run so many stores, how they distribute those stores across their infrastructure, and the motivation for building their own internal platform on top of Kubernetes. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

“Serverless” is a word used to describe functions that get deployed and run without the developer having to manage the infrastructure explicitly. Instead of creating a server, installing the dependencies, and executing your code, the developer just provides the code to the serverless API, and the serverless system takes care of the server creation, the installation, and the execution. Serverless was first offered with the AWS Lambda service, but has since been offered by other cloud providers. There have also been numerous open source serverless systems. On SE Daily, we have done episodes about OpenWhisk, Fission, and Kubeless. All of these are built on the Kubernetes container management system. Kubernetes is an open-source tool used to build and manage infrastructure, so it is a useful building block for higher level systems. Chad Arimura is the VP of serverless at Oracle, where he runs the Fn project, an open source serverless platform built on top of Kubernetes. In the past, he ran Iron.io, a message broker platform. Matt Stephenson also joins the show–he is a senior principal software engineer at Oracle and has experience from Amazon and Google, where he worked on Google App Engine (which was arguably one of the first “serverless” platforms). We discussed why there are so many different serverless tools built on Kubernetes, and the tradeoffs that these serverless tools are exploring. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When you interact with your bank, it probably feels different than when you interact with a software technology company. That’s because the biggest banks in the world were started before software became such a universally important tool. Their core competency is banking–not consumer software. Today, most banks make consumer-facing software. But these banks were not founded by engineers. The software development process at a typical bank does not look like the software development process at a software company like Netflix. Monzo is a digital bank that focuses on high quality engineering. Since it was started in 2015, Monzo has always thought of itself as a software company. This gives it certain advantages over older banks. Today’s guest Richard Dingwall is an engineer at Monzo, and he joins the show to describe Monzo’s software architecture, the engineering strategy, and its migration to Kubernetes. Richard has prior experience at several different banks and financial institutions. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Crocodile Browser is a fast browser built by Osine and Anesi Ikhianosime, a pair of brothers from Nigeria. I interviewed them 3 years ago, and in this episode I caught up with Osine to learn what he and his brother have been working on since then. Osine and Anesi have become friends of mine since we had a conversation several years ago. I met Osine for the first time at the Facebook F8 conference last year, and it was one of the first times I had met someone from another continent on the Internet, then got to hang out with them in person. There were some issues with network connectivity, so I decided to release this show on the weekend with no ads. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Searching through all of the videos on the Internet is not a simple problem. In order to search through all the videos, you need to build a search index. In order to build a search index, you need to build a web crawler. Video files are large. To store all of the actual video files would cost far too much money. In order to build an index in a cost-efficient manner, you need to have a way of storing information about a video without storing the entire video itself. You might be thinking “hasn’t Google already solved video search? Why are we even talking about this?” Google has solved some aspects of video search–but a different set of challenges is being tackled by a video search company called Pex. In order to explain what Pex is building, we should first explain the problem set they are trying to tackle. Videos across the internet are consumed on a variety of platforms such as YouTube, Instagram, Facebook, and Vimeo. These videos are sliced up, bootlegged, and repurposed from one platform to another. For content creators who earn their living from their hosted video streams, this can be a nightmare. Imagine you are a musician, and you make lots of money from music videos. You upload your cool new video to YouTube, and it instantly gets bootlegged by other users and shared across the internet in hundreds of different places. When people watch the stolen versions of your video, you are not getting compensated. If you could locate all of those stolen videos, you could order them to take it down, or claim the video so that you are paid for it. And here is the engineering problem–how can you find all those re-posted videos? By crawling the web and building a search index for every video on the web. Rasty Turek is the CEO of Pex, and in this episode he describes how to build a system that crawls the Internet and indexes videos. It’s a large scale engineering challenge, and there are lots of tradeoffs to be made between financial cost, speed, accuracy, and engineering complexity. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Different browsers consume JavaScript in different ways. When a new version of JavaScript comes out, developers are eager to use the new functionality of that language version. But if you are writing frontend JavaScript code, that code needs to be interpretable by every browser that might consume it–whether the consumer is on an iPhone running Safari or a Windows machine running Internet Explorer 11. Babel is a transpiler for JavaScript. Babel allows new versions of JavaScript to be consumed by older browsers by translating new language features of JavaScript into code that is readable by an older JavaScript interpreter. Babel does this by parsing JavaScript code, creating an abstract syntax tree, and manipulating the AST to make that code comply with the old browser. Henry Zhu is a core maintainer of Babel and a full-time open source developer. In today’s episode, Henry explains how Babel works and its various applications. He also talks about life as a full-time open source developer, where he earns a living through Patreon and OpenCollective. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Over the last decade, cloud computing made it easier to programmatically define what infrastructure we have running, and perform operations across that infrastructure. This is called “infrastructure as code.” Whether you want to backup a database, deploy a new version of a service, or introduce a new tier of load balancers, the changes that we make across our infrastructure can be done programmatically, instead of through a series of manual steps. As infrastructure got turned into code, operations people started working more like developers, and developers began to do the work of operations–a convergence known as “devops.” At Google, this “devops” movement was manifested in a role called “site reliability engineer.” In previous shows, we have explored site reliability engineering culture. Laine Campbell is a senior VP of engineering at Fastly, and the author of the book Database Reliability Engineering. In this book, Laine describes how the ideas of site reliability engineering can be extended to databases. Laine joins the show to discuss the book, and how engineering teams can build effective workflows around databases. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Rust is a systems programming language with a distinct set of features for safety and concurrency. In previous shows about Rust, we explored how Rust can prevent crashes and eliminate data races through its approach to type safety and memory management. Rust’s focus on efficiency and safety makes it a promising language for networking code. Tokio is a set of networking libraries built on Rust. Tokio enables developers to write asynchronous IO operations by way of its multithreaded scheduler. Tokio’s goal is to make production-ready clients and servers easy to create by focusing on small, reusable components. Carl Lerche is an engineer at Buoyant, a company that makes the popular Linkerd and Conduit service mesh systems. Kubernetes developers deploy service mesh to their distributed applications as sidecar proxies. These proxies need to be low-latency and highly reliable. In that light, it makes sense that Conduit (the more recent service mesh from Buoyant) is built using Rust. Carl joins the show to describe why Rust is useful for building networked services. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Twenty years ago, all of the data in an organization could fit inside of relational databases. Imagine a company like Proctor and Gamble. P&G is a consumer packaged goods company with hundreds of business sectors–shaving products, toothpaste, shampoo, laundry detergent. Twenty years ago, if the chief financial officer of P&G wanted to answer a question about the revenue projections within the enterprise, that CFO would ask a VP to find the answer. The VP would contact the business analysts in all the different departments within Procter and Gamble, and those business analysts would all work with database administrators to answer questions for their business sector. In that world, it might have taken weeks or months for the CFO to get the answer about revenue projections. Today, data engineering has improved dramatically. Data sets within an enterprise are updated more rapidly. The tooling has advanced thanks to the Hadoop project leading to a wide range of open source projects that feed into one another. But data problems across an enterprise still exist. Business analysts, data scientists, and data engineers struggle to communicate with each other. The CFO still can’t get a question about revenue projections answered instantly. Instead of instant answers, we live in a world of friction, batch processing, and monthly reports. And this is not just true of old enterprises like P&G. It is true of newer startups like Uber, Airbnb, and Netflix. It seems that no amount of engineers and financial windfall can completely cure the frictions of the modern data platform. Tomer Shiran started Dremio to address the long-lived problems of data management, data access, and data governance within an enterprise. Dremio connects databases, storage systems, and business intelligence tools together, and uses intelligent caching to make commonly used queries within an organization more readily accessible. Dremio is an ambitious project that spent several years in stealth before launching. In today’s episode, Tomer gives a history of data engineering, and provides his perspective on how the data problems within an organization can be diminished. Full disclosure: Dremio is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Evolutionary algorithms can generate surprising, effective solutions to our problems. Evolutionary algorithms are often let loose within a simulated environment. The algorithm is given a function to optimize for, and the engineers expect that algorithm to evolve a solution that optimizes for the objective function given the constraints of the simulated environment. But sometimes these results are not exactly what we are looking for. For example, imagine an evolutionary algorithm that tries to evolve a creature that do a flip within a simulated physics engine that mirrors the real world. You could imagine all sorts of evolutionary traits. Maybe the creature will evolve to have legs that are like springs, and let the creature jump high enough to do a flip. Maybe the creature will develop normal legs with strong muscles that propel the creature high enough to flip. But you wouldn’t expect the creature to evolve to be extremely tall–so tall that the creature can merely lean over fast enough so that the top of its body flips upside down. In one experiment, this is exactly what happened. In another, similar experiment, the evolving creature discovered a bug in the physics engine of the simulated environment. This creature was able to exploit the problem with this physics engine to be able to move in ways that would not be possible in our real-world physical universe. Evolutionary algorithms sometimes evolve solutions in ways that we don’t expect. Researchers usually throw those results away, because they don’t contribute to the result that the researchers are looking for. The consequence is that lots of interesting anecdotes get lost. Joel Lehman, Dusan Misevic, and Jeff Clune are the lead authors of the paper “The Surprising Creativity of Digital Evolution.” The paper was a collection of anecdotes about strange results within the world of digital evolution. They join the show to describe what digital evolution is, and some of the strange results that they surveyed in their paper. Joel and Jeff are engineers at Uber’s artificial intelligence division–so this topic has applicable importance to them. Machine learning is all about evolution within simulated environments, and developing safe algorithms for AI requires an understanding of what can go wrong in a poorly defined evolutionary system. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

If you have ever stayed in a short-term rental (like an Airbnb, HomeAway, or CouchSurfing), you have probably used the wifi network at that rental property. Why wouldn’t you? It’s no different than hopping on an open wifi network at an airport, or a Starbucks, or your friend’s house, right? One major difference: the hardware is easily accessible to previous guests at the short-term rental. Previous guests could tamper with the software on a router, and use that tampering to do some malicious, surveillant things. Jeremy Galloway is a security engineer at Atlassian. In today’s show, he explains the risk of using wifi at a short-term rental like an Airbnb–including an explanation of how easy it is to take over a wifi network as a guest at a rental property. A broader point we discuss: large attack surfaces are difficult to secure. Whether we are talking about Airbnb, or another sharing economy app like Uber, or a large corporate network like Atlassian, or even your own personal life. Jeremy offers some best practices and philosophies for how to respond to the modern world of security. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Relational databases have been popular since the 1970s, but in the last 20 years the amount of data that applications need to collect and store has skyrocketed. The raw cost to store that data has decreased. There is a common phrase in software companies: “it costs you less to save the data than to throw it away.” Saving the data is cheap, but accessing that data in a useful way can be expensive. Developers still need rapid row-wise and column-wise access to the data. Accessing an individual row of a database can be useful if a user is logging in and you want to load all of that user’s data, or if you want to update a banking system with a new financial transaction. Accessing an entire column of a database can be useful if you want to aggregate summaries of all of the entries in a system–like the sum of all financial transactions in a bank. These different kinds of transactions are nothing new, but with the growing scale of data, companies are changing their mentality from thinking in terms of individual databases to thinking about distributed “data platforms.” In a data platform, the data across a company might be put into a variety of storage systems–distributed file systems, databases, in-memory caches, search indexes–but the API for the developer is kept simple. And the simplest, most commonly understood language is SQL. Marco Slot is an engineer with Citus Data, a company that makes Postgres scalable. Postgres is one of the most common relational databases, and in this episode Marco describes how Postgres can be used to service almost all of the needs of a data platform. This isn’t easy to do, as it requires sharding your growing relational database into clusters and orchestrating distributed queries between those shards. In this show, Marco and I discuss Citus’s approach to the distributed systems problems of a sharded relational database. This episode is a nice complement to previous episodes we have done with Ozgun and Craig from Citus, in which they gave a history of relational databases, and explained how Postgres compares to the wide variety of relational databases out there. Full disclosure: Citus Data is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the tech industry, we have all grown to fear “lock-in.” Lock-in is a situation in which you have no choice but to pay a certain provider for some aspect of your computer services. Since computers are so fundamental to our lives, we sometimes have no choice but to pay the provider of that lock-in service. Think of a few service providers in your life who have no serious competition. What is your relationship to that service provider? Do you feel like you are paying too much money? Do you wish that you could switch? This is how many people feel about their Internet service provider. An Internet Service Provider is the company that provides you with the “last mile” of physical infrastructure that connects you to the rest of the Internet. Different forms of ISP include cable ISPs, satellite ISPs, fiber ISPs, and copper/DSL ISPs. The medium of delivery varies, but the functionality is the same. This company is crucial to your Internet access. In many geographic locations, there are very restricted options for which ISP you could use. Why is that? Many people assume that there is some physical or regulatory barrier to starting an ISP. In fact, there are fewer barriers than you might think. Adam Montgomery is a co-founder of Necto, a company that provides an ISP starter kit. If you want to start your own ISP in an apartment building or in your neighborhood or wherever you are, the Necto ISP starter kit can help you get off the ground. That might sound like a crazy idea, but in this episode Adam explains why it is not so crazy–why the technology around ISPs is more broadly accessible than many people believe. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Big blocks or small blocks: this is the fundamental question of Bitcoin scalability. The argument for big blocks is also known as “on-chain scalability.” Under this strategy, each block in the append-only chain of Bitcoin transaction blocks would grow in size to be able to support lower transaction fees and higher on-chain throughput. A set of Bitcoin users who supported this idea forked Bitcoin to create Bitcoin Cash, a version of Bitcoin that has a larger block size. The argument for small blocks asserts that scaling Bitcoin does not require a larger block size. Under this model, the scaling demands of the Bitcoin blockchain would be handled by sidechains. A sidechain is a network of person-to-person payment channels that only reconcile with the Bitcoin blockchain to checkpoint batches of transactions. These sidechains can be connected together to form the “lightning network.” Lightning network is hard to implement. To implement a lightning network requires solving real-world distributed systems problems that are unprecedented. It’s much more complicated than deploying a blockchain with a larger block size. In addition, opponents of lightning network suggest that this will lead to a centralized banking system being constructed on top of Bitcoin. Opponents of lightning network fear that instead of a decentralized payments network, the world with lightning network will be a lower cost version of the present financial system, in which JP Morgan and Blockstream partner up to battle Coinbase in a centralized war for control of the unbanked. These big blockers argue that the new banks on the lightning network will be just like the old banks–censorious of transactions and held in the domineering palms of the global financial kleptocracy. So why bother with the lightning network approach? Why are we building this inelegant, kludgey system of off-chain, potentially centralized banking 2.0 complexity? Why not just increase the block size indefinitely and keep things simple? And even if we increased the block size today, couldn’t we still deploy lightning network in the future while appeasing the transaction volume of today? One major reason is that growing the block size does have a cost. The bigger the block size, the more demands it places on any node that wants to maintain a record of those blocks. And if you grow the block size today, you forego the experiment of seeing whether a small block size plus lightning network could in itself handle the transaction volume of a global financial system. The framing of “big blockers versus small blockers” is a conveniently polarized reduction of a much more granular reality. To believe that there is no subtlety between the two sides of this debate is to underestimate the number of dimensions to this argument. It’s an unfortunate side effect of rigidly programmed Twitter bots, and a political atmosphere in which your lines in the sand are demarcated by which subreddit you choose to affiliate with. That said–my impression is that the more experienced engineers are overwhelmingly on the side of small blocks plus lightning network as the most promising approach to scaling Bitcoin. Take whatever side of the debate you want. A single line of Bitcoin core code speaks much louder than an avalanche of tweets. In today’s episode, Jameson Lopp joins the show to explain why lightning network is an appealing engineering construct. We play the devil’s advocate and contrast lightning network with a big block approach, as well as a big block plus lightning network approach. Jameson also describes his experience working within the Ethereum ecosystem, and gives a sober explanation of some of the issues that Ethereum scalers may themselves encounter.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Investing is an infinite game. In a game, a player can formulate a strategy based on the available resources, the apparent variance of the environment, and the metagame of the other actors involved. For an investor, the game board includes companies, currencies, and people. A successful game player can model their actions mathematically. They can describe a thesis for an in-game decision with clear language. Game players who reason through “gut feeling” do not perform well (unless their “gut” is aligned with correct mathematical heuristics). The same is true for investors. An investor who is going to be successful in the long term will be able to explain their investment thesis crisply. Each investment represents a bet with net positive expected value. The expected value of an investment is the sum of all potential probability-weighted future outcomes of a business. Each of those potential expected outcomes is the anticipated outcome times the probability that the investment works out in the anticipated way. Brian Singerman is a computer scientist and partner at Founder’s Fund. He is on the board of Affirm, AltSchool, Emerald Therapeutics, and a variety of other companies in disparate areas. He also plays lots of board games. Brian was a lot of fun to talk to because he was willing to field questions from an expansive range of topics–and he answered them so quickly and concisely that I started to get nervous that I was going to run out of things to ask him. Many of the businesses Brian has invested in do not have a well-defined historical precedent. If a venture capital investor was trying to make bets in defined “sectors” that investor would probably overlook a business like Forward (a vertically integrated healthcare company) or Cloud9 (a collection of esports teams). If an investment does not have a historical precedent, it’s harder to reason about it by analogy. You have to judge it by fundamental reasoning: the current market, the capability of the founders, and the economics of the business model. In many professions, reasoning by analogy will work out perfectly fine. You can pattern match on the past, and use that to justify decisions for the future. But if your professional livelihood depends on reasoning by fundamental principles, you get trained to assess situations that do not have precedent. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Moore’s Law states that the number of transistors in a dense integrated circuit doubles about every two years. Moore’s Law is less like a “law” and more like an observation or a prediction. Moore’s Law is ending. We can no longer fit an increasing amount of transistors in the same amount of space with a highly predictable rate. Dennard scaling is also coming to an end. Dennard scaling is the observation that as transistors get smaller, the power density stays constant. These changes in hardware trends have downstream effects for software engineers. Most importantly–power consumption becomes much more important. As a software engineer, how does power consumption affect you? It means that inefficient software will either run more slowly or cost more money relative to our expectations in the past. Whereas software engineers writing code 15 years ago could comfortably project that their code would get significantly cheaper to run over time due to hardware advances, the story is more complicated today. Why is Moore’s Law ending? And what kinds of predictable advances in technology can we still expect? John Hennessy is the chairman of Alphabet. In 2017, he won a Turing award (along with David Patterson) for his work on the RISC (Reduced Instruction Set Compiler) architecture. From 2000 to 2016, he was the president of Stanford University. John joins the show to explore the future of computing. While we may not have the predictable benefits of Moore’s Law and Dennard scaling, we now have machine learning. It is hard to plot the advances of machine learning on any one chart (as we explored in a recent episode with OpenAI). But we can say empirically that machine learning is working quite well in production. If machine learning offers us such strong advances in computing, how can we change our hardware design process to make machine learning more efficient? As machine learning training workloads eat up more resources in a data center, engineers are developing domain specific chips which are optimized for those machine learning workloads. The Tensor Processing Unit (TPU) from Google is one such example. John mentioned that chips could become even more specialized within the domain of machine learning. You could imagine a chip that is specifically designed for a LSTM machine learning model. There are other domains where we could see specialized chips–drones, self-driving cars, wearable computers. In this episode, John describes his perspective on the future of computing, and offers some framework for how engineers can adapt to that future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A database stores data to an underlying section of storage. If you are an application developer, you might think of your persistent storage system as being the database itself–but at a lower level, that database is writing to block storage, file storage, or object storage. A container orchestration system manages application containers. If you want to run WordPress (a blogging platform) on Kubernetes, that means you also need to run a database to store your blog posts in a persistent way. To run a database, you need to have an underlying storage medium–which could be a disk that is at your on-prem data center, or block storage on a disk at a cloud provider. Kubernetes is not the only container orchestrator. There’s also Cloud Foundry, Mesos, Docker Swarm, and several others. Each of these container orchestrators needs to be able to run a variety of persistent workloads (such as a MySQL database or a Kafka cluster). Each of these persistent workloads needs to be able to use different types of backing storage. With the range of container orchestrators and the range of backing storage types, a problem arises. Every storage type would have to write custom code to connect to each container orchestrator. The solution to this is the CSI: the container storage interface. The CSI is an interface layer between the container orchestrator and the backing storage system. In today’s episode, Jie Yu from Mesosphere describes the motivation for the CSI, and gives an overview for its design principles. There are great lessons here for anyone working with containers or distributed systems in general. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When software is performing suboptimally, the programmer can use a variety of tools to diagnose problems and improve the quality of the code. A profiler is a tool for examining where a program is spending time. Every program consists of a set of different functions. These functions call each other. The total amount of time that your program runs is the sum of the time your program spends in all of the different functions. When you run a program, you can execute a profiler on that program, and the profiler will give you a breakdown of which of the different functions time is being spent in. If you have function A, B, and C, your profiler might say that your program is spending 30% of its time in function A, 20% of its time in function B, and 50% of its time in function C. Julia Evans is a software engineer at Stripe, and the creator of a Ruby profiler called rbspy. rbspy can execute on a running Ruby program and report back with a profile. As Julia explains, a profiler turns out to be a non-trivial piece of software to build. To introspect a Ruby program, you need to understand how the Ruby interpreter is translating Ruby code into C structs for execution. This episode is about profilers–but in order to talk about profilers, we also have to talk about Ruby, the Ruby interpreter, and the way that executing programs are laid out in memory. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Applications of artificial intelligence are permeating our everyday lives. We notice it in small ways–improvements to speech recognition; better quality products being recommended to us; cheaper goods and services that have dropped in price because of more intelligent production. But what can we quantitatively say about the rate at which artificial intelligence is improving? How fast are models advancing? Do the different fields in artificial intelligence all advance together, or are they improving separately from each other? In other words, if the accuracy of a speech recognition model doubles, does that mean that the accuracy of image recognition will double also? It’s hard to know the answer to these questions. Machine learning models trained today can consume 300,000 times the amount of compute that could be consumed in 2012. This does not necessarily mean that models are 300,000 times better–these training algorithms could just be less efficient than yesterday’s models, and therefore are consuming more compute. We can observe from empirical data that models tend to get better with more data. Models also tend to get better with more compute. How much better do they get? That varies from application to application, from speech recognition to language translation. But models do seem to improve with more compute and more data. Dario Amodei works at OpenAI, where he leads the AI safety team. In a post called “AI and Compute,” Dario observed that the consumption of machine learning training runs is increasing exponentially–doubling every 3.5 months. In this episode, Dario discusses the implications of increased consumption of compute in the training process. Dario’s focus is AI safety. AI safety encompasses both the prevention of accidents and the prevention of deliberate malicious AI application. Today, humans are dying in autonomous car crashes–this is an accident. The reward functions of social networks are being exploited by botnets and fake, salacious news–this is malicious. The dangers of AI are already affecting our lives on the axes of accidents and malice. There will be more accidents, and more malicious applications–the question is what to do about it. What general strategies can be devised to improve AI safety? After Dario and I talk about the increased consumption of compute by training algorithms, we explore the implications of this increase for safety researchers. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cryptocurrency infrastructure is a new form of software. Thousands of developers are submitting transactions to Bitcoin and Ethereum, and this transaction volume tests the scalability of current blockchain implementations. The bottlenecks in scalability lead to slow transaction times and high fees. Over the last twenty years, engineers have learned how to scale databases. We’ve learned how to scale Internet applications like e-commerce stores and online games. It’s easy to forget, but there was a time when those systems didn’t perform well either. Scaling a blockchain is different than scaling a relational database or a microservices infrastructure. Blockchains are peer-to-peer databases with an append only ledger shared by thousands of nodes. With different scalability solutions, there are tradeoffs between decentralization, scalability, and security. As an example, in Bitcoin, the core developers are working towards deployment and adoption of lightning network. Some would argue that this approach favors scalability over decentralization. Today’s show is about scaling Ethereum. Raul Jordan and Preston Van Loon are developers who are part of Prysmatic Labs, a team building a sharding implementation for the Go Ethereum client. In this episode, we discuss Ethereum’s approaches to scaling, including sharding and Plasma. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ten years ago, a biology researcher was limited by the software tools available. Most of the electronic record keeping was done using Excel and other general purpose tools. Benchling is a suite of software tools that were designed to simplify the lives of life science researchers. Benchling helps with sample tracking, experiment design, and workflow management. Sherwin Yu is an engineering manager at Benchling, and he joins the show to discuss the workflows of the life scientist–how experiments are designed and managed. Life science researchers in both academia and industry use Benchling, and Sherwin spends time talking to them and understanding what they need from their tools. We also talked about the impact of CRISPR, robotic cloud laboratories, and other future developments. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Containers have improved deployments and resource utilization. Kubernetes created a platform to manage those containers and orchestrate them into distributed applications. In today’s episode, we explore tools that improve the workflow of the application developer who is working with Kubernetes, including Helm, Draft, and Brigade. Helm is a package manager for Kubernetes, which allows users to find, share, and use software that is built for Kubernetes. The unit of installation for Helm users is a Helm Chart. Installing a Helm Chart can simplify the deployment of a database, load balancer, or continuous integration tool. Draft is a tool for simplifying the containerization process. When a developer runs Draft, a Dockerfile is created to containerize the application, and a Helm Chart is created to enable the application to be easily deployed. Brigade is a tool for creating and running Kubernetes workflows. Brigade allows for event-driven scripting on top of Kubernetes. Chatops, continuous integration systems, and complex big data pipelines can all be defined with Brigade. Brigade is exciting, because it is a higher level tool on top of Kubernetes–in some ways similar to the “serverless on Kubernetes” systems we have covered in the past. Ralph Squillace is a principal program manager with Microsoft, where he works on containers, Linux, and cloud products. Ralph joins the show to talk about how developing with containers has changed in the last few years, and how it will continue to evolve in the near future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ad blockers in the browser protect us from the most annoying marketing messages that the Internet tries to serve to us. But we still pay a price for these ads. We pay the bandwidth costs of requesting these pages. Our browsers are slowed down by these extra requests. Pi Hole is a hardware based ad blocker. Pi Hole acts as a DNS server for all of the traffic that makes its way onto your network. Pi Hole has a blacklist of all the URLs to block–including tracking systems and ad networks. Pi Hole stops these URLs from communicating with all the devices on your network–including your cell phone. Jacob Salmela is the developer of Pi Hole, which he describes as a black hole for advertiser traffic. In this episode, we explain how traditional ad blocking in the browser works, and how things are improved with a piece of dedicated hardware doing the ad blocking. It was also a useful review of the relationship between URLs, IP addresses, your home network, and the broader Internet. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Self-driving, electric cars will someday outnumber traditional automobiles on the road. As transportation becomes autonomous, it is hard to imagine an industry that will not be affected by the downstream effects of this change. These cars will likely be managed by fleet operators like Lyft and Uber. We will need fewer cars, and the amount of space dedicated to those cars will shrink dramatically. Parking lots, massive roads, and gas stations will be reclaimed or repurposed. City planning departments will have to devise entirely new strategies. As the self-driving cars reach consumer availability, an intricate supply chain for these cars will develop. When smartphones became mass-produced, the costs of GPS devices, accelerometers, and other small components dropped steeply. A consequence of the smartphone supply chain was that other devices like consumer drones became affordable. The self-driving car supply chain will lead to the mass production of building blocks for other new devices. With fewer automotive fatalities, the economics of the car insurance industry might collapse completely. At a minimum, the costs of car insurance will likely shift to the fleet operators, who can purchase that car insurance at prices factoring in their large risk pool. Frank Chen is a deal and research partner with Andreessen Horowitz. In a series of presentations on the Autonomy Ecosystem, Frank explores the effects of our impending shift to self-driving electric cars. His analysis considers changes to energy infrastructure, the competitive landscape of software companies, and a range of other topics. Frank joins the show to discuss autonomous vehicles and the side effects of widespread autonomous deployments. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a user takes a ride on Uber, the app on the user’s phone is communicating with Uber’s backend infrastructure, which is writing to a database that maintains the state of that user’s activity. This database is known as a transactional database or “OLTP” (online transaction processing). Every active user and driver and UberEATS restaurant is writing data to the transactional data store. Periodically, that data is copied from the transactional data system to a different data storage system, where that data can be queried for large-scale data analysis. For example, if a data scientist at Uber wants to get the average amount of miles that a given user rode in February, that data scientist would issue a query to the analytical data cluster. Uber uses the Hadoop distributed file system (HDFS) to store analytical data. On this file system, Uber has a version history of all of the company’s useful historical data. Trip history, rider activity, driver activity–every data point that is in the transactional database–but in a file format that is easier to query for large scale processing. This file format is known as Parquet. Data scientists, machine learning engineers, and real-time application developers all depend on the massive quantities of data that are stored in these Parquet files on Uber’s HDFS cluster. To simplify the access of that data by many different clients, Uber uses Presto, an analytical query engine originally built at Facebook. Presto translates SQL queries into whatever query language is necessary to access the underlying storage medium–whether that storage system is an ElasticSearch cluster, a set of Parquet files, or a relational database. Presto is useful because it simplifies the relationship between data engineers and the application developers who are building on top of the data engineering infrastructure. In today’s show, Zhenxiao Luo joins to give an end-to-end description of Uber’s data infrastructure–from the ingest point of the OLTP database to the OLAP data storage system on HDFS, to the wide range of data systems and applications that run on top of that OLAP data. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The world of software moves faster than the laws that regulate it. When software companies do get regulated, that regulation is often enforced unevenly among different companies. Software continually presents the legal system with new requirements. Consumer data privacy needs to enforced on a granular level. Software developers need a system of protecting their intellectual property. When a company becomes dominant, our legal system needs to scrutinize that company for potential antitrust violations. Micah Kesselman is a lawyer specializing in software IP prosecution. Prior to becoming a lawyer, he studied computer science. He joins the show to discuss a range of issues at the intersection of software and the law–including GDPR, software patents, and self-driving cars. These are topics we will cover in more detail in the future, but it was great to have Micah bring the perspective of a lawyer to the show. Massachusetts Autonomous Vehicles Working Group Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Deploying software to a container presents a different security model than deploying an application to a VM. There is a smaller attack surface per container, but the container is colocated on a node with other containers. Containers are meant to have a shorter lifetime than VMs, so there are generally fewer consequences if a container needs to be destroyed and rebuilt due to a potential security vulnerability. Maya Kaczorowski works on container security at Google. In a recent talk at KubeCon, Maya discussed runtime security of containers on Kubernetes. Maya joins the show to discuss container security, and what it means to software developers and operators. Maya also gives guidelines for evaluating the security of your own cluster. We talked about the security benefits of a managed Kubernetes provider, and also explored how some container security vendor software works. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A sample of the human voice is a rich piece of unstructured data. Voice recordings can be turned into visualizations called spectrograms. Machine learning models can be trained to identify features of these spectrograms. Using this kind of analytic strategy, breakthroughs in voice analysis are happening at an amazing pace. Rita Singh researches voice at Carnegie Mellon University. Her work studies the high volume of latent data that is available in the human voice. As she explains, just a small fragment of a human voice can be used to identify who a speaker is. Your voice is as distinctive as your fingerprint. Your voice can also reveal medical conditions. Features of the human voice can be strongly correlated with psychiatric symptom severity, and potentially heart disease, cancer, and other illnesses. The human voice can even suggest a person’s physique–your height, weight, and facial features. In this episode, Rita explains the machine learning techniques that she uses to uncover the hidden richness of the human voice. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data Skeptic is a podcast about machine learning, data science, and how software affects our lives. The first guest on today’s episode is Kyle Polich, the host of Data Skeptic. Kyle is one of the best explainers of machine learning concepts I have met, and for this episode, he presented some material that is perfect for this audience: machine learning for software engineers. Second Spectrum is a company that analyzes data from professional sports, turning that data into visualizations, reports, and futuristic sports viewing experiences. We had a previous show about Second Spectrum where we went into the company in detail–it was an excellent show, so I wanted to have Kevin Squire, an engineer from Spectrum, come on the show to talk about how the company builds machine learning tools to analyze sports data. If you have not seen any of the visualizations from Second Spectrum, stop what you are doing and watch a video on it! This year we have had three Software Engineering Daily Meetups: in New York, Boston, and Los Angeles. At each of these Meetups, listeners from the SE Daily community got to meet each other and talk about software–what they are building and what they are excited about. I was happy to be in attendance at each of these, and I am posting the talks given by our presenters. The audio quality is not perfect on these, but there are also no ads. Thanks to Telesign for graciously providing a space and some delicious food for our Meetup. Telesign has beautiful offices in Los Angeles, and they make SMS, voice, and data solutions. If you are looking for secure and reliable communications APIs, check them out. We’d love to have you as part of our community. We will have more Meetups eventually, and you can be notified of these by signing up for our newsletter. Come to SoftwareDaily.com and get involved with the discussion of episodes and software projects. You can also check out our open source projects–the mobile apps, and our website.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Voice interfaces are a newer form of communicating with computers. Alexa is a voice interface platform from Amazon. Alexa powers the Amazon Echo, as well as Alexa-enabled cars, refrigerators, and dishwashers. Any developer can build a device with a voice interface using a Raspberry Pi. Paul Cutsinger works on Echo and Alexa at Amazon. He’s focused on growing the market of developers who are building voice interfaces. In this episode, Paul describes how to design and implement a voice application for the Amazon Alexa platform. The market for voice powered apps is so new, and there has yet to be a “killer app.” If you like to tinker on new platforms, you will like this episode–and I was surprised by how easy it sounds to build a voice app. Personally I use voice interfaces all the time–to set timers, to find out how to tell if a cucumber has gone bad, to ask what temperature to cook a potato at. Sometimes, when I am lying in bed trying to get to sleep, I will ask my nearest device to read me a Wikipedia article. These are great use cases, but I’m sure we will see something much more groundbreaking in the future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Message broker systems decouple the consumers and producers of a message channel. In previous shows, we have explored ZeroMQ, PubNub, Apache Kafka, and NATS. In this episode, we talk about another message broker: Apache Pulsar. Pulsar is an open source distributed pub-sub message system originally created at Yahoo. It was used to scale products with high volumes of users–such as Yahoo Mail. There are three components of a Pulsar deployment: the Pulsar broker (which handles the message brokering), Apache Bookkeeper (which handles the durable storage of the messages), and Apache Zookeeper, which manages the distributed coordination. Lewis Kaneshiro joins the show to describe how Apache Pulsar works, and how it compares to other messaging systems like Apache Kafka. Lewis is the CEO of Streamlio, a company that builds messaging and stream processing systems for enterprises, and uses Pulsar in its core product. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Gloo is a function gateway built on top of the popular open source project Envoy. The goal of Gloo is to decouple client-facing APIs from upstream APIs. Gloo is similar to an API gateway, which is a tool that software companies can use to collect all their APIs and one place and impose security, monitoring, and other standards around those APIs. The goal of Gloo is to provide all the tools necessary to glue together traditional and cloud-native applications. Idit Levine is the CEO of Solo.io, a company that is building Gloo and several other projects. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

YouTube runs a large MySQL database to hold the metadata about its videos. As YouTube scaled, the database was sharded, and applications within YouTube had to write queries that were aware of the sharding layout of that database. This is problematic, because it pushes complexity to the application developer. An application developer shouldn’t have to be aware of how a database is laid out among different nodes. The developer should be able to issue a query, and have the cluster simply return the data. Vitess is an open source system for scaling large MySQL databases. Sugu Sougoumarane co-created Vitess at YouTube. Since YouTube is owned by Google, Vitess was able to leverage the Borg cluster manager developed at Google. Once Kubernetes came to market, it became more viable to make Vitess accessible to open source developers. Sugu joins the show to talk about the scalability problems that YouTube’s database infrastructure encountered and the motivations for building Vitess. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Kubernetes ecosystem consists of enterprises, vendors, open source projects, and individual engineers. The Cloud Native Computing Foundation was created to balance the interests of all the different groups within the cloud native community. CNCF has similarities to the Linux Foundation and the Apache Foundation. CNCF helps to guide open source projects in the Kubernetes ecosystem–including Prometheus, Fluentd, and Envoy. With the help of the CNCF, these projects can find common ground where possible. KubeCon is a conference organized by the Cloud Native Computing Foundation. I attended the most recent KubeCon in Copenhagen. KubeCon was a remarkably well-run conference–and the attendees were excited and optimistic. As much traction as Kubernetes has, it is still very early days and it was fun to talk to people and forecast what the future might bring. At KubeCon, I sat down with Chris Aniszczyk and Dan Kohn, who are the COO and director of the CNCF. I was curious about how to scale an organization like the CNCF. In some ways, it is like scaling a government. Kubernetes is growing faster than Linux grew, and the applications of Kubernetes are as numerous as those of Linux. Different constituencies want different things out of Kubernetes–and as those constituencies rapidly grow in number, how do you maintain diplomacy among competing interests? It’s not an easy task, and that diplomacy has been established by keeping in mind lessons from previous open source projects. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Mesos is a system for managing distributed systems. The goal of Mesos is to help engineers orchestrate resources among multi-node applications like Spark. Mesos can also manage lower level schedulers like Kubernetes. A common misconception is that Mesos aims to solve the same problem as Kubernetes, but Mesos is a higher level abstraction. Ben Hindman co-founded Mesosphere to bring the Mesos project to market. Large enterprises like Uber, Netflix, and Yelp use Mesosphere for resource management. Before he started the company, Ben worked in the Berkeley AMP Lab, a research program where the Spark and Tachyon projects were also born. At this point, he has spent significant time in both academia and industry. This conversation spans distributed systems theory, history, and practice. Ben and I spoke at KubeCon 2018 in Copenhagen–which was an amazing conference. We were both amazed at how big the audience for Kubernetes has gotten, and the pace at which the technology is advancing. Today, Kubernetes is mostly used for scheduling containerized applications that engineers have built themselves. But there will be higher level tools that use Kubernetes as a building block. Much like Zookeeper was used as a building block for Hadoop, Kubernetes will be used to build serverless applications and distributed databases. Once you are using a distributed database built on Kubernetes, you don’t want to think about the container orchestration–you want to think about the raw storage and CPU requirements for that database. This is one reason why Mesos is so compelling. Since Kubernetes creates an increased cardinality of distributed systems, it’s good to know that there is a framework built to manage those higher level applications. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Algorithms for building neural networks have existed for decades. For a long time, neural networks were not widely used. Recent changes to the cost of compute and the size of our data have made neural networks extremely useful. Our smart phones generate terabytes of useful data. Lower storage costs make it economical to keep that data. Cloud computing democratized the ability to do large scale machine learning across GPUs. Over the last few years, these trends have been driving widespread use of deep learning, in which neural nets with a large series of layers are used to create powerful results in various fields of classification and prediction. Neural networks are a tool for making sense of unstructured data–text, images, sound waves, and videos. “Unstructured” data is data with high volume or high dimensionality. For example, an image has a huge collection of pixels, and each pixel has a color value. One way to think about image classification is that you are finding correlations between those pixels. A certain cluster of pixels might represent an edge. After doing edge detection on pixels, you have a collection of edges. Then you can find correlations between those edges, and build up higher levels of abstraction. Yinyin Liu is a principal engineer and head of data science at the Intel AI products group. She studies techniques for building neural networks. Each different configuration of a neural network for a given problem is called a “topology.” Engineers are always looking at new topologies for solving a deep learning application–such as natural language processing. In this episode, Yinyin describes what a deep learning topology is and describes topologies for natural language processing. We also talk about the opportunities and the bottlenecks in deep learning–including why the tools are so immature, and what it will take to make the tooling better. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud computing lowered the cost and improved accessibility to tools for storing large volumes of data. In the early 2000s, Hadoop caused a revolution in large scale batch processing. Since then, companies have been building ways to store and access their data faster and more efficiently. At the same time, the sheer volume of data has increased and machine learning has given rise to methods of extracting signal from seemingly inconsequential data points. This confluence of factors gave rise to the role of the data engineer. A data engineer defines the data pipeline and supports data scientists and machine learning engineers. Tobias Macey hosts the “Data Engineering Podcast,” where he covers the fast moving world of data engineering–including databases, cloud providers, and open source tools. Tobias and I covered a range of topics in the data engineering space and also spent significant time discussing the world of software engineering podcasting. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Starting an Internet business is harder than it should be. You need to incorporate, create an operating agreement, set up a system to accept payments, and many other straightforward tasks. In the 1990s, this was how it felt to set up anything on the Internet. You always had to stand up a web server on your own infrastructure, before you could get to the interesting part–which was building an actual product. With the popularization of cloud computing, it became massively easier to stand up a server. Because of that lower activation energy, millions of applications and thousands of software businesses got started. But the activation energy required to start a business remains higher than necessary. It feels like standing up a web server in the 90s–lots of tedium and reinventing the wheel that has been done by people before you. This is the motivation behind Stripe Atlas, a project to simplify the process of starting an Internet business. Patrick McKenzie works on Atlas at Stripe. He was previously on the show to discuss his experience leaving a large corporation to work on his own small software companies. And his name has become synonymous with the modern phenomenon of the small software company–he has been writing about this topic for over a decade at Kalzumeus.com. It was great to talk to Patrick once again about Internet businesses, and I’m excited to see Stripe Atlas become something huge. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When I buy a mattress online, I pay for it with my credit card. Behind the scenes, a complex series of transactions occur between a payment gateway, the credit card company, and a few banks. There are problems with this process–it is slow, complex, and involves the synchronization of several different parties. Some consumers will not want to purchase the mattress because they do not have cash, and the lending rates they get offered are higher than they are willing to spend. If these consumers were presented with more intelligent loan rates, the lender could still make money, the mattress company could still make money, and the consumer would get a new mattress. It’s a missed opportunity all around. Affirm is a consumer financial services company. Their first product offers loans to consumers making purchases. In today’s episode, Affirm CTO Libor Michalek explains how Affirm decided to build this product, and what they have done to scale it. The conversation took me by surprise. Affirm was started by Max Levchin, who was a co-founder of PayPal. I assumed that when Affirm was created, they already knew exactly what they were going to build–because Affirm is a payments company and Max has had knowledge of the payments industry going back two decades. In reality, Affirm started out with more vague ideas around what they were building. They spent some time running small experiments as they looked for product/market fit–just like a bootstrapped startup would have. It was inspiring to know that even an experienced team is willing to go through the humble process of searching for a product within a space they are deeply familiar with. We didn’t get to all the questions I was planning to explore, but I hope to do another show about Affirm in the future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Superpedestrian is a robotic bicycle wheel that learns how you pedal and personalizes your bicycle ride. The engineering challenges of Superpedestrian are at the intersection of robotics, software, and real-time analytics. The first half of today’s show is about Superpedestrian. Goss Nuzzo Jones and Matt Cole are engineers at Superpedestrian. The slides for their presentation are also in the show notes. The second half of today’s show is about HubSpot, a massive business with lots of infrastructure challenges. Thomas Petr explained how HubSpot’s engineering has matured, and some of the scaling problems they have tackled. Last month, we had three Software Engineering Daily Meetups: in New York, Boston, and Los Angeles. At each of these Meetups, listeners from the SE Daily community got to meet each other and talk about software–what they are building and what they are excited about. I was happy to be in attendance at each of these, and I am posting the talks given by our presenters. The audio quality is not perfect on these, but there are also no ads. Thank you to HubSpot for hosting this Meetup–they have beautiful offices and if you are looking for a job (or if you want to host a technology Meetup in the Boston area) I strongly recommend checking them out. We’d love to have you as part of our community. We will have more Meetups eventually, and you can be notified of these by signing up for our newsletter. Come to SoftwareDaily.com and get involved with the discussion of episodes and software projects. You can also check out our open source projects–the mobile apps, and our website. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Phones are constantly tracking the location of a user in space. Devices like cars, smart watches, and drones are also picking up high volumes of location data. This location data is also called “geospatial data.” The amount of geospatial data is rapidly increasing, and there is a growing demand for software to perform operations over that data. Geospatial data sets are often massive–so it is non-trivial to perform operations over this data. Geospatial data can consist of something as simple as a set of latitude/longitude data points. A single lat/long coordinate pair can be enriched with information about what ZIP code it is in, how far that data point is from the other data points in the set, and where the nearest coffee shop is in relation to that data point. Ram Sriharsha created Magellan, a geospatial analytics library for Spark. In today’s show, Ram describes the set of problems within the domain of geospatial analytics engineering. Ram also works as a product manager for Apache Spark at Databricks. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Last year, the WannaCry ransomware attack shut down hospitals, public transportation systems, and governments, demanding payment to unlock key computer systems. A programmer named Marcus Hutchins was able to stop WannaCry by registering a DNS entry buried in the WannaCry code. Not long after he stopped the WannaCry attack, Marcus Hutchins was arrested at a security conference in Las Vegas. Marcus’s arrest was due to actions that were unrelated to WannaCry. He is accused of writing a piece of malware called Kronos. Marcus volunteered his time to help stop WannaCry–a piece of ransomware that threatened to cause billions of dollars in damages. Whether or not he was a black hat in the past, perhaps Marcus should be absolved of his past actions. Reeves Wiedeman is a journalist with New York Magazine, and he joins the show to tell the story of WannaCry’s Gray Hat: Marcus Hutchins. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Alexis Le-Quoc started Datadog in 2010, after living through the Internet boom and bust cycle of the late 90s and early 2000s. In 2010, cloud was just starting to become popular. There was a gap in the market for infrastructure monitoring tools, which Alexis helped fill with the first version of Datadog. Since 2010, the number of different cloud infrastructure products has proliferated. There were new databases, queueing systems, virtualization and containerization tools. Web 2.0 took off, and thousands of new Internet businesses got started. Many of these businesses used Datadog to monitor their increasingly wide range of infrastructure configurations–and Datadog began to scale. On today’s show, Alexis tells the story of how Datadog grew from its first product into a variety of tools–infrastructure monitoring, logging, and application performance monitoring. Monitoring is a unique challenge–there is a ton of data, the data is latency sensitive, and the data is operationally important. These engineering constraints provide for a great conversation. Alexis is the CTO of Datadog, and we talked about cloud providers, building a business, infrastructure, and how to scale engineering management. Full disclosure: Datadog is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Technology is pushing us rapidly toward a future that is impossible to forecast. We try to imagine what that future might look like, and we can’t help having our predictions shaped by the media we have consumed. 1984, Terminator, Gattaca, Ex Machina, Black Mirror–all of these stories present a dystopian future. But if you look around the world, the most successful technologists are mostly guided by a sense of optimism. Technologists themselves are mostly idealistic–they see the future through a utopian lens. Popular media largely tells a different story: that we are headed for a dystopian world. Why is there such a gulf in the level of idealism between technologists and the media? Mike Solana found himself asking that question on a regular basis during his work at Founder’s Fund, where he is a vice president. Founder’s Fund has a bias toward funding difficult, cutting-edge technology like gene editing, robotics, and nuclear energy. This technology that Mike was seeing made him excited about the future–which led to his creation of the podcast “Anatomy of Next.” “Anatomy of Next” has explored biology, robotics, nuclear energy, superintelligence, and the nature of reality. Soon the podcast will be exploring how our civilization will explore and settle the solar system–specifically Mars. I’ve listened through the entire first season of the show twice and enjoyed it so much because Mike explores questions that are on the border of philosophy and technology–questions about the nature of reality, and what makes us human–and nobody can give perfect answers to these questions. But Mike interviews top experts on the show, which provides us with a framework. Guests on “Anatomy of Next” include Nick Bostrom (the author of Superintelligence), George Church (a pioneer in gene editing), and Palmer Luckey (the founder of VR company Oculus). Mike joins the show to talk about why he started “Anatomy of Next,” and his own perspective on the future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Podcasting about cryptocurrencies is a strange occupation. You get emails all the time from companies doing a token sale that you would never want to be affiliated with. You get angry tweets from anonymous Twitter accounts that are on one side of the Bitcoin scaling debate. You get to interview extreme personalities, and the technical discussions can be highly educational. Brian Fabian Crain started the Epicenter podcast four years ago. Podcasting about cryptocurrencies allows a podcaster to report on a wide range of areas: economics, software, philosophy–and the stories within the blockchain world itself. Epicenter is one of my favorite podcasts about cryptocurrencies because Brian is always prepared enough to ask sophisticated questions. In this episode, we talked about ICOs–when does an ICO make sense? It seems that many token economies could function just as well without a token involved. We discussed the scaling approaches of Bitcoin and Ethereum–why are these two blockchains taking very different approaches to their scaling plans? And we talked about Chorus, the company that Brian founded to build infrastructure for proof-of-stake cryptocurrencies. I enjoyed talking to Brian about all these different subjects, and look forward to having him on again in the future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Keybase is a platform for managing public key infrastructure. Keybase’s products simplify the complicated process of associating your identity with a public key. Keybase is the subject of the first half of today’s show. Michael Maxim, an engineer from Keybase gives an overview for how the technology works and what kinds of applications Keybase unlocks. The second half of today’s show is about Clarifai. Clarifai is an AI platform that provides image recognition APIs as a service. Habib Talavati explains how Clarifai’s infrastructure processes requests, and the opportunities for improving the efficiency of that infrastructure. Last month, we had three Software Engineering Daily Meetups: in New York, Boston, and Los Angeles. At each of these Meetups, listeners from the SE Daily community got to meet each other and talk about software–what they are building and what they are excited about. I was happy to be in attendance at each of these, and I am posting the talks given by our presenters. The audio quality is not perfect on these, but there are also no ads. Thanks to Datadog for graciously providing a space for our Meetup, and for being a sponsor of SE Daily. You can sign up for Datadog and get a free t-shirt by going to softwareengineeringdaily.com/datadog. We’d love to have you as part of our community. We will have more Meetups eventually, and you can be notified of these by signing up for our newsletter. Come to SoftwareDaily.com and get involved with the discussion of episodes and software projects. You can also check out our open source projects–the mobile apps, and our website. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google’s central system for managing compute resources is called Borg. On Borg, millions of Linux containers process a wide variety of workloads. When a new application is spun up, Borg provides that application with the resources it needs. Workloads at Google usually fall into one of two distinct categories: long-running application workloads (such as Gmail) and batch workloads (such as a MapReduce job). In the early days of Google, the long-lived workloads were scheduled onto a system called “BabySitter” and the batch workloads were scheduled onto a system called “Global Work Queue.” Borg was the first cluster manager at Google designed to service both long-running and batch workloads from a single system. The second cluster manager at Google was Omega, a project that was created to improve the engineering behind Borg. The innovations of Omega improved efficiency and architecture of Borg. More recently, Kubernetes was created as an open source implementation of the ideas pioneered in Borg and Omega. Google has also built a Kubernetes as a service offering that companies use to run their infrastructure in the same way that Google does. Brian Grant is an engineer at Google who has seen the iteration of all three cluster management systems that have come out of Google. He joins the show to discuss how the workloads at Google have changed over time, and how his perspective on how to build and architect distributed systems has evolved. Full disclosure: Google is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Rajat Monga is a director of engineering at Google where he works on TensorFlow. TensorFlow is a framework for numerical computation developed at Google. The majority of TensorFlow users are building machine learning applications such as image recognition, recommendation systems, and natural language processing–but TensorFlow is actually applicable to a broader range of scientific computation than just machine learning. TensorFlow has APIs for decision trees, support vector machines, and linear algebra libraries. The current focus of the TensorFlow team is usability. There are thousands of engineers building data intensive applications with TensorFlow, but Rajat and the rest of the TensorFlow team would like to see millions more. In today’s show, Rajat and I discussed how TensorFlow is becoming more usable, as well as some of the developments in TensorFlow around edge computing, TensorFlow Hub, and TensorFlow.js, which allows TensorFlow to run in the browser. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

There are hundreds of different databases. There are tens of continuous delivery products. There is an ocean of cloud providers and CRM systems and monitoring platforms and sales prospecting tools. The range of available software products is so diverse that it can be overwhelming to figure out which products to buy. Siftery is a company that was started to index software products and help buyers make decisions. Siftery can build a data set from your web site or from your Google account, assess your software stack, and compare those software products to others on the market. In a previous show with Ayan Barua, we discussed how engineers should explore the question of build vs. buy. In today’s episode, Ayan joins the show to discuss how Siftery has evolved, and the engineering behind Siftery products. A newer Siftery product called Track can ingest banking transactions, QuickBooks records, or other transaction histories and use that information to compile the cost structure of your software company, and we spent the latter part of our conversation discussing why and how they built it. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A message broker is an architectural component that sends messages between different nodes in a distributed system. Message brokers are useful because the sender of a message does not always know who might want to receive that message. Message brokers can be used to implement the “publish/subscribe” pattern, and by centralizing the message workloads within the pub/sub system, it lets system operators scale the performance of the messaging infrastructure by simply scaling that pub/sub system. Derek Collison has worked on messaging infrastructure for 25 years. He started at TIBCO, then spent time at Google and VMWare. When he was at VMWare, he architected the open source platform Cloud Foundry. While working on Cloud Foundry, Derek developed NATS, a messaging control plane. Since that time, Derek has started two companies–Apcera and Synadia Communications. In our conversation, Derek and I discussed the history of message brokers, how NATS compares to Kafka, and his ideas for how NATS could scale in the future to become something much more than a centralized message bus. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Stripe processes payments for thousands of businesses. A single payment could involve 10 different networked services. If a payment fails, engineers need to be able to diagnose what happened. The root cause could lie in any of those services. Distributed tracing is used to find the causes of failures and latency within networked services. In a distributed trace, each period of time associated with a request is recorded as a span. The spans can be connected together because they share a trace ID. The spans of a distributed trace are one element of observability. Others include metrics and logs. Each of these components of observability make their way into services like Lightstep and Datadog. The path traveled by different elements of observability is called the observability pipeline. In an episode last year, Cory Watson explained how observability works at Stripe. In today’s episode, Cory describes how observability is created and aggregated. It’s a useful discussion for anyone working at a company that is figuring out how to instrument their systems for better monitoring. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Bitcoin and Bitcoin Cash are two cryptocurrencies with similar properties. But the supporters of each of these Bitcoin versions have strongly divergent opinions on the direction of the Bitcoin project. At the center of this debate is the subject of block size. Bitcoin’s block size determines how many transactions fit into each block that is mined. A larger block size leads to faster transactions and lower fees, but creates higher demands on mining hardware. A smaller block size leads to a slower on-chain network and higher fees, but allows the full nodes on the network to be run on low performance hardware like Raspberry Pi. Bitcoin Cash has a large block size. Bitcoin Core has a smaller block size. Proponents of the smaller block size argue that Bitcoin’s scaling can be achieved by the off-chain “lightning network” solution. Roger Ver is a Bitcoin entrepreneur and investor. Since he discovered the currency, he has been buying it and evangelizing it. More recently, Roger has become an ardent supporter of Bitcoin Cash–emphasizing that Bitcoin Cash is Bitcoin. In this episode, Roger describes his economic ideology, and explains why Bitcoin is so important to him. We explore how vested interests can shape the narrative and the direction of Bitcoin, and talk about the future of how corporations, governments, and individuals might be using cryptocurrencies. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Most new frontend webapps today use ReactJS. An increasing number of mobile apps are created using the cross-platform components of React Native. GraphQL, Facebook’s open source data-fetching middleware tool is being used by more and more companies, who are finding that it simplifies their development. Facebook’s open source suite of technologies created a new developer ecosystem. There is an increased demand for engineers who know how to build software with React, ReactJS, and GraphQL. This was the reasoning behind Gabe Greenberg starting G2i, a developer marketplace of engineers who write ReactJS, React Native, and GraphQL applications. In this episode, Gabe, Lee Johnson, and Chris Severns from G2i join the show to discuss React and the other Facebook open source technologies–as well as the ecosystem around them. We explored the architecture of a developer marketplace business, and how to scale a consulting company. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning tools are rapidly maturing. TensorFlow gave developers an open source version of Google’s internal machine learning framework. Cloud computing provides a cost effective, accessible way of training models. Edge computing allows for low latency deployments of models. But even if you are a kid with a laptop who has learned all the machine learning algorithms, read all of the deep learning textbooks, and figured out how to use AWS, all of the tooling and education in the world doesn’t change the fact that you still need data to build models. This illustrates why we need data-as-a-service. A kid with a laptop has access to infrastructure-as-a-service, platform-as-a-service, and software-as-a-service. As these tools build on each other, there has been an explosion of high-leverage software products. But the world of data sets remains crude and underdeveloped. Think about some data sets you could take advantage of: the number of emergency room patients that come into a hospital with chest pain; the size of the average coffee mug; the principal component breakdown of sidewalk concrete in San Francisco. SafeGraph is a company that offers data sets as a service. Auren Hoffman is the CEO of SafeGraph, and he joins the show to discuss why he started building SafeGraph and how he thinks about the state of publicly accessible data. Auren was previously on the podcast, and I always enjoy talking to him–this was a great episode and I think you will like it as well. Full disclosure: LiveRamp is a sponsor of Software Engineering Daily, LiveRamp being the company that Auren created prior to SafeGraph. Raj Chetty economic papers Paul Graham “Keep Your Identity Small” Auren Hoffman on Quora Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Let’s Talk Bitcoin is one of the most popular podcasts about cryptocurrencies. Adam B. Levine started it after three other podcasts he started did not get the traction he had hoped for. Adam parlayed the success of Let’s Talk Bitcoin into a network of podcasts–the Let’s Talk Bitcoin Network–which also includes one of my favorite shows, Epicenter. Adam joins me on today’s episode for a discussion of so many topics: the culture around cryptocurrencies, the art of podcasting, blockchain scalability, and ICOs. The conversation around ICOs was particularly exciting–if you have been listening to recent episodes, you have heard interviews with companies who have done ICOs. Some ICO companies are now facing legal ramifications for their token sales–and Adam and I have some disagreement over whether these ICO companies deserve much sympathy. It was a debate that I enjoyed and I hope to have Adam back on the show in the future for more debates. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Monitoring a Kubernetes cluster allows operators to track the resource utilization of the containers within that cluster. In today’s episode, Ilan Rabinovitch joins the show to explore the different options for setting up monitoring, and some common design patterns around Kubernetes logging and metrics gathering. Ilan is the VP of product and community at Datadog. Earlier in his career, Ilan spent much of his time working with Linux and taking part in the Linux community. We discussed the similarities and differences between the evolution of Linux and that of Kubernetes. In previous episodes, we have explored some common open source solutions for monitoring Kubernetes–including Prometheus and the EFK stack. Since Ilan works at Datadog, we explored how hosted solutions compare to self-managed monitoring. We also talked about how to assess different hosted solutions–such as those from a large cloud provider like AWS versus vendors that are specifically focused on monitoring. Full disclosure: Datadog is a sponsor of Software Engineering Daily. 8 Surprising Facts About Real Docker Adoption – Datadog Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Laura Shin is the host of Unchained, a podcast about cryptocurrencies and decentralized technology. For every episode, Laura does significant research and preparation, so the content turns out polished and high quality. Her enthusiasm for the subject of cryptocurrencies comes through in her reporting. Podcasting about cryptocurrencies requires walking a fine line. Cryptocurrencies have a mixture of drama and exciting technology–which are both great for a journalist. But you can’t get too deep in the drama, because the podcast will feel like a tabloid. And you can’t get too deep in the technical weeds, because the listener will fall asleep. Laura joins the show to discuss how she got into reporting on cryptocurrencies, why she got so obsessed with the subject, and her experience as a solo entrepreneurial journalist. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Social networks can make you feel connected to a global society. But those social networks are controlled by a corporate entity. The profit motivations of the corporation are not directly aligned with the experience of the users. Mastodon is an open source, decentralized social network. Eugen Rochko started building Mastodon in response to his dissatisfaction with centralized social networks like Facebook and Twitter. In the Mastodon model, users can run their own nodes, and other users can connect to them. You can follow users whose accounts reside in other nodes. Eugen joins the show to discuss how Mastodon works, and how its thousands of users interact on the platform. We explore the open source community that is building Mastodon, and speculate on the future of social networks. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Go is a language designed to improve systems programming. Go includes abstractions that simplify aspects of low level engineering that are historically difficult—concurrency, resource allocation, and dependency management. In that light, it makes sense that the Kubernetes container orchestration system was written in Go. Erik St. Martin is a cloud developer advocate at Microsoft, where he focuses on Go and Kubernetes. He also hosts the podcast “Go Time,” and has written a book on Go called Go In Action. Recently, Erik helped build the virtual Kubelet project, which allows Kubernetes nodes to be backed by services outside of that cluster. If you want your Kubernetes cluster to leverage abstractions such as serverless functions and standalone container instances, you can use Virtual Kubelet to treat these other abstractions as nodes. Erik also discussed his experience using Kubernetes at Comcast—which was a great case study. Near the end of the show, he also talked about organizing Gophercon, a popular conference for Go programmers—if you are organizing a conference or thinking about organizing one, it will be useful information to you. Full disclosure: Microsoft, where Erik works, is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Tammy Butow has worked at Digital Ocean and Dropbox, where she built out infrastructure and managed engineering teams. At both of these companies, the customer base was at a massive scale. At Dropbox, Tammy worked on the database that holds metadata used by Dropbox users to access their files. To call this metadata system simply a “database” is an understatement–it is actually a multi-tiered system of caches and databases. This metadata is extremely sensitive–this is metadata that tells you where the objects across Dropbox are located–so it has to be highly available. To encourage that reliability, Tammy helped institute chaos engineering–inducing random failures across the Dropbox infrastructure, and making sure that the Dropbox systems could automatically respond to those failures. If you are unfamiliar with the topic, we have covered chaos engineering in two previous episodes of Software Engineering Daily. Tammy now works at Gremlin, a company that does chaos engineering as a service. In this show we talked about her experiences at Dropbox, and how to institute chaos engineering across databases. We also explored how her work at Gremlin–a smaller startup–compares to Dropbox and Digital Ocean, which are larger companies. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software engineers have interacted with operations teams since software was being written. In the 1990s, most operations teams worked with physical infrastructure. They made sure that servers were provisioned correctly and installed with the proper software. When software engineers shipped bad code that took down a software company, the operations teams had to help recover the system—which often meant dealing with the physical servers. During the 90s and early 2000s, these operations engineers were often called “sysadmins,” “database admins” (if they worked on databases), or “infrastructure engineers.” Over the last decade, virtualization has led to many more logical servers across a company. Cloud computing has made infrastructure remote and programmable. The progression of infrastructure led to a change in how operations engineers work. Since infrastructure can be interacted with through code, operations engineers are now writing a lot more code. The “DevOps” movement can be seen through this lens. Operations teams were now writing software—and this meant that software engineers could now work on operations. Both software engineers and operators could create deployment pipelines, monitor application health, and improve the system scalability—all through written code. Site reliability engineering (or SRE) is a newer point along the evolutionary timeline of operations. Web applications can be unstable sometimes, and SRE is focused on making a site work more reliably. This is especially important for a company that makes business applications which other companies rely on. Mike Hiraga is the head of site reliability engineering at Atlassian. Atlassian makes several products that many businesses rely on—such as JIRA, Confluence, HipChat, and Bitbucket. Since the infrastructure is at a massive scale, Mike has a broad set of experiences from his work managing SRE at Atlassian. One particularly interesting topic is Atlassian’s migration to the cloud. Atlassian was started in 2002, before the cloud was widely used, and they have more recently made a push to move applications into the cloud. Full disclosure: Atlassian is a sponsor of Software Engineering Daily—and they are hiring, so if you are looking for a job, check out Atlassian jobs, or send me an email directly and I’m happy to introduce you to the team. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Interplanetary File System (IPFS) is a decentralized global, peer-to-peer file system. IPFS combines ideas from BitTorrent, Git, and Bitcoin, creating a new way to store and access objects across the Internet. When you access an object on almost any website, you are accessing the object via a location address—a URL. The URL tells you where to find the object. If the object is a photo on Facebook that you are linking to, the URL will have an address of somewhere on Facebook. Other objects that we access through URLs include web pages, videos, and JavaScript import packages. URLs seem natural to us. You look up an object based on where that object is being stored. Why would you do anything differently? A downside of location addressing is that if the location disappears, you can no longer access that object. If a government decides to censor a website that I wanted to visit, the government can shut down access to the server where that website sits, and my link will break. This happened in Turkey—where Wikipedia was shut down last year. Objects in IPFS are content addressed—you access an object by giving IPFS a cryptographic hash of the object, and IPFS will find someone on the network who has a copy of that object, and give you access to it. To look up a webpage in an IPFS browser, you put the content address in the address bar. When the HTML for that page is received, that page might have lots of other content-addressed files referred to on the page. Your browser can also grab all of those content-addressed files from the IPFS P2P network. In this episode, David Dias explains how IPFS is designed. David is an engineer at Protocol Labs, the company building out IPFS. This episode is a great companion to our previous show with Juan Benet, the creator of IPFS. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Ethereum community started as a small group of dedicated engineers. It has ballooned to thousands of engineers, entrepreneurs and investors, all of whom have a stake in the direction of Ethereum. Ethereum is an open source project, and the direction of a popular open source project can get complex. Ethereum is figuring out how to govern itself. It’s not clear what the perfect model is, but there are a few historical examples to think about: namely Linux and Bitcoin. Linux is similar to Ethereum in that there is a clear leader—Linux has Linus Torvalds and Ethereum has Vitalik Buterin. Linux is massively successful, and the Linux development team does have a top-down, hierarchical approach. But does a hierarchy with clear leadership make sense for a project like Ethereum, which has decentralization at its core? Bitcoin is headless—Satoshi disappeared in 2010, and there is not an official leader. Bitcoin has succeeded without a well-defined hierarchy–depending on what your definition of success is. Bitcoin development does not move as fast as Ethereum (this is by design)—but there is more widespread trust that the integrity of the system cannot be compromised by its creator. Hudson Jameson is an Ethereum developer and entrepreneur who has been part of the community since the early days. He works on Ethereum governance, which defines how changes to the Ethereum project are proposed, accepted, and implemented. Hudson joins the show today to talk about Ethereum governance, smart contracts, and the DAO hack. We did not discuss on-chain vs. off-chain governance, but I am hoping to cover that in a future episode. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The pubsub pattern allows a developer to create channels, which messages can be written to and read from. Pubsub messaging is useful for multicast messaging–when you want to publish messages from a producer, and have multiple consumers who are subscribed to the publisher receive those messages. Almost any application that reaches a high level of complexity will need a pubsub system of some kind. The pubsub system itself can be complex. A pubsub system needs to scale up and down to handle different numbers of consumers and producers, and different volumes of messages. Back in 2010, the growth of mobile and cloud was leading to many new applications with high throughput, multi-user interactions. Developers were standing up their own instances of open source pubsub message queueing systems like RabbitMQ and ZeroMQ. Once the MQ systems needed to scale, the developer would need to handle the scaling. Stephen Blum started his company PubNub around this time, to create automatically scaling APIs for messaging. Stephen joins the show to discuss the infrastructure choices around building a large scale pubsub service, and how the company has scaled over time. He also talks about the management, product development, and business side of running the company. PubNub has built several additional technologies on top of the core infrastructure that was originally for messaging. Full disclosure: PubNub is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Most technology companies rely on open source software projects. But open source software projects are often maintained by a group of people that is not affiliated with any particular company. When an open source project develops too much technical debt, it can become a tragedy of the commons. Who is responsible for maintaining these open source projects? This is the motivation for open source bounties. Companies and individuals who rely on open source create bounties, which are financial incentives for developers to solve problems within the open source project. Kevin Owocki is the creator of Gitcoin, a platform for open source bounties that is mediated by an Ethereum smart contract. Kevin joins the show to discuss his experience building Gitcoin–as well as some of the problems with the blockchain space, such as rampant ICOs. Gitcoin is NOT a cryptocurrency or token itself–it is a platform for open source software to be built more efficiently. Kevin was an awesome guest and you will enjoy the conversation. Gitcoin is a nice example of a real-world Ethereum use case–it uses Ethereum for escrow: if I post a $25 bounty for someone to fix a bug in my open source project, I will lock up ether in a smart contract. When the bug is fixed, the programmer who fixed it will submit a pull request on Github, and I will release the ether from the smart contract to pay them. We would love for you to fill out our listener survey at softwareengineeringdaily.com/survey. This will help us decide what other content to focus on. Of course–you can also send me an email at any time, jeff@softwaredaily.com. And in the meantime, if you are completely sick of cryptocurrencies, check out our back catalog of episodes at softwaredaily.com, or by downloading our Software Engineering Daily apps, which have all of our episodes including our Greatest Hits, which is a curated set of the most popular shows. The apps will soon have offline downloads and bookmarking. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Zcash is a payment and consensus system that allows users to transfer money to each other with strong guarantees of privacy. Zcash implements the same core features of Bitcoin, with the added functionality of shielded payments. Shielded payments are private, and they are enabled by a novel cryptographic technique called zk-SNARKS: zero knowledge succinct non-interactive argument of knowledge. A zk-SNARK allows for the proof that a certain piece of information is valid without revealing any information other than the validity of that information itself. Before you listen to this episode, it might be useful to go back to our previous episode about Zcash with Nathan Wilcox, in which he gives an overview of the technology. This episode is a deeper dive into how Zcash transactions work, and why zk-SNARKS are important. We would love for you to fill out our listener survey at softwareengineeringdaily.com/survey. This will help us decide what other content to focus on. Of course–you can also send me an email at any time, jeff@softwaredaily.com. And in the meantime, if you are completely sick of cryptocurrencies, check out our back catalog of episodes at softwaredaily.com, or by downloading our apps, which have all of our episodes including our Greatest Hits, which is a curated set of the most popular shows. The apps will soon have offline downloads and bookmarking. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

“The Federal Reserve System is fraudulent. Whatever its stated purpose, its effective purpose is to create a mechanism of deficit spending by politicians, through the insidious invisible taxation of monetary debasement (aka inflation).” These are the words of Erik Voorhees, the CEO of crypto financial exchange ShapeShift. Long before he started ShapeShift, Erik was opposed to some of the core principles of the global financial system, in which he sees the US dollar as a means of control. As an early adopter of Bitcoin, he saw a way to make financial transactions without using fiat currency. Erik’s company ShapeShift allows users to convert different digital currencies between each other. Because it only makes exchanges of currencies and does not hold much currency at any time, ShapeShift is resilient to hacking. In this episode, Erik and I discussed his economic philosophy, and how that informs his affinity for cryptocurrencies. Erik also describes the architecture of ShapeShift, and gives some advice for how to think about building businesses around cryptocurrencies. ShapeShift has had a few near-death experiences, like any startup, and there is a useful story in this episode about how to survive and recover from a serious business setback. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

We sign many different types of contracts throughout our lives. We sign a mortgage to get a loan for a house. When we go to the hospital, we sign a piece of paper that defines how our medical data can be shared between organizations. These pieces of paper represent our opting into an agreement that will be mediated and enforced by computer interactions. We can’t see the code behind those computer interactions, and we can’t verify that it is abiding by the contract we agreed to. Smart contracts allow for programmatic execution of contractual agreements. Code is law, and there is less ambiguity. The most widely used smart contract platform is the Ethereum blockchain–but several large enterprises are creating their own smart contracts. Should all smart contracts be decentralized, or do enterprise consortium blockchains make sense? In this episode, Marley Gray from Microsoft joins the show to discuss enterprise smart contracts–why you would want to use them and how they can be architected. Marley has worked on banking and financial technology for over a decade, and makes some strong arguments for why banks will adopt smart contracts, and the timeline for how that might take place. We would love for you to fill out our listener survey at softwareengineeringdaily.com/survey. This will help us decide what other content to focus on. Of course–you can also send me an email at any time, jeff@softwaredaily.com. And in the meantime, if you are completely sick of cryptocurrencies, check out our back catalog of episodes at softwaredaily.com, or by downloading our apps, which have all of our episodes including our Greatest Hits, which is a curated set of the most popular shows. The apps will soon have offline downloads and bookmarking. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ethereum is a system for running decentralized smart contracts. In the current implementation of Ethereum, every smart contract gets deployed to every full node. Whenever a user wants to call a smart contract, that smart contract gets executed on each full node–across the entire network. The current model for smart contract execution needs to be made more scalable. In today’s episode, Christian Reitwiessner joins the show to describe Plasma–a system for scaling smart contracts. Christian is a developer who has worked extensively on Solidity, the most popular smart contract programming language in Ethereum. For the last month, we have focused on blockchain related topics, and we will soon be shifting to other subjects. Some of the listeners have not enjoyed the blockchain focus, other people have loved it–for everyone listening, we would love for you to fill out our listener survey at softwareengineeringdaily.com/survey. This will help us decide what other content to focus on. Of course–you can also send me an email at any time, jeff@softwaredaily.com. And join our Slack at softwareengineeringdaily.com/slack. And if you are completely sick of cryptocurrencies, check out our back catalog of episodes at softwaredaily.com, or by downloading our apps, which have all of our episodes including our Greatest Hits, which is a curated set of the most popular shows. The apps will soon have offline downloads and bookmarking. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Soumya Basu is a PhD student at Cornell, where he studies distributed systems problems associated with cryptocurrencies. Soumya is advised by Emin Gun Sirer, a Cornell professor who previously appeared on the show to discuss smart contract security. Soumya joins the show today to talk about a variety of issues in the cryptocurrency space. We first explored the degree to which Bitcoin and Ethereum are actually decentralized–which might be less than you think. Because of the centralization of mining pools, much of the transaction processing is also centralized. After talking about decentralization, we got into Soumya’s research focus–cryptocurrency networking and block propagation. Bitcoin transactions are collected into blocks. When a Bitcoin full node solves the cryptographic puzzle associated with a block of transactions, that full node broadcasts the new block to all the other nodes in the network. It is important for that block broadcast to be fast and efficient, so that the other full nodes in the network can be made aware of the new block as soon as possible, and they can start working from the updated chain. The problem of making all nodes in the network aware of a new block is known as “block propagation.” Block propagation can be accelerated through the use of relay nodes. A relay node is a node that is dedicated to communicating these new blocks throughout the blockchain. Soumya is working on a relay node architecture called Falcon–and in this episode we talk all about what Falcon is. If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Consensus protocols are used to allow computers to work together. A consensus protocol lets different servers agree on the state of a system. For decades, these protocols have been used to establish consensus among database nodes, application servers, and other infrastructure that runs within an enterprise. More recently, new consensus protocols have been invented to allow cryptoeconomic systems to agree on the state of a financial system. The first cryptoeconomic consensus protocol to reach wide adoption was Nakamoto consensus–the proof-of-work system used for consensus of Bitcoin. Since then, other systems have been developed, with different tradeoffs in security, speed, and formal verifiability. Ethan Buchman is the CTO at Tendermint, a consensus system for blockchains. In addition to working on Tendermint, Ethan works on Cosmos, a network of blockchains. In this episode, we talk about different consensus systems–for centralized, trustworthy systems as well as for trustless systems like currencies. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The DAO was a system of smart contracts on the Ethereum blockchain that investors put millions of dollars into. Back in May 2016, it was the largest crowdfunding event in history, and we discussed it in detail in a previous episode with Matt Leising. The DAO was hacked due to a security vulnerability, and this event led to a hard fork of Ethereum. The DAO was organized by a company called Slock.it. Slock.it’s original goal was to allow people to connect devices to the Ethereum blockchain. If you could connect smart locks, cars, and electricity systems to the blockchain, it could create decentralized systems for sharing these devices. To raise money, Slock.it created the DAO. Although the initial scope of the DAO was to raise money for Slock.it, over time it expanded in scope to become a decentralized system for venture capital. When the DAO was hacked, the events that followed shook the Ethereum community. The hard fork lowered the financial damage inflicted on the investors–but there was still outrage within the community. How was it possible for an open source crowdfunding project to launch with a security vulnerability? As the Ethereum world looked for someone to blame, they turned to Slock.it. Thus began a very difficult period in the life of Christoph Jentzsch. Christoph is the CEO of Slock.it, and he has been involved in the Ethereum community since the early days. When people think of Slock.it, they might imagine a group of people that move fast and break things. But in fact, Christoph’s early work on Ethereum was around rigorous unit testing of different Ethereum clients. He was obsessed with testing, and consistency between the different Ethereum interfaces. In today’s episode, Christoph and I talk about his early experiences with Ethereum, his reflections on the events of the DAO, and the direction that Slock.it is going today. Since the events of the DAO, the company has refocused its efforts on the original mission–to connect devices to the Ethereum blockchain. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data streams about the weather can be used to predict how soybean futures are going to change in price. Satellite data streams can take pictures of the number of cars on the road, and judge how traffic patterns are changing. Search engines can aggregate data from different queries and determine what people are most interested in. Data streams define how the world is changing over time. Technology companies process these data streams and make decisions based on that stream. The most direct example of this might be financial trading companies, which use all kinds of data streams to predict economic price changes. When Henri Pihkala worked on algorithmic trading systems, he saw how useful these data streams are, and decided to build products around data streaming. Eventually, Henri started working on Streamr, a platform for data streams to be bought and sold on top of the Ethereum network. Streamr is an adaptation of technology that Henri worked on before he started working on the decentralized version. The original technology is a user interface for connecting data streams and building applications on top of them, and he acquired several customers for that platform. Today, the Streamr platform is still mostly centralized, but Henri and his team are working on building out the decentralized infrastructure. Streamr raised an ICO worth ~25 million Euros. Most startups would not raise this amount of money before series B, much less before they have a product with a large user base. In this episode, Henri discusses why they raised so much money, and explains why ICOs are different than equity raises. The investors who participated in the Streamr ICO received the DATAcoin token. Henri also explained why it makes sense for this ecosystem to have its own token. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

To use a web application, you probably open a web browser or a mobile app. To access an Ethereum application, many people use an Ethereum browser. In previous episodes, we explored Metamask and Mist, which are Ethereum browsers for the desktop. In today’s episode, we explore Status, a mobile Ethereum browser. Status founders Jarrad Hope and Oskar Thoren join the show to talk about the engineering of Status. How Status connects to the Ethereum blockchain, what people want from Ethereum applications, and the engineering of the Status app itself. Status is built using React Native–which is working out quite well for them. We also talked some about the mechanics of an ICO. Status has raised $100m in their ICO for the Status Network Token. An ICO differs from raising equity in several ways. Rather than representing a direct stake in the business, a token represents a stake in the ecosystem that is being built. Through their ICO, Status raised much more than a startup at a similar stage in company development would have–and the vesting schedule for the founders is 2 years. After two years, their stake will be liquid. This illustrates another way that the ICO can contrast with a traditional startup equity offering. In a traditional startup, there is not a liquid open market for equity prior to the company going public. This can be good, as it forces the founders to maintain their skin in the game until they have proven the business. But it can also be bad–founders should arguably be able to take some money off of the table even if their business model is not completely worked out. In the interview, Jarrad explained that he anticipates the open source community around Status to be contributing more to the Status app over time, because the community has a stake in the app by purchasing the Status token. I hope this is the case–it would be very cool to see more consumer-facing open source applications. Status is a consumer facing app–and it did make me think that it is strange that there is so much open source software for building applications (think about React Native, Kubernetes, Kafka), but there are fewer consumer-facing open source apps. There’s not an open source Uber, an open source Facebook, or an open source Google. Why is that? Maybe that’s because we are still in the days where someone has to pay for the backend compute layer. In other words–open source code is free to host, but running the actual application infrastructure still requires the owner to pay–so it makes sense that consumer applications are still developed and maintained by central actors. With Ethereum, maybe that will change and we will see more consumer facing, open source, decentralized applications. That is certainly the world that Status.im is hoping for. Speaking of consumer facing open source applications: check out our Software Engineering Daily apps on the iOS or Android app store. All 700 episodes of Software Engineering Daily are in the app–we’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Anthony Diiorio was involved with Ethereum since the earliest days. He was one of the first people to see the Ethereum ideas presented by Vitalik Buterin, and he invested deeply in Ethereum–both financially and by helping to establish the early Ethereum community. Anthony started Decentral in 2014, which is a hub for his projects in the cryptocurrency space, the most impactful project being Jaxx. Jaxx is a blockchain wallet that can hold multiple different cryptocurrencies. It works by connecting a small client-side application to remote full nodes. The user interface is simple, and Jaxx maintains the full node instances that the small client-side application connects to. We discuss the architecture of Jaxx in more detail during this episode. We also talk about Anthony’s background–which includes a wide range of businesses: marketing, patio door manufacturing, real estate, and eventually blockchains. Anthony had a wealth of information to provide around entrepreneurship–both inside and outside of the blockchain space. If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A financial exchange is an operationally intensive business. You have customers making a high volume of transactions, your service has to be low latency and highly available, and you are dealing with a lot of money. A cryptocurrency exchange has all of the complexity of a typical financial exchange–and then some additional complexity. Shapeshift is a cryptocurrency exchange that allows users to buy and sell digital assets–Bitcoin, Ethereum, Litecoin, and lots of other currencies. Shapeshift also has a set of tools and APIs that allow developers to build higher level applications that transact in cryptocurrencies. Shapeshift’s CEO is an early cryptocurrency entrepreneur named Erik Voorhees, who will appear on the show in the near future. Today’s guest Jon is the COO of Shapeshift–he handles the operations of the company. He prefers not to use his last name, because Shapeshift is particularly sensitive to social engineering attacks. We’ll get into why that is in the episode–and explore lots of other topics too. How to scale a cryptocurrency exchange, the products Shapeshift offers, and some of the near-death experiences that Shapeshift has had. After all–it is a startup, and every startup has moments where it seems like the company will die. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cryptocurrency speculation has pulled in a large population of people who do not know what they are investing in. If you hear about an investment of $1000 turning into $1M, it’s tempting to get sucked in yourself. For most of these everyday people, the game is completely rigged. A large percentage of market activity is driven by “pump and dumps.” A pump and dump is a conspiracy to trick investors into buying a currency. An insider group commits the pump and dump. This is accomplished by purchasing the currency ahead of time, then promoting it via Twitter, Telegram, and reddit. The outsiders fall victim to the promotion of the currency, and buy it after the fast run-up in value. The currency then crashes, and the outsiders are left “holding the bag.” Pump and dumps are not a new phenomenon—they have happened with worthless penny stocks. One thing that is new is the ease with which new cryptocurrencies are being created. Launching an ICO is easy. Marketing it is cheap. Pumping and dumping has never been more accessible. And buying them is quite easy as well. This has led to a perfect storm of naive investment capital. Bruno Skvorc is the CEO and owner of Bitfalls, a site with blog posts, news, and information about cryptocurrencies. He wrote a post called “The Anatomy of a Pump and Dump Group,” which details how cryptocurrency pump and dumps have been used to swindle investors out of millions of dollars. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the finance industry, many people have a computer on their desk called a Bloomberg terminal. A Bloomberg terminal contains news, stock prices, communication tools, and other features that make it worth a high subscription price. And people in finance can afford to pay that high subscription because their decisions can cause a gain or loss of thousands of dollars. Cryptocurrency investors have a similar set of informational problems as traditional financiers. There is a flood of information. Financial quotes are inconsistent across different exchanges. Opinions from Twitter and reddit can be tremendously useful—if they are captured and leveraged correctly. Santiment is a platform that is working to build a Bloomberg terminal for the cryptocurrency investor. Santiment has raised 45k Ether in their ICO last July, which was originally an amount equal to ~$11m. Valentin Mihov is the CTO of Santiment, and he joins the show to explain what Santiment’s product does, and how the token holders will ultimately derive value from Santiment’s ecosystem. If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Most applications today run on a cloud provider like AWS. They are built with a framework like Ruby on Rails. They use a set of APIs like Stripe and Twilio for middleware services. This is the era of “web 2.0.” With decentralized systems, we are starting to get a feel for what “web 3.0” might feel like. The futuristic idea of “web 3.0” works off of the following idea: instead of using a centralized service owned by a single company, you might purchase your computation and storage from a network of nodes. The nodes will be running peer-to-peer software that competes on price. Fabian Vogelsteller works on Web3.js, a JavaScript library for interfacing with the Ethereum blockchain. He also works on Mist, a browser for Ethereum. Fabian joins the show to discuss the difference between decentralized apps and centralized apps—and to explain why we need to build a bridge between those two worlds. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Decentralized applications can be built on the Ethereum blockchain. Just as the Bitcoin blockchain is a distributed, append-only ledger of financial transaction history, Ethereum is a distributed, append-only ledger of computational transaction history. New kinds of applications can be built on the Ethereum blockchain—and just like every new technology, we need an interface to bridge that new technology and our existing technology. We can use a pure Ethereum browser like Mist—or we can use a Chrome extension like Metamask to turn our normal browser into an Ethereum interface. Dan Finlay is the lead developer of Metamask. In today’s episode, we explore why you would want to interface with decentralized applications, and the different ways of doing so. A few examples we explore—simple transactions like transferring Ether from one person to another; or transacting with a smart contract. My personal anecdote: I recently used Metamask for the first time to fund a GitCoin issue. GitCoin is a way to put up financial rewards for people solving open source issues. I locked up $42 in an Ethereum smart contract, and it became the bounty of that issue. The issue was solved, and I released the $42 from the smart contract to be sent to the developer who solved it. In this example, Ethereum served as a simple escrow service. To send my Ether, I used the Metamask plugin on my Chrome browser. If you are a little confused—don’t worry. We explain it all in this episode. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Decentralized applications might someday offer alternatives to modern monopolies. Uber, Airbnb, Facebook, Amazon—all of these services could be recreated on a decentralized stack of technologies like Ethereum, IPFS, and Golem. Fully decentralized services could be more transparent, cheaper, and more efficient. But let’s be realistic. Today, even the simplest applications of fully decentralized blockchains don’t work as well as we need them to. Cryptokitties offered a glimpse into how a simple viral application can limit the throughput of Ethereum. And don’t forget that these technologies are in some ways still subject to centralization in their current form. Miners form the decentralized consensus layer—and that mining activity is physically centralized in large server farms. The decentralized future is possible. In order to get there, we need to make progress on the low-level tools that such a world will be built upon. This is the realization that today’s guest Karl Floersh had. Karl is a researcher for the Ethereum Foundation. He was initially excited about the prospect of decentralized apps—such as a decentralized Uber. But as he looked more closely at the space, he realized how early we are, and how much work there is to be done on foundational technologies. Proof of Stake is the central topic of discussion in today’s conversation with Karl. Proof of Stake is a consensus mechanism that is an alternative to Proof of Work. In Proof of Work, miners race to validate blocks of transactions. This results in duplicated effort and perhaps wasted energy. In Proof of Stake, validators are chosen to approve transactions. These validators lock up an amount of currency that they are willing to “stake.” If a validator acts badly, the validator will lose their entire stake. This mechanism could be more efficient—and we will explain why that is in this episode. If Proof of Stake works, it could lead to a faster, truly decentralized Ethereum blockchain. That’s a remarkable potential outcome. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

For a decade, Bitcoin’s proof-of-work system has run without disruption. In a proof-of-work scheme, Bitcoin miners compete to solve a cryptographic puzzle associated with a block of transactions. Every ten minutes, all the Bitcoin miner nodes race to be the first to solve a block of transactions. Only one miner wins each block, meaning the other nodes’ time was ultimately wasted. There is also a massive expense of electricity. Bitcoin is a system with low transaction throughput—about 7 transactions per second. Computer scientists have wondered—is there an alternative way of doing consensus? What if we took all the wasted compute power from proof of work, and allocated it in a way that makes transactions get processed faster? But Bitcoin’s governance tends to be extremely conservative. A change to the consensus mechanism probably won’t happen any time soon in Bitcoin. Ethereum’s consensus mechanism is modeled after that of Bitcoin—proof-of-work mining. But Ethereum’s governance ethos is quite different. Ethereum is in the process of planning and implementing proof of stake, an alternative consensus mechanism in which trusted validators are chosen to validate blocks of transactions. Subhan Nadeem is a student at the University of Waterloo where he studies computer science and business. He is the author of several popular articles on Medium that explain blockchain concepts. He joins the show to talk about crypto from the point of a student—and gives us a great walk through of different consensus mechanisms. To find all of our old episodes about cryptocurrencies, check out our apps in the iOS or Android app store. They have all 700 of our episodes, with recommendations, related links, discussions and more. And it’s all open source–if you are looking for an open source project to contribute to, come check us out at github.com/softwareengineeringdaily. We welcome all kinds of contributors–new developers and experts. Engineers and non-technical people. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Humans organize into groups. There are lots of group types: religions, corporations, national governments, state governments, citizenries, clubs, musical bands. Every group has governance. Governance defines the rules, and the ways that rules change. The United States requires citizens to pay taxes. A corporation requires you to show up to work, but they have to pay you a salary. Most groups today are managed by people. If you break a law, you have to go to court and sit in front of a judge and jury, who decide how you will be punished. If you work at a corporation, and you have a problem with your manager, you go to HR to arbitrate it. These organizations are centralized. There is a governing body who sets the rules. If there is an ambiguity, the person who happens to be in power gets to decide how the ambiguity is resolved. Power is centralized in that governing body. These organizations are run by people. The governance of these organizations is enforced only to the extent that the human government carries out its duties. A decentralized autonomous organization is a group that can run with neither centralized nor human intervention. It is decentralized and autonomous. It is a DAO. Aragon is a platform for running and managing decentralized autonomous organizations. Luis Cuende is the founder of Aragon, and joins the show to explain what a DAO is and why people want to create them. We also talk about the engineering of Aragon and the structure of its ICO—which raised $25m via token sale. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Smart contracts are programs that run on the Ethereum blockchain. A smart contract developer pays Ether to deploy the contract. When a contract is deployed, every full node on the Ethereum blockchain has a copy of the contract code in that node’s address space. Every full node needs to hold a copy of every smart contract. This allows every full node to process every call to any smart contract. If you want to call a smart contract, that contract will execute on every full node. When you call a smart contract, you are initiating a transaction. Like Bitcoin transactions, these Ethereum transactions get batched into blocks. Ethereum full nodes compete to solve the cryptographic puzzle associated with a block. But instead of mere financial transactions, these are computational transactions. Raine Revere is a smart contract engineer and cofounder at Maiden and she joins the show to describe smart contract creation and deployment. It’s a great introduction to some Ethereum fundamentals. To find all of our old episodes about cryptocurrencies, check out our apps in the iOS or Android app store. They have all 700 of our episodes, with recommendations, related links, discussions and more. And it’s all open source–if you are looking for an open source project to contribute to, come check us out at github.com/softwareengineeringdaily. We welcome all kinds of contributors–new developers and experts. Engineers and non-technical people. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Joseph Bonneau is co-author of Bitcoin and Cryptocurrency Technologies, a popular textbook. At NYU, he works as an assistant professor exploring cryptography and security. His YouTube lessons teaching Bitcoin have hundreds of thousands of views. His material offers clear explanations of how Bitcoin works. Since Joseph has a clear understanding of the objective facts around Bitcoin, he is the perfect person to ask about the more subjective topics: the common misunderstandings of Bitcoin; the governance tradeoffs between Ethereum and Bitcoin; proof of work vs. proof of stake. Joseph believes that the early mainstream cryptocurrency solutions will be largely centralized—and that we are likely to move beyond Bitcoin to more efficient currencies. I enjoyed hearing his reasons behind this perspective. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Farms have lots of data. A corn farmer needs to monitor the chemical composition of soil. A soybean farmer needs to track crop yield. A chicken farmer needs to count the number of eggs produced. If this data is captured, it can be acted upon—for example, a dry farm can automatically turn up its irrigation system. Or the data can simply be studied. If you work in a pure software business, you might take for granted how easy it is to track your metrics. On the farm, you need to use sensors and drones to gather your data. Mike Prorock is the CTO of Mesur.io, a company that makes sensors and software infrastructure for agriculture. He joins the show to describe the use cases for agriculture technology, and the architecture behind it. Today’s episode is a great complement to our recent episodes on streaming data. Mesur.io offers a case study in how streaming systems can be put into practice. Mike will also be speaking at the upcoming Strata Data Conference in San Jose. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Dogecoin was started in 2013 as a joke. Jackson Palmer forked Bitcoin and created his cryptocurrency as a play off the “doge” meme. The currency became popular as a means of reddit users “tipping” each other. If I made a comment on reddit that you liked, you might send me some Dogecoin. This use case allowed people to share the idea of Dogecoin virally, and Dogecoin became valuable, even though the currency did not have any technical properties that made it significantly different than Bitcoin. As Dogecoin was becoming popular, an experienced Internet scam artist took notice and started a Dogecoin exchange called Moolah. Moolah was used to steal money from its customers and investors, and the CEO was arrested. Jackson Palmer was not involved in this scheme, but it soured his feelings about Dogecoin and the entire Bitcoin space. His coin, which had been created as a joke, had been repurposed as a weapon to steal money. Jackson left the Dogecoin community in 2015 to focus on other things. But as Bitcoin entered the mainstream conversation, Jackson has been pulled back into the world of cryptocurrency. Jackson’s YouTube channel has over 20,000 subscribers, who tune into learn about consensus protocols, new tokens, and cryptocurrency news. In today’s episode, Jackson and I discuss his experiences with Dogecoin, and how that compares with the scams around low-quality ICOs that are pulling in retail investors today. We also discuss more positive things–such as proof-of-stake and newer consensus protocols. If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Also–Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

There are two factors that limit the rate at which transactions are accepted into the Bitcoin blockchain: block time and block size. Block time defines how often a new block is appended onto the blockchain. Block size defines how many transactions fit into a new block. As of March 2018, the current block time and block size allow for about 7 transactions per second to be accepted into the Bitcoin blockchain. In today’s episode, we discuss the technical limitations of the Bitcoin blockchain, and some potential solutions to scalability: SegWit and lightning network. Today’s guest is Peter Ullrich, the host of Explain Blockchain. Explain Blockchain is a podcast I have found tremendously useful as I have started to learn about blockchains. He provides thorough, technical explanations of complicated topics, and I recommend subscribing to his show, and listening to the episodes multiple times, because there is a lot of content condensed into a short amount of time. Over the next month, we will be exploring a variety of blockchain-based technologies. Some interviews will be high-level conversations that assume only a familiarity with cryptocurrencies. Some of them will be deeply technical, and assume a strong understanding of Bitcoin and Ethereum. And some episodes, like today’s episode, will be aimed at the developer who is in the process of “going down the rabbit hole.” If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Bitcoin is an immutable, append-only blockchain ledger that reaches consensus through proof-of-work. The contents of the ledger are financial transactions–people sending and receiving Bitcoin currency to each other. Since Bitcoin, there have been other cryptocurrencies that have similar properties–like Ethereum and the IPFS/Filecoin system. Similar to Bitcoin, they use a decentralized, proof of work based system with a currency reward system–but the ledger being maintained is not purely financial. A currency is a necessary component to maintaining the blockchain’s validity. Over the next month, we will be exploring a variety of blockchain-based technologies. Some interviews will be high-level conversations that assume only a familiarity with cryptocurrencies. Some of them will be deeply technical, and assume a strong understanding of Bitcoin and Ethereum. And some episodes, like today’s episode, will be aimed at the developer who is in the process of “going down the rabbit hole.” If you are finding yourself reading about Bitcoin and Ethereum a few hours every day, but you are still struggling to grasp the basics, this episode is for you. It is meant to be a complement to other introductory resources, such as “Mastering Bitcoin,” by Andreas Antonopoulos. Cryptocurrency systems are revolutionary–they will unlock completely new applications in the very near future. It’s If you are trying to understand these decentralized currencies and applications, the best place to start is with Bitcoin. That’s why I was happy to have Daniel Van Flymen back on the show. Daniel was previously on for one of our most popular episodes–”Blockchain Building,” in which he talked about how useful it can be to build a blockchain based system for practice. Today, Daniel discusses the basics of Bitcoin transactions. What happens when you send money? How are transactions represented on the blockchain? How do full nodes and light clients interact with each other? These are difficult topics to discuss purely over audio, so this episode is best listened to as a companion resource for someone who is studying cryptocurrencies. If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The easiest way to train a computer to recognize a picture of cat is to show the computer a million labeled images of cats. The easiest way to train a computer to recognize a stop sign is to show the computer a million labeled stop signs. Supervised machine learning systems require labeled data. Today, most of that labeling needs to be done by humans. When a large tech company decides to “build a machine learning model,” that often requires a massive amount of effort to get labeled data. Hundreds of thousands of knowledge workers around the world earn their income from labeling tasks. An example task might be “label all of the pedestrians in this intersection.” You receive a picture of a crowded intersection, and your task is to circle all the pedestrians. You have now created a piece of labeled data. Scale API is a company that turns API requests into human tasks. Their most recent release is an API for labeling data that has been generated from sensors. As self-driving cars emerge onto our streets, the sensors on these cars generate LIDAR, radar, and camera data. The cars will interpret that data in real time using their machine learning models, and then they will send that data to the cloud so that the data can be processed offline to improve the machine learning models of every car on the road. The first step in that processing pipeline is the labeling–which is the focus of today’s conversation. Alexandr Wang is the CEO of Scale, and he joins the show to discuss self-driving cars, labeling, and the company he co-founded. A few notes before we get started. We just launched the Software Daily job board. To check it out, go to softwaredaily.com/jobs. You can post jobs, you can apply for jobs, and it’s all free. If you are looking to hire, or looking for a job, I recommend checking it out. And if you are looking for an internship, you can use the job board to apply for an internship at Software Engineering Daily. Also, Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Spark is a system for processing large data sets in parallel. The core abstraction of Spark is the resilient distributed dataset (RDD), a working set of data that sits in memory for fast, iterative processing. Matei Zaharia created Spark with two goals: to provide a composable, high-level set of APIs for performing distributed processing; and to provide a unified engine for running complete apps. High-level APIs like SparkSQL and MLlib enable developers to build ambitious applications quickly. A developer using SparkSQL can work interactively with a huge dataset, which is a significant improvement on batch Hive jobs running on Hadoop. A developer training a machine learning model can put the model through multiple steps in the training process without checkpointing the data to disk. The second goal of Spark–a “unified engine for running complete apps”–was the focus of my conversation with today’s guest Matei Zaharia. Matei is the CTO of Databricks, a company that was started to implement his vision for Spark and to build highly usable products on top of the technology. Databricks Delta is a project that combines a data warehouse, data lake, and streaming system–all sitting on top of Amazon S3 and using Spark for processing. In our recent episodes about streaming, we explored some common streaming architectures. A large volume of data comes into the system and is stored in Apache Kafka. Backend microservices and distributed streaming frameworks read that data and store it in databases and data lakes. A data warehouse allows for fast access to the large volumes of data–so that machine learning systems and business analysts can work with data sets interactively. The goal of Databricks Delta is to condense the streaming system, the data lake, and the data warehouse into a single system that is easy to use. If you listened to the previous episodes, you will have an idea for the level of complexity that is involved in managing these different systems. For some companies, it makes complete sense to manage a Kafka cluster, a Spark cluster, a set of S3 buckets, and a data warehouse like Amazon Redshift. But we probably don’t want all of that management to the lowest barrier to entry. Delta will hopefully reduce that barrier to entry and make it easier for enterprises to set up large systems for processing data. A few notes before we get started. We just launched the Software Daily job board. To check it out, go to softwaredaily.com/jobs. You can post jobs, you can apply for jobs, and it’s all free. If you are looking to hire, or looking for a job, I recommend checking it out. And if you are looking for an internship, you can use the job board to apply for an internship at Software Engineering Daily. Also, Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Steve Herrod led engineering at VMWare as the company scaled from 30 engineers to 3,000 engineers. After 11 years, he left to become a managing director for General Catalyst, a venture capital firm. Since he has both operating experience and a wide view of the technology landscape as an investor, he is well-equipped to discuss a topic that we have been covering on Software Engineering Daily: the integration of cloud and edge computing. Today, we think of the cloud as a network of large data centers operated by big players like Google, Amazon, and Microsoft. The cloud is where most of the computation across the world takes place. My smartphone and laptop are “edge” devices. They are lightweight computers that don’t perform much complex processing. I would not be able to run a large production database or a 3 terabyte MapReduce job on my laptop. The current division of labor makes sense in this world of smart clouds and low-power, low-bandwidth devices. But the devices are getting cheaper, smarter, and more proliferate. Cars, drones, security cameras, sensors and other devices can serve as points of computation that are geographically between the edge devices and the cloud. With more devices between you and the cloud, there is an opportunity to put computation on those devices. Everyone knows that cloud and edge computing will become intermingled in the coming years. But predicting just how it will play out is nearly impossible. And as an investor, if you bet on something too early, you get the same result as someone who was wrong altogether. A good analogy for the “cloud and edge” space of investments might be the “smart home.” Everyone knows the smart home is coming eventually, but it’s very hard to tell how long it will be before smart home systems are in widespread use–so it is an open question how to invest in the space. Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to internships@softwareengineeringdaily.com. We’d love to hear from you. If you haven’t seen what we are building, check out softwaredaily.com, or download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

On Software Engineering Daily, we have been covering the “serverless” movement in detail. For people who don’t use serverless functions, it seems like a niche. Serverless functions are stateless, auto-scaling, event driven blobs of code. You might say “serverless sounds kind of cool, but why don’t I just use a server? It’s a paradigm I’m used to.” Serverless is exciting not because of what it adds but because of what it subtracts. The potential of serverless technology is to someday not have to worry about scalability at all. Today, we take for granted that if you start a new company, you are building it on cloud infrastructure. The problem of maintaining server hardware disappeared for 99% of startups, which unlocked a wealth of innovation. The cloud also simplified scalability for most startups–but there are still plenty of companies that struggle to scale. Significant mental energy is spent on the following questions: How many database replicas do I need? How do I configure my load balancer? How many nodes should I put in my Kafka cluster? Serverless functions are important because they are an auto-scaling component that sits at a low level. This makes it easy to build auto scaling systems on top of them. Auto scaling databases, queueing systems, machine learning tools, and user applications. And since the problem is being solved at such a low level, the pricing competitions will also take place at the low level, meaning that systems built on serverless functions will probably see steep declines in costs in the coming years. Serverless compute could eventually become free or nearly free, with the major cloud providers using it as a loss leader to onboard developers to higher level services. All of this makes for an exciting topic of discussion, that we will be repeatedly covering. Today’s show is with Eduardo Laureano, the principal program manager of Azure Functions. It was a fantastic conversation and we covered applications of serverless, improvements to the “cold start problem,” and how the Azure Functions platform is built and operated. Full disclosure: Microsoft is a sponsor of Software Engineering Daily. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to internships@softwareengineeringdaily.com. We’d love to hear from you. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Earlier this year we did several shows about Cloud Foundry, followed by several shows about Kubernetes. Both of these projects allow you to build scalable, multi-node applications–but they serve different types of users. Cloud Foundry encompasses a larger scope of the application experience than Kubernetes. Kubernetes is lower level, and is actually being used within newer versions of Cloud Foundry to give Cloud Foundry users access to the Kubernetes abstractions. Recording those shows gave me a wide understanding of how infrastructure is managed and how it has evolved. Today’s episode provides more context on Cloud Foundry–how the project got started, how people use it, and where Cloud Foundry is going. Today’s guest Mike Dalessio is a VP of engineering on Pivotal Cloud Foundry, and we had a great time talking about his work. Engineering leadership is a fine art, and conversations with engineering leaders are consistently interesting–this was no exception. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kafka is at the center of modern streaming systems. Kafka serves as a database, a pubsub system, a buffer, and a data recovery tool. It’s an extremely flexible tool, and that flexibility has led to its use as a platform for a wide variety of data intensive applications. Today’s guest is Gwen Shapira, a product manager at Confluent. Confluent is a company that was started by the creators of Apache Kafka–Jay Kreps, Neha Narkhede, and Jun Rao, who have all been on the show in previous episodes. In those shows, we discussed the inner workings of Kafka. This episode is more about practical use cases and design patterns. Gwen explores a few use cases. First, reconciling data between different servers. A massive, international, multi-user game like World of Warcraft needs to keep its users in sync despite the fact that those users are pinging different server locations. Kafka can help reconcile data between the multiple servers. This discussion reminded me of the awesome show we did with Yan Cui about scalable multiplayer games. Other examples we discussed include log management, data enrichment, and large scale analytics. It was a great conversation and I think you will enjoy it as well. Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to internships@softwareengineeringdaily.com. We’d love to hear from you. If you haven’t seen what we are building, check out softwaredaily.com, or download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Streaming architecture defines how large volumes of data make their way through an organization. Data is created at a user’s smartphone, or on a sensor inside of a conveyor belt at a factory. That data is sent to a set of backend services that aggregate the data, organizing it and making it available to business analysts, application developers, and machine learning algorithms. The velocity at which data is created has led to widespread use of the “stream” abstraction–a never ending, append-only array of data. To deal with this volume, streams need to be buffered, batched, cached, mapreduced, machine learned, and munged until they are in a state where they can provide value to the end user. There are numerous ways that data can travel this path, and in today’s episode we discuss the streaming systems, data lakes, and data warehouses that can be used to build an architecture that makes use of streaming data. Ted Dunning is a chief application architect at MapR, and he joins the show to discuss the patterns that engineering teams are using to build modern streaming architectures. Full disclosure: MapR is a sponsor of Software Engineering Daily. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to internships@softwareengineeringdaily.com. We’d love to hear from you. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When you go to a website where a video is playing, and your video lags, how does the website know that you are having a bad experience? Problems with video are often not complete failures–maybe part of the video loads, and plays just fine, and then the rest of the video is buffering. You have probably experienced sitting in front of a video, waiting for it to load as the loading wheel mysteriously spins. Since problems with video are often not complete failures, troubleshooting a problem with a user’s video playback is not as straightforward as just logging whenever a crash occurs. You need to continuously monitor the video playback on every client device and aggregate it in a centralized system for analysis. The centralized logging system will allow you to separate problems with a specific user from problems with the video service itself. A single user could have bad wifi, or have 50 tabs open with different videos. To identify problems that are caused by the video player rather than the user, you need to capture the playback from every video and every user. Scott Kidder works at Mux, where he builds a streaming analytics system for video monitoring. In this episode, Scott explains how events make it from a video player onto the backend analytics system running on Kinesis and Apache Flink. Events from the browser are constantly added to Kinesis (which is much like Kafka). Apache Flink reads those events off of Kinesis and map reduces them to discover anomalies. For example, if 100 users watch a 20 minute cat video, and the video stops playing at minute 12 for all 100 users, there is probably some data corruption in that video. You would only be able to discover that by assessing all users. Scott and I discussed the streaming infrastructure that he works on at Mux, as well as other streaming systems like Spark, Apache Beam, and Kafka. This episode is the first in a short series about streaming data infrastructure. I wanted to do some shows in preparation for Strata Data conference in March in San Jose, which I will be attending thanks to a complimentary ticket from O’Reilly. O’Reilly has been kind enough to give me free tickets since Software Engineering Daily started and did not have the money to attend any conferences. If you want to attend Strata, you can use promo code PCSED to get 20% off. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Tugdual Grall is an engineer with MapR. In today’s episode, we explore use cases and architectural patterns for streaming analytics. Full disclosure: MapR is a sponsor of Software Engineering Daily. In past shows, we have covered data engineering in detail–we’ve looked at Uber’s streaming architecture, talked to Matei Zaharia about the basics of Apache Spark, and explored the history of Hadoop. To find all of our episodes about data engineering , download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Pinterest is a visual feed of ideas, products, clothing, and recipes. Millions of users browse Pinterest to find images and text that are tailored to their interests. Like most companies, Pinterest started with a large monolithic application that served all requests. As Pinterest’s engineering resources expanded, some of the architecture was broken up into microservices and Dockerized, which make the system easier to reason about. To serve users with better feeds, Pinterest built a machine learning pipeline using Kafka, Spark, and Presto. User events are generated from the frontend, logged onto Kafka, and aggregated to build machine learning models. These models are deployed into Docker containers much like the production microservices. Kinnary Jangla is a senior software engineer at Pinterest, and she joins the show to talk about her experiences at the company–breaking up the monolith, architecting a machine learning pipeline, and deploying those models into production. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Over 12 years of engineering, Box has developed a complex architecture of services. Whenever a user uploads a file to Box, that upload might cause 5 or 6 different services to react to the event. Each of these services is managed by a set of servers, and managing all of these different servers is a challenge. Sam Ghods is the cofounder and services architect of Box. In 2014, Sam was surveying the landscape of different resource managers, deciding which tool should be the underlying scheduler for deploying services at Box. He chose Kubernetes because it was based on Google’s internal Borg scheduling system. For years, engineering teams at companies like Facebook and Twitter had built internal scheduling systems modeled after Borg. When Kubernetes arrived, it provided an out-of-the-box tool for managing infrastructure like Google would. In today’s episode, Sam describes how Box began its migration to Kubernetes, and what the company has learned along the way. It’s a great case study for people who are looking at migrating their own systems to Kubernetes. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When Box started in 2006, the small engineering team had a lot to learn. Box was one of the earliest cloud storage companies, with a product that allowed companies to securely upload files to remote storage. This was two years before Amazon Web Services introduced on-demand infrastructure, so the Box team managed their own servers, which they learned how to do as they went along. In the early days, the backup strategy was not so sophisticated. The founders did not know how to properly set up hardware in a colocated data center. The frontend interface was not the most beautiful product. But the product was so useful that eventually it started to catch on. Box’s distributed file system became the backbone of many enterprises. Employees began to use it to interact with and share data across organizations. The increase in usage raised the stakes for Box’s small engineering team. If Box’s service went down, it could cripple an enterprise’s productivity, which meant that Box needed to hire experienced engineers to build resilient systems with higher availability. And to accommodate the growth in usage, Box needed to predict how much hardware to purchase, and how much space in a data center to rent–a process known as capacity planning. As Box went from 3 engineers to 300, the different areas of the company went from being managed by individuals, to teams, to entire departments with VPs and C-level executives. Jeff Quiesser is an SVP at Box, and one of the earliest employees. He joins the show today to describe how Box changed as the company scaled. We covered engineering, management, operations, and culture. In previous shows, we have explored the stories of companies like Slack, Digital Ocean, Giphy, Uber, Tinder, and Spotify. It’s always fun to hear how a company works–from engineering the first product to enterprises with millions of users. To find all of our episodes about how companies are built, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Employees often find themselves needing to do work outside of the office. Depending on the sensitivity of your task, accessing internal systems from a remote location may or may not be OK. If you are using a corporate application that shows the menu of your company’s cafe on your smartphone, your workload is less sensitive. If you are accessing the proprietary codebase of your company’s search engine, your workload is more sensitive. As Google grew in headcount, the different cases of employees logging in from different places grew as well. Google developed a fine-grained, adaptive security model called BeyondCorp to allow for a wide variety of use cases. Whether you are an engineer logging in from a Starbucks or a human resources employee logging in from your desk, the BeyondCorp system uses the same access proxy to determine your permissions. The BeyondCorp architecture is also built around the assumption of a zero-trust network. A zero-trust network is a modern enterprise security architecture where internal servers do not trust each other. Zero-trust networks assume that the network has already been breached. If you are writing an internal application, your default assumption should be to distrust an incoming request from someone else on the network. The zero-trust model is in contrast to an outdated model of enterprise security–that of the hard outer defense of a firewall, that purports to prevent attackers from ever making their way into the vulnerable inside of a network. The firewall model assumes that all of these servers within the firewall can trust each other. Several papers have come out of Google discussing the BeyondCorp security model. These papers describe the network architecture, and the security philosophies of BeyondCorp. Since the release of these papers, an ecosystem of security providers has sprung up to provide implementation services for companies that want BeyondCorp security in their enterprise. Google has also productized its BeyondCorp system with an identity-aware proxy that is tied into their Google Cloud product. Max Saltonstall is the technical director of information technology in the office of the CTO at Google, where he has helped to facilitate the widespread adoption of the BeyondCorp program. In this episode, we talk about enterprise security–from remote employee access to zero-trust networks. We also talk about implementing the BeyondCorp model–why enterprises should consider it, and how to do it. We have done lots of past shows about security–from car hacking to smart contract vulnerabilities to discussions with luminaries like Bruce Schneier and Peter Warren Singer. To find all of our episodes about security, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Applications need to be ready to scale in response to high-load events. With mobile applications, this can be even more important. People rely on mobile applications such as banking, ride sharing, and GPS. During Black Friday, a popular ecommerce application could be bombarded by user requests–you might not be able to complete a request to buy an item at the Black Friday discount. If you attend the Superbowl, and then try to catch an Uber after leaving, all the other people around you might be summoning a car at the same time, and the system might not scale. In order to prepare infrastructure for high volume, mobile development teams often create end-to-end load tests. After recording incoming mobile traffic, that mobile traffic can be replicated and replayed, to measure a backend’s response to the mobile workload. Paulo Costa and Rodrigo Coutinho are engineers at OutSystems, a company that makes a platform for building low-code mobile applications. In this episode, Paulo and Rodrigo discuss the process of performing end-to-end scalability testing for mobile applications backed by cloud infrastructure. We talked about the high level process of architecting the load test, and explored the tools used to implement it. Full disclosure: OutSystems is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Matt Leising is a journalist at Bloomberg who has covered financial markets for 15 years. Today, his reporting has been completely engulfed by cryptocurrencies. There are so many dramatic stories, it’s hard to pick what to focus on. Today, we discuss two topics he has covered recently: Ripple and Tether. Ripple is a company that makes enterprise blockchain solutions for global payments. That sounds like the future, and it is no surprise that people would want to buy into Ripple if possible. Ripple has been around for 7 years, and they have a strong team, and relationships with major financial institutions. One of Ripple’s early projects was a currency called XRP. The goal of XRP was to make a fast, scalable digital asset that would facilitate currency exchange among banks. We covered Ripple and XRP in previous episodes with David Schwartz and Greg Kidd. XRP remains in circulation, but Ripple the company has shifted development resources away from XRP, and towards RippleNet, which seeks to replace the aging SWIFT code system for banks. Today, XRP is being experimented with by several money transfer companies, but the digital currency is not widely used for anything–well, other than speculation. In the tremendous cryptocoin bull run of early 2018, XRP shot up as sharply as almost any other coin. In an article about Ripple, Matt Leising tried to get to the root explanation for why this occurred. Was it a sudden market recognition of some long term value of XRP? Was it a stampeding herd of people who did not know the state of XRP? Was it a pump and dump? A few days after publishing his article about Ripple, Matt wrote about Tether. Tether purports to be a “stablecoin”–a digital currency which is pegged to the value of something less volatile. Stablecoins are useful in that they can reduce friction of exchange between tokens. Without a stablecoin, you might have to transfer from one cryptocurrency to USD, which probably involves the US banking system. There’s a good discussion of stablecoins in our episode with Vlad Zamfir and Haseeb Qureshi on cryptoeconomics. This was Matt’s second appearance on the show, and it was a blast to have him back on. In his last episode, he discussed the infamous DAO hack, which led to an Ethereum fork. To find that episode as well as links to learn more about the topics described in the show, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Over the last decade, computation and storage has moved from on-premise hardware into the cloud data center. Instead of having large servers “on premise,” companies started to outsource their server workloads to cloud service providers. At the same time, there has been a proliferation of devices at the “edge.” The most common edge device is your smartphone, but there are many other smart devices that are growing in number–drones, smart cars, Nest thermostats, smart refrigerators, IoT sensors, and next generation centrifuges. Each of these devices contains computational hardware. Another class of edge device is the edge server. Edge servers are used to facilitate faster response times than your core application. For example, Software Engineering Daily uses a content delivery network for audio files. These audio files are distributed throughout the world on edge servers. The core application logic of Software Engineering Daily runs on a WordPress site, and that WordPress application is distributed to far fewer servers than our audio files. “Cloud computing” and “edge computing” both refer to computers that can serve requests. The “edge” is commonly used to refer to devices that are closer to the user–so they will deliver faster responses. The “cloud” refers to big, bulky servers that can do heavy duty processing workloads–such as training machine learning models, or issuing a large distributed MapReduce query. As the volume of computation and data increases, we look for better ways to utilize our resources, and we are realizing that the devices at the edge are underutilized. In today’s episode, Kenton Varda explains how and why to deploy application logic to the edge. He works at Cloudflare on a project called Cloudflare Workers, which are a way to deploy JavaScript to edge servers, such as the hundreds of data centers around the world that are used by Cloudflare for caching. Kenton was previously on the show to discuss protocol buffers, a project he led while he was at Google. To find that episode, and many other episodes about serverless, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

How do you build resilient, failure tested systems? Redundancy, backups, and testing are all important. But there is also an increasing trend towards chaos engineering–the technique of inducing controlled failures in order to prove that a system is fault tolerant in the way that you expect. In last week’s episode with Kolton Andrus, we discussed one way to build chaos engineering as a routine part of testing a distributed system. Kolton discussed his company Gremlin, which injects failures by spinning up a Gremlin container and having that container induce network failures, memory errors, and filled up disks. In this episode, we explore another insertion point for testing controlled failures, this time from the point of view of Linkedin. Linkedin is a social network for working professionals. As Linkedin has grown, the increased number of services has led to more interdependency between those services. The more dependencies a given service has, the more partial failure cases there are. That’s not to say there is anything wrong with having a lot of service dependencies–this is just the way we build modern applications. But it does suggest that we should try to test the failures that can emerge from so many dependencies. Bhaskaran Devaraj and Xiao Li are engineers at Linkedin, and are working on a project called Waterbear, with the goal of making the infrastructure more resilient. Linkedin’s backend system consists of a large distributed application with thousands of microservices communicating between each other. Most of those services communicate over Rest.li, a proxy for standardizing interactions between services. Rest.li can assist with routing, AB testing, circuit breaking, and other aspects of service-to-service communication. This proxy can also be used for executing controlled failures. As services are communicating with each other, creating a controlled failure can be as simple as telling your proxy not to send traffic to downstream services. If that sounds confusing, don’t worry, we will explain it in more detail. In this episode, Bhaskaran and Xiao describe their approach to resilience engineering at Linkedin–including the engineering projects and the cultural changes that are required to build a resilient software architecture. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The number of ways that applications can fail are numerous. Disks fail all the time. Servers overheat. Network connections get flaky. You assume that you are prepared for such a scenario, because you have replicated your servers. You have the database backed up. Your core application is spread across multiple availability zones. But are you really sure that your system is resilient? The only way to prove that your system is resilient to failure is to experience failure, and to make swift responsiveness to failure an integral part of your software. Chaos engineering is the practice of routinely testing your system’s resilience by inducing controlled failures. Netflix was the first company to discuss chaos engineering widely, but more and more companies are starting to work it into their systems, and finding it tremendously useful. By inducing failures in your system, you can discover unknown dependencies, single points of failure, and problematic state conditions that can cause data corruption. Kolton Andrus worked on chaos engineering at Netflix and Amazon, where he designed systems that would test system resiliency through routine failures. Since then, he founded Gremlin, a company that provides chaos engineering as a service. In a previous episode, Kolton and I discussed why chaos engineering is useful, and he told some awesome war stories about working at Amazon and Netflix. In this show, we explore how to build a chaos engineering service–which involves standing up Gremlin containers that institute controlled failures. To find the previous episode I recorded with Kolton, as well as other supplementary materials described in this show, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

On this show, we spend a lot of time talking about CI/CD, data engineering, and microservices. These technologies have only been widely talked about for the last 5-10 years. That means that they are easy to adopt for startups that get founded in the last 5-10 years, but not necessarily for older enterprises. Within a large enterprise, it can be challenging to make significant changes to how technology is used. Many of the listeners might even take it for granted that your source code is in git–but if you work at an enterprise that started building software in 1981, you might be moving source code around on FTP servers or floppy disks. The difficulty of changing the technology within an enterprise gets compounded by culture. Culture develops around specific technologies. That is one interpretation of “Conway’s Law”–that the way an organization uses software informs an organization’s communication structure. This is no surprise–if your organization manages code using FTP servers and floppy disks, it will slow down your innovation. Zhamak Dehghani is an engineer at ThoughtWorks, where she consults with enterprises to modernize their software and culture. She works off of a blueprint that describes specific steps that an enterprise can take towards modernizing: continuous integration; building a data pipeline; building a system of experimentation. In some ways, this conversation fits nicely with our shows about DevOps a few years ago. Full disclosure: ThoughtWorks is a sponsor of Software Engineering Daily. To find all of our shows about DevOps, as well as links to learn more about the topics described in the show, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Sue Loh is a software engineer and author of a book with the goal of breaking developer stereotypes. Stereotyping among developers leads to bad outcomes. When incorrect assumptions are made about certain populations, those populations feel marginalized and engineering resources get misallocated. From the perspective of Sue, the main problem is about how children are socialized. Young girls in particular are discouraged from programming, leading to a steady decline in the percentage of women in the computing workforce. With her book, Sue is hoping to create a piece of literature that will expose young female and minority students to the world of technology, and help them realize how cool engineering can be. Sue created a Kickstarter around her book, and has raised more than $40,000. Sue also describes her experiences as an engineer who has risen through the ranks at Microsoft, and how her perspective on engineering has evolved as she has become a principal engineer. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Functional programming can improve the overall design of an application architecture. Runar Bjarnason has been exploring how writing in a functional style increases modularity and compositionality of software for many years. He is co-author of Functional Programming in Scala, a book that explores the relationship between functional programming and software design. In this interview with guest host Adam Bell, Runar explains how writing in a functional style involves limiting side effects, avoiding exceptions and using higher order abstractions. Writing in this style places constraints on what a module in a software system may do, but by constraining modules in this way, the software modules themselves become endlessly composable. Functional Programming In Scala Constraints Liberate, Liberties Constrain Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Training a deep learning model involves operations over tensors. A tensor is a multi-dimensional array of numbers. For several years, GPUs were used for these linear algebra calculations. That’s because graphics chips are built to efficiently process matrix operations. Tensor processing consists of linear algebra operations that are similar in some ways to graphics processing–but not identical. Deep learning workloads do not run as efficiently on these conventional GPUs as they would on specialized chips, built specifically for deep learning. In order to train deep learning models faster, new hardware needs to be designed with tensor processing in mind. Xin Wang is a data scientist with the artificial intelligence products group at Intel. He joins today’s show to discuss deep learning hardware and Flexpoint, a way to improve the efficiency of space that tensors take up on a chip. Xin presented his work at NIPS, the Neural Information Processing Systems conference, and we talked about what he saw at NIPs that excited him. Full disclosure: Intel, where Xin works, is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At KubeCon, I had the chance to catch up with Adam Stacoviak of the Changelog, a podcast that was an inspiration for starting Software Engineering Daily. Changelog has long been one of my favorite podcasts about engineering, thanks in part to Adam’s personality. This was a spontaneous conversation, but it was a good one.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A modern farm has hundreds of sensors to monitor the soil health, and robotic machinery to reap the vegetables. A modern shipping yard has hundreds of computers working together to orchestrate and analyze the freight that is coming in from overseas. A modern factory has temperature gauges and smart security cameras to ensure workplace safety. All of these devices could be considered “edge” devices. Over the last decade, these edge devices have mostly been used to gather data and save it to an on-premise server, or to the cloud. Today, as the required volumes of data and compute scale, we look for ways to better utilize our resources. We can start to deploy more application logic to these edge devices, and build a more sophisticated relationship between our powerful cloud servers and the less powerful edge devices. The soil sensors at the farm are recording long time series of chemical levels. The pressure sensors in a centrifuge are recording months and years of data. The cameras are recording terabytes of video. These huge data sets are correlated with labeled events–such as crop yields. With these large volumes of data, we can construct models for responding to future events. Deep learning can be used to improve systems over time. The models can be trained in the cloud, and deployed to devices at the edge. Aran Khanna is an AI engineer with Amazon Web Services, and he joins the show to discuss workloads at the cloud and at the edge–how work can be distributed between the two places, and the tools that can be used to build edge deep learning systems more easily. To find all of our shows about machine learning and edge computing, as well as links to learn more about the topics described in the show, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

After two weeks of episodes about Kubernetes, our in-depth coverage of container orchestration is drawing to a close. We have a few more shows on the topic before we move on to covering other aspects of software. If you have feedback on this thematic format (whether you like it or not), send me an email: jeff@softwareengineeringdaily.com Today’s episode fits nicely into some of the themes we have covered recently–Cloud Foundry, Kubernetes, and the changing landscape of managed services. Sean McKenna works on all three of these things at Microsoft. We spent much of our time discussing the use cases of container instances versus Kubernetes. Container instances are individual managed containers–so you could spin up an application within a container instance without having to deal with the Kubernetes control plane. Container instances might be described as “serverless containers,” since you do not have to program against the underlying VM at all. This begs the question–why would you want to use a managed Kubernetes service if you could just use individual managed containers? Sean explores this question, and gives his thoughts on where this ecosystem is headed. Full disclosure: Microsoft is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Last month, Software Engineering Daily had our 4th Meetup at Cloudflare in San Francisco. For this Meetup, the format was short interviews with security specialists from Pinterest, Cloudflare, and Segment. Each of these companies has unique security challenges, but they also have overlap in their security strategies. Nick Sullivan, Amine Kamel, and Evan Johnson are all seasoned engineers, and it was a privilege to sit down with each of them. Some topics we discussed: cryptography, secret management, incident response, and social network security. In 2018, I am hoping to travel to several tech hubs and do Meetups. I wanted to do more of these last year, but did not plan effectively. So this year I’d like to plan them far in advance. Some locations I have in mind are New York, Los Angeles, Austin, and Seattle. If you have suggestions, or if you know of a venue that could comfortably host us, send me an email–jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Creating a podcast is still too difficult. One of the main barriers to entry is the editing process. After recording a podcast, the podcast producer needs to line up soundwaves in a digital audio workstation and clip the raw audio files to remove sections that need to be removed. As someone who has edited a lot of podcasts, I know that this is difficult and tedious. One way of simplifying the editing process is to use speech-to-text to produce a transcription of an audio file, and aligning the text output with the audio. After that alignment, you have a mapping between the text and the audio–so you can delete text and have the corresponding audio be deleted as well. SpeechBoard is a project by Craig Cannon and Ramon Recuero Moreno. SpeechBoard is an easy way to edit podcasts by deleting transcribed words that are mapped to an audio interview. In this episode, Craig, Ramon and I discuss how SpeechBoard is built and why this product hasn’t existed until recently. We also discuss the podcast world, which Craig is deeply familiar with as the host of Y-Combinator’s podcast. The YC podcast is one of my favorite shows, and if you like SE Daily, you will probably like the YC podcast, so check it out. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In 2011, platform-as-a-service was in its early days. It was around that time that Gabe Monroy started a container platform called Deis, with the goal of making an open source platform-as-a-service that anyone could deploy to whatever infrastructure they wanted. Over the last six years, Gabe had a front row seat to the rise of containers, the variety of container orchestration systems, and the changing open source landscape. Every container orchestration system consists of a control plane, a data plane, and a scheduler. In the last few weeks, we have been exploring these different aspects of Kubernetes in detail. Last year, Microsoft acquired Deis, and Gabe began working on the Azure services that are related to Kubernetes–Azure Container Service, Kubernetes Service, and Container Instances. In this episode, Gabe talks about how containerized applications are changing, and what developments might come in the next few years. Kubernetes, functions-as-a-service, and container instances are different cloud application runtimes, with different SLAs, interfaces, and economics. Gabe provided some thoughts on how different application types might use those different runtimes. Full disclosure: Microsoft is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Oliver Gould worked at Twitter from 2010 to 2014. Twitter’s popularity was taking off, and the engineering team was learning how to scale the product. During that time, Twitter adopted Apache Mesos, and began breaking up its monolithic architecture into different services. As more and more services were deployed, engineers at Twitter decided to standardize communications between those services with a tool called a service proxy. A service proxy provides each service with features that every service would want: load balancing, routing, service discovery, retries, and visibility. It turns out that lots of other companies wanted this service proxy technology as well, which is why Oliver left Twitter to start Buoyant, a company that was focused on developing software around the service proxy–and eventually the service mesh. If you are unfamiliar with service proxies and service mesh, check out our previous shows on Linkerd, Envoy, and Istio. Kubernetes is often deployed with a service mesh. A service mesh consists of two parts: the data plane and the control plane. The “data plane” refers to the sidecar containers that are deployed to each of your Kubernetes application pods. Each sidecar has a service proxy. The “control plane” refers to a central service that aggregates data from across the data plane and can send communications to the service proxies sitting across that control plane. The Linkerd service mesh was built in Java, and the project started before Kubernetes had become the standard for container orchestration. More recently, Buoyant built Conduit, a service mesh built using Rust and Go. In this episode, we explore how to design a service mesh and what Oliver learned in his experience building Linkerd and Conduit. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern applications store most of their data on hosted storage solutions. We use hosted block storage to back databases, hosted object storage for objects such as videos, and hosted file storage for file systems. Using a cloud provider for these storage systems can simplify scalability, durability, and availability–it can be less painful than taking care of storage yourself. One downside: the storage systems offered by the cloud providers are not open source. The APIs might vary from provider to provider. Wiring your application to a particular storage service on a particular cloud could tightly couple you to that cloud. Rook is a project for managing storage, built on Kubernetes. If you use a Rook cluster for your storage, you can port that storage model to any cloud, and have a consistent API for object, block, and file storage. In this episode, Bassam Tabbara describes the state of cloud storage, and why he started the Rook project. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A common problem in a distributed system: how do you take a snapshot of the global state of that system? Snapshot is difficult because you need to tell every node in the system to simultaneously record its state. There are several reasons to take a snapshot. You might want to take a picture of the global state for the purposes of debugging. Or you might want to take a comprehensive snapshot of your system (including the database) and port your system from one cloud to another. Or you might just need to take a snapshot for disaster recovery. When a Kubernetes application is deployed, its initial configuration is described in config files. After a deployment, the state of the application might change–some nodes die, some services get scaled up. At any given time, the current state of a Kubernetes cluster is described by etcd, a distributed key-value store. Niraj Tolia is CEO of Kasten, a company that provides data management, backups, and disaster recovery for Kubernetes applications. Niraj joins the show to describe how Kubernetes deployments manage state, and what the modern business environment is around Kubernetes. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the last four years, CoreOS has been at the center of enterprise adoption of containers. During that time, Brian Harrington (or “Redbeard”) has seen a lot of deployments. In this episode, Brian discusses the patterns he has seen among successful Kubernetes deployments–and the pitfalls of the less successful. How should you manage configuration? How can you avoid IP address overlap between containers? How should you log and monitor your Kubernetes cluster–and whose responsibility is it to set all that stuff up? Brian also discusses the motivation for multi-cloud deployments, and how to implement multi-cloud Kubernetes. CoreOS offers a distributed systems management tool called Tectonic, which uses Kubernetes for container orchestration. In a time where there are lots of options to choose from when it comes to managed Kubernetes providers, it was great to hear Brian describe some of the architectural decisions for building Kubernetes into Tectonic. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A backend application can have hundreds of services written in different programming frameworks and languages. Across these different languages, log messages are produced in different formats. Some logging is produced in XML, some is produced in JSON, some is in other formats. These logs need to be unified into a common format, and centralized for any developer who wants to debug. The popularity of Kubernetes is making it easier for companies to build this kind of distributed application, where different services of different languages are communicating over a network, with a variety of log message types. Fluentd is a tool for solving this problem of log collection and unification. In today’s episode, Eduardo Silva joins the show to describe how Fluentd is deployed to Kubernetes, and what the role of Fluentd is within a Kubernetes logging pipeline. We also discuss the company where Eduardo works–Treasure Data. The story of Treasure Data is unusual. The team started out doing log management, but has found itself moving up the stack, into marketing analytics, sales analytics, and customer data management. This story might be useful for anyone who is open source developer thinking about how to evolve your project into a business. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes has become the standard way of deploying new distributed applications. Most new internet businesses started in the foreseeable future will leverage Kubernetes (whether they realize it or not). Many old applications are migrating to Kubernetes too. Before Kubernetes, there was no standardization around a specific distributed systems platform. Just like Linux became the standard server-side operating system for a single node, Kubernetes has become the standard way to orchestrate all of the nodes in your application. With Kubernetes, distributed systems tools can have network effects. Every time someone builds a new tool for Kubernetes, it makes all the other tools better. And it further cements Kubernetes as the standard. Google, Microsoft, Amazon, and IBM each have a Kubernetes-as-a-service offering, making it easier to shift infrastructure between the major cloud providers. We are likely to see Digital Ocean, Heroku, and longer tail cloud providers offer a managed, hosted Kubernetes eventually. In this editorial, I explore the following questions: Click here to read the full “Gravity of Kubernetes” editorial by Jeff Meyerson.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes has become the standard system for deploying and managing clusters of containers. But the vision of the project goes beyond managing containers. The long-term goal is to democratize the ability to build distributed systems. Brendan Burns is a co-founder of the Kubernetes project. He recently announced an open source project called Metaparticle, a standard library for cloud-native development: Metaparticle builds on top of Kubernetes primitives to make distributed synchronization easier… It supplies language independent modules for locking and leader election as easy-to-use abstractions in familiar programming languages. After decades of distributed systems research and application, patterns have emerged about how we build these systems. We need a way to lock a variable, so that two nodes will not be able to write to that variable in a nondeterministic fashion. We need a way to do master election, so that if the master node dies, the other nodes can pick a new node to orchestrate the system. We know that just about every distributed application needs locking and leader election–so how can we build these features directly into our programming tools, rather than bolting them on? With Kubernetes providing a standard operating system for distributed applications, we can start to build standard libraries that assume we have access to underlying Kubernetes primitives. Instead of calling out to external tools like Zookeeper and etcd, a standard library like Metaparticle will abstract them away. An example: if I am writing a system to do distributed mapreduce, I would like to avoid thinking about node failures and race conditions. Brendan’s idea is to push those problems down into a standard library–so the next developer who comes along with a new idea for a multi-node application has an easier time. Brendan Burns currently works as a distinguished engineer at Microsoft, and he joins the show to discuss why it is still hard to build distributed systems and what can be done to make it easier. This is the second time we have had Brendan on the show. The first time he came on, he discussed the history of Kubernetes, and some of the design decisions of the system. This episode was more about the future. Full disclosure: Microsoft (where Brendan is employed) is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ben Sigelman began working on distributed tracing when he was at Google and authored the “Dapper” paper. Dapper was implemented at Google to help debug some of the distributed systems problems faced by the engineers who work on Google infrastructure. Today, a decade after he started thinking about distributed tracing, Ben Sigelman is the CEO of Lightstep, a company that provides distributed tracing and other monitoring technologies. Lightstep’s distributed tracing model still bears a resemblance to the same techniques described in the paper–so I was eager to learn the differences between open source versions of distributed tracing (such as OpenZipkin) and enterprise providers such as Lightstep. The key feature of Lightstep that we discussed: garbage collection. If you are using a distributed tracing system, you could be collecting a lot of traces. You could collect a trace for every single user request. Not all of these traces are useful–but some of them are very useful. Maybe you only want to keep track of traces that take an exceptionally long latency. Maybe you want to keep every trace in the last 5 days, and destroy them over time. So, the question of how to manage the storage footprint of those traces was as interesting as the discussion of distributed tracing itself. Beyond the distributed tracing features of his product, Ben has a vision for how his company can provide other observability tools over time. I spoke to Ben at Kubecon–and although this conversation does not talk about Kubernetes specifically, this topic is undoubtedly interesting to people who are building Kubernetes technologies. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Since Kubernetes came out, engineers have been deploying clusters to Amazon. In the early years of Kubernetes, deploying to AWS meant that you had to manage the availability of the cluster yourself. You needed to configure etcd and your master nodes in a way that avoided having a single point of failure. Deploying Kubernetes on AWS became simpler with an open-source tool called kops (short for Kubernetes Operations). Kops automates the provisioning and high-availability deployment of Kubernetes. In late 2017, AWS released a managed Kubernetes service called EKS. EKS allows developers to run Kubernetes without having to manage the availability and scaling of a cluster. The announcement of EKS was exciting, because it means that all of the major cloud providers are officially supporting Kubernetes. Arun Gupta is a principal open source technologist at AWS, and he joins the show to explain what is involved in deploying and managing a Kubernetes cluster. Arun describes how to operate a Kubernetes cluster, including logging, monitoring, storage, and updates. If you are convinced that you want to use Kubernetes, but you aren’t sure yet how you want to deploy it, this will be useful information for you. We also discussed how Amazon built EKS, and some of the architectural decisions they made. AWS has had a managed container service called ECS since 2014. The development of ECS was instructive for the AWS engineers who built EKS. Amazon wanted to make EKS able to integrate with both open source tools and the Amazon managed services. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A single user request hits Google’s servers. A user is looking for search results. In order to deliver those search results, that request will have to hit several different internal services on the way to getting a response. These different services work together to satisfy the user request. All of these services need to communicate efficiently, they need to scale, and they need to be secure. Services need to have a consistent way of being “observable”–allowing logging and monitoring. Services need to have proper security. Since every service wants these different features (like communication, load balancing, security), it makes sense to build these features into a common system that can be deployed to every server. Louis Ryan has spent his years at Google working on service infrastructure. During that time, he has seen massive changes in the way traffic flows through Google. First, the rise of Android, and all of the user traffic from mobile phones. And second, the rise of Google Cloud Platform, which meant that Google was now responsible for nodes deployed by users outside of Google. These two changes–mobile and cloud–led to an increase in the amount of traffic and the type of traffic. All of this traffic leads to more internal services communicating with each other. How does service networking change in such an environment? Google’s adaptation to the new networking conditions is to introduce a “service mesh”. A service mesh is a network for services. It provides observability, resiliency, traffic control, and other features to every service that plugs into it. Each service needs to plug into the service mesh. In Kubernetes, services connect to the mesh through a sidecar. Let me explain the term “sidecar.” Kubernetes manages its resources in pods, and each pod contains a set of containers. You might have a pod that is dedicated to responding to any user that is requesting a picture of a cat. Within that pod, you not only have the container that serves the cat picture–you also have other “sidecar” containers that help out an application container. You could have a sidecar that gets deployed next to your application container that handles logging, or a sidecar that helps out with monitoring, or network communications. If you are using the Istio service mesh, that means that you are using a sidecar called Envoy. Envoy is a sidecar called a “service proxy” that provides configuration updates, load balancing, proxying, and lots of other benefits. If we get all that out of Envoy, why do we need a separate abstraction of a “service mesh”? Because it helps to have a tool that aggregates and centralizes all the different communications among these proxies. Every service gets a sidecar for a service proxy. Every service proxy communicates with the centralized service mesh. Louis Ryan joins this episode to explain the motivations for building the Istio service mesh, and the problems it solves for Kubernetes developers. For the next two weeks, we are covering exclusively the world of Kubernetes. Kubernetes is a project that is likely to have as much impact as Linux–and it is very early days. Whether you are an expert in Kubernetes or you are just starting out, we have lots of episodes to fit your learning curve. To find all of our old episodes about Kubernetes (including a previous show about Istio), download the Software Engineering Daily app for iOS or for Android. In other podcast players, only the most 100 recent episodes are available, but in our apps you can find all 650 episodes–and there is also plenty of content that is totally unrelated to Kubernetes! Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

With the community centralizing on Kubernetes, developers are able to comfortably bet big on open source projects like Istio, Conduit, Rook, Fluentd, and Helm, each of which we will be covering in the next few weeks. The centralization on Kubernetes also makes it easier to build enterprise companies, who are no longer trying to think about which container orchestration to support. There is a wide array of Kubernetes-as-a-service providers offering a highly available runtime–and a variety of companies offering observability tools to make it easier to debug distributed systems problems. Despite all of these advances–Kubernetes is less usable than it should be. It still feels like operating a distributed system. Hopefully someday, operating a Kubernetes cluster will be as easy as operating your laptop computer. To get there, we need improvements in Kubernetes usability. Today’s guest Joe Beda was one of the original creators of the Kubernetes project. He is a founder of Heptio, a company that provides Kubernetes tools and services for enterprises. I caught up with Joe at KubeCon 2017, and he told me about where Kubernetes is today, where it is going, and what he is building at Heptio. Full disclosure–Heptio is a sponsor of Software Engineering Daily. For the next two weeks, we are covering exclusively the world of Kubernetes. Kubernetes is a project that is likely to have as much impact as Linux. Whether you are an expert in Kubernetes or you are just starting out, we have lots of episodes to fit your learning curve. To find all of our old episodes about Kubernetes, download the Software Engineering Daily app for iOS or for Android. In other podcast players, only the most 100 recent episodes are available, but in our apps you can find all 650 episodes–and there is also plenty of content that is totally unrelated to Kubernetes! Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the first 10 years of cloud computing, a set of technologies emerge that every software enterprise needs; continuous delivery, version control, logging, monitoring, routing, data warehousing. These tools were built into the Cloud Foundry project, a platform for application deployment and management. As we enter the second decade of cloud computing, another new set of technologies are emerging as useful tools. Serverless functions allow for rapid scalability at a low cost. Kubernetes offers a control plane for containerized infrastructure. Reactive programming models and event sourcing make an application more responsive and simplify the interactions between teams who are sharing data sources. The job of a cloud provider is to see new patterns in software development and offer tools to developers to help them implement those new patterns. Of course, building these tools is a huge investment. If you’re a cloud provider, your customers are trusting you with the health of their application. The tool that you build has to work properly and you have to help the customers figure out how to leverage the tool and resolve any breakages. Onsi Fakhouri is the senior VP of R&D for cloud at Pivotal, a company that provides a software and support for Spring, Cloud Foundry and several other tools. I sat down with Onsi to discuss his strategy for determining which products Pivotal chooses to build. There are a multitude of engineering and business elements that Onsi has to consider when allocating resources to a project. Cloud Foundry is used by giant corporations like banks, telcos and automotive manufacturers. Spring is used by most enterprises that run Java, including most of the startups that I have worked at in the past. Cloud Foundry has to be able to run on premise and in the cloud providers like AWS, Google and Microsoft. Pivotal also has its own cloud, Pivotal Web Services, and all of these stakeholders have different technologies that they would like to see built. Onsi’s job is to determine which ones have the highest net impact and make a decision on those and allocate resources towards them. I interviewed Onsi at Spring One Platform, which is a conference that is organized by Pivotal who, full disclosure, is a sponsor of Software Engineering Daily. This week’s episodes are all conversations from that conference, and if there’s a conference that you think I should attend and do coverage at, let me know. Whether you like this format or not, I would love to get your feedback. We have some big developments coming for Software Engineering Daily in 2018 and we want to have a closer dialogue with the listeners. Please send me an email, jeff@softwareengineeringdaily.com or join our Slack channel. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the 1980s and the 1990s, most applications used only a relational database for their data management. In the early 2000s, software projects started to use an ever increasing number of data sources. MongoDB popularized the document database, which allows storage of objects that do not have a consistent schema. The Hadoop distributed file system enabled the redundant storage and efficient querying of high volumes of data that are spread out across multiple commodity disks. The Cassandra Database is a hybrid between key-value storage and column-oriented storage. The benefit of these different data systems is that you can choose a system that gives you the read and write performance that you need. The downside is that each of these databases has different querying semantics. If you’re a developer trying to access data from your application, you often need to know how to access that data from the specific data source and whether that data needs to be queried with SQL, or with the document style query, or with a MapReduce job. Spring Data is a project to standardize the programming model for data access within Spring. The vision for the project is to give Spring developers a consistent way to access their data from any database, or retaining the performance characteristics of those databases. Spring is a Java framework for writing web applications, but this conversation is useful even for people who are not building these Spring applications. Whatever application you’re building, you are probably pulling from multiple data sources. The question of how to abstract away the complexity of those multiple data sources is also being tackled by projects such as GraphQL and Falcor. John Blum is a staff engineer who works on the Spring Data Project at Pivotal. He joins the show to discuss how to design a data access layer. We discussed the API between a database and the Spring Data layer and also talked about reactive programming. Reactive programming allows the application layer to respond to changes in the underlying data layer. I interviewed John at SpringOne Platform, which is a conference that is organized by Pivotal, who full disclosure is a sponsor of Software Engineering Daily. This week’s episodes are all conversations from that conference. If there’s a conference that you think I should attend and do some coverage at, please let me know. Whether you like this format or not, I would love to get your feedback. We have some big developments coming for Software Engineering Daily in 2018, and we want to have a closer dialogue with the listeners. Please send me an e-mail jeff@softwareengineeringdaily.com, let me know what’s up. Or join our Slack channel. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud Foundry is an open-source platform as a service for deploying and managing web applications. Cloud Foundry is widely used by enterprises who are running applications that are built using Spring, a popular web framework for Java applications, but developers also use Cloud Foundry to manage apps built in Ruby, Node and any other programming language. Cloud Foundry includes routing, message brokering, service discovery, authentication and other application level tooling for building and managing a distributed system. Some of the standard tooling in Cloud Foundry was adopted from Netflix open-source projects, such as Hystrix, which is the circuit breaker system; and Eureka, which is the service discovery server and client. When a developer deploys their application to Cloud Foundry, the details of what is going on are mostly abstracted away, which is by design. When you’re trying to ship code and iterate quickly for your organization, you don’t want to think about how your application image is being deployed to underlying infrastructure. You don’t want to think about whether you’re deploying a container or a VM, but if you use Cloud Foundry enough, you might have become curious about how Cloud Foundry schedules and runs application code. BOSH is a component of Cloud Foundry that sits between the infrastructure layer and the application layer. Cloud Foundry can be deployed to any cloud provider because of BOSH’s well-defined interface. BOSH has the abstraction of a stem cell, which is a versioned operating system image wrapped in packaging for whatever infrastructure as a service is running underneath. With BOSH, whenever a VM gets deployed no your underlying infrastructure, that VM gets a BOSH agent. The agent communicates with the centralized component of BOSH called the director. This role of director is the leader of the distributed system. Rupa Nandi is a director of engineering at Pivotal where she works on Cloud Foundry. In this episode we talked about scheduling an infrastructure, the relationship between Spring and Cloud Foundry and the impact of Kubernetes, which Cloud Foundry has integrated with so that users can run Kubernetes workloads on Cloud Foundry. I interviewed Rupa at SpringOne Platform, a conference that is organized by Pivotal who, full disclosure, is a sponsor of Software Engineering Daily, and this week’s episode are all conversations from that conference. Whether you like this format or don’t like this format, I would love to get your feedback. We have some big developments coming for Software Engineering Daily in 2018 and we want to have a closer dialogue with the listeners. Please send me an email, jeff@softwareengineeringdaily.com or join our Slack channel. We really want to know what you’re thinking and what your feedback is, what you would like to hear more about, what you’d like to hear less about, who you are. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published October 22, 2015 Dwarf Fortress is a construction and management simulation computer game set in a procedurally generated fantasy world in which the player indirectly controls a group of dwarves, and attempts to construct a successful underground fortress. Tarn Adams works on Dwarf Fortress with his brother Zach. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published January 6, 2016 Brian Kernighan is a professor of computer science at Princeton University and the author of several books, including The Go Programming Language and The C Programming Language, a book more commonly referred to as K&R. Professor Kernighan also worked at Bell Labs alongside Unix creators Ken Thompson and Dennis Ritchie and contributed to the development of Unix. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published November 18, 2015 Seth Godin is a writer, speaker, and entrepreneur. He is the author of many books, including most recently, What To Do When It’s Your Turn. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published November 10, 2015 Wolfram Research makes computing software powered by the Wolfram language, a knowledge-based programming language that draws from symbolic and functional programming paradigms. Stephen Wolfram is the Founder and CEO of Wolfram Research, and also the author of A New Kind of Science. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Originally published November 17, 2015 Technical debt, referring to the compounding cost of changes to software architecture, can be especially challenging in machine learning systems. D. Sculley is a software engineer at Google, focusing on machine learning, data mining, and information retrieval. He recently co-authored the paper Machine Learning: The High Interest Credit Card of Technical Debt. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Military force is powered by software. The drones that are used to kill suspected terrorists can identify those terrorists using the same computer vision tools that are used to identify who is in an Instagram picture. Nuclear facilities in Iran were physically disabled by the military-sponsored Stuxnet virus. National intelligence data is collected and processed using the MapReduce algorithm. The military keeps up with technology more effectively than lawmakers. It is common to read a quote from a senator or a judge that shows a basic misunderstanding of cybersecurity. Many politicians do not even use email. There is a large and growing knowledge gap between military capability and the technological savvy of policymakers. On the whole, government is not prepared for modern warfare. Just like in social media information wars, the instigators of conflict have an advantage. And the ability to instigate such a conflict is democratized. Social media, open source software, and cloud computing give a technologist superpowers. Cryptocurrencies can anonymize the financial transactions to pay for such tools, and basic encryption can anonymize the terroristic acts that occur over a remote internet connection. Peter Warren Singer is a political scientist who formerly worked in the United States advisory committee on International Communications and Information Policy. He is also an author, whose books include Wired for War, Cybersecurity and Cyberwar: What Everyone Needs to Know, and Ghost Fleet: A Novel of the Next World War. Peter writes about the circumstances that could lead to global warfare, and how military actors might behave in a third world war. In this episode, Peter shares a dark, but realistic vision that we should all hope to avoid. If you like this episode, we have done many other shows on related topics–including drones, IoT security, and automotive cybersecurity. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern frontend development is about components. Whether we are building an application in React, Vue, or Angular, components are the abstractions that we build our user interfaces out of. Today, this seems obvious, but if you think back five years ago, frontend development was much more chaotic–partly because we had not settled around this terminology of the component. React has become the most popular frontend framework, and part of its growth is due to the ease and reusability of components across the community. It’s easy to find building blocks that you can use to piece together your frontend application. Do you need a video player component? Do you need a news feed component? A profile component? All of these things are easy to find. As you build a React application, you take some open source components off the shelf, and you build others yourself. To keep things looking nice and consistent, you need to style your components. If you are not careful with how you manage your stylesheets, you can end up with inconsistent stylings and namespace conflicts. Max Stoiber is the creator of styled-components, a project to help enforce best practices around styling components. He has also a founder of Spectrum, a system that allows people to build online communities. Spectrum has similar design and engineering challenges to Slack or Facebook, so it made for a great discussion of modern software architecture. In today’s episode, Max and I had a wide-ranging conversation about frontend frameworks, components, and the process of building a product. Max also describes the advantages of using GraphQL and the Apollo toolchain. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

“Management is about human beings. Its task is to make people capable of joint performance. To make their strengths effective and their weaknesses irrelevant.” That quote is from Peter Drucker. It is one of the many useful quotes collected in Ron Lichty’s book “Managing the Unmanageable”—and it illustrates why we work in teams. When we collaborate with each other, we make each other’s strengths effective, and our weaknesses become irrelevant. To collaborate effectively, we need leaders. We need management. Ron Lichty spent 6 years managing engineers at Apple, and many more years in management and director roles elsewhere. In his book, Ron lays out the lessons he learned in 30 years of engineering management. Ron also describes concrete strategies for how to manage engineers productively. An engineer who becomes a manager needs to learn new skills. And the hardest skills to master have nothing to do with technology. Prioritizing the right projects, allocating engineering resources, making architectural decisions—all of those skills are important. But the art of relationships—of diplomacy and language—is harder to learn than any technical skill. How do you motivate an engineer to do something that is boring? How do you have a difficult conversation with an engineer who needs to improve? When a conflict between engineers comes up, do you confront the conflict head-on, or do you wait for those engineers to resolve it among themselves? These questions do not have easy answers. The best way to learn how to react to these situations is to live through them. The second best way to learn is to read and listen to people who have seen so much of the management dynamic that they can distill it into anecdotes and aphorisms. In today’s show, Ron shares several stories that changed how I think about management. Ron and I did not have time to discuss everything I wanted to, and I recommend checking out his podcast episode on Software Engineering Radio for more detail. And also check out his book—Managing the Unmanageable. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The New York Times makes most of its money off of subscriptions. Facebook makes its money off of native advertising. Hacker News is funded by Y-Combinator. Each of these business models creates biases in the information that gets promoted on the respective platforms. This is why I like to know the origin story and the business models behind the publications that I read. Published content is shaped by the profit motive of the publication. And yet, last month, I repeatedly found myself reading high quality content on a Medium publication that I did not know the origin of: Hacker Noon. Hacker Noon is a popular Medium publication that syndicates curated content written about software. Let me explain “syndication.” Imagine that I just spent three days on a Medium post about functional programming, and I have zero followers on social media. How can I get people to read awesome post? The answer is syndication. I can submit my Medium post to Hacker Noon. This gives me free distribution, and it gives Hacker Noon free content—a win-win relationship. But why was it worth it for Hacker Noon to spend time curating content? That syndication process takes time. You have to read through lots of submissions, sometimes you have to send it back to the author to have it edited. And this is all to build a following on Medium. I have not heard of Medium being a profitable platform to build a business. It’s worth pointing out the difference between Medium and WordPress. On WordPress, this model of curated syndication has worked to massive success—for example, the Huffington Post and TechCrunch. These businesses make millions of dollars from advertising networks, because they are built on WordPress, and WordPress is an open model. A publisher on WordPress can install plugins that serve ads from third party providers like Outbrain and Taboola. A WordPress site can also install any kind of data collection scripts, to gather data on visitors, and sell it to the highest bidder. The lack of third party plugins is the blessing and the curse of Medium. Because there is no third party ecosystem, reading content on Medium is a beautiful experience. The page loads quickly and predictably. There are no random scripts that are blocking the page as they hog your browser’s resources. When you go to close the page, there is never a popup that asks you to subscribe to a newsletter. When I read content on Medium, I am not getting slapped across the face with ads for reverse mortgages and açaí berries. I am not being tagged for retargeting. It’s a beautiful experience. But Medium seems like an ecosystem that would not allow for the content syndication business like Hacker Noon. I wanted to know who was running Hacker Noon, how the business works, and what it says about Medium as a publishing platform. Hacker Noon turns out to be part of a network of Medium publications called AMI. AMI’s network includes sites like Art + Marketing, Future Travel, and Fit Yourself Club–all of which are distinct syndication platforms. David Smooke is the CEO of AMI, and he joins this episode to explain how his business works, how he has scaled the content syndication business, and why he is betting on Medium. It was a detailed look into the state of online publishing and where it might be headed. If you don’t read Hacker Noon already, one article to start with that shows off the quality of content is Learn Blockchains By Building One. I interviewed the author of that article, Daniel Van Flymen, and it has been one of the most popular episodes of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When engineers are writing code, they are manipulating objects. You might have a user object represented on your computer, and that user object has several different fields—a name, a gender, and an age. When you want to send that object across the network to a different computer, the object needs to be turned into a sequence of 1s and 0s that will travel efficiently across the network. This is known as “serialization.” As the user object sits on your computer, it is represented in 1s and 0s. You could just send that same representation over the wire. But we use efficient serialization to send it over the network in a more compact format. We also have to make sure that when we send that object to another service, the other service knows how to deserialize it, and turn it back into a format that we can operate on at the application level. Protocol buffers are a serialization protocol that originated at Google. Protocol buffers created a standardized interface for efficiently passing data between services. When Kenton Varda worked at Google, he was the tech lead for protocol buffers, and he joins the show to explain how protobufs work—and a newer serialization protocol that Kenton led: Cap’n Proto. You can expect to walk away from this episode with an understanding of how serialization protocols work, and the design tradeoffs you can make when creating a serialization protocol. We also touched on a startup that Kenton founded, called Sandstorm, and how he eventually found himself at Cloudflare, where he works on Cloudflare workers. With these topics, we did not go as deep as I would have liked, and I look forward to having Kenton back on in the near future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google Docs is used by millions of people to collaborate on documents together. With today’s technology, you could spend a weekend coding and build a basic version of a collaborative text editor. But in 2004 it was not so easy. In 2004 Steve Newman built a product called Writely, which allowed users to collaborate on documents together. Initially, Writely was hosted on a single server that Steve managed himself. All of the reads and writes to the documents went through that single server. Writely rapidly grew in popularity, and Steve went through a crash course in distributed systems as he tried to keep up with the user base. In 2006, Writely was acquired by Google, and Steve spent his next four years turning Writely into Google Docs. Eventually he moved onto other projects within Google—“Cosmo” and “Megastore Replication.” When Steve left the company in 2010, he took with him the lessons of logging and monitoring that keep Google’s infrastructure observable. Large organizations have terabytes of log data to manage. This data streams off the servers that are running our applications. That log data gets processed in a “metrics pipeline” and turned into monitoring data. Monitoring data aggregates log data in a more presentable format. Most of the log messages that get created will never be seen with human eyes. These logs get aggregated into metrics, then compressed, and (in many cases) eventually thrown away. Different companies have different sensitivity around their logs, so some companies may not garbage collect any of their logs! When a problem occurs in our infrastructure, we need to be able to dig into our terabytes of log data and quickly find the root cause of a problem. If our log data is compressed and stored on disk, it will take longer to access it. But if we keep all of our logs in memory, it could get expensive. To review: if I want to build a logging system from scratch today I need to build: a metrics pipeline for converting log data into monitoring data; a complicated caching system, a way to store and compress logs; a query engine that knows how to ask questions to the log storage system; a user interface so I don’t have to inspect these logs via command line… The list of requirements goes on and on—which is why there is a huge industry around log management. And logging keeps evolving! One example we covered recently is distributed tracing, which is used to diagnose requests that travel through multiple endpoints. After Steve Newman left Google, he started Scalyr, a product that allows developers to consume, store, and query log messages. I was looking forward to talking to Steve about data engineering, and the query engine that Scalyr has architected, but we actually spent most of our conversation talking about the early days of Writely, and his time at Google—particularly the operational challenges of Google’s infrastructure. Full disclosure: Scalyr is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Duolingo is a language learning platform with over 200 million users. On a daily basis millions of users receive customized language lessons targeted specifically to them. These lessons are generated by a system called the session generator. Andre Kenji Horie is senior engineer at Duolingo. He wrote about the process of rewriting the session generator, moving from Python to Scala and changing architecture at the same time. In this episode Adam Bell talks with him about the reasons for the rewrite, what drove them to move to Scala and the experience of moving from one technology stack to another. Rewriting Doulingo’s Engine in Scala Jobs at Duolingo Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The values system of a company guides the actions of the engineers who work at that company. Some companies value open communication and a flat organization where anybody can talk to anyone else. Other companies encourage hierarchy and secrecy, so that employees are focused on their specific section of the company. Some companies take themselves seriously, and have a work environment that is as stoic as the military. Other companies pride themselves on having good beer and a friendly, laid back atmosphere. When company values are properly defined, the values can be used as reference points when making decisions. At Amazon, one of the core company values is “bias for action.” As an engineer, you are often in a situation where you can wait for more information, or you can start a project with an incomplete picture for how you will finish it. The “bias for action” lets you know that you should usually start the project despite having an incomplete picture. Another use of a company values system is for hiring. When a company publishes their values, prospective employees can use those stated values as a way to know if they would be a good cultural fit. For example “move fast and break things” was a value that allowed Facebook to ship new products faster than any other company before it. But the speed of movement is not for everyone. Some engineers like to have their code unit tested, and free of all bugs before shipping to production. Every company has values that define their company. And every engineer has values that define how they want to work. Lynne Tye started her company Key Values as a platform to index companies by their values systems. This allows engineers to find companies that are a good cultural fit for their values system. Lynne joins the show today to explain how engineers and companies define their values systems, and how that affects the outcomes of engineering organizations. Lynne also talks about her time at HomeJoy, one of the first companies in the “gig economy”. HomeJoy was an on-demand house cleaning service that grew extremely fast, but ultimately went under due to lawsuits. The challenges of HomeJoy were a predictor of the challenges later faced by Uber and Airbnb, and it was fascinating to hear Lynne reflect on her time spent managing operations at HomeJoy–which was about as operationally intensive a company as you can imagine! Thanks to Courtland Allen for the intro to Lynne, and if you haven’t checked out the Indie Hackers podcast, which is hosted by Courtland, you should subscribe to it. Indie Hackers breaks down the engineering and business models behind small software companies–it’s one of my favorite shows. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ten years ago, if you wanted to build software, you probably needed to know how to write code. Today, the line between “technical” and “non-technical” people is blurring. Website designers can make a living building sites for people on WordPress or Squarespace–without knowing how to write code. Salesforce integration experts can help a sales team set up complicated software–without knowing how to write code. Shopify experts can set up an ecommerce store to your exact specifications–without knowing how to write code. WordPress, Squarespace, Salesforce, and Shopify are all fantastic services–but they are not compatible with each other. I can’t install a WordPress plugin on Salesforce. Now imagine this from the point of view of plugin creators. Plugin creators make easy ways to integrate different pieces of software together. Take PayPal as an example. PayPal wants to make it easy for software builders to integrate with their API. One plugin that PayPal has is a button that says “Pay with PayPal.” If I am a developer at PayPal, and I am building a button that people should be able to easily put on their webpage so that their users can pay with PayPal, I have to create a button that is compatible with WordPress, and Squarespace, and Wix, and Weebly, and GoDaddy, and Blogger, and all the other website builders that I might want to integrate with. In 2014, Zack Bloom started a company called Eager. Eager was a cloud app marketplace which allowed app developers to make flexible plugins that non-technical users could drag and drop into their site without technical expertise. In order for these non-technical users to add any apps from the Eager marketplace to their webpage, they had to drop in a line of JavaScript–which is, unfortunately, a significant hurdle for a nontechnical user. Eager proved to be a useful distribution mechanism for plugin developers who could write a plugin once and get distributed to multiple plugin marketplaces. But Eager was not as widely used as a way to directly drag and drop plugins onto sites. The question was: how do you build a marketplace for non-technical users to add plugins to any website without forcing the non-technical user to write code? How do you make editing any website as easy as a WYSIWYG editor? The CDN turns out to be the perfect distribution platform for these kinds of apps. Users already integrate with a CDN, so the CDN can do the work of inserting the code that allows the plugins to be added to a user’s webpage. Because of the opportunity for the integration between a plugin marketplace and a CDN, Eager was acquired by Cloudflare, and Eager became Cloudflare apps. Zack Bloom joins the show today to discuss the motivations for his company, the engineering behind building a cloud app marketplace, and the acquisition process of his company Eager. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Remember when the best game you could play on your phone was Snake? In 1998, Snake was preloaded on Nokia phones, and it was massively popular. That same year Half-Life won game of the year on PC. Metal Gear Solid came out for Playstation. The first version of Starcraft also came out in 1998. In 1998, few people would have anticipated that games with as much interactivity as Starcraft would be played on mobile phones twenty years later. Today, mobile phones have the graphics and processing power of a desktop gaming PC from two decades ago. But one thing still separates desktop gaming from mobile gaming: the network. With desktop gaming, users have a reliable wired connection that keeps their packets moving over the network with speeds that let them compete with other users. With mobile gaming, the network can be flaky. How do we architect real-time strategy games that can be played over an intermittent network connection? Yan Cui is an engineer at Space Ape Games, a company that makes interactive multiplayer games for mobile devices. In a previous episode, Yan described his work re-architecting a social networking startup where the costs had gotten out of control. Yan has a skill for describing software architecture and explaining the tradeoffs. When architecting a multiplayer mobile game, there are many tradeoffs to consider. What do you build and what do you buy? Do you centralize your geographical deployment to make it easier to reconcile conflicts, or do you spread your server deployment out globally? What is the interaction between the mobile clients and the server? The question of interaction between client and server for a mobile game has lessons that are important for anyone building a highly interactive mobile application. For example, think about Uber. When I make a request for a car, I can look at my phone and see the car on the map, slowly approaching me. The driver can look at his phone and see if I move across the street. This is accomplished by synchronizing the data from the driver’s phone and my phone in a centralized server, and sending the synchronized state of the world out to me and the driver. How much data does the centralized server need to get from the mobile phones? How often does it need to make those requests? The answers to these questions will vary based on bandwidth, device type, phone battery life, and other factors. There are similar problems in mobile game engineering, when users are in different players on a virtual map. They are fighting each other, trying to avoid enemies, trying to steal power ups from each other. Mobile games can be even more interactive than a ridesharing app like Uber, so the questions of data synchronization can be even harder to answer. On Software Engineering Daily, we have explored the topic of real-time synchronization in our past shows about the infrastructure of Uber and Lyft. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Yan Cui’s new video course: AWS Lambda in Motion Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Internet was designed as a decentralized system. Theoretically, if Alice wants to send an email to Bob, she can set up an email client on her computer and send that email to Bob’s email server on his computer. In reality, very few people run their own email servers. We all send our emails to centralized services like Gmail, and connect to those centralized services using our own client—a browser on our laptop or a mobile application on our smart phone. Gmail is popular because nobody wants to run their own email server—it’s too much work. With Gmail, our emails our centralized, but centralization comes with convenience. Similar centralization happened with online payments. Decentralization is a desirable feature of computer systems. So how do we make more of our applications decentralized? Martin Kleppman is a distributed systems researcher and the author of Data Intensive Applications. Martin is concerned by the centralization of our computer networks, and he works on CRDT technology in order to make it easier for people to build peer-to-peer applications. Most of the people who know how to build systems with CRDTs are distributed systems PhDs, database experts, and people working at huge internet companies. How do you make developer-friendly CRDTs? How do you allow random hackers to build peer-to-peer applications that avoid conflicts? Start by making a CRDT out of the most widely used, generalizable data structure in modern application development: the JSON object. In today’s episode, Martin and I talk about conflict resolution, CRDTs, and decentralized applications. This is Martin’s second time on the show, and his first interview is the most popular episode to date. You can find a link to that episode in the show notes for this episode, or you can find it in the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Developers can build networked applications today without having to deploy their code to a server. These “serverless” applications are constructed from managed services and functions-as-a-service. Managed services are cloud offerings like database-as-a-service, queueing-as-a-service, or search-as-a-service. These managed services are easy to use. They take care of operational burdens like scalability and outages. But managed services typically solve a narrow use case. You can’t build an application entirely out of managed services. Managed services are scalable and narrow. Functions-as-a-service are scalable and flexible. With managed services, you make remote calls to a service with a well-defined API. With functions-as-a-service, you can deploy your own code. But functions-as-a-service execute against transient, unreliable compute resources. They aren’t a good fit for low latency computation, and the code you run on them should be stateless. Managed services and functions-as-a-service are the perfect complements. Managed services provide you with well-defined server abstractions that every application needs—like databases, search indexes, and queues. Functions as a service offer flexible “glue code” that you can use to create custom interactions between the managed services. The term “serverless” is used to describe the applications that are built entirely with managed services and functions as a service. Serverless applications are dramatically simpler to build and easier to operate than cloud applications of the past. The costs of managed services can get expensive, but the costs of functions as a service can cost 1/10th of what it might take to run a server that is handling your requests. Whether the size of your bill will increase or decrease as your company becomes “serverless” is less of an issue than the fact that your employees will be more productive: serverless applications have less operational burden, so developers spend more time architecting and implementing software. It has been 5 years since the Netflix infrastructure team was talking about the aspirational goal of a “no-ops” software culture. Your software should be so well-defined that you do not need regular intervention of ops staff to reboot your servers and reconfigure your load balancers. Serverless is a newer way of moving operational expense into capital expense. Today’s guest Randall Hunt is a senior technical evangelist with Amazon Web Services. He travels around the world meeting developers and speaking at conferences about AWS Lambda, the functions as a service platform from Amazon. Randall has given some excellent talks about how to architect and build serverless applications (which I will add to the show notes), and today we explore those application patterns further. Serverless Services – Randall Hunt Randall Hunt at AWS Summit Seoul Serverless, What is it Good For? Randall Hunt Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A company’s approach to data can make or break the business. In the past, data was static. There was not much data, it sat in Excel, and it was interacted with on a nightly or monthly basis. Now, data is dynamic, real time and huge. To tap into available data, many industries have oriented themselves to becoming data intensive. With many new industry sectors becoming data driven, a new field called data science emerged. As a new field, data science has attracted a lot of attention from professionals with diverse backgrounds. Describing what is data science and who is a data scientist is not easy. As technologies surrounding the field continue to evolve and new verticals are added, the discourse surrounding the field has attracted different voices putting forward their definition of the field. In this episode, Zacharias Voulgaris joins guest host Sid Ramesh to discuss the developments in the field. He is the author of several data science books, and in today’s conversation Zacharias explains what he means by the data science mindset–including trends and misconceptions that people have on the field. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When I log into my bank account from my laptop, I first enter my banking password. Then the bank sends a text message to my phone with a unique code, and I enter that code into my computer to finish the login. This login process is two-factor authentication. I am proving my identity by entering my banking password (the first factor) and validating that I am in control of my phone (the second factor) by receiving that text message. But in order to log in from my laptop, I need to be in control of my laptop. The laptop itself is a factor. With the laptop and my password, I have two factors. I might not actually need the phone as a factor. Praneet Sharma is the CEO of Keyless, a product that moves 2-factor authentication into the browser. Praneet joins the show to discuss how all kinds of authentication work: multi-factor authentication, single sign on, and Yubikey. We use this discussion of authentication methods to help explain why it actually could make sense for some people to be doing 2-factor authentication without requiring people to take out their phone. We also explore recent security breaches like Target, Equifax and Yahoo–and the industry of security software sold to developers. I see giant banners for security software companies every time I go into the San Francisco airport, and Praneet explained to me some of the products that these kinds of companies are selling. Praneet has joined the show in a previous episode to talk about advertising fraud. He also works with Shailin Dhar at Method Media Intelligence. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Functions as a service are deployable functions that run without an addressable server. Functions as a service scale without any work by the developer. When you deploy a function as a service to a cloud provider, the cloud provider will take care of running that function whenever it is called. You don’t have to worry about spinning up a new machine and monitoring that machine, and spinning the machine down once it becomes idle. You just tell the cloud provider that you want to run a function, and the cloud provider executes it and returns the result. Functions as a service can be more cost effective than running virtual machines or containerized infrastructure, because you are letting the cloud provider decide where to schedule your function, and you are giving the cloud provider flexibility on when to schedule the function. The developer experience for deploying a serverless function can feel mysterious. You send a blob of code into the cloud. Later on, you send a request to call that code in the cloud. The result of the execution of that code gets sent back down to you. What is happening in between? Rodric Rabbah is the principal researcher and technical lead in serverless computing at IBM. He helped design IBM Cloud Functions, the open source functions-as-a-service platform that IBM has deployed and operationalized as IBM Cloud Functions. Rodric joins the show to explain how to build a platform for functions as a service. When a user deploys a function to IBM Cloud Functions, that function gets stored in a database as a blob of text, waiting to be called. When the user makes a call to the function, IBM Cloud Functions takes it from the database and queues the function in Kafka, and eventually schedules the function onto a container for execution. Once the function has executed, IBM Cloud Functions stores the result in a database and sends that result to the user. When you execute a function, the time spent scheduling it and loading it onto a container is known as the “cold start problem”. The steps of executing a serverless function take time, but the resource savings are significant. Your code is just stored as a blob of text in a database, rather than sitting in memory on a server, waiting to execute. In his research for building IBM Cloud Functions, Rodric wrote about some of the tradeoffs for users who build applications with serverless functions. The tradeoffs exist along what Rodric calls “the serverless trilemma.” In today’s episode, we discuss why people are using functions-as-a-service, the architecture of IBM Cloud Functions, and the unsolved challenges of building a serverless platform. Full disclosure: IBM is a sponsor of Software Engineering Daily. OpenWhisk Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Most user interfaces that we interact with are not animated. We click on a button, and a form blinks into view. We click a link and the page abruptly changes. On the other hand, when we interact with an application that has animations, we can feel the difference. The animations are often subtle. If you aren’t sure what I’m talking about, pay attention the next time you use Slack or Facebook Messenger or iMessage. Airbnb values animation so much that they built Lottie, a library for animation. In an animated application, the user interface feels alive. When a software team takes the time to build animations into small interactions, the user perceives the animations as polish and attention to detail. Sarah Drasner has been evangelizing the value of animations for years, and she is an expert at implementing complex and beautiful animations on the web. She works at Microsoft as a developer advocate and joins the show to talk about how to build animations. If you are building a web application and want to create a unique UI, you might find this show useful. JavaScript supports detailed animations, often through the manipulation of SVG files. SVG stands for “scalable vector graphics” a file format that represents an image in its own DOM. SVG is so flexible because of this DOM format, which defines the different parts of the SVG. This is in contrast to a bitmap, which is just a simple matrix of dots, without any rich metadata. You could manipulate SVG with raw JavaScript—but most people use a frontend JavaScript framework like React, Angular, or VueJS. Sarah has been implementing most of her recent web animations using Vue, and she is a member of the Vue core team. Vue has an entertaining story, because it gained popularity in a time when Google was supporting AngularJS and Facebook was supporting ReactJS. The first version of Vue was created from scratch by a single developer, Evan You. If you are a Vue developer looking for an open source project to hack on, you can check out softwaredaily.com, which is an open source platform to consume content about software. In addition to the Vue web app, we also have the Software Engineering Daily app for iOS and for Android. All of these apps are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Are we a media company or a technology company? Facebook and the New York Times are both asking themselves this question. Facebook originally intended to focus only on building technology–to be a neutral arbiter of information. This has turned out to be impossible. The Facebook newsfeed is defined by algorithms that are only as neutral as the input data. Even if we could agree on a neutral data set to build a neutral newsfeed, the algorithms that generate this news feed are not public, so we have no way to vet their neutrality. Facebook is such a powerful engine for distribution, it has allowed for a rise in the number of publishers who can get their voice heard. As a result, large media companies have lost market share because Facebook has replaced their distribution. The New York Times has always been a media company–but the standards for media consumption have shot up. Millions of people produce content for free, and that content is distributed through high quality experiences like Twitter, YouTube, Medium, and Facebook. When a page takes too long to load on NewYorkTimes.com, it doesn’t matter how good the content is–the user is going to navigate away before they read anything. Today, the New York Times has built out an experienced engineering team. In a previous episode, we reported how the Times uses Kafka to make its old content more accessible. In today’s show, we talk about how the Times uses React and GraphQL to improve the performance and the developer experience of engineers who are building software at the New York Times. Scott Taylor and James Lawrie are software engineers at the New York Times. In this episode, they explain how the New York Times looks at technology. The user experience on New York Times rivals that of a platform company like Facebook, and this is assisted by technologies originally built at Facebook: React, Relay, and GraphQL. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Functions as a service let developers deploy stateless application logic that is cheap and scalable. Functions as a service still have some problems to overcome in the areas state management, function composition, usability, and developer education. Kubernetes is a tool for managing containerized infrastructure. Developers put their apps into containers on Kubernetes, and Kubernetes provides a control plane for deployment, scalability, load balancing, and monitoring. So–all of the things that you would want out of a managed service become much easier when you put applications into Kubernetes. This is why Kubernetes has become so popular–and it is why Kubernetes itself is being offered as a managed service by many cloud providers–including IBM. For the last decade, IBM has been building out its cloud offerings–and for two of those years, Jason McGee has been CTO of IBM Cloud Platform. In this episode, Jason discusses what it is like to build and manage a cloud, from operations to economics to engineering. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Thumbtack is a marketplace for real-world services. On Thumbtack, people get their house painted, their dog walked, and their furniture assembled. With 40,000 daily marketplace transactions, the company handles significant traffic. On yesterday’s episode, we explored how one aspect of Thumbtack’s marketplace recently changed, going from asynchronous matching to synchronous “instant” matching. In this episode, we zoom out to the larger architecture of Thumbtack, and how the company has grown through its adoption of managed services from both AWS and Google Cloud. The word “serverless” has a few definitions. In the context of today’s episode, serverless is all about managed services like Google BigQuery, Google Cloud PubSub, and Amazon ECS. The majority of infrastructure at Thumbtack is built using services that automatically scale up and down. Application deployment, data engineering, queueing, and databases are almost entirely handled by cloud providers. For the most part, Thumbtack is a “serverless” company. And it makes sense–if you are building a high-volume marketplace, you are not in the business of keeping servers running. You are in the business of improving your matching algorithms, your user experience, and your overall architecture. Paying for lots of managed services is more expensive than running virtual machines–but Thumbtack saves money from not having to hire site reliability engineers. Nate Kupp leads the technical infrastructure team, and we met at QCon in San Francisco to talk about how to architect a modern marketplace. This was my third time attending QCon and as always I was impressed by the quality of presentations and conversations I had there. They were also kind enough to set up some dedicated space for podcasters like myself. The most widely used cloud provider is AWS, but more and more companies that come on the show are starting to use some of the managed services from Google. The great news for developers is that integration between these managed services is pretty easy. At Thumbtack, the production infrastructure on AWS serves user requests. The log of transactions that occur get pushed from AWS to Google Cloud, where the data engineering occurs. On Google Cloud, the transaction records are queued in Cloud PubSub, a message queueing service. Those transactions are pulled off the queue and stored in BigQuery, a system for storage and querying of high volumes of data. BigQuery is used as the data lake to pull from when orchestrating machine learning jobs. These machine learning jobs are run in Cloud Dataproc, a managed service that runs Apache Spark. After training a model in Google Cloud, that model is deployed on the AWS side, where it serves user traffic. On the Google Cloud side, the orchestration of these different managed services is done by Apache Airflow, an open source tool that is one of the few pieces of infrastructure that Thumbtack does have to manage themselves on Google Cloud. To find out more about the Thumbtack infrastructure, check out the video of the talk Nate gave at QCon San Francisco, or check out the Thumbtack Engineering Blog. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The labor market is moving online. Taxi drivers are joining Uber and Lyft. Digital freelancers are selling their services through Fiverr. Experienced software contractors are leaving contract agencies to join Gigster. Online labor marketplaces create market efficiency by improving the communications between buyers and sellers. Workers make their own hours, and their performance is judged by customers and algorithms, rather than the skewed perspective of a human manager. These marketplaces for human labor are in different verticals, but they share a common problem: how do you most efficiently match supply and demand? Perfect marketplace matching is an unsolved problem. Hundreds of computer science papers have been written about the problems of stable matching, which often turn out to be NP-Complete. The stock market has been attempting to automate marketplace matching for decades, and inefficiencies are discovered every year. Today’s show is about matching buyers and sellers on Thumbtack, a marketplace for local services. For the first seven years, Thumbtack was building liquidity in its 2-sided market. During those years, the model for job requests was as follows: let’s say I was on Thumbtack looking for someone to paint my house. I would post a job that would say I am looking for house painters. The workers on Thumbtack that paint houses could see my job and place a bid on it. Then I would choose from the bids and get my house painted. This was the “asynchronous” model. The actions of the buyer and seller were not synchronized. There was a significant delay between the time when the buyer posted a job and the time when a seller places a bid, and then another delay before the buyer selects from the sellers. Thumbtack recently moved to an “instant matching” model. After gathering data about the people selling services on the platform, Thumbtack is now able to avoid the asynchronous bidding process. In the new experience, a buyer goes on the platform, requests a house painter, and is instantly matched to someone who has a history of accepting house painting tasks that fit the parameters of the buyer. From the user’s perspective, this is a simple improvement. From Thumbtack’s perspective, there was significant architectural change required. In the asynchronous model, the user requests lined up in a queue, and were matched with pros who placed bids on the items in that queue. In the instant matching model, a user request became more like a search query–the parameters of that request hit an index of pros and returns a response immediately. Xing Chen is an engineer from Thumbtack, and joins the show to describe the rearchitecture process–how Thumbtack went from an asynchronous matching system to synchronous, instant matching. We also explore some of the other architectural themes of Thumbtack, which we dive into in further detail in tomorrow’s episode about scaling Thumbtack’s infrastructure, which uses both AWS and Google Cloud. On Software Engineering Daily, we have explored the software architecture and business models of different labor marketplaces–from Uber to Fiverr. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook serves interactive content to billions of users. Google serves query requests on the world’s biggest search engine. Uber handles a significant percentage of the transportation within the United States. These services are handling radically different types of traffic, but many of the techniques they use to balance loads are similar. Vivek Panyam is an engineer with Uber, and he previously interned at Google and Facebook. In a popular blog post about load balancing at scale, he described how a large company scales up a popular service. The methods for scaling up load balancing are simple, but effective–and they help to illustrate how load balancing works at different layers of the networking stack. Let’s say you have a simple service where a user makes a request, and your service sends them a response with a cat picture. Your service starts to get popular, and begins timing out and failing to send a response to users. When your service starts to get overwhelmed, you can scale up load by creating another service instance that is a copy of your cat picture service. Now you have two service instances, and you can use a layer 7 load balancer to route traffic evenly between those two service instances. You can keep adding service instances as the load scales and have the load distributed among those new instances. Eventually, your L7 load balancer is handling so much traffic itself that you can’t put any more service instances in front of it. So you have to set up another L7 load balancer, and put an L4 load balancer in front of those L7 load balancers. You can scale up that tier of L7 load balancers, each of which is balancing traffic across a set of your service instances. But eventually, even your L4 load balancer gets overwhelmed with requests for cat pictures. You have to set up another tier, this time with L3 load balancing… In this episode, Vivek gives a clear description for how load balancing works. We also review the 7 networking layers before discussing why there are different types of load balancers associated with the different networking layers. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As a system becomes more complex, the chance of failure increases. At a large enough scale, failures are inevitable. Incident response is the practice of preparing for and effectively recovering from these failures. An engineering team can use checklists and runbooks to minimize failures. They can put a plan in place for responding to failures. And they can use the process of post mortems to reflect on a failure and take full advantage of the lessons of that failure. Emil Storlarsky is a production engineer at Shopify where his role shares many similarities with that of Google’s site reliability engineers. In this episode, Emil argues that the academic study of emergency management and industries such as aerospace and transportation have a lot to teach software engineers about responding to production problems. In this interview with guest host Adam Bell, Emil argues that we need to move beyond tribal knowledge and incorporate practices such as an incident command system and rigorous use of checklists. Emil suggests that we need to move beyond a mindset of “move fast and break things” and toward a place of more deliberate preparation. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Incident Response Insights Talk The Human Side Of Post Mortems Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

There is a quote from Jeff Bezos: “70% of the work of building a business today is undifferentiated heavy lifting. Only 30% is creative work. Things will be more exciting when those numbers are inverted.” That quote is from 2006, before Amazon Web Services had built most of their managed services. In 2006, you had no choice but to manage your own database, data warehouse, and search cluster. If your server crashed in the middle of the night, you had to wake up and fix it. And you had to deal with these engineering problems in addition to building your business. Technology today evolves much faster than in 2006. That is partly because managed cloud services make operating a software company so much smoother. You can build faster, iterate faster, and there are fewer outages. If you are an insurance company or a t-shirt manufacturing company or an online education platform, software engineering is undifferentiated heavy lifting. Your customers are not paying you for your expertise in databases or your ability to configure load balancers. As a business, you should be focused on what the customers are paying you for, and spending the minimal amount of time on rebuilding software that is available as a commodity cloud service. Rich Archbold is the director of engineering at Intercom, a rapidly growing software company that allows for communication between customers and businesses. At Intercom, the engineering teams have adopted a philosophy called Run Less Software. Running less software means reducing choices among engineering teams, and standardizing on technologies wherever possible. When Intercom was in its early days, the systems were more heterogeneous. Different teams could choose whatever relational database they wanted–MySQL or Postgres. They could choose whatever key/value store they were most comfortable with. The downside of all this choice was that engineers who moved from one team to another team might not know how to use the tools at the new team they were moving to. After switching teams, you would have to figure out how to onboard with those new tools, and that onboarding process was time that was not spent on effort that impacted the business. By reducing the number of different choices that engineering teams have, and opting for managed services wherever possible, Intercom ships code at an extremely fast pace with very few outages. In our conversation, Rich contrasts his experience at Intercom with his experiences working at Amazon Web Services and Facebook. Amazon and Facebook were built in a time where there was not a wealth of managed services to choose from, and this discussion was a reminder of how much software engineering has changed because of cloud computing. To learn more about Intercom, you can check out the Inside Intercom podcast. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

If I am building a mobile app to play podcast episodes, and I make a change to the user interface, I want to have manual quality assurance (QA) testers run through tests that I describe to them, to make sure my change did not break anything. QA tests describe high level application functionality. Can the user register and log in? Can the user press the play button and listen to a podcast episode on my app? Unit tests are not good enough, because unit tests only verify the logic and the application state from the point of view of the computer itself. Manual QA tests ensure that the quality of the user experience was not impacted. With so many different device types, operating systems, and browsers, I need my QA test to be executed in all of the different target QA environments. This requires lots of manual testers. If I want manual testing for every deployment I push, that manual testing can get expensive. RainforestQA is a platform for QA testing that turns manual testing into automated testing. The manual test procedures are recorded, processed by computer vision, and turned into automated tests. RainforestQA hires human workers from Amazon Mechanical Turk to execute the well-defined manual tests, and the recorded manual procedure is used to train the machines that can execute the same task in the future. Russell Smith is the CTO and co-founder of RainforestQA, and he joins the show to explain how RainforestQA works: the engineering infrastructure, the process of recruiting workers from mechanical turk, and the machine learning system for taking manual tasks and automating them. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A popular software application serves billions of user requests. These requests could be for many different things. These requests need to be routed to the correct destination, load balanced across different instances of a service, and queued for processing. Processing a request might require generating a detailed response to the user, or making a write to a database, or the creation of a new file on a file system. As a software product grows in popularity, it will need to scale these different parts of infrastructure at different rates. You many not need to grow your database cluster at the same pace that you grow the number of load balancers at the front of your infrastructure. Your users might start making 70% of their requests to one specific part of your application, and you might need to scale up the services that power that portion of the infrastructure. Today’s episode is a case study of a high-volume application: a monitoring platform called Raygun. Raygun’s software runs on client applications and delivers monitoring data and crash reports back to Raygun’s servers. If I have a podcast player application on my iPhone that runs the Raygun software, and that application crashes, Raygun takes a snapshot of the system state and reports that information along with the exception, so that the developer of that podcast player application can see the full picture of what was going on in the user’s device, along with the exception that triggered the application crash. Throughout the day, applications all around the world are crashing and sending requests to Rayguns servers. Even when crashes are not occurring, Raygun is receiving monitoring and health data from those applications. Raygun’s infrastructure routes those different types of requests to different services, queues them up, and writes the data to multiple storage layers–ElasticSearch, a relational SQL database, and a custom file server built on top of S3. John-Daniel Trask is the CEO of Raygun and he joins the show to describe the end-to-end architecture of Raygun’s request processing and storage system. We also explore specific refactoring changes that were made to save costs at the worker layer of the architecture. This is useful memory management strategy for anyone working in a garbage collected language. If you would like to see diagrams that explain the architecture and other technical decisions, the show notes have a video that explains what we talk about in this show. Full disclosure: Raygun is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As the gig economy grows, that growth necessitates innovations in the online infrastructure powering these new labor markets. In our previous episodes about Uber, we explored the systems that balance server load and gather geospacial data. In our coverage of Lyft, we studied Envoy, the service proxy that standardizes communications and load balancing among services. In shows about Airbnb, we talked about the data engineering pipeline that powers economic calculations, user studies, and everything else that requires a MapReduce. In today’s episode, we explore the business and engineering behind another online labor platform: Fiverr. Fiverr is a marketplace for digital services. On Fiverr, I have purchased podcast editing, logo creation, music lyrics, videos, and sales leads. I have found people who will work for cheap, and quickly finish a job to my exact specification. I have discovered visual artists who worked with me to craft a music video for a song I wrote. Workers on Fiverr post “gigs”–jobs that they can perform. Most of the workers on Fiverr specialize in knowledge work, like proofreading or gathering sales leads. The workers are all over the world. I have worked with people from Germany, the Philippines, and Africa through Fiverr. Fiverr has become the leader in digital freelancing. The staggering growth of Fiverr’s marketplace has put the company in a position similar to an early Amazon. There is room for strategic expansion, but there is also an urgency to improve the infrastructure and secure the market lead. Gil Scheinfeld is the CTO at Fiverr, and he joins the show to explain how the teams at Fiverr are organized to fulfill the two goals of strategic, creative growth and continuous improvement to the platform. One engineering topic we discussed at length was event sourcing. Event sourcing is a pattern for modeling each change to your application as an event. Each event is placed on a pub/sub messaging queue, and made available to the different systems within your company. Event sourcing creates a centralized place to listen to all of the changes that are occurring within your company. For example, you might be working on a service that allows a customer to make a payment to a worker. The payment becomes an event. Several different systems might want to listen for that event. Fiverr needs to call out to a credit card processing system. Fiverr also needs to send an email to the worker, to let them know they have been paid. Fiverr ALSO needs to update internal accounting records. Event sourcing is useful because the creator of the event is decoupled from all of the downstream consumers. As the platform engineering team works to build out event sourcing, communications between different service owners will become more efficient. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In an event driven application, each component of application logic emits events, which other parts of the application respond to. We have examined this pattern in previous shows that focus on pub/sub messaging, event sourcing, and CQRS. In today’s show, we examine the intersection of event driven architecture and serverless architecture. Serverless applications can be built by combining functions-as-a-service (like AWS Lambda) together with backend as a service tools like DynamoDB and Auth0. Functions-as-a-service give you cheap, flexible, scalable compute. Backend as a service tools give you robust, fault-tolerant tools for managing state. By combining these sets of tools, we can build applications without thinking about specific servers that are managing large portions of our application logic. This is great–because managing servers and doing load balancing and scaling is painful. With this shift in architecture, we also have to change how data flows through our applications. Danilo Poccia is the author of AWS Lambda In Action, a book about building event-driven serverless applications. In today’s episode, Danilo and I discuss the connection between serverless architecture and event driven architecture. We start by reviewing the evolution of the runtime unit–from physical machines to virtual machines to containers to functions as a service. Then, we dive into what it means for an application to be “event driven.” We explore how to architect and scale a serverless architecture, and we finish by discussing the future of serverless–how IoT and edge computing and on-premise architectures will take advantage of this new technology. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Large-scale data analysis was pioneered by Google, with the MapReduce paper. Since then, Google’s approach to analytics has evolved rapidly, marked by papers such as Dataflow and Dremel. Dremel combined a column-oriented, distributed file system with a novel way of processing queries. A single Dremel query is distributed into a tree of servers, starting with the root server, splitting into the intermediate servers, and ending with the leaf servers talking to the file system. Once the data is pulled from the file system into the leaves, the data propagates back to the root server, and is shuffled along the way so that the root server receives a sorted response. When Google started turning its internal services into customer-facing cloud products, the effort to productize Dremel began, and BigQuery was born. Jordan Tigani is an engineering lead who works on BigQuery, and he joins the show to discuss the evolution of the data warehouse. Large scale distributed queries still can take a long time–but queries get faster every year. Queries that required a nightly Hadoop job 10 years ago can be viewed in a frequently updated user-facing dashboard. Power users of BigQuery talk about the speed and the query interface as being two of its most valuable differentiating features. As the job of a large scale data analyst becomes less technically intensive, tools like BigQuery will continue to rise in popularity. We have done some great shows about Google papers like Spanner, Dremel, and Dataflow. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Shout out to today’s featured contributor Shreyans Sheth. Shreyans has worked on the Software Engineering Daily search API, and has also helped us understand open source best practices, which we are still learning. Thanks again Shreyans for your work. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Justin Kan has been building startups for a decade, and in that time he has interacted with lots of lawyers. From incorporation to fundraising to selling his company Twitch, the interactions with lawyers consistently seemed less transparent and less efficient than would be optimal. For an engineer like Justin, the natural inclination here was to build software and sell it to lawyers. But there would be so much resistance–you would have to convince the lawyers to change their pricing model to fixed-pricing, which would give them the incentive to buy software and work more efficiently. Instead, Justin teamed up with a few entrepreneurial lawyers who were willing to start a new law firm from scratch, and use software on day 1. The software company is called Atrium Legal Technology Services (or Atrium LTS for short), and the law firm that uses the software is Atrium LLP. Both of these companies are very new, and were publicly announced a few months ago. The two companies work side-by-side in undecorated office in downtown San Francisco. When I took the elevator up to see the company, the elevator doors opened and revealed two paper signs pointing to opposite ends of the office. On the Atrium LTS side of the office, engineers were writing software to extract the meaning from documents. Today, lawyers at old law firms are paid hundreds of dollars an hour to fill in document templates by editing a text document. As the Atrium LTS software gets better, document preparation will be done through web applications, with the variable names disambiguated from the parts of the document that never change from client to client. On the other side of the office sat Atrium LLP. The legal team was dressed a little more formally than their engineer counterparts, but there was nothing close to the formality of a traditional Silicon Valley law firm. Far from the decor of a Menlo Park law firm, the office space was actually more spartan than most well-funded startups, signaling to the employees that this is an unproven business strategy, and there is a ton of work to be done to validate it. This sentiment was echoed in my conversation with Justin. It’s possible (even plausible) that Atrium LLP could become the biggest law firm in the world, but the road to getting there will take patience and steady execution. I enjoyed hearing Justin explain the motivation for starting Atrium LTS, and look forward to covering the company in the future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

An engineer who wants to start a business using investment capital needs to understand the expectations of investors. The market for the business needs to be huge. The team needs to have a differentiated understanding of the market, or a differentiated product. The CEO needs to have the determination to continue operating the company even when it gets very difficult. And the price needs to be right for the investor. Even if you are just working at a startup, or considering joining a startup, you must understand how the investment market works. From a raw financial standpoint, it only makes sense to spend your time at a startup that has equity with a high expected value. Your equity will only have high expected value if the company continues to exist long enough to have an exit–the company must either go public or get acquired. In order to make it down the long and winding road to an exit, a technology company often needs to raise money on multiple occasions. That money is used to pay employees like you! If the company can’t earn enough revenues or raise money, you are going to get fired. Then, you may not have the spare cash to execute your stock options, and you might lose the rights to the equity that you worked so hard for. The best way to avoid this is to learn to think like an investor–because as an engineer working for equity, you are an investor. Semil Shah is an early stage seed investor with Haystack, a fund that he started. He also works with GGV Capital, a venture firm investing out of the United States and China. Semil has been blogging about technology for many years, and eventually evolved from a commentator to an investor. In this episode, we explore the dynamics between investors and founders of early-stage technology companies. We also explore the strange market of podcasting. Semil worked at a company called Concept.io, which was acquired by Apple for $30M. We have done some great shows with other engineering investors like Chris Dixon and Adrian Colyer. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cryptocurrencies give us a decentralized financial system. OpenBazaar is a decentralized commerce system. A merchant can log onto OpenBazaar and post a listing for an item–for example, a t-shirt that I want to sell for $15. My item listing will spread throughout the OpenBazaar P2P network. A shopper can download the OpenBazaar desktop application and see my listing for a t-shirt. The shopper can pay me $15 in bitcoin, and I will send the t-shirt to their address. If I were selling that shirt on Amazon, the corporation would take a cut of that transaction. OpenBazaar has no transaction costs–so users get to save some money. However, users also miss out on the benefits of a corporate marketplace. Amazon makes sure that the seller will send the item to the buyer, and makes sure that the buyer pays the seller. On OpenBazaar, an escrow system is needed to place money in the hands of a neutral third party until the goods are delivered. Amazon ensures that the distributor sends the item to the customer. On OpenBazaar, users need to figure out how to send the goods to each other. Brian Hoffman was the first developer to start working on OpenBazaar. The project has grown significantly since his initial commit, and OpenBazaar now has buyers, sellers, and open source committers. There is a clear desire for an open system of commerce. Brian is also the CEO of OB1, a company that provides services on top of OpenBazaar. OpenBazaar is a protocol–and other companies will undoubtedly emerge to build on top of it as well. In our conversation, Brian discussed how OpenBazaar works–the peer-to-peer protocol, the escrow system, the dispute resolution, and the open source community management. It is a fascinating, unique project, and I hope you learn something about it from this episode. To find all of our old episodes about decentralized technology and blockchains, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Shout out to today’s featured open source contributor Justin Lam. He has been working on improving the iOS codebase, and I know all the SE Daily mobile users appreciate his effort. Thanks Justin! Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Netflix API is accessed by developers who build for over 1000 device types: TVs, smartphontes, VR headsets, laptops. If it has a screen, it can probably run Netflix. On each of these different devices, the Netflix experience is different. Different screen sizes mean there is variable space to display the content. When you open up Netflix, you want to efficiently browse through movies. The frontend engineers who are building different experiences for different device types need to make different requests to the backend to fetch the right amount of data. This was the engineering problem that Vasanth Asokan and his team at Netflix was tasked with solving: how do you enable lots of different frontend engineers to get whatever they need from the backend? This problem led to the development of a “serverless-like platform” within Netflix, which Vasanth wrote about in a few popular articles on Medium. This platform enables frontend developers to write and deploy backend scripts to fetch data, decoupling the responsibilities of frontend engineers and backend engineers. The tight coupling of frontend and backend engineering was problematic to the development velocity of Netflix. We have done many shows about Netflix engineering, covering topics like data engineering, user interface design, and performance monitoring. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Serverless architecture is software that runs without an addressable server. Serverless is made possible by two types of technology: platform as a service providers like Auth0, and functions as a service like AWS Lambda. With both of these technologies, we can program logic that runs without being deployed to a server. Functions as a service are cheap and scalable. Write your code for a Serverless function, and the cloud provider will cheaply deploy and execute that function on some server somewhere. The difficult part is maintaining state. Since Serverless compute instances are ephemeral, you aren’t dealing with a system that will keep track of your state—it is going to disappear eventually. The ephemeral nature of Serverless code requires us to shift our thinking—but the dramatic cost and simplified scalability make it well worth the effort. Serverless functions can add complexity in exchange for lower price. Serverless “platform as a service” often lowers complexity at a slightly higher price. A Serverless database like Firebase handles database scaling and gives you a nice web interface. A Serverless machine learning platform like Google CloudML gives your models scalability and controlled deployment. A Serverless authentication service like Auth0 manages your authentication. In addition to authentication, Auth0 has built a set of tools to allow SaaS companies to extend their platforms into a sandboxed code execution environment. Bobby Johnson is an engineer at Auth0, and he joins the show to describe the toolbox that Auth0 has developed: authentication, webtasks, and extensibility–and how the world of “serverless” architecture is evolving. Full disclosure, Auth0 is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In 2003, Paul Martino co-founded Tribe.net, one of the earliest social networking sites. Tribe had significant traction, with hundreds of thousands of users. In the early 2000s, hundreds of thousands of users was enough traffic to pose a company with engineering challenges. Paul had studied computer science, and was able to use his knowledge of high-performance computing to write an efficient graph database, and solve the other technical puzzles that the company faced–but the business did not ultimately work out. The failure of Tribe made the founders even hungrier for success–and it taught them lessons that they carried into subsequent businesses. Paul went on to start Aggregate Knowledge, a marketing technology company that sold for $119 million. His Tribe co-founder Mark Pincus went on to start Zynga, the multi-billion dollar gaming company. Another Tribe employee co-founded Yammer, which sold to Microsoft for a billion dollars. Since his exit from Aggregate Knowledge, Paul Martino started Bullpen Capital, which makes post-seed investments. The Bullpen Capital portfolio is appealing to me–partly because of the number of Internet gambling companies. Paul and I talked about gambling and other taboo business sectors–as well as what makes a good investment in the “post-seed” category. I enjoyed speaking to Paul because he has a straightforward, no-nonsense way of talking about things–it’s very charismatic and uncommon. We have done some great shows with other engineering investors like Chris Dixon and Adrian Colyer. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Shout out to today’s featured contributor Kurian Vithayathil. He has made significant contributions to the Software Engineering Daily Android app. Thanks again Kurian for your work. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Bob Hoffman’s long career in advertising included CEO positions at three different agencies. He helped huge brands craft their messaging, and grab consumer attention. In Bob’s world of advertising, lots of money was spent on creativity. Were the campaigns successful? That depends who you ask. In the old world of advertising, everyone acknowledged that success was subjective. If you could imagine the opposite of what Bob Hoffman built his career doing, it might look like search advertising. Bob’s campaigns were about creating a brand’s voice, with colorful art and subtlety and ambient messaging. Advertising was about turning a brand into an entity you recognize, teaching the consumer to associate Nike with fitness, or Dove soap with clean hands, or Cheetos with cheesy, salty attitude. Search advertising, on the other hand, is just text. You enter a search query, you are looking for black socks, and the top link that comes back is a line of text that says “cheap black socks.” Search advertising catches people who have an intent to do something. They have stated their intent by typing into a box. With search advertising, a brand might not even need a sexy, flashy creative. As money poured into adtech, user tracking, and Google, brands started to care more about metrics. When Bob met with a brand, the brand wouldn’t be asking about the cool new advertising campaign featuring a young actress drinking a Coca Cola. The brand would be asking about the click-through rate of a display advertising campaign. Brands moved their focus to statistics, and away from creativity. And technology companies were happy to provide them with statistics. Whether those statistics were true or not is another story altogether. The industry was moving from creative BS to outright lying, and Bob decided to leave. In today’s episode, Bob explains how the state of advertising became so problematic, and the ways in which it harms us Internet users. We have done lots of reporting about advertising fraud for the last year, and it is a popular topic because people are often shocked to find that online advertising is inextricably linked to organized crime, surveillance, and Twitter botnets. That’s not to say that online advertising doesn’t work–it certainly does! But understanding the dark underbelly of the Internet’s cash cow is a necessary precondition to finding a solution. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Augmented reality applications are slowly making their way into the world of the consumer. Pokemon Go created the magical experience of seeing Pokemon superimposed upon the real world. IKEA’s mobile app lets you see how a couch would fit into your living room, which has a significant improvement on the furniture buying process. Augmented reality applications can have even more dramatic impact on industrial enterprises. Have you ever set up a factory? You might need to build a conveyor belt. You might need to put together the parts of a giant machine that extrudes steel. You might need to fix a silicon wafer fabrication machine. It takes an expert to set up these heavy, complicated machines. ScopeAR is a company that builds augmented reality tools. One of the ScopeAR products allows users to telepresence with each other to collaborate on the construction and maintenance of heavy machinery. Imagine I am setting up my factory, and I have a complicated piece of machinery (let’s say a conveyor belt) in front of me. I have never constructed a conveyor belt before. I put on a HoloLens, and set up a VoIP call with an expert who has experience with that piece of machinery, and they point out what I need to do by superimposing 3-D arrows, text, and other instructions on my field of vision. They can share my experience and help guide me through the process. This is such a flexible tool–you can imagine applications for augmented reality assistance being useful in medicine, construction, education and other fields. Scott Montgomerie is the CEO of ScopeAR and in today’s episode, we talk about the state of AR, how the AR tools from Apple and Google compare, and how the similarity between tools used for mapping the world in AR relate to the tools used to map the world by autonomous cars. Scott was a great guest, and I hope to have him back on in the future. We have done some great shows about how to build augmented reality and virtual reality applications. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Shout out to today’s featured contributor Edgar Pino. He is working on a real-time chat application for Software Engineering Daily, so that we can have chat rooms for people to discuss the episodes easily. Innovative work! Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Computational load is the amount of demand that is being placed on a computer system. “Load” can take the form of memory, CPU, network bandwidth, disk space, and other finite resources. When we design systems, we need to prepare for high-load events. On a social network, people are much more active in the mornings. On an e-commerce site, Black Friday causes many more users to come online for discount shopping. Our distributed application must be able to scale in response to these spikes in traffic. Cloud computing has changed the popular software architecture patterns, and load balancing has changed along with it. With on-demand, infinite infrastructure, we don’t need to worry about ordering servers and provisioning. With infrastructure as code, it becomes simpler to manage lots of deployable units–so we can break up our monolith into microservices, and have hundreds or thousands of virtual machines or containers running. Enterprises that were started before cloud computing have large on-premise server deployments–but today, many of them also use the cloud. The cloud can be used to augment their classic on-prem deployments with cloud platform-as-a-service features. The cloud can also be used as a reliable way to scale during high load events. Today, a common architectural pattern is to have your application broken up into services. Each of those services has multiple instances. When the load on a particular service is under lots of demand, you create more instances to handle the increased load. How do you monitor the load on each service? How do you know when to spin up new instances of the service? Load analysis and load balancing across different services can be implemented by placing “agents” throughout your infrastructure. These agents gather data about services and service instances, and route that data to a centralized place. The centralized “control plane” can be used to make decisions about load-balancing and traffic routing. Ranga Rajagopalan worked on networking at Cisco for a decade before co-founding Avi Networks as CTO. Avi Networks builds modern load balancing software, and in today’s episode, Ranga describes the requirements of load balancing. We talked about the evolution of network infrastructure, the impact of the cloud, and the technical decisions that his team has made when architecting Avi Networks. Full disclosure: Avi Networks is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The New York Times is a newspaper that evolved into a digital publication. Across its 166 year history, The Times has been known for longform journalistic quality, in addition to its ability to quickly churn out news stories. Some content on the New York Times is old but timeless “evergreen” content. Readers of the New York Times website are not only looking for the most recent news–they want to know what the headlines were the day after Pearl Harbor. They want to read editorials about Martin Luther King. Over the last 30 years, New York Times has moved itself online, bringing old material with it. Since the 90s, several different content management systems (CMS) have been used by journalists within The Times. These different sources of content store data in different formats. This is a data management problem. Users want to search over the entire history of articles published by The Times, which means that The Times needs to unify those articles in a single index. These are articles from the 1920s that were digitized using OCR, articles from 1998 that were written on a legacy CMS, and articles from 2017 that use the latest CMS. Boerge Svingen is the director of engineering at NYT, and he wrote about this problem and its solution on Medium. This story describes the flexibility of Kafka; in contrast to the applications of Kafka as a place to buffer high volumes of data, the New York Times uses Kafka as a place to unify data and allow for other specific materialized views to be built on top of it. We have covered Kafka in the past with interviews of some of its creators–including Jay Kreps and Neha Narkhede. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A cryptocurrency has a distributed ledger called a blockchain. The blockchain keeps track of every transaction that occurs across the cryptocurrency. This blockchain must stay up-to-date and verified–which requires someone in the network to do that validation. Bitcoin and Ethereum use the proof-of-work algorithm. Miners do computational work to validate the legitimacy of transactions across the network, and in return they are given cryptocurrency as a reward for that computational work. In the future, cryptocurrencies could move towards a proof-of-stake model. If you own a significant amount of cryptocurrency, you have incentive to keep the validity of the blockchain up to date. Proof-of-stake algorithms can be significantly less energy intensive. Vlad Zamfir is a researcher for the Ethereum Foundation, and he joins Haseeb Qureshi for a conversation about cryptoeconomics. This is an in-depth conversation between two active blockchain developers. We hope you enjoy it. You can send us feedback on the show by emailing me jeff@softwareengineeringdaily.com or joining us on the Slack channel at softwareengineeringdaily.com/slack. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In America, the tech companies we focus on are commonly known as FAANG: Facebook, Amazon, Apple, Netflix, Google. We all know what these companies do because they impact our daily lives. In Asia, there are three giant tech companies that have similar scale: Baidu, Alibaba, and Tencent, otherwise known as BAT. Technology within a location is shaped by the pressures of that location. You might think we live in a global society, but tech in Asia is dramatically different than it is in America. Differences in culture lead to differences in product development. In China, a different political system contributed to more rapid adoption of online payments. Because there is more payment data, people can be given loans more efficiently. Less of the population is “unbanked.” Online payments are mostly handled by WeChat, a social networking product from Tencent, and Alibaba, an ecommerce giant. If you live in the West, imagine that Facebook and Amazon handled most of your payments for everything. You would have a different relationship with those companies. Bernard Leong is the host of Analyse Asia, a podcast about Asian developments in technology in business. After studying materials science in Singapore and theoretical physics at Cambridge, he made his way into business and journalism, and developed an interest in the Singularity–a subject that few people took seriously until recently (one topic we explored in this show is Masayoshi Son, the Japanese tycoon who wants to invest nearly a trillion dollars into technology companies; Masayoshi believes firmly that the Singularity is coming). Shenzhen: The Silicon Valley of Hardware (Full Documentary) | Future Cities | WIRED In the Plex by Steven Levy The Hidden Forces Behind Toutiao: China’s Content King Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

It’s 9pm at night, and you are hungry. You order a pizza from Domino’s. You live on a street that’s dark, and so you have installed a smart lightbulb in front of your mailbox that lights up the address. When the pizza at Domino’s is ready, you want the lightbulb on your mailbox to light up so that the delivery person can read your address when they arrive in front of your house with the pizza. The Internet should make it possible to have this kind of event-driven, connected world. Anything that is connected to the Internet should be able to send signals to anything else on the Internet, so that our lives gradually become more automated. This is what IFTTT does. Users of IFTTT can easily create applets to wire different services together. You can use IFTTT to trigger an email whenever three of your friends retweet something on Twitter. You can use IFTTT to flash the lights in your house when Bitcoin hits new market highs. You can use IFTTT to order a pizza whenever Bitcoin crashes. IFTTT makes it easy to connect different services together, and a lot of work goes into the infrastructure that enables these billions of events to process correctly. Nicky Leach from IFTTT’s engineering team joins the show to describe how IFTTT allows for integrations between services that were not built to integrate–and he talks about the scheduling, data engineering, and monitoring of the company’s software stack. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In 2015, eleven years had passed since MapReduce was first published, and companies were still having data problems. Tomer started working on Dremio, a company that was in stealth for another two years. I interviewed Tomer two years ago, when he still could not say much about what Dremio was doing. We talked about Apache Drill, an open-source project related to what Dremio eventually built. Earlier this year, two of Tomer’s colleagues Jacques Nadeau and Julien Le Dem came on to discuss columnar data storage and interoperability. What I took away from that conversation was that today, data within an average enterprise is accessible, but the different formats are a problem. Some data is in MySQL, some is in Amazon S3, some is in ElasticSearch, some is on HDFS stored in Parquet files. Different teams will set up different BI tools and charts that read from a specific silo of data. At the lowest level, the different data formats are incompatible–you have to transform MySQL data in order to merge it with S3 data. On top of that, engineers doing data science work are using Spark, Pandas, and other tools that pull lots of data into memory–if the in-memory formats are not compatible, the data teams can’t get the most out of their work. On top of THAT, at the highest level, data analysts are working with different data analysis tools, so there is even more siloing. Now I understand why Dremio took two years to bring to market. They are trying to solve data interoperability by making it easy to transform data sets between different formats. They are trying to solve data access speed by creating a sophisticated caching system. And they are trying to improve the effectiveness of the data analysts by providing the right abstractions for someone who is not a software engineer to study the different data sets across an organization. Dremio is an exciting project because it is rare to see a pure software company put so many years into up-front stealth product development. After talking to Tomer in this conversation, I’m looking forward to seeing Dremio come to market. It was fascinating to hear him talk about how data engineering has evolved to today. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Public key encryption allows for encrypted, private messages. A message sent from Bob to Alice gets encrypted using Alice’s public key. Public key encryption also allows for signed messages–so that when Alice signs a message, Alice uses her private key and Bob can verify it if Bob has her public key. In both cases, Bob needs Alice’s public key! If Bob gets that public key from an email message, Bob is trusting that the email message is secure–and if Bob can’t ever verify that first message containing the key, he has no way to verify the messages that come after it. This is the problem of key distribution. Key distribution undermines the usability of PGP encryption. Serious encryption advocates will sometimes meet in person to exchange pieces of paper containing public keys. Keybase is a company that attempts to solve the problem of key distribution by having users connect social media accounts and devices to Keybase, in order to collectively verify who you are, and then give you the power to share your public key. Max Krohn is a founder of Keybase, and was previously a founder of SparkNotes and OKCupid. Max was on the show a few years ago to discuss the basics of Keybase, and in this episode he explores some of the abstractions that Keybase has built on top of its core identity tool–Keybase File System, Keybase Teams, and Keybase Git. We do break down the basics of Keybase, but if you want a more thorough explanation, you might like to check out that older episode, you can download the Software Engineering Daily app on iOS or Android to find all of our old episodes. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Computer chips have physical limitations. When transistors get too small, electrons start to behave in ways that make the hardware modules less reliable. Our reliable technological progress has been enabled by Moore’s Law: the idea that the number of components we can fit on a chip doubles roughly every 12-18 months. We can’t keep shrinking the size of these components, because physics is no longer complying. Quantum computing allows us to operate on qubits rather than bits, giving us better parallelism and continued reliable technological progress. Quantum computing is still mostly an area of research rather than production systems–but it is rapidly approaching usability, and Zlatko Minev joins the show to explain how quantum computing works, and why software engineers should care. Zlatko is a PhD candidate at the Yale Quantum Information Lab. Today he describes how qubits work, which algorithms quantum computing impacts, and which parts of modern computer architecture will work on a quantum computer. We may have to throw out the Von Neumann architecture when it comes to quantum! Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A smart contract is a program that allows for financial transactions. Smart contracts are usually associated with the Ethereum platform, which has a language called Solidity that makes it easy to program smart contracts. Someday, we will have smart contracts issuing insurance, processing legal claims, and executing accounting transactions. Smart contracts involve money, and they are likely to transact with cryptocurrencies. That makes them ripe targets for attackers. What are the vulnerabilities of smart contracts? What can we do to ensure the safety of a high throughput, automated financial system? In today’s episode, Haseeb Qureshi talks to Emin Gün Sirer, a professor at Cornell University where he is co-director of the Initiative for Cryptocurrencies and Contracts. They discuss how smart contracts work and how to secure them. Haseeb and Emin are both working full-time on cryptocurrencies, which makes for a detailed technical discussion. In our previous episode about the DAO hack, Emin Gün Sirer was one of the protagonists of the story. You can find that episode as well as all of our old episodes by downloading the Software Engineering Daily app for iOS and for Android. We also have several other episodes with Haseeb. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Interviewing engineers is not a solved problem. Quite the opposite–everyone in the software industry will tell you their own personal issues with the hiring process. One reason that technical interviews have not evolved significantly is the lack of standardized tooling. Some companies give you one phone screen, some give you two. Some companies have you solve brain teasers (“how many golf balls fit in a school bus”) and some make you fix bugs in their production codebase. During the on-site interview, some companies use whiteboards, some let you use a laptop. Software companies do so much–they should be outsourcing the things that are not their core competency. Certainly they cannot outsource the entire hiring process–but they can outsource parts of it to a company like Interviewing.io. Engineers come to Interviewing.io to practice their interview skills, where other engineers from top companies practice with them as an interviewer. When an engineer has practiced interviewing enough, they can use Interviewing.io to interview for real with real companies and find a job. Aline Lerner is the CEO of Interviewing.io, and she knows about the software interviewing and recruiting process as much as anyone. After working as an engineer, she started studying recruiting, consulting with top companies to help them improve their process. From her observations, she created Interviewing.io. In this episode, we dissect the workflow that she created for engineers to improve at interviewing and find jobs, and also explore the insights that led her to starting Interviewing.io. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning models can be built by plotting points in space and optimizing a function based off of those points. For example, I can plot every person in the United States in a 3 dimensional space: age, geographic location, and yearly salary. Then I can draw a function that minimizes the distance between my function and each of those data points. Once I define that function, you can give me your age and a geographic location, and I can predict your salary. Plotting these points in space is called embedding. By embedding a rich data set, and then experimenting with different functions, we can build a model that makes predictions based on those data sets. Yufeng Guo is a developer advocate at Google working on CloudML. In this show, we described two separate examples for preparing data, embedding the data points, and iterating on the function in order to train the model. In a future episode, Yufeng will discuss CloudML and more advanced concepts of machine learning. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

How would you build a system for indexing and monitoring the entire Internet? Start by breaking the Internet up into IP address ranges. Give each of those address ranges to servers distributed around the world. On each of those servers, iterate through your list of IP addresses, sending packets to them. Depending on what sorts of packets those IP addresses respond to, and what those responses are, you can build a map of the devices on the Internet: what is running on those devices, and what they respond to. Qadium is a company that indexes and monitors devices on the Internet, to help organizations understand the devices that are within corporate networks. If you are a large corporation, Qadium can probably do a better job of figuring out your Internet footprint than you can. Matt Kraning is the CTO of Qadium, and in today’s show he describes the process by which Qadium maps the Internet. Matt used to work on data infrastructure at DARPA, and has deployed Hadoop in Afghanistan–so the infrastructure of Qadium seems relatively manageable. Our data conversations in this episode spam from talking about Storm and Hadoop to Google BigQuery, BigTable, and DataFlow. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Scala is a functional and object oriented programming language built on the JVM. Scala Native takes this language, loved by many, and brings it to bare metal. Scala Native is an optimizing ahead-of-time compiler and lightweight managed runtime designed specifically for Scala. Denys Shabalin (dennis shuh-blin) is a Research Assistant at the EPFL and the primary creator of Scala Native. In this episode, Adam Bell interviews Denys about the motivations behind the Scala Native project, how it was implemented and future directions. He also briefly touches on how Scala Native made cold compilation times of Scala code twice as fast. If you are interested in functional programming, compiler design, or want to learn some interesting tidbits about garbage collector design and trade offs you will like this episode. The mobile apps are open sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to hack on, we would love to get your help! We are building a new way to consume software engineering content. We have the Android app, the iOS app, a recommendation system, and a web frontend–and more projects are coming soon. If you have ideas for how software engineering media content should be consumed, or if you are interested in contributing code, check out github.com/softwareengineeringdaily, or join our Slack channel (there’s a link on our website)–or send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

You have heard the phrase: every company is becoming a software company. An insurance company is now supposed to turn into a software company that sells insurance. A clothing retailer needs to reinvent itself to be able to build software to manage the production and distribution of its clothing. Software applications provide so much leverage to an organization, it seems smart to develop in-house software teams to build those applications. But does it really make sense? Is there a better alternative? In the 90s outsourcing was a common solution to this problem. If you didn’t have software expertise at your company, you would hire a large consulting firm. These firms would often hire inexperienced offshore developers, and the resulting code quality was not so great. Because of the bad experiences of the first Internet boom, companies became more cautious about outsourcing their engineering work–which led to today, where the standard is to hire your own software team. The world has changed in ways that have made outsourcing a more viable solution. Programming best practices are more widely understood. There is an international community of software engineers that share information on places like Stack Overflow, Quora, and Twitter. Off-the-shelf collaboration tools make it much easier to communicate the requirements of a project to a team of developers. Gigster is a company that is working to optimize the engineering of software projects. Large enterprises come to Gigster to build new projects from scratch–whether that project is a marketplace, a mobile application, or a machine learning model. Roger Dickey is the CEO of Gigster, and he joins the show to describe how Gigster works, and why it often makes sense for companies to focus on their core competency and outsource software engineering. Some of our most popular episodes of Software Engineering Daily describe how leading software companies are being built–we have covered Giphy, Netflix, Digital Ocean, Stripe, and many others. Download the Software Engineering Daily app for iOS or Android to hear all of our old episodes. They are easily organized by category, and as you listen, the SE Daily app gets smarter, and recommends you content based on the episodes you are hearing. If you don’t like this episode, you can easily find something more interesting by using the recommendation system. The mobile apps are open sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to hack on, we would love to get your help! We are building a new way to consume software engineering content. We have the Android app, the iOS app, a recommendation system, and a web frontend–and more projects are coming soon. If you have ideas for how software engineering media content should be consumed, or if you are interested in contributing code, check out github.com/softwareengineeringdaily, or join our Slack channel (there’s a link on our website)–or send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A blockchain is a data structure that provides decentralized, peer-to-peer data distribution. Bitcoin is the most well-known blockchain, but in the next decade we will see many more blockchains. Most listeners probably know that you could just fork the code of Bitcoin to start your own blockchain–but wouldn’t it be nice to know how to build a blockchain from scratch? Daniel van Flymen is the author of the Medium article Learn Blockchains by Building One. In his post, he walks you through how to write the code for a blockchain–just like any other web app. He starts with raw Python code, defines the data structures, and stands up his simple blockchain app on a web server to give a toy example for how nodes in a blockchain communicate. For me, this was a great article to read. I have reported on blockchains for over a year, but had not seen such a clear example with executable, simplified code. Stay tuned at the end of the episode for Jeff Meyerson’s tip about making the most of a new job: brought to you by Indeed Prime. To find all of our coverage of cryptocurrencies, download the Software Engineering Daily app for iOS or Android to hear all of our old episodes. They are easily organized by category, and as you listen, the SE Daily app gets smarter, and recommends you content based on the episodes you are hearing. If you don’t like this episode, you can easily find something more interesting by using the recommendation system. The mobile apps are open sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to hack on, we would love to get your help! We are building a new way to consume software engineering content. We have the Android app, the iOS app, a recommendation system, and a web frontend–and more projects are coming soon. If you have ideas for how software engineering media content should be consumed, or if you are interested in contributing code, check out github.com/softwareengineeringdaily, or join our Slack channel (there’s a link on our website)–or send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ethereum is a decentralized transaction-based state machine. Ethereum was designed to make smart contracts more usable for developers. Smart contracts are decentralized programs that usually allow for some a transaction between the owner of the contract and anyone who would want to purchase something from the contract owner. For example, I could set up a smart contract where a listener sends my smart contract some ether and I send the listener a podcast episode automatically. Smart contracts can also interact with each other, to network together complex transactions. In the same way that web development has been made easier by PaaS and SaaS, smart contracts will make building financial systems simple. Preethi Kasireddy is a blockchain developer who writes extensively about cryptocurrencies. She joins the show to describe how the Ethereum platform works, including the steps involved in a smart contract transaction. This episode covers some advanced topics of Ethereum, and if you are out of your comfort zone, don’t worry–you aren’t alone. The mobile apps are open sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to hack on, we would love to get your help! We are building a new way to consume software engineering content. We have the Android app, the iOS app, a recommendation system, and a web frontend–and more projects are coming soon. If you have ideas for how software engineering media content should be consumed, or if you are interested in contributing code, check out github.com/softwareengineeringdaily, or join our Slack channel (there’s a link on our website)–or send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Visa processes 1,600 transactions per second. PayPal processes 193 transactions per second. Bitcoin processes only 3-4 transactions per second. In order to fulfill the dreams of financial programming–in order to get decentralized, peer-to-peer micropayments–Bitcoin needs a much higher transaction throughput. Bitcoin’s scalability issues have led to debates within the community and changes in the software. In this episode, Jordan Clifford gives an overview of some of the scaling limitations of Bitcoin, and discusses SegWit, a change to the Bitcoin protocol that improves scalability. Jordan was previously on the show to discuss the basics of Ethereum and Bitcoin. This episode covers some advanced topics of Bitcoin, and if you are out of your comfort zone, don’t worry–you aren’t alone. Stay tuned at the end of the episode for Jeff Meyerson’s tip about assessing cultural fit at a company: brought to you by Indeed Prime. The mobile apps are open sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to hack on, we would love to get your help! We are building a new way to consume software engineering content. We have the Android app, the iOS app, a recommendation system, and a web frontend–and more projects are coming soon. If you have ideas for how software engineering media content should be consumed, or if you are interested in contributing code, check out github.com/softwareengineeringdaily, or join our Slack channel (there’s a link on our website)–or send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Tinder is a rapidly growing social network for meeting people and dating. In the past few years, Tinder’s userbase has grown rapidly, and the engineering team has scaled to meet the demands of increased popularity. On Tinder, you are presented with a queue of suggested people that you might match with, and you swipe left or right to indicate that you like or dislike them. Creating that queue of suggestions is a complex engineering problem. Many factors go into the suggestions that Tinder gives you: geotargeting, food preferences, your favorite band, your photos, and the people you have swiped on in the past. Bryan Li is an engineering manager at Tinder, and he joins the show to describe the interaction between the mobile client, backend servers, and the offline analytics and machine learning. We also talk about managing different teams and how to reorganize smoothly as a company grows. The mobile apps are open sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to hack on, we would love to get your help! The Software Engineering Daily open source community is building a new way to consume software engineering content. We have the Android app, the iOS app, a recommendation system, and a web frontend. If you are interested in contributing, check out github.com/softwareengineeringdaily–or send me an email: jeff@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Despite all the problems with online advertising, ads are not going away. Advertising is fundamental to the modern Internet economy. In previous episodes of Software Engineering Daily, we have mostly dissected the problems of adtech–bots, tracking, fraud, brand safety. We have talked about some solutions–for example, JavaScript tags that you can put on a page to identify a bot before you serve it an ad. But these solutions don’t get the job done completely, because it isn’t possible to reliably identify bots. Today we explore another solution for adtech: the whitelist. Marc Goldberg is the CEO of Trust Metrics, a company that provides whitelisting for advertisers. A whitelist is a list of domains that are acceptable to run your advertisements on. In order to build a whitelist, you need to review thousands of sites to judge which ones are reasonable places to publish an advertisement. Marc joins the show to describe how to build and scale a system for reviewing websites and judging whether they are safe to run ads against. The mobile apps are open sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to hack on, we would love to get your help! The Software Engineering Daily open source community is building a new way to consume software engineering content. We have the Android app, the iOS app, a recommendation system, and a web frontend. If you are interested in contributing, check out github.com/softwareengineeringdaily–or send me an email: jeff@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Advertising fraud continues to plague the Internet. We do not know the scope and scale of that fraud. How many ads on the Internet are viewed by bots? Estimations range from 2% to 99%. Advertisers are slowly becoming more educated about fraud, thanks in part to Dr. Augustine Fou. Dr. Fou is a full-time advertising fraud researcher. He looks at data sets of billions of ad impressions to figure out how fraud works and help victims of ad fraud make their case. Last year, Dr. Fou came on the show to give an overview of his perspective on the world of ad fraud. Today, we dive into the importance of Twitter in ad fraud schemes. We also talk about the severity of fraud on mobile apps. If you downloaded a flashlight app, or an alarm clock app, or a keyboard, that app could be displaying hidden ads that never actually show up. Stay tuned at the end of the episode for Jeff Meyerson’s tip about getting into a new job: brought to you by Indeed Prime. The mobile apps are open sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to hack on, we would love to get your help! The Software Engineering Daily open source community is building a new way to consume software engineering content. We have the Android app, the iOS app, a recommendation system, and a web frontend. If you are interested in contributing, check out github.com/softwareengineeringdaily–or send me an email: jeff@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A customer engages with a company across a variety of channels–email, Zendesk, Salesforce, online advertising. Unifying those data sources and getting a dashboard into the entire customer experience is the goal of Usermind, a customer engagement hub. If you can get all of that data unified in one place, it creates a tool that salespeople, customer service, and marketing can all look at to see how users are engaging with a company. Michel Feaster is the CEO of Usermind, and she joins the show to describe how Usermind works and the engineering behind the product. To connect all the different APIs from all those other companies makes this a complicated integration problem, and hearing the Usermind strategy for managing integrations will be useful to anyone listening who is building a product with lots of external API integrations. Stay tuned at the end of the episode for Jeff Meyerson’s tip about launching a side business: brought to you by Indeed Prime. If you like listening to this podcast, download the Software Engineering Daily app for iOS and Android to hear all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that. If you don’t like this episode, you can easily find something more interesting by looking at the recommendations in the app. The mobile apps are open sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to hack on, we would love to get your help! The Software Engineering Daily open source community is building a new way to consume software engineering content. We have the Android app, the iOS app, a recommendation system, and a web frontend. If you are interested in contributing, check out github.com/softwareengineeringdaily–or send me an email: jeff@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

42 is tuition-free developer school for students from 18-30. It was started by Xavier Niel, a French billionaire who wanted to encourage a new model of software education. 42 has campuses in France and Silicon Valley. 42 has very high standards for the students it admits, because the students that get in are not paying tuition, but they have 24/7 access to high quality computers and a beautiful campus. Unlike coding bootcamps, 42 lasts 3-5 years. Students who graduate are equipped with both computer science theory and the practical ability to see projects through on their own. Brittany Bir is the chief operating officer of 42 Coding School in Silicon Valley. She joins the show to talk about how 42 works and the future of programming education. At 42, the focus is on peer-to-peer learning. Students complete projects of their own choosing, and are also required to take on internships and get real world work experience. Stay tuned at the end of the episode for Jeff Meyerson’s tip about building your brand brought to you by Indeed Prime. If you like listening to this podcast, download the Software Engineering Daily app for iOS and Android to hear all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that. If you don’t like this episode, you can easily find something more interesting by looking at the recommendations in the app. The mobile apps are open sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to hack on, we would love to get your help! The Software Engineering Daily open source community is building a new way to consume software engineering content. We have the Android app, the iOS app, a recommendation system, and a web frontend. If you are interested in contributing, check out github.com/softwareengineeringdaily–or send me an email: jeff@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

React is a programming model for user interfaces. ReactJS is for building user interfaces for web applications. React Native is for building UI on Android or iOS. ReactVR is for building user interfaces in virtual reality. React Native was originally developed to make it easier to maintain parity between the web, iOS, and Android teams at Facebook. If I build an application for the web with ReactJS, I can rewrite that application for React Native on iOS or Android and reuse some of my code from the web application. It is not a 1-click level of portability between platforms, but it helps share user interface components between different platforms. ReactVR brings React development to virtual reality. Andrew Imm is a ReactVR developer at Facebook, and he joins the show to discuss how ReactVR works. We talk about the support for VR in the browser: WebGL, WebVR, and ThreeJS. We also explore some of the key React components that you might use to build an interface in ReactVR, and we wrap up the show by exploring VR more broadly–how consumers use VR today and how they might use it in the near future. The iOS app is the first project to come out of the Software Engineering Daily Open Source Project. There are more projects on the way, and we are looking for contributors–if you want to help build a better SE Daily experience, check out github.com/softwareengineeringdaily. We are working on an Android app, the iOS app, a recommendation system, and a web frontend. Help us build a new way to consume software engineering content at github.com/softwareengineeringdaily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A basketball game gives off endless amounts of data. Cameras from all angles capture the players making their way around the court, dribbling, passing, and shooting. With computer vision, a computer can build a well-defined understanding for what a sport looks like. With other machine learning techniques, the computer can make predictions by combining historical data with a game that is going on right now. Second Spectrum is a company that builds products for analyzing sports. At major basketball arenas, Second Spectrum cameras sit above the court, recording the game and feeding that information to the cloud. Second Spectrum’s servers crunch on the raw data, processing it through computer vision and putting it into deep learning models. The output can be utilized by teams, coaches, and fans. Yu-Han Chang and Jeff Su are co-founders of Second Spectrum. They join the show to describe the data pipeline of Second Spectrum from the cameras on the basketball court to the entertaining visualizations. After talking to them, I am convinced that machine learning will completely change how sports are played–and will probably open up a platform for new sports to be invented. The iOS app is the first project to come out of the Software Engineering Daily Open Source Project. There are more projects on the way, and we are looking for contributors–if you want to help build a better SE Daily experience, check out github.com/softwareengineeringdaily. We are working on an Android app, the iOS app, a recommendation system, and a web frontend. Help us build a new way to consume software engineering content at github.com/softwareengineeringdaily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

An alert is a signal of problematic application behavior. When something unusual happens to your application, an alert can bring that anomaly to your attention. In order to detect unusual events, you need to define the norm. In order to define both normal and problematic behavior, you need metrics. Metrics are measurements of the behavior in your application. Metrics get created from logs and other data. These high volumes of data get aggregated and collected into easily digestible metrics. This aggregation process that reduces data to metrics is often called a “metrics pipeline.” Clement Pang is the chief architect of Wavefront, a company that builds metrics and alerting software for enterprises such as Box and Lyft. Clement joins the show to discuss how enterprises use alerting and metrics and how to build a company around metrics and alerting–including the data engineering involved in constructing a metrics pipeline. The iOS app is the first project to come out of the Software Engineering Daily Open Source Project. There are more projects on the way, and we are looking for contributors–if you want to help build a better SE Daily experience, check out github.com/softwareengineeringdaily. We are working on an Android app, the iOS app, a recommendation system, and a web frontend. Help us build a new way to consume software engineering content at github.com/softwareengineeringdaily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Playing a video on the Internet seems simple. You press play, the video gets delivered, and boom–you are watching Game of Thrones, right? It’s a bit more complicated. Unless you have built an application that involves video, you probably have not dealt with the world of codecs, bitrates, and streaming. Depending on the bandwidth between the user and the server, you might want to use different compression rates. Think about all of the different use cases–different connection speeds, device types, operating systems, video players, cloud providers. As a developer, you just want videos in your application to play quickly and reliably. But it takes a lot of engineering, monitoring, and re-engineering to get it right. Matt McClure and Jon Dahl are the founders of Mux, a company that makes video infrastructure technologies. Previously they built Zencoder, a product for encoding and delivering video. This episode was a fascinating discussion of why building video products for the modern Internet is still so hard. Download the Software Engineering Daily app for iOS to hear all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that. The iOS app is the first project to come out of the Software Engineering Daily Open Source Project. There are more projects on the way, and we are looking for contributors–if you want to help build a better SE Daily experience, check out github.com/softwareengineeringdaily. We are working on an Android app, the iOS app, a recommendation system, and a web frontend. Help us build a new way to consume software engineering content at github.com/softwareengineeringdaily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A dashboard is a data visualization that aggregates metrics in a way that we can quickly understand. In a modern software company, everyone uses dashboards–from salespeople to DevOps to HR. Each dashboard represents a query that must be updated frequently, so that anyone looking at it is getting up-to-date information. The data set being queried might be getting updated quickly in the case of time series or log data. Some queries require joins between disparate data sources. How do you keep dashboards accurate? How do you keep query latency down? Tom O’Neill is the CTO of Periscope Data, a company that makes popular dashboarding tools. In this episode, Tom explains the data engineering that underlies Periscope data. We explore topics such as caching, columnar data, and Redshift. The iOS app is the first project to come out of the Software Engineering Daily Open Source Project. There are more projects on the way, and we are looking for contributors–if you want to help build a better SE Daily experience, check out github.com/softwareengineeringdaily. We are working on an Android app, the iOS app, a recommendation system, and a web frontend. Help us build a new way to consume software engineering content at github.com/softwareengineeringdaily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Static analysis is the process of evaluating code for errors, memory leaks, and security vulnerabilities. The “static” part refers to the fact that the code is not running. This differentiates it from unit tests and integration tests, which evaluate the runtime characteristics of code. If you use an IDE or a linter, you are using a basic form of static analysis all the time. More sophisticated static analysis tools can be used to analyze code in sensitive domains like healthcare or automobiles. During static analysis, we can discover problems in the code by evaluating the structure of a program. Buffer overruns can be identified before they turn into a vulnerability like Heartbleed. Null pointer exceptions can be fixed before they cause a segmentation fault. Concurrency issues can be serialized before they result in a problematic race condition. Today’s guest Paul Anderson is the VP of engineering at GrammaTech, where he works on CodeSonar, a static analysis tool. We discussed how static analysis works, why it is useful, and how it fits into a modern software delivery pipeline. Full disclosure: GrammaTech is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The underlying cause of failure for many startups is that the founders are afraid of discomfort. An environment where everyone is comfortable is unlikely to be an environment where personal growth and value creation is occurring. When you are in a startup, calibrating the right amount of discomfort is often about calibrating risk. What are your risks? Can you quantify them? Can you enumerate them? Multiplying out the probability of surviving each of those risks, then multiplying that number times the sum of the discounted future cash flows of your business will give you the expected value of your business. Under the right circumstances, entrepreneurship has much higher expected value than a stable engineering job. The important difference is variance. Your business needs to be able to withstand the variance that bad luck can provide. And entrepreneurs themselves need to be able to withstand the variance implied by the fact that their business can completely fail and go to zero. Leo Polovets is a partner with Susa Ventures. He worked as an early engineer at Linkedin, Google, and Factual, and he blogs at Coding VC. In this episode we talked about the proper mindset for founding a company–how to think about risk, mistakes, discomfort, and finance. Coding VC blog Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Tinder is a popular dating app where each user swipes through a sequence of other users in order to find a match. Swiping left means you are not interested. Swiping right means you would like to connect with the person. The simple premise of Tinder has led to massive growth, and the app is now also used to discover new friends and create casual meetings. Every social network knows–if you are not growing, then you are dying. Growth is so important to Tinder, they have a large engineering organization devoted to five facets of growth: new users, activation, retention, dropoff, and anti-spam. These five segments cover the entire Tinder user lifecycle, and there is a sub-team in charge of each of the five areas. No matter what kind of Tinder user you are, there are growth engineers focused on your experience. Alex Ross is the director of engineering for the growth team at Tinder. His job requires a mix of data science, data engineering, psychology, and setting proper KPIs (key performance indicators). Each subteam has KPIs that determine how well they are doing with growth–and if the wrong KPI is set, it can create bad incentives. For example, a growth team that is focused only on getting users to spend more time engaging with Tinder would have an incentive to create so-called “dark patterns” that trigger addiction. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Online advertising enables free content and services of the Internet. One of the free services that is powered by advertising is the browser. 60% of web browsing is done through Chrome, which is owned by Google, which is powered by advertising. The application that most of us use to explore the web is made by a company that relies on ads, so it is unsurprising that the default of that browser is to allow close tracking of user behavior. When you hit a website, a variety of trackers are logging your data for the purpose of serving you better ads. Some people don’t like ads, and they don’t like being tracked–but what is the alternative? How else can we get all the content we want? Since the 90’s, engineers have envisioned an Internet powered by micropayments. A micropayments system in your browser would allow users to pay for content with money instead of adtech. Brave is a web browser built with a modern view of advertising, privacy, and economics. Brave users can pay for content with their money OR by paying attention to ads. This system is formalized through the Basic Attention Token (BAT), a cryptocurrency that can be used to purchase user attention. Jonathan Sampson is a senior developer relations specialist with Brave Software. He joins the show to talk about the problems with the browsing experience and what Brave is doing to stop it. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The applications that demand deep learning range from self-driving cars to healthcare, but the way that models are developed and trained is similar. A model is trained in the cloud and deployed to a device. The device engages with the real world, gathering more data. That data is sent back to the cloud, where it can improve the model. From the processor level to the software frameworks at the top of the stack, the impact of deep learning is so significant that it is driving changes everywhere. At the hardware level, new chips are being designed to perform the matrix calculations at the heart of a neural net. At the software level, programmers are empowered by new frameworks like Neon and TensorFlow. In between the programmer and the hardware, middleware can transform software models into representations that can execute with better performance. Milena Marinova is the senior director of AI solutions at the Intel AI products group, and joins the show today to talk about modern applications of machine learning and how those translate into Intel’s business strategy around hardware, software, and cloud. Full disclosure: Intel is a sponsor of Software Engineering Daily. Question of the Week: What is your favorite continuous delivery or continuous integration tool? Email jeff@softwareengineeringdaily.com and a winner will be chosen at random to receive a Software Engineering Daily hoodie. Data Skeptic podcast: Generative Adversarial Networks Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Spotify is a streaming music company with more than 50 million users. Whenever a user listens to a song, Spotify records that event and uses it as input to learn more about the user’s preferences. Listening to a song is one type of event–there are hundreds of others. Opening the Spotify app, skipping a song, sharing a playlist with a friend–all of these are events that provide valuable insights to Spotify. These are not the only types of events that Spotify cares about. There are also events that occur at the infrastructure level–for example a logging server that runs out of disk space. There are events that are relevant to all the users on Spotify–for example a new album release from Taylor Swift. An “event” is an object that needs to be registered within a system. Since there are so many events on a platform like Spotify, delivering and processing them reliably requires significant investment. Modern Internet companies are built by connecting cloud services, databases, and internal tools together. These different systems might respond to different events in different ways. Each system subscribes to the types of events that it wants to hear. Since there are so many events, and they might be received at uneven bursts, a modern architecture has a scalable queueing system to buffer events. To put an event on the queue, the event producer “publishes” that event to the queue. The event is then received by each “subscriber.” That’s why queueing is often known as pub/sub–publish/subscribe. Igor Maravic is an engineer with Spotify. In this episode, he explains why pub/sub is a key element of Spotify’s infrastructure–and he describes the migration that Spotify has made from Apache Kafka to Google Cloud Pubsub. If you like this episode, we have done many other shows about cloud infrastructure. You can check out our back catalog by downloading the Software Engineering Daily app for iOS, where you can listen to all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Moat is one of the most successful advertising technology companies in history. After building a business from measurement of ad impressions, Moat was sold to Oracle for $850 million. Advertising powers the free content on the Internet. Measurement makes it easier for publishers to monetize their content. At Software Engineering Daily, we know this from firsthand experience. The podcast ecosystem has barely any ability to measure success–and that can make it hard to entice advertisers. In podcasting, it is very difficult to understand if an advertising campaign is a success. This illustrates why Moat is important. Improving the analytics on advertising helps publishers, brands, ad agencies, and adtech companies decide how to allocate their capital. Why is it hard to measure advertising success? Why is this a difficult engineering problem? Because there are so many players in the space with conflicting incentives. A brand wants to show ads to people who will buy a product. A publisher wants to display an ad that will maximize revenue. Adtech companies and ad agencies want to take the biggest cut possible from the transactions between brands and publishers. In the midst of all of this, fraudulent traffic providers offer cheap services that drain money from anyone who is not keeping a close eye on their deal flow. In this fog of war, Moat’s goal is to provide transparency where possible. Moat CEO Jonah Goodhart joins this episode to talk about advertising analytics, viewability, and fraud. If you like this episode, we have done many other shows about adtech and advertising fraud. You can check out our back catalog by going to softwareengineeringdaily.com or by downloading the Software Engineering Daily app for iOS, where you can listen to all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

If I have a picture of a dog, and I want to search the Internet for pictures that look like that dog, how can I do that? I need to make an algorithm to build an index of all the pictures on the Internet. That index can define the different features of my images. I can find mathematical features in each image that describe that image. The mathematical features can be represented by a matrix of numbers. Then I can run the same algorithm on the picture of my dog, which will make another matrix of numbers. I can compare the matrix representing my dog picture to the matrices of all the pictures on the internet. This is what Google and Facebook do–and we covered this topic in our episode about similarity search a few weeks ago. Today, we evaluate a similar problem: searching images within Squarespace. Squarespace is a platform where users can easily build their own website for blogging, e-commerce, or anything else. Neel Vadoothker is a machine learning engineer at Squarespace, and he joins the show to talk about how and why he built a visual similarity search engine. If you like this episode, we have done many other shows about machine learning. You can check out our back catalog by going to softwareengineeringdaily.com or by downloading the Software Engineering Daily app for iOS, where you can listen to all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software gives us new ways of communicating with each other. Engineers build scalable systems for e-commerce, helpdesk, and video sharing–and these systems do scale, to millions of people. But software alone cannot serve all of the demands of all the users and customers on these platforms. We need customer service representatives to address unexpected demands. We need design specialists to evaluate the interface that made sense to the engineers but not the users. We need sales people to connect our strange software to an impatient prospective customer. Engineers sometimes joke about firing all the non-engineers in the company. As engineers, it is easy to discount all of the work that non-engineers do–it can seem unscalable, or non-quantifiable, or mechanical. But most companies would fall over immediately without support, sales, design, operations, and the multitude of other non-engineering roles. More to the point–people in non-technical roles can drive the success of an en organization. Some of the most influential leaders in tech came from a non-technical background: Stuart Butterfield of Slack; Brian Chesky of Airbnb; Sheryl Sandberg of Google and Facebook. A liberal arts education can foster the perfect set of skills to thrive in a technology company. George Anders is an author whose most recent book is called You Can Do Anything: The Surprising Power of a “Useless” Liberal Arts Education. George is one of my favorite business writers, and some of his past writing includes pieces about Sequoia Capital, Amazon, Linkedin, and a ton of other topics on Quora. If you like this episode, we have done many other shows about business with guests like Seth Godin and Tyler Cowen–indeed many of the shows on Software Engineering Daily are not deeply technical. You can check out our back catalog by downloading the Software Engineering Daily app for iOS, where you can listen to all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machines understand the world through mathematical representations. In order to train a machine learning model, we need to describe everything in terms of numbers. Images, words, and sounds are too abstract for a computer. But a series of numbers is a representation that we can all agree on, whether we are a computer or a human. In recent shows, we have explored how to train machine learning models to understand images and video. Today, we explore words. You might be thinking–”isn’t a word easy to understand? Can’t you just take the dictionary definition?” A dictionary definition does not capture the richness of a word. Dictionaries do not give you a way to measure similarity between one word and all other words in a given language. Word2vec is a system for defining words in terms of the words that appear close to that word. For example, the sentence “Howard is sitting in a Starbucks cafe drinking a cup of coffee” gives an obvious indication that the words “cafe,” “cup,” and “coffee” are all related. With enough sentences like that, we can start to understand the entire language. Adrian Colyer is a venture capitalist with Accel, and blogs about technical topics such as word2vec. We talked about word2vec specifically, and the deep learning space more generally. We also explored how the rapidly improving tools around deep learning are changing the venture investment landscape. If you like this episode, we have done many other shows about machine learning with guests like Matt Zeiler, the founder of Clarif.ai and Francois Chollet, the creator of Keras. You can check out our back catalog by downloading the Software Engineering Daily app for iOS, where you can listen to all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that. Question of the Week: What is your favorite continuous delivery or continuous integration tool? Email jeff@softwareengineeringdaily.com and a winner will be chosen at random to receive a Software Engineering Daily hoodie. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Decentralized Autonomous Organization (DAO) was a digital form of venture capital. It was an ambitious idea–to provide a new decentralized business model for organizing corporations on top of the Ethereum blockchain. Few people in the crypto community were opposed to this premise–but the timeline was short, the code requirements were tremendous, and in retrospect, a vulnerability was inevitable. The DAO launched in May 2016, setting the record for the largest crowdfunding event in history. The following month, the DAO was hacked, millions of dollars of Ether were stolen, and the reverberations of the event were a referendum on how the Ethereum community governs itself. Matt Leising is a reporter for Bloomberg who has chronicled the DAO in his article The Ether Thief. He continues to follow cryptocurrencies closely, as the Internet of money fractals increasingly into the public consciousness. If you like this episode, we have done many other shows about cryptocurrencies and their implications. You can check out our back catalog by downloading the Software Engineering Daily app for iOS, where you can listen to all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that. Errata: Coinbase now supports Bitcoin Cash. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

You have probably missed some of the best episodes of Software Engineering Daily. If you listen to just a few episodes a week, it can be difficult to identify the high quality shows. And if you are new to the podcast, you have no idea how to find episodes that might appeal to you. Software Engineering Daily has a discovery problem. We have 600 episodes, and much of the content is evergreen. The shows we did a year ago on Apache Spark, or Ethereum, or ReactJS are still relevant today, and they get plenty of listens. Keith and Craig Holliday built a recommendation system for Software Engineering Daily. Then they built a Software Engineering Daily iOS app to improve the experience of SE Daily listeners. You can use the SE Daily app to find the most popular episodes of this podcast, and to find episode recommendations based on what you have listened to. In this episode, Keith and Craig join the show to explain why they built an app for Software Engineering Daily. You can find all the code for the SE Daily app at github.com/softwareengineeringdaily in case you want to fork it for your own podcast–or if you want to contribute to it. Question of the Week: What is your favorite continuous delivery or continuous integration tool? Email jeff@softwareengineeringdaily.com and a winner will be chosen at random to receive a Software Engineering Daily hoodie. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a cyber attack occurs, how do we identify who committed it? There is no straightforward answer to that question. Even if we know Chinese hackers have infiltrated our power grid with logic bombs, we might not be able to say with certainty whether those hackers were state actors or rogue Chinese hackers looking for an offensive asset to sell to their government. Even if we know someone in Russia launched an attack on the banking system in Ukraine, we might not know whether that attack came from the government or from aggressive non-governmental forces. Accurate cyberattack attribution is key to preventing diplomatic mistakes in the modern battleground of the Internet. Today’s guest John Davis is one of the authors of the report called “Stateless Attribution: Toward International Accountability in Cyberspace”. John is a senior information scientist with RAND Corporation, a non-profit institution that helps improve policy and decisionmaking through research and analysis. This report was commissioned by Microsoft, and it provides a deep assessment of our current ability to attribute a cyberattack to the perpetrator of that attack. If you like this episode, we have done many other shows about security, with guests like Bruce Schneier and Samy Kamkar. You can check out our back catalog by downloading the Software Engineering Daily app for iOS, where you can listen to all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ransomware and DDoS attacks happen all the time. Sometimes they affect large swaths of users. WannaCry ransomware froze the computer systems in hospitals. Mirai botnet DDoS attacks took down a DNS provider, making Netflix and Twitter inaccessible for a short period of time. These are innocent attacks compared to what we could face from a world where cars, heart rate monitors, and other safety critical machinery become connected to the Internet. This is not a new subject–we have covered it in previous episodes about security. But it’s a deep subject, and there is much ground to cover. Chris Craig joins the show for this episode–he is a security researcher at Oak Ridge National Lab. He studies network and cloud security, and in this episode he brings his broad expertise to subjects like IoT security, car security, and the question of standards–what do we need to standardize and certify as the internet becomes connected to physical infrastructure? Thanks to Jared Smith for the introduction. When Safety and Security Become One Standardisation and Certification of the ‘Internet of Things’ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software companies that have been around for a decade have a ton of data. Modern machine learning techniques are able to turn that data into extremely useful models. Salesforce users have been entering petabytes of data into the company’s CRM tool since 1999. With its Einstein suite of products, Salesforce is using that data to build new product features and APIs. Simon Chan is the senior director of product management with Einstein. He oversees the efforts to give longtime Salesforce customers new value, and the efforts to build brand new APIs for image recognition and recommendation systems, which can form the backbone of entirely new businesses. Companies spend billions of dollars on sales and marketing, and I wanted to understand where the best opportunities for Salesforce were. Simon and I spent much of our time exploring higher level applications, but we got to lower level engineering eventually. There are 600 episodes of Software Engineering Daily, and it can be hard to find the shows that will interest you. If you have an iPhone and you listen to a lot of Software Engineering Daily, check out the Software Engineering Daily mobile app in the iOS App Store. Every episode can be accessed through the app, and we give you recommendations based on the ones you have already heard. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

We write code in a language that looks like English. Whether it is JavaScript, Fortran, or assembly language, that code is an abstraction on top of layers of intermediate languages, binary, transistors, and physics. 100 years ago, this would have seemed like magic. Most of us know about Alan Turing, who described the vision of a multipurpose computer with the concept of the Turing machine. Less well known is the scientist Claude Shannon, who laid the groundwork of information theory. With information theory, we can compress data and communicate it efficiently. Jimmy Soni and Rob Goodman are the authors of “A Mind at Play,” a biography of Claude Shannon. Claude’s unique insights about information were made possible by his willingness to involve himself in lots of different areas–science, art, juggling, warfare. This interview gives insights for how we can think of new ideas by synthesizing disparate subjects. There are 600 episodes of Software Engineering Daily, and it can be hard to find the shows that will interest you. If you have an iPhone and you listen to a lot of Software Engineering Daily, check out the Software Engineering Daily mobile app in the iOS App Store. Every episode can be accessed through the app, and we give you recommendations based on the ones you have already heard. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Automation will make healthcare more efficient and less prone to error. Today, machine learning is already being used to diagnose diabetic retinopathy and improve radiology accuracy. Someday, an AI assistant will assist a doctor in working through a complicated differential diagnosis. Our hospitals look roughly the same today as they did ten years ago, because getting new technology into the hands of doctors and nurses is a slow process–just ask anyone who has tried to sell software in the healthcare space. But technological advancement in healthcare is inevitable. Cosima Gretton is a medical doctor and a product manager with KariusDX, a company that is building diagnostic tools for infectious diseases. She writes about the future of healthcare, exploring the ways that workflows will change and how human biases could impact the diagnostic process–even in the presence of sophisticated AI. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Loans give people more financial security. If people know that they can receive a loan, they will be more willing to take intelligent risks. A loan can allow for a short-term investment that pays off enough to justify the interest rate on that loan. For the lender, a loan can be a fantastic return on capital–as long as the lendee does not default. When banks were the rulers of the financial infrastructure, most of them would err on the side of caution when it came to lending. They would adhere strictly to credit scores, and a wanting customer would be out of luck if they did not have a credit score, or if their credit score had gotten lower than acceptable. Newer fintech companies are taking advantage of data sources other than credit scores. They are using machine learning in conjunction with these new data sources to find viable lendees who would be overlooked by traditional institutions. Ofer Mendelevitch is the VP of data science at LendUp. He joins the show to explain why loans are important, how LendUp functions, and the machine learning systems that power an intelligent system of lending. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Sensors are being attached to trains, lightposts, and all kinds of factory equipment. Industrial machinery gives off high volumes of data that can be captured, stored, and processed with machine learning in order to improve workflows and ensure safety. Jayson Delancey works at GE, which is building tools and systems to manage large IoT deployments. The full stack for enterprise IoT involves tools for managing thousands of sensors; databases for storing all the data that is coming off of these devices; authentication and authorization systems for enforcing security. There is a lot to do. In this episode, Jayson surveys some of the technology GE is building with Predix, its industrial IoT platform. He also talked about some of the large scale IoT deployments he has seen. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Most products do not sell themselves. Salespeople bridge the gap between a product creation and a customer who purchases it. People can make a good living on the internet selling niche products–if they can find their customers. The process of taking a large group of potential customers and narrowing it down to only the subset of those customers who will buy your product is known as the sales funnel. The sales funnel consists of multiple stages–the first of which is known as “prospecting.” A salesperson doing prospecting is casting a wide net, sending emails to hundreds or thousands of people, looking for anyone who has some small probability of being interested. Without a tool for prospecting, the process can be very labor intensive. Jean-Baptise Escoyez is the CTO at Prospect.io, a tool for sales prospecting. In this episode, we explored the process of building Prospect.io, from the high level product design to the engineering details of how it is implemented. I use Prospect.io to sell two different products so it was enjoyable to find out how one of my favorite tools works. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Applications built in the cloud are often serving requests from all around the world. A user in Hong Kong could have written to a database entry at the moment just before a user in San Francisco and a user in Germany simultaneously try to read from that database. If the user in San Francisco is allowed to see a different database entry than the user in Germany, that database is not strongly consistent. Strongly consistent databases work such that two users who read the same entry at the same time will receive the same result. Weakly consistent or “eventual consistent” databases are suitable for applications where transaction ordering is not important–photo sharing apps and ecommerce shopping carts, for example. Bank accounts, on the other hand, often need to be strongly consistent. CockroachDB is a scalable, survivable, strongly consistent database. Alex Robinson is an engineer at Cockroach Labs and he joins the show to explain the data model for CockroachDB and how it maintains strong consistency. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Religious extremists use technology to recruit vulnerable individuals to a violent cause. Google is developing ways to combat this extremism through its platforms, namely YouTube. When a user looks for inflammatory religious or supremacist content, YouTube’s “Redirect Method” instead sends those users toward anti-terrorist videos. Google’s fight against extremism compelled writer Lochlan Bloom to write an article called “The Coming Battle: AI, Extremism, and the New War of Ideas.” Lochlan joins the show to discuss the societal implications of giant internet providers controlling our information flow. Lochlan is a science fiction writer, most recently of The Wave, a book that mixes existentialism with quantum physics. Our reality is defined by what we observe, and this theme courses through our conversation–from religion to Twitter to artificial intelligence. The Coming Battle: AI, Extremism, and the New War of Ideas Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Content websites are supported by advertising. Most of the advertisements around the internet are dynamic ad slots that change depending on the user who visits the site. Those dynamic ad slots are available to a variety of different bidders. For each ad slot, an auction occurs. The highest bidder gets to serve an ad for that slot. Praneet Sharma is the co-founder of Method Media Intelligence, which he founded with Shailin Dhar, who has been on the show several times to discuss his investigations into the world of ad fraud. I wanted to have his partner Praneet on the show to get his perspective on ad fraud and how to clean up the advertising ecosystem. One advance in dynamic advertising that we discussed is header bidding, and an open source library called PrebidJS. When an ad-supported website gets delivered to your web browser, the HTML begins to load and the JavaScript on the page begins to execute. Some of that JavaScript is calling out to advertising networks looking for the highest bidder. Until the page receives a callback for what to put in the ad slots on the page, the page will not finish loading. Sites that do not manage their ad requests appropriately suffer performance issues. Header bidding is a technique to wrap all of the requests to different advertising exchanges in a single serialized blob of code at the top of the page. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Internet runs on advertising. Advertising is subject to fraud–but then again, so is every system of online transactions. The amount of money lost in electronic payments fraud and ecommerce scamming is probably much greater than what is lost due to ad fraud. So why do we keep covering advertising fraud on Software Engineering Daily? More of our audience needs to know about ad fraud. Few people realize how much fraud there is in online advertising. In previous episodes of Software Engineering Daily, we have explained, how advertising fraud works, why it is absurd and disgraceful, and why nobody talks about. We also cover ad fraud because I personally find it interesting and sometimes hilarious. Those are the same reasons I invited Shailin Dhar to speak at the third Software Engineering Daily Meetup. Shailin has been on the show twice before and he will be on again in the future. He has made it a full time job to expose ad fraud, and he gives a great presentation on the topic in this episode. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Querying a search index for objects similar to a given object is a common problem. A user who has just read a great news article might want to read articles similar to it. A user who has just taken a picture of a dog might want to search for dog photos similar to it. In both of these cases, the query object is turned into a vector and compared to the vectors representing the objects in the search index. Facebook contains a lot of news articles and a lot of dog pictures. How do you index and query all that information efficiently? Much of that data is unlabeled. How can you use deep learning to classify entities and add more richness to the vectors? Jeff Johnson is a research engineer at Facebook. He joins the show to discuss how similarity search works at scale, including how to represent that data and the tradeoffs of this kind of search engine across speed, memory usage, and accuracy. Notes: Jeff’s blog post about similarity search Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Augmented reality is coming at us fast. Every large tech company is rumored to be building an AR product. Microsoft HoloLens is already available to developers. Pokemon Go, the most popular augmented reality product today, was made by a company that was spun out of Google. But Apple seems to be ahead of everyone. Apple’s ARKit is a set of tools for developers to build augmented reality applications. The applications people are building with ARKit are remarkable, and two of those early adopters join the show today for an interview. Jesse Bounds and Siyu Song work at Mapbox, a company that makes mapping, navigation, and location search SDKs. Location is natural companion to augmented reality. If I am walking down the street with a pair of augmented reality glasses on, those glasses can augment the world with information based on my location. Because the fit between AR and mapping is so natural, Mapbox has been rapidly experimenting to build up an expertise in AR. As a result, Jesse and Siyu make for great guests to talk about what engineers can build with ARKit today and what might be possible in the future. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a user experiences an error in an application, the engineers who are building that application need to find out why that error occurred. The root cause of that error may be on the user’s device, or within a piece of server-side logic, or hidden behind a black box API. To fix a complex error, we need a stack trace of contextual information so that we can correlate events across all layers of an application. James Smith is the CEO of Bugsnag, a company that makes crash reporting and error tracking software. In this episode, he describes how to diagnose errors in modern applications. He also explains how the company functions and how Bugsnag itself is built. The product consumes and stores millions of events which makes for a good discussion of software architecture. Full disclosure: Bugsnag is a sponsor of SE Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

GatsbyJS is a framework for building web applications for JavaScript. Gatsby’s original goal was to allow users to create super fast static web sites that could be hosted and served efficiently at a low cost. Most web pages have components from a framework like React or Angular that need to render after the user requests them. This rendering can sometimes require additional requests to external data sources, causing the page to take longer to load. Gatsby uses GraphQL to pull in data at build time and pre-render as much of a site as possible using React’s server side rendering. When a page built with Gatsby is served to a user, as much of the page has been rendered as possible, so that the browser can quickly load everything on the page without additional network requests. Kyle Mathews is the creator of GatsbyJS. He joins the show to describe why he created Gatsby–the high level goals and low level engineering decisions. We also discuss how Kyle intends to take Gatsby beyond just an open source project and turn it into a business. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Building and nurturing a developer community is hard work but it is vital for the growth of a country’s technology ecosystem. When communities coalesce around programming languages, tools or programming methods, what follows is a network of conferences, meet ups and other similar events. Juan Pablo Buriticá, VP of Engineering at Splice, has spent the last decade building developer communities in his home country of Colombia and across Latin America as well as running distributed engineering teams. He has helped to launch a number of conferences, meet-ups and more recently, an online meet-up providing advanced technical information for native Spanish speakers. In today’s episode, Juan returns to Software Engineering Daily to speak with Carl Mungazi about the benefits of having a distributed engineering team and hiring developers from developing nations. He discusses how Colombia came to have the largest Spanish-speaking JavaScript community in the world and the importance of good communication when building software with a team of developers based around the world. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Quality assurance testing is a form of testing that closely mirrors user behavior. Sometimes it is manual, sometimes it is automated. Automated QA tests are scripts that validate correct data representation as the application mechanically runs through high-level workflows–like a login page. Manual QA testers act out use cases of an application to see if there are any bugs that were missed during automated test cases. Manual QA testing is often necessary for complex applications where it is not possible to enumerate all potential workflows within a script. Different companies have radically different workflows for QA testing. There are a variety of ticketing systems, testing frameworks, and team chat applications that play a role in a tester’s daily life. QASymphony is a platform for testing tools that integrates with other popular technologies to centralize a QA testing workflow. Jonathan Alexander is the CTO at QASymphony. He’s also the author of Codermetrics: Analytics for Improving Software Teams. He joins the show to discuss the past and present of QA and his strategies for managing the team that is building QASymphony. Thanks to Kevin Wolf for the intro. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook was rapidly outgrowing its infrastructure in 2009. Classic data center design was not up to the task of the rapid influx of new users and data, photos and streaming video hitting Facebook’s servers. A small team of engineers spent the next two years designing a data center from the ground up to be cheaper, more energy efficient, and more ergonomic for the engineers who worked within. That data center design was open sourced in 2011. Intel, Rackspace, and Goldman Sachs were the first three large organizations to join Facebook in the Open Compute Project, an effort to bring the benefits of open source collaboration to data centers. Steve Helvie works on the Open Compute Project and he joins the show to describe how the project has evolved in the last six years–how it has affected data center design and the implications for the future. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Slack is an application for team communication. Users chat across mobile devices, web browsers, and a desktop application, which means Slack has three places to deploy on rather than two. And the desktop apps on Windows, Mac, and Linux are not identical, so Slack has even more places to deploy. With so many different runtime environments, Slack needs to make technology choices that reduce the chance of errors. TypeScript allows for static typing of JavaScript. The extra compilation step checks the types of variables being passed between different places–so the errors will be discovered at compile time. In an untyped world, those errors might occur at runtime. TypeScript also unlocks the ability to put JavaScript code in an IDE, allowing for more efficient development. Felix Rieseberg is a desktop engineer at Slack, and in today’s episode he explains the unique challenges of building Slack, and why the team moved from JavaScript to TypeScript. TypeScript at Slack – engineering blog Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Animations make an application more fun and engaging. For most apps, animation is an afterthought. Developers are concerned with getting the functionality right, and designers have enough work to do simply getting icons, text formatting, and page layout correct. There is also the issue of cross-device compatibility. iOS, Android, and web have different ways of doing animation, with no unifying standard–except gifs, and gifs are not interactive, they simply play from start to finish. Airbnb’s emphasis on design makes it the right company to work on the problem of cross-device, interactive animations. Brandon Withrow and Gabriel Peal are engineers who work on Lottie, a library for animations in iOS, Android, web, and React Native. This episode is about how and why Lottie was built, and how Lottie gets used within Airbnb. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

JavaScript is moving so fast. It’s not easy to keep up with all of the frameworks, build tools, and packages. No other language spans frontend to backend, mobile to web to server. Sacha Greif is an independent designer and developer most prominent in his roles as co-author of Discover Meteor and community builder at Sidebar.io, a design newsletter with over 35,000 subscribers, and Hacker News Kansai. He is currently best known in the Javascript community as the maintainer of VulcanJS, and for his annual State of Javascript survey which is now open for 2017. In this episode, Shawn Wang guests hosts a discussion about both projects and Sacha’s thoughts on independent web design and development. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Internet of Things is the concept that traditionally analog objects, like thermostats and lightbulbs, can be given digital guts and connected to the internet to create more value for users. From Nest thermostats to Phillips Hue lightbulbs, these connected things are starting to enter the mainstream. According to recent estimates by Gartner, over eight billion connected “Things” will be in use in 2017, with that number ballooning to over twenty billion by 2020. Jeremy Foster is a Technical Evangelist at Microsoft and the host of Microsoft Virtual Academy’s “Introduction to Azure IoT” course. In this episode, host Jared Porcenaluk joins Jeremy to discuss developing for the Internet of Things. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Serverless computing reduces the cost of using the cloud. Serverless also makes it easy to scale applications. The downside: building serverless apps requires some mindset shift. Serverless functions are deployed to transient units of computation that are spun up on demand. This is in contrast to the typical model of application delivery–the deployment of an application to a server or a container that stays running until you shut it down. Robin Weston develops large projects with AWS Lambda, and he joined me for a discussion of how to build applications for serverless environments and how to do continuous delivery with serverless functions. One big appeal for continuous delivery fans is that serverless deployments are often smaller–the user is deploying something as small as a function. Full disclosure: ThoughtWorks GoCD is a sponsor of Software Engineering Daily. Serverless Architectures and Continuous Delivery by Robin Weston Robin Weston at Pipeline Conf Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

After raising $18 million, social networking startup Yubl made a series of costly mistakes. Yubl hired an army of expensive contractors to build out its iOS and Android apps. Drama at the executive level hurt morale for the full-time employees. Most problematic, the company was bleeding cash due to a massive over-investment in cloud services. This was the environment in which Yan Cui joined Yubl. The startup did have traction. There were social media stars who would announce on Twitter that they were about to go on Yubl, and Yubl would be hit by an avalanche of traffic. 50,000 users suddenly logging on to interact with their favorite celebrity was a significant traffic spike. How do you deal with a traffic pattern like that? Serverless computing. AWS Lambda allowed the company to scale up quickly in a cost efficient manner. Yan began refactoring the entire backend infrastructure to be more cost efficient, heavily leveraging AWS Lambda. Unfortunately, Yan’s valiant effort was not enough to save the company. But there are some incredible engineering lessons from this episode–how to build cost-effective, scalable infrastructure. It’s also a case study worth looking at if you work at a startup, whether or not you are an engineer. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Quantum computing is based on the system of quantum mechanics. In quantum computing, we perform operations over qubits instead of bits. A qubit is a vector, which can take on many more values than 0 or 1. The technology used to implement quantum computers is advancing such that it has its own Moore’s Law, but it can also leverage the classical advancements of Moore’s Law. If classical computing advances at the exponential pace of 2^n, quantum computing advances at the pace of 2^2^n. Quantum computing will advance technology in ways that will take us by surprise. If things feel like they are moving fast now, just wait until developers have access to quantum processing units. Machine learning, simulated chemical synthesis, and NP-complete problems are ripe for quantum computers. Vijay Pande is a partner at Andreessen Horowitz and a board member at Rigetti Computing, a quantum computer company. In this episode, we explored what software engineers today need to know about quantum computers and some of the application domains that developers will be working on as quantum computers become available. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Continuous delivery is a model for deploying small, frequent changes to an application. In a continuous delivery workflow, code changes that are pushed to a repository set off a build process that spins up a new version of the application. Testing is performed against that new build before advancing it to production, merging it with the existing codebase. Many continuous delivery products are getting built today because it is a wide open space–much like cloud providers or monitoring tools. There are subjective product and engineering decisions to be made depending on the audience for the product. Heroku Flow is a continuous delivery platform built on top of Heroku, a platform as a service. Andy Appleton is an engineer at Heroku and he joins the show to describe how Heroku Flow was built. Two years of work went into the project from initial conception to launch. Full disclosure: Heroku is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Patents allow individuals and company to lay creative claim for an invention. A patent can provide protection from having its idea being used without giving credit to its creators. Of course, is that patents can be filed and not turned into products, inhibiting innovation. Patents can also be used offensively in a practice known as patent trolling. Large companies like IBM and Google have so many patents that they have trouble keeping track of them all. And if your company has many different hardware and software products, how can you be sure that your patent collection protects you from a patent troll? Nicole Shanahan is the CEO of ClearAccessIP, a product that indexes patents, looks for vulnerabilities in a corporation’s patent strategy, and finds opportunities in a patent collection for further value. The large text corpus of a patent collection is the perfect place to apply machine learning. We discussed the nature of patents, the intersection between law and software, and the product development process of ClearAccessIP. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Wearables are everywhere. In the medical field they are transforming lives. Haiyan Zhang, Innovation Director at Microsoft Research, created a wearable for a young graphic designer that developed Parkinson’s. Parkinson’s is a condition that inhibits movement, and this wearable allows the Parkinson’s patient to write and draw again. Haiyan explained the research process and the technical aspects of how it works. Edaena Salinas of The Women in Tech Show interviewed Haiyan for this episode. They talked about the Internet of Things, the components of these systems and the technical challenges. Haiyan also explained her path from software engineering to design and the process of commercializing products that come from research. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Self-driving cars are here. Fully autonomous systems like Waymo are being piloted in less complex circumstances. Human-in-the-loop systems like Tesla Autopilot navigate drivers when it is safe to do so, and lets the human take control in ambiguous circumstances. Computers are great at memorization, but not yet great at reasoning. We cannot enumerate to a computer every single circumstance that a car might find itself in. The computer needs to perceive its surroundings, plan how to take action, execute control over the situation, and respond to changing circumstances inside and outside of the car. Lex Fridman has worked on autonomous vehicles with companies like Google and Tesla. He recently taught a class on deep learning for semi-autonomous vehicles at MIT, which is freely available online. There was so much ground to cover in this conversation. Most of the conversation was higher level. How do you even approach the problem? What is the hardware and software architecture of a car? I enjoyed talking to Lex, and if you want to hear more from him check out his podcast Take It Uneasy, which is about jiu jitsu, judo, wrestling, and learning. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A decade ago, a Microsoft developer might have been defined by the fact that they built C# applications on Windows. Today, a Microsoft developer is just as likely to be writing JavaScript for Linux. The company has repositioned itself to focus on cloud services, SaaS products, and enterprise artificial intelligence. Jason Young and Carl Schweitzer host the MS Dev Show, a popular podcast about Microsoft developers and technologies. On their show, they explore the rapidly expanding marketplace of services on Microsoft Azure and talk to experts about how these services are built and what they are used for. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Building a startup requires constant evaluation of tradeoffs. At the earliest stage, the founders evaluate different ideas. Once an idea is settled on, the company develops strategies for finding early customers and growing. As the company develops traction, the operators consider ways to scale further or partner with an acquirer. Joseph Jacks and Greg Koberger are two founders who have been on the show previously. JJ started Kismatic, the earliest company completely focused on Kubernetes. Kismatic was acquired by Apprenda last year. Greg runs Readme.io, a company that provides beautiful documentation as a service. In this episode, Greg and Joe share their thoughts on running and scaling startups–engineering concerns, scaling strategies, and discussions of what to build and what to buy. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Access to education is something everyone strives for but not all achieve–especially education that leads to meaningful and well-paying work. In today’s world where software is eating all sorts of industries, access to a good technical education is still out of the reach of many people. Laboratoria is a social enterprise which teaches women from low-income backgrounds in Peru, Mexico and Chile how to code and helps place them in coding jobs. It was started in Peru by couple Mariana Costa (CEO) and Herman Marin (CTO) along with a friend after they found it difficult to hire developers for a web agency they had started. In today’s episode, Mariana talks to Carl Mungazi about how Laboratoria is using software engineering to change the lives of the women in Latin America whilst also meeting a demand for good technical talent. She discusses the challenges faced by her students, who sometimes spend hours traveling to the school, and her plans for training 10,000 developers over the next five years. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Containers are widely used in projects that have adopted Docker, Kubernetes, or Mesos. Containers allow for better resource isolation and scalability. With all of the adoption of containers, companies like Red Hat, Google, and CoreOS are working on improved standards within the community. Standards are important to this community because of its pace of growth and the number of concurrent projects. If you heard our recent episode about the Linux Kernel’s open source governance, you know that having some rules in place will help encourage the right kind of creativity to thrive. In the world of containers, networking is not well addressed as it is highly environment specific. The Container Networking Interface is an effort to add specifications around how networks of containers can form. Dan Williams is an engineer at Red Hat. In today’s episode, he explores the ideas behind the container networking interface, which gives insights into how the broader community of cloud native technologies is evolving as a whole. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Reinforcement learning is a type of machine learning where a program learns how to take actions in an environment based on how that program has been rewarded for actions it took in the past. When program takes an action, and it receives a reward for that action, it is likely to take that action again in the future because it was positively reinforced. Michal Kempka is a computer scientist work works on VizDoom, an AI research platform for reinforcement learning, with co-creators Marek Wydmuch, Grzegorz Runc, Jakub Toczek, Wojciech Jaśkowski. VizDoom is based on the first-person dungeon game Doom. In VizDoom, an autonomous agent navigates through a maze avoiding enemies. Reinforcement learning is a widely used tool for machine learning, and we will be doing more shows in the future that explain how it works in further detail. Cornell University Library: VizDoom Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In its most basic definition, machine learning is a tool that makes takes a data set, finds a correlation in that data set, and uses that correlation to improve a system. Any complex system with well-defined behavior and clean data can be improved with machine learning. Several precipitating forces have caused machine learning to become widely used: more data, cheaper storage, and better tooling. Two pieces of tooling that have been open sourced from Google help tremendously: Kubernetes and TensorFlow. Kubernetes is not a tool for machine learning, but it simplifies distributed systems operations, unlocking more time for engineers to focus on things that are not as commodifiable–like tweaking machine learning parameters. TensorFlow is a framework for setting up machine learning systems. Machine learning should affect every aspect of our lives–including tuxedo fitting. Generation Tux is a company that allows customers to rent apparel that historically has required in-person fitting. Using machine learning, they have developed a system that allows customers to get fit for an outfit without entering a brick-and-mortar store. In this episode, Colan Connon and Thomas Bell from Generation Tux join to explain how Generation Tux adopted Kubernetes and TensorFlow, and how the company’s infrastructure and machine learning pipeline work. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software engineers have a skill set that can be applied to solve problems outside of a codebase. Analytical skills can be used to evaluate investment opportunities. Creative thinking can be used to build businesses. Communication skills can be used to build and enhance relationships. John Sonmez is a software engineer who created the Simple Programmer, a community of developers who discuss strategies around software, business, and life. He joined me on the show to discuss these topics and others, as well as his new book The Complete Software Developer’s Career Guide. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com SE Radio: John Sonmez, Marketing Yourself and Managing Your Career Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every software company backs up critical data sources. Backing up databases is a common procedure, whether a company is in the cloud or on-prem. Backing up virtual machine instances is less common. Rubrik is a company that is known for building backup infrastructure for enterprises. Their main product is an appliance that sits on prem at an enterprise and stores snapshots of virtual machines running within the enterprise. If a virtual machine dies, Rubrik can quickly restore the VM snapshot. The appliance also backs up to the cloud. Kenny To is a founding engineer at Rubrik, and he joins the show to discuss backups and how Rubrik is engineered. Enterprises that start backing up to the cloud through Rubrik start a path towards potentially more cloud services. For enterprises that have not been able to move to the cloud yet, this can be an appealing opportunity. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Shopify is a company that helps customers build custom online storefronts. Shopify has built upon the same Ruby on Rails application since the founding of their business 12 years ago starting with Rails 0.5 and moving all the way to Rails 5. MRuby is a lightweight implementation of the Ruby language. Shopify made the decision to use mruby to allow customers to create custom scripts that are run every time a customer adds items to their cart. However, since mruby was a language implementation that was not widely used, Shopify opted to post a Bug Bounty to the HackerOne bug bounty platform to find security vulnerabilities in their use of mruby. What followed was a payout of over $500,000 as report after report flooded in of security vulnerabilities inside mruby itself. There was so many reports that Shopify made the decision to sandbox the mruby execution into separate processes and decreased the bounty awards by 90%. In this episode, Jeremy Jung interviews Daniel Bovensiepen (BOH-ven-see-pen) about mruby and the Shopify bug bounty. Mruby: http://mruby.org/ The $500,000 release: http://mruby.sh/201703270126.html HackerOne bounty page: https://hackerone.com/shopify-scripts American Fuzzy Lop: http://lcamtuf.coredump.cx/afl/ Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At Coinbase, security is more important than anything else. Coinbase is a company that allows for storage and exchange of cryptocurrencies. Protecting banking infrastructure is difficult, but in some ways the stakes are higher with Coinbase, because bitcoin is fundamentally unregulated. If a hacker were able to syphon all of the money out of Coinbase accounts, Coinbase would have no recourse–which means this is a more sensitive problem than the regulated banking system, where transactions can often be reversed. Philip Martin is the director of security at Coinbase. He joins the show today to explain why his love of complex and high-stakes security challenges brought him to Coinbase. Philip has some specific points about Coinbase and some more abstract points about security that were very useful to me. This is the third and final episode in our series about Coinbase. Our first two episodes covered the currencies of Coinbase and the fraud prevention techniques the company uses. We’d love to hear your thoughts on this series, and any other suggestions or feedback you have. Send me an email–jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Coinbase is a platform for buying and selling digital currency: bitcoin, ethereum, and litecoin. Every payments company deals with fraud, but a cryptocurrency company has a harder job than most payments companies, because bitcoin transactions are anonymous and non-reversible. This is in contrast to a bank, which deals with a regulated, reversible transaction system. Soups Ranjan is the director of data science at Coinbase. In this episode, he walks through the challenges of preventing fraud and describes how machine learning and humans in the loop are used to deal with bad actors. From the data ingestion to the data engineering to the data science, this episode is a great overview of antifraud at Coinbase, and is a nice complement to the presentation that we previously aired from Soups. This is the second episode in our series about Coinbase. Yesterday we discussed how Coinbase makes cryptocurrencies easier to work with. Tomorrow we dive into the security infrastructure of Coinbase. We’d love to hear your thoughts on this series, and any other suggestions or feedback you have. Send me an email–jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cryptocurrencies have seen a surge of value recently. People are starting to see that bitcoin, ethereum, and other currencies are not just for speculation. At worst, they are a store of value–like digital gold. At best, they are a tool for micropayments, smart contracts, and an entire decentralized financial platform. Coinbase is a company for buying and selling cryptocurrencies. This episode is the first of three interviews with different members of Coinbase. In this episode, Linda Xie and Jordan Clifford explain why cryptocurrencies are important, and how products that Coinbase builds make cryptocurrencies easier to use. This is the first in a series of episodes about Coinbase. Tomorrow we will discuss antifraud with Soups Ranjan, director of data science at Coinbase. We’d love to hear your thoughts on this series, and any other suggestions or feedback you have. Send me an email–jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software deployment evolves over time. In the 90s, a “deployment” might have meant issuing a new edition of your software via CD-ROM. Today, a deployment is often a multi-stage process. A new software build will undergo automated unit tests and integration tests, before being deployed to users. The deployment might only go out to a small percentage of total users initially, with that percentage going up as the deployment proves not to have bugs. Avi Cavale is the CEO of Shippable, a platform for DevOps. In this episode, we discussed deployments in the context of containers, including a discussion of what has become easier: microservices, feature flagging, and continuous delivery. He also discussed his experience building Shippable. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Kafka is an open-source distributed streaming platform. Kafka was originally developed at LinkedIn, and the creators of the project eventually left LinkedIn and started Confluent, a company that is building a streaming platform based on Kafka. Kafka is very popular, but is not easy to deploy and operationalize. That is why Confluent has built a Kafka-as-a-service product, so that managing Kafka is not the job of an on-call DevOps engineer. Neha Narkhede is the CTO of Confluent and she has been on the show twice before to discuss Kafka. In our last episode, we discussed event sourcing and CQRS with Kafka. In this episode, we explore more common enterprise uses for Kafka, and Neha talks about the engineering complexities of building a managed Kafka-as-a-service product. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A cryptocurrency exchange faces a uniquely difficult fraud problem. A hacker who steals my credentials can initiate a transfer of all my bitcoin to another wallet, and it is a non-reversible, non-identifiable payment. So it is really important to prevent those kinds of fraudulent transactions. At the third Software Engineering Daily Meetup, Coinbase director of data science Soups Ranjan explained how Coinbase stays ahead of fraudsters, and he describes some of the cutting-edge social engineering attacks that are being used to try to steal cryptocurrency–including cell phone takeover attacks. Next week, we will be airing three shows I did on-site at Coinbase–interviews with engineers from three different teams. Check out those shows for a deep dive into cryptocurrency uses, fraud, and infrastructure. Coinbase is an exciting company, and it was a lot of fun getting a panorama for how several parts of the organization function. The next Meetup will be in New York. We don’t know when it will be yet, but sign up to follow us at softwareengineeringdaily.com/meetup. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Airbnb is a company that is driven by design. New user interfaces are dreamed up by designers and implemented for web, iOS, and Android. This implementation process takes a lot of resources, but it used to take even more before the company started using React Native. React Native allows Airbnb to reuse components effectively. React Native works by presenting a consistent model for the user interface regardless of the underlying platform, and emitting a log of changes to that user interface. The underlying platform translates those changes into platform specific code. Leland Richardson is an engineer at Airbnb. In today’s episode, he explains how Airbnb uses React Native, how React Native works, and the future of the platform. Check out our new topic feeds, in iTunes or wherever you find your podcasts. We’ve sorted all 500 of our old episodes into categories like business, blockchain, cloud engineering, JavaScript, machine learning, and greatest hits. Whatever specific area of software you are curious about, we have a feed for you. Check the show notes for more details. React Europe Talk Airbnb article on react-sketchapp Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

React Native allows developers to reuse components from one user interface on multiple platforms. React Native was introduced by Facebook to reduce the pain of teams who were rewriting their user interfaces for web, iOS, and Android. Nader Dabit hosts React Native Radio, a podcast about React Native. Nader also trains companies to user React Native through his company React Native Training. In this episode, we explore what a developer can and cannot do with React Native, when a developer needs to use native APIs, and some speculation on the future of React Native. This episode is a good preface for tomorrow’s episode about React Native Interfaces with Leland Richardson of Airbnb. In that episode we will dive deeper into how React Native works and just how big of a change it could be for cross-platform developers. Check out our new topic feeds, in iTunes or wherever you find your podcasts. We’ve sorted all 500 of our old episodes into categories like business, blockchain, cloud engineering, JavaScript, machine learning, and greatest hits. Whatever specific area of software you are curious about, we have a feed for you. Check the show notes for more details. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

“Culture fit” is a term that is used to describe engineers that have the right personality for a given company. In the hiring process, “lack of culture fit” is used to turn away engineers who are good enough at coding but just don’t seem right for the company. As today’s guest Ammon Bartram says, “lack of culture fit” usually means “lack of enthusiasm for what a company does.” Ammon is the co-founder of Triplebyte, a company that is debugging the interviewing process. Triplebyte has interviewed thousands of engineers, and is discovering which aspects of the current hiring process make sense and which are based on superstition, or tradition. We had a great conversation about what culture really means, and how to hire effectively. Check out our new topic feeds, in iTunes or wherever you find your podcasts. We’ve sorted all 500 of our old episodes into categories like business, blockchain, cloud engineering, JavaScript, machine learning, and greatest hits. Whatever specific area of software you are curious about, we have a feed for you. Check the show notes for more details. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The history of computing can be thought of as a series of ideas rather than objects. From Aristotle’s formalization of the syllogism, to Alan Turing’s model for an all-purpose computing machine, to Satoshi Nakamoto’s distributed transaction ledger–these breakthroughs did not come in the form of polished, tangible objects. In fact, the objects which end up changing computing fundamentally are often built from ideas that seemed trivial at first glance. Chris Dixon is a general partner at venture capital firm Andreessen Horowitz and is the author of the article How Aristotle Created the Computer. One job of a venture capitalist is to be early in identifying the ideas that will evolve into influential, tangible objects. In this article, Chris examined several instances in the history of computing where ideas that looked weird and impractical at first glance ended up being world-changing. Recent examples we discussed are blockchains and neural networks. Chris recently wrote a great article about crypto tokens. Check out our new topic feeds, in iTunes or wherever you find your podcasts. We’ve sorted all 500 of our old episodes into categories like business, blockchain, cloud engineering, JavaScript, machine learning, and greatest hits. Whatever specific area of software you are curious about, we have a feed for you. Check the show notes for more details. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Instacart is a grocery delivery service. Customers log onto the website or mobile app and pick their groceries. Shoppers at the store get those groceries off the shelves. Drivers pick up the groceries and drive them to the customer. This is an infinitely complex set of logistics problems, paired with a rich data set given by the popularity of Instacart. Jeremy Stanley is the VP of data science for Instacart. In this episode, he explains how Instacart’s 4-sided marketplace business is constructed, and how the different data science teams break down problems like finding the fastest route to groceries within a store, finding the best path to delivering groceries from a store to a user, and personalizing recommendations so people can find new items to try. Are you looking for old episodes of Software Engineering Daily, but don’t know how to find the ones that are interesting to you? Check out our new topic feeds, in iTunes or wherever you find your podcasts. We’ve sorted all 500 of our old episodes into categories like business, blockchain, cloud engineering, JavaScript, machine learning, and greatest hits. Whatever specific area of software you are curious about, we have a feed for you. Check the show notes for more details. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The code in the Linux kernel changes all the time–11k lines are added, 5.8k lines are removed, and 2k lines are modified DAILY. Linux is an open source operating system that has been worked on for 25 years, and one reason the project is able to move so fast is its governance and release structure. Greg Kroah-Hartman is a fellow at the Linux Foundation, where he takes part in many of the developments in the kernel. This episode was a dive into how open source software gets built at scale and what is in store for the future. The Kubernetes project has drawn comparable attention to the size of Linux, and the Kubernetes project is learning how to manage open source from the Linux community. Check out our new topic feeds, in iTunes or wherever you find your podcasts. We’ve sorted all 500 of our old episodes into categories like business, blockchain, cloud engineering, JavaScript, machine learning, and greatest hits. Whatever specific area of software you are curious about, we have a feed for you. Check the show notes for more details. Kernel Development Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern software applications are often built out of loosely coupled microservices. These services can be written in different languages, by different people, but communication between services needs to be standardized. For this reason, a service proxy is useful. A service proxy is a sidecar container that sits next to a service and facilitates communications with other services. Once every service has a sidecar proxy, that sidecar can be used as a way to communicate with a centralized control plane. The sidecar can report telemetry data to the control plane, and the control plane can be used to set policies across services, such as rules for scaling and load balancing which might vary from service to service. Istio is an open platform to connect, manage, and secure microservices. Istio is a service mesh that uses Envoy service proxies. If all of this sounds confusing–don’t worry, we’ll explain it all in today’s interview with Varun Talwar and Louis Ryan, who work on Istio at Google. Check out our new topic feeds, in iTunes or wherever you find your podcasts. We’ve sorted all 500 of our old episodes into categories like business, blockchain, cloud engineering, JavaScript, machine learning, and greatest hits. Whatever specific area of software you are curious about, we have a feed for you. Check the show notes for more details. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Containers make it easier for engineers to deploy software. Orchestration systems like Kubernetes make it easier to manage and scale the different containers that contain services. The popular container infrastructure powered by Kubernetes is often called “cloud native.” On Software Engineering Daily, we have been exploring cloud native software to get a complete picture of the problems in the space, and the projects that are being worked on as solutions. One area of interest: how should services communicate with each other? What should be standardized? How can you make it easy to identify problems and avoid cascading failures? One solution is the service mesh, a tool that allows services to communicate with each other more safely and effectively. William Morgan was an engineer who helped scale Twitter in the early days when the company was dealing with lots of outages. He was on the show previously to discuss scaling Twitter, and in today’s episode we go into the company that he is running, Buoyant, where he works on building a service mesh called Linkerd. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com Scaling Twitter What’s a Service Mesh and Why Do I Need One? Buoyant is hiring: email william@buoyant.io Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Search engines run our lives. The path we take to information is dictated by Google, Facebook, Amazon, and other forms of search. Search engines feel objective and truthful, but are built through ongoing experimentation and subjective decision making. That’s what has kept Danny Sullivan writing about search engines for twenty years. The content Google prioritizes, the ads that we see, the way that a product review changes how highly a search result appears on a search; these are the topics that Danny studies. He is the founder of Search Engine Land, an invaluable resource for news and updates about search engines and marketing. I’ve been reading Search Engine Land since college, so it was a treat to sit down for a conversation with him. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Professional hackathon programmers travel around the hackathon circuit, winning merchandise and small cash prizes. There are enough hackathons that some programmers actually do this as a full-time job. For example, Peter Ma, a programmer who describes himself as a “rapid prototype specialist.” Peter is a great programmer, and he has received lots of offers to work at big tech companies. What drives him to stay independent and work on hackathon projects? There are other types of corporate hackathons. Many of us are familiar with the hackathon where a manager orders pizza and suggests that everyone stays at the office late fixing bugs. Some hackathons are held for kids, to get them exposed to technologies early on. Lizette Chapman is a reporter at Bloomberg, where she writes about technology, business, and news. I was fascinated by her story about hackathons, and it was great to have her on the show to talk about the characters of the hackathon circuit, and why corporations sponsor hackathons. Lizette has co-hosted the Bloomberg Decrypted podcast in an episode about hackathons. Decrypted is one of my favorite podcasts and I recommend checking it out. These Hackathon Hustlers Make Their Living from Corporate Coding Contests Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software Engineering Daily has been around for almost two years. In this episode Pranay Mohan and Erika Hokanson join me for a reflection on where we have been and where we are going. Pranay was the producer of Software Engineering Daily for the first year, after which he left and went to work at Snapchat. Erika joined the show 9 months ago to work on operations, ad sales, and expansion plans. The thesis of Software Engineering Daily has always been that serious, in-depth material about software provides value. Right now we are a podcast about software engineering. We are planning expansion into a larger media company with video, a mobile app, a desktop platform, more podcasts, and more journalism. Pranay, Erika, and I have crafted the vision for Software Engineering Daily. We want serious, inspiring, technical content to be more widespread. We know that you do too. If you have any suggestions you would like to see from us, you can always email me: jeff@softwareengineeringdaily.com. It’s already been 500 episodes and I am even more excited about Software Engineering Daily than I was when I began–looking forward to episode 1000. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software architecture address the challenge of communicating and navigating large, complex systems to stakeholders, both technical and non-technical. Over the years software architecture has gone in and out of fashion. Today we discuss why software architecture is important, what it means to have software architecture, and how to properly structure teams and incorporate architecture. Today’s show is guest hosted by David Curry. David sits down with Simon Brown to discuss the importance of having a common language for software systems. Simon is an independent consultant specializing in software architecture, he is the author of Software Architecture for Developers, and founder of Structurizr. If you are interested in hosting a show, check out softwareengineeringdaily.com/host Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A self-driving car needs to be able to quickly respond to changes in driving conditions. A factory needs to be able to quickly respond to changes in workplace safety. For these kinds of applications, we need processing power closer to the user of the application. If we put all of our application logic in the cloud, we will have to make a network round trip for every request. Servers in the cloud are powerful, but so are the computers at the edge–smartphones, sensors, drones, cars, and on-prem servers. Edge computing is giving us more computation outside the data center. Olivier Bloch works on Microsoft Azure IoT Edge, a set of services for edge computing. Azure IoT Edge includes on-prem versions of Microsoft Azure technologies. Tools that were previously accessible only in the cloud can be deployed and hosted on premise. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

John Looney spent more than 10 years at Google. He started with infrastructure, and was part of the team that migrated Google File System to Colossus, the successor to GFS. Imagine migrating every piece of data on Google from one distributed file system to another. In this episode, John sheds light on the engineering culture that has made Google so successful. He has very entertaining stories about clusterops and site-reliability engineering. Google’s success in engineering is due to extremely high standards, and a culture of intellectual honesty. With the volume of data and throughput that Google responds to, 1-in-a-million events are likely to occur. There isn’t room for sloppy practices. John now works at Intercom, where he is adjusting to the modern world of Google infrastructure for everyone. This conversation made me feel quite grateful to be an engineer in a time where everything is so much cheaper, so much easier, and so much more performant than it was in the days when Google first built everything from scratch. I had a great time talking to John, and hope he comes back on the show again in the future because it felt like we were just scratching the surface of his experience. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A data-driven organization is more efficient because the company can learn what to focus on. In this episode, Edaena Salinas from The Women in Tech Show interviews Rya Sciban, Product Manager at Periscope Data, who explains the needs of data teams in an organization. We talked about what data analysis is and how this changes as the amount of data grows. Rya explained what analytics clusters are and effective ways of sharing data between the organization. Periscope Data has been successful in retaining women in product and development teams. We talked about effective strategies for this and for having more women in leadership positions. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Deep learning allows engineers to build models that can make decisions based on training data. These models improve over time using stochastic gradient descent. When a model gets big enough, the training must be broken up across multiple machines. Two strategies for doing this are “model parallelism” which divides the model across machines and “data parallelism” which divides the data across multiple copies of the model. Distributed deep learning brings together two advanced software engineering concepts: distributed systems and deep learning. In this episode, Will Constable, the head of distributed deep learning algorithms at Intel Nervana, joins the show to give us a refresher on deep learning and explain how to parallelize training a model. Full disclosure: Intel is a sponsor of Software Engineering Daily, and if you want to find out more about Intel Nervana including other interviews and job postings, go to softwareengineeringdaily.com/intel. Intel Nervana is looking for great engineers at all levels of the stack, and in this episode we’ll dive into some of the problems the Intel Nervana team is solving. Related episodes about machine learning can be found here. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Modern architectures often consist of containers that run services. Those containers scale up and down depending on the demand for the services. These large software systems often use a technique known as event sourcing, where every change to the system is kept in an event log. When an event on the log is processed, several different data stores might be updated in response. In these architectures, containers are interacting with each other. Multiple databases are responding to events in the event log. To connect these systems together, engineers can write small functions to pass data around–you might call these small connecting functions “glue.” Glue functions are a great use for a serverless tool such as AWS Lambda or Google Cloud Functions. As these glue functions grow in popularity, there is an increased need for an open source way to deploy serverless functions. Sebastian Goasgoen works on Kubeless, a serverless execution tool built on top of Kubernetes. In this episode, we explore his take on the “serverless on Kubernetes” problem. This is a great companion episode to yesterday’s interview with Soam Vasani. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes is an orchestration system for managing containers. Since it was open sourced by Google, Kubernetes has created a wave of innovation in the infrastructure technology space. Another recent innovation has been the “serverless” execution tools–such as AWS Lambda and Google Cloud Functions. Serverless execution, otherwise known as functions-as-a-service, allows a developer to execute code against cloud servers without specifying which cloud servers they are executing on. Serverless execution is a cheap, flexible resource that any large company wants to have access to. But AWS Lambda and the other popular serverless tools are closed source. This led Soam Vasani to work on Fission, a serverless executor that sits on top of Kubernetes. If you have not heard about either Kubernetes or Serverless, you can check out our previous episodes about either topic. If you are familiar with the two topics, I think you’ll enjoy this episode, in which Soam explains the motivation for serverless on Kubernetes, and the architecture of Fission. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Microsoft’s past is full of stories. It’s early period of corporate domination in the 1990s was followed by a period of government antitrust scrutiny, and a period of unsure product direction. Today, Microsoft’s focus on cloud has allowed the company to regain its footing with a clear trajectory for growth. Since 2002, Richard Campbell has chronicled the Microsoft developer community as co-host of .NET Rocks!, a podcast that was originally about the C# .NET framework. Richard also founded Humanitarian Toolbox, an open source set of tools for assisting disaster relief organizations. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes makes it easier for engineering teams to manage their distributed systems architecture. But it’s still not simple to deploy and operate a Kubernetes cluster. Google Container Engine (GKE) is a managed control plane for Kubernetes. Just as developers can use Google App Engine to easily deploy monolithic apps against a platform as a service, we can use Google Container Engine to deploy microservices against a platform as a service. David Aronchick and Chen Goldberg work on Google Container Engine, and they join the show to explain why platform as a service container engines are useful. Google is not the only cloud provider with a container engine–Amazon ECS and Azure Container Engine also allow you to run containers in a managed fashion. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Engineers have plenty to be skeptical about. We look to data sets to give us something resembling objective truth. Some areas of research have so many variables that it is hard to isolate facts. Kyle Polich hosts the popular data science show Data Skeptic, where he examines problems and solutions around data, and he is one of the guests today in our round table discussion. There are some big unanswered questions in our world that might eventually be solved with enough data and the right scientific approach: nutrition, or drug discovery, or image classification. The hiring process is like this. How can you predict whether an engineer will make for a good hire? Ammon Bartram of Triplebyte is working on solving the hiring process for engineering organizations and he is the other guest for this roundtable episode. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

DNS stands for domain name system. This is the naming system that maps the entire internet. It associates information with domain names. More specifically, DNS specifies mappings between numerical IP addresses and domain names. Most engineers know these basic facts about DNS, but they may not know how much engineering a complex company like Etsy or Zappos puts into their DNS configuration. Dynamic DNS allows for intelligent response, so that a resource is served from the most efficient place–even in the face of a DDoS attack, or just routine failure of cloud servers. Phil Stanhope is the VP of technology at Oracle Dyn and he joins the show to explain how modern DNS works and the role of a DNS provider. Full disclosure: Dyn is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Video object segmentation allows computer vision to identify objects as they move through space in a video. The DAVIS challenge is a contest among machine learning researchers working off of a shared dataset of annotated videos. The organizers of the DAVIS challenge join the show today to explain how video object segmentation models are trained and how different competitors take part in the DAVIS challenge. A good companion to this episode is our discussion of Convolutional Neural Networks with Matt Zeiler. Software Engineering Daily is looking for sponsors for Q3. If your company has a product or service, or if you are hiring, Software Engineering Daily reaches 23,000 developers listening daily. Send me an email: jeff@softwareengineeringdaily.com Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

On January 31st 2017, GitLab experienced a major outage of their online repository hosting service. The primary database server experienced data loss due to a combination of malicious spam attacks and engineering mistakes that occurred while trying to respond to those spam attacks. GitLab responded to the event transparently. The company put up a postmortem describing the event in detail. In subsequent posts, GitLab expressed sympathy for the employee who made engineering mistakes that led to the deletion of data. The employee was not judged or disciplined for an understandable error. The response from the developer community was very positive. Engineers know that building cloud services is hard. Engineering is as much about avoiding errors as it is about appropriately responding to the inevitable mistakes. GitLab is a developer platform that combines repository hosting with several other features–issue tracking, code review, and CD. Today’s guest is Pablo Carranza, who works on infrastructure at GitLab. In this episode, he walks us through GitLab’s product, the engineering stack, and a postmortem of the outage. We also discuss working at Amazon, and the importance of postmortems, which I first encountered at Amazon. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Different databases have different access patterns. Key-value, document, graph, and columnar databases are useful under different circumstances. For example, if you are a bank, and you have a database of customers and the transactions they have performed, the ideal access pattern for aggregating the total amount of all transactions might be a columnar store. If the transaction amounts are all in one column, it helps to have all of the columnar entries close together on disk. But if you want to look at your bank as a social network, and you want to be able to map how money flows between the different people who use your bank, you might want to map your data as a graph database. That would make it easier to query for the connections across the different users in the bank. CosmosDB is a database from Microsoft that allows for multiple data models and multiple well-defined consistency models. Today’s guest Andy Hoh is a product manager at Azure CosmosDB and he joins the show to describe the product. Microsoft unveiled CosmosDB at Build, their annual developer conference, which is where I performed this interview. It was a pleasure hanging out at Build in the podcast booths they set up, so thanks to Microsoft for inviting me. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

With a fast-growing field like data science, it is important to keep some amount of skepticism. Tools can be overhyped, buzzwords can be overemphasized, and people can forget the fundamentals. If you have bad data, you will get bad results in your experimentation. If you don’t know what statistical approach you want to take to your data, it doesn’t matter how well you know Spark or TensorFlow. And if you aren’t passionate about the work you are doing, you are unlikely to finish the projects you start (whether we are talking data science or otherwise). Kyle Polich hosts the Data Skeptic podcast, a show at the intersection of data science and skepticism. As a podcaster, Kyle takes himself seriously and is prepared for his shows–which I admire. Having met him recently at the Microsoft Build conference, he’s a great guy and I look forward to doing more podcasts with him in the future. In this episode, Kyle is interviewed by Sid Ramesh, a data engineering correspondent for Software Engineering Daily. https://dataskeptic.com http://openhouseproject.co Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apple controls the iOS ecosystem. As an accident of history, Apple also controls the podcasting ecosystem. Unlike most ecosystems within Apple’s dominion, podcasts remain open. A podcaster merely has to record an mp3, distribute it via RSS feed, and submit that RSS feed to the iTunes podcast portal. Podcasting has thrived in recent years, but very few technology companies have managed to take advantage of that growth. Libsyn is the most popular place to host a podcast. Libsyn is a combination of a CDN, a hosting service, analytics, and place to get an RSS feed for a podcaster. There have been many clones of Libsyn over the years, but the company remains the industry standard. For people who are confused–iTunes does not host any audio files. It is just an index of feeds. A podcaster needs to host audio files somewhere in order to give iTunes access. Today’s guest Rob Walch joins the show to talk about podcasts–including his podcast Today in iOS. I had a great time meeting Rob at the Microsoft Build conference. Special thanks to Bharat Bhat for organizing the podcast booths at Microsoft Build. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Social networks like Facebook and Twitter facilitate interactions between individuals. Every message I send to you on Facebook goes through Facebook’s servers before reaching you. This is known as the client-server model. Since the early days of the internet, engineers have always envisioned a peer-to-peer model, where I could communicate to you directly, without a company brokering that relationship. Andre Staltz works on Scuttlebutt, a peer-to-peer system for social graphs, identity, and messaging. Scuttlebutt is used by a group of open-source hackers, many of whom live off-grid and do not have constant access to the Internet. In this episode, we discuss why someone would want a peer-to-peer social network, how to build one, and the progress that has been made on Scuttlebutt by some of the most talented open-source engineers in the world. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Everyone in the world should have some basic level of guaranteed healthcare. This is not controversial. But what should that basic level of healthcare be? Should it extend into the later years of your life, when the majority of your health costs are incurred? And how much has modern technology driven down the cost of what it should cost to treat a patient? Healthcare today has lots of problems with bureaucracy and poorly aligned incentives. But the potential of vastly better healthcare is clear to technologists, and advances in software in hardware that have benefited other enterprises will eventually make their way into healthcare–reducing cost, improving oversight, and leading to better health. Watsi is a non-profit with the goal of seeing a world with universal healthcare. Watsi facilitates crowdfunding of patients who need low-cost, high-impact treatment. Y-Combinator Research recently funded a study in collaboration with Watsi to study using technology to improve the quality and reduce the cost of healthcare. Thomas Bukowski is a software engineer with Watsi, and he joins me for an interview about what universal healthcare means and what the roadmap to getting there might look like. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Relay is a JavaScript framework for building data-driven React applications. Facebook open sourced Relay around the same time they open sourced GraphQL, and Facebook expected Relay to be the more popular of the two projects. However, the reality was reversed. Open source companies like Meteor quickly began to build GraphQL tools and a few businesses were started around GraphQL. One year later, the excitement for GraphQL had completely surpassed the excitement for Relay which had aged poorly in a newborn ecosystem of GraphQL tooling. At the same time, Facebook was also starting to integrate Relay into their React Native apps. But Relay was performing poorly on low-end Android devices. This led the Relay team to the conclusion that they needed to rewrite Relay. Both to better fit into the growing GraphQL ecosystem, and to be built with performance in low-end React Native environments at the top of mind. Relay Modern is the new version of Relay. It was released to the open source community at this year’s F8. In this episode Caleb Meredith is joined by Lee Byron, the co-creator of GraphQL, and Joe Savona, a founding member of the Relay team, to discuss Relay Modern. The discussion includes a conversation about the commercial GraphQL ecosystem, the story of why Facebook decided Relay needed to be rewritten, and a look at the future of UI development from some trends seen in Relay Modern. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Sponsors Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Healthcare is a complex business. Oscar is a company that wanted to build a new insurance provider–but realized that healthcare is so interconnected that in order to build a new insurance provider, realized it actually needed to build an entire healthcare business too, complete with patient management and facilities. Since Oscar is a modern technology company, the focus on customer service, engineering, and data management offers an optimistic view into what healthcare might look like in the near future. Every time a patient interacts with the healthcare system, their insurance provider collects data on that interaction. Isaac Councill helped architect the infrastructure at Oscar that manages and analyzes the patient data. In this show, we talk about the healthcare system, data engineering, and Apache Mesos, which Oscar uses to manage its applications. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Many companies are transitioning from a monolith to microservices architecture. Tools for cloud computing, containerization, and continuous delivery are making this easier. But there are still technological and organizational challenges that a company will encounter while making this transition. Cassandra Shum is an engineer with ThoughtWorks. She has worked with major financial institutions and other large companies to architect their migrations from monolith to microservices. Also, she regularly puts on workshops to engineers who are seeking to make this migration at the company they work at. In this episode, she describes some of her experiences and recommendations around transitioning from a monolith to microservices. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud computing changed how we develop applications for the web. Over the last decade, engineers have been learning how to build software in this new paradigm. The costs have gone down, but our nodes can fail at any time. We no longer have to manage individual servers, but the layers of virtualization and containerization require new strategies for communicating between services. As we have adjusted to this new way of building applications, the term “cloud-native” has become a useful descriptor. Cloud native applications are modern distributed systems capable of scaling to tens of thousands of self healing multi-tenant nodes. Open source projects such as Kubernetes, Prometheus, and Linkerd draw on the lessons of big technology companies like Google, Twitter, and SoundCloud. Engineers today don’t need to reinvent the key infrastructure of those companies. We can combine the open source cloud-native technologies and use them to build powerful systems. Dan Kohn is the executive director of the Cloud Native Computing Foundation, and he joins me for an interview about the projects within the CNCF, how they fit together, and the future of computing. Stay tuned at the end of the episode for Jeff Meyerson’s tip about job searching brought to you by Indeed Prime. Measuring the Popularity of Kubernetes Using BigQuery Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Public data is not always so accessible. It is nice when you can request data simply by making an API call, but that is the exception rather than the rule–especially when we are talking about data managed by the government. Oil and gas drilling data falls into this category. Oseberg is a company that is building a tool for analyzing oil and gas data. Oseberg is a rich dashboard for knowledge workers to query and visualize the data. Evan Anderson is the CEO of Oseberg, and he joins me to discuss building a business where the data is hard to acquire. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Firebase is a backend-as-a-service. The key efficiency of a backend-as-a-service is that it enables developers to go from having a 3-tier architecture (client, server, database) to a 2-tier architecture (client, backend-as-a-service). The team who started Firebase built it as a pivot. They had started a social network, and then they realized there wasn’t a good backend for chat tools. And so they started a chat-as-a-service tool, for people who wanted to include chat in their applications. And that led them to the fundamental realization that chat is actually representative of a broader category of real-time synchronization problems. Firebase was eventually acquired by Google. Doug Stevenson is a senior developer advocate with Google and the host of Meet Firebase, a YouTube talk show about Firebase. It was a pleasure to sit down for a conversation with him, especially because I recently started using Firebase in my own application as a backend for real-time chat. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When you purchase an ebook you must agree to the Terms of Service that tell you what you can do with it. What is actually in that terms of service? What are you agreeing to when you buy an ebook? The answers might surprise you. In this episode, Srini Kadamati interviews Chris Groskopf on how the rise of digital products has eroded the idea of traditional ownership. They discuss digital ownership from the point of view of the legal system, consumers, and the companies creating these products. Chris Groskopf is a data journalist who uses data, graphics, and storytelling to build compelling news experiences. He’s worked on multiple pioneering teams at organizations like the Chicago Tribune and NPR. He’s currently the first data editor at Quartz, a digital-first Atlantic publication, where he’s written about how complex systems like the stock market fail and how most of the world’s art is locked away in museums. Outside of journalism, he’s created multiple Python data libraries, like agate, proof, and csvkit. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Artsy is an online art marketplace. This might sound like a simple engineering problem–you just set up a basic ecommerce site, list some pieces of art, and start making money, right? The art world is complicated. There are four major pillars: patrons, art fairs, galleries, and auctions. Bringing these different parts online is not trivial. And in order to do so, Artsy has to work with the existing ecosystem. It is not like the taxi industry, where you can aggressively compete against the pre-existing businesses. The art world is built around relationships and trust. The engineering is hard too. An art auction results in a transaction for millions of dollars. In this way, building an auction system is like building a trading system. Latency needs to be very low, and you can’t make any mistakes or else customers could suffer to the tune of millions of dollars. Daniel Doubrovkine is the CTO of Artsy, and he joins me to describe the complexities of the art market and the engineering challenges that come with building a software company around it. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Humans have now been defeated by computers at heads up no-limit holdem poker. Some people thought this wouldn’t be possible. Sure, we can teach a computer to beat a human at Go or Chess. Those games have a smaller decision space. There is no hidden information. There is no bluffing. Poker must be different! It is too human to be automated. The game space of poker is different than that of Go. It has 10^160 different situations–which is more than the number of atoms in the universe. And the game space keeps getting bigger as the stack sizes of the two competitors gets bigger. But it is still possible for a computer to beat a human at calculating game theory optimal decisions–if you approach the problem correctly. Libratus was developed by CMU professor Tuomas Sandholm, along with my guest today Noam Brown. The Libratus team taught their AI the rules of poker, they gave it a reward function (to win as much money as possible), and they told it to optimize that reward function. Then they had Libratus train itself with simulations. After enough training, Libratus was ready to crush human competitors, which it did in hilarious, entertaining fashion. There is a video from Engadget on YouTube about the AI competing against professional humans. In this episode, Noam Brown explains how they built Libratus, what it means for poker players, and what the implications are for humanity–if we can automate poker, what can’t we automate? Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Many countries in the developing world are undergoing a technological revolution which is shaping how they tackle problems around infrastructure, health, education and finance. Young people are at the forefront of developing solutions to the problems in the developing world. These young people creating technology and businesses to foster innovation and growth. Countries in the Middle East are no exception to this. Despite the difficulties the region faces such as the wars in Syria and Yemen, its populations are well-informed, plugged in and are using technology to build the future of their societies. In the book Startup Rising published in 2013, Washington D.C.- and New York City-based entrepreneur and venture investor Chris Schroeder wrote about how the rest of the world, especially the West, needed to wake up to what was happening in the Middle East. In today’s episode, Chris talks to Carl Mungazi about what he has seen throughout his travels in the Middle East. He discusses the successes and challenges faced by the entrepreneurs who shared their stories with him and what this means for those watching in the West. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Convolutional neural networks are a machine learning tool that uses layers of convolution and pooling to process and classify inputs. CNNs are useful for identifying objects in images and video. In this episode, we focus on the application of convolutional neural networks to image and video recognition and classification. Matt Zeiler is the CEO of Clarifai, an API for image and video recognition. Matt takes us through the basics of a convolutional neural network–you don’t need any background in machine learning to understand the content of the episode. He also discusses the subjective aspects of image and video recognition, and some of the tactics Clarifai has explored. This is far from a solved problem. Matt also discusses the infrastructure of Clarifai–how they use Kubernetes, how models are deployed, and how models are updated. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view the transcript for this episode. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software consultancies solve problems involving management and software engineering. A large company might hire a software consulting company to give an external opinion on software architecture, or on an organizational structure. Sometimes a consultancy is brought in to help integrate a new technology, or do a major refactoring. Scaling a software consultancy to meet the varying demands of clients presents a unique challenge. Software companies that make money from media, or software-as-a-service, or advertising technology are primarily focused on scaling the technology. For a software consulting business, scaling and updating the team is arguably more important than any particular piece of software. Rachel Laycock is the head of technology for ThoughtWorks North America. She joins the show to discuss how to manage and grow a large software consulting organization. It’s a great discussion of culture, technology, and how the nature of work is changing. If you are interested in hosting a show for Software Engineering Daily, we are looking for engineers, journalists, and hackers who want to work with us on content. It is a paid opportunity. Go to softwareengineeringdaily.com/host to find out more. The Software Engineering Daily store is now open if you want to buy a Software Engineering Daily branded t-shirt, hoodie, or mug and support the show. Let’s get on with the show. Go to softwareengineeringdaily.com/store. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Suddenly, automation is changing our world faster than anyone anticipated. For technologists, the world is becoming convenient and high-leverage. For non-technologists, the job market is evaporating. Haseeb Qureshi and Quincy Larson join me for a roundtable discussion on automation, jobs, and artificial intelligence. Haseeb and I have had numerous discussions about this topic before, and Quincy is the founder of Free Code Camp, which teaches people to learn programming for free. If there is one upside of all these jobs being automated away, its that it will lead to massive user growth for Free Code Camp. I enjoyed talking to Quincy and Haseeb as always. If you are interested in hosting a show for Software Engineering Daily, we are looking for engineers, journalists, and hackers who want to work with us on content. It is a paid opportunity. Go to softwareengineeringdaily.com/host to find out more. The Software Engineering Daily store is now open if you want to buy a Software Engineering Daily branded t-shirt, hoodie, or mug and support the show. Let’s get on with the show. Go to softwareengineeringdaily.com/store. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

CRISPR is a technique for altering the human genome. It might be the most powerful tool for biological modification that we have ever discovered. In this episode, we explore CRISPR: how it works, why it exists in the natural world, and the implications for being able to modify DNA so easily. Geoff Ralston is a partner at Y-Combinator. He wrote an article entitled Hacking DNA: The Story of CRISPR, Ken Thompson, and the Gene Drive. Since Geoff is not a biologist, he is the perfect person to explain CRISPR to an audience of non-biologists. Since he is an investor, he is also great at explaining the pace at which CRISPR might make it to market, and how it might converge with some of the other futuristic trends we are seeing so regularly today. If you are interested in hosting a show for Software Engineering Daily, we are looking for engineers, journalists, and hackers who want to work with us on content. It is a paid opportunity. Go to softwareengineeringdaily.com/host to find out more. The Software Engineering Daily store is now open if you want to buy a Software Engineering Daily branded t-shirt, hoodie, or mug and support the show. Go to softwareengineeringdaily.com/store. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Washington Post was acquired by Amazon CEO Jeff Bezos in 2013. Since then, the newspaper has started thinking more like a software company, opting to build new software rather than buy off-the-shelf third party solutions. Arc Publishing is a CMS built by The Washington Post to produce and display content. When you visit washingtonpost.com, you are viewing a site built with Arc Publishing. The Washington Post has also brought its advertising technology in-house. Jarrod Dicker is the head of commercial product and technology at The Washington Post. He joins the show to discuss the transformation that has occurred since Bezos purchased the company. We also explore the problems in digital advertising that have been covered in recent episodes of Software Engineering Daily. Thanks to listener Mani Gandham for introducing me to Jarrod, and if you have a suggestion for a guest you want to hear, please send me an email. If you are interested in hosting a show for Software Engineering Daily, we are looking for engineers, journalists, and hackers who want to work with us on content. It is a paid opportunity. Go to softwareengineeringdaily.com/host to find out more. The Software Engineering Daily store is now open if you want to buy a Software Engineering Daily branded t-shirt, hoodie, or mug and support the show. Let’s get on with the show. Go to softwareengineeringdaily.com/store. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

There are certain experiences when a product solves a problem so thoroughly and elegantly that it lifts a weight off of your shoulders that you didn’t even know was there. Dropbox did this with file storage. Slack did this with group collaboration. Zencastr does this for recording podcasts. Before I used Zencastr to record my podcasts, like most podcasters, I used a Skype plugin. There were a number of inconveniences in the podcaster workflow from Skype. Zencastr solved all of these by creating a podcast recording tool in the browser and presenting a simple user interface. Josh Nielsen joins the show to talk about the challenges of building a podcasting tool in the browser, and the new technologies that make it easier, such as WebRTC. We would love to get your feedback on Software Engineering Daily. Please fill out the listener survey, available on softwareengineeringdaily.com/survey. Also–Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to download the transcript for this episode. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Listeners have had difficulty finding the Software Engineering Daily content they want to listen to. We are creating new podcast feeds to address this. The content on each podcast feed is mutually exclusive from the other feeds, except for the main feed and “Greatest Hits.” You can now find the following podcast feeds in iTunes and Google Play: Every new episode goes into 2 feeds: An episode can potentially be in 3 feeds if it is also in Greatest Hits. If you subscribe to all of the feeds including Software Engineering Daily (main feed), you will receive 2 notifications for every new episode. With the different feeds, we hope you can curate your own ideal SE Daily listening experience. Thanks for listening to Software Engineering Daily and please let us know what you think of the new feeds. A few recommendations: We hope the rollout of these new feeds goes smoothly. If you have issues, please email me, notify us in Slack, or tweet at us. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A new programmer learns to build applications using data structures like a queue, a cache, or a database. Modern cloud applications are built using more sophisticated tools like Redis, Kafka, or Amazon S3. These tools do multiple things well, and often have overlapping functionality. Application architecture becomes less straightforward. The applications we are building today are data-intensive rather than compute-intensive. Netflix needs to know how to store and cache large video files, and stream them to users quickly. Twitter needs to update user news feeds with a fanout of the president’s latest tweet. These operations are simple with small amounts of data, but become complicated with a high volume of users. Martin Kleppmann is the author of Data Intensive Applications, an O’Reilly book about how to use modern data tools to solve modern data problems. His book includes high-level discussions about architectural strategy, and lower level discussions like how leader election algorithms can create problems for a data intensive application. If you are interested in hosting a show for Software Engineering Daily, we are looking for engineers, journalists, and hackers who want to work with us on content. It is a paid opportunity. Go to softwareengineeringdaily.com/host to find out more. The Software Engineering Daily store is now open if you want to buy a Software Engineering Daily branded t-shirt, hoodie, or mug and support the show. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Most popular music today uses a computer as the central instrument. A single musician is often selecting the instruments, programming the drum loops, composing the melodies, and mixing the track to get the right overall atmosphere. With so much work to do on each song, popular musicians need to simplify–the result is that pop music today consists of simple melodies without much chord progression. Magenta is a project out of Google Brain to design algorithms that learn how to generate art and music. One goal of Magenta is to advance the state of the art in machine intelligence for music and art generation. Another goal is to build a community of artists, coders, and machine learning researchers who can collaborate. Engineers today are happy to outsource server management to a cloud service provider. Similarly, a musician can use Magenta for creation of a melody, so she can focus on other aspects of a song, such as instrumentation. Doug Eck is a research scientist at Google. In today’s episode, we explore the Magenta project and the future of music. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. We would love to get your feedback on Software Engineering Daily. Please fill out the listener survey, available on softwareengineeringdaily.com/survey. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to download this show’s transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

We view our iPhones as inanimate objects. But when we see robots such as the Boston Dynamics machines that move with a motion that seems like an animal, the robot comes alive. We feel more sympathy and connection towards it. Today’s episode is about the distinction between inanimate machines and machines that seem alive. Peeqo is a robot assistant similar to Amazon Echo or Google Home. It was built by Abhishek Singh as part of his thesis. Abhishek wanted to explore the border between inanimate electronics and the electronics we personify. Peeqo is a cylindrical tube that bends slightly, like the human neck. At its head, it has a smartphone-sized screen that displays gifs to reflect its feedback visually. Abhishek also has a company called Svrround, which makes 360 degree video experiences. We explore the changes to engineering that allow someone to be involved in two cutting-edge projects at once despite having very few employees working with him. This was an energizing conversation, and I greatly enjoyed meeting Abhishek. I hope to have him on again in the future. We would love to get your feedback on Software Engineering Daily. Please fill out the listener survey, available on softwareengineeringdaily.com/survey. Also–Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ransomware uses software to extort people. A piece of ransomware might arrive in your inbox looking like a PDF, or a link to a website with a redirect. Ransomware is often distributed using social engineering. The email address might resemble someone you know, or a transactional email from a company like Uber or Amazon. Tim Gallo and Allan Liska are authors of the O’Reilly book Ransomware: Defending Against Digital Distortion. They join me to describe the 5 stages of ransomware: deployment, installation, command and control, destruction, and extortion. Tim and Allan describe conditions under which it might make sense to pay the extortion, and some frightening recent cases of ransomware impacting the real world. We would love to get your feedback on Software Engineering Daily. Please fill out the listener survey, available on softwareengineeringdaily.com/survey. Also–Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to download the Ransomware transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Spring Framework is an application framework for Java and JVM languages. Spring was originally built around dependency injection, but grew to become an entire ecosystem of tools and plugins for Java developers. Spring was originally released 15 years ago, and since then a lot has changed around application development. For example, many engineers deploy applications to the cloud in microservices architectures. The expectations around frameworks has also changed, with the rise of Django, Ruby on Rails, and NodeJS. Spring Boot takes an opinionated view of building production-ready Spring applications. By taking an opinionated view, Spring Boot gets engineers up and running faster than the traditional Spring framework. Josh Long is a Spring Developer Advocate at Pivotal and he joins the show to discuss Spring Boot and the history of the Spring Framework. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. We would love to get your feedback on Software Engineering Daily. Please fill out the listener survey, available on softwareengineeringdaily.com/survey. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to download the Spring Boot transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The online advertising industry is a giant casino. Giant technology companies are the casino owners, online publishers are the casino employees, the brand advertisers are the victims who keep returning to the casino to lose their money, and the small adtech companies are the sharks who make lots of money exploiting the inefficiencies of the system. One of these smaller adtech companies is called eZanga. eZanga sells “pre-filtered traffic.” Pre-filtered traffic means traffic that will pass through bot detection filters. A publisher can purchase traffic to their website so that the ads on that website get viewed. eZanga describes this technology as “marketing,” and has won a giant contract with United States government to handle advertising for the GSA. Advertising fraud does not just promote misinformation–it is now taking our tax dollars and spending it on paid traffic. If any of this is confusing to you, don’t worry. We explain it all in today’s episode with Shailin Dhar, the advertising fraud expert who wrote a detailed report about eZanga and its contract with the US Government. Shailin was previously on the show to give an overview of ad fraud, and what his work as an ad fraud investigator entails. Also, Shailin will be a speaker at our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Ad Fraud in Our Own Backyard: The Dhar Method Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to download the eZanga transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software Engineering Daily examines the world through the lens of software engineering. In most episodes, an expert in a particular topic joins the show as a guest, and we go into deep technical detail. Occasionally we like to do episodes where we survey a collection of topics. In today’s topic roundtable, Caleb Meredith and Courtland Allen join me for a discussion of several questions: would it make sense for Facebook to build an operating system? Does online advertising work? How can you work productively on an engineering company with your brother as a co-founder? Courtland is the founder of IndieHackers.com, which was recently acquired by Stripe. Caleb Meredith is the lead JavaScript correspondent of Software Engineering Daily. It was a blast talking to both of them, and we plan to do more round table episodes in the future. We would love to get your feedback on Software Engineering Daily. Please fill out the listener survey, available on softwareengineeringdaily.com/survey. Also–Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to download the Topic Roundtable transcript. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Bitcoin is underappreciated even to this day. The public focus is usually on the speculative value, but Bitcoin has functional value as a technology platform. If I want to make 100 transactions with my bank for 1 cent, the bank won’t allow it. Our current financial infrastructure is not set up for micropayments. Bitcoin is built with micropayments in mind. As Bitcoin works through its governance issues and its scalability problems, we will see gradual improvement in financial liquidity between people and machines. 21 is a company that has raised $120M to make Bitcoin useful to developers. This is a long term project, and the first step of that project is to get Bitcoin in the hands of users. To fulfill that end, 21 is developing services that encourage people to make small digital transactions. The first service is the 21 messaging service, where users can pay to send messages to people who are unlikely to respond to an unsolicited email otherwise. For example, if I want to send an email to a venture capitalist pitching my company, I am more likely to get a response if I pay that venture capitalist $20 to read my message through 21, rather than if I sent a cold email from my email address. Balaji Srinivasan is the CEO of 21, and he joins me for a conversation about the potential of Bitcoin and the objectives of the company he is building. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The word “microservices” started getting used after a series of events–companies were moving to cloud virtual machines. Those VMs got broken up into containers, and the containers can fit to the size of the service. Services that are more narrowly defined take up smaller containers, and can be packed more densely into the virtual machines–hence the term “microservices.” As this change to software architecture has occurred, the DevOps movement has encouraged organizations to have better relationships between development and operations. Continuous deployment leads to fewer painful outages. Improved monitoring tools make it easier for developers to take on some of the pain that was previously centralized in operations. Several months ago, I attended the Microservices Practitioner Summit, which brings together engineers who are working with microservices at their companies. The conference was organized by Austin Gunter and Richard Li of Datawire. In this episode, they joined me for a conversation about microservices. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Swift is a language that is most commonly used to write apps for Apple client devices, such as iPhones. Since being released in 2014, Swift has become one of the most popular languages due to its high performance and developer ergonomics. In 2015, Swift was open sourced, creating the opportunity for Swift to be used outside of the Apple ecosystem. If you write an iPhone app today, your frontend is in Swift and your backend is probably in NodeJS, Java, or Ruby. Engineers are working to port Swift to the server so that the Swift developer experience is isomorphic: the same language on the backend and the frontend. Chris Bailey is an engineer at IBM working on Kitura, a Swift web framework. In this episode, we discuss the history of Swift, why it is so appealing to developers, and why Swift could become a server side language with as much popularity as Java. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Most people in Africa never had a desktop computer. The first computer they owned was a smart phone. This is why Africa is referred to as a “leap frog” place with regard to computers–Africa leapfrogged the desktop to the smart phone. The banking system in Africa also followed a trajectory that is different than the West. Westerners are used to banking on their desktop computers. African e-commerce has developed around the smartphone as the computer for banking. As a result, Africa is in a technological development phase with tremendous opportunities in the financial sector and beyond. Mymookh.com is a Kenyan end-to-end payment platform that allows small- to medium-sized businesses to easily set up online stores and sell to consumers directly from their social media pages. It is one of the many startups across Africa leveraging the accessibility of mobile payments to meet the particular needs of merchants and consumers there. Africa’s development and use of mobile payment systems is expected to grow as more countries on the continent create and adopt this technology. In this episode, George Gachui, co-founder of Mymookh, talks to Carl Mungazi about how Kenya became a leading player in the mobile payment space with its m-pesa system. He explains how it has revolutionized the way Kenyans approach commerce and payment technology and discusses the challenges of creating a unified platform system that transcends borders and currencies. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Bots on the internet can be malicious, helpful, and everything in between. A bot that responds to all of your tweets might call you a socialist–that is malicious. Google crawls the web to index Google search. That is helpful. Social media marketing bots schedule 200 Twitter posts to go out throughout the day. That is either a little annoying or a little helpful depending on who you are. Bots are being used to amplify political viewpoints. An amplified viewpoint can serve as a gravity well for like-minded individuals, and help a sparsely supported political cause find its footing. Sometimes that amplified viewpoint is completely fictional or unfalsifiable. Real people believe that Hillary Clinton is a lizard alien because they have seen that story shared by enough Twitter bots. So-called “fake news” is a topic that has been discussed on so many other podcasts. What is not reported is the connection between link bait and advertising fraud. When a botnet is able to make an article go viral, thousands of people organically click on the link to that article. That organic traffic is used to launder fake clicks. These bots that are spreading “fake news” might be controlled by conspiratorial Russians. But it doesn’t have to be that complicated. Anyone who wants to make money in online advertising fraud is incentivized to make salacious media–whether it is real or fake. Samuel Woolley is the director of research at Political Bots. He works with Jigsaw, a division of Alphabet that seeks to make the Internet safer. In today’s episode, we talk about political bots, advertising fraud, and the connection between the two. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook’s open source projects include React, GraphQL, and Cassandra. These projects are key pieces of infrastructure used by thousands of developers–including engineers at Facebook itself. These projects are able to gain traction because Facebook takes time to decouple the projects from their internal infrastructure and clean up the code before releasing them into the wild. Facebook has high standards for what they are willing to release. Tom Occhino manages the React team at Facebook and works closely with engineers to determine what projects make sense to open source. In this episode, Preethi Kasireddy interviews Tom about how Facebook thinks about open source–what went right with React, why it makes sense for Facebook to continue to release new open source projects, and how full-time employees at Facebook interact with that open source codebase. We would love to get your feedback on Software Engineering Daily. Please fill out the listener survey, available on softwareengineeringdaily.com/survey. Also–Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Pete Hunt: React: Rethinking best practices — JSConf EU 2013 Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Internet is decreasing in privacy and increasing in utility. Under some conditions, this tradeoff makes sense. We publicize our profile photo so that people know what we look like. Under other conditions, this tradeoff does not make sense. We do not want a television that costs less to purchase because it is silently recording all of the conversations that take place in the room and selling them to the highest bidder. The example of the TV that records everything you say (which is a real thing) illustrates a tradeoff of the Internet. The advertising industry pushes us towards lower marginal costs for products and services in exchange for less privacy. Someday we will live in a world where it will be easy for consumers to control the dial on the tradeoff between privacy and the price of their services. Until then, we have almost zero control over what information the advertising surveillance industrial complex knows about us. Bill Budington is a security engineer with the Electronic Frontier Foundation. In today’s episode, Bill describes some of the current techniques used by the advertising industry to track your activity through the web. Bill works on encryption tools as well as Panopticlick, a project that allows users to see what trackers they are vulnerable to. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Search is a common building block for applications. Whether we are searching Wikipedia or our log files, the behavior is similar: a query is entered and the most relevant documents are returned. The core data structure for search is an inverted index. Elasticsearch is a scalable, resilient search tool that shards and replicates a search index. Philipp Krenn from Elastic joins the show today to discuss how search works and how Elasticsearch scales. We use Wikipedia as a running example for how a query is processed and how documents are stored. If you’ve ever wondered how search works–or if your company uses Elasticsearch and you want to know more about it–this is a great episode for you. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

React Native has unlocked native mobile development to web engineers who may now apply their skills to build iOS and Android applications in JavaScript. For the first time, cross platform JavaScript-based applications feel as if they were written in the native language of choice for the platforms. Businesses who choose to adopt React Native for their native app development also see great benefits such as the ability to push new JavaScript code without going through the app store review process, and the ability to share code and business behaviors across the iOS and Android platforms. Expo is building a cross-platform native runtime for React Native. Expo brings the benefits of deployment and iterative development to native without sacrificing any user experience costs. Expo plans to do this with their native SDK, custom development environment, and tools built in collaboration with Facebook like create-react-native-app. React Native has the incredible potential to revolutionize all user interface development with its core set of cross-platform UI primitives, and React’s popular declarative rendering pattern. So in this episode Brent Vatne and Adam Perry join Caleb Meredith to first discuss Expo and the future of React Native to try and answer the question: can React Native become the one UI framework to rule them all? Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every program gets compiled down to 1s and 0s before it can be executed against hardware. Before being translated to machine code, programs that are written in a language like Rust, Swift, or Java spend time in an intermediate representation. In Java, this intermediate representation is Java bytecode. Many different languages–such as Scala–translate to Java bytecode, because there has been lots of optimization written to speed up Java bytecode. Java bytecode runs on the JVM–the Java Virtual Machine. LLVM is a project that draws inspiration from the Java Virtual Machine. LLVM originally meant “low level virtual machine” but today it is just called LLVM and describes a set of compiler tools. In today’s interview with Morgan Wilde, we explore how compilers work, how different processor hardware architectures present a problem for compilers, and why LLVM’s intermediate representation creates a layer of interoperability for any language that compiles down to that intermediate representation. Whether you are new to compilers or have experience, this episode will appeal to you. Morgan is an excellent teacher and his enthusiasm for the subject comes through. He has a 30-minute YouTube video–A Brief Introduction to LLVM that I highly recommend. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Engineers in Silicon Valley see a world of constant progress. Our work is creative and intellectually challenging. We are building the future and getting compensated quite well for it. But what if we are actually achieving far less than what is possible? What if, after so many years of high margins, gourmet lunch, and self-flattery, we have lowered our standards for innovation? And if Silicon Valley has been lulled into complacency, what does that say about the rest of the United States? American exceptionalism has faltered and complacency has risen in its wake. Today’s guest Tyler Cowen is an economist and author. His new book The Complacent Class is the final book in a trilogy that describes a decline of American output and a decline in American mindset. Complacent America has lost its ability to assess risk. Children are prevented from playing tag for risk of injury. College students protest against speakers who might present challenging ideas. The number of Americans under 30 who own a business has fallen by 65% since the 1980’s–millennials are too busy going to business school to start businesses. In his books, Tyler weaves together history, philosophy, and contemporary culture. He presents hard data about many different fields, and theorizes about how the trends in those fields relate to each other. He also has a podcast, Conversations with Tyler, and in this episode I tried to mirror his interview style. If you like this episode, you should check out his show–he has interviewed people like Ezra Klein, Peter Thiel, and Kareem Abdul-Jabbar. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cryptocurrencies are not only a financial instrument–they are a new platform for building applications. The blockchain allows for new solutions to digital property management, micropayments, hedge fund incentives, and ad fraud. The cryptocurrency platforms with the most traction are Bitcoin and Ethereum. Bitcoin has no central leader and is going through some growing pains with governance issues. Ethereum is led by the charismatic Vitalik Buterin. Bitcoin and Ethereum are not competing instruments. They fulfill different technical purposes. Under current conditions of algorithm development and network infrastructure, neither Bitcoin nor Ethereum can accomplish the dreams that will one day be realized, because of the problem of distributing transaction information across nodes in the system. If we compared cryptocurrencies to the Internet, we would not even be in the days of dial-up yet. ConsenSys is a venture production studio that is working on several projects within the blockchain space. Mike Goldin is a software developer with ConsenSys and joins the show to talk about blockchain applications in 2017–where we are and where we are going. It was a wide ranging conversation and I hope to have Mike back in the future so we can go deeper on some of the topics we glossed over. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

There are various standards at play when creating and consuming Application Program Interfaces (APIs). These standards, though, are mostly technical and mostly lower-level than the content of the API. Andy Beier has experienced the broad range of API quality in his role with Domo in creating integrations with other businesses. He has made standardization of good practices in creating APIs his mission, with an emphasis on making the right information easily accessible without having to download more than necessary. He has traveled to meet with leaders in the field to promote standards and to make APIs easier to create and to consume. In this episode, Andy joins Dave Rael for a conversation about API design standards, what makes for a good API, and steps in moving the broader technical community toward more useful and secure APIs. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In this episode, I gathered questions from listeners in our Slack channel and Twitter feed. The questions I answered include: We always want more feedback and questions. Please email me, join Slack, and fill out our survey! Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A hedge fund is a collection of investors that make bets on the future. The “hedge” refers to the fact that the investors often try to diversify their strategies so that the direction of their bets are less correlated, and they can be successful in a variety of future scenarios. Engineering-focused hedge funds have used what might be called “machine learning” for a long time to predict what will happen in the future. Numerai is a hedge fund that crowdsources its investment strategies by allowing anyone to train models against Numerai’s data. A model that succeeds in a simulated environment will be adopted by Numerai and used within its real money portfolio. The engineers who create the models are rewarded in proportion to how well the models perform. Xander Dunn is a software engineer at Numerai and in this episode he explains what a hedge fund is, why the traditional strategies are not optimal, and how Numerai creates the right incentive structure to crowdsource market intelligence. This interview was fun and thought provoking–Numerai is one of those companies that makes me very excited about the future. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Brendan Eich created the first version of JavaScript in 10 days. Since then JavaScript has evolved, and Brendan has watched the growth of the web give rise to new and unexpected use cases. Today Brendan Eich is still pushing the web forward across the technology stack with his involvement in the WebAssembly specification and the Brave browser. For all of its progress, JavaScript struggles to run resource-intensive programs like complex video games. With JavaScript falling short on its charge to be the “assembly language for the web” the four major browser vendors started collaborating on the WebAssembly project to allow programming languages a faster, lower level compile target when deploying to the web. Brendan is the CEO of Brave which aims to provide a faster and safer browsing experience by blocking ads and trackers by default in a new browser. The Brave browser is also helping publishers monetize in interesting new ways while also giving a share of ad revenue to its users. Caleb Meredith is the host of this show. He previously guest hosted a popular episode on Inferno, a fast, React-like JavaScript framework. As we bring on more guest hosts, please send us feedback. We want to know what every host is doing well, and what we can improve on. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Big technology companies have so much going on at any given time that a journalist can tell any type of story they want to about it. Depending on what angle you observe the company from, you can write a story depicting that company as good, evil, growing, or about to crash. The truth only becomes apparent to outsiders with time. Amazon’s culture and business strategy were detailed in The Everything Store, a 2013 book by Brad Stone. I read The Everything Store before working at Amazon, then I read it a second time after working at Amazon. The book is an accurate and balanced depiction of Amazon’s ethos. Brad’s new book The Upstarts documents the rise of Uber and Airbnb, two companies with a similar controversial valence to Amazon. It was a pleasure to speak to Brad because I admire his engrossing storytelling as an author and his strategic analysis as a business journalist. After discussing business and technology with him, we explored journalism–Brad is senior executive editor at Bloomberg. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Servers in a data center fail. Sometimes entire data centers have a power outage. Bugs in an application make it into production. Human operators make mistakes and cause data to be deleted. Failure is unavoidable. We make backups and replicate our servers so that when a failure occurs, we can quickly respond to it without making the user feel much pain. But how can we test that our response will work before an actual catastrophe occurs? Kolton Andrus is CEO of Gremlin, a company that works on failure injection as a service. Gremlin is based on ideas around planned failure that Kolton learned from his years at Amazon and Netflix. We ended up talking as much about the culture of Netflix and Amazon as we did about how and why to build failure injection. It’s always nice to share war stories with other people who have worked at Amazon because the culture is so distinct. If you want to know more about Amazon’s culture, check out the episode tomorrow with Brad Stone, author of The Everything Store. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Designers and software engineers need to communicate with each other. From Apple to Slack to Uber, the emphasis on visual design within a product is rising in importance. Much like development and operations siloes have been bridged with the DevOps movement, design and engineering teams are working more closely together to align the vision of the designers with the realities of code. InVision is a product for prototyping designs and product workflows. I categorize InVision with high-level productivity tools like Trello, Asana, and github. When teams start using InVision, it often becomes an integral part of the entire product workflow. A designer will mock up their product vision on InVision and share it with the rest of the team for criticism and commentary. Since the product is the perfect bridge between engineers and designers, InVision the company is expanding as a platform. Bjorn Freeman-Benson is the CTO at InVision and his task is to scale the 100% remote engineering team while developing new product features. Bjorn describes himself as a software psychologist. We talked about the psychology of managing engineers and the technical challenges of building a large, popular web app for technology products. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Developing a product requires careful balance between engineering, sales, design, and customer service. The founding CTO of a company often needs to take on each of these responsibilities, because when the company only has a few people there is nobody to delegate these different tasks to. Cullen Zandstra is the CTO at FloQast, a SaaS tool for accounting close management. It isn’t important what that means–we do describe it in this episode, but the actual software product that we are discussing is less important than the general discussions we have. Cullen gives detailed explanations for how to develop a software product. In the past, Cullen worked at MySpace. After we discussed his current company FloQast, we discussed some of what he learned at MySpace, and why the social network proved to be less durable than Facebook. We also had some great discussion as to why Google+ was ALSO less durable than Facebook. Also–I should mention that FloQast is hiring engineers in their Los Angeles office. And thanks to my friend Patrick Mathieson for connecting Cullen and myself. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Net neutrality is the principle that internet service providers and governments regulating the internet should treat all data on the internet the same. Debates around Net Neutrality can be as contentious as subjects like global warming, or tabs vs. spaces. To a hardcore free market economist, Net Neutrality sounds suspicious. Why would it be good for the government to regulate prices on the data that passes around wires owned by a corporation? The problem is that a large portion of the United States can only be served by wires owned by a single company. In those instances, the company that owns the wires has monopoly pricing power over the data delivered to homes in those geographic regions. Quincy Larson is the founder of Free Code Camp and a frequent author of internet-based articles on Medium. He returns to Software Engineering Daily to make the case for Net Neutrality. I was convinced by some of his arguments and less convinced by others. In any case, I think you will find this episode entertaining and informative. If you are like me, after this episode you will have at least a slightly better understanding of the issues surrounding Net Neutrality. Inside the Invisible War for the Open Internet Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In bitcoin, every transaction in the shared ledger has the sender, recipient, and value. This ledger gets appended infinitely and is shared within a peer-to-peer network. Zcash is a cryptocurrency with all the features of bitcoin, plus encrypted transactions. The sender, recipient, and value fields are all encrypted. If bitcoin is HTTP, Zcash is like HTTPS, a secure transport layer. Nathan Wilcox works on Zcash, and in this episode we discuss why an encrypted version of bitcoin is useful, how mining works, and how Zcash the company is structured. Nathan also gives some context for the current state of the bitcoin community, and where Zcash fits in. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Wearables have become more accessible to the public. Snap’s spectacles, Google Glass, FitBit, and Apple Watch suggest a future in which many people will be wearing a smart device. In this episode Asta Roseway, Research Designer at Microsoft Research, gives insights into other categories of wearables like tattoos, scarves, and cosmetics. Asta talked about her work on DuoSkin, a wearable that looks like a beautiful metallic tattoo. We talked about its capabilities and how it was built, and why it is still too early for connected tattoos. We also talked about the intersection between health, wearables, technology, and fashion and how wearables might look in the near future. This episode is guest hosted by Edaena Salinas, who also hosts The Women in Tech Show. Asta Roseway’s Website Asta Roseway Microsoft Research MoodWings LightWear Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Multiagent systems involve the interaction of autonomous agents that may be acting independently or in collaboration with each other. Examples of these systems include financial markets, robot soccer matches, and automated warehouses. Today’s guest Peter Stone is a professor of computer science who specializies in multiagent systems and robotics. In this episode, we discuss some of the canonical problems of multiagent systems, which have some overlap with the canonical problems of distributed systems–for example, the problems of coordinating between different agents with varying levels of trust resembles the problem of establishing consistency across servers in a database cluster. Peter has recently contributed to the 100 year study of artificial intelligence, so we also had a chance to discuss the opportunities and roadblocks for AI in the near future. And since Peter teaches computer science at my alma mater, UT Austin, I had to ask him a few questions about the curriculum. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Biology research is complex. The sample size of a biological data set is often too small to make confident judgments about the biological system being studied. During Jason Knight’s PhD research, the RNA sequence data that he was studying was not significant enough to make strong conclusions about the gene regulatory networks he was trying to understand. After working in academia, and then at Human Longevity, Inc Jason came to the conclusion that the best way to work towards biology breakthroughs was to work on the computer systems that enable those breakthroughs. He went to work at Nervana Systems on hardware and software for deep learning. Nervana was subsequently acquired by Intel. In this episode, we discuss how machine learning can be applied to biology today, and how industrial research and development is key to enabling more breakthroughs in the future. The main lesson I took away from this show is that while we have seen phenomenal breakthroughs in certain areas of health–like image recognition applied to diabetic retinopathy or skin cancer–the challenges of reverse engineering our genome to understand how nucleic acids fit together into humans are still out of reach, and improving the hardware used for deep learning will be necessary to tackle these kinds of informational challenges. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every company that deals with payments deals with fraud. The question is not whether fraud will occur on your system, but rather how much of it you can detect and prevent. If a payments company flags too many transactions as fraudulent, then real transactions might accidentally get flagged as well. But if you don’t reject enough of the fraudulent transactions, you might not be able to make any money at all. Because fraud detection is such a difficult optimization problem, it is a good fit for machine learning. Today’s guest Michael Manapat works on machine learning fraud detection at Stripe. This conversation explores aspects of both data science and data engineering. Michael seems to benefit from having a depth of knowledge in both aspects of the data pipeline, which made me question whether data science and data engineering are roles that an engineering organization wants to separate. This is the third in a series of episodes about Stripe engineering. Throughout these episodes, we’ve tried to give a picture for how Stripe’s engineering culture works. We hope to do more experimental series like this in the future. Please give us feedback for what you think of the format by sending us email, joining the Slack group, or filling out our listener survey. All of these things are available on softwareengineeringdaily.com. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

If you are building a service that processes payments, your software architecture has a lot of requirements. Not only do you need to be highly available, consistent, and fast–you need to be PCI compliant. In this episode, we explore the infrastructure of Stripe with Evan Broder, who has been with the company for five years. Stripe started as a small payments company catering to developers with a monolithic code base. Some of those aspects of Stripe have changed, and others have stayed the same. In our last episode, we covered how observability works at Stripe. In this episode, we explore what is being observed–the actual infrastructure itself, and how different engineers are organized around managing the infrastructure. In tomorrow’s episode, we’ll talk to Michael Manapat, about Stripe’s machine learning pipeline for detecting and preventing fraudulent transactions. Throughout these episodes, you will get a sense for how Stripe’s engineering culture works. We hope to do more experimental series like this in the future. Please give us feedback for what you think of the format by sending us email, joining the Slack group, or filling out our listener survey. All of these things are available on softwareengineeringdaily.com. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Observability allows engineers to understand what is going on inside their systems. In its most raw form, observability comes from log data. Modern systems have many layers of logs–virtualized cloud infrastructure, container orchestration, the container runtime itself, and the application logic running within the container. With all of these layers, it is not practical for a developer to have to sift through layers of logs every time a bug occurs in production, or a deployment fails integration tests. Higher level observability tools include charts, distributed tracing tools, and monitoring services. With proper observability, developers can save time during incident response. Day-to-day software development becomes safer and more comfortable. Stripe is a payments company for developers. This episode is the first in a series of episodes profiling different aspects of the company. Our guest Cory Watson leads the observability team at Stripe. In subsequent episodes, we will explore infrastructure and machine learning at Stripe. Throughout these episodes, you will get a sense for how Stripe’s engineering culture works. We hope to do more experimental series like this in the future. Please give us feedback for what you think of the format by sending us email, joining the Slack group, or filling out our listener survey. All of these things are available on softwareengineeringdaily.com. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Donald Trump has either alarmed or excited everyone in the engineering community. Some of the debates are based around innovation. Opponents to Trump say that his anti-immigration policies will reduce the innovation that relies on H1-B visas. Supporters of Trump say that his anti-regulation policies will unlock innovation that is restricted by bureaucracy. Other debates are around philosophy and free speech. Opponents to Trump say that his views on women are retrograde and that he promotes racism and white nationalism. Supporters of Trump say that he is brave enough to say what people are really thinking, and that he is truly independent from the polarized, ossified political system. Software Engineering Daily is a show about the world through the lens of the software engineer, and today our world is being shaped by Donald Trump. We needed to do an episode on this subject because so many of my day-to-day conversations with other engineers were becoming about Trump. Brad Taylor is a senior engineer with Optimizely who has formed Tech Stands Up, a grassroots movement giving a voice to the rapidly growing concerns about the Trump administration’s policies affecting the tech community and its users. We had a stimulating conversation about the tradeoffs of a Trump administration from an engineer’s perspective. Don’t worry–this is not becoming a show about politics. Tech Stands Up: Pi Day Rally Tickets TechStandsUp.org Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Making the right engineering choices in today’s wide landscape of cloud technologies is hard. Predicting the future in order to invest in companies in this space has the same level of complexity. The cost of cloud computing is going down but the volume of total required space and processing power is going up. The open source community is growing and improving but people are increasingly willing to buy software that will save them time. Capital is flowing into Silicon Valley at a faster rate than can be sensibly absorbed by the number of quality companies. A decent product won’t have trouble raising money. But a decent investor has to choose wisely among the huge selection of available opportunities. Lenny Pruss works at Amplify Partners where he is currently focused on the cloud native space. In this episode, we talk about what cloud native means and how to navigate the complex landscape, whether you are an engineer or an investor. Cloud Native Landscape Project Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Command Query Responsibility Segregation (CQRS) is a powerful concept that has the potential to make for reliable and maintainable systems. It is also broadly misunderstood and means different things to different people. Derek Comartin learned about the idea after viewing some talks by Greg Young and has since successfully applied the approach with great success and it has transformed the way he views features, business requirements, and dependencies. The result is a system that is easier to maintain and faster to enhance. Among his key lessons are that slices are better than layers, mediators improve dependency management, and cohesion is better applied to business concerns than to technical ones. In this episode, Derek joins Dave Rael for a conversation about CQRS and applying it to make your code your own and to separate it from technical concerns in order to make your software development operation work better and faster. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning has improved both in tools and accessibility. Frameworks like TensorFlow create the right abstractions for developers to work efficiently. Educational programs like Metis and Insight Data Science provide a place for developers to learn these tools. As a result, artificial intelligence is becoming easier to develop and more widespread. Rumman Chowdhury works on artificial intelligence at Accenture. Before her current role, she taught data science at Metis. In this episode, we talk about the current state of artificial intelligence, from the tools available to the long term implications–such as the robot tax recently proposed by Bill Gates. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Load testing measures performance of a system undergoing a large volume of requests. Before an application is pushed to production, engineers will often load test their software to ensure it is resilient in the face of high traffic. As web applications have changed, the requirements around load testing have changed as well. External APIs, internal undocumented APIs, and proprietary databases are black boxes that you might not be able to test reliably with unit tests or integration tests. Having an end-to-end load testing system can provide a measure of insurance against unknown unknowns before users start engaging with a production version. Mark Gilbert works on performance engineering at Apica, and he joins the show to discuss how load testing software is built and when engineers should use it. Full disclosure: Apica is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

There is too much mobile engineering work to be done and not enough mobile engineers. As a result, a talented mobile engineer will often make more money than a similarly talented web developer. There are many other disconnects between the world of mobile engineering and the world of backend and web development. We have reported on web development far more than mobile on Software Engineering Daily, and, I wanted to get a holistic view of the mobile ecosystem. Nathan Esquenazi is a co-founder of CodePath, which is a continuing education program for professional engineers and designers. There was much to explore–from the economics of mobile, to the state of cross platform, to the increasingly blurry line between mobile and IoT. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Thursday February 23rd was a big day in security news: details were published about the Cloudbleed bug, which leaked tons of plaintext requests from across the Internet into plain view. On the same day, the first collision attack against SHA-1 was demonstrated by researchers at Google, foretelling the demise of SHA-1 as a safe hashing function. What does this mean for the average engineer? What are the implications for regular internet users? Haseeb Qureshi interviews Max Burkhardt, a security researcher at Airbnb, to get to the bottom of what exactly happened, what it means, and how it affects the security of web applications. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Manufacturing electronics is not a simple task and because of its complexity it is also quite expensive. Companies like Apple are able to pull their costs down through economies of scale. Since Apple is always placing huge bulk orders on chips, sensors, and other components, the company can get lower prices for those components than an individual hardware hacker that wants to build a prototype. Herein lies the opportunity for Tempo Automation, a company that allows rapid prototyping of electronics. By aggregating demand for electronics prototyping, Tempo Automation can offer competitive pricing and speed to anyone who needs a hardware prototype. I visited Tempo Automation and was given a tour of the production floor by CEO Jeff McAlvay. He showed me the huge machines that are used to build hardware prototypes and explained how they work in serial to produce prototypes. He also mentioned that anyone can schedule a tour of Tempo Automation’s factory by going to their website, and I highly recommend it. After the tour, Jeff and I sat down for an interview about what Tempo Automation is building and their plans for the future. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When an engineer is offered a job a tech company, their compensation is often partly in cash and partly in equity–shares of the company. How should an engineer evaluate that offer? How should they negotiate? In the world of equity compensation, costly and avoidable mistakes are routine, and this hurts both companies and employees. Josh Levy was on Software Engineering Daily previously to talk about the Amazon Web Services open guide, which was one of the most popular episodes we have ever had. In this episode, Josh and I are joined by Joe Wallin, a lawyer who has been involved in startups for many years. Josh and Joe are the authors of the The Open Guide to Equity Compensation, which is a resource designed to clear up the confusion around stock, options, and fundraising. It’s a tremendously useful and concise overview of what an engineer or founder needs to know when it comes to equity financing. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Parse was a backend as a service company built in 2011 before being acquired by Facebook in 2013. Building a backend as a service for developers requires walking a thin line between giving engineers lots of control and preventing those engineers from shooting themselves in the foot. While she was at Parse, Charity Majors learned about the operational burdens of managing a service with high uptime requirements and deeply technical edge cases that could take down a user’s entire system. Charity took the lessons in systems engineering that she learned at Parse and cofounded Honeycomb.io, a service for observability and monitoring. Honeycomb is described as a tool for your systems like an IDE is to your code. Parse was eventually shut down because the service did not have a place in the strategic plans of Facebook. Charity and I also discussed the lessons learned from how the Parse acquisition panned out–a useful conversation for anyone who is considering selling a company or acquiring a startup. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When an application is using all of its available resources, that application needs to be scaled. Scaling an application means giving it more resources–typically servers. Autoscaling is an engineering practice where an application is automatically given more or less resources based on how healthy the application performance is at a given time. Applications on Heroku have access to autoscaling. Heroku users don’t need to worry about provisioning new servers manually because the platform does it for them. In this episode, we explore how Heroku built autoscaling. Andrew Gwozdziewycz [@apgwoz] is an operational experience engineer with Heroku. As he describes, autoscaling requires frequent health checks of an application. Since thousands of applications are running on Heroku, a metrics pipeline using Kafka and Cassandra supports the high volume of health check data. That data feeds into the decision process for when an application needs to scale. Full disclosure: Heroku is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the mid 90s, data warehousing might have meant “using an Oracle database.” Today, it means a wide variety of things. You could be stitching together a big data pipeline using Kafka, Hadoop, and Spark. You could be using managed tools like BigQuery from Google. How did we get from the simple days of Oracle databases to the wealth of options available today? Mark Rittman writes and podcasts about data engineering and data warehousing on his site Drill to Detail. Today, we explore the past, present, and future of data warehousing and touch on many of the trends that have been explored in recent episodes of Software Engineering Daily. Google BigQuery, and Why Big Data is About to Have Its Gmail Moment Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At the first Software Engineering Daily Meetup, the speakers explored a range of topics. A few weeks ago, we published Courtland Allen’s talk about how to build a small software business. In today’s episode, we are publishing Haseeb Qureshi’s talk, called “Everything That Rises Must Converge: Why Engineers Disagree About Everything (And Why Fraudsters Do Too).” This talk explores philosophy, poker, software engineering, fraud, and a basket of other topics. These topics might sound like a random collection, but Haseeb currently works as an engineer on Airbnb’s Risk team and he is the author of How To Be a Poker Player, which is a book on the philosophy of poker. The audio is taken from the Meetup and we cleaned it up as best we could. We will do a better job with the audio from the next Meetup–which is Thursday, March 9th at Galvanize! Check out our Meetup page for more details. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Security vulnerabilities are an important concern in systems. When we specify that we want certain information hidden, for example our phone number or our date of birth, we expect the system to hide the information. However, this doesn’t always happen due to human error in the code because programmers have to write checks and filters across the program. In this episode, Edaena Salinas interviews Jean Yang, Assistant Professor at the Computer Science Department at Carnegie Mellon, who presents Jeeves, a language that allows programmers to specify security policies more intuitively, making it harder to leak information that is meant to be protected. Jean explained how Jeeves was implemented and how it can be used. We also talked about what it takes to bring research concepts from academia to the industry and at the end we had a very interesting conversation on how to educate a broader audience on the importance of security. Jean was also named one of the 35 innovators under 35 by MIT Technology Review. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Expectations for mobile apps have gone up steadily since the iPhone was released. But the choice of databases built for mobile apps has remained limited mostly to SQLite. RealmDB was created as a new option for mobile developers on iOS, Android, or any other mobile platform. Realm is not just a database. It is a database platform, offering a variety of systems such as authentication, sync, and an event framework. In this interview with RealmDB VP of engineering Brian Munkholm, he explains why we need not only a new mobile database type, but an entire database platform. We talk about the business of mobile, IoT, cloud, and cross platform. It’s a wide ranging conversation, but also deeply technical. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Online advertising is a system of transactions that involve many different players. The user visits a publisher’s website; the publisher notifies an exchange that the user is on the website; the exchange presents an opportunity to a marketplace that can buy that opportunity to show the end user the ad. And this a simple example. The transactions in online advertising are as opaque and rife with fraud as Wall Street–but less regulated. Blockchain technology presents an opportunity to bring more transparency to the advertising ecosystem using a shared ledger. Ken Brook is the CEO of VidRoll, a video advertising company. Using his experience in the adtech business, Ken is working on Adchain, a shared ledger for adtech. In this episode, we explore adtech, blockchains, and Adchain. Russian Cyberforgers Steal Millions a Day with Fake Sites Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The impact of artificial intelligence on our everyday lives will be so profound that our modern institutions will change completely. Employment, government, romance, social norms–all of these things will be upended. To see the signs of this coming, you no longer have to read science fiction. Every week, there are blog posts, news stories, and videos chronicling our strange, exciting time. Rob May is an investor in artificial intelligence companies and is CEO of the AI company Talla. Every week, he puts out the Technically Sentient newsletter, a compilation of the best pieces of information about AI over the past week. Each newsletter also contains a short essay by Rob in which he gives a big picture perspective on what he is seeing in the AI space. This was an illuminating conversation with Rob about the implications of artificial intelligence and the topics he writes about in Technically Sentient. I first saw Rob speak at the Launch Scale Festival, which Jason Calacanis puts on as a free event for people who have started companies. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

If you wanted to build a machine learning model to understand human health, where would you get the data? A hospital database would be useful, but privacy laws make it difficult to disclose that patient data to the public. In order to publicize the data safely, you would have to anonymize it, so that a patient’s identity could not be derived from data about that patient–and true anonymization is notoriously difficult. In every industry where privacy is a concern there is a similar challenge. If there is no place with public data sets, there is no place where the machines can go to learn. The possible machine learning applications that we can build are limited by the data sets that are available. Auren Hoffman started his company SafeGraph to unlock data sets so that machine learning algorithms can learn from that data. In this episode, we talk about the machine learning landscape in both the short- and long-term time horizons. We also discussed some of Auren’s strategies for building companies, which have been crucial for me in thinking about how to build Software Engineering Daily. Where Should Machines Go to Learn? Quoracast Episode Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning frameworks like Torch and TensorFlow have made the job of a machine learning engineer much easier. But machine learning is still hard. Debugging a machine learning model is a slow, messy process. A bug in a machine learning model does not always mean a complete failure. Your model could continue to deliver usable results even in the presence of a mistaken implementation. Perhaps you made a mistake when cleaning your data, leading to an incorrectly trained model. It is a general rule in computer science that partial failures are harder to fix than complete failures. In this episode, Zayd Enam describes the different dimensions on which a machine learning model can develop an error. Zayd is a machine learning researcher at the Stanford AI Lab, so I also asked him about AI risk, job displacement, and academia versus industry. Why ML is hard Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data scientists need flexible interfaces for displaying and manipulating data sets. Data engineers need to be able to visualize how their data pipelines wire together databases and data processing frameworks. DevOps engineers need dashboards to understand their monitoring data at a high level. All of these programmers are building data applications. Data applications let us visualize and manipulate data sets effectively. In today’s episode, Dave King joins the show to describe the growing importance of data applications. Many knowledge workers use flexible tools like spreadsheets and business intelligence applications. But when you build a domain specific data application for a knowledge worker, you can unlock a higher degree of leverage for that knowledge worker. We discuss data applications and the future of knowledge work in this episode with Dave King. Full disclosure: Exaptive is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Most tech companies are moving toward a highly distributed microservices architecture. In this architecture, services are decoupled from each other and communicate with a common service language, often JSON over HTTP. This provides some standardization, but these companies are finding that more standardization would come in handy. At the ridesharing company Lyft, every internal service runs a tool called Envoy. Envoy is a service proxy. Whenever a service sends or receives a request, that request goes through Envoy before meeting its destination. Matt Klein started Envoy, and he joins the show to explain why it is useful to have this layer of standardization between services. He also gives some historical context for why Envoy was so helpful to Lyft. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Infrastructure is a term that can mean many different things: your physical computer, the data center of your Amazon EC2 cluster, the virtualization layer, the container layer–on and on. In today’s episode, podcasters Chris Wahl and Ethan Banks discuss the past, present, and future of infrastructure with me. Ethan and Chris host Datanauts, a podcast about infrastructure. In each episode, Datanauts goes deep on a topic such as networking, serverless, or OpenStack. As someone who hosts a similar podcast, I find it entertaining and educational to hear their points of view on a regular basis. If you like Software Engineering Daily, you might like Datanauts. And if you like Datanauts, you will love this episode of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Deep learning uses neural networks to identify patterns. Neural networks allow us to sequence “layers” of computing, with each layer using learning algorithms such as unsupervised learning, supervised learning, and reinforcement learning. Deep learning has taken off in the last few years, but it has been around for much longer. Adam Gibson founded Skymind, the company behind Deeplearning4j. Deeplearning4j is a distributed deep learning library for Scala and Java. It integrates with Hadoop and Spark, and is specifically designed to run in business environments on distributed GPUs and CPUs. Adam joins the show today to discuss the history and future of deep learning. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data science is typically done by engineers writing code in Python, R, or another scripting language. Lots of engineers know these languages, and their ecosystems have great library support. But these languages have some issues around deployment, reproducibility, and other areas. The programming language Golang presents an appealing alternative for data scientists. Daniel Whitenack transitioned from doing most of his data science work in Python to writing code in Golang. In this episode, Daniel explains the workflow of a data scientist and discusses why Go is useful. We also talk about the blurry line between data science and data engineering, and how Pachyderm is useful for versioning and reproducibility. Daniel works at Pachyderm, and listeners who are more curious about it can check out the episode I did with Pachyderm founder Joe Doliner. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Engineering managers face a different set of problems than engineers themselves. Whether they are hiring new employees, firing underperformers, or guiding a team of existing engineers, engineering management is all about people. Empathy is paramount because management is like being a professional therapist for the members of your team. Mike Borozdin has been managing engineers for more than a decade and blogs about engineering management. In this episode, we cover the core responsibilities of an engineering manager and some strategies around how to do it right. This episode is useful for anyone looking to improve their relationship with their manager, move into management, or improve as a manager. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Open source software is publicly available code that is worked on in the open by large crowds of developers. Almost all new software today uses some open source software in its code. But most people never contribute to open source themselves. Some people would love to get involved in open source, but they don’t know how to get started. Shubheksha Jalan found herself in this position. In her early attempts to break into open source, she failed to get much traction and found the open source community hostile to newcomers. She would fix beginner issues, but had trouble figuring out how to make more complex contributions. Eventually she found success and wrote about it. In this episode she takes us through that journey. If you are wondering how to get into open source, this episode is for you. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Giphy is a search engine for gifs, the short animated graphics that we see around the Internet. Giphy is also a creative platform where people create new gifs. Every search engine requires the construction of a search index, which is a data structure that responds to search queries efficiently. Since Giphy is a search engine for graphics, there is almost no text inherently associated with the each document. Giphy uses a pipeline of different labeling techniques in order to make a gif indexable by the search engine. In my conversation with Giphy CTO Anthony Johnson, we discuss how to scale a search engine, why Giphy needs to build new techniques for image processing, how human labeling for machine learning is evolving, and the future of Giphy–both as a creative medium and an advertising platform. This was an exciting and wide-reaching interview. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud computing has pushed computation away from our own private servers and into virtual machines running on a data center. In the world of cloud computing, processing is centralized in these data centers, and our smartphone and laptop application performance suffers from having high latency between the client and the cloud server. As machine learning proliferates, the current model of cloud computing will become too slow. A small difference in the time it takes to refresh a machine learning model for a drone or car could be the difference between life and death. Computation will move to the edge. The same drones, cars, and IoT devices that need their models updated quickly will form a peer-to-peer network with which to distribute time-sensitive tasks. One bellwether for this real time peer-to-peer network might be Uber’s Ringpop, a fault-tolerant application layer sharding system. In such an edge computing model, your device could federate a complex request out to other nearby devices that have spare processing capacity, pay for those compute cycles using Bitcoin, and receive a response without any request to a centralized cloud server. The cloud servers would still be around, but they would be responsible for doing offline computation across large data sets. This prediction was described by Peter Levine, a partner at Andreessen Horowitz in his talk “The End of Cloud Computing”. In this episode, Peter discusses the pressures that our pushing toward edge computing and away from the cloud. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

What is the relationship between your brain and your conscious experiences? This is is the fundamental question of the work of Donald Hoffman, a professor of computer science and cognitive science at UC Irvine. When Hoffman was a child, he wondered whether there was a cognitive dividing line between humans and machines, and that curiosity has driven him to his current work–building a mathematical framework which we can use to model consciousness. Humans would like to believe that evolution selects for traits that allow us to understand the world as it is. From his experimental simulations, Hoffman has shown the opposite–that evolution pushes us towards a mistaken version of reality. For listeners who are interested in theories about whether we live in a simulation, this episode is for you. If you are skeptical of the simulation theory then this episode will also be useful for you, as Hoffman gives one of the most nuanced, comprehensive explanations of reality that I have heard. TED Talk The Case Against Reality Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software developers succeed by combining technical ability, communication skills, and well-reasoned philosophies to craft information systems. Where this podcast focuses on the information systems, Developer On Fire focuses on the engineers who built them. Dave Rael started his podcast as a way to overcome fear and access an entrepreneurial side of himself. In his interviews with prominent engineers, Dave identifies patterns and strategies that have enabled his guests to succeed. He also finds the human side–namely the fallibilities and failures–of legendary developers, which makes it easier to imagine ourselves accomplishing the feats of his guests. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Back in 2008, the range of tools that engineers could use to connect computer systems together were getting quite good. Cloud computing was democratizing access to servers. But the telephony ecosystem was still inaccessible to the average developer. If you needed your program to make a phone call and connect a user to a customer service representative, there was no easy way to do that. Twilio was started to make it easy for developers to connect to telephone systems using simple API calls. This has unlocked many important use cases: from Uber’s communication systems to the widespread adoption of 2-factor authentication. In this episode, Twilio VP of product management Pat Malatack joins the show to explain how the company builds and scales the telephony systems that underpin applications which we use every day. We also talked about how Twilio’s culture shapes how engineering proceeds at the company. Full disclosure: Twilio is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Engineers today have a variety of career options. You could go work for a large corporation, you could raise money and start a startup, you could freelance and move from job to job with freedom–or you could start a business with the goal of quickly becoming profitable. Courtland Allen was a guest on Software Engineering Daily a few months ago, when he discussed Indie Hackers, a platform he built to share the stories of engineers building business on their own and making money. We only touched the tip of the iceberg in our conversation, so I was excited to invite him to the first Software Engineering Daily Meetup, which occurred earlier this month. Today we are republishing his talk, and I would love to hear your feedback on this format. We will be experimenting more with new hosts and formats throughout 2017, and if you have ideas for the show or you are interested in hosting a show, please send me an email. Also–the first Software Engineering Daily Meetup was fantastic–there were ~200 people showing up so we may have to cap attendance on the next one. Please join the Meetup group if you are interested, and we will let you know when we schedule our next event. Courtland Allen slides in PDF. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Many elderly people live with unhealthy levels of isolation. Social isolation is a problem for anybody, but younger people can use technology to alleviate their isolation with tools like Skype and Facebook. How can we bridge the generational gap and give elderly people access to the same technological tools that younger people find easy to use? Voice interfaces are an important new medium for communicating with computers. Amazon and Google are at the forefront of the voice interface movement, but many other devices are emerging. One of these devices is Elliq, from Intuition Robotics. Elliq consists of a microphone voice interface that sits on a table next to a small tablet computer. An elderly person can talk to the voice interface and have it start a voice call with someone on Skype, or see new Facebook photos or YouTube videos. The discussion of Elliq’s hardware is as interesting as that of its backend software. As Itai Mendelsohn of Intuition Robotics explains, Elliq got to market very quickly as a result of the company’s adoption of high-level managed services, like Firebase and Google’s managed machine learning. ELLIQ – The active aging companion Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Whatever engineering problem you have right now, the solution is probably not to write a new programming language. But sometimes it does make sense. JetBrains makes IDEs–the interactive development environments that many people code in, like IntelliJ and Webstorm. And all of these IDEs are written in Java. So the JetBrains team is very familiar with Java and the JVM. Since JetBrains spends so much time working on tooling for developers, they also have an intimate understanding of the problems that developers encounter. Kotlin was developed as an alternative to JVM languages such as Java and Scala. In this episode, Hadi Hariri from JetBrains explains why the company decided to build a new JVM language. We discuss many of its features, such as safety, concision, and its ability to compile to JavaScript. Why Kotlin is My Next Programming Language: Ode to the Language You’ve Never Heard Of, by Mike Hearn YouTube: GOTO 2016 Conference, Kotlin – Ready for Production, Hadi Hariri Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Translation is a classic problem in computer science. How do you translate a sentence from one human language into another? This seems like a problem that computers are well-suited to solve. Languages follow well-defined rules, we have lots of sample data to train our machine learning models. And yet, the problem has not been solved–largely because languages don’t always follow rules. We have idioms and subtle contextual clues that make it hard to provide a computer with hard and fast rules for translation. Unbabel is a company whose solution to translation puts a human in the loop to correct the error-prone translations that computers often make. In this episode, Vasco Pedro joins the show to explain Unbabel’s approach to translation, its technology stack, and the business applications for translation. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The goals of microservices are the same as what we have pursued in software engineering for decades: isolation, decoupling, maintainability, scalability. The reason that we use the term microservices is not because we have a completely different idea of what a service is than we used to. We use the term microservices because we are signaling that we need to achieve these architectural goals in a different way than we needed to 10-15 years ago. Markus Eisele is a developer advocate at Lightbend. He joins the show to discuss how enterprises are moving from monolithic architectures to microservices architectures, which has been touched on in previous shows. Lightbend makes a framework called Lagom, that suggests a opinionated strategy for moving towards microservices using message passing, CQRS, and other patterns that we explore. Reactive Microservices Architecture Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Building a mobile app requires developers to build a separate version for Android and iOS. The approval process for app stores makes it difficult to deploy quickly and iterate in small batches for developers who are making native apps. These frictions cause fewer developers to write mobile apps than we would have if the smartphone platform was unified. Since the early days of the smartphone, different cross platform solutions have been tried, and many developers got burned after they implemented an app that was supposed to work on iOS and Android, when the performance ended up being poor on both. Today, the cross-platform solutions are finally getting good with frameworks like React Native and Ionic. Ionic allows developers to write apps in AngularJS and deploy them to both Android and iOS. In this episode, Adam Bradley from Ionic gives a brief history of cross-platform apps, and explains what is different today–and why cross platform apps are exploding in popularity, particularly in large enterprises. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Urbit is a completely new way of looking at computing. Every user gets a personal server, which runs your apps, wrangles your connected devices, and defines your secure identity. Your urbit presents your whole digital life as a single web service. Urbit feels foreign and confusing for those of us coming from the traditional web because the normal paradigm is to iterate and paper over the problems of the current platform with new things built on top. Curtis Yarvin, the creator of Urbit, argues that the current model is too fundamentally broken for that to work. As he says: “the Internet as a client-server network has won. The Internet as a peer-to-peer network has failed.” This sounds like yet another quirky, overambitious developer side project–but Urbit has serious legs. The github repo has had 51 committers over its four years of activity. Last year, a public crowdsale of Urbit address space raised more than $200,000. Peter Thiel was an early investor in the project, perhaps partly due to the combination of persistence, technical skill, and unusual opinions of Curtis Yarvin. In this episode, Curtis and Galen Wolfe-Pauly join me for a conversation about Urbit–its strange computing platform and the contrarian philosophies that drive its creators. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Over the past few years, React has become the most popular front end JavaScript framework. As React has matured, the open source community around React has identified areas for improvement. Since React itself is too mature to refactor completely, new projects have been started to take the best aspects of React and start from scratch. Inferno is an extremely fast, React-like JavaScript library for building modern user interfaces. Dominic Gannaway is the creator of Inferno and in this episode he joins Caleb Meredith for an interview about Inferno and other React-like JavaScript libraries. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Product managers are responsible for guiding the design and overall functionality of software. The relationship between product managers and engineers is complimentary: a PM is viewing the product from a perspective that is closer to the customer, so the PM often has the responsibility of navigating high-level tradeoffs in the functionality of a product. Suzie Prince is the head of product at ThoughtWorks and she joins the show to discuss how she navigates tradeoffs as a PM. We also explored communication strategies for how PMs can work productively with engineers, and the overall product development process. Full disclosure: SnapCI is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Medical imaging is used to understand what is going on inside the human body and prescribe treatment. With new image processing and machine learning techniques, the traditional medical imaging techniques such as CT scans can be enriched to get a more sophisticated diagnosis. HeartFlow uses data from a standard CT scan to model a human heart and understand blockages of blood flow using simulations of fluid dynamics. In today’s episode, Razik Yousfi and Leo Grady from HeartFlow describe the data processing pipeline for the company and what their technology stack looks like. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data visualization tools are required to translate the findings of data scientists into charts, graphs, and pictures. Understanding how to utilize these tools and display data is necessary for a data scientist to communicate with people in other domains. In this episode, Srini Kadamati hosts a discussion with Jake VanderPlas about the Python ecosystem for data science and the different attempts at creating a data visualization library. Jake VanderPlas is the Director of Research for Physical Sciences at the University of Washington’s eScience institute, where he also received his PhD in Astronomy. In addition to contributing to many Python data science libraries like scikit-learn, scipy, numpy, and matplotlib, he’s written multiple books that have been published by O’Reilly and has given many talks on data science tools and techniques. He’s also the co-creator of the Altair project, which is a declarative data visualization library for Python built on the Vega-Lite visualization grammar. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Column-oriented data storage allows us to access all of the entries in a database column quickly and efficiently. Columnar storage formats are mostly relevant today for performing large analytics jobs. For example, if you are a bank, and you want to get the sum of all of the financial transactions that took place on your system in the last week, you don’t want to iterate through every row in a database of transactions. It is more efficient to just look at the column for the amount of money, and ignore things like timestamp and user id. Julien Le Dem co-created Parquet, a file format for storing columnar data on disk. Jacques Nadeau is a VP of Apache Arrow, a format for in-memory columnar representation. They are both part of Dremio, and they join the show to talk about how columnar data is stored, processed, and shared between systems like Spark, Hadoop, and Python. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Free Code Camp is an online learning platform that takes people from knowing nothing about code to having enough knowledge to build software for a living. We have already done a show with Quincy Larson, the founder of Free Code Camp, in which we discussed his motivation for starting the organization. The economics of running a free interactive platform with thousands of users informs the architectural strategy. Free Code Camp does not even have ads, so it needs to be frugal. Berkeley Martinez is the CTO of Free Code Camp, and in today’s episode he discusses how Free Code Camp works from a technical perspective, including how the site’s sandboxed coding environment is built. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software engineers are often one of the last lines of defense against potentially dangerous and unethical practices. Every software company encounters situations where the line between right and wrong is not clearly drawn. Back in the year 2000 Bill Sourour was asked to code a project to help market a pharmaceutical product in a misleading way. Reflecting on this experience, Bill recently wrote about it on Medium, suggesting that developers should avoid doing things that compromise our ethics in small ways. The more software defines our real world the more relevant the conversation around ethics becomes. In this episode, Bill and I explore the ethics of modern software development. The Code I’m Still Ashamed Of Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A company like Pinterest has millions of transactional emails to send to people. The scalability challenges of sending high volumes of email mean that it makes more sense for most companies to use an email as a service product rather than building their own. Chris McFadden is the VP of engineering and cloud operations at SparkPost and he joins the show to explain the architecture of SparkPost’s email as a service product. SparkPost started as an on-premise email technology for large enterprises, and evolved into a SaaS product. In 2014, the company migrated to the cloud, which has changed its infrastructure as well as its operational model. Full disclosure: SparkPost is a sponsor of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Vulnerabilities exist in every computer system. As a system gets bigger, the number of vulnerabilities magnifies. The web is the biggest, most complex computer system we have–but fortunately, the steps we can take to secure our web applications are often quite simple. Jared Smith is a cyber security research scientist with Oak Ridge National Laboratory. He joined me on the show to discuss web application security, but I really wanted to know his position on some of the more grandiose topics–Stuxnet, our power grid, Russian hacking, and corporate backdoors. This was a wide ranging discussion and I enjoyed it a lot. For a presentation Jared gave at Nodevember about Web Security, check out this YouTube video. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Meetup is an online service that allows people to gather into groups and meet in person. Since 2002, the company has been growing and its technology stack has been changing. Today, they are in the process of migrating to the cloud, using both Amazon Web Services and Google Compute Platform. Yvette Pasqua is the CTO of Meetup and she joins the show to explain how Meetup’s technology stack works and how the teams are organized. The discussion of multiple clouds is particularly interesting–Yvette describes GCP and AWS as both having distinct, well-defined use cases at Meetup. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a useful new technology comes out, companies that are in a position to adopt that new technology can gain an edge over competitors. As our industry grows and moves faster, these kinds of changes are coming faster–some recent examples are Docker, ReactJS, and Kubernetes. Evolutionary architecture supports incremental, guided change as a first principle along multiple dimensions. A company with an evolutionary architecture is structured to evolve in response to changes inside the company (such as a decision to change the product) or outside the company (such as the emergence of Docker). Neal Ford is an architect at ThoughtWorks and one the creators of the evolutionary architecture concept. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every digital system has vulnerabilities. Cars can be hacked, locked computers can be exploited, and credit cards can be spoofed. Security researchers make a career out of finding these types of vulnerabilities. Samy Kamkar’s approach to security research is not just about dissection–it’s also about creativity. For many of the technologies he hacks on, Samy open-sources code that summarily describes the vulnerability he has been working on. For example, in his project PoisonTap, Samy open-sourced code that you can run on a $5 Raspberry Pi, and plug into a locked computer to exploit it. Our conversation covered the art of deconstructing technologies for vulnerabilities and Samy’s goals as a security researcher. We also touched on some of the broader issues of modern security. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Self-contained systems is an architectural approach that separates the functionality of a system into many independent systems. Each self-contained system is an autonomous web application, and is owned by one team. Communication with other self-contained systems or 3rd party systems is asynchronous where possible. As Eberhard Wolff explains in this episode, self-contained systems is not the same thing as microservices, but they are not mutually exclusive. Organizations often adopt a mix of architectural ideas, and it is worth understanding these different models so you can decide which of them to apply to your own projects. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Many people find themselves going down a career path that does not bring them satisfaction. A lawyer finds himself constantly working cases he doesn’t care about. A student in medical school gets completely burned out from hospital bureaucracy. An investment banker no longer finds joy in the accounting statements that she used to enthusiastically study. Startups offer a different career path. Within a startup, an employee can often find creativity, limited bureaucracy, and highly variable reward structures. These ingredients make the world of startups both refreshing and intimidating to someone who feels stuck in a career that no longer gives them joy. Breaking Into Startups is a podcast about people coming from non-traditional backgrounds and making their way into startups. Breaking Into Startups is run by Artur and Timur Meyster and Ruben Harris, who join me for a conversation about how they migrated from more traditional careers to startups–and how many other people are doing the same. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the 1990s, the barriers to starting a company were significant. Not only did you need an idea, you needed $200,000 for servers and Oracle licenses. With cloud computing, the up-front financial costs of getting a company off the ground have been mostly eliminated–but the idea of starting a company is still perceived as risky. The process of building software has changed dramatically in the last twenty years, but many of the challenges of managing a software company remain timeless. How do you hire properly? How do you keep track of a software stack that is growing in complexity? How do you handle dissenting opinions from employees? Mike Wolfe has been building software companies since the 1990s and joins the show to discuss how to build products and manage engineering teams. It’s a wide-ranging discussion including technological trends, interpersonal skills, and startup financing.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A biologist wants to study the genetic makeup of an organism. A pharmaceutical researcher wants to test the effects of an experimental drug. These types of experiments require a deep knowledge of the scientific domain as well as the lab techniques to produce the data that will eventually yield a result. Transcriptic is a robotic biology laboratory that allows you to make requests for experiments to run remotely. Scientists who use Transcriptic do not have to perform wet lab experimentation. The robotic laboratory process has the potential to do for biology what Amazon Web Services did for computing. Max Hodak is a cofounder of Transcriptic and joins me today to explain how a robotic laboratory works. There are a wide variety of challenges, from hardware integration to software reverse engineering–but we also talk about why it is worth it to overcome these challenges, including a discussion of the “reproducibility crisis” that is undermining faith in our scientific experiments.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Application performance monitoring helps an engineer understand what is going on with an application. An application on a single machine is often monitored by inserting bytecode instructions into the application after it has been interpreted. Distributed cloud applications with functionality broken up across multiple servers often use distributed tracing. Andi Grabner from Dynatrace joins today’s show to explain how monitoring software is built, and how engineers use it to solve problems. Monitoring is core to every business, whether the goal is to understand top-level business processes of to dissect and debug a specific engineering problem. And because monitoring is important at every layer of the stack, there is a plethora of monitoring tools for sale.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Online marketplaces and social networks often have a trust and safety team. The trust and safety team helps protect the platform from scams, fraud, and malicious actors. To detect these bad actors at scale requires building a system that classifies every transaction on the platform as safe or potentially malicious. Since every social platform has to build something like this, Smyte decided to engineer trust and safety as a service. Josh Yudaken joins the show today to discuss how Smyte engineered its platform to provide machine learning models for any organization that wants to take advantage of Smyte for its trust and safety. The tools we discuss include Kubernetes, RocksDB, and Kafka, and Smyte is solving some problems that have not been solved before, so this is a great episode for anyone interested in data engineering or fraud detection–or how to use cloud services and open source tools in unique ways.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every song you hear on the radio is written with a computer. Computer musicians mostly use synthesizers and samples to compose these songs. A sample is a snippet of recorded sound, sometimes taken from a songs, a movie, or another source. The more samples a musician has access to the better. SampleFocus is a platform where musicians upload and download samples to build songs with. Daniel Trostli is the engineer who is building SampleFocus, and he joins the show to discuss how people use samples to write music and how he built SampleFocus. I met Daniel at the Launch Scale event, a conference about building businesses, and when I was having lunch with Daniel, he told me about working on this personal project full-time, and some of the different business models he was exploring. It’s a great episode for anyone who is thinking about turning a side project into a business.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In the last five years, companies started hiring data engineers. A data engineer creates the systems that manage and access the huge volumes of data that are accumulating on cheap cloud servers. As the saying goes, “it’s more expensive to throw out the data than to store it.” Pete Soderling joins the show to discuss the rise of the data engineer, and how that role interacts with data scientists, software engineers, and machine learning experts. Pete runs Hakka Labs, a site that aggregates material for data science and data engineering. He also organizes DataEngConf, a conference for data engineering.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A huge percentage of online advertisements are never seen by humans. They are viewed by bots–automated scripts that are opening web pages in a browser and pretending to be a human. Advertising scammers set up web pages, embed advertisements on those pages, and then pay for bot traffic to come and view those advertisements. This aspect of the internet is bizarre and alarming. Think about it–how much of the Internet economy depends on online advertising? A lot! How many of those advertisements are consumed by robots? Some estimates say as much as 80%. The more time I spend looking at the online advertising industry, the more perplexed and curious I become. Augustine Fou also has this curiosity. He got a PhD at MIT in material science and engineering–but today he spends his time researching advertising fraud and working as an independent consultant.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

For many years, software companies have been breaking up their applications into individual services for the purpose of isolation and maintainability. In the early 2000s, we called this pattern “service-oriented architecture”. Today we call it “microservices”. Why did we change that terminology? Did the services get smaller? Not exactly. Jonas Boner suggests that the movement towards cloud and the increased prevalence of mobile changes how we look at these services–so much that we needed to change the terminology necessary to even talk about them. And once the conversation has shifted to “microservices”, what steps do we need to take to implement them properly? The reactive manifesto is a collection of principles for how to build applications. When the reactive manifesto is applied to the idea of microservices, we get reactive microservices, which Jonas and I discuss in today’s episode.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Some tasks are simple, but cannot be performed by a computer. Audio transcription, image recognition, survey completion–these are simple procedures that almost any human could execute, but the machine learning models have not gotten consistent enough to do them accurately. Scale is an API for human labor, created by Lucy Guo and Alexandr Wang. Similar to Amazon Mechanical Turk, Scale sends small, simple tasks to workers who can complete those tasks. Scale provides an interface that is easy for developers to use, unlike Mechanical Turk, which requires a dashboard. Similar to how Stripe allows developers to build software off of payments systems easily, Scale allows developers to build human-driven, manual tasks into their code–which unlocks a wide range of potential applications, which Lucy and Alexandr discussed with me.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Over the next few years, bots will pervade our lives more and more. These smart, conversational text interfaces provide a new way of engaging with the computer systems that we have been mostly interacting with through web and mobile app interfaces for the last decade. Bots are a necessary complement to the voice interfaces of the future, because we don’t always want to talk to the computer, and natural language processing is not yet good enough to always translate our vocal request accurately. Bots are not toys, they aren’t trivial, and they aren’t going away any time soon. Jon Bruner is the host of the O’Reilly Bots Podcast, a show that has quickly become one of my favorites because it covers both theoretical and applied artificial intelligence, while remaining approachable to listen to. Jon hosts it with Pete Skomoroch, who works on a bot startup full time. Jon and Pete were also organizers of O’Reilly Bot Day, which I enjoyed tremendously and got a lot of knowledge from.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Mesos and Kubernetes are tools for distributed systems management. Kubernetes is built with an emphasis on running services, whereas Mesos is commonly used for a wider variety of workloads, including data infrastructure like Spark and Kafka. Mesos can also be used as a platform to provide resource management for Kubernetes. Dharmesh Kakadia is the author of Apache Mesos Essentials, and has spent time studying both Mesos and Kubernetes. He joins the show to explain the strengths of both platforms, and how they can be used together. In addition to dissecting Mesos and Kubernetes, we had a wide-ranging discussion, from cloud providers to software architecture to business strategy.

If you are in the Bay Area on January 11th, come to the Software Engineering Daily meetup. There will be awesome speakers, food, and a positive atmosphere. Find more information on softwareengineeringdaily.com or at the Software Engineering Daily meetup page.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Advertising fraud is easy, legal, and extremely profitable. A fraudster can set up a website, scrape content from the internet, and run programmatic advertisements against that website. The fraudster can then purchase bot traffic. Those bots will visit the page, consume advertisements, and return profit to the owner of the page. In a past life, Shailin Dhar worked for a company that set up these types of advertising fraud schemes. He was fascinated by the industry in the same way that plenty of people get fascinated with the fast-moving market dynamics that are enabled by modern software. But over time, the novelty wore off and Shailin realized how big the fraud problem is and how much it was hurting people. In today’s episode Shailin and I discuss how bots and poorly aligned incentives lead to systemic failures and significant financial loss for brands who purchase advertising.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud Foundry is an open-source platform as a service. Companies use Cloud Foundry as a control plane to deploy and manage applications. It provides abstractions for microservices and continuous integration. Cornelia Davis joins the show to discuss Cloud Foundry. I interviewed her at the DevOps Enterprise Summit, where enterprises share their stories of improving their culture and their technology stack. Cornelia explained how and why a large enterprise adopts Cloud Foundry, and why they often hire a company like Pivotal to help them as they migrate their infrastructure and change their processes.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Caching is a fundamental concept of computer science. When data is accessed frequently, we put that data in a place where it can be accessed more quickly–we put the data in a cache. When data is accessed less often, we leave it in a place where the access time is slow or expensive. Netflix has a huge variety of data, and a huge variety of access patterns for how that data gets retrieved from storage. In today’s episode, Scott Mansfield gives an overview of Netflix’s caching architecture, including EVCache, the ephemeral, volatile cache built for Netflix’s cloud architecture. As with other episodes about Netflix architecture, this show is a deeply technical case study.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software Engineering Daily was started a year and a half ago, based on what I learned from my podcasting experience on Software Engineering Radio. Last week, I interviewed Robert Blumen, the editor of Software Engineering Radio, about how that podcast is produced. In today’s episode, Robert interviews me about this podcast. If you are thinking about starting a podcast about software engineering, this podcast will be useful to you. We discuss the processes behind preparing for a show, how to conduct an interview, why people listen to podcasts, and advertisers. I also give some of the history behind Software Engineering Daily–why I started it and what my goals are.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When you are working on a program, a lot of things are going through your head. In some sense, you become part machine when you are programming. Learnable Programming is a concept that facilitates this, by showing developers what the computer is doing in real time, before compiling. In this episode, Josh Varty, co-founder of Code Connect Inc., talks to Edaena Salinas about incorporating concepts from Learnable Programming into Visual Studio and C# by showing developers what the computer is thinking while they are typing. Code Connect lets a developer immediately see what value a variable is taking, and provide test values to troubleshoot a portion of code to understand how different code paths are going to work. This is instead of having to compile and run the full program. If you have ever thought about making developer tools for an IDE or a programming language, this episode provides some great information.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a human passes away, we create a tombstone as a memorial. Friends and family visit a grave to remember the times they had with that person while they were still alive. Memorial bots are another way to celebrate the life of someone who has passed away. A memorial bot is created by taking the messages sent by a deceased person and passing it through a machine learning model in order to make a bot that replicates the deceased person. Eugenia Kuyda is the CEO of Luka, a company that builds AI products. When her friend Roman Mazurenko suddenly died, she worked with her team to make a bot that replicates his speech patterns. In our interview, we discussed memorial bots, deep learning, and the product Luka is working on–Replika, a personal AI friend for anyone.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When you are deciding who to marry, you are using an algorithm. The same is true when you are looking for a parking space, playing a game of poker, or deciding whether or not to organize your closet. Algorithms To Live By is a book about the computer science of human decisions. It offers strategies for how to think through everyday life like a computer scientist. Brian Christian has a background in computer science and philosophy, and is an author of Algorithms to Live By. He joins the show to explain how the same algorithms and data structures we use for our computer programs can be applied to the real world.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

There are multiple paths to constructing a piece of software from start to finish. An individual programmer can build an entire product from scratch in a couple days. A giant corporation can commission a project and delegate responsibilities to hundreds of people. An open source community can use the wisdom of the crowds to efficiently build an operating system. Today’s episode is about another path for building software, and my own experience traveling that path. I designed and contracted an app called Adforprize that is now available on the iOS app store. Designing an app and contracting it out to an engineer to build is a worthwhile exercise for most people in the tech industry–whether you are an engineer or not. This post has two main arguments: As with the previous monologue episodes such as 10 Philosophies for Engineers and You Are Not a Commodity, there are no advertisements on this episode. Please email me any feedback you have–jeff@softwareengineeringdaily.com Everyone has an app idea, and many of them are pretty good. You have probably had a conversation with a friend where one of you says “hey, wouldn’t it be great if there was a 2-sided marketplace for bacon?” Or “hey, wouldn’t it be great if there was an app where users could take a selfie and instantly see what celebrity they resemble the most?” These things would be cool! But most ideas end with the ideation. Who has the time to implement these things? Most of us are busy at work, where we spend our time and mental energy dealing with the technical complexity of our jobs. If you are a “smart creative”, when you get home from your job, you probably don’t want to spend your leisure hours reading iOS documentation and slowly learning to write the necessary code for your bacon marketplace app. Smart creatives who have a busy full-time job often have a hobby with more immediate gratification, like playing an instrument or blogging. Some of us also write apps on the side, but if you don’t already have all the requisite skills for writing an app, the prospect of reading tutorials and documentation can make the feeling of ever publishing an app seem so distant.

There are many people listening to this who are better at sales, marketing, design, or product management than they are at programming–they want to build an app. One common narrative is that if these people want to see something built, they need to learn to program, and that’s not true.

There is a more fun, fulfilling way to build an app than learning to code in Swift or React Native–contracting out the programming to someone else. Designing an app is a fun exercise. Building a prototype with tools like Sketch, InVision, and Keynote is free and frictionless. Most people listening to this probably spent some time drawing in KidPix or Microsoft Paint or PhotoShop when they were a kid. Most of us know how to draw at some basic level, because we spend all day seeing and understanding images–drawing is second nature to anyone who can understand shapes. Similarly, we all know how to design apps because we spend all day using them. We understand the purpose of a button or a slider or a swipe. But most of us have no idea how to configure the code around this user experience–the buttons and sliders and TextViews. Along with the design process, the business ideation process is also fun. It’s fun to imagine business models and viral loops and ways to drive user engagement. But without the capability to build the app that embodies your business ideas, it is less fun to think about those ideas. And again–most of us don’t relish the idea of struggling through the compiler errors and Stack Overflow posts that an app implementation requires. We’ve all become product designers. We are using our phones all day–most of us have naturally developed a good intuition for how an app should function. Product design ability is naturally acquired by being steeped in technology all day. If you are an employe... Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Eight years ago, MongoDB was an internal project at 10gen, a company that was trying to build a platform-as-a-service out of open-source components. The team at 10gen realized that the platform-as-a-service play would be too complex, and difficult to build. Since MongoDB was the most valuable component of that project, they narrowed their focus to this new document-oriented database. In today’s episode, MongoDB CTO Eliot Horowitz describes the history of MongoDB the open source project, as well as the company–which recently released a managed cloud service. Eliot explained how the company has architected the MongoDB Atlas service on top of AWS, and why developers often want a managed service as their database rather than managing database servers themselves. Full disclosure–MongoDB is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Four years ago, I started volunteering for a popular podcast about software–Software Engineering Radio. For the next two years, I learned about the process of making a quality podcast about engineering. With its emphasis on preparation, timeless engineering principles, and attention to the listener, Software Engineering Radio continues to be one of the most popular shows about engineering. Software Engineering Daily is my effort to bring the quality of SE Radio on a daily basis. In today’s show, I talk to Robert Blumen, the editor of Software Engineering Radio. Robert is responsible for guiding SE Radio, and without my experience producing shows alongside him, Software Engineering Daily probably wouldn’t exist. If you listen to SE Daily, you should definitely subscribe to SE Radio. I still produce content for SE Radio because the process and the community help me get internal feedback.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

C# .NET is the framework that is most often used to write software for the Microsoft Windows operating system. For many years, the C# .NET framework was closed source, but Microsoft’s recent push towards open source has led to the creation of .NET Core, a fork of C# .NET composed of a small subset of features from the original C# .NET stack. This episode takes us through a history of .NET, with two program managers who have worked on .NET for many years–Immo Landwerth and Lee Coward. We also explored the present and future of .NET–discussing .NET Core and .NET usage on operating systems other than Microsoft Windows, which is another recent development.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Biology research at Microsoft is focused on three main areas: molecular programming, synthetic biology, and stem cell biology. At the intersection of biology and computing there are implications for health, medicine, and efficient computing techniques. The field of Biological Computation is in its early days, and there is still lots of work to be done. Colin Gravill works in the computational science group at Microsoft Research, and today he explains how Microsoft is investigating biology with the same curiosity and pragmatism that Microsoft investigates computer science. I’d love to do more shows at the border between software and biology–if you have suggestions, please send me an email. jeff@softwareengineeringdaily.com

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Rust is a systems level language that is built to prevent crashes and eliminate data races. A language like C++ gives you high speed and lots of control, but it is easy to have segfaults, data races, and other problems if you aren’t careful. On this spectrum of control versus safety, we can plot other languages like Java, Go, and Haskell–but none of these languages have the unique feature set of Rust. Rust’s concurrency model is built on the foundation of ownership–a piece of data can only be owned by a single thing at a time. In today’s interview, Alex Crichton explains ownership, and the other abstractions that give Rust its unique model of concurrency.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Microservices are a widely adopted pattern for breaking an application up into pieces that can be well-understood by the individual teams within the company. Microservices also allow these individual pieces to be scaled independently and updated in isolation. Past Software Engineering Daily episodes have covered the microservice architectures of Twitter, Netflix, Google, Uber and other companies. In today’s episode, I sit down with Rafael Schloming, who is building tools for microservices at Datawire.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

There is a hierarchy of ways to access and store data in a computer system. The cheapest, slowest way to store and retrieve data is disk. On the faster end, we have memory. As we architect systems with increasing complexity, we have additional considerations–network latency, transient compute nodes, and numerous caching layers. Apache Geode is a distributed, in-memory system for the JVM. It provides fast data storage and retrieval. Swapnil Bawaskar is an engineer who works on Geode, and he joins the show today to explain the architecture of Geode, and how Geode is different from other in-memory systems that get used for caching, such as memcache or Redis.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Many programmers listening to this podcast are working at a big company, and they would prefer to be running their own software business. Patrick McKenzie has been writing about this topic for several years on his blog Kalzumeus.com. Almost a decade ago, he was working as an enterprise developer at a large company in Japan. Over time, his side projects started making enough money to justify his leaving to work on them full-time. Starting a small successful software business is not about luck, or taking big risks, or having your muse deliver some brilliant insight. It is about studying the markets where you can build software that can deliver value. Patrick and I discuss this topic, as well as Stripe–where he now works today. He sold his small businesses to go work full time on Stripe Atlas, so his story of transitioning back to a big company is fascinating as well.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Botnets have a massive influence on the Internet. As we have seen recently with the Mirai Botnet, IOT bots can take down companies as big as Netflix. In our recent episodes about advertising fraud, we’ve talked about how bots are being used to take billions of dollars of revenue from advertisers. Derek Muller is one of those advertisers who has spent money on ads and gotten nothing but fake traffic in return. Two years ago, he posted a video on YouTube about his experience purchasing advertising traffic on Facebook, and getting Likes from accounts that were clearly from fake accounts. Derek is the host of Veritasium, an awesome YouTube channel about science, truth, and technology–so we also talked some about how he built a successful YouTube business.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Slack is a chat client that has reached wide adoption. The rise of Slack has coincided with the rise of chatbots. A chatbot is a simple, conversational interface into a computer program that may have simple functionality, like telling you some simple statistics, or more complex functionality, like helping you manage your continuous integration pipeline. Bot design and engineering is a new field, and a vast array of resources and techniques are available for developers looking to hack on it. Amir Shevat is the director of developer relations at Slack, and is responsible for communicating with developers about the best ways to build bots for Slack. After seeing his talk at O’Reilly Bot Day, I had a number of questions about where we are with bots today and where we are going. I enjoyed this conversation with Amir, and if you aren’t already convinced that bots are an important platform for engineers to understand, this conversation will convince you.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Large technology companies have no shortage of data. But raw data itself does not provide a competitive advantage. Many companies are bottlenecked by a shortage of data scientists who can query that data effectively. This results in an organizational dysfunction where people lining up to ask questions of the data science team are unable to move as fast as they want to. Tomasz Tunguz saw this when he was working at Google, and he continues to see it today from his position at Redpoint, where he works as a venture capitalist looking at companies. The problems with the “data supply chain” led him to write “Winning With Data”, a book about how technology companies can successfully operationalize, explore, and act on their data. We’ve done many shows about the interactions between software engineers, data engineers, and data scientists. This episode is a great complement to previous episodes, providing a holistic view into the way data moves through an organization–and I highly recommend Tomasz’s book “Winning with Data”.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Amazon Web Services changed the economics of building an internet application. Instead of having to invest tens of thousands of dollars up front for hardware, developers can pay for services over time as their application scales. As AWS has grown to be a gigantic platform, the documentation about how to use cloud infrastructure has become insufficient. As an answer to this, Joshua Levy initiated The Open Guide to Amazon Web Services, an open-source collection of resources available on github. Joshua has experience at a variety of companies, including Viv, the conversational interface that was recently acquired by Samsung. In our conversation, Josh brought his years of experience to the table to explain the risks, benefits, and alternatives of Amazon Web Services. Josh has become a friend over the last year, and if you get the chance to have a conversation with him, I highly recommend it. He will be at the re:Invent conference in Vegas at the end of this month (November) and if you are interested in talking to him about the AWS Open Guide, or anything else, reach out to him.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Bot Day was an O’Reilly conference featuring talks from some of the leading figures in the bot industry. Before I attended Bot Day, I knew there were lots of applications for chatbots, but I didn’t realize how good the tooling has gotten–it’s very easy to get started with chatbots today so if you are a developer and you have a basic idea, I suggest checking out the videos from Bot Day, as they are a great resource for how to get started. When I was at the conference, I had a number of conversations with Edaena Salinas, who hosts The Women In Tech Show. We also caught up a few weeks after the conference to discuss our reflections on Bot Day.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The blueprint for a typical startup involves investing heavily in cloud services–either from Amazon, Google, or Microsoft. The high costs can quickly eat away at all of the money that startup has raised. In today’s episode, Avi Freedman outlines some of the infrastructure mistakes that can set back a company severely–cloud jail, hipster tools, and lack of monitorability. Avi is the CEO of Kentik, a network traffic, performance, and security visibility company, so he gets lots of perspective on the infrastructure of the average company. We also discussed the business of running a software company, contrasted with the life of a poker player. Avi and I both have some experience playing high stakes poker, so it was a great opportunity to get his perspective on the parallels between the two fields.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When Facebook acquired Instagram, one of the first systems Instagram plugged into was Facebook’s internal spam and fraud prevention system. Pete Hunt was the first Facebook engineer to join the Instagram team. When he joined, the big problems at Instagram were around fake accounts, harassment, and large volumes of spammy comments. After seeing the internal Facebook spam prevention tools clean up Instagram, Pete decided to start a company to build products that would allow this type of spam and fraud prevention as a service. Smyte provides protection against bad actors on the Internet. Complex marketplaces like Tilt, Meetup, and TaskRabbit plug into Smyte to filter their transactions for suspicious behavior. In this episode, we talk about how this type of behavior manifests, and the event-driven software architecture that the team at Smyte has built.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ben Northrop was sitting at a tech conference, listening to a presentation about a new piece of technology, when he was struck by the sense that history was repeating itself. For the twenty years that Ben has worked as a software engineer, he has been hearing about new technologies that claim they will be able to disrupt everything, and he has relentlessly been keeping up as best he can with the latest trends. Of course, Ben is all of us. Nobody can keep up. As a daily software podcaster, I think I am as well positioned as anyone to be tuned into the breadth of the software engineering ecosystem–but I am constantly behind as well. In his blog post “Reflections of an Old Programmer”, Ben theorized about the causes of this feeling of fatigue, and the constant struggle to keep up with the world of software engineering. His post contains enduring lessons that will be of value to new programmers and will feel familiar to programmers who have been around for awhile.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Jeff Bezanson’s university thesis described the motivation for a new programming language. He discussed the shortcomings of “array based programming environments” and his desire to create a more performant language with the best qualities of Lisp, Python, Ruby, Perl, Mathematica, R, and C. The Julia Language is a high performance language designed to suit technical users that crave the flexibility to pick their own notation. The language has support for a wide variety of operators, allowing scientists to create domain specific equations that can be elegantly expressed as code. Jeff Bezanson joins the show to discuss his motivations for Julia, and how the language has evolved since he started working on it.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Legacy code is code without automated tests. Most companies have lots of legacy code, and most developers don’t like working on legacy code. Why is that? What is it that makes legacy code so difficult to work with? And why does a large amount of legacy code slow down an organization so severely? Andrea Goulet is the CEO of Corgibytes, a consultancy that specializes in remodeling legacy codebases. Andrea’s company developed out of an observation that there are developers who actually love working on legacy code. Since founding Corgibytes, she has become an expert on legacy code and published a recent article about Building Technical Wealth. The article presents ways that companies can confront the realities of legacy code.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Indie Hackers is a website that profiles independent developers who have made profitable software projects, usually without raising any money. These projects make anywhere from a few hundred dollars a month to more than $100,000 as in the case with park.io, one of the services profiled by Indie Hackers. Courtland Allen is the creator, engineer, and interviewer behind Indie Hackers. For each business that is profiled by Indie Hackers, Courtland conducts a short interview with the founder. Courtland joins the show to discuss the changing trends that are making it easier to bootstrap a software business if you are a capable developer–or even if you are a nontechnical person who understands how software works. Since Courtland and I are both in the business of interviewing engineers, we had a lot to talk about, and this is a fantastic episode.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When you visit a web page, that web page can write data to a file on your computer, known as a cookie. Scripts on that page can also read from your cookie file to understand where you have been in the past. All of this data about you is getting shared between advertising companies like Google, Facebook, and AppNexus. Ghostery is a browser extension that allows you to limit what you share with these online tracking companies. Larry Furr develops products at Ghostery, and on this episode he takes us through the process of how we are tracked through the Internet. We also explore the topic of ad fraud, which is a theme we will continue to explore on SE Daily. If you have any information on ad fraud, or have recommendations for guests, please email me at jeff@softwareengineeringdaily.com

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Chat bots are your newest co-worker. Slack, HipChat, and other chat clients allow developers and other team members to communicate more dynamically than the limits of email. Companies have started to add bots to their chat rooms. These bots can give you technical information, restart a server, or notify you that a build has finished. Jason Hand is the author of ChatOps: Managing Infrastructure in Group Chat. He joins the show today to discuss how ChatOps improves development and operations by centralizing lots of functionality in group chat. Edaena Salinas is the host for today’s show. She also hosts the excellent Women In Tech Show–a podcast we highly recommend. Since we are on the subject of bots–we want to thank O’Reilly Media for recently providing Software Engineering Daily a ticket to Bot Day.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Online advertising is heavily affected by a set of delusions and fraudulent practices that few people in the adtech industry have an interest in stopping. This is the curious, perverse nature of the world of online media. Some of the online advertising we see fulfills its job, when an ad successfully conveys a meaningful message from a marketer to a consumer. But there are reasons to be extremely skeptical of the way that online advertising works. In this episode, and several more to come, Software Engineering Daily will be diving deep into the adtech business to understand the bot fraud and mass psychosis that bolsters our tech media giants. Bob Hoffman is the author of The Ad Contrarian, a long-running blog about advertising. After selling the advertising agency he was running, he left the industry. In today’s episode, Bob discusses his experience in the ad industry, and the delusions he saw when he was working in it.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Music collaboration software that works over the Internet is a software challenge that has not been fully tackled. On today’s Internet, users collaborate intensively on programming projects, journalism, and other projects, but the tools for collaborating on music online have not yet become popular. Blend.io is a social music collaboration tool–a github for musicians. I have been using it myself and enjoying it tremendously. Today’s guest Alan Grow joins the show to talk about how to build a social version control system for musicians, as well as where music is headed.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Advertising fraud takes billions of dollars out of the economy every year. We don’t know exactly how much money is being lost, because we don’t know what percentage of Internet users are bots. Are You A Human is a company designed to solve that exact problem, and provide a service for verifying whether a user is real or automated. Ben Trenda is the CEO of Are You A Human. In this episode, we discuss some of the ways that bots pose as humans–clicking on ads, pretending to read ads, and defrauding marketers and publishers. We also talk about how Are You A Human combats this. This episode is part of a series of episodes about advertising fraud. If you know anything about the area, send me an email: jeff@softwareengineeringdaily.com

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Blockchain technologies like Bitcoin and Ethereum have not impacted the lives of most consumers today. The theoretical breakthroughs that blockchain enables will eventually happen–I will be able to pay 1 cent to a knowledge worker in Africa without having to pay a 5 cent transaction fee. My servers will be able to pay other servers for small compute jobs. We will have decentralized versions of sharing economy systems like Airbnb and Uber. But first there are many infrastructure and policy issues to sort through. The technology is not quite there yet, and people running even simple mining operations are threatened by our current legal framework. Coin Center is an organization dedicated to education of policymakers and media about blockchain technology. Peter Van Valkenburgh directs research at Coin Center, and joins the show to talk about the issues that stand in the way of that utopian dream of micropayments, IoT, and other blockchain miracles.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kafka is a distributed log for producers and consumers to publish messages to each other. We’ve done many shows about Kafka as a key building block for distributed systems, but we often leave out the discussion of the complexities of setting up Kafka and monitoring it. Kafka deployments can be a complex piece of software to manage. Tom Crayford is an engineer at Heroku, where he helped engineer the recent Heroku Kafka product, which is a managed version of Apache Kafka. Our conversation explored the use cases of Kafka and how to build Kafka as a cloud service at scale. For more information about Heroku Kafka, check out an upcoming webinar. Full disclosure: Heroku is a sponsor of Software Engineering Daily. That said, this is a topic I am genuinely interested in–it is often difficult to get cloud providers to talk in detail about how they are architecting their services.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Netflix has a highly interactive user interface. As I move my mouse around the page, hovering over titles and inspecting movie descriptions, there is a lot going on under the hood. One component of this UI is RxJS, a library for building reactive JavaScript. Reactive programming uses the observer pattern to create objects that emit streams of events. We can compose these streams together to create elegant abstractions. Reactive programming may seem confusing at first, but it can simplify certain patterns that may be hard to describe with imperative programming. Ben Lesh, a senior software engineer at Netflix, joins the show to explain why reactive programming is useful, and how RxJS is used at Netflix.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Bot fraud, the New York tech scene, RethinkDB and open source; these topics and more are discussed in today’s episode. Two of the most popular guests return to the show to explore a variety of topics. Ben Halpern is the creator of The Practical Dev, a massively popular Twitter account and blog that you may recognize from its parody O’Reilly book covers. Haseeb Qureshi is an engineer at Airbnb, and a blogger who is well known for his writings on salary negotiation and coding boot camps. This episode was an experiment–if you like it, please let me know what you think.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google Compute Engine is the public cloud built by Google. It provides infrastructure- and platform-as-a-service capabilities that rival Amazon Web Services. Today’s guest Joe Beda was there from the beginning of GCE, and he was also one of the early engineers on the Kubernetes project. Google’s internal systems have made it easy for employees to spin up compute resources, but it was not a simple task to make this internal cloud consumable by the public–not to mention competitive with AWS. In order for a cloud provider to be successful, it needs to offer self-healing, self-managing infrastructure that can run microservices.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud computing was something much different in 2011, when Brian Gracely and Aaron Delp started The Cloudcast, a podcast I listen to on a regular basis. The Cloudcast features technical discussions about cloud infrastructure technology, and one of the most recent shows was a monologue by Brian Gracely where he explained his perspective on the industry rumblings about a Docker fork. The utility of a container for so many different purposes leads to different organizations having differing preferences for what use case is optimized for. The impression that I took away from this conversation, as well as the next episode that will air with Joe Beda, is that the diverse opinions and products in the container and orchestration ecosystem is quite healthy. Brian does a great job explaining his perspective on The Cloudcast, and he discusses his beliefs further in this episode.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Simpsons is a classic, beloved television show. The scripts of The Simpsons have been made publicly available, and include dialogue, location, and character information. Todd Schneider used these scripts and other information sources as a corpus to analyze The Simpsons and find interesting statistics–such as who the most important supporting characters were, and how the ratings of the show have trended over time relative to other TV shows that have declined in ratings. Todd works at Genius in New York, and I took the opportunity to ask him other questions about pop culture, given that Genius is a place where pop culture and data collide. If you haven’t seen the site, it started out as a wiki for rap music, and quickly grew to more aspects of pop culture. Todd also has a variety of other side projects, involving betting markets, analysis of taxis vs Lyft and Uber, and a systematic study of the wedding section of the New York Times.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data engineering is the software engineering that enables data scientists to work effectively. In today’s episode, we explore the different sides of data engineering–the data science algorithms that need to be processed and the implementation of software architectures that enable those algorithms to run smoothly. The PANCAKE STACK is a 12-letter acronym that Chris Fregly gave to a collection of data engineering technologies including Presto, Cassandra, Kafka, Elastic Search, and Spark. In his current life, Chris travels around the world giving workshops on how to deploy and use the PANCAKE STACK. Before that, he was an engineer at Netflix, where he received an Emmy for Streaming Engineering Excellence.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a user of a social network updates her profile, that profile update needs to propagate to several databases that want to know about such an update–search indexes, user databases, caches, and other services. When Neha Narkhede was at LinkedIn, she helped develop Kafka, which was deployed at LinkedIn to help solve this very problem. Using Kafka as an event queue, LinkedIn adopted the CQRS architectural pattern together with event sourcing. Event sourcing is an architectural pattern that allows changes to our application model to be represented as events. Each event is published to an event queue, and is pulled off of the queue by each of the various services that need to consume that event. Event sourcing and the related architectural pattern CQRS allow for a flow of information through an application that is easy to reason about, and has several other desirable properties. In today’s episode, Neha explains how to use Kafka for event sourcing and how related software patterns are improving the architectures of companies like Netflix and Uber. For more information, check out this Confluent blog entry.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The intent of the DevOps movement is to get organizations moving faster and more effectively by breaking down siloes, and improving communication. Gene Kim’s book The Phoenix Project illustrated this by telling the fictional story of a company adopting a DevOps mentality. Although that book was fiction, Gene is an experienced engineer, having worked as founder and CTO of Tripwire, a software company that makes security and compliance automation software. In his new book The DevOps Handbook, Gene presents a practical companion to The Phoenix Project. Together with his co-authors, Gene has written a guide for how to move an organization toward DevOps, and in this episode we explore some of the topics from his book. Check out an excerpt from Gene’s book here: DevOps Handbook

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At Netflix, developers write applications with a variety of requirements–from simple requests for a list of movies to more resource-intensive requests like a complex machine learning workflow. Netflix wants developers to be able to request the resources they need from a compute cluster and receive those resources on-demand, without thinking too much about the state of that pool of resources they are drawing from. At the cluster level, this means scheduling the application requests intelligently. Sharma Podila, a distributed systems software architect at Netflix, explains how Netflix has built a cloud native scheduling system on top of Mesos.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Building a monitoring system is a complex distributed systems problem. Events are produced from different points in an application and must be aggregated in order to form metrics. These events are often ingested by a time series database, which forms the backbone of our monitoring system. Theo Schlossnagle is the CEO of Circonus, where he has been working on architecting the company’s monitoring software for six years. In this episode, we talk about how to build a monitoring system and the requirements for the underlying time-series database, as well as what monitoring even is.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In order to move software updates from the development team to production, companies do a variety of things. Some teams might email files to each other or use FTP or even floppy disks. Most companies today at least use version control systems like Git together with separate servers for development and production. When code is ready to move to production, a build that is on the development server gets copied over to the production servers, and the production servers begin serving real users. This process is known as deployment, and over the last few decades companies have started deploying more rapidly (even “continuously”), leading to faster iterations and better feedback loops between the software development team and the users of the product. A particularly effective version of this workflow is known as continuous delivery. In today’s episode, David Rice from ThoughtWorks joins the show to give a short history of continuous delivery, and how continuous delivery actually looks in practice.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kafka Streams is a library for building streaming applications that transform input Kafka topics into output Kafka topics. In a time when there are numerous streaming frameworks already out there, why do we need yet another? To quote today’s guest Jay Kreps “the gap we see Kafka Streams filling is less the analytics-focused domain these frameworks focus on and more building core applications and microservices that process data streams.” Jay is the CEO of Confluent, a company that is building Kafka technology, and he is one of the original authors of Kafka. We began with a brief history of why Kafka evolved to be the message broker of choice for so many data engineering stacks, then we got into a discussion of streaming–what is streaming? How does Kafka communicate with streaming frameworks? Our conversation concluded with a discussion of Kafka Streams and the future of the Kafka data platform.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Scala is a functional programming language built on the JVM. For more than a decade, this didn’t mean anything to front end web developers. More recently, ScalaJS has brought Scala to the front end. ScalaJS is a project that compiles any Scala program down to JavaScript–so that all of your Scala programs can run on the browser. Haoyi Li has worked on ScalaJS extensively and has written an online book about ScalaJS. If you are a front end developer looking for a safer way to write your web apps, you will like this episode–or if you are a fan of functional programming.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software teams are traditionally composed of roles such as project manager, developer, QA, and manager. What happens if you throw out all of those titles, hire mostly engineers, and ask them to do whatever they think is best? That is the core idea behind Fred George’s idea of Developer Anarchy. In today’s episode, David Curry guest hosts an interview with Fred George. They talk about the structure of teams and the idea of developer anarchy. For a great complement to this episode, check out the Software Engineering Radio episode about developer anarchy.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Platform as a service can mean different things to different people. The most prominent feature of a PaaS is the ability to abstract away issues that every developer within an organization has to deal with. As an example, developers today don’t need to fear scalability and load balancing issues as much as engineers of the 90’s and early 2000s. We can develop our applications without worrying about the scaling that is going on under the covers. Sinclair Schuller is the CEO of Apprenda and he joins the show today to discuss what is involved in building a PaaS, particularly one targeted at enterprises. His company was started in 2007, just a few years after AWS got started. We also talked about the potential Docker Fork, for which Sinclair had useful business and engineering perspectives.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Docker containers changed how engineers think about building software, and the company most responsible for the widespread adoption of containers is Docker itself. Since containerization has caught on in the mainstream, companies like RedHat, Google, Huawei, and many other big players have built platform products that utilize Docker containers. Docker containers are the unit that many engineers use to deploy their applications, but the servers that run those containers are usually on infrastructure providers like Amazon, Google, or Rackspace. This raises the question–how will Docker the company make money? Having raised $180M in equity, Docker has plenty of time to find something that works, but there are suspicions in the open source community that aggressive changes to the open source Docker project are being made by Docker the company with the long-term objective of monetization. These changes–namely the default support for Docker Swarm–are not nefarious, but they might be less likely to occur of Docker were just an open source project, as opposed to an open source project being tended by a well-financed company. The New Stack broke the story about the potential Docker Fork, and Alex Williams and Joab Jackson of The New Stack join me on today’s episode. We go deep into the facts and speculations about why a fork might occur, the motivations of different key players, and the disputes within the community that have led us to this moment.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Africa has a huge reservoir of untapped technical talent. Tech companies have a huge amount of work they need skilled employees for. Andela recruits the most talented technologists across Africa, shapes them into world-class software developers, and places them with tech companies worldwide such as Google and Microsoft. Brice Nkengsa is the director of engineering at Andela, and he joins the show to discuss Andela. This episode touches on many of the themes of Software Engineering Daily–boot camps, scaling an organization, microservices, and the future of work. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

After nine years at Microsoft, Scott Berkun left to become an author. One of his books on project management was read by Matt Mullenweg, the creator of the WordPress blogging tool that runs a large percentage of the internet (including Software Engineering Daily). Scott became friends with the WordPress founder, who is also the CEO of Automattic, a company that sells WordPress hosting and support. Around that time, Automattic was run as a completely flat company. Everyone reported to Matt. This was unsustainable, and eventually Matt decided that Automattic should try management, and he convinced Scott to take a break from his writing career to return to the world of software engineering as Automattic’s first manager. Scott accepted the role under the condition that he could write a book about the experience, and the result of that agreement is “The Year Without Pants: WordPress.com and the Future of Work”, which is a brilliant depiction of what it is like to work at a completely remote software company. Scott joins the show today to talk about his book and his career journey–from Microsoft, to writing, to WordPress, and back to writing. He also shares his thoughts on the workplace dynamics that we spend our days in at software companies.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Mark Zuckerberg may be the most powerful person in the world. At no other time in history has a single human had such fine-grained control over the most influential tool for media. Today’s guests are Michael Zimmer and Nick Proferes, the creators of The Zuckerberg Files, an index of every recorded word that Mark Zuckerberg has said in text, video, or audio.

Why would someone create this? I was partly expecting Michael and Nick to be anti-Facebook, because The Zuckerberg Files sounds like a tool to hold the Facebook CEO accountable. In reality, Michael and Nick have a nuanced view of Facebook and Zuckerberg–they are Facebook users themselves, and they initially built The Zuckerberg Files as a way to portent product directions that Facebook might lean towards–particularly those related to privacy.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Scikit-learn is a set of machine learning tools in Python that provides easy-to-use interfaces for building predictive models. In a previous episode with Per Harald Borgen about Machine Learning For Sales, he illustrated how easy it is to get up and running and productive with scikit-learn, even if you are not a machine learning expert. Srini Kadamati hosts today’s show and interviews Andreas Mueller, a core committer to scikit-learn. Srini and Andreas discuss the background and implementation of scikit-learn and walk through some prototypical workflows for using it.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Containers have become the unit of infrastructure that many technology stacks deploy to. With the shift to containers, the attack surface of an application has changed, and we need to reconsider our security models; the resource allocation of our containers, the interactions between different containers on a single machine, and the big picture–how the external web may interact with our containers. Phil Estes joins the show to discuss container security, as well as the OCI, container orchestration, as well as other container related topics.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Security for the popular chat application Slack is a major focus for the company. A corporate Slack account is as valuable to a hacker as a corporate email account. In today’s episode, Ryan Huber and I talk through Slack’s approach to security–from philosophical discussions of how to company approaches security to the technical practices of logging and monitoring, and why Slack has a separate Amazon Web Services account just for the security team. Ryan works on security at Slack and has written a Medium post called Distributed Security Alerting, that I recommend checking out. If you haven’t heard it, you might also like the previous episode we did about Slack’s Architecture, with Keith Adams from Slack.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The role of “tech lead” is a combination of a software engineer, a project manager, and an architect. A tech lead might spend 30% of her time coding and 70% managing a project, resolving conflicts, and planning. In today’s episode, we discuss the structure of software teams, and when it might make sense to have a tech lead on your team, in contrast to the more common team structure of project managers, engineers, and engineering managers. Jeff Norris, an engineer from ThoughtWorks joins the show to discuss the idea of the tech lead. This is a great show for anyone in management who is looking for alternative team structures, and this episode might also appeal to engineers who are looking to transition to a role with more variety.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Devoxx4Kids is an organization that is inspiring children through robotics, programming, and engineering. Kids who attend a Devoxx4Kids workshop are exposed to entertaining and educational activities, such as writing a Minecraft mod or programming an Arduino. Arun Gupta is a software engineer and the president of the board for Devoxx4Kids USA. In this episode we discuss coding education for kids, and what happens at a Devoxx4Kids workshop. If you have a kid who you want to expose to technology, this episode has some great suggestions for how to do that.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google builds cloud services for developers, such as PubSub, Cloud Storage, BigQuery, and Cloud DataStore. On Software Engineering Daily, we’ve done lots of shows about how these types of services are built. In this episode, we are zooming in on the interaction between the developer using a cloud service and the design and engineering of the client APIs. To build a useful cloud service, Google has to make a common way of interacting with that service from any programming language on any device. Jon Skeet is a longtime engineer at Google, and he joins the show to explain how Google uses generated code to make the creation of those APIs more streamlined. We talk about gRPC, protocol buffers, C#, and lots of other topics.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When the US government hacks its own citizens, The Electronic Frontier Foundation is often the best source of reporting to find out what laws the government has broken. When a change to the privacy policy of Google or Facebook is made, the Electronic Frontier Foundation is the best place to find out how that change in privacy exploits users. The Electronic Frontier Foundation provides legal defense and editorialism at the intersection of law and technology. Nate Cardozo joins the show today to discuss the mission of the EFF and his work at the organization. We have a wide ranging conversation, ranging from governments to corporations to Stuxnet to how the internal discussions at the EFF lead to the stances taken by the EFF.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Batch and stream processing systems have been evolving for the past decade. From MapReduce to Apache Storm to Dataflow, the best practices for large volume data processing have become more sophisticated as the industry and open source communities have iterated on them. Dataflow and Apache Beam are projects that present a unified batch and stream processing system. A previous episode with Frances Perry discussed how they work in detail. In today’s episode, Eric Anderson discusses Cloud Dataflow, a service from Google that lets users manage their data processing pipelines without having to spin up individual servers to run them on. Cloud Dataflow–like the “serverless” movement we have done several shows on–represents a growing shift towards cloud providers offering services that abstract away the operational challenges of managing compute nodes. If you have suggestions for topics in this area, please do send me an email.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

A previous episode of Software Engineering Daily called “You Are Not A Commodity” received a lot of feedback, both negative and positive. The episode was a monologue I wrote about why engineers should build products on their own as a default career path, rather than work at a large corporation as a default career path. A reddit thread about the episode was almost entirely negative. Most of the emails I received about the episode were very positive. Other regular listeners wrote in and said they had mixed feelings. One of those listeners was Preethi Kasireddy, a previous guest of the show. In this episode, Preethi gives some counterarguments and questions the ideas that were presented in the “You Are Not A Commodity” episode. This will be a useful discussion for anyone who is thinking about their career path, and whether to go work for a company or do something on their own.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Unikernels allow us to specify the minimum features of an operating system we need to deploy our applications. We’ve had many shows about containers, which allow you to deploy your application on top of a segregated portion of an operating system. Unikernels are different because they can be deployed directly to bare metal or to a hypervisor. Idit Levine works on Unikernels at EMC. We had an interview a few weeks ago with her colleague Scott Weiss, and today’s conversation goes deeper into the motivations for Unikernels, the problems they solve, and the project she is leading called Unik, which seeks to add the degree of usability and adoption to Unikernels that Docker brought to containers.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Microsoft was the dominant technology company in the 1990’s, until it came under fire for anticompetitive practices. Internet Explorer was tightly coupled to the Windows operating system, which prevented Netscape Navigator–a competing browser–from reaching users on the dominant platform. This episode is about antitrust–what businesses can and cannot do in the name of competition, what the impact of Microsoft’s legal battles in the late 90’s was, and how the law might respond to potential technology monopolies in the near future–such as Facebook, Google, and Uber.

Our guest Harry First is a professor of law at New York University School of Law. He wrote a textbook called “The Microsoft Antitrust Cases: Competition Policy for the Twenty-First Century”.Sponsors

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Slack is a chat application that is rapidly growing in popularity. The focus of Slack is to create a polished, responsive tool for productivity that cuts down on the emailing, context switching, and useless meetings that take place at a typical enterprise. Keith Adams, the chief architect at Slack, joins the show to explain how those high level principles translate into engineering decisions. Keith previously worked for 7 years at Facebook, contributing heavily to the tools that make PHP easier to develop with at scale. Slack’s core product is also built with PHP, and Keith discusses the similarities and differences between scaling Facebook and Slack.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a company switches the relational database it uses, you wouldn’t expect the news of the switch to go viral. Most engineers are not interested in the subtle differences between MySQL and Postgres, right? Uber recently switched from having Postgres as its main relational database to using MySQL. Evan Klitzke wrote a detailed blog post about the migration, and post got very popular for at least three reasons: If you are even slightly interested in distributed systems or databases, I recommend reading Evan’s blog post in detail.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Relational databases are used by most applications. MySQL, Postgres, Microsoft SQL Server, and other products implement the core features of a relational database in different ways. A developer who has never studied this space in detail may not know the differences between these databases, and in this episode we describe some tradeoffs that relational databases can make. Craig Kerstiens is an engineer at Citus Data, a company that makes scalable Postgres. We talk about the requirements for a relational database, ACID compliance, how different databases handle different distributed systems problems, and the recent blog post from Uber about the company’s switch from Postgres to MySQL.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Learning to program is about self-driven exploration. Universities help guide you, coding boot camps provide a rigorous environment to work in, and online coding courses provide content for you to study. But none of this will turn you into a great programmer unless you have the drive to improve and the curiosity to explore. The Recurse Center is a place where people can come to become better programmers. Nick Bergson-Shilcock is a founder of The Recurse Center and he joins the show to discuss how it works and why he started it. This episode is a continuation of our exploration of coding boot camps, online courses, and universities, which are the pillars of programming education. The Recurse Center presents another unique model for improving as a programmer.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook users provide lots of information about the structure of their relationship graph. Facebook uses that information to provide content and services that are expected to be important to users. If Facebook knows who the most important people in my life are, Facebook can use that knowledge to serve me content that is more relevant to me. Jon Kleinberg studied Facebook network structures together with Lars Backstrom, creating a paper called “Romantic Partnerships and the Dispersion of Social Ties: A Network Analysis of Relationship Status on Facebook”. The goal of this study was to find a metric that could help rank the strength of relationships on Facebook, and the results have implications for sociology as well as the way that we think about building social networks. Jon is a professor of Computer Science at Cornell, and wrote the textbook “Algorithm Design” which I used in college, so it was a privilege to get to talk to him.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Drones will become a central part of our lives. Drones are delivering packages, surveying cell phone towers, providing wi-fi, or fertilizing crops. Drones are assisting humans in dangerous work, and serving as an entirely new computing platform, providing services that were previously nonexistent. Airware is a company that is building a full-stack drone platform. In this episode, Buddy Michini takes us through the software architecture of a drone. Airware’s drones have two operating systems–one for the real-time flight critical aspects and one for application developers who want to build their own software for drones.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning can be used to generate music. In the case of Feynman Liang’s research project BachBot, the machine learning model is seeded with the music of famous composer Bach. The music that BachBot creates sounds remarkably similar to Bach, although it has been generated by an algorithm, not by a human. BachBot is a research project on computational creativity. Feynman Liang created BachBot using Python machine learning tools to build a long-short term memory model. Our conversation explores artificial intelligence, music, and his approach to this research project.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

You have probably read a news article that was written by a machine. When earnings reports come out, or a series of sports events like the Olympics occurs, there are so many small stories that need to be written that a news organization like the Associated Press would have to use all of its resources to write enough content to cover it all. Wordsmith is a tool for automated content generation, and today’s guest Robbie Allen is the CEO of Automated Insights, the company that makes Wordsmith. He talks today about the wide range of uses for automated content, as well as how to engineer a product that takes data from a spreadsheet and turns it into a human-readable sentence. Robbie is also speaking at the O’Reilly Artificial Intelligence Conference in New York, September 26-27.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Haskell programming language is often thought of as an academic tool, useful for teaching students about monads and functors, but not much else. But there are advantages to using Haskell as a production backend language. Better is a company built with Haskell on the backend, and Carl Baatz wrote a blog post detailing his experiences using Haskell. He joins the show to give a detailed explanation of why a company might want to use Haskell on the backend, from software architecture, to testing, to hiring.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google’s infrastructure has been the source inspiration for research papers, software projects, and entire companies. Google pioneered the idea that we care less about the individual machines we are running our applications on, and more about the applications themselves. Containers are the abstraction we use to separate the concerns of the application from those of the underlying hardware. CoreOS is an operating system built with this paradigm shift in mind. In a data center, the main job of the operating system is to be a platform for containers to run smoothly on. Brandon Philips is the CTO of CoreOS and he joins the show to explain what CoreOS does differently to power the applications that get deployed on top of it.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Research in artificial intelligence takes place mostly at universities and large corporations, but both of these types of institutions have constraints that cause the research to proceed a certain way. In a university, basic research might be hindered by lack of funding. At a big corporation, the researcher might be encouraged to study a domain that is not squarely in the interest of public good–such as targeted advertising. Oren Etzioni is the CEO of the Allen Institute for Artificial Intelligence, and in this episode we discuss AI research–from the doomful premonitions of Nick Bostrom to the unbridled optimism of Ray Kurzweil, as well as the realities of how AI research actually proceeds. Projects at the Allen Institute are defined and structured to solve problems in an intelligent, scalable fashion, so that engineering can proceed steadily from the local maxima of a problem domain to the global maxima. The Allen Institute seeks to bridge the gap by providing ample funding for open source AI research for the common good. Oren is also speaking at the O’Reilly Artificial Intelligence Conference in New York, September 26-27.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Uber has a software architecture with unique requirements. Uber does not have the firehose of user engagement data that Twitter or Facebook has, but each transaction on Uber is both high value and time-sensitive. Users are paying for transportation that they expect to be available and reasonably close by. When Uber’s system is trying to match a rider with a driver, availability is favored over consistency. It is important that the rider can always get some driver, even if it is not the best driver. Ringpop is a system built at Uber to provide scalable, fault-tolerant, application layer sharding. Ringpop consists of a membership protocol, consistent hashing, and forwarding capabilities. Jeff Wolski is a software engineer at Uber working in Ringpop, and he joins the show to explain how Ringpop brings coordination to distributed applications.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes is a cluster management tool open sourced by Google. On Software Engineering Daily, we’ve done numerous shows on how Kubernetes works in theory. Today’s episode is a case study in how to deploy Kubernetes to production at a company with existing infrastructure. GolfNow is a fifteen year-old application written in C# .NET. It is a successful, growing business that is a division of NBC Sports. As GolfNow has grown, it has encountered scalability issues, and the engineering team at GolfNow decided to move its entire monolithic infrastructure to microservices running in Docker containers, managed by Kubernetes. Sheriff Mohamed joins the show today to discuss migrating his company’s application to Kubernetes. It’s a great show for anyone who is moving a large team to Kubernetes, or considering the technology for their application.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In a microservices architecture, a user request will often make its way through several different services before it returns a result to the end user. If a user experiences a failed request, the root cause could be in any of the services along that request path. Even more problematic is the challenge of debugging latency in this kind of request chain. Reshmi Krishna joins the show today to discuss distributed tracing, the process of tracking the path of a request through multiple services in order to determine the root cause of latency or errors. A popular tool for distributed tracing is Zipkin, which is largely based off of a paper published by Google called “Dapper”. Reshmi is also speaking at the upcoming O’Reilly Velocity Conference in New York, September 20-22, so check that out if you are interested in web performance, continuous delivery, or anything else related to web and mobile development.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

“Serverless” usually refers to an architectural pattern where the server side logic is run in stateless compute containers that are event-triggered and ephemeral. Mike Roberts has written a series of articles about serverless computing, in which he discusses theories and patterns around serverless architecture. In this episode, Mike and I discuss how to reimagine our software architecture using functions-as-a-service. We go into the costs, benefits, and modern limitations of current serverless platforms like AWS Lambda.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Akka is a toolkit for building concurrent, distributed, message-driven applications on the JVM. Akka provides an implementation of the actor model of concurrency, which simplifies concurrency by adding a lighter weight abstraction than threads and thread pools. Konrad Malawski joins the show today to discuss Akka and reactive streams. Reactive streams is an initiative to provide a standard for asynchronous stream processing. This show goes deep into modern concurrent programming and is a great companion to some of the shows we have done about reactive programming.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Unbounded data streams create difficult challenges for our application architectures. The data never stops coming, and we are forced to assume that we will never know if or when we have seen all of our data. Some streaming systems give us the tools to deal partially with unbounded data streams, but we have to complement those streaming systems with batch processing, in a technique known as the Lambda Architecture. Apache Beam is a unified model for defining and executing data processing workflows, and Frances Perry joins the show to explain how Beam provides a way for us to model our data processing, agnostic of whether we choose to run those workflows on Spark, Flink, or Google’s Dataflow.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

TensorFlow is Google’s open source machine learning library. Rajat Monga is the engineering director for TensorFlow. In this episode, we cover how to use TensorFlow, including an example of how to build a machine learning model to identify whether a picture contains a cat or not. TensorFlow was built with the mission of simplifying the process of deploying a machine learning model from research to production, so we also talk about that, as well as how TensorFlow can be used effectively in combination with Google’s open-source cluster manager, Kubernetes.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data Validation is the process of ensuring that data is accurate. In many software domains, an application is pulling in large quantities of data from external sources. That data will eventually be exposed to users, and it needs to be correct. Radius Intelligence is a company that aggregates data on small businesses. In order to ensure that business addresses and phone numbers are correct, Radius uses human data validation to ensure that their machine-gathered data is correct. On today’s episode, Srini Kadamati interviews Dan Morris about human data validation, and how it fits into a machine learning pipeline.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning has become simplified. Similar to how Ruby on Rails made web development approachable, scikit-learn takes away much of the frustrating aspects of machine learning, and lets the developer focus on building functionality with high-level APIs. Per Harald Borgen is a developer at Xeneta. He started programming fairly recently, but has already built a machine learning application that cuts down on the time his sales team has to spend qualifying leads. What I found most interesting about this episode was that machine learning gets used by a single developer to solve a simple business problem and deliver solid value. This is in contrast to how many of us think about machine learning–as an intimidating domain that requires a large team to build anything meaningful.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Flexport is a technology company that makes logistics, supply chain management, and freight forwarding software. Shipping freight across the world requires container ships, airplanes, trains, warehouses, and trucks. Flexport’s software integrates with many of these different shipping companies, and provides a dashboard for the end user to understand how their products are being shipped around the world. Amos Elliston is the CTO of Flexport, and he joins the show to discuss building software for the global supply chain.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When you hear about massive data breaches like the recent ones from LinkedIn, MySpace, or Ashley Madison, how can you find out whether your own data was compromised? Troy Hunt created the website HaveIBeenPwned.com to answer this question. When a major data breach occurs, Troy acquires a copy of the stolen data and provides a safe way for individuals to check if their credentials have been stolen. Troy is an expert on data breaches, and he works as a regional director at Microsoft. Our conversation explores passwords, IoT security, Stuxnet, and the dark, bizarre world of data breaches.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Linux kernel of many popular operating system distributions contains 200-500 million lines of code. The average user never touches many of the libraries that are contained in these operating system distributions. For example, if you spin up a virtual machine on a cloud service provider, the virtual machine will have a USB driver. This is wasted space, because you can’t even interact with the USB port on a virtual machine. Unikernels are a way to rethink our usage of operating systems. A unikernel uses a stripped down operating system called a library operating system–it contains only the libraries you need for the applications you are running. Today’s guest Scott Weiss joins the show to talk about unikernels, and a project he is working on called UniK, a tool for compiling application sources into unikernels.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Prometheus is a tool for monitoring our distributed applications. It allows us to focus on the services we are deploying rather than the individual machines that make up instances of that service. The monitoring service itself is a portion of a distributed system that is treated differently than the services we are monitoring. We don’t want to use a consensus-based tool like Zookeeper or Consul because we can’t afford the strong consistency. Brian Brazil’s company Robust Perception is built around Prometheus, and he joins the show to discuss why and how to use Prometheus.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The technology underlying podcasts is simple–a podcaster publishes mp3 files to an RSS feed, and the listener subscribes to that feed, receiving mp3s whenever the feed is updated. Unfortunately, the simplicity of podcasts makes it difficult to build automated advertising infrastructure on top of that simple RSS model. This lack of rich automated advertising has kept podcasting from flourishing. aCast is a company that is trying to change that by providing a better podcast experience for both the publisher and the listener. Today’s guest Mikael Emtinger is a creative technologist at aCast, and we discuss the infrastructure around podcasts, and how aCast is trying to improve it.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

GraphQL was open sourced out of Facebook, and gave developers a way to unify their different data sources into a single endpoint. Although the promise of GraphQL is appealing, the process of setting up a GraphQL server that can communicate with each disparate data source can prove to be complex.

Scaphold.io provides GraphQL as a service, and today’s guests are the creators of Scaphold, Vince Ning and Michael Paris. Scaphold.io lets developers configure their schema, and hosts their data. Vince and Michael explain the basics of GraphQL, and also discuss how they are building a GraphQL as a service platform.Sponsors

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Alec Ross worked in the White House as a Senior Policy Advisor to Hillary Clinton. His book Industries of the Future explores the biggest technological opportunities and threats to our society. The industries addressed in his book include robotics, genetics, and cybersecurity. Technological familiarity is increasingly correlated with an individual’s optimism. Cyberwarfare presents attack vectors that are difficult to insulate against. Arguments about surveillance center disproportionately on governmental surveillance rather than that of the private sector. In our conversation, Alec discusses these topics and others.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Clojure is a dynamically typed functional language that runs on the JVM. Today’s guest Alex Miller gives us an overview of Clojure’s core functionality. Alex is a developer of Cognitect, and a founder of the the Strange Loop conference. We discuss the data structures, garbage collection, and concurrency support. How does Clojure compare to other JVM languages like Scala and Groovy? How does Clojure copy immutable data structures without copying all of the data? How does a Clojure program get evaluated and converted to Java bytecode? These questions, and many others are discussed in this episode.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Odd Networks is building a platform for anyone to launch their own over-the-top streaming video service. With Odd Networks, you can deploy your own video channel using a Roku, Amazon Fire TV, Apple TV, and other services.

Creating a streaming video service with interoperability between these different platforms presents numerous technical challenges, and today’s guest Kris Walker explains how Odd Networks is addressing those challenges. We discuss the open source projects of Odd Networks, including oddworks, which encompases the SDKs, stores, services, and middleware, and the event bus, called oddcast.Sponsors

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

MobyCraft is a client-side Minecraft mod to manage and visualize Docker containers. MobyCraft was created by Aditya Gupta. I met him at DockerCon, where he gave a presentation about his project. He also discussed his interaction with the Netflix team, who integrated MobyCraft with their container management tool called Titus. You can watch a video online of Titus managing hundreds of 3-D containers within MineCraft. In this episode, we discuss how and why Aditya built MobyCraft, and how he got started programming at such a young age.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Prometheus is an open-source monitoring tool built at SoundCloud. It can be used to produce detailed time-series data about a distributed architecture. Prometheus is based on the monitoring system inside Google’s infrastructure, called Borgmon. Julius Volz is the creator of Prometheus, and he joins the show to explain why he built Prometheus and how it differs from previous monitoring tools. Prometheus is widely used to monitor Kubernetes clusters, just like Google uses Borgmon to monitor its Borg clusters.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Financial technology has changed significantly in the last decade, and companies both new and old are adapting to that change. Newer companies like TransferWise and Stripe are often called “fintech” companies–short for “financial technology”. Established companies like banks may not refer to themselves as fintech companies, but the way that they do business is changing, due to technological advances like blockchain. Ed Donner is the CEO of untapt, a hiring platform for fintech companies, and he joins the show to talk about the fundamental changes that are causing so many new fintech companies to be created. Before starting untapt, Ed worked at JP Morgan Chase, where he spent much of his time leading engineers and hiring engineers, which makes him well-equipped to build a platform for hiring software engineers into fintech companies, driven by machine intelligence.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

github has grown to have 10 million users and 30 million repositories. Getting to this scale has required innovation in many places–github has significantly altered the code for git itself, and has built unique infrastructure and written low level code to architect for git repository management at scale. Despite the need for cutting-edge technologies to support github, the development culture at github values tried and true technologies, and today’s guest Sam Lambert explains the value of maintaining a Rails monolith as the frontend of github, and the other battle-tested tools that github uses.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Pixar has made some of the most successful movies of all time: Toy Story, WALL-E, Monsters Inc, and many others. These movies are made with cutting-edge computer animation techniques that Pixar often has to invent in order to tell the story it wants to tell. Pixar has teamed up with Khan Academy to teach anyone who wants to learn the basics of computer animation–Pixar-style. The collaboration with Khan Academy is called “Pixar in a Box”, and Kitt Hirasaki joins the show to talk about it. Kitt has worked two stints at Pixar, starting in 1996 and again in 2008, and today he works at Khan Academy–so we also get into his experiences at Pixar, and how the software engineering at Pixar works.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Open source software has become the rule for how software is written rather than the exception. OSCON is O’Reilly’s open source conference, where companies and individuals talk about where the open-source world is going. Rachel Roumeliotis is the chair of OSCON, and she joins the show today to talk about the state of open source, and how the conference has developed since she started working at O’Reilly. We are giving away a free ticket to O’Reilly’s Velocity 2016 conference in New York. If you want to be entered to win that ticket, send a tweet about your favorite SE Daily episode about dev ops or web performance, and tag software_daily, as well as #velocityconf.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Finally–the Android operating system has been put on an iPhone, and today’s guest is Nick Lee, who accomplished that feat. Nick works at Tendigi, a design and engineering firm. In the past, Nick has put Windows 95 on an Apple Watch. Why would you do something like this? In today’s interview with Nick, we talk about the technical challenges of bringing Nick’s bizarre sense of technological irony into reality.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Container management systems like Kubernetes and Docker Swarm give us a higher level management tool for architectures built out of distributed containers. Container platforms like Rancher provide a higher layer of usability, and today’s guest Darren Shepherd of Rancher Labs takes us through what a container platform is. This interview is part of our continued coverage of Kubernetes, Docker, and the other components of the growing container ecosystem.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In a typical data analytics system, there are a variety of technologies interacting. HDFS for storing files, Spark for distributed machine learning, pandas for data analysis in Python–each of these different technologies has a different format for how data is represented. Serialization and deserialization between these different formats causes significant latency across the overall system. Apache Arrow is a tool for improving performance of in-memory analytics systems, and today’s guest Uwe Korn explains how Arrow enables these systems with interoperability.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

EconTalk is a weekly economics podcast that has been going for a decade. On EconTalk, Russ Roberts brings on writers, intellectuals, and entrepreneurs for engaging conversations about the world as seen through the lens of economics. Russ Roberts is today’s guest, and it is a treat because I have been listening to EconTalk since 2006 and it was a central point of inspiration for what Software Engineering Daily has become. On this episode, we talk about how software impacts the world economically, from bitcoin’s promise of zero cost transactions to the opportunities and regulatory challenges of the software-enabled gig economy.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

On smart thermostats, sensor-driven assembly lines, and electronically monitored farms, the internet of things is producing huge volumes of data. To take advantage of that data, an application needs tools for storing and analyzing that data. Today’s guest is Jean-Cristophe Cimono, the CTO of mnubo, a cloud platform for connected objects. Today we walk through the architecture of mnubo and the use cases of an IoT analytics platform. Thanks to Manuel Vonthron and Eduardo Siman for contributing to the preparation of this show.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At software conferences, I like to walk around the vendor booths and talk to the representatives from different companies. By talking to the vendors about their marketing pitches, I get an idea of how those companies are positioning themselves for the future, and the complex business landscape of software becomes slightly easier to understand. At recent conferences, many of the big vendors have been talking about their cloud platform. With Cloud Foundry, OpenShift, Kubernetes, OpenStack and so many other cloud platforms, it is hard to keep track of the different offerings, and how they differentiate. Bridget Kromhout joins the show today to talk about these different platforms, and how they have changed modern software development. We also talk about her podcast Arrested DevOps, a great show where she interviews some of the luminaries of the operations and software development world. You can also hear Bridget speak at O’Reilly’s Velocity 2016 conference in New York. We are giving away a free ticket to Velocity New York. If you want to be entered to win that ticket, send a tweet about your favorite SE Daily episode about dev ops or web performance, and tag software_daily, as well as #velocityconf.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Lee Atchison spent seven years at Amazon working in retail, software distribution, and Amazon Web Services. He then moved to New Relic, where he has spent four years scaling the company’s internal architecture. From his decade of experience at fast growing web technology companies, Lee has written the book Architecting for Scale, from O’Reilly. As an application scales, it becomes significantly more complicated while at the same time receiving more traffic. The intersection of these two problems leads to a variety of discussions around availability, risk management, and microservices. Lee and I didn’t have time to get through everything in his book Architecting for Scale, but if you enjoy this episode, check out the book. Lee also spoke recently at the O’Reilly Velocity conference in Santa Clara, so you can check out his talk. We are giving away a free ticket to O’Reilly’s Velocity 2016 conference in New York. If you want to be entered to win that ticket, send a tweet about your favorite SE Daily episode about dev ops or web performance, and tag software_daily, as well as #velocityconf.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Scheduling is the method by which work is assigned to resources to complete that work. At the operating system level, this can mean scheduling of threads and processes. At the data center level, this can mean scheduling Hadoop jobs or other workflows that require the orchestration of a network of computers. Adrian Cockcroft worked on scheduling at Sun Microsystems, eBay, and Netflix. In each of these environments, the nature of what was being scheduled was different, but the goals of the scheduling algorithms were analogous–throughput, response time, and cache affinity are relevant in different ways at each layer of the stack. Adrian is well-known for helping bring Netflix onto Amazon Web Services, and I recommend watching the numerous YouTube videos of Adrian talking about that transformation.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Call centers are a vulnerable point of attack for large enterprises. Fraud accounts for more than $20 billion in lost money every year, and a significant portion of that fraud is due to customer service representatives being fraudulent social engineering attacks. Chris Halaschek joins the show today to discuss how Pindrop Security is addressing this attack vector. Every phone call that gets made to a call center has a unique phoneprint, and the machine learning model at Pindrop Security uses these phoneprints to assign a risk score to each call. Chris also discusses the challenges associated with scaling a cloud security company.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At most universities, there is not a course titled “cloud computing”. Most students leave college without an understanding of distributed systems, cloud service providers, and the fundamentals of how a data center works. Kasper Nissen and Martin Jensen are changing that with KubeCloud, a small tangible cloud computing cluster that runs on Raspberry Pis. Kasper and Martin started KubeCloud as a masters thesis, and it is grown to a textbook-sized treatise on cloud computing. KubeCloud is both software and a curriculum to teach students microservices, containers management, and the real-world problems of distributed systems.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Transferring money from one country to another is expensive, and the banks that facilitate money transfer have tricked us into believing that it should be expensive. On today’s show, Harsh Sinha explains the peer-to-peer system of transferring money with TransferWise, where he works as VP of engineering. Harsh also discusses the larger picture of FinTech companies. The emergence of so many companies at the intersection of finance and technology is no accident. The 2008 financial crisis created a loss of trust in the existing financial system. Simultaneously, smart phones and cheap cloud computing has created opportunities for newer companies like TransferWise to position themselves as a new option for consumer banking.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Cassandra is a distributed database that can handle large amounts of data with no single point of failure. Since 2008, Cassandra has been widely adopted and the software and the community around it have grown steadily. A software developer interacting with Cassandra uses CQL, the Cassandra Query Language. ScyllaDB is another open-source database that has been created to be totally compatible with CQL. By complying with CQL, the internals of ScyllaDB can be a vastly different rewrite from Cassandra. ScyllaDB uses C++, whereas Cassandra uses Java. ScyllaDB improves upon the performance characteristics of Cassandra, by optimizing for modern hardware, and Dor Laor joins the show today to discuss how ScyllaDB does all of this.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In order to use a remote desktop experience, software engineers have a limited number of options, and most of them are proprietary, like VMWare or Oracle. Remote desktop is a functionality that many engineers use every day, so it is surprising that the open source world has taken awhile to displace the functionality of proprietary software. In 2010, Mike Jumper started working on Guacamole, a way to access remote desktops through your browser. Over the last six years, Mike has worked continuously to create a simple, open-source software tool to access desktops remotely, and this year Guacamole joined the Apache Incubator and became Apache Guacamole. In this episode, we discuss the past, present, and future of remote desktop, and the technical internals of Apache Guacamole.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Six years ago, Twitter was experiencing outages due to high traffic. Back in 2010 Twitter was built as a monolithic Ruby on Rails application. Twitter migrated to a microservices architecture to fix these problems. During this migration, the engineers at Twitter learned how to build and scale highly distributed microservice architectures. William Morgan was an engineer at Twitter during that time, and he is now the CEO of Buoyant.io, a company building open-source microservices infrastructure. Some of the big problems at Twitter were solved at the communication layer, using an RPC library called Finagle. At Buoyant, those lessons are being applied to a project called Linkerd, an RPC proxy.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Mass customization is the process of making customized, personalized products that are accessible to individuals and small businesses. The process involves manufacturing, assembly lines, supply chains, and software at every step along the way. Today’s guests are Jim Sokoloff and Maarten Wensveen, who work on infrastructure and technology at Cimpress, a mass customization platform. Cimpress has t shirt printers, warehousing machines, supply chain management tools, and lots of other computers that come together in the computer-integrated manufacturing process. The company has been around for a few decades, and more recently they have moved to microservices for many of the reasons that have been discussed in previous episodes. If you work at a big company with some monolithic characteristics, this episode might give you some good arguments to bring to your manager about why and how to move to microservices.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The unit of computation has evolved from on premise servers to virtual machines in the cloud to containers running in those virtual machines. Serverless computation is another stage in the evolution of computational unit management. With a serverless architecture, a function call to the cloud spins up a transient container, calls the function on that container, and then spins down the container. Ryan Scott Brown joins the show today to discuss the benefits and consequences of serverless computing. With containers and VMs, we still have to worry that the resources we are spinning up in the cloud will run without being utilized. Serverless computing gives us more control over these compute resources, so that we don’t have unused servers that we are paying for.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Algorithmia is marketplace for algorithms. A software engineer who writes an algorithm for image processing or spam detection or TF-IDF can turn that algorithm into a RESTful API to be consumed by other developers. Different algorithms can be composed together to build even higher level applications. Diego Oppenheimer is the CEO of Algorithmia, and he joins the show today to explain how Algorithmia works. The company has developed its own container orchestration and management service, and operates similarly to the serverless computing paradigms that we have discussed on recent episodes of Software Engineering Daily. Diego also talks about the marketplace dynamics of building a platform for developers to sell algorithms to each other.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Internet of Things is becoming a reality. Factories are being outfitted with sensors, temperature monitors, and other data gathering devices. In agriculture, farms are becoming more efficient thanks to soil monitoring devices and automated pesticide regulation. In our homes, refrigerators, alarm clocks, and mirrors are becoming “smart”. Steve Busby joins the show today to talk about the big picture: how the Internet of things works, from data ingestion to processing to feedback. Steve works at Microsoft in the Azure IoT division, and he discusses the problems which companies are having and the solutions that are available today–and where we are going in the future.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Every software application has secrets. User passwords and database credentials must be managed carefully, because poor access controls can lead to disaster scenarios. Vault is a tool for secret management, developed at Hashicorp, a company that builds software tools for application delivery and infrastructure management. Seth Vargo is a software engineer and open source advocate at Hashicorp, and in today’s episode he discusses the advantages of having a single tool to manage all of your secrets. If you aren’t a security expert, don’t worry, we discuss some of the basics of security. And if you are a security expert, you will appreciate the comparisons we discuss between Hashicorp and other tools that have been used for secret management.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google’s site reliability engineers are responsible for maintaining the highly available services that power the Google software that we all use on a regular basis. O’Reilly recently published the book “Site Reliability Engineering: How Google Runs Production Systems”, and the book provides a comprehensive window into how the site reliability engineering role works. Todd Underwood is a director of site reliability engineering. On today’s episode, Todd explains how the role of a SRE relates to devops. We discuss the relationship between the engineers who are developing Google services, and the SREs who are maintaining it. Google’s internal data center operating system “Borg” is also discussed.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google researcher Jennifer Wang co-wrote a paper called “Gender Differences in Factors Influencing Pursuit of Computer Science and Related Fields”. The paper focuses on a survey of 1700 high school and college students, and takes a statistical approach to understanding why women are not pursuing computer science. In our conversation, Jennifer talks about the two influences that lead to fewer women in computer science: encouragement and exposure. The problem of encouragement: women often do not receive encouragement to go into computer science. The problem of exposure: women are often unaware that computer science even exists. On this episode, we explore the roots of these problems, and other results of her demographic study of young students.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

JavaScript programming usually is done through the use of frameworks, such as ReactJS, AngularJS, and EmberJS. These frameworks abstract away some of the messy details of JavaScript, and simplify web development so that engineers can build products at a faster pace. When we build software using JavaScript frameworks, we are missing out on some of the richness of the JavaScript language itself.

Kyle Simpson is the author of “You Don’t Know JS”, a series of books that suggests that JavaScript developers should start from the ground up, not from the top down. By learning the basics of JavaScript, a software engineer can learn the timeless fundamentals that will not disappear with the creation of next week’s hottest framework. After exploring the idea of frameworks versus raw JavaScript, Kyle and I discuss asynchronous JavaScript, from concurrency to the observer pattern. Sponsors

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Virtual machines were the unit of cloud computation for many years. Amazon Web Services pioneered the democratized model of allowing anyone to deploy a service to the cloud, running on a virtual machine on Amazon’s servers. After virtual machines, containers have become the unit of scale in the cloud. We break up our virtualized servers into even smaller units of computation called containers. Today, the unit of compute is getting reduced even more, with the introduction of serverless architecture. Serverless architectures started getting talked about after Amazon Web Services released a service called AWS Lambda, which allows users to have pieces of code run in response to events. Programmers write a function and hand it off to Amazon, and Amazon will run that function call, and only charge the programmer when the function is actually called. This is in contrast to the cost model of containers or virtual machines, which users pay for even while they are running idle. Today’s guest Austen Collins believes that serverless computing is the model of the future, and he created a company called Serverless around this idea. His company Serverless provides an application framework for building applications exclusively on Amazon Web Services Lambda.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Engineering managers start out as engineers. Eventually, there is a fork in their career road where an engineer can choose to move up into management or continue on as an engineer in a more senior role. Changing to management involves an increase in responsibilities, a different set of goals to focus on. Jon Emerson was working at Google as an engineer when a project he was working on started to get more attention. He moved into management, and spent several years at Google as a manager. Today, he works at Hired as a director of engineering, leading four different teams. Full disclosure: Hired is a sponsor of Software Engineering Daily, but this episode is not about Hired itself. We do start out with a discussion of the hiring process, because of Jon’s domain expertise around hiring, but most of our conversation focuses on the role of a manager, and the role of a director. Most of the episodes of Software Engineering Daily focus on the day-to-day life of an engineer, so it was interesting to get a perspective from someone higher up the management chain, who has more visibility over entire software projects.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The war against spam has been going on for decades. Email spam blockers and ad blockers help protect us from unwanted messages in our communication and browsing experience. These spam prevention tools are powered by machine learning, which catches most of the emails and ads that we don’t want to see. TrueCaller is a company that is bringing this quality of spam detection to our phone call systems. Umut Alp is the CTO of TrueCaller, and he joins the show today to break down the engineering problems of preventing telephone call spam. Users of TrueCaller install it on their phone, and the software allows users to report when they have received a spam call. Using this reporting mechanism, and other learning algorithms, TrueCaller is able to learn what types of calls it should block from being accepted by your phone. Today on Software Engineering Daily, we discuss cell phone spam prevention.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The software industry has a severe lack of women. There are numerous root causes of this diversity problem. Families do not encourage women to enter math and science. The media portrays most programmers as white males. Our industry often picks up on the signals of the broader society and perpetuates them. Reversing this trend of low female involvement in computer science could have tremendous positive impact, and Jenine Beekhuyzen joins the show today to discuss how she she is taking action to improve the pipeline of young women entering computer science.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Smart phone apps have better performance than web apps. When we have an application that we use on a regular basis, we download that application to a smart phone rather than using the browser based version on our mobile browser. Google’s Polymer Project wants to improve the gap between native app performance and mobile web app performance. The key problem with mobile web is that we are sending huge JavaScript bundles to mobile devices, which inhibits performance. The Polymer Project is working to build more functionality into our mobile browsers so that it is easier to load these heavy web applications.

Rob Dodson is a developer advocate with Google. Today’s episode explores the past, present, and future of web application development, from jQuery to React to progressive web apps. The Polymer project also represents a push towards better support for people in developing countries, where internet connections are less reliable. Sponsors

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Most programmers spend lots of their time reading content about software. Since our field changes so rapidly, engineers consume news and editorials voraciously, trying to keep up with the impossibly fast pace. The Practical Dev is a collection of blog posts, editorials, and interviews that was created to help with that end. Ben Halpern is the creator of Practical Dev, and he joins the show to discuss software editorialism. The goal of the Practical Dev to help developers grow and learn, and Ben is working towards that goal by providing a platform for engineers to write long-form content.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ten years ago, databases were much simpler. Most companies would only have one or two types of databases in production. Today, the age of one-size-fits-all is over. Companies have multiple databases to deal with different types of use cases, and databases have become distributed to multiple nodes in order to be scalable. Ozgun Erdogan of Citus Data joins the show to give us a modern look at databases. Ozgun suggests that PostgreSQL alone can perform most of the work that we are trying to get from our variety of databases. We discuss how Citus Data scales Postgres, and Ozgun contrasts an all-Postgres architecture with other types of databases such as the “NewSQL” class of databases, like MemSQL and VoltDB.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes is the container management platform that came out of Google’s experiences managing data centers. Kubernetes abstracts away many of the frustrations of distributed systems management. OpenShift is a platform built on top of Kubernetes to provide an additional layer of usability. Clayton Coleman is the lead engineer of OpenShift, and in our conversation today, we start with the basics of Kubernetes, then talk about OpenShift–Clayton explains why we need another abstraction on top of Kubernetes. Near the end of the conversation, we discuss the current state of cloud products–which can be confusing. Mesosphere, Swarm, Kubernetes, OpenStack, ECS–why do we need all of these different products? Clayton gives his perspective, and explains why it is not going to get any less confusing any time soon.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Coding boot camps are a subject of controversy. Critics of boot camps defend the conventional university system, and argue that boot camp graduates do not have enough experience to write quality software. But the reality is that some boot camp graduates have found success from this new educational path. After graduating high school, Kenny Tran attended one coding boot camp, then spent some time living at home absorbed in his personal projects. Eventually, he went to second coding boot camp–his form of graduate school. During his second boot camp, Kenny worked on PurifyCSS, a module that can reduce the size of front-end projects by 60%. Today, Kenny works at the groundbreaking company Mesosphere–completing a career arc that proves coding boot camps are enough of an education to rival traditional university learning.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Infrastructure-as-code is a trend that has been popularized over the past decade, as cloud computing and distributed systems have become a part of every technology company. Tools like Salt, Puppet, Chef, and Ansible allow us to manage servers and processes from the command line. David Boucha works at SaltStack, the company that makes Salt. Salt is a platform that provides configuration management and remote execution. In our conversation today, we discuss how the distributed systems architecture of Salt works, and how it can be used in practice. We also discuss infrastructure-as-code in the context of modern infrastructure tools like containers and Kubernetes.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Solar energy is a growing market. Improvements in hardware have led to some people predicting that solar energy will be powering the world within the next few decades. Undoubtedly, a large percentage of our current energy infrastructure will be replaced by solar in the near future. Replacing our old, inefficient power grid requires massive investment. On Software Engineering Daily, the resources we usually talk about are memory, bandwidth, storage, and other computer science concepts. David Reiss joins us today to talk about energy, which is a resource so fundamental that we usually don’t even consider it. David is the CTO of Wunder Capital, a fintech company that facilitates investments in solar power. Much of this conversation centers on the economics of the solar energy market, but we also discuss software topics–machine learning and why Wunder Capital’s software architecture is a monolith by design today. Sponsors

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Minecraft is a sandbox video game in which players build constructions out of 3-D cubes in a procedurally generated world. Minecraft is the best-selling PC game of all time. But Minecraft is not just a game. It is a platform for creativity, used by players within the game as well as programmers outside of it. Gabriel Simmer is a 16-year-old programmer who build NodeMC, a tool that wraps around the Minecraft server process. NodeMC can be used to build dashboards for Minecraft, and spin up additional Minecraft servers. In our conversation, Gabriel explains why people are so excited about Minecraft, how people are hacking Minecraft, and what the future of Minecraft is now that Microsoft has acquired it.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Rust is a systems programming language being developed at Mozilla. Rust has features of a high-level functional language like Scala and a low-level, performance-driven language like C++. Steve Klabnik is a developer program member with Mozilla. In this episode, he discusses how Rust looks at memory management, type safety, mutability, and concurrency. We also dive into a discussion of the low level virtual machine, also known as the LLVM, which the language Swift is built on.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The process of building a software project requires us to make so many architectural decisions. Which programming languages should be used? Which cloud service provider? Which database? A newer type of building block is the analytics platform. Companies need to track events, aggregate metrics, and change the user’s experience based on aggregated data. Dan Kador is a co-founder of Keen IO, and he joins us to discuss the rise of analytics platforms. Keen’s architecture is based on Storm, Cassandra, and Kafka, and Keen uses these different building blocks to create a scalable, reliable analytics backend. On today’s episode, we explore the usage of analytics, the architecture of Keen’s backend system, and the business model of an analytics as a service company.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Erlang is a programming language with primitives that help software engineers build distributed systems. When a process is malfunctioning in Erlang, the philosophy of the language is to let the process crash–and in a distributed system where unexpected faults happen on a regular basis, this philosophy of “let it crash” simplifies how we reason about an Erlang system. Other distributed systems advantages of Erlang include the garbage collection strategy. Each process in Erlang has its own garbage collector which means makes it easier to construct systems without a stop-the-world garbage collection. Francesco Cesarini is the founder of Erlang Solutions, and he joins the show today to discuss the book that he wrote with Steve Vinoski, called Designing for Scalability with Erlang/OTP.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google has built a microservices architecture on top of the internal container management system called Borg. These services communicate over an internal protocol known as Stubby. Borg and Stubby are tightly coupled to Google’s infrastructure–it would not make sense for Google to open source them–but Google has worked with the open source community to develop open source projects with the core functionality of Borg and Stubby. These projects are known as Kubernetes and gRPC.

Sandeep Dinesh works on the Google Cloud Platform team as a developer advocate. Our conversation explores how a client application request from an app like Gmail would communicate with Google’s servers, where the request is handled by a network of microservices. We also talk about where Google Cloud Platform is evolving, and how it offers a competitive, differentiated model from Amazon Web Services.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At Netflix, 500 billion events and 1.3 petabytes of data are ingested by the system per day. This includes video viewing activities, error logs, and performance events. On today’s episode, we dive deep into the data pipeline of Netflix, and how it evolved from their 1.0 version to the modern 2.0 version. Before listening to this episode, check out the blog post that inspired it.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Dropbox has been storing files on Amazon Web Services for 8 years, and Dropbox’s core business is storing files. For the past three years, Dropbox has been working on a project to migrate its file storage from Amazon Web Services to its own custom built infrastructure. Magic Pocket is the name of Dropbox’s new infrastructure layer, and it gives Dropbox more control and improved economics. James Cowling leads the storage team at Dropbox. On today’s episode, James takes us into the architecture of Dropbox and explains how the team moved all of the user file storage from Amazon S3 to Dropbox’s Magic Pocket infrastructure. Dropbox’s architecture is built with a focus on simplicity–and there are numerous challenges to maintaining that simplicity in the face of an extremely complex problem like this.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Almost a year ago, Software Engineering Daily aired a week of shows about decentralized technologies like Bitcoin, Ethereum, and IPFS. Bitcoin has established itself as a stable network, but it can only be used for financial transactions. Ethereum is a global computer built on a blockchain, but it does not have the adoption of Bitcoin. IPFS is a distributed data store with an incentivization layer where you can pay strangers to store your content. Today’s guest is Karl Floersh, an engineer working on decentralized technology. Karl has written several posts about how to build decentralized applications, as well as how the future might look once these decentralized technologies have gotten traction.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Distributed systems products are often marketed with terms like “real-time data” and “hassle-free scaling”, but what do those terms actually mean? Is data in a distributed system ever reliably “real time”? Do we ever have strong enough plans about our scalability strategy to say that scaling will be “hassle free”? Camille Fournier joins us today to discuss distributed systems in practice. Like everything in else in computer science, distributed systems are all about tradeoffs–and picking the right sets of tradeoffs in our distributed system will affect the entire organization that is building that system. We also discuss the Cloud Native Computing Foundation, which is similar to the Apache Foundation, but specifically for cloud technologies. The CNCF is likely to have strong impact on the way we build software for a long time to come.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Distributed databases are difficult to operate, and Crate.io wants to change that. Crate is a fast, scalable, easy-to-use SQL database that is built to run in containerized environments. An average software company runs several databases–MySQL for relational store, MongoDB for a document database, HDFS for blob storage and data warehouse, elastic search for search. On today’s show, Jodok Batlogg from Crate discuss the distributed architecture of Crate, and breaks down the use cases, from Microservices to data warehousing.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Microsoft has built a suite of technologies on top of its Azure infrastructure as a service. Today, we discuss Azure Stream Analytics, a real-time event processing engine developed at Microsoft. Azure Streaming allows for constant querying of incoming data streams, and my guest Santosh Balasubramanian discusses Azure and the movement from batch processing to stream processing.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Spark is a framework for fast, distributed, in-memory analysis. Apache Cassandra is a distributed database management system that provides high availability and fast throughput. Today, we are collecting fast, big data streams from user behavior, smart phones and sensors, and the disk checkpointing of and query language of Hadoop MapReduce is no longer adequate. Tim Berglund from Datastax came on Software Engineering Daily to explain how Apache Cassandra in a popular episode a few weeks ago. On this episode, Tim returns to discuss how Spark and Cassandra can be used together to provide a stack with the analytics and storage we need for today’s distributed computing environment.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

React started as just a view layer–it was the V in MVC. React has moved down the stack, with Flux, Redux, GraphQL, and Relay providing opinions for how React applications should structure their data flow. Jared Forsyth works at Khan Academy, which uses React on the front end. At Khan Academy, Jared has experimented with many different ways of handling data flow for a React application, and in today’s episode we not only discuss the conventional tools for React applications, but also ClojureScript, Reframe, and Om/next, which are solutions for React data handling, that are outside of the world of raw JavaScript.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Applications on your phone can talk to each other, but the way that apps interact is still far from the way that web pages interact. Mike Wakerly is the CTO of Button, a service that connects mobile apps to each other. By embedding a component from Button into your app, you can easily request the functionality of an app while you are in a different app. In this episode, Mike and I discuss the future of mobile, and the problems of app stores today. We also explore how mobile application context switching has improved, and how that creates business opportunities that are much more interesting than advertising.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Changelog is one of the most popular Software Engineering Podcasts in existence. Open source software moves fast, and The Changelog helps developers keep up with that rapid pace by publishing consistent podcasts and newsletters. When I started Software Engineering Daily, Changelog was one of the shows that I looked at for inspiration on how to succeed as a software podcaster. Adam Stacoviak and Jerod Santo are the hosts of The Changelog, and they join me today to talk about how open source has changed since they started the podcast in 2009, and how it will change in the near future.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Virtual reality is a new platform for software engineers to work with. Best practices for VR development have not become widespread throughout the developer community. If you are developing VR software, you need to know how to avoid making the user sick, and this requires an understanding of the hardware limitations. Andreia Gaita is a software engineer at github. On today’s episode of Software Engineering Daily, Andreia takes us through the developer experience of VR, and gives some analysis and predictions about consumer VR. For anyone who is looking for an introduction to virtual reality development–this episode is for you.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Patients with Type-1 diabetes need to frequently pump insulin into their bodies. In order to know when to pump insulin, these patients have a continuous glucose monitor alarm which detects incorrect levels of blood glucose. When the alarm goes off, the diabetes patient administers insulin manually through the pump. In an ideal world, the alarm would communicate with the insulin pump, creating a closed loop. But the world of outdated medical devices is not ideal. Dana Lewis is a developer of an open-source, closed-loop artificial pancreas called the Do-It-Yourself Pancreas system. Dana joins the show to discuss how she hacked together her artificial pancreas using a Raspberry pi plugged into these medical devices. This is a fascinating story of reverse engineering, internet of things, and the hacker mentality. Sponsors

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

At Netflix, building a user interface requires data about movies, actors, users, accounts, and much more. If each piece of data is coming from a different source, a programmer has to call each of the services that has access to that data. Falcor is a piece of technology designed at Netflix to solve this problem. Falcor creates a unified JSON model of all of the data you could want–and Netflix engineer Brian Holt joins us today to discuss how Falcor works. We also discuss React, ES6, and JavaScript at Netflix, and Brian explains why Falcor is extremely similar to GraphQL, and why that is OK.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes is an open-source distributed operating system that allows normal engineers to manage their resources at the scale that Google does. With Docker providing the APIs to make containers easy to work with, Kubernetes allows processes to be easily distributed and replicated across a data center. Kubernetes was created at Google, and today’s guest Kelsey Hightower is a staff developer advocate working at Google. He joins me in a conversation about how Kubernetes works, and why it is important to an average developer. We touch on how software engineering is going to look in the next 5-10 years, and how a lot of the work around distributed systems will become much easier over time.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Exception monitoring services and log management services are two sides of a gradient. Exception monitoring services capture and aggregate the problems that occur on your application. Log management services aggregate all of your logs, so that you can decide for yourself what constitutes a problem. Brian Rue from Rollbar joins the show today to talk about Rollbar’s exception monitoring architecture, and the competitive landscape of these technology products. Every software engineer wants track the problems with an application, but some developers need more information than others–and that ends up changing how these error aggregation services are architected. This is an interesting conversation on the business of SaaS products for developers, and the architecture of a distributed system designed to monitor and aggregate errors.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

As software engineering gets more popular, the resources that we use to read and understand software are growing and improving. Scale Your Code is an organization that seeks to improve accessibility to the world’s programming knowledge, and today’s guest Christophe Limpalair joins me to discuss Scale Your Code. Christophe is a podcaster and blogger who runs Scale Your Code.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Docker is an open platform for distributed applications. Docker allows for standardized container runtimes and will be an important piece of infrastructure for many years to come. But if you are new to Docker, the technology can be confusing and intimidating. What is a container? Why do I want to put my application in a container? Today’s guest Prakhar Srivastav wrote a long tutorial explaining the fundamentals of Docker. Prakhar explains why Docker is useful, how Docker works, and how you can get started.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Kafka has become the most popular open-source solution for persistent replicated messaging in the Hadoop ecosystem. But some software engineers who are working with “big data” don’t want to deal with the configuration and set up of Kafka. One way to side step this problem is to go with a managed solution, like Microsoft Azure Event Hubs. Dan Rosanova is today’s guest. He joins us to discuss persistent replicated messaging, and the features that Azure Event Hubs provide. This is an interesting discussion about distributed pub sub messaging, and we also get into a great conversation about platform as a service versus open source software, and the future of cloud software.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes has been a focus of several previous shows on Software Engineering Daily. The architecture and management of our distributed systems have changed, and widespread adoption of containers and container management systems is just beginning. Joseph Jacks from Kismatic joins the show today to give us a perspective on the past, present, and future of container management. He also talks about his company Kismatic, which provides enterprise support for Docker and Kubernetes.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

“Graphing when your Facebook Friends are awake” was trending at the top of Hacker News when Alex Hogue first posted it. Alex wrote a blog post about this piece of software he built that interacts with public Facebook APIs to detect when any of his friends are awake or asleep. As I read that post, I wanted to talk to Alex, because it was such a creative and technical piece of software–but it is surprising that someone would take the time to randomly inspect the elements of the Facebook page, reverse engineer the way Facebook sends data about statuses, mine data from that reverse engineering, and create a bunch of graphs about that data. Today, Alex joins me to explain how and why he built a tool to graph when people are awake or asleep using Facebook data.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Applications crash, and engineers need to be able to quickly find the root cause of a crash. Apps have become distributed, and debugging workflows have changed. Developers need better tools to identify and troubleshoot problems with their apps. Ben Curtis joins us today to discuss Honeybadger, an exception and monitoring tool for web applications. We discuss how software tools can be used to detect problems faster and improve the debugging workflow.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kubernetes is an open source system for automating deployment, operations, and scaling of containerized applications. Google developed Kubernetes after fifteen years of running containers in production. Brendan Burns is a founder of the Kubernetes project, and he joins us to talk about the lessons learned as Google has built containerized applications to distribute across its massive infrastructure. We talk about Docker, Borg, Kubernetes, and other distributed systems technologies. Applications crash, and engineers need to be able to quickly find the root cause of a crash. Apps have become distributed, and debugging workflows have changed. Developers need better tools to identify and troubleshoot problems with their apps.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Elixir is a programming language built on top of the Erlang virtual machine. Elixir allows metaprogramming, polymorphism, and a web framework called Phoenix that has drawn positive comparisons to Ruby on Rails. Jose Valim is today’s guest. He built Elixir to augment a language that he loved–Erlang. On Software Engineering Daily, we interviewed Joe Armstrong, the creator of Erlang, which was a very popular show–I encourage any listeners who are fans of Erlang to check it out. Erlang was built with concurrency in mind, and it has been rising in popularity as more of our applications are written to be distributed. In today’s episode we will discuss what Jose is building on top of Erlang.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Engineers who want to add search to their application usually deploy Elasticsearch, or write their own search engine that uses TF-IDF. These solutions work well for large documents, but are less effective for large volumes of small records–which is how many modern web and mobile applications are structured. In today’s show, Julien Lemoine discusses how his company Algolia thinks about search. Algolia is a search as a service company that gives developers an easier way to search on their websites and applications.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

CDN stands for content delivery network. A content delivery network is a system of distributed servers that delivers web pages and other web content. Without CDNs, the internet would be much slower, because CDNs function as a caching layer for most web resources. Carl Gustas is an engineer at CacheFly, a popular content delivery network. He joins us today to discuss how CDNs work, and the different methods an engineer can use to take advantage of caching in a CDN.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Google has published papers on distributed systems such as BigTable, Chubby, and the Google File System. During this episode, we focus on a product that takes inspiration from Google’s Spanner project, a database that is built on a distributed monolithic sorted map. CockroachDB is a scalable, survivable, consistent SQL database. Today’s guest Ben Darnell is the CTO of Cockroach Labs, and he joins us today to discuss SQL and NoSQL distributed databases. Ben explores the tradeoffs between these database types, and explains how CockroachDB provides many of the best features of both a SQL and a NoSQL distributed database.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Wiki technology was invented decades ago to improve how software developers communicate. Today, Wikipedia has taken the ideas of the wiki to a new level, creating a free knowledge graph for the world to learn from. Ward Cunningham developed WikiWikiWeb in 1994. He joins us today to discuss the first wiki, and how wikis have changed the way information propagates. Ward is not just an expert on how to write software–he is an expert on the software development experience, and we discuss how to build a community where software developers can overcome the fear of building something that has not been built before.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software applications are constantly generating logs. These logs are necessary to understand how an application is functioning, and logs are key to debugging. As applications have gotten more complex, logging infrastructure has become complex as well. Storing and managing all of our log data is such a big task that several companies have been started to tackle this problem. Today’s guest is Christian Beedgen, CTO at Sumo Logic. Sumo Logic is a cloud-based log management company. We discuss the elastic log-processing platform Sumo Logic has built to help software engineers with log management. It’s a great conversation about distributed systems, machine learning, and debugging applications.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Frontend web development was simpler in the past–CSS, HTML, and JavaScript were all you needed to know. Today, we have mobile web, React, Angular, PHP, JQuery, and so much more. Marc Grabanski focuses on what he believes is timeless–pure JavaScript. Marc is the founder of Frontend Masters, a training site for developers who want to build quality web interfaces. He joins us today to discuss the shifting nature of front end development.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Nadia Eghbal joins us as the host of today’s show to discuss the past, present, and future of open-source software. This monologue was adapted from Nadia’s article, We’re in a brave, new post open source world. In this episode, Nadia describes the origins of open-source in the Free software movement, its rise to popularity, and today’s golden era where mainstream and popular technologies like React, Spark and Docker are all open source projects. Yet there are several serious issues that threaten the foundation upon which the current open-source renaissance is happening. Nadia discusses what these are, and how to resolve these issues so that we can continue to have a vibrant ecosystem for developers. Thank you to Nadia Eghbal for coming on the show and for standing up for open source software.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When you spend money online, you expect a receipt to come in your email. When you register for a new web site, you need to verify your sign up in your email. These types of emails are called “transactional email” and sending these types of email at scale is a complex engineering task. J.R. Jasperson is the chief architect at SendGrid, a transactional email platform. On this episode, we discuss how email works–from the basics to the massive scale that SendGrid operates on. We also talk about email spam and fraud in detail.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Git is the most popular version control system. If you have been programming for less than a decade, it’s likely that you haven’t used any other method of version control. The git workflow of a software team defines how that team collaborates, builds, and ships software. Tim Pettersen is a developer advocate at Atlassian, where he has been building software around git for many years. He joins us on today’s episode to talk about strategies for git–including branching, merging, continuous integration, and software as a service.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Application delivery has become more complex as software architectures have moved into the cloud. Data center infrastructure has turned into code to be manipulated, and software engineering teams are adjusting their strategies. HashiCorp is a company that builds open-source software for application development and deployment. Mitchell Hashimoto is the founder of HashiCorp, and he joins us to discuss a modern approach to application delivery, and the tools HashiCorp is developing.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Code analysis tools can help a developer understand code. One tool for code analysis is bitHound, which provides code and dependency analysis for NodeJS applications. On today’s episode, we discuss how to use a code analysis tool–and we also talk about how to build one, by breaking down the distributed architecture of bitHound’s backend.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Uber is a transportation company with a high volume of temporal spacial data, constantly being collected from the devices of its users. At any given time, the engineers and data scientists at Uber need to be able to query the system, and understand what is going on with drivers and riders. The unique real-time engineering requirements of Uber lead to an interesting architecture. Danny Yuan joins us today to discuss Uber’s data engineering stack and how the company makes use of its streaming data.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The last editorial we did was 10 Philosophies for Engineers. Listeners enjoyed that episode, so we decided to do another. 10 Philosophies was a collection of beliefs I have about the software engineering industry, and how we can find fulfillment in our work as engineers. The philosophies that I discussed were rooted in my experiences working as a computer science student and as an engineer at several different companies. Before writing any lines of code, I played poker competitively for five years, from the age of 15-20. This was a formative experience. Playing poker changed my perspective on money, risk, and statistics. It has been 7 years since I stopped playing poker full time, but the lessons of the game are still with me. In this editorial, I will discuss some beliefs I have that relate to both poker and software engineering. As with the 10 Philosophies episode, these beliefs my own opinions. If you disagree with them, I would love to know why. At Software Engineering Daily we actually want to know any thoughts you have on our content. We make content for our listeners and our readers. Please tell us how we can improve by emailing us, or by filling out the listener survey. As a poker player becomes a software engineer, certain trends about human-computer interaction become apparent: This post explores each of these trends, explaining why these trends are important to poker players, software engineers, and everyone else. Automated Games In 2008, poker was the perfect sport for human-computer symbiosis. What Tyler Cowen said about freestyle chess also applied to poker: In poker, a human with a statistical “heads-up” display can make decisions that are more mathematically justified than a human without such a tool. Heads-up displays create the poker version of “human-plus-machine teams”. One thesis of Average is Over is that a human will only be employable in the future by finding a career where human reasoning provides defensible value to the problem solving process of a computer. If the human’s responsibilities are not defensible, the human will beobviated. In a subsequent blog post, Cowen addresses the “flip” that can occur when a computational problem no longer requires human assistance: Poker players have been increasingly defeated by machines for the past 10 years. It is no surprise that Google’s AlphaGo has defeated human champion Lee Sedol. If Google decided to beat humans at poker, it would be a trivial exercise for the researchers. Poker seems different than Go or Chess, since there is nondeterminism. You start with two cards, but you don’t know how the board will develop. It would seem that fate is in control, unlike Go and Chess, which have no random elements. With just 4 suits and 13 ranks, a poker game has a trivial branching factor. The nondeterminism is so minimal for a computer to plan around that it is effectively deterministic. Imagine if AlphaGo had to learn to play a version of Go with the following rule: at the beginning of each turn, flip a coin. If you lose the flip, you don’t get to move. Adjusting to this rule would be trivial. That is the magnitude of nondeterminism within poker. Poker, Chess, and Go have small decision spaces. The rules never change, the game pieces never change, there is minimal nondeterminism. A computer can assess a hand of poker as it would a hidden Markov model, but it will take work on par with the AlphaGo team for a computer to be trained to build a model accurately. The job of a professional poker player has been a bad long-term choice for a human to pursue for more than a decade, because it is vulnerable to automation. Games like Go, Poker, and Chess can be automated with machine learning techniques we understand today. The rules, game piece schema, and objectives are easy to define, so these games are ripe for supervised learningand reinforcement learning. Yann LeCun protested against the hype around the AlphaGo victory: Poker is vulnerable t... Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Applications can and will crash — it is increasingly important for developers to have visibility into the reasons how and why the crash occurred. James Smith is the guest on the show today, and joins Jeff to discuss why modern applications crash, and how developer products are improving to tighten the gap between QA/support and dev. James is the founder of Bugsnag, an automated crash monitoring system.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ten years ago, building a highly scalable image delivery service would require millions of dollars in upfront costs, and hours of work configuring hardware server infrastructure. Today, it is possible to bootstrap this type of service, with minimal investment. Today’s episode is about building a content delivery network for images and video. Today’s guest is Itai Lahan, CEO of Cloudinary. We discuss Cloudinary’s early product infrastructure, and how they have evolved as a company since then. We also talk in detail about the venture capital landscape of Silicon Valley today, and how to strategize about raising money. Full disclosure, Cloudinary is a sponsor of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Memory is king. The cost of memory and disk capacity are both decreasing every year–but only the throughput of memory is increasing exponentially. This trend is driving opportunity in the space of big data processing. Alluxio is an open source, memory-centric, distributed, and reliable storage system enabling data sharing across clusters at memory speed. Alluxio was formerly known as Tachyon. Haoyuan Li is the creator of Alluxio. Haoyuan was a member of the Berkeley AMPLab, which is the same research facility from which Apache Mesos and Apache Spark were born. In this episode, we discuss Alluxio, Spark, Hadoop, and the evolution of the data center software architecture.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloud service providers like Amazon, Google, and Microsoft provide both infrastructure as a service and platform as a service. Infrastructure as a service gives developers access to virtual machines, servers, and network infrastructure. Platform as a service is the software that runs on top of that infrastructure as a service–this includes things like Amazon DynamoDB, Microsoft Azure Machine Learning, and Google App Engine. OpenStack is an open-source cloud operating system. Today’s guest is John Purrier, a founder of OpenStack and the CTO of Automic Software.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In today’s episode, Ben Halpern interviews Juan Buritica, VP of Engineering at Ride. They discuss Ride’s migration from a monolith to microservices, the challenges of running a distributed team and preventing developer burnout.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

GreyOrange Robotics builds robots for warehouse automation of logistics and ecommerce companies for quicker deliveries. Today’s episode features Akash Gupta, the CTO of GreyOrange.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

DevOps has been the subject of many episodes of Software Engineering Daily. And yet–the question we continue to ask ourselves is “what is DevOps?” In order to understand the present, we must look into the past. On today’s show, we break down the history of dev ops with Demir Selmanovic, the lead technical editor at TopTal. From waterfall software development, to the agile manifesto. From lean manufacturing to the productive pinnacle of DevOps nirvana. Demir also wrote a blog post called What the Hell is DevOps, which he posted on the TopTal blog. Full disclosure: TopTal is a sponsor of our show, but this episode doesn’t discuss TopTal very much so hopefully that doesn’t impact the trust of our listeners.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Today’s guest is Calvin French-Owen, the CTO of Segment, a tool that companies use to aggregate their analytics into once place. As Segment has scaled, the company has had to restructure its etire technical architecture. Microservices, containers, Amazon Web Services, and dev ops are a few of the topics that Calvin and I explore in our conversation, so this is a great episode for anyone who is trying to understand the relationships between those different subjects. Segment’s product unifies analytics from different services and puts them into one centralized place. Full disclosure: Segment is a sponsor of Software Engineering Daily. For most of this episode, we don’t even talk about the product, we talk about the back end engineering behind the product.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

CodeNewbie is a community of programmers and people learning to code. There are so many people learning about software today, and CodeNewbie gives them a place to hang out, socialize, and become comfortable with the world of software. CodeNewbie has an excellent podcast, and if you like Software Engineering Daily you should check it out. Saron Yitbarek is the guest today, she is the creator of CodeNewbie. In this episode of Software Engineering Daily, Saron sits down to talk about learning to code after she had already had several careers, spanning biology, public radio, and marketing.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Mic.com is a media company focused on news for millennials. Anthony Sessa is the VP of product at Mic.com, and he joins us to talk about the engineering of a modern news organization tailored to young people. We discuss data engineering, frontend technologies, and how to migrate away from WordPress. We also explore how to build a successful media company today, and how millennials want to consume news and software.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software developers have been socializing on chat rooms for decades. In the nineties, we began using IRC and AOL instant messenger. In the early 2000s, we turned to Google Hangouts and Yammer. Today, we are using modern apps like Slack and Hipchat. On today’s show, we take a deep dive into Gitter, the chat client specifically designed for developers. Our guests are Mike Bartlett and Andrew Newdigate, the creators of Gitter. Gitter is a highly scalable, real time social application for developers to talk about writing their software. This is a great episode that spans topics like front end development, back end distributed systems, how to compete with Slack, and how to scale a chat room to tens of thousands of active users.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Military veterans have the right set of skills to become programmers. Technical expertise, emotional resilience, psychological persistence, and teamwork–these are the qualities of the US army and they are the qualities of the best programmers. More and more veterans who leave the army are becoming coders, and the mission of Operation Code is to help veterans make that transition. Dave Molina is the founder of Operation Code. His organization helps army veterans become coders. Dave came on the show to discuss coding boot camps, military, software education, and what it’s like to become a civilian technologist after spending years in the army.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Building a software business today requires lots of decision making. Building software isn’t just about choosing a programming language, or a framework, or a database. Developers have to choose the right cloud service provider, the best issue tracking service, the best hosted code repository, the clearest data visualization tools. We need a platform to compare the options between different products, all in one place. In today’s episode, we talk to Ayan Barua from Siftery, a platform for software products and the companies who use them. Siftery’s mission is to help technologists find the best products for their job, whether they are a sysadmin or an advertising account manager. My discussion with Ayan covers the question of build versus buy, how to build a recommender system, and how software will be built in the future; this is also a great episode for anyone who is thinking of how to build a two-sided marketplace business.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software engineering is a deterministic field. We write lines of code, and feed data into that code, expecting to get a certain answer. Computing is deterministic because humans developed it–we understand computers from top to bottom. The same cannot be said about biology. Matt Might is an associate professor with a PhD in computer science. When his son was diagnosed with an extremely rare illness, he was confronted with the uncertainties of human biology. In this episode, we discuss Matt’s quest to solve the puzzle of his son’s disease–computer science meets genetics, on this episode of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In many ways the engineering job market is broken. Engineers cannot find the right jobs where they are valued properly and enjoy their work. Employers have difficulty filling their positions and finding the right people that fit their cultures. Traditional matchmakers like Monster and generic recruiting agencies don’t cut it — you know this by the generic LinkedIn messages you get looking for a Java expert… when you have JavaScript as a skill on your profile. Hired.com is trying to fix this broken process by using a high-touch approach to recruiting. Hiten Parmar, an engineer and product strategist at Hired joins us today to discuss the issues at hand, as well as how Hired is approaching the solution. We chat about the challenges facing engineers, which skills are in demand, and the inner workings of Hired. Full disclosure: Hired is a sponsor of the show, but we would still have done this interview because we genuinely want to discuss the job market, and figure out how companies are addressing this issue.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Stack Overflow is used by developers to find out how to build software. Stack Overflow is both a tool and a community, and today’s guest Jeff Atwood has made a career out of building tools and communities. As the co-founder of Stack Exchange and Discourse.org, Jeff has been solving the problem of civilized online communication for seven years. In today’s episode of Software Engineering Daily, Jeff Atwood talks about building online communities from the perspective of an engineer as well as a sociologist.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

On Software Engineering Daily, we often discuss big data in terms of data engineering and data science. Data engineering is the infrastructure and pipelines that handle massive amounts of data and puts that data in a data lake. With a data infrastructure in place, a data scientist can study the data and take action on it. A data scientist can also create visualizations–which is the subject of today’s episode. Aurelia Moser has a specialty in mapping and data visualization. We discussed how to create effective visualizations of our data, and the tools that can be used to collect and present our data. Whether you work at the New York Times or a small tech company, data visualizations are important, because visualizations can be used to communicate important trends across an organization. Aurelia will also be speaking at the upcoming Strata + Hadoop World Conference in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Big data is yesterday–fast data is now. FiloDB is a reactive columnar OLAP database that is built on Cassandra and Spark. Today’s guest is Evan Chan, creator of FiloDB. In our discussion today, we talk about the use cases of an OLAP data store. Evan explains how to tackle the problem of video analytics–if you have ever found yourself asking how a company like YouTube or Netflix or Ooyala performs analytics on millions of users watching millions of videos, this episode is for you. By combining the database features of Cassandra with the data processing power of Spark, Evan created FiloDB to help solve this type of analytics problem. Evan will also be presenting at Strata + Hadoop World in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Cassandra can serve as both the real-time data store for online transactional applications, as well as the read-intensive database for data warehousing operations. In order to combine these two use cases into a single database, Apache Cassandra required lots of innovation. In today’s episode of Software Engineering Daily, we discuss the internals of Cassandra. Tim Berglund takes us through how Cassandra performs reads and writes, how Cassandra offers tunable consistency, and what inspirations Cassandra took from the Amazon Dynamo paper as well as the Google BigTable paper. Tim will also be presenting at Strata + Hadoop World in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Hadoop was created in 2003. In the early years, Hadoop provided large scale data processing with MapReduce, and distributed fault-tolerant storage with the Hadoop Distributed File System. Over the last decade, Hadoop has evolved rapidly, with the support of a big open-source community. Today’s guest is Mike Cafarella, co-creator of Hadoop. Mike takes us on a journey from past to present. Hadoop was based on the Google File System and MapReduce papers, and so Mike and I talk about what it was like to work on a distributed file system in 2004, and the challenges of implementing real software systems based on white papers. We also discuss YARN, and the wave of innovation that YARN enabled within the Hadoop ecosystem. Mike will also be presenting at Strata + Hadoop World in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

When a company gets big enough, there is so much data to be processed that an entire data engineering team becomes responsible for managing this data and making it available to other teams. Airbnb is one such company. Max Beauchemin works on the data engineering team at Airbnb, where he creates infrastructure and tooling for managing data. In this episode of Software Engineering Daily, we talk about AIrflow, a workflow scheduler that assists in job processing. If you don’t know what a workflow is, or a job, we will explain that in this episode. Max and I also talk about Panoramix, a data slicing and visualization tool that helps data scientists and business analysts understand large volumes of data. Max will also be presenting at Strata + Hadoop World in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Diagnosing illness today requires the trained eye of a doctor. With machine learning, we might someday be able to diagnose illness using only a data set. Today on Software Engineering Daily, we are joined by David Kale, a researcher at the intersection of machine learning and clinical data. We discuss the machine learning and research techniques he is using to diagnose illnesses using neural networks, and we also talk about the challenges of performing data science in hospitals, where the data is mostly confidential. David will also be presenting at Strata + Hadoop World in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ruby on Rails has had the most commits made by its creator, David Heinemeier Hansson. The next most frequent contributor to Rails is Aaron Patterson, our guest on today’s episode of Software Engineering Daily. Topics discussed in today’s episode include Ruby relative to JavaScript, Rails relative to Linux, and DHH relative to Tenderlove.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Microsoft and Amazon are the tech giants of the Pacific Northwest. These two companies shape Seattle technology, and Todd Bishop has been writing about them for over a decade. Todd is the co-founder of GeekWire, an online media company focused on technology in the Pacific Northwest. We talk about the past and present of Microsoft and Amazon, what it means for an engineering company to be good or evil, and what it’s like to be a software journalist.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Socket.io enables realtime bidirectional communication. But what does “realtime” actually mean? Today’s guest is Guillermo Rauch, the creator of Socket.IO, a widely used technology for client server communication. We discuss the nature of real-time apps like Uber and Google Docs, and talk about the API and usage of Socket.IO.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Technology careers are not as accessible to certain demographics, such as women. Sarah Allen is working to change that through education and outreach. Sarah Allen created RailsBridge and Bridge Foundry, a pair of organizations working to make technology more accessible to people who are underrepresented in tech. RailsBridge workshops are a fun and free way to get started with Ruby, Rails or other technologies. RailsBridge was so successful that Sarah created Bridge Foundry, which has an even wider variety of free workshops, from Clojure to Go to Mobile. This is an inspirational episode of Software Engineering Daily, and is a great learning opportunity for anyone interested in building a community.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Continuous integration and deployment are important tools for modern software development. With continuous integration and deployment, individual engineers can push code without waiting to synchronize with the rest of the team on a big software release. Today on Software Engineering Daily, Flo Motlik, the CTO of Codeship, joins us to discuss continuous integration, dev ops, and microservices. In full disclosure, Codeship is a sponsor of Software Engineering Daily, but we would be doing this interview whether or not that was the case, because Flo has a lot to say about software engineering. In this show, we get into conversations and case studies of how software teams take continuous deployment from theory into practice.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Monsanto is a company that is known for its chemical and biological engineering. It is less well known for its data science and software engineering teams. Tim Williamson is a data scientist at Monsanto, and on today’s show he talked about how he and a small group of engineers at Monsanto dramatically shifted the culture around data science-driven genetic engineering. In this episode, Tim explains how useful graph databases are for modeling the genetic lineages, and talks about how Monsanto manages simulations and experiments on their genomics software pipeline. Tim also talks about how just a few engineers can create a cultural shift within a large company like Monsanto using the leverage allowed by software.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

This episode is a republication from my interview with Leslie Lamport on Software Engineering Radio. Leslie Lamport won a Turing Award in 2013 for his work in distributed and concurrent systems. He also designed the document preparation tool LaTex. Leslie is employed by Microsoft Research, and has recently been working with TLA+, a language that is useful for specifying concurrent systems from a high level. The interview begins with a definition: a distributed system is a multiprocessor system in which the time required for interprocess communication is large compared to the time for events within a single processor–in other words, it takes longer for interprocess communication than it does for a process to look at its own memory. Alternatively, a distributed system is one in which processors communicate by sending messages. Leslie goes on to talk about how he became interested in distributed systems, and describes the story behind his paper about the Paxos algorithm. The goal of Paxos is to maintain consensus in an environment with unexpected faults (otherwise known as Byzantine faults). After the discussion of Paxos, Jeff asks Leslie about his recent talk “Thinking for Programmers,” which emphasizes the benefit of having a specification prior to writing actual code. “Specification” can mean a variety of things, but predicates and next-state relationships provide a mathematical rigor that is well-suited to distributed and concurrent systems. The conversation concludes with Jeff asking Leslie about how a programmer can build the mental resolve to work through a difficult problem.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Spark is replacing MATLAB in the domain of computational neuroscience. The constraints of running MATLAB on a single machine can’t support the demands of neuroscience, which has huge collections of images and time-series data sets. Jeremy Freeman is a computational neuroscientist who is adopting Apache Spark to be able to analyze these giant data sets that do not fit on a single machine. But Apache Spark was not designed with neuroscience in mind. For this reason, Jeremy has helped to build several libraries on top of Spark. Thunder is a library for standard, distributed representation of data. Lightning is an API for reproducible web visualizations. These abstractions sit on top of Spark, and add a layer of usability. As it turns out, solving these problems for neuroscience have produced tools that are useful in a variety of other domains. In our discussion with Jeremy Freeman, we talk about Apache Spark, neuroscience, and the technological and cultural problems faced by traditional academic research.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

University research produces numerous papers about software engineering. Unfortunately, many of the problems explored by these software engineering researchers have no actual application. These papers fall into The Black Hole of Software Engineering Research. Andy Ko is an associate professor at the University of Washington. When he worked as a CTO of AnswerDash, he experimented with some of the software engineering research coming out of academia and industry. Today on Software Engineering Daily, we discuss Andy’s results. Our conversation spans the topics of education, big companies, and startups.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

NewSQL is a class of modern relational databases that seek to provide the same scalable performance of NoSQL systems for OLTP, while still maintaining the ACID guarantees of a traditional database system. VoltDB is a NewSQL database, designed in part by Turing Award Winner Michael Stonebraker. Today’s guest is John Hugg, a founding engineer at VoltDB. Today’s show covers the internals of VoltDB, including concepts like tunable consistency, database replication, and the advantages of a deterministic logical log.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

React is a set of technologies started by Facebook and open-sourced in 2013. Since then, it has become widely accepted as the best JavaScript library for building user-interfaces. A dedicated community around React has developed as well, and with any large software community, conferences are a necessity. React.js Conf is a yearly conference in San Francisco where React developers talk about the latest developments in the React ecosystem. React started with user interfaces, but has evolved to be so much more. From data flow patterns like Redux, to hardware libraries and virtual reality–React is no longer just a JavaScript library; it is a philosophy for how to build futuristic technologies today. Brent and Jeff discuss all of this, as well as Exponent, a web development platform for building native mobile apps with JavaScript – using React Native. Brent Vatne is the organizer of React.js Conf and a software engineer at Exponent.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Internet Explorer, Google Chrome, Firefox–it’s easy to forget that these modern browsers descended from the war between Microsoft and Netscape. Today, we hear from a software engineer who was on the front lines of that war, back in 1992. Eric Sink was one of the original developers of Spyglass, a browser that was licensed to both Microsoft AND Netscape. If you want to understand why that happened, and what the inside story of the browser wars is, and what the business of selling browser licenses was like in 1992, check out this episode of Software Engineering Daily.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In this episode, Jeff and Jeremy discuss Internet.org and the criticism it has received from organizations like the EFF. Internet.org is a program created by Facebook whose purpose is to expand the reach of the internet by connecting people in developing nations like India. However, it has come under fire for its failure to provide “the whole internet” and limiting access to a subset of websites which have to be approved by Facebook. Critics call this a violation of net neutrality and argue that it betrays what the internet stands for – openness and fair access to all destinations. This episodes jumps into the nuances of the debate, with Jeff playing counterpoint to the EFF’s stance – that Internet.org is not the internet. Jeremy Malcolm is a technologist and lawyer who works for EFF’s international team on issues such as intellectual property, network neutrality, Internet governance, and trade.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Design is more important than ever. There are so many websites being created every day, that it is not enough to differentiate your product with rock solid engineering. Your product has to be beautiful, intuitive, and tested for usability. You don’t want all of the hard work of the product and engineering teams to go to waste, so on this episode, the focus is design. Tracy Osborn is the creator of WeddingLovely, a beautifully designed two sided marketplace website, where people who are getting married connect with wedding vendors. Today, Tracy discusses “design for nondesigners”. Whether you are an engineer, a marketer, or a business analyst, just learning a few simple tactics about design can take you far.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

JavaScript is everywhere, on the browser, the server, and now on hardware. Finally–the Internet of Things is upon us — and it is powered by JavaScript. Even if you consider yourself a web developer, and you have no interest in “maker” culture, keep your eye on this space. Factories in Shenzhen are making better, more modular parts, and the same supply chain economics that are driving down the cost of smart phones and smart cars, will give hobbyist engineers all the simple hardware components they could ever need. This will power a future of proliferated devices that have internet connectivity and “talk” to each other to enable user experiences we wouldn’t dream of today. And naturally, with this will come problems that we cannot anticipate as well. On today’s episode, Andrew discusses how JavaScript can be used on hardware. Jeff and Andrew talk about the players in the hardware space, as well as how the big cloud players like AWS and Azure are positioning themselves to be the data lakes for the internet of things. Andrew is giving a talk about this at the 2016 O’Reilly Fluent Conference. If you want to win a free ticket to Fluent, tweet about your favorite episode of Software Engineering Daily and tag @fluentconf and @software_daily. Andrew Chalkley is a hardware hobbyist, software engineer and a full-time teacher at Treehouse.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Disabled individuals account for over a billion people worldwide. This represents the world’s largest minority on the Web, with $220 billion in discretionary spending power in the United States alone. Yet, the accessible web is more than a financial issue – it is also a moral issue, and it is crucial that we make the web accessible to ensure an equitable information platform. Today’s guest Nic Steenhout joins us to explain how to develop software with accessibility in mind. At the upcoming Fluent Conference, he will be giving a talk entitled Don’t turn off that JavaScript just yet, referencing JavaScript’s history of causing accessibility problems. If you want to win a free ticket to Fluent, tweet about your favorite episode of Software Engineering Daily and tag @fluentconf and @software_daily. Nic Steenhout is a software engineer and an accessibility consultant at Simply Accessible.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Reactive programming emphasizes writing code that is readily responsive to events. It is an increasingly popular paradigm with highly interactive websites like Netflix. It draws on the value of functional programming calls like map, reduce, and flatmap. Matt joins us today to chat about reactive programming, the observer pattern, and his thoughts on where the future is going. If you are a fan of functional programming or JavaScript, you will enjoy this show. Matt is also speaking at the 2016 O’Reilly Fluent Conference. If you want to win a free ticket to the conference, tweet about your favorite episode of Software Engineering Daily and tag @fluentconf and @software_daily. Matt Podwysocki is the author ReactiveX JS and a software engineer at Microsoft.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Node.js powers an increasing number of applications in the modern web. As node’s popularity grew, npm evolved in parallel as its default package manager and it has become a robust system for sharing and developing node programs. Yet today, npm is growing beyond its roots, and poised to become a generalized framework for all kinds of workflows within web development. Laurie will be giving a talk at the 2016 O’Reilly Fluent Conference. If you want to win a free ticket to the conference, tweet about your favorite episode of Software Engineering Daily and tag @fluentconf and @software_daily. Laurie Voss is the CTO of npm, Inc.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Following the successful experiment of History of Hadoop, we are doing another Saturday experiment: an editorial podcast. Let us know your thoughts via Slack, Twitter, or email! Our podcast errs on the side of technical rigor. Whether the topic is distributed databases, microservices, Soylent, Uber, or Dwarf Fortress, we try to separate hype from substance, deferring the narrative to the guest. With that deference there is editorial objectivity on the part of Software Engineering Daily. One Slack channel member said “when Software Engineering Daily’s opinions are announced, that takes time away from the guest.” Fair enough. However, as with any journalistic organization, we have opinions. On SE Daily, we stream objectivity and batch subjectivity. This episode is willfully subjective. It has no guest. It is a monologue editorial inspired by Developer Tea. Immutable laws are rare in software engineering, and when an engineer claims to have found one, that engineer is usually regarded with skepticism. General principles are more welcome. In this post and podcast episode, I convey some loose philosophies about modern software engineering. These are strong opinions weakly held. I welcome debate and discussion. Software is a new field and nobody knows how to do it. If someone says you are unqualified and therefore you must do maintenance work, you should question that person. We have an upside down system where the people who are paid the least do the crappiest work. They tend to be young and naive. This is not an axiom. The narrative that is sold to young engineers by giant companies is the following: take your $80k/yr job, do the software maintenance which makes the company $1 million, and hate your life. After you have spent enough time in the first tier of the intellectual strip mine, we will make you an SDE 2, where you can do slightly higher level refactoring for $150k a year, which will make the giant company $5 million. This is what is called an arbitrage. We have an assembly line mindset left over from the industrial age. Software is more of an artisanship. Don’t believe that you are a replaceable cog. Don’t believe the one-size-fits-all interview process with whiteboarding problems. These serve to grind away your individuality and make you feel like an assembly line worker. The planning and design process is an art, but once the requirements are in place you can proceed more deterministically. The same is true for the other quantitative activities I have taken part in–poker, music, and writing. As Michael Rosenthal and I discussed, the question of art vs. science is the same question as strategy vs. tactics. I learned this very early on playing poker when I had to leave that career, and I had tightly coupled poker to my identity. If you make your job the same thing as who you are, then your self-worth is defined by those who are judging you in your job. Your job is a means to service your own higher purpose. When you take an action on your smartphone, there is latency before that action is ingested. Servers sometimes will lie to you, but servers tend towards eventual consistency. The world works the same way. In the short term, human systems lie to us all the time, but the world tends toward eventual consistency–the truth eventually presents itself. A decent analogy is the efficient market hypothesis: slowly efficient markets are an eventually consistent process. The world is a distributed system–what is the consequence of this? We have to do the arduous risk and reward calculations that are mandatory for every distributed systems programmer. Long-tail failures can and do occur. In a distributed system, we often prioritize safety over liveness. In a distributed operating system, the programmer takes all precaution to avoid data loss. Similarly, if a real life scenario presents a small probability of giant downside risk, you should take huge precaution. If someone offers you to roll a die with 1000... Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Lin Clark is today’s guest on Software Engineering Daily, and she joins Jeff to talk about Code Cartoons, a webcomic that explains Facebook’s open source projects like Flux and Relay with the same elegance and creativity as XKCD. Lin explains why cartoons can be a surprisingly effective way to introduce technologies to people without the fuss and intimidation of written tutorials or documentation. At the 2016 O’Reilly Fluent Conference, Lin Clark will be speaking about Code Cartoons and the technologies that have sprung up around React JS. Software Engineering Daily is giving away a free ticket to the Fluent 2016 conference, which takes place on March 8-10 in San Francisco. If you want to be entered to win this free ticket, tweet about your favorite episode of Software Engineering Daily and tag @fluentconf and @software_daily. Lin Clark is the creator of Code Cartoons, and a Senior Developer Tools Engineer at Mozilla.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Prepare for the post-browser world, says today’s guest Jonathan Stark. We are moving beyond the browser, into a world where the web is consumed by a variety of front ends. Jonathan is a speaker at the upcoming O’Reilly Fluent Conference in San Francisco, and you can win a free ticket to the conference. To be entered into a random drawing for that ticket, send us a tweet about your favorite episode of Software Engineering Daily between now and February 22nd. Include the hashtag #fluentconf and and tag us @software_daily to make sure we can see your tweet. Fluent is a conference about the web platform–which certainly includes the browser, but also mobile apps, VR experiences, and smart objects connected to the web. In today’s episode, Jeff and Jonathan explore what adjustments developers should make in this post-browser world. Jonathan Stark is a mobile consultant who helps consumer brands thrive in the post-PC era.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

You keep hearing about containers, and maybe you have even used Docker in production. Now, it’s time to move beyond Docker. In today’s episode of Software Engineering Daily, Steve Pousty talks about OpenShift, a platform from RedHat that helps engineers leverage the power of containers and the dev ops harmony of microservices. To hear these buzzwords, you can always listen to Software Engineering Daily. But to see these buzzwords demonstrated in live presentations, the 2016 O’Reilly Fluent Conference is coming up March 8-10 in San Francisco, and you can get a chance to win a free ticket by tweeting at us about your favorite episode of Software Engineering Daily. Include the hashtag #fluentconf and and tag us @software_daily to make sure we can see your tweet. Steve Pousty is a Developer Advocate (and PaaS Dust Spreader) at Red Hat.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In today’s episode of SEDaily, Jeff and Austin discuss why designing for a drunk user is a competitive advantage. Austin also explains why engineers need to think about UX and how they can incorporate UX considerations into development workflows. Austin Knight is a designer and UX engineer at Hubspot. Austin will be speaking at the upcoming Fluent Conference, and Software Engineering Daily is giving away a ticket to Fluent, which will be held March 8-10 in San Francisco. To be entered in a random drawing for this free ticket, tweet about your favorite episode of Software Engineering Daily between now and February 22nd. Make sure to tag us @software_daily and Fluent @fluentconf so we see your tweet.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

This week of shows on Software Engineering Daily focuses on O’Reilly’s Fluent Conference, a yearly software conference that focuses on the web and the tools used to build modern applications. Software Engineering Daily is giving away a ticket to The Fluent Conference, which will be held March 8-10 in San Francisco. To be entered in a random drawing for this free ticket, tweet about your favorite episode of Software Engineering Daily between now and February 22nd. Include the hashtag #fluentconf and and tag us @software_daily to make sure we can see your tweet. In today’s show, Jeff chats with Peter Cooper and Simon St Laurent, the organizers of Fluent. They discuss how the web is transitioning beyond the desktop and how to keep up with ever evolving javascript frameworks. This was also unique chance as well to learn why software conferences are so important.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

This episode is different from the traditional interview format of Software Engineering Daily, and focuses on the history of Hadoop. Thanks to Marco Bonaci for allowing us to republish this in audio format. You can find the original post here: History of Hadoop

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Operating a data center has been turned into a service. For most new companies, managing real-world servers is no longer an issue. But one exception is if your startup is a company offering the data center as a service. In this episode, Jeff and Sam talk about the engineering that goes on at DigitalOcean, and how the company differentiates itself from its competitors. Sam Kottler is an engineering manager at DigitalOcean, and previously worked at Red Hat.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Monoliths versus microservices – the architectural debate continues across the internet. SoundCloud is a popular music company that experienced the rapid development benefits of a monolith, as well as the long-term technical debt. That technical debt has since been relieved by a move towards microservices. In this episode of Software Engineering Daily, Lukasz joins Jeff to talk about the realities of moving from a monolith to a microservices architecture, and walks through the lessons learned at SoundCloud. Lukasz Plotnicki is a consultant and software engineer at ThoughtWorks, which helps clients address their software development challenges and needs.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Stream processing engines are often the component of data engineering stacks with the most variety, with the big data ecosystem offering several compelling options. Assessing the differences between projects like Flink, Storm, and Spark Streaming is difficult without an agreed upon set of metrics to compare them on. Fortunately, the Yahoo engineering team created a set of benchmarks to do exactly this. On today’s episode, Jeff and Bobby compare streaming frameworks through the performance on several of Yahoo’s benchmarking tests. Bobby Evans is an architect at Yahoo working on streaming frameworks, primarily on Apache Storm. He is also the Apache Storm Incubating PMC at The Apache Software Foundation.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software engineering combines art, science, and philosophy. On the Developer Tea podcast, Jonathan Cutrell explores how engineers can improve their technical ability as well as their interpersonal skills and mental clarity. On this episode of Software Engineering Daily, Jeff and Jonathan about focus, career development, and software podcasting. Jonathan Cutrell is the founder and host of Developer Tea. He is also the Director of Technology at Whiteboard.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Matplotlib is a python plotting library inspired by MATLAB. It allows developers to easily visualize their data using simple functions. In this episode, Jeff and Ben discuss why visualizing data is integral for the human experience, and how they it can make drawing conclusions from data more meaningful. Ben relates this to his experience studying atmospheric data in the wild, and how his work as a researcher drives his need to visualize the vast amounts of raw data generated by weather systems. Ben Root is a core contributor to Matplotlib and works as a scientific programmer for Atmospheric and Environmental Research Inc.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Keras is a minimalist, highly modular neural networks library, written in Python and capable of running on top of either TensorFlow or Theano. It was developed with a focus on enabling fast experimentation. In this episode, François discusses the state of deep learning, and explains why the field is experiencing a cambrian explosion that eventually may taper off. He explains the need for Keras and why its simplicity and ease makes it a useful deep learning library for developers to experiment and build with. François Chollet is the author of Keras and the founder of Wysp, learning platform for artists. He currently works for Google as a deep learning engineer and researcher.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Software is changing every domain including Marketing, Healthcare, and even Agriculture. Government is one specific area where the potential for improvement is obvious. Governments are often defined by abstract rules, transactions, permissions and hierarchies, grounded on some fundamental principles – this is striking similar to the same tenets that software and computer science are based on. In that sense, could software be leveraged in this unique industry to create massive change? OpenGov is one company trying to find out through application of software in local city and state governments. The company currently offers a web-based financial visualization software that focuses on making government budget and financial data accessible by modeling and presenting it in a user-friendly way. But it is not farfetched to see the potential being greater and broader in scope. In this episode, Jeff discussed with Andrew how OpenGov is using technology to improve government. Their conversation focuses in on the front-end technologies used to create the webapp, as well as “microservice fatigue” and the other various architectural concerns within the team. Andrew Clark is a software engineer at OpenGov.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In this episode, Jeff and Derrick sit down to discuss the state of tech journalism, and how it has kept up with the rapid pace of technology innovation. Additionally, they discuss the evolution of the “big data” stack and where it is headed. The conversation focuses in on Apache Mesos and Mesosphere, the company building the Datacenter Operating System (DCOS). Derrick Harris is a long time tech journalist, having worked at GigaOm and Fortune as a writer covering tech topics. He currently works at Mesosphere, where he is a tech evangelist.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Incarceration in the United States is often a punitive exercise rather than a rehabilitative one. As a consequence, recidivism is the norm – according to a 2012 report by the California Department of Corrections and Rehabilitation, just over 65% of those released from California’s prison system return within three years. San Quentin State Prison in California is challenging this issue by trying to equip inmates with valuable skills while they are incarcerated. The Last Mile is a program that works with prisoners to help them build relevant skills in technology so that they can more easily transition to productive employment once they are out of prison. Specifically, it teaches inmates how to program, oftentimes starting at the basics like how to use a computer, and how hypertext works. In addition to the challenges of teaching students who may not know computer basics, the program has to work with the constraint of not having internet access. In this episode, Wes and Jeff discuss the realities of incarceration in the United States, and how teaching inmates how to code is a remarkably empowering way to combat and weaken the prison-industrial complex. Wes Bailey is the Director of Program Operations at The Last Mile, and a self-taught developer.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Governments are often laggards in the adoption of new technology. Every aspect of life, from transportation to healthcare, are being affected by the convergence of several trends including connected mobile devices, cloud storage and distributed systems. Yet federal services are in many cases still incapable of delivering the best and most affordable services to citizens. The most visible instance of the U.S. government struggling to modernize was the implementation of Healthcare.gov. What was originally a modest budget for deploying the web marketplace of the Affordable Care Act ballooned into a costly implementation that required hiring external consultants and developers to fix. In response to this dysfunction, 18F was formed with the mission to simplify the government’s digital services. Sarah joins Software Engineering Daily to explain how and why 18F was formed, and how it plans to improve government from within in the years to come. Sarah Allen is a Presidential Innovation Fellow who serves as a developer and product manager for 18F. She is also a co-founder of Bridge Foundry, an organization that helps encourage and empower people who are learning to code.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Riak is a distributed NoSQL key value data store. It was created by Basho, taking inspiration from the Amazon Dynamo paper. In this episode, Jon and Jeff discuss the benefits and different implementations of non-relational distributed data stores, and why Riak is a compelling option. Jon Meredith is the chief architect at Basho.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Spark has skyrocketed in popularity lately, arguably surpassing Hadoop as the hottest big data technology as of late. Even IBM has thrown its weight behind the framework, calling it the “most important new open source project in a decade”. In this episode, Holden joins Software Engineering Daily to discuss why Spark is growing in popularity and how developers can begin learning the framework. Holden Karau is a principal engineer at IBM working with Apache Spark. She is also an author of Learning Spark, a technical guide for developers new to the data processing framework.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Drill is a schema-free SQL query engine for Hadoop, NoSQL, and cloud storage. Drill is unique in that it uses a JSON data model instead of a relational model like many traditional engines. Additionally, it supports on-the-fly schema discovery, which allows execution to begin without knowing the structure of the data. As a result, Drill can work with data that is constantly updating or changing. Tomer Shiran is the founder of the Apache Drill project and the CEO of Dremio.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Machine learning is something that many business are starting to tack onto their existing processes. Yet, to add machine learning capabilities after the fact is often a fool’s errand. Joshua argues that machine learning cannot be an afterthought, but rather must be custom developed to suit the specific problem or question that each company is trying to answer. His company, Wise.io, tackles this challenge of helping business build ground up machine learning applications that generate accurate predictions for use in an array of business processes. Joshua Bloom is the cofounder and CTO of Wise.io. He is also an astrophysicist, and a professor of astronomy at UC Berkeley.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data science has emerged as an integral discipline alongside software engineering. In this episode, Jeff and Srini discuss the rise of the field, what it means to be a data scientist, and how the field will evolve as more businesses adopt data driven decision making. Srini also explains his thoughts on the essential qualities of a data scientist, and how educating data scientists is a growing challenge in a world that is flooded with “big data”. Srini Kadamati is a data scientist at Dataquest, where he creates content to help people learn data science.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

WebTorrent is a streaming torrent client for the browser that can be used without any additional plugins, extensions, or installations. It is written entirely in JavaScript, and uses WebRTC to handle peer-to-peer communication. There are some limitations to making BitTorrent work over the web using WebRTC, including the inability for a browser-based WebTorrent client or “web peer” to communicate directly with a torrent client. However, the potential for every browser to contribute to the hosting and streaming of files is unprecedented. Imagine if large content sites like Netflix or YouTube did not have centralized hosting, and instead relied on its users to seed the content to each other using their web clients. This enables not only more efficient networks, but also the ability for each user to contribute to and be part of the content delivery network. Feross Aboukhadijeh is the creator of WebTorrent. Previously he built PeerCDN, a peer-to-peer content delivery network, which was acquired by Yahoo in 2013.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Django is an open source web development framework that was initially released in 2005, around the same time as Ruby on Rails. In many ways, the two frameworks are similar – they expedited web development by modularizing common components of web apps and enabling developers to quickly prototype and launch their products. They both encouraged the use of the MVC architectural pattern, with each respectively acting as the controller, allowing developers to use their own front end and database technologies. But the path of the two frameworks diverged in other parameters, including mainstream adoption. Ruby on Rails became the sweetheart in Silicon Valley and elsewhere, while Django was loved by a core constituency of pythonistas who had used Python in other applications and fallen in love with it. At the peak of the frameworks’ popularity, there was a common question from new developers asking whether they should first learn Django or Rails – the answer often boiled down to whether you wanted to use Python or Ruby. In this episode, Frank discusses the history and growth of Django, and explains how it evolved in comparison with other frameworks. Frank Wiles is the President of the Django Software Foundation, and partner at Revolution Systems. He worked at the Lawrence Journal-World newspaper, where Django was initially conceived and created.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ruby on Rails is a web app framework released 10 years ago that influenced the way websites were being built. Rails skyrocketed to popularity in the late 2000’s and empowered many small companies to quickly build and maintain robust web apps. While it is still a mainstay in web development, it has been overshadowed as of late by JavaScript, and Node.js on the backend. David joins Software Engineering Daily to discuss how Ruby on Rails has evolved to keep up with newer programming paradigms and why it is still an excellent choice to build web apps with nowadays. David Heinemeier Hansson is the creator of Ruby on Rails and a partner at Basecamp. Rails was extracted from the work done at Basecamp, and open-sourced for the public to commit to and use for their applications.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Marty Haught is a director of Ruby Central, a nonprofit dedicated to the support and advocacy of the worldwide ruby community. Ruby Central also hosts the RubyConf and RailsConf software conferences.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Michael Hartl is the author of The Ruby on Rails Tutorial, a widely acclaimed guide to learning how to build Ruby on Rails webapps.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Soylent is a meal replacement beverage, available in both liquid and powdered forms. Soylent is designed with the goal of meeting all nutritional requirements for an average adult. John Coogan is the CTO of Soylent.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Druid is a column-oriented distributed database that excels as a data warehousing solution for fast queries on large data sets. Fangjin Yang is a core contributor to Druid, and is currently co-founder and CEO of Imply, which helps build interactive analytics powered by Druid. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Brian Kernighan is a professor of computer science at Princeton University and the author of several books, including “The Go Programming Language” and “The C Programming Language”, a book more commonly referred to as K&R. Professor Kernighan also worked at Bell Labs alongside Unix creators Ken Thompson and Dennis Ritchie and contributed to the development of Unix.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kitematic is an open source project built to simplify and streamline using Docker on a Mac or Windows machine. It allows users to get up and running on Docker with a single click, and provides a user interface that makes running and managing Docker files easy. Kitematic was acquired by Docker in March 2015. Sean Li is the co-founder and Chief Product Officer of Kitematic. He is now working on product design and developer experience at Docker.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Distributed computing is becoming an increasingly important and complex field within software engineering, with numbers of new data engineering and infrastructure frameworks being released each year. For a beginner, it can be tough to know where to start, so this episode may be considered a primer to learning about distributed systems independently. Alvaro Videla is a core developer for RabbitMQ and a distributed systems blogger, and he joins Software Engineering Daily to explain some of the core concepts of distributed systems.

Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Michael Hausenblas is a developer and cloud advocate with Mesosphere, which builds the Datacenter Operating System (DCOS), a distributed OS that uses Apache Mesos as its kernel. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Vue.js is a lightweight front-end JavaScript framework that makes it simple to begin prototyping and building web interfaces. Vue.js provides a flexible API for MVVM data bindings, and can serve as an alternative to other front-end frameworks like Angular and React. Evan You is the creator of Vue and a core developer at Meteor. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Quora is a Q&A website where questions are asked, answered, edited and organized by its community of users. Shreyes Seshasai is the Director of Engineering at Quora. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Triplebyte is a technical hiring platform that vets engineers using a comprehensive evaluation platform and connects them to companies that are interesting in hiring them. Triplebyte was part of the Y Combinator summer class of 2015. Ammon Bartram is the Chief Data Officer and cofounder of Triplebyte. Previously, Ammon was lead video developer at Justin.tv and also cofounded SocialCam. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Eric Elliott is a JavaScript advocate, and the author of Programming JavaScript Applications. He is also an entrepreneur tackling social issues including homelessness and inequality in the tech industry. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Tessel is an open-source microcontroller that is programmable in JavaScript and compatible with Node.js. Kelsey Breseman is a hardware engineer and Steering Committee Member on the Tessel Project. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The stream processing paradigm is increasingly being adopted by applications that need to process and handle large volumes of data. Apache Kafka is an open-source distributed publish-subscribe messaging system that is built to support streaming data processing. Neha Narkhede is the one of the creators of Apache Kafka, which she built to address engineering challenges while working at LinkedIn. She is also the co-founder of Confluent, which builds enterprise products around Kafka. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Code Fellows in-person code school offering web and mobile development training in Seattle, Portland, and Chicago. Dave Parker is the CEO of Code Fellows and organizer of Seattle Startup Week. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Scott Hanselman is a technologist, teacher and podcaster who works for Microsoft on its web platform team. Scott hosts several podcasts including Hanselminutes and This Developer’s Life, which cover various topics in software. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

TensorFlow is an open source machine learning library intended to bring large-scale, distributed machine learning and deep learning to everyone. Google recently released the framework to the public as a second-generation API, having learned from the successes and failures of DistBelief. Greg Corrado is a senior research scientist and tech lead at Google, where he focuses on the research areas of machine intelligence, machine perception and natural language processing. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Rebecca Garcia is the creator of GeekGirlWeb and founder of CoderDojo NYC. She spends much of her time educating people about the importance of STEM education, particularly in the domain of programming. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Todd Bishop from The GeekWire Podcast recently interviewed Jeff for a discussion on various topics including programming languages, coding bootcamps and the philosophical question of whether programming is an art or a science. Thanks to GeekWire for the opportunity! You can find the show and more at Geekwire: http://www.geekwire.com/2015/is-software-development-art-or-science-geekwire-podcast-with-software-engineering-daily/ Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Spotify is a streaming music service that uses data science and machine learning to implement product features such as recommendation systems and music categorization, but also to answer internal questions. Boxun Zhang is a data scientist at Spotify where he focuses on understanding user behavior within the product. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The Internet of Things era is upon us, and with it the related concerns of security, privacy and reliability of the connected systems. This episode explores these issues and how companies can prevent these problems through better development processes and lifecycle management. Anders Wallgren is the CTO of Electric Cloud, which builds products to help companies optimize their software build, test, and deployment process. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

John K. Paul is an engineering manager and speaker, who has given several talks on Javascript, including JavaScript, the Real Bad Parts. He is the CTO of InRhythm and also an organizer of NYCHTML5. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Richard Golden is the host of Learning Machines 101, a podcast that covers artificial intelligence and machine learning topics. Dr. Golden is also a full-time Professor of Cognitive Science and Electrical Engineering at UT Dallas. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Real-time stream processing of data is becoming widely adopted in the efforts to manage and process “big data”. Some of the top frameworks for processing data streams include Storm, Spark, Samza and Flink. Satish Mittal is an architect at InMobi, an ad platform that needs to deal with processing large volumes of ad data with tight time demands. He recently conducted an analysis of streaming frameworks to determine which of the tools was right for the application at InMobi. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Uber is a transportation and logistics company that manages many aspects of its ride-sharing services through mobile apps and distributed technology. Uber faces unique challenges in rapidly scaling its services internationally, and at one point increased its developer headcount from 200 to over 1000 in less than a year. Matt Ranney is the Chief Systems Architect at Uber and was previously a founder and CTO of Voxer. At QCon San Francisco, he gave a talk called Scaling Uber. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

React Native is leading to a future where a developer can build native experiences on web, iOS and Android platforms without having to write 3x the code. Christopher Chedeau is an engineer at Facebook who works on the React team. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Data engineering is a newly emerging field that exists at the intersection of software engineering and data science. Data engineers are increasingly in demand for their abilities to use the various open source big data frameworks to build efficient and robust data pipelines. David Drummond and Austin Ouyang are program directors for the Insight Data Engineering Fellows Program, where they help train data engineers that can immediately start contributing in the industry. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

This is episode 100 of Software Engineering Daily, and is a behind the scenes episode on the show that explores the challenges and motivations behind building a daily software podcast. Pranay Mohan is the Producer of Software Engineering Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Let’s Encrypt is a free, automated, and open certificate authority developed by the Internet Security Research Group (ISRG). The ISRG is a non-profit whose mission is to reduce financial, technological, and education barriers to secure communication over the Internet. Josh Aas is the Co-founder and Executive Director of the ISRG. He is also a Senior Technology Strategist at Mozilla. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Free Lossless Image Format (FLIF) is a novel lossless image format which outperforms PNG, lossless WebP, lossless BPG and lossless JPEG2000 in terms of compression ratio. Jon Sneyers is one of the core contributors to FLIF. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Popcorn Time was a free, open-source application that allowed for streaming of movie and television show torrents. Niv Sardi is an activist and developer based in Argentina, who previously worked on the main fork of Popcorn Time, popcorntime.io. Niv is also the founder of the Butter Project, a spin out project from Popcorn Time. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Information retrieval and search engineering are becoming more intertwined with machine learning and natural language processing, leading to a wealth of work to be done in the field. Grant Ingersoll is a founder and CTO of LucidWorks, which helps clients make sense of their data and deploy search applications built on Apache Lucene and Solr. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

CodeCombat is a multiplayer live coding strategy game where the players write code to control their characters. Nick Winter is the CEO and Co-founder of CodeCombat. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Seth Godin is a writer, speaker, and entrepreneur. He is the author of many books, including most recently, What To Do When It’s Your Turn. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Technical debt, referring to the compounding cost of changes to software architecture, can be especially challenging in machine learning systems. D. Sculley is a software engineer at Google, focusing on machine learning, data mining, and information retrieval. He recently co-authored the paper Machine Learning: The High Interest Credit Card of Technical Debt. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

MotherCoders is a program that helps mothers gain technical skills with the support of a community and benefits like on-site childcare. Tina Lee is the Founder of MotherCoders, having created the program to meet her own need after trying to find existing support systems while learning how to code. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

SQLite is an SQL database engine that can be used in applications as a library for quick read-write operations. Unlike other database management systems, SQLite is not a client-server database designed to run in a data center. D. Richard Hipp is the architect and primary author of SQLite and the Fossil SCM. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Trisha Gee is a Java developer and Java Champion. She is currently a Developer Advocate at JetBrains, where she focuses on educating developers on various tools and IDEs. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Apache Flink is an open-source framework for distributed stream and batch data processing. Stephan Ewen is a committer and PMC member of the Flink project, and the CTO of Data Artisans. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Wolfram Research makes computing software powered by the Wolfram language, a knowledge-based programming language that draws from symbolic and functional programming paradigms. Stephen Wolfram is the Founder and CEO of Wolfram Research, and also the author of A New Kind of Science. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Jane Street Capital is a quantitative trading firm known for its emphasis on technology and specifically functional programming. OCaml is the main programming language used at Jane Street, chosen for its performance, correctness and dynamism. Yaron Minsky is the Head of Technology at Jane Street and was responsible for introducing OCaml to the company and transitioning its architecture. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Caml is a functional programming language that is a dialect of the ML programming language family, developed at INRIA and formerly at ENS. Gérard Huet is a senior researcher at INRIA, the French Institute for Research and Automation. He helped develop the Caml programming language in the 80s, and has a variety of other accomplishments in the world of computer science, including developing the Coq Proof Assistant System. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Haskell is a purely functional programming language that employs lazy evaluation. Lennart Augustsson is a computer scientist who is heavily involved with Haskell, and created the first publicly available Haskell compiler. He is also the author of the Cayenne programming language. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Racket is a functional programming language similar to Lisp and Scheme. Matthew Flatt is a member of the Racket core development team and also a professor of computer science at the University of Utah. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Elm is a functional programming language for web browsers. Richard Feldman and Srinivas Rao are front-end developers who work with Elm at NoRedInk. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Erlang is a functional, concurrent programming language that was originally designed within Ericsson in the 1980’s. It was built to support distributed, fault-tolerant, non-stop applications suitable for telecommunications infrastructure. Joe Armstrong is one of the designers of Erlang, and the chief architect of the Open Telecom Platform (OTP), a framework for building Erlang applications. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Treehouse is an online school that teaches software development, design and strategy. Ryan Carson is the CEO and Founder of Treehouse. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Galvanize Data Science is a 12-week data science program born out of Zipfian Academy, which was acquired by Galvanize in 2014. Jonathan Dinu and Ryan Orban were the founders of Zipfian, and are now continuing their work to teach data science at Galvanize. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Free Code Camp is an open source community dedicated to teaching people how to code while also helping non-profits Quincy Larson is the Founder of Free Code Camp, and previously was the director of several schools in the U.S. and China. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Steve Kinney is the co-director of academics at Turing School of Software and Design. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Haseeb Qureshi is an former high-stakes poker player who pivoted to web development. He is an instructor at App Academy in San Francisco. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Dwarf Fortress is a construction and management simulation computer game set in a procedurally generated fantasy world in which the player indirectly controls a group of dwarves, and attempts to construct a successful underground fortress. Tarn Adams works on Dwarf Fortress with his brother Zach. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kudu is an open-source storage engine for the Hadoop ecosystem. It balances the advantages of both HDFS and HBase, by allowing for performant random-access queries while also providing fast writes and scans for analytics. Todd Lipcon is a software engineer at Cloudera who leads the development of Kudu. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Our conversation covered the world of software podcasting, including a discussion of one of our favorite software podcasts, JavaScript Jabber, a weekly podcast about all things JavaScript, including Node.js, front-end technologies, careers, and teams. Charles Wood is the host of JavaScript Jabber, Ruby Rogues and several other software podcasts. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ethereum is a decentralized platform for applications that are guaranteed to run exactly as expected via smart contracts. Gavin Wood is a Co-founder and the Director of Development at Ethereum. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Léonie Watson is an accessibility engineer and a director at the British Computer Association of the Blind. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Derek Sivers is a programmer, musician, and writer. He has created several companies and products, including CD Baby, which became the largest seller of independent music online. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Genie is an open-source tool that provides job and resource management for the Hadoop ecosystem in the cloud. Tom Gianos is a Senior Software Engineer at Netflix focusing on its big data platform. He is one of the core contributors in charge of maintaining and improving Genie. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Pachyderm is a containerized data analytics platform that seeks to replace Hadoop. Joe Doliner is the Co-founder and CEO of Pachyderm. Previously he worked as a software engineer at RethinkDB and AirBnB. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

DemocracyOS is an open-source application for online decision making. The organization’s goal is to enable more responsive and transparent communication with government using modern technology. Santiago Siri is the president of DemocracyOS. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

ReadMe is simplifying the process of writing documentation. The platform provides a readymade developer hub with the ability to integrate API endpoints into documentation. Greg Koberger is a developer and designer, and the founder of ReadMe. He previously founded a traveling incubator and worked for Mozilla. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

It provides automated code grading, plagiarism checking, code quality analysis and additional course management tools. Prah Veluvolu is the CEO and Co-founder of Mimir. He studied at Purdue University and participated in the Boiler Startup Accelerator before graduating from the Y Combinator Summer 2015 class. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Fullstack Academy is a 13-week immersive program for students to learn software engineering. Their flagship course focuses primarily on Javascript and the associated frameworks used to build real-world web applications. David Yang and Nimit Maru are the Co-founders of Fullstack Academy. They were part of the Y Combinator summer 2012 class. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Yhat is an enterprise software company tackling the challenge of how data science gets done. Their products enable companies and users to easily deploy data science environments and translate analytical models into production code. Greg Lamp is the Co-founder and CTO of Yhat and previously worked as a product manager in financial services. Yhat was part of the Y Combinator winter 2015 class. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kaggle is a platform for data scientists to collaborate and compete on machine learning problems with the opportunity to win money from the competitions’ sponsors. Ben Hamner is the co-founder and CTO of Kaggle. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Many companies find themselves drowning in data. The quantity of data matters far less than the right questions in the pursuit of actionable insights. Edwin Chen is a data scientist and blogger. He previously led data science teams at Dropbox and Google. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Yad Faeq returns to the podcast to discuss data science at a high level, and rescue Software Engineering Daily from the threat of the hype vortex. Yad is a software engineer, currently working on machine learning applications. Yad first appeared on Software Engineering Daily on Episode 0, the inaugural show. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Dataquest is an in-browser platform for learning data science that is tackling this problem. Vik Paruchuri is the founder of Dataquest. He was previously a machine learning engineer at EdX and before that a U.S. diplomat. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Successfully solving a problem often demands a strong model for communication and well-trained robots. Stefanie Tellex is an assistant professor at Brown University whose research focuses on constructing robots that use natural language to communicate with humans. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Crocodile Browser with Anesi and Osine Ikhianosime Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

ChickTech: High School builds girls’ confidence, teaches them vital skills, and introduces them to a wide range of technological interests, from robotics to game design to programming. Janice Levenhagen is the Founder and Executive Director of ChickTech. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Pivotal makes software platforms and database products to enable enterprises to make use of their data. Sarah Aerni is principal data scientist at Pivotal. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Well-known tech companies of Silicon Valley sponsor the next generation of female software engineers emerging from these dinners. Sukrutha Bhadouria is a software engineer at Salesforce and the Managing Director of Bay Area Girl Geek Dinners. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Former Andreesen-Horowitz partner Preethi Kasireddy saw excited software engineers coming through the doors every day, pitching their ideas. This was her impetus to take the leap and make the career switch to software engineering. She is attending Hack Reactor, a 12-week coding boot camp. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Businesses can make changes to apps using Apptimize’s powerful and friendly editor, and free up developers’ time for more impactful work. Nancy Hua is the Founder and CEO of Apptimize. She worked in high-frequency trading prior to starting a company. This is the first show of Women in Tech Week, where we will discuss the inequalities women in tech face, and see what solutions may exist for this problem. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Mobile devices use a virtual machine to interpret the JavaScript into native code. Tadeu Zagallo is a Facebook engineer who works on React Native for iOS. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Fluxxor is a set of tools for building JavaScript data layers using the Flux architecture. It was one of the earliest open-source implementations of Flux. Brandon Tilley is a software engineer who works at The Minerva Project. He is the creator of Fluxxor. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

These technologies complement the paradigm shift of React and other Facebook technologies. Nick Schrock is the co-creator of GraphQL and an engineer on product infrastructure at Facebook. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Redux is a predictable state container for JavaScript to use with React or any other view library. Dan Abramov currently works full time on Redux, React Hot Loader, and React Transform. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Facebook engineering has recently produced a powerful suite of loosely coupled tools for development: React, GraphQL, Relay, React Native, and Flux Architecture. Ben Alpert is an engineer on the React Core team at Facebook. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

It offers users a way to build full-fledged web apps from their React components. Michael Jackson works on React Router and is the founder of ReactJS Training. He will be speaking at the upcoming QCon San Francisco. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Security researchers and organizations have to stay vigilant in this cat-and-mouse game. Shuman Ghosemajumder is the VP of Product at Shape Security, which defends applications from malware and bots. He is the former click fraud czar at Google, and he will be speaking at QCon San Francisco. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Christopher Kelly is a computer scientist and a pro mountain biker who now works full-time as a health hacker. He is a co-founder of Nourish Balance Thrive, where he works alongside two medical doctors, a food scientist and a registered nurse to help people feel and perform better using biomedical testing together with diet and lifestyle hacks. At QCon San Francisco, he will be giving a talk called Debug Me. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

But life can be made easier with tactics such as the actor pattern and the use of conflict-free replicated data types (CRDTs). Caitie McCaffrey is a distributed systems engineer who currently works at Twitter. She previously worked on Halo 4 at Microsoft and 343 Industries. At QCon San Francisco, she will be hosting the track Taming Distributed Architecture. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Scala, Clojure, and Akka are functional tools built on the Java Virtual Machine. Jessica Kerr is a functional developer on the JVM. She currently works at Monsanto. At QCon San Francisco, she will be giving a talk called Contracts in Clojure: Settling Types vs. Tests. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Phil Haack is a former PM on Microsoft’s ASP.NET MVC Framework, as well as NuGet, an open-source package manager. He currently works on Github Desktop at Github. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Adrián Lamo is a threat analyst, hacker, and writer. In the early 2000’s, Adrián was a hobbyist white-hat hacker, breaking into companies to expose vulnerabilities and fix them. In 2010, Adrian informed the US Army that Chelsea Manning had provided more than 260,000 documents to Wikileaks. This interview does not discuss the Manning case, because we covered that topic in our Quoracast interview several months ago. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Max Krohn is the co-founder of Keybase, and previously co-founded OKCupid and SparkNotes. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Bruce Schneier is a security researcher and author of Data and Goliath. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Reverse engineers have begun to dissect car security. Craig Smith is the author of The Car Hacker’s Handbook and the founder of Theia Labs, a research and consulting firm. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

David Schwartz is Chief Cryptographer for Ripple Labs. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

The hosts of DevOps Cafe joined Software Engineering Daily for a conversation about DevOps culture and misconceptions. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Carmen DeArdo is technology director at Nationwide Insurance. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run. This allows for quick flexibility and scaleability in business applications, a key driver of the DevOps methodology. James Turnbull is VP of engineering at Kickstarter, an advisor at Docker and author of The Docker Book. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Kohsuke Kawaguchi is the primary developer of Jenkins CI and the CTO of CloudBees, a provider of enterprise Jenkins. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Containers are a central component of the DevOps movement. Joyent provides simple, secure deployment of containers with bare metal speed on container-native infrastructure Bryan Cantrill is the CTO of Joyent, the father of DTrace and an OS kernel developer for 20 years. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

IPFS enables the creation of completely distributed applications. Juan Benet is the creator of IPFS and the founder of Protocol Labs. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

VoltDB provides streaming analytics with transactions. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Technical and financial criticisms facing Ethereum are as real as those faced by bitcoin. This episode is a departure from Databases Week. Stephan Tual is CCO of Ethereum and founder of Ursium, a blockchain consulting company. Thanks to Reddit and Quora for questions. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ryan Boyd is a developer advocate for Neo4j, an open-source graph database. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Time-series data can be used by for metrics and analytics. Paul Dix is the CEO of InfluxDB. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Derek Nelson is the CEO of PipelineDB. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

RethinkDB pushes changes to the application rather than waiting for a request. Slava Akhmechet is the CEO of RethinkDB. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

MemSQL with Nikita Shamgunov Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Rusty Russell is an Australian free software programmer and advocate, known for his work on the Linux kernel’s networking subsytem and the Filesystem Hierarchy Standard. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Aaron Davis works on MetaMask, which brings Ethereum apps to the web browser. In this monologue episode, Aaron dives deep into the Ethereum internals, philosophy, and developer experience. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Christian Ander is CEO of Goobit AB. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Andreas Antonopoulos is a bitcoin researcher, journalist, and evangelist. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Ripple with Greg Kidd Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Melanie Swan is a science and technology innovator and philosopher at the MS Futures Group. She founded the Institute for Blockchain Studies, and is the author of Blockchain: Blueprint for a New Economy. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Faisal Khan is a banking, payments, and fintech consultant with a background in electrical engineering. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Michael Miller is the author of “The Ultimate Guide To Bitcoin” and 150 other books. Michael is an expert in explaining complex topics in ways which are comprehensible to the layperson. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Christopher Berner works on Presto at Facebook. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Venkatesh Seetharam is a software engineer at Hortonworks. He has worked on several Apache projects, including Hadoop, Falcon, and Atlas. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Flavio Junqueira is a committer and PMC of Apache ZooKeeper, and former VP of ZooKeeper. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Guozhang Wang is an engineer at Confluent, which offers a stream data platform built using Kafka. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Eric Sammer is the co-founder and CTO of Rocana. At Cloudera, he served as an Engineering Manager responsible for tools and partner integrations. Within that role, he developed many of Cloudera’s best practices for developing large, distributed, data processing infrastructure. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Matei Zaharia created Spark, and is the co-founder of Databricks, a company using Spark to power data science. Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Cloudera allows enterprises to leverage their data through its Hadoop platform. Eli Collins is the Chief Technologist at Cloudera. Topics include: Links: Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Yunong Xiao is a senior engineer at Netflix. He works on the website’s Node.js front-end and middle-tier services. Topics discussed: Links: Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Questions include: Links: Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

restify is a node.js module built to enable correct REST web services. Netflix uses restify to gain performance and visibility. Yunong Xiao, senior engineer at Netflix, talks about the incident which incited Netflix’s service migration to restify. The discussion starts with an explanation of flame graphs and evolves into a conversation about restify, and the importance of choosing the correct RESTful API. Links: Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

MongoDB is a cross-platform document-oriented database. Bryan Reinero is a developer advocate at MongoDB. Questions include: Links: Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

TypeScript is a typed superset of JavaScript that compiles down to regular JavaScript. Jon Turner maintains the TypeScript package at Microsoft. Links: Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

Meteor is a full-stack web framework for building isomorphic JavaScript applications. Questions include: Links: Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

React.js is a JavaScript library for building user interfaces. Facebook and Instagram maintain the open-source repository and use the technology to build front-end components. Questions include: Learn more about your ad choices. Visit megaphone.fm/adchoices

View Details

In Episode 0 of SE Daily, Yad and Jeff give a prologue to many of the topics that will be covered in JavaScript Week. Right-click to download the episode. Links: Learn more about your ad choices. Visit megaphone.fm/adchoices

Software Daily: Recent Episodes

SoftwareDaily.com