All Content – Software Engineering Daily: Recent Episodes

All Content – Software Engineering Daily

PCSX2 is an open-source PlayStation 2 emulator that allows users to play PS2 games on modern hardware. The emulator is remarkable for simulating the complex architecture of the PS2, which includes the Emotion Engine CPU, Graphics Synthesizer, and specialized subsystems.

The emulator just hit a major milestone with the release of PCSX2 version 2.0. The release brings many changes including a Qt-based interface, big picture mode, auto-selection of graphics APIs, and native support for MacOS.

TellowKrinkle is a developer for PCSX2 who ported the emulator to MacOS, among other contributions. In addition to his work on PS2 emulation he has also worked on Dolphin, which emulates the Nintendo GameCube and Wii. Tellow joins the podcast with Joe Nash to talk about how he got started in emulation, the PS2 architecture, the challenges of rendering PS2 games on modern GPUs, and more.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post PlayStation 2 Emulation with TellowKrinkle appeared first on Software Engineering Daily.

Taiwan Semiconductor Manufacturing Company, or TSMC, produces a significant portion of the global supply of advanced semiconductors. Its cutting-edge technology powers everything from smartphones to high-performance computing, and its customers include Apple, NVIDIA, and ARM.

TSMC’s dominance in chip production has made Taiwan a critical player in the global tech supply chain, drawing attention from major economies like the U.S. and China. This has escalated geopolitical tensions, with concerns over the stability of Taiwan and the potential risks to global tech industries if chip production were disrupted due to regional conflicts.

Tim Culpan is an independent technology journalist and author of the forthcoming book “The World’s Smallest Superpower — inside the rise of TSMC, Foxconn and a nation of Taiwan technology titans.”

He’s based in Taipei and has been covering the semiconductor and electronics hardware industry for 25 years, including 18 years as a journalist and Columnist at Bloomberg. Most recently his work can be found at timculpan.substack.com.

Tim has written extensively about TSMC, and recently broke news on the developments at TSMC’s Arizona factory. He joins us today to discuss what’s happening at TSMC and what that means for the US, Taiwan and China’s chip industries.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post TSMC and the Geopolitics of the Chip Industry with Tim Culpan appeared first on Software Engineering Daily.

Factorio is a construction and management simulation game focused on resource-gathering with real-time strategy and survival elements. The player survives by locating and harvesting resources to craft various tools and machines, which in turn create more advanced materials that allow for the progression to more sophisticated technologies.

The game was released in 2020 and has been hailed as a manufacturing masterpiece. Factorio’s Space Age expansion just released so we took the opportunity to speak with Michal Kovařík, also known as kovarex, who is the Founder and Director of Wube Software which developed Factorio. Michal joins the show with Joe Nash to talk about the origins of the game, the new expansion, and everything in between.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Factorio with Michal Kovařík appeared first on Software Engineering Daily.

Slack is a team communication platform that originated as an internal tool within Tiny Speck, a game development company. When the company realized that their game would not achieve commercial success, they changed direction and repurposed the communication tool into a new product which eventually became Slack. Slack was launched in 2013 and is now ubiquitous in workplaces around the world.

Shruti Kapoor is a Lead Member of the Technical Staff at Slack. She’s worked on features including Huddles, the recent redesign of Slack, and currently works on accessibility. She joins the podcast to talk about her path into frontend engineering, the frontend tech stack at Slack, the developer tooling, how Slack evaluates new technologies, and more.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Frontend Engineering at Slack with Shruti Kapoor appeared first on Software Engineering Daily.

Boston Dynamics is a robotics company known for creating advanced robots with highly dynamic movement and agility, designed to navigate complex environments. Their robots, such as the quadruped Spot and the humanoid Atlas, have applications in industries ranging from logistics to public safety. They also garner widespread attention with their impressive videos showcasing robots performing complex tasks with precision.

Matthew Malchano is Boston Dynamics‘ Vice President of Software. For more than 20 years, Matt has been a technical contributor and leader on robotics projects such as Spot, BigDog, LS3, and SandFlea. He has led efforts in areas including software, product, and robotics autonomy, perception, and control. Matt joins the podcast with Sean Falconer to talk about his wide-ranging work at Boston Dynamics.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Boston Dynamics with Matt Malchano appeared first on Software Engineering Daily.

TypeScript is a superset of JavaScript that adds static typing with optional type annotations. It was created at Microsoft and first released in 2012.

TypeScript ESLint enables ESLint and Prettier to run on TypeScript code.

Josh Goldberg is a host for Software Engineering Daily, the author of Learning TypeScript by O’Reilly, and a Microsoft MVP. He’s also a maintainer of TypeScript ESLint. In this episode, Josh speaks with Paweł Borkowski about the history of TypeScript, why it was created, some of its current limitations, the TypeScript ESLint project, and more.

Paweł is the founder at flat.social the world’s first ‘flatverse’ start-up and glot.space, an AI-powered language learning app. Pawel’s background is as a full-stack software engineer with a lean and experimental approach towards product development. With a strong grounding in computing science, he spent the last decade getting early-stage products off the ground – both in startup and corporate settings. Follow Paweł on Twitter, LinkedIn and his personal website – pawel.io.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post TypeScript ESLint with Josh Goldberg appeared first on Software Engineering Daily.

Modern engineering teams often face challenges with unpredictable delivery and limited visibility into their performance. This can make it difficult to track progress, identify bottlenecks, and understand how efficiently time and resources are being used. The lack of clear insights commonly prevents teams from aligning their work with broader business goals.

Sleuth is designed to be an operating system for engineering, and help teams achieve more predictable delivery and align with business needs.

Dylan Etkin is the Founder and CEO of Sleuth. Dylan is an Atlassian alum who has spent the last 15 years building dev tools with Jira, Bitbucket, and Statuspage. He joins the podcast to talk about the challenges faced by modern engineering teams and innovative strategies to overcome them.

Full Disclosure: This episode is sponsored by Sleuth.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Sleuth and the Future of Engineering Teams with Dylan Etkin appeared first on Software Engineering Daily.

Python 3.13 was just released and brings fundamental changes to the language including a new interactive interpreter, experimental support for running in a free-threaded mode, and a Just-In-Time compiler, or JIT. There are also updates to the Python type system, module removals, and docstring improvements among many other changes.

Łukasz Langa is the CPython Developer in Residence at the Python Software Foundation. He joins the show with Sean Falconer to talk about maintaining Python, the evolution of the language, Python optimization, the most impactful changes in Python 3.13, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post The Big Changes in Python 3.13 with Łukasz Langa appeared first on Software Engineering Daily.

Argo is an open-source suite of tools to enhance continuous delivery and workflow orchestration in Kubernetes environments. The project had its start at Applatix and was accepted to the Cloud Native Computing Foundation in 2020.

Michael Crenshaw and Zach Aller are both lead maintainers for Argo. They join the show with Lee Atchison to talk about the origins of the project, what problems Argo solves, the four core tools in Argo, and more.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Argo and Kubernetes with Michael Crenshaw and Zach Aller appeared first on Software Engineering Daily.

In software engineering, C++ is often used in areas where low-level system access and high-performance are critical, such as operating systems, game engines, and embedded systems. Its long-standing presence and compatibility with legacy code make it a go-to language for maintaining and extending older projects. Rust, while newer, is gaining traction in roles that demand safety and concurrency, particularly in systems programming.

We wanted to explore these two languages side-by-side, so we invited Herb Sutter and Steve Klabnik to join host Kevin Ball on the show. Herb works at Microsoft and chairs the ISO C++ standards committee. Steve works at Oxide Computer Company, is an alumnus of the Rust Core Team, and is the primary author of The Rust Programming Language book.

We hope you enjoy this deep dive into Rust and C++ on Software Engineering Daily.

Kevin Ball or KBall, is the vice president of engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript meetup, and organizes the AI inaction discussion group through Latent Space.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Rust and C++ with Steve Klabnik and Herb Sutter appeared first on Software Engineering Daily.

CRISPR is a powerful tool in biotechnology that allows scientists to precisely edit genes, much like editing lines of code in a computer program. Just as developers can remove or alter specific parts of a code to fix bugs or enhance functionality, CRISPR enables researchers to modify DNA to correct genetic disorders, improve crops, or develop new treatments. The development of CRISPR-based editing was recognized by the Nobel Prize in Chemistry in 2020 awarded to Emmanuelle Charpentier and Jennifer Doudna.

Profluent Bio is an AI-first protein design company that recently developed OpenCRISPR-1, which is an AI-generated, CRISPR-like protein that does not occur in nature. Importantly, the company also released the protein and nucleic acid sequences for OpenCRISPR-1.

Aadyot Bhatnagar is an ML Scientist at Profluent Bio and previously worked at Salesforce. He joins the podcast with Sean Falconer to talk about OpenCRISPR-1 and how it was made.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Engineering an Open Source CRISPR with Aadyot Bhatnagar appeared first on Software Engineering Daily.

SoundCloud is an online platform and music streaming service where users can upload, promote, and share their music or audio creations. It was founded in 2007 and is known for its community-driven approach, allowing artists to interact directly with their fans and receive real-time feedback on their work.

Matthew Drooker is the Chief Technology Officer at SoundCloud. He previously worked at Turner and has deep experience as a technologist and leader in the media industry. Matthew joins the show with Jordi Mon Companys to talk about his background, the evolution of the SoundCloud platform, its current tech stack, and much more.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post SoundCloud’s Platform Architecture with Matthew Drooker appeared first on Software Engineering Daily.

GraphQL is an open-source query language for APIs and a runtime for executing those queries. It was developed by Facebook to address the problem of over-fetching or under-fetching data, which is a common issue with traditional REST APIs.

Matt Bessey is a Principal Engineer and Software Architect. Earlier this year Matt wrote a blog post titled “Why, after 6 years, I’m over GraphQL”. The post put words to many users’ frustrations with the technology, and it went viral on Hacker News.

Matt joins the show today to talk about GraphQL, the problems it solves, its security vulnerabilities, and why it might not be a good fit for backend engineering today.

You can find a link to Matt’s blog posts here.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post The End of GraphQL with Matt Bessey appeared first on Software Engineering Daily.

Asahi Linux is a project that aims to port Linux to Apple Silicon chips, which use a custom ARM-based architecture.

The project is fundamentally important given the popularity of Apple Silicon Macs, and it’s also a heroic effort because Apple Silicon is an entirely undocumented platform.

Alyssa Rosenzweig is a well-known computer scientist who describes herself as a graphics developer passionate about software freedom. She is currently a contractor at Valve where she develops open source software to improve Linux gaming. Alyssa is also a contributor to Asahi Linux and works on reverse-engineering the Apple M1 GPU, among other contributions to the project. Alyssa joins the podcast to talk about reverse engineering hardware, Asahi Linux, new advances in gaming on Asahi, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Linux on Apple Silicon with Alyssa Rosenzweig appeared first on Software Engineering Daily.

DOOM is a pioneering first-person shooter that needs no introduction.

The game was released in 1993 for DOS and was an instant success. This led to ports of the game to other major platforms including Windows, PlayStation, and Sega Saturn.

One of the most remarkable ports was to the Super Nintendo, with development being led by legendary engineer Randy Linden. In addition to his work on the SNES port of DOOM, Randy developed PlayStation and Dreamcast emulators, and worked at Microsoft on the Xbox 360 and Kinect.

Limited Run Games and Bethesda recently announced a new version of DOOM for SNES, that Randy also worked on. It has performance improvements, new features, and uses a new version of the SuperFX chip that can handle full-motion video.

Randy joins the show today to talk about his career, re-implementing video games, the new SNES DOOM port, and more.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post DOOM on Super Nintendo with Randy Linden appeared first on Software Engineering Daily.

Authentication is a key requirement for any B2B software application, especially if software vendors are selling to enterprise clients who are likely to have strict authentication requirements for the vendors they use. However, building authentication for a B2B application is typically complex and resource-intensive due to the data models required, the provisioning and managing accounts, and additional security and scale concerns.

Julianna Lamb is the Co-Founder and CTO of Stytch which is building an all-in-one platform for identity and access management. She joins the podcast with Gregor Vand to talk about the platform.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Identity and Access Management with Julianna Lamb appeared first on Software Engineering Daily.

J. P. Morgan Payments is one of the leaders in payments processing with a staggering $10 trillion in payments handled daily. The company recently released its Payments Developer Portal, or PDP, which serves as a gateway for developers to build and test payment APIs, and accept, manage, and send payments on their own platforms. Developing financial APIs at a global scale presents unique engineering challenges, in large part because there is no margin for error.

Jack Gibson is a Managing Director and Head of Payments Engineering, Architecture and APIs at J. P. Morgan Payments. He joins the podcast with Sean Falconer to explore the design and engineering behind the company’s payments API offering.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Building Secure Payments Infrastructure with Jack Gibson appeared first on Software Engineering Daily.

MongoDB Atlas is a managed NoSQL database that uses JSON-like documents with optional schemas. The platform recently released new vector search capabilities to facilitate building AI capabilities.

Ben Flast is the Director of Product Management at MongoDB. He joins the show to talk about the company’s developments with vector search.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post MongoDB Vector Search with Ben Flast appeared first on Software Engineering Daily.

Darío and Wiseguy are Nintendo 64 hackers and modders. They are the creators of multiple projects, including RT64 which is an N64 graphics renderer for enhancing games in emulators and native ports.

This year, they released N64: Recompiled, which is a tool to statically recompile N64 binaries into C code that can be compiled for any platform. The advance promises to usher in a revolution in N64 native ports for PC. To demonstrate the tool’s capability, it was used to produce a working copy of The Legend of Zelda: Majora’s Mask on PC.

Darío and Wiseguy join the podcast to talk about their journeys into the N64 hacking scene, the N64: Recompiled project, obscure N64 game bugs, and more.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post N64 Recompiled with Darío and Wiseguy appeared first on Software Engineering Daily.

The Stack Overflow Developer Survey is an annual survey conducted by Stack Overflow that gathers comprehensive insights from developers around the world. It offers a valuable snapshot of the global developer community, covering a wide range of topics such as preferred programming languages, tools, and technologies.

Erin Yepis is a Senior Analyst and Ryan Polk is the Chief Product Officer at Stack Overflow. They join the show with Sean Falconer to talk about the results of the 2024 Developer Survey, which just released this summer.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post The 2024 Stack Overflow Developer Survey with Erin Yepis and Ryan Polk appeared first on Software Engineering Daily.

Software supply chain attacks exploit interdependencies within software ecosystems. Security in the supply chain is a growing issue, and is particularly important for companies that rely on large numbers of open source dependencies.

Chainguard was founded in 2021 and offers tools and secure container images to improve the security of the software supply chain.

Matt Moore is the Founder and CTO of Chainguard. He started his career in compiler optimization at Microsoft and worked at Google before starting Chainguard. He joins the show with Gregor Vand to talk about container security.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Container Security with Matt Moore appeared first on Software Engineering Daily.

The Google Maps Platform JavaScript API is a fundamental web technology that’s used to build dynamic and interactive map features in web apps.

Matt Toon is a Solutions Engineering Manager for the Google Maps Platform. He joins the podcast with Josh Goldberg to talk about his background working with geospatial data, the development of Google Maps Platform, bringing 3D Maps to the JavaScript API, and much more.Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Google Maps Javascript API with Matt Toon appeared first on Software Engineering Daily.

LLMs are becoming more mature and accessible, and many teams are now integrating them into common business practices such as technical support bots, online real-time help, and other knowledge-base-related tasks. However, the high cost of maintaining AI teams and operating AI pipelines is becoming apparent.

Maxime Armstrong and Yuhan Luo are Software Engineers at Dagster, which is an open source platform for orchestrating data and AI pipelines. They join the show with Sean Falconer to talk about running cost-effective AI pipelines.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post AI Pipelines with Maxime Armstrong and Yuhan Luo appeared first on Software Engineering Daily.

Data is at the center of many business decisions and advances today, including AI-driven capabilities. This requires companies to have well-governed data that is easy for users to find, use and understand. In moving to the cloud, Capital One modernized its data ecosystem and adopted a “You Build, Your Data” model to equip its data stakeholders with self-service capabilities to use and build data applications.

Jim Lebonitte is a Senior Distinguished Engineer at Capital One leading technical architecture and strategy for enterprise data platforms. He has over 15 years of experience building platforms focused on data and software delivery experiences. Jim joins the podcast to talk about how to empower data users at scale while keeping data well-governed, building data pipelines and applications, and much more.

Full Disclosure: This episode is sponsored by Capital One**.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Data Management at Capital One with Jim Lebonitte appeared first on Software Engineering Daily.

ESLint is a static code analysis tool for identifying and fixing problems in JavaScript code. It helps developers maintain code quality and consistency by enforcing coding standards and detecting issues such as syntax errors, stylistic problems, and bugs.

Nicholas Zakas is the creator of ESLint. He joins the show to talk about working at Yahoo in the early days of the web, learning from JSLint, creating ESLint, separation of parsing and rules, and more.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Creating ESLint with Nicholas Zakas appeared first on Software Engineering Daily.

The Humane pin is a multi-modal wearable device designed by Humane Inc., a startup co-founded by former Apple employees Imran Chaudhri and Bethany Bongiorno. This wearable is part of a broader vision to create more seamless and integrated interactions between humans and technology, moving away from traditional screens.

George Kedenburg III is a Software Designer at Humane, and Josh Dickens is a Software and Product Designer at Humane. They join the podcast with Sean Falconer to talk about Humane and the technology the company is developing.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Humane with George Kedenburg III and Josh Dickens appeared first on Software Engineering Daily.

In the modern enterprise, high-performance applications require a solid foundation that can handle complex, high-volume, real-time data. VMware Tanzu GemFire is an in-memory data grid (IMDG) designed to deliver the speed, low latency, and scalability needed for such applications. Though it has been in use for decades, GemFire is experiencing a resurgence, as it was purpose-built for the challenges of real-time data processing and management.

A Wall Street HeritageGemFire’s roots trace back to the early 2000s, when Wall Street traders needed software capable of propagating real-time trading floor data—where every millisecond counts. “Wall Street traders were looking to build software that could propagate real-time trading floor data where every millisecond counts,” explains Ivan Novick, Product Manager for Data Products at the Tanzu Division of Broadcom.

Developing such intricate solutions can be prohibitively complex for independent software vendors. GemFire emerged as a platform for application development rather than a traditional turnkey business application, addressing limitations found in legacy systems. In many ways, it was ahead of its time—built to tackle the challenges of real-time information delivery across the enterprise.

Moving Beyond Two-Tier ArchitecturesHistorically, enterprises relied on two-tier database architectures to manage data. However, as workload demands increased, these systems struggled to meet real-time, low-latency requirements. The need to synchronize data across multiple databases and applications created significant bottlenecks. As information volumes grew beyond the capacity of single databases, synchronization issues became more pronounced.

GemFire, however, was designed with these challenges in mind. It excels in handling unstructured data like text, images, and video. Through vector embedding, GemFire transforms this data into multi-dimensional mathematical graphs, grouping similar items together for efficient access and analysis.

Key Features of GemFireGemFire offers a range of powerful features that make it ideal for real-time applications:

Sub-millisecond response times
In-memory data and compute grid
Single source of truth across distributed systems
Ultra-high speed
Scalability on demand

One common issue with systems that manage complex data across multiple nodes is the risk of inconsistencies when multiple writers and readers are involved. GemFire mitigates this with active-active, multi-site Wide Area Network (WAN) replication. Whether a business operates in New York, London, or Tokyo, GemFire ensures data is synchronized and up-to-date across all locations. “On our grid, data is synchronized and always up-to-date; the system updates information millions of times a second,” says Novick.

Where Should Tanzu GemFire be Deployed?GemFire is particularly well-suited for organizations with demanding IT workloads—those for whom off-the-shelf software is insufficient. These companies need custom-built solutions capable of processing vast amounts of data in real-time.

For example, consider a large airline managing millions of passengers at an airport like LAX. They need to track ticket sales, check-ins, seating arrangements, loyalty programs, flight schedules, and aircraft locations. Each component interacts with multiple websites, applications, and devices. GemFire’s architecture enables them to manage and synchronize all these moving parts seamlessly.

Financial Services and Government ApplicationsFinancial services and government sectors are experiencing significant growth in the adoption of GemFire. These industries require real-time, event-driven systems for complex calculations and rapid decision-making.

A large bank may process billions of dollars in transactions every day. They need to assess the movement of funds in real-time, detect potential fraud within seconds, and run statistical analyses on historical data to make informed decisions. GemFire’s real-time capabilities allow financial institutions to manage these processes effectively.

Expanding to Industrial and Edge ComputingAs edge computing gains traction, GemFire is finding a home in industrial applications. Edge computing reduces the load on central servers by processing data closer to the source, lowering WAN infrastructure overhead and costs.

Companies are deploying small clusters of two to four nodes at the edge to perform custom processing for industrial tasks. These clusters require high availability, parallelism, and replication, all of which GemFire provides. As deployments scale, GemFire ensures that transactions are accurately synchronized across hundreds of systems.

A Platform Poised for GrowthGemFire’s evolution from a custom-built IMDG tool for financial services into a sophisticated real-time platform has positioned it as a leader in modern application development. With its vector database and real-time processing capabilities, GemFire continues to attract interest across industries and is ideal for handling emerging AI needs. As organizations look to build more complex, real-time solutions—especially at the edge—GemFire’s role will only continue to grow.

The post VMware Tanzu GemFire and Next-Generation Real-Time Application Development appeared first on Software Engineering Daily.

Online transaction processing, or OLTP, is designed for managing high volumes of short, fast, and concurrent transactions, such as data entry and retrieval operations. OLTP systems solve the problem of efficiently handling numerous simultaneous transactions, making them essential for sectors like banking and retail.

Joran Greef is the Founder and CEO of TigerBeetle, which is developing an open-source financial transactions database focused on mission critical safety and performance. He joins the podcast to talk about the TigerBeetle’s technology and problems it solves.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Building a Fast Financial Transactions Database with Joran Greef appeared first on Software Engineering Daily.

Mozilla Firefox is an open-source web browser developed by the Mozilla Foundation. Since its first major release in 2004, it has stood out on the browser landscape for its emphasis on privacy, security, and customization.

Brian Grinstead is a Senior Principal Engineer at Mozilla. He joins the podcast with Kevin Ball to talk about the Firefox architecture, the recent UI refactor, web performance, the role of Rust in the browser, and more.

Kevin Ball or KBall, is the vice president of engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript meetup, and organizes the AI inaction discussion group through Latent Space.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Firefox Software Architecture with Brian Grinstead appeared first on Software Engineering Daily.

Google needs no introduction, and is renowned for its data and analytics capabilities.

Gerrit Kazmaier is the VP and GM for Database, Data Analytics and Looker at Google. He has a long history in the space, and in this episode he speaks with Sean Falconer about data and analytics in the AI era.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post AI Data Analytics at Google with Gerrit Kazmaier appeared first on Software Engineering Daily.

DevOps is a powerful model for managing the building and operational aspects of modern applications. Most developers are now familiar with DevOps, and the adoption of DevOps practices is widespread and growing.

Adam Jacob was the original author of Chef, a popular early DevOps tool. He’s now the CEO of System Initiative, which develops an open-source collaborative tool designed to remove the many pain points from DevOps work.

Adam joins the show to talk about the history of DevOps, current strategies in DevOps, System Initiative’s collaborative platform, and more.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Building a Collaborative DevOps Platform with Adam Jacob appeared first on Software Engineering Daily.

React is an open-source front-end JavaScript library maintained by Meta. It was first released in 2013 and is now the most popular web framework.

Ilya Gurevich is a Senior Software Engineer at The New York Times. Last winter, his team set out to implement React 18 for The Times’ flagship core news site. Ilya recently wrote a viral blog post about this decision, and how the team tackled some of the engineering challenges along the way.

Today he joins the podcast with Gregor Vand to talk about adopting React 18 to enhance the performance of the The New York Times website.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post React 18 at The New York Times with Ilya Gurevich appeared first on Software Engineering Daily.

Feature flagging tools have grown in popularity as a way to decouple releases and deployment, but they can introduce their own long-term problems and tech debt.

Lekko is a startup democratizing the practice of dynamic configuration. Their motivating idea is to empower engineers to focus on software releases, and business teams and other stakeholders to shape deployment.

Konrad Niemiec is the Founder and CEO at Lekko. He previously worked at Uber where an internal tool called Flipr enabled dynamic configuration management, and which today serves as a key design inspiration for Lekko. Konrad joins the show with Sean Falconer to talk about his company and the technology they’re developing.

Full Disclosure: This episode is sponsored by 10K Media**.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Lekko and Dynamic Configuration with Konrad Niemiec appeared first on Software Engineering Daily.

Harold Halibut is a 2024 narrative adventure video game developed by German developer Slow Bros.

The game has a distinct look owing to its use of stop motion animation with 3D scans of physical sets and puppets.

Onat Hekimoglu worked on Harold Halibut as the Director, Game Designer, Composer, and Person of Many Hats. He joins the podcast with Joe Nash to share the story and technical details of how he and his team developed their unique game.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Crafting a Stop Motion Video Game with Onat Hekimoglu appeared first on Software Engineering Daily.

Stack Overflow is a legendary question-and-answer site for programmers, and is likely well known to most SEDaily listeners.

Svelte is an open-source front-end framework that was released in 2016 and continues to grow rapidly in popularity.

Giamir Buoncristiani is a Staff Software Engineer at Stack Overflow. He is also the tech lead for the Stacks design system. Giamir joins the podcast to talk about modernizing Stack Overflow’s front-end user interface and why the team has embraced Svelte.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Why Stack Overflow Uses Svelte with Giamir Buoncristiani appeared first on Software Engineering Daily.

The Google Sheets calculation engine was originally written in Java and launched in 2006. In the early days of the product, all calculation happened on the server. However, starting in 2013 the engine has run in the browser using JavaScript.

Google Sheets is now one of the first products at Google to use WebAssembly Garbage Collection, or WasmGC, on Chrome.

Michael Thomas is the Multiplatform Lead for Google Workspace and Thomas Steiner is a Developer Relations Engineer at Google. They join the podcast to tell the story of why and how Google Sheets ported its calculation worker from JavaScript to WasmGC.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post How Google Sheets Uses WasmGC with Michael Thomas and Thomas Steiner appeared first on Software Engineering Daily.

Large datasets require large computational resources to process that data. More frequently, where you process that data geographically can be just as important as how you process it.

Expanso provides job execution infrastructure that runs jobs where data resides, to help reduce latency and improve security and data governance.

David Aronchick is the CEO of Expanso. He previously worked at Google on the Kubernetes team, which influenced his decision to start Expanso. David joins the show to talk about his company.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Containers at the Edge with David Aronchick appeared first on Software Engineering Daily.

Douglas Crockford is an American computer programmer who is involved in the development of the JavaScript language. He specified the JSON data format, and has developed various JavaScript related tools such as the static code analyzer JSLint. Crockford is also a game developer and worked at Atari.

Douglas joins the podcast to talk about his career and work in computer science.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Evolving JavaScript with Douglas Crockford appeared first on Software Engineering Daily.

Damien Filiatrault is the founder and CEO of Scalable Path, a software staffing agency that matches companies and startups with vetted, remote software developers. The company was founded in 2010, and since then has worked on hundreds of client projects and has built a freelance network with 35,000 remote developers in 177 countries.

Damien joins the podcast to talk about software engineering management, the state of the software engineering job market, the challenge of hiring engineers, measuring productivity, and more.

Full Disclosure: This episode is sponsored by Scalable Path**.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Developer Productivity with Damien Filiatrault appeared first on Software Engineering Daily.

Tanzu GemFire is a distributed, in-memory, key-value store that performs read and write operations at fast speeds. It offers highly available parallel message queues, continuous availability, and a scalable event-driven architecture. It was developed to have sub-millisecond response times and accordingly found early application in automated trading environments on Wall Street.

Ivan Novick is the Product Manager for GemFire at the Tanzu Division of Broadcom. He joins the show to talk about Tanzu GemFire and its applications.

Full Disclosure: This episode is sponsored by VMware**.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post GemFire with Ivan Novick appeared first on Software Engineering Daily.

In 2022, Stefan Li and Stew Fortier envisioned a document editor with language model features built in. They founded Type.ai, received backing from Y Combinator, and have since been at the frontier of building a next-generation document editor. However, to ensure a robust and performant frontend, Type.ai needed to take advantage of many modern browser features.

Stefan Li is the CTO of Type.ai, and he joins the show to talk about the state of frontend dev, the service worker API, IndexedDB, the SharedWorker interface, Web Locks, and more.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Modern Frontend Engineering with Stefan Li appeared first on Software Engineering Daily.

A major challenge in applied AI is out-of-distribution detection, or OOD, which is the task of detecting instances that do not belong to the distribution the classifier has been trained on. OOD data is often referred to as “unseen” data, as the model has not encountered it during training.

Bayan Bruss is the VP of AI Foundations at Capital One and in this role he works with academic researchers to translate the latest research to address fundamental problems in financial services. Bayan joins the show with Sean Falconer to talk about OOD, the importance of bringing AI research to real world applications, and more.

Full Disclosure: This episode is sponsored by Capital One**

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post AI Research at Capital One with Bayan Bruss appeared first on Software Engineering Daily.

DuckDB is an open-source column-oriented relational database that was first released in 2019. It’s designed to provide high performance on complex queries against large databases, and focuses on online analytical processing workloads.

Hannes Mühleisen is the Co-Creator of DuckBD, and is the CEO and Co-Founder of DuckDB Labs. He joins the show to talk about drawing inspiration from SQLite, why DuckDB was written in C++, the novel data processing scenarios it enables, and more.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post DuckDB with Hannes Mühleisen appeared first on Software Engineering Daily.

Rerun is an open source SDK and viewer for visualizing and interacting with multimodal data streams. The SDK lets you send data from anywhere, and the viewer collects the data and aligns it so the user can scroll back and forth in time to interpret it. The tools have been applied in spatial computing, augmented reality, virtual reality, and mixed reality.

Emil Ernerfeldt is the Co-Founder and CTO of Rerun. Emil is also the creator of egui which is a popular GUI library written in Rust. He joins the podcast to talk about his history in game development, building super fast tools, and developing Rerun.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Creating GUIs in Rust with Emil Ernerfeldt appeared first on Software Engineering Daily.

Chroma is an open-source AI application database.

Anton Troynikov is a Founder at Chroma. He has a background in computer vision and previously worked at Meta. In this episode Anton speaks with Sean Falconer about Chroma, and the goal of building the memory and storage subsystem for the new computing primitive that AI models represent.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Chroma’s Vector Database with Anton Troynikov appeared first on Software Engineering Daily.

In 79 AD, in the ancient Roman town of Herculaneum, twenty meters of hot mud and ash buried an enormous villa once owned by the father-in-law of Julius Caesar. Inside, there was a vast library of papyrus scrolls.

The scrolls were carbonized by the heat of the volcanic debris, but they were trapped underground where they remained preserved.

It wasn’t until the 1750s that the scrolls were discovered, but they were fragile and resistant to being opened and read.

Then, in 2015, researchers used X-ray tomography and computer vision to virtually unwrap the scrolls.

Last year, the Vesuvius Challenge was launched by Nat Friedman, Daniel Gross, and Brent Seales to crowdsource the process of reconstructing the text from the scrolls.

Juli Schilliger and Youssef Nader are two members from the winning team. They join the show to talk about the computational approaches they used to reconstruct the scroll text.

For interested listeners, the 2024 Vesuvius Challenge is now live, with new challenges and prizes. Check out ScrollPrize.org to learn more.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post The Vesuvius Challenge with Juli Schilliger and Youssef Nader appeared first on Software Engineering Daily.

Traditionally, security checks and testing are performed towards the end of the software development lifecycle. However, discovering vulnerabilities at that stage can be costly and time-consuming.

This observation has led to the shift-left movement, which advocates for implementing security testing earlier in the software development process.

HoundDog AI is a startup focused on software to enable shift-left security practices. Amjad Afanah and Sudipta Mukherjee are Co-Founders of HoundDog, and they join the show to talk about their company.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Shift-Left Security and Code Scanning with Amjad Afanah and Sudipta Mukherjee appeared first on Software Engineering Daily.

Uber handles billions of trips and deliveries, and tens of billions of financial transactions across drivers, couriers, users, and merchants every quarter.

LedgerStore is an immutable storage solution at Uber that provides verifiable data completeness and correctness guarantees to ensure data integrity for its transactions.

Kaushik Devarajaiah is the Tech Lead for LedgerStore at Uber. He joins the show to talk about scaling Uber’s data and storage infrastructure.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Uber’s LedgerStore and its Trillions of Indexes with Kaushik Devarajaiah appeared first on Software Engineering Daily.

One of the fastest areas of growth in observability is frontend observability, or real user monitoring. This is the practice of monitoring and analyzing the performance, behavior, and user experience of web applications from the user’s perspective.

Purvi Kanal is a Senior Software Engineer at Honeycomb. She joins the podcast to talk about the evolution and status of real user monitoring.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Frontend Observability with Purvi Kanal appeared first on Software Engineering Daily.

Inworld is a company that provides tools for game studios to add AI-driven gameplay. They are at the leading edge of using generative AI in game development, and have worked with companies such as Xbox, Ubisoft, and NVIDIA.

Igor Poletaev is the VP of AI and Nathan Yu is the Director of Product and GM of Labs at Inward. They join the show to talk about using AI in game development.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post AI Tools for Game Development with Igor Poletaev and Nathan Yu appeared first on Software Engineering Daily.

Static analysis is the examination of code without executing the program. It’s used to identify potential errors, code quality issues, security vulnerabilities, and adherence to coding best practices.

Abbas Sabra is a Principal Engineer at Sonar, which creates tools to help developers produce clean code. Abbas specializes in C++ static analysis, and began his career in the financial industry, where he identified inefficiencies within the C++ tooling ecosystem. He joins the show to talk about static analysis and static analysis tool development.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post C++ Static Analysis with Abbas Sabra appeared first on Software Engineering Daily.

Bison Ventures invests in frontier technology companies that use innovative science and deep technology. A key pillar of their investment portfolio is climate technology.

Tom Biegala is a co-founder of Bison Ventures. Prior to starting Bison Ventures, Tom worked at Cascade Asset Management Company, the investment office that manages the assets of the Bill & Melinda Gates Foundation Trust and Gates family. He joins the show to talk about why he started Bison, the climate tech startup landscape, and much more.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Climate Tech Investing with Tom Biegala appeared first on Software Engineering Daily.

Luma AI develops technologies at the forefront of AI and graphics. They created a text-to-3D tool that functions like Midjourney but for generating 3D models. Another tool makes photorealistic environments by reconstructing any scene in 3D from just a few photos.

Karan Ganesan is a Software Engineer and Barkley Dai is the Product and Growth Lead at Luma AI. They join the show to talk about the origin of the company and the technologies it uses.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Luma AI with Barkley Dai and Karan Ganesan appeared first on Software Engineering Daily.

Redis is an in-memory database that can be used for caching, vector search, and as a message broker.

Brian Sam-Bodden is a Senior Applied AI Engineer at Redis. He joins the show to talk about his work and AI at the company.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post AI at Redis with Brian Sam-Bodden appeared first on Software Engineering Daily.

David Szymanski is a video game developer focused on short retro horror games. He created the hit FPS Dusk, along with Iron Lung, Chop Goblins, and the upcoming Butcher’s Creek. He’s also involved in the production of the upcoming Iron Lung film.

David joins the podcast to talk about his work, how to capture an appealing retro game feel, why he makes short games, developing in Unity, looking beyond Unity, and more.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Dusk and the Art of Making Short Games with David Szymanski appeared first on Software Engineering Daily.

Retool is a platform to help engineers quickly build internal frontends. It does this by abstracting away repetitive aspects of frontend development. The platform was started in 2017 and has received funding from Sequoia, Stripe Co-Founders, and Nat Friedman.

David Hsu is the founder and CEO of Retool. He joins the show to talk about why he started coding, studying philosophy and computer science, Retool’s tech stack, and more.

Paweł is the founder at flat.social the world’s first ‘flatverse’ start-up and glot.space, an AI-powered language learning app. Pawel’s background is as a full-stack software engineer with a lean and experimental approach towards product development. With a strong grounding in computing science, he spent the last decade getting early-stage products off the ground – both in startup and corporate settings. Follow Paweł on Twitter, LinkedIn and his personal website – pawel.io.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Fast Frontend Development with David Hsu appeared first on Software Engineering Daily.

ChatGPT has been out for more than a year and has since become the centerpiece of intense discussion and debate about AI.

Christian Hubicki is a renowned robotics research scientist and an Assistant Professor of Mechanical Engineering at Florida State University. In 2023, he was a guest on Software Engineering Daily, where he discussed ChatGPT and its implications with Sean Falconer. Christian now joins Sean again to check in about the state of AI and its future directions.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post One Year of ChatGPT with Christian Hubicki appeared first on Software Engineering Daily.

Databases underpin almost every user experience on the web, but scaling a database is one of the most fundamental infrastructure challenges in software development. PlanetScale offers a MySQL platform that is managed and highly scaleable.

Sam Lambert is the CEO of PlanetScale and he joins the show to talk about why he started the platform, scaling databases, using Vitess for SQL shard orchestration, and more.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Hyperscaling SQL with Sam Lambert appeared first on Software Engineering Daily.

GV, or Google Ventures, is an independent venture capital firm backed by Alphabet.

Erik Nordlander is a General Partner at GV and invests across enterprise software and frontier technology, focusing on developer tools, cloud infrastructure and machine learning. He has backed companies like Cockroach, Warp and Neo4j. Prior to joining GV in 2010 and opening up the firm’s New York City office, Erik was at Google and led development of the company’s next-gen display and serving system, and built statistical and machine learning models for Google’s ad businesses.

Erik joins the podcast to talk about his work.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Google Ventures with Erik Nordlander appeared first on Software Engineering Daily.

A common challenge for developers of SaaS products is integrating with existing services, including services that customers might already be using. For example, a SaaS product might need to integrate with customers using Salesforce, HubSpot, or another CRM system. However, this can be demanding for developers when 3rd party APIs are poorly documented or inconsistent.

Lauren Long is a co-founder at Ampersand which is a developer platform for SaaS integrations. She joins the show to talk about smoothing out API connectivity to make SaaS interoperable.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post The Challenge of API Design with Lauren Long appeared first on Software Engineering Daily.

A headless software architecture decouples the frontend, or the “head”, from the backend. This separation allows developers to manage the UI layer independently of the backend logic and data management.

Hydrogen is Shopify’s open-source headless framework for building custom storefronts. It’s React-based and is focused on performance and flexible UI components.

Ben Sehl is a Senior Product Lead at Shopify where he works on Hydrogen and the storefront developer experience. He joins the show to talk about his engineering background, the motivation for creating Hydrogen, Hydrogen versus the Liquid templating language, and much more.

Be sure to check out the 2024 Shopify Editions Dev Newsroom and Brochure.

Full Disclosure: This episode is sponsored by Shopify**

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Shopify’s Hydrogen Framework with Ben Sehl appeared first on Software Engineering Daily.

Celeste is a critically acclaimed action platformer developed by Extremely OK Games. They recently released Celeste 64 which is an open source project to commemorate the 6th anniversary of the original Celeste, and development of their next major game, Earthblade, is underway.

Noel Berry is a programmer at Extremely OK Games and he joins the show to talk about developing Celeste and Earthblade, gameplay engineering, the state of C# versus C++ for game development, and more.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Celeste and Platform Game Engineering with Noel Berry appeared first on Software Engineering Daily.

DataStax is a generative AI data company that provides tools and services to build AI and other data-intensive applications.

Ed Anuff is the Chief Product Officer at DataStax. He joins the show to talk about making Apache Cassandra accessible, adding vector support at DataStax, envisioning the future application stack for AI, and more.

Full Disclosure: This episode is sponsored by DataStax**

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post DataStax with Ed Anuff appeared first on Software Engineering Daily.

Kong is a software company that provides open-source platforms and cloud services for managing, monitoring, and scaling APIs and microservices.

Marco Palladino is the CTO of Kong, and he joins the podcast to talk about the platform and APIs as the building blocks of the digital world.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post It’s APIs All the Way Down with Marco Palladino appeared first on Software Engineering Daily.

Bitwarden is an open-source password management service that securely stores passwords, passkeys, website credentials, and other sensitive information

Matt Bishop is a Principal Architect at Bitwarden. He joins the show to talk about the platform and his work there.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Bitwarden with Matt Bishop appeared first on Software Engineering Daily.

Codecademy is an online platform that offers classes on languages including Python, Go, JavaScript, C++, and many others.

Zoe Bachman is the Senior Curriculum Director at Codecademy and designs courses for the platform. She joins the podcast to talk about her work there.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Codecademy with Zoe Bachman appeared first on Software Engineering Daily.

Web development has exploded in complexity and dynamism over the last decade. Of all the paradigms created during this period, two have bubbled to the top: GraphQL and REST. Each has significantly impacted how we build things digitally, and each has its unique approach to data retrieval and exchange. This article aims to demystify these technologies, offering a clear comparison to help developers make informed decisions. Let’s start with a high-level overview of GraphQL and REST.

Understanding GraphQLWhat is GraphQL?GraphQL is a query language developed by Facebook in 2012, aimed at making data fetching more efficient and tailored to the needs of modern web applications. It allows clients to request exactly the data they need, nothing more, nothing less, which can significantly reduce the amount of data transferred between the client and server.

Imagine you’re at a buffet (the server) where an extensive array of dishes (data) is available. In a traditional buffet, you might be given a plate (the API response) with portions of every dish available, regardless of your taste or appetite. This is akin to traditional REST APIs, where the server defines what data is available in each endpoint, often leading to over-fetching (getting more data than you need) or under-fetching (needing to make additional requests for more data).

With GraphQL, you’re free to choose exactly what dishes you want and in what quantity. You can select your preferred dishes (specific data fields), specify the amount of each (query depth and complexity), and even mix ingredients from different dishes to create a custom plate (combining data from multiple sources in one request). This “choose-what-you-need” approach reduces waste (bandwidth and memory usage) and ensures you get exactly what you want (the data needed for the application).

At the heart of a GraphQL endpoint is the schema. Designing a GraphQL schema is a foundational step in building a GraphQL API, necessitating a collaborative and strategic approach among cross-functional teams, including product managers, developers, and designers. This process not only outlines the structure of the API but also ensures that the product’s data needs align with the user experience and business objectives.

A GraphQL schema serves as a contract between the client and the server, detailing the types of data available and the ways clients can interact with that data. It is defined using GraphQL’s Schema Definition Language (SDL). It specifies queries for retrieving data, mutations for modifying data, and subscriptions for real-time updates, ensuring a comprehensive blueprint for data interaction.

If you are working with an engineering team that divides itself into many domains, then the schema design process encourages early and ongoing collaboration between product managers, developers, and designers across the domains. This multidisciplinary approach ensures that the schema reflects the product’s requirements and user needs accurately. Designing with the end user in mind ensures that the schema is intuitive and aligns with user expectations, leading to a better user experience.

Exploring RESTWhat is REST?REST stands for Representational State Transfer, an architectural style defining a set of constraints for creating web services. It operates through standard HTTP methods and treats server objects as resources that can be created, retrieved, updated, or deleted.

Each endpoint in a REST API corresponds to a specific resource or a collection of resources. The design revolves around resource identification, representation, and the actions that can be performed on them. Designing endpoint paths with consideration for resource relationships and hierarchy can make the API more intuitive but should be used judiciously to avoid overly complex URLs.

Identifying the resources and modeling the domain are critical first steps, necessitating the product team to have a thorough comprehension of the business entities. This can lead to more upfront discussions on endpoint structure, resource naming conventions, and resource relationships. REST involves designing multiple endpoints, each corresponding to a specific resource or a collection of resources. This requires more collaboration between frontend and backend teams to ensure the endpoints match client application workflows.

Comprehensive documentation is essential for ensuring that developers understand how to use the API. Tools like Swagger (OpenAPI) can automate documentation and provide interactive interfaces for testing endpoints.

Similarities and DistinctionsCommon GroundGraphQL and REST are both firmly rooted in the client-server model. This architecture separates the user interface concerns from the data storage concerns, allowing developers to build more scalable and maintainable systems. At their core, both GraphQL and REST are about simplifying the communication process between clients (such as web or mobile applications) and servers. They provide a structured way to query and manipulate data over the internet, making it easier for developers to build interactive, data-driven applications.

When to Use GraphQLGraphQL’s single evolving schema allows for easier iterative development and feedback loops. Teams can add fields and types without affecting existing queries, enabling smoother evolution of the API as user needs and business requirements change. For example, frontend teams can specify exactly what data they need for their own feature, they can work on specific portions of the schema without impacting other teams, which can lead to more efficient discussions on data requirements and less back-and-forth communication.

Consider a financial application for managing corporate expense cards where a user’s dashboard might display a feed of transactions, notifications, payment cards, and money transfers, all of which are interconnected data points. With GraphQL, the frontend can make a single query to fetch all required data simultaneously.

For example:

Let’s say, for instance, the transaction team wants to support new features such as categorizing transactions or adding merchant details. In GraphQL, this can be achieved without impacting the existing schema or the work of other teams. Let’s say our current schema definition looks like the following:

The transaction team could opt to extend the Transaction type with a new Merchant type:

Then update the resolver for the Transaction type:

Now, say the Card Issuance team wants to support a set of Security features that involves authorized access to card details (i.e. cvv / pin ). They would work on extending the Card type.

When updating the resolver, the can not only quickly add another resolver for the field but also apply field-level permissions to the team a newly created field.

When to Use RESTREST tends to be best suited for a few particular scenarios. For example, applications that are straightforward or that heavily rely on caching. It also tends to be best with data models with a relatively flat structure, where resources (the entities or objects in your application) are easily identified and accessed directly through URLs. REST can also be a very effective approach if your application’s data can be represented as a collection of simple, independent entities without complex interrelations.

Other areas where REST is helpful are if the use case requires clear resource identification. Each resource in a RESTful API is identified by a unique URL, making it clear what each endpoint represents and how it can be interacted with. This simplicity in identifying and accessing resources contributes to the overall straightforwardness of using REST. Given its standardized nature, REST is often preferred for APIs where extensive documentation and ease of consumption are crucial.

Example Scenario with RESTIn a RESTful API architecture, resources such as transactions, notifications, cards, and transfers are typically accessed through distinct endpoints. Implementing new features like SecurityFeatures for a resource involves considering both the endpoint design and the security model, ensuring that sensitive information is only accessible to authorized users. Here’s how this could be structured and secured:

RESTful Endpoints

Transactions: GET /api/users/{userId}/transactions
Notifications: GET /api/users/{userId}/notifications
Payment Cards: GET /api/users/{userId}/cards
Money Transfers: GET /api/users/{userId}/transfers

For a comprehensive dashboard view, the client would need to make separate requests to each of these endpoints. This could be particularly challenging in scenarios with limited bandwidth or high latency, as the cumulative delay from multiple round trips could degrade the user experience significantly. To populate this view, the app makes four separate API calls:

Fetch transactions to show recent spending activity.
Fetch notifications to alert the user to any important updates or actions required.
Fetch payment cards to allow the user to select a card for new transactions or to manage existing cards.
Fetch money transfers to review recent transfers or to initiate new ones.

Each call waits for a response before the dashboard can fully render, leading to a visible delay for the end user.

From a development perspective, adding the SecurityFeatures section to the existing card resource endpoint is different from the GraphQL implementation. This section would include sensitive data related to a payment card, such as CVV or PIN management capabilities, which should only be available to authenticated and authorized users. One common approach to securing this data with REST is using middleware in the server that processes requests. Middleware can intercept requests before they reach the endpoint logic, allowing the server to authenticate and authorize requests.

Choosing the Right ToolThe choice between GraphQL and REST is not a matter of superiority but of suitability. GraphQL offers unparalleled flexibility and efficiency for complex, data-intensive applications, while REST shines in its simplicity, cacheability, and standardization. The decision should be guided by the project’s specific requirements, available development resources, and scalability considerations.

The post GraphQL vs. REST: What Are They, and Which Is Better for You? appeared first on Software Engineering Daily.

Akash Network is a decentralized cloud computing platform that leverages unused compute capacity around the world. It makes this capacity available to others, and provides a decentralized peer-to-peer model for managing and paying for these resources in an online marketplace.

Greg Osuri is the CEO for OverClock Labs which created Akash Network. He joins the show to talk about Akash.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post A Decentralized Compute Marketplace with Greg Osuri appeared first on Software Engineering Daily.

Linting is the process of checking source code for programmatic as well as stylistic errors. Ruff is a highly popular Python linter written in Rust. It was developed by Charlie Marsh, who also founded Astral, which is focused on next-generation Python tooling.

Charlie joins the podcast to talk about open source development, Ruff, the UV package installer, and much more.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Ruff and Next-Generation Python Tooling with Charlie Marsh appeared first on Software Engineering Daily.

Gabriel Gambetta is a Senior Software Engineer at Google where he works on YouTube. He’s an expect in computer graphics and game development, and is famous for his articles on engineering fast-paced multiplayer games.

Gabriel joins the show to talk about his history with game development, client-server game architecture, rubber-banding, raytracing, rasterizers, and much more.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Engineering Fast-Paced Multiplayer Games with Gabriel Gambetta appeared first on Software Engineering Daily.

The U.S. government recently released a report calling on the technical community to proactively reduce the attack surface area of software infrastructure. The report emphasized memory safety vulnerabilities, which affect how memory can be accessed, written, allocated, or deallocated.

The report cites this class of vulnerability as a common theme in the some of the most infamous cyber events, such as the Morris worm of 1988, the Heartbleed vulnerability in 2014, and the Blastpass exploit of 2023.

Herb Sutter works at Microsoft and chairs the ISO C++ standards committee. He joins the show to talk about C++ safety.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post C++ Safety with Herb Sutter appeared first on Software Engineering Daily.

Figma is a cloud-based design and product development platform that is widely used in UI and UX work. It allows users to collaborate in real-time, a key feature that has helped drive its popularity. Figma is an impressive engineering feat, in part because of how far it pushes what’s possible in a web browser.

Abhi Mathur is the VP of Platform Engineering at Figma. He joins the show today to talk about his path to Figma, architecture, scaling, team organization, and more.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Figma Engineering with Abhi Mathur appeared first on Software Engineering Daily.

The majority of enterprise data exists in heterogenous formats such as HTML, PDF, PNG, and PowerPoint. However, large language models do best when trained with clean, curated data. This presents a major data cleaning challenge.

Unstructured is focused on extracting and transforming complex data to prepare it for vector databases and LLM frameworks.

Crag Wolfe is Head of Engineering and Matt Robinson is Head of Product at Unstructured. They join the podcast to talk about data cleaning in the LLM age.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Unstructured Data and LLMs with Crag Wolfe and Matt Robinson appeared first on Software Engineering Daily.

Red Hat Enterprise Linux is a Linux distribution developed by Red Hat for the commercial market. The company recently announced changes to the availability of its source code.

In response, CIQ, Oracle and SUSE formed the Open Enterprise Linux Association, or OpenELA, which aims to provide a consistent and secure upstream location for Enterprise Linux distribution sources.

Wim Coekaerts is Executive Vice President of Software Development at Oracle, and he joins the podcast to explain the recent changes in the Linux enterprise space, and discuss the OpenELA and its importance to the Linux ecosystem.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post The Changing Enterprise Linux Ecosystem with Wim Coekaerts appeared first on Software Engineering Daily.

LLMs have become one of the most important technologies to emerge in recent years. Many of the most prominent LLM tools are closed source, which has led to great interest in developing open-source tools.

Antonio Velasco Fernández is a Data Scientist and Jose Pablo Cabeza García is a Lead Data Engineer, both at Elastacloud. In this episode, recorded in 2023, they joined the podcast to talk about LLMs and the importance of community development for LMMs.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post LLM Community Development with Antonio Velasco Fernández and Jose Pablo Cabeza García appeared first on Software Engineering Daily.

A deepfake is a synthetic media technique that uses deep learning to create or manipulate video, audio, or images to present something that didn’t actually occur. Deepfakes have gained attention in part due to their potential for misuse, such as creating forged videos for political manipulation or spreading misinformation.

Ryan Ofman is a Lead Engineer and Head of Science Communication at DeepMedia, which is a platform for AI-powered deepfake detection. He joins the show to talk about the state of deepfakes, their origin, and how to detect them.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Detecting Deepfakes with Ryan Ofman appeared first on Software Engineering Daily.

Decompilation is the process of translating a compiled program’s bytecode back into a higher-level programming language, like C. There’s a vibrant and growing scene of engineers working to decompile classic video games, and some of the most prominent projects have focused on the Nintendo 64. Recent successes include Super Mario 64, The Legend of Zelda: Ocarina of Time, and Paper Mario.

Ethan Roseman and Mark Street are both software engineers with experience in the decompilation scene. In addition to their work on specific games, they’re active in creating open-source tooling for the decompilation community, including Splat which is a binary splitting tool, and Decomp.me which is a collaborative decompilation and reverse engineering site.

Ethan and Mark join the podcast to talk about N64 game decompilation, surprising discoveries in the game code, tool development, and much more. Be sure to check out the Decomp.me Discord to learn more.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Bonus Episode: The N64 Decompilation Scene with Ethan Roseman and Mark Street appeared first on Software Engineering Daily.

CSS, or Cascading Style Sheets, is a fundamental technology in web development that defines the presentation and layout of HTML documents. It serves as a styling language that allows developers to control the appearance of web pages.

Rachel Andrew is a Staff Technical Writer on the Google Chrome Team. Before that she worked for Mozilla and was also the Editor in Chief at Smashing Magazine. She is a member of the CSS Working Group, the Web Standards Project, and Open Web Docs. Rachel is our guest today.Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post The State of CSS with Rachel Andrews appeared first on Software Engineering Daily.

Security Operations, or SecOps, refers to the collaboration between security and operations teams to secure an organization’s systems, applications, and data.

Maxime Lamothe-Brassard is a Co-Founder of LimaCharlie which is a cloud SecOps platform. He has a background in security and has previously worked at the Canadian Intelligence service, Crowdstrike, Google, and Google X. He joins the podcast to talk about modern security operations.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Modern SecOps with Maxime Lamothe-Brassard appeared first on Software Engineering Daily.

Microsoft Copilot is a chatbot developed by Microsoft that launched in 2023 and is based on a large language model.

Justin Harris is a Principal Software Engineer at Microsoft and has an extensive background in classical machine learning and neural networks, including large language models. He joins the show to talk about Microsoft Copilot, natural language processing, ML team organization, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Microsoft Copilot with Justin Harris appeared first on Software Engineering Daily.

The interplanetary filesystem, or IPFS, is a peer-to-peer network that uses a distributed and decentralized model. Functionally, IPFS allows users to store and share files without having to rely on a single source of truth for those files.

Matt Ober is the Co-Founder & CTO of Pinata. He joins the show to talk about IPFS and Pinata.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Pinata and the Interplanetary File System with Matt Ober appeared first on Software Engineering Daily.

Animal Well is a Metroidvania game developed as a solo project by Billy Basso over the course of seven years. It’s the first game released by publisher Bigmode, which was founded by Jason “Dunkey” Gastrow.

Billy joins the show to talk about creating Animal Well’s engine from scratch, how the game handles animation, fine-tuning character movement, and more.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Animal Well with Billy Basso appeared first on Software Engineering Daily.

One of the most promising applications of large language models is giving non-experts the ability to easily query their own data. A potential positive side effect is reducing ad-hoc data analysis requests that often strain data teams.

Sarah Nagy is the Co-founder and CEO at Seek which is using natural language processing to change how teams work with data. She joins the podcast to talk about the platform and providing a natural language interface to databases.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer . Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post LLMs for Data Queries with Sarah Nagy appeared first on Software Engineering Daily.

Augmented reality and extended reality, or AR and XR, are in the early stages of being integrated into design workflows. But they are likely to transform design in the years to come. Jenna Fizel has a background in architecture and desktop software engineering. She sat down with Tyson Kunovsky to talk about her work at the design firm IDEO and how they’re using AR and XR for industrial design prototyping.

TK: Hi, Jenna. And welcome to Software Engineering Daily.

JF: Thanks, Tyson. Happy to be here.

TK: Jenna, before we start, I’d love to learn a little bit more about you and your software engineering background. How did you become a software engineer? And what did your academic journey look like?

JF: Sure. I came to software actually through architecture originally. My academic background is in computational geometry. And so, I, for a while, was writing code and developing algorithms to create things like building facades and realized that I was a lot more interested in the systems I was building than the fact that they turned into buildings in the end.

I shifted careers a little bit. And for about a decade, was a desktop C++ developer. Creating custom software for lots of different kinds of institutions. From companies, to libraries to help them explain complex data to various audiences through sort of massive installations that ran across multiple servers connected to many different kinds of backends. And then display that all in novel interfaces located in places. That’s kind of where my architecture background remained. I always like to build things for specific human contexts.

TK: Well, that makes sense, because now you’re a senior director of emerging technology for a really interesting and unique company called IDEO. And, Jenna, for those that aren’t familiar with IDEO as an organization, what kinds of work does IDEO do? And who do you do it for?

JF: Yeah. IDEO is a design consultancy, which means that we help our clients with problems that can be framed as design questions. Now, that’s vague, because we do a lot of different things. We’ll work on anything from the physical design of something like a guitar capo to government policy. And we do this in a really cross-disciplinary way. We have dedicated teams for each client project where we combine different skill sets and different backgrounds. You might see somebody like myself with a software engineering background. Somebody with a background in organizational change or somebody with a business design background alongside of people who are experts in graphic design or interaction design all working together on a single problem over a concentrated period of time.

TK: That’s quite the breadth of solutions that you offer. Given we’re software podcast, we don’t usually have a lot of folks on talking about designing guitar capos and government policy. But I’m guessing that taking this design centric worldview and applying it to the problems that you are solving is a unique and powerful way to create novel solutions that you might not be able to arrive at otherwise.

Given that IDEO is so cross-disciplinary, how do you see the role of actual software engineering in the design work that you do?

JF: Yeah. I think there’s sort of two approaches here. One is simply our interdisciplinarity. We always value the expertise of our team members and bringing sort of the lens of the kind of craft that they’re bringing into the situation in understanding the design problem. For software, that’s a lot about thinking systematically or thinking about real-world implications of design decisions. Or actually building software. I still do that. And my colleagues do that as well. But we do it in a slightly different way that I think we’ll probably dig into.

And then I think there is the sort of expertise level. Because we work on a lot of digital products. Or we work on systems or questions that are adjacent to digital product development. And so, bringing the sort of experience lens of having worked myself on both desktop and web-based products and other people with a variety of other kinds of experience. We can bring in the sort of needs of the client context more vitally into the room when the design solution is being worked through.

TK: It almost sounds like I don’t know quite what to call this. But your approach seems more as if it’s applied software engineering for the real-world, which is interesting. Because when you think about why companies typically build or hire others to build software, it’s usually for some very specific and often, for lack of a better term, boring business purpose. Given the work that IDEO does is so diverse, why does IDEO build software?

JF: Yeah. I would say that there’s sort of three reasons why we build software. One is to create a prototype. To sort of answer the question, “How could this thing work?” And that is especially relevant these days in emerging technology where if we’re designing like an augmented reality experience, it’s not really enough to just talk about it or even draw it. You kind of have to make it or make at least some limited aspect of it.

And, often, we’ll come to clients who have large engineering organizations. And maybe they have some prototypers that they’re often working inside of relatively complex systems. And we take the approach that you should only build what you need to answer the question you’re trying to ask with your prototype.

Step one, or approximately step one, is to define that question. And then step two is to try to build something and see how hard that is. And then maybe go revise step one again until you get to something that leads you to a better question. And so, that’s what a prototype is. It’s a machine for asking better questions. That’s one thing.

Another one is experiences. One of our co-founders, Bill Moggridge, has this quote I love that the only way to experience the experience is to experience it, which a little bit silly. But I think it’s also really true. Sometimes we need ourselves and our clients to sort of feel a future scenario or even feel the possibility in an existing design. And, often, the only way to do that is to actually make the thing to some degree.

Again, we always are trying to be lean with how much effort we’re putting into any of our work, including engineering work. We think through like how much of this experience? And in what ways does it need to be real to convey the emotion or that connection to others? Or the impact of the design choices we’re making. Yeah, four experiences.

And then I think, thirdly, to support with strategic choices. Sometimes, or really often, even when we’re doing product design, it’s in the context of larger decisions a company is making. What markets to enter? Or what new teams to stand up? And it can be really powerful to build a little bit of the future of what those teams might accomplish in order to help feel the consequences of different strategic choices.

TK: It sounds like software then is just yet another tool in your belt to help facilitate understanding. And I absolutely love that quote. The only way to experience the experience is to experience it. I’m definitely going to have to steal that.

Now that we know a little bit more about your overall approach to utilizing software, let’s change direction and talk about something that I know you have deep experience with, which is augmented reality, or XR, as it’s commonly referred to for those that are unfamiliar. When it comes to emerging technologies like XR, how are those technologies used by the engineers and designers at IDEO today?

JF: Yeah. It’s a good question. First of all, IDEO’ers are curious people. And they’re always sort of looking out for the next new thing. And so, a lot of us bring in our interests, especially in emerging technology, naturally to our design work. We also do have some slightly more formal structures that encourage this.

Part of my role actually is running our internal learning group on emerging technology that I started during the pandemic actually to help with collaboration. It was kind of a wonderful – or wonderful and tragic confluence of factors where we’d had VR headsets and Microsoft HoloLenses lenses in our studios. But one or two of them that project teams could take on and use.

But we were all in our own homes, and apartments, and things when the Meta Quest 2 came out at a low enough price point that we could conceivably send out hardware to more people who wanted them. And that created an opportunity to make a sort of obligation between the people who raised their hands for this hardware and a learning community.

I’ve sort of engaged with learning communities before. And it is very important to make sure that everyone is getting something out of learning things. It was an amazing opportunity to have this sort of like new capability and hardware to send to people in exchange for their commitment to use it and share back their learnings.

And that has really blossomed into a now – well, sort of about 80-person active membership where we have folks trying out different virtual reality, augmented reality. And, of course, new sort of AI tools. And then sharing back their experiences, which we do every Thursday. And we try to keep those sharebacks relevant to concerns within client work sometimes directly supporting a specific piece of work. But, also, often driven by sort of what is happening inside of the realm of feasibility itself. I think this is a little bit underappreciated source of inspiration, which is actually sort of like what is possible? What can we build? And what can we build relatively easily? And that is one of the purposes of this group.

TK: Could you give a couple of examples of things that you’ve learned about XR during these sharebacks? And then perhaps use that to speak about how these learnings help you and your team better solve problems for your customers.

JF: Yeah. I think um a really powerful example is what I’ve seen happen inside of the industrial design community. They’ve really found a few different high value approaches to using XR in particular. Although, actually today, this afternoon, one of my industrial design colleagues will be sharing back about using AI tools to go from sketches to rendering. We have a variety of interests.

But I think the first thing I saw for that community was reality capture. Using mostly their phones. But, also, occasionally, dedicated hardware to take physical mockups or even physical objects in their environment. Scan them and then share them. This is maybe a little bit less relevant now that a lot of them share a physical location. But, certainly, when everyone was separated, this was incredibly useful. And that was really used sort of like inside the design process to get to results more quickly and with a higher fidelity.

It has since morphed into a storytelling technique in a lot of ways. Whereas before, teams would always present renderings at the end of a design project. They’re now often also sharing augmented reality models with each other and with the client themselves, which I think really lets you see the design in context in a way that you previously would have to make a high fidelity physical model before. And so, that’s really exciting.

This is also a little bit to do with industrial design. But we do a lot of what’s called design research, which is going out uh into the contexts of the people who will eventually be using the things that we design and asking them questions or engaging them in activities that help us understand their habits, their needs, their desires relevant to whatever it is we’re designing.

Um, and we’ve been able to, uh, engage with people again, especially people who have constraints on where they can physically be through – especially augmented reality. Sometimes we work on medical devices. And sometimes those medical devices are for people who have immune disorders or ill in other ways. And they are perhaps using like very complicated physical devices to help with their conditions. And to sort of produce a bunch of mockups of those physical devices. Sterilize them and send them. Or is a thing that we sometimes engage in but is very costly.

And with a fiducial that you can print out on your home printer and tape to a book, you can get an augmented reality experience where you can sort of understand things like where might a button sit. What might the sort of indications on a display be to help you understand how to walk through a treatment protocol or what have you? Or even to just understand does this item fit in with my life? Does it feel intimidating? Does it feel friendly? These are things that you can kind of get at without um sending a physical object and instead using a virtual overlay.

TK: Those are some interesting applications of XR. And it sounds like you’re working really closely with both technical and non-technical stakeholders to implement them. I want to talk more deeply about XR in a moment. But since we’re on the topic, I’m curious to get your perspective on collaboration between technical and non-technical stakeholders. Because I know it’s an issue for a lot of companies. How do you get non-technical folks to engage and provide helpful input, especially when maybe they don’t have the right technical expertise necessary to contribute effectively?

JF: Yeah. That’s a great question and an eternal struggle. But an exciting one. I really think that there’s sort of stereotypically a divide between the people who can sort of make things for real and then everybody else. It’s the consumer versus the creator.

And for a lot of these technologies, that line is getting blurrier and blurrier. It actually is relatively possible with a little bit of support and a sort of like friendly guardrails to help people without a lot of background make the leap into being productive creators.

But everybody has to be a little bit bought-in. And that is part of why having the learning group is so powerful. It contains many more folks than just the sort of software industrial design and interaction design people. We have people who work on business development in it. We have, again, design researchers in it.

And I try to balance the programming so that we have inspiration inside of the expertise of the various folks who are engaged in the community. I’ll often have – I had last year an artist called Grace Boyle do a short series of workshops with us. And she really cares about multi-sensory design and designing for emotions, dreams. Sort of ideas and concepts that might appeal to people with backgrounds other than engineering. But she then executes that work in a very precise and fairly engineering-heavy way. And so, that was sort of an attempt to get that inspiration in.

And then through a series of structured activities actually help people build and make. And I always try to offer a few different ways of building from something that you can do simply by clicking on things in a browser. To, yeah, forking a template that I’ve started for those who would like to write some TypeScript.

TK: You’re learning groups sound like an effective way of getting non-technical folks involved. Because you make it easy for them to become creators themselves. And I know a lot of companies try to do similar things with their lunch and learns, competency groups, and other activities but with varying degrees of success.

Before we get back to XR, what advice would you have for folks at organizations that might want to spin up and start running their own learning groups? I guess, specifically, what have you seen work to make folks feel like this isn’t yet just another job responsibility on top of everything else?

JF: I think there have to be a couple preconditions. One, I think people have to be interested. That sounds silly. But I think it’s actually really important. You have to either have the topic be attractive enough. You have to bring in interesting outsiders. Or you have to offer something. And so, I feel really lucky that I was able to offer those VR headsets when we were starting up.

And then the second part of your question, there has to be value there. This learning group’s been going for more than two years now. And that’s because what we do has real tangible impact for our clients. And while I value the sort of opportunity to think a little bit more broadly and with a little bit frankly lower stakes in some of these sessions, because we’re not necessarily building to exactly a deadline or for a high-stakes presentation, it’s really critical that people are gaining skills that then they can use in those high-stake situations or higher stake situations anyway.

And I try to do that both through connecting with leaders across IDEO to understand what are they seeing in the market. What are we seeing within our project teams? And then inviting sort of the closed loop of a shareback from a project team who’s used something, learned in the learning group to demonstrate to everybody that, actually, you can learn valuable stuff here. I think both being attractive and explicitly demonstrating value are both really important.

And I think there is a third way to do both of those things, which is to help people become just a tiny bit famous. My colleague, Danny Durant, started first a Medium blog. And now has graduated to be a part of IDEO’s website called Edges, currently, where we help people publish uh sort of this interesting work that they’re doing um at the edges of what IDEO does. That is both client work and independent work that happens.

And a great source for that has been for us the learning group. If you share something sufficiently interesting in that group, I will bother you to turn it into an article that we’ll then publish. Or I’ll bother you to submit to conferences. Juliette Laroche, one of the participants created a share a shareback about envisioning new kinds of materials using image generators. And it was a really compelling and profound presentation. And off of that, I actually pitched her to a conference organizer. She’s spoken a couple of different times about this topic and is actually um getting some of her images included in a textbook that’s going to be published next year. It is possible to sort of get some personal motivation into a learning group as well. But you have to put in that effort to help people find those opportunities before you get a little engine going and they become sort of easier and more natural.

TK: Some great advice there. Thanks for sharing. Okay. Now that we’ve talked about collaboration and learning groups, let’s circle back and talk about the topic of XR. When you think about the future, what gets you most excited about XR?

JF: I think we’ve seen, especially over the last couple of years, XR really used as a sort of explanatory tool or a tool to create imaginative leaps in our clients. We’ve done sort of large-scale simulations of things like retail stores of the future. We did this in work for Canada Goose that launched actually just pre-pandemic, uh which was a sort of rethink of a zero-inventory store with a bunch of immersive experiences. And because we sort of did all that previous work, they actually implemented basically everything that we put into our final design. It was an amazing thing to see the final physical results.

We’ve also done this for transportation. Sort of making simulations of personal vehicles up through things the size of airplanes. And that has been super successful. But what I’m really excited about these days is that more and more XR products are actually coming to market. And now there are all of these other questions around the products that companies are beginning to ask.

And so, we’re able to bring this sort of relatively deep expertise in personal usage across a lot of different kinds of skill sets to thinking about questions of onboarding people, helping them understand what it feels like to have a social interaction inside of an immersive 3D space instead of in some boxes on a screen. And helping clients from startups to the large tech companies, you might imagine, think through these sort of secondary consequence questions of the products that they’re developing.

TK: Can you talk to us a little bit about the actual software development process for XR? Not having actually done any XR Dev myself, how does it work? What languages are you coding in? What frameworks do you use? How does prototyping development work? Give us the rundown.

JF: Sure. There’s actually quite a few approaches that I found to be pretty viable here. Again, it depends a little bit on who I’m encouraging to build. Or if I’m building myself. Or with the engineering organizations of my clients. I think the sort of typical place you might think to start is game engines. Both Unreal and Unity have good support for XR platforms. And there’s actually an open standard called Open XR that you can build from that’s supported by meta devices. And the Vive and sort of all the usual suspects.

That can be really helpful if you’re working with somebody who is already kind of familiar with one of those game engines as many people are. I would say the biggest barrier there is setup. There’s many, many, many YouTube tutorials. And actually a big effort at the beginning of the learning group was to build out a specific tutorial that works on a Mac, because we’re a Mac shop, to get you from Unreal Engine onto your Meta Quest headset. And I think that was worth it. We build a lot of nice things off of that workflow.

If we’re talking about Apple, very well integrated in their ecosystem. If you are already an Apple – especially an Apple mobile developer, it’s sort of almost trivially easy to engage with like both a Vision Pro or just simply an on-phone augmented reality experience. And this is actually the first place that I would point out that you don’t need to have really almost any coding expertise to still engage in these kinds of ecosystems.

There’s an app called Reality Composer that works on any iOS device that will let you compose scenes, assign behaviors to different parts of the scenes. Not super complicated ones. But like tap behaviors. Or proximity behaviors. And you can easily import assets that have animations already baked into them that you’ve perhaps built in programs like Blender, or even purchased from an online service, or downloaded from Apple’s own pretty robust library. And that lets almost anybody quite easily sketch-out what a scene or an interaction might feel like in XR, which can then actually be smoothly passed over to a developer and imported it into Xcode. And that process is actually not completely foolproof. But you’re not going to lose all the effort that that original person put in. That’s kind of game engines and native stuff.

There’s also actually a relatively good sort of web platform called WebXR that has good support from platforms like Three.js. Or Frame, which is actually Three.js under the hood anyway. But that’s simple to create. Like, low complexity environments, I would say, with complicated behavior that are connected to complex backends, which is often kind of my sweet spot. I tend to build web experiences most often.

I’ve also gotten a lot of excitement out of combining a lot of large language model or image generator services into XR environments, which is super easy to do in a web app. Because all the infrastructure is already there to take advantage of combining these other services into one thing.

TK: You beat me to it. The very next question that I was going to ask was about AI applications within the context of XR. When you think about the confluence of LLMs, AI, and XR, I bet there are some pretty useful and interesting applications. Can you give us a couple examples of how you’re able to pull in and harmoniously leverage these technologies together?

JF: Yeah. I mean I think the biggest and most obvious one is voice control. That is the sort of – at least sort of the support so far inside of VR headsets, especially for interaction, is very much game-like. You have sort of controllers that feel a lot like a separated game controller. You have buttons and joysticks. And that, first of all, just isn’t very intuitive or comfortable to a lot of people. And I think you know once you put on a headset, you start to feel the Star Trek Holodeck dream and you want to talk to the computer.

And while I am a little bit skeptical of voice control as a panacea for user interface problems, I do think it’s a really, really interesting thing to add to the mix. And it’s now almost trivially easy to do so, especially in a web platform. Obviously, there are blockbuster things like Whisper that are relatively easy to access. There’s also even locally-run models. Like, Picovoice comes to mind there that can run even on a mobile device like a VR headset to good quality and get you reasonable transcription that you can then flow into sort of actions within your overall environment.

And so, I find that to be like the sort of like no question. We should be engaging with this. We should be thinking through how these interactions might happen. And then actually building them. I think to one of my points earlier where sometimes you need to prototype to answer a question, I think a big question in UX these days is what does voice really mean for my app? And I think, these days, it’s easy to do that in any context. But maybe a little bit surprisingly, it’s also easy to do that inside of XR apps.

TK: You’ve mentioned a few different kinds of hardware, frameworks, and companies that are helping to move XR forward. What are some of your favorite organizations that are doing interesting things with XR today?

JF: Yeah. Great question. This is not a sort of commercial company or not a sort of mass consumer company. But I think one of the most interesting explorations in XR is this company called aNUma. They started by designing sort of transcendental experiences for groups of people to have inside of headsets. Sort of group-guided meditation.

But meditation in a really interesting way. Because instead of sort of imagining energy flowing through you, you can simply experience energy flowing through because you’ve replaced all of your visual and auditory inputs. And they started with sort of a more general application. And now they are focused on helping people through end-of-life experiences and letting people be together in these meaningful ways with their loved ones when they might not be physically able to otherwise. Obviously, that’s only relevant to a small portion of people. But I think that is the most powerful experience I’ve had the privilege of engaging with in XR.

I think beyond that, there are a lot of the sort of more artistic or more emotionally-driven XR experiences are really interesting to me. You retain a lot more from an experience that completely replaces your senses than you do from almost any other kind of experience. Combining that with something of high emotion is really powerful.

There’s the artist I already mentioned, Grace Boyle. She has mostly site-specific pieces. You’d have to like see what she’s up to and where it’s being shown at any particular time. There’s another artist named Wyatt Roy who does a lot with reality capture both for memory and to sort of explore the living places and the working places of himself and of other artists that I find really compelling.

And then, yeah, I think I’ll shout out two slightly older applications. One is called Notes on Blindness. This was actually I think from originally back from maybe 2017 or even earlier, which explores the diaries of man who went blind in adulthood and sort of visualizes what it feels like to slowly lose your sight in interesting ways. And then, finally, an experience called Goliath, which tries to create empathy for the experience of having schizophrenia. Yeah. Sort of emotionally impactful subject matter. But I think conveyed in really, really interesting and compelling ways.

TK: Wow. All of those are really wonderful applications of XR. And I’m sure a lot of folks, myself included, have never thought about applying XR to empathize with those that are losing their sight or to be with their loved ones towards end-of-life.

Given that XR has so many broad applications, on a more practical level, what advice would you have for companies that are looking to gain XR competency and integrate XR or MR into their own product offerings?

JF: Yeah. I would say to really consider what having more context about the space around as if you’re exploring AR or MR, mixed reality, and which I think most people would probably be looking in that direction. If you’re no longer constrained to your black rectangle, what is it about that environment that you would want to really integrate with? Because, ultimately, I think the question of designing for XR is not that different from creating architectural designs, which is part of why I’m so excited about this change in the world. But it does require more ambient thinking and more sort of consideration of what might be around the person when they’re engaging with your product.

And so, I think making mockups like in real-life. Use some paper and tape and try to put interface elements physically around a person and see how that feels. See in what ways would you want to sort of touch and interact. And, also, what do you need to know about? Do you need to know – there’s a lamp sitting on the desk in front of me right now. Is it useful to your service to know that there’s a lamp there? What would you want to do with that lamp? Is it reasonable to collect information about it? Could you connect to it? There become all of these sort of spidery ecosystem questions that arise when you’re thinking about integrating with the world. And, also, frustrations that might naturally happen for people.

There’s not a good definition of sort of like what’s in scope and what’s out of scope for these interactions. And so, defining a couple of really high-value ones and then figuring out how to communicate them to your users I think make for much more satisfying XR experiences than just putting a hat on something. Or sort of like just taking your buttons and putting them into space in front of you. Not that that can’t be fun. But I think if you actually want to invest in this space, thinking about that holistic understanding of your surroundings is really important.

TK: For me, the image of just putting a hat on something brings to mind that wonderful scene from the show Silicon Valley where that company is trying to poach Richard with their super-secret XR mustache technology. Definitely worth the watch if you haven’t seen it.

But, Jenna, I just want to say thank you so much for taking time to talk to us about your experience at IDEO and your work with XR. If folks are interested in learning more about the work that you and IDEO do, what’s a good way to connect?

JF: Sure. I mean, first of all, our website is ideo.com. We have a contact forum. You’re certainly welcome to reach out there. I think the most – the easiest way to reach me and my colleagues who are interested in emerging technology is to email ai@ideo.com. We have of course chosen the very shortest possible email for that one.

You will be able to talk to us about more things than just large language models and image generators. But we’re happy to talk about those as well.

TK: Awesome. Well, Jenna, thank you so much for your time and for coming on Software Engineering Daily.

JF: Thanks for having me, Tyson.

The post Product Design using AR and XR with Jenna Fizel appeared first on Software Engineering Daily.

Almost every application or system involves some sort of user onboarding. Increasingly, companies must implement know-your-customer and know-your-business compliance, or KYC and KYB, as part of that process. In addition, they often handle personal identifiable information, or PII.

Footprint is a developer platform that was co-founded by Alex Grinman for handling identity, security, fraud, and authentication.

Alex joins the show to talk about identity verification, security, compliance, Footprint’s frontend and backend design, and much more.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Implementing KYC and User Verification with Alex Grinman appeared first on Software Engineering Daily.

Vue is a popular JavaScript frontend framework, and Nuxt is an open source meta-framework on top of Vue.

Anthony Fu is a Framework Developer on the Nuxt team. He joins the show to talk about Vue, Nuxt, open source development, and more.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Nuxt JS with Anthony Fu appeared first on Software Engineering Daily.

The size of ML models is growing into the many billions of parameters. This poses a challenge for running inference on non-dedicated hardware like phones and laptops.

Argmax is a startup focused on developing methods to run large models on commodity hardware. A key observation behind their strategy is that the largest models are getting larger, but the smallest models that are commercially relevant are getting smaller. The company was started in 2023 and has raised money from General Catalyst and other industry leaders.

Atila Orhon is the founder of Argmax and he previously worked at Apple and NVIDIA. He joins the show to talk about working in computer vision, building ML tooling at Apple, optimizing ML models, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Scaling Large ML Models to Small Devices with Atila Orhon appeared first on Software Engineering Daily.

Anaconda is a popular platform for data science, machine learning, and AI. It provides trusted repositories of Python and R packages and has over 35 million users worldwide.

Rob Futrick is the CTO at Anaconda, and he joins the show to talk about the platform, the concept of an OS for AI, and more.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Anaconda and Accelerating AI Development with Rob Futrick appeared first on Software Engineering Daily.

Panic has created games such as Firewatch and Untitled Goose Game. They recently ventured into gaming hardware with the Playdate. The console is unique for its inputs, which include a hand crank, and because Panic provides a free SDK, so anyone can develop games for it.

James Moore is a DevOps Engineer and Dave Hayden is an Engineer at Panic. They join the show to talk about developing the Playdate handheld.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Engineering the Playdate Gaming Handheld with James Moore and Dave Hayden appeared first on Software Engineering Daily.

Machine learning models learn patterns and relationships from data to make predictions or decisions. The quality of the data influences how well these models can represent and generalize from the data.

Nihit Desai is the Co-founder and CTO at Refuel.ai. The company is using LLMs for tasks such as data labeling, cleaning, and enrichment. He joins the show to talk about the platform, and how to manage data in the current AI era.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Using LLMs for Training Data Preparation with Nihit Desai appeared first on Software Engineering Daily.

Simplifying Application Development Security

Security is an integral and often challenging component in application development. Developers focus on the functionality of the frontend and backend, but they may be unaware of potential security design flaws in their implementations and APIs. OAuth and OpenID Connect (OIDC) provide a standardized security framework for protecting applications.

Authlete is a service solution that streamlines development of security components. Software Engineering Daily’s Gregor Vand recently spoke with the Principal Architect at Authlete, Justin Richer, to discuss common authorization, identity, and security challenges, and the solution landscape. You can find their full conversation here.

The Gap Between Developers and Security Professionals

Developers’ interest, and often their time, is focused on adding functionality to applications. The security of these systems can sometimes seem obtuse, since security is a specialty that many are not totally familiar or comfortable with. As a consequence, design problems often emerge as applications are built.

“Sometimes, the security team shows up and says, you can’t connect everything together because that’s not secure,” explained Authlete’s Justin Richer. Security professionals often start with the desire to lock the system down and protect confidential information. Consequently, security gets in the way of the application developer, who wants to provide functionality to connect data, people, and processes. The functionality and security viewpoints may possibly clash because their goals are different.

Security Systems Have to be Collaborative

As a result, a fresh outlook and approach are needed. As Gregor noted, “the juxtaposition of collaboration and security does not initially sound like they can coexist, but they have to.”

Humans use the security solutions, and often how people interface with these products is not in the way that vendors designed or even considered. So, enterprises must create security checks that keep information secure but do not prevent people from completing their work, and this is often a difficult balancing act.

What are OAuth and OpenID Connect?

Two key building blocks of modern application security are OAuth, from the IETF, and OpenID Connect, from the OpenID Foundation. Each of these standards plays a key role in ensuring security, but the specifications can be challenging for developers to work with.

While often thought of as an authorization protocol, OAuth is fundamentally a delegation protocol. “OAuth excels whenever you have multiple systems and need to provide an abstract security layer,” noted Authlete’s Justin Richer. In applying a delegation protocol, a user possesses a set of system rights that perform certain tasks, such as the ability to access a record or update certain metadata. Users delegate specific subsets of their rights to applications that perform them on their behalf. How does OAuth accomplish this? OAuth creates an artifact called an access token that encompasses not only the end user’s rights but also the identity of the delegated software.

Here are the steps in the delegation process:

The OAuth delegation process. Image from OAuth 2 In Action by Justin Richer and Antonio Sanso. Manning Publications 2017. Used with permission.A user, known as the Resource Owner (RO) in OAuth terminology, has access to an API, known as the Protected Resource hosted on the Resource Server (RS). The RO wants a piece of software, known as the Client, to act on their behalf at the RS. The RO can authenticate to the system to prove who they are and that they have access, but we don’t want the RO to give those credentials to the Client. OAuth provides an Authorization Server (AS) to facilitate the RO giving the client delegated access, through the use of an access token. The access token represents the combination of all the parties in the OAuth protocol and the access being given, and it can be used by the Client to call the RS.

The OAuth standard simplifies typical API integration work by allowing the Client and RS to never deal with the user’s credentials, and it allows the user to be very deliberate about what kinds of access they want to give the Client application.

The second component, OpenID Connect, works with OAuth. OAuth enables users to securely delegate certain rights to pieces of software, but OAuth does not tell the software who the user is. OpenID Connect works by allowing the user to delegate their own identity information to the client software from the AS, which becomes an Identity Provider (IdP) in OIDC. The IdP provides information about who the user is alongside the access token from OAuth. “Let’s say an app needs all of the extended profile information, such as address, email, and other personal data. A developer writes an API call to do that. That API call is protected by an OAuth access token and it’s the same OAuth access token that I got when I made the call that said who is the user. That access token could also be used for many other functions: getting their calendar information, looking at their bank account history, checking their health record, or any other type of transaction. All these features are possible, but developers need to create them in order to use them.”

A Hosted Security Solution

When building out a protected system, developers need to have dedicated components like the AS to provide key security functionalities. Building an OAuth AS and an OIDC IdP is a formidable task, but many deployments need a level of flexibility and power that only building can provide. But how could you go about building such an important security component?

Authlete is a unique product because it’s a hosted service that provides OAuth and OpenID Connect support functions. What is different from other services is that Authlete does not host the entire platform that handles user authentication, accounts, or attributes. Instead, it supplies support for the OAuth and OpenID Connect protocol pieces where and when they are needed.

“Authlete really shines in the space where developers want to build out something custom and have it run in their environment and programming language,” explained Authlete’s Richer. A developer needs to build an IdP or AS. A request comes in to the developer’s implementation, say a token endpoint request or an authorization endpoint request. The software then calls the Authlete service, which examines the request: Is this request formed correctly? Is everything formatted properly? Which extensions are valid? What should the response be?

Authlete provides developers with an SDK that features an OAuth library they can call and use as needed, but instead of being a static library, “the solution is backed by a live service,” said Authlete’s Richer. Authlete stores the protocol state, which allows it to manage data over time. It handles tokens, keys, and other information so the application developer doesn’t need to account for these directly. Authlete also provides libraries to ease integration, “our libraries turn around and they call the right endpoint with a properly formatted REST request. The API itself is really straightforward.” Some of Authlete’s largest customers have been able to take advantage of this by integrating directly with the API without needing to use an SDK, which wasn’t available in their preferred development platform.

In essence, the developer builds their own authorization server. They provide the application rights and privileges and the OAuth and OpenID parts. Therefore, they concentrate on shipping the features of their application or service. The Authlete service provides only the security layers that matter to their applications.

As a result, instead of taking an off-the-shelf software project and building their own security system on top of it, developers have a world-class system on which they build out their own customized functionality. Nor do developers need to give over all control to a fully hosted service. With the Authlete product, customers retain the control and flexibility that a custom solution provides but don’t have to become experts in the details of the interoperability layer that OAuth and OpenID provide.

Prioritizing Security and Data Control

The solution has found adoption in multiple industries, with particular use in the banking industry. “In the financial sector, Authlete gives enterprises the ability to hold all of the sensitive information close to themselves,” explained Richer. “We don’t store user accounts. We don’t store names and passwords and addresses and account numbers and all of that other stuff. Instead, they tell us, I need an access token for the following subject identifier with the following rights and we deliver it.” Privacy-driven regulations and demand for privacy protections from consumers is changing how companies architect and deploy their systems, especially with regard to hosted services. Letting go of personal data to a cloud service is not a choice to be taken lightly, especially in highly regulated verticals. Authlete’s semi-hosted service architecture allows its customers to retain control over sensitive financial data and provide service that is hosted and controlled by the customer’s systems, while at the same time providing a state-of-the-art implementation of security protocols to protect these systems.

Security is an important but difficult aspect of modern systems, and deploying security protocols like OAuth and OIDC brings its own benefits and challenges. At the end of the day, developers want to provide the best functionality for their users, and Authlete provides a compelling and unique approach to securing that functionality.

Check out https://events.authlete.com/sed to try out Authlete for an extended 90-day free trial.

Full Disclosure: Authlete is a sponsor of Software Engineering Daily

The post Making OAuth and OIDC Accessible to Developers appeared first on Software Engineering Daily.

Kentik is a network observability platform that focuses on letting users easily ask questions and get answers about their network.

Avi Freedman is the CEO of Kentik and he joins the podcast to talk about the platform, his observability philosophy, the role of AI in observability, and much more.

Full Disclosure: This episode is sponsored by 10K Media (Kentik).**

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post AI-Driven Observability at Kentik with Avi Freedman appeared first on Software Engineering Daily.

Solid.js is a popular JavaScript framework known for its reactive and efficient rendering system. Instead of using a Virtual DOM, it compiles its templates to real DOM nodes and updates them with fine-grained reactions.

Ryan Carniato is the creator of SolidJS, and he joins the show to talk about the framework.

Taylor Nodell is a software engineer specializing in front end web technologies. He’s particularly passionate about accessibility and how the internet is changing our relationships to each other. He spends his weekends hiking in the Australian wilderness and making heavy metal.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post SolidJS with Ryan Carniato appeared first on Software Engineering Daily.

Sammy Steele is a Senior Staff Engineer at Figma, and the tech lead for their databases team. She previously worked at Dropbox, where she built out their petabyte-scale metadata storage and search systems.

Sammy recently published a blog called “How Figma’s databases team lived to tell the scale”. The blog went viral and made it to the top of Hacker News. We invited Sammy on the podcast to learn more, and she is our guest today.

Check out Figma Config at https://config.figma.com/.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Database Scaling at Figma with Sammy Steele appeared first on Software Engineering Daily.

Software development has historically been a time-intensive and often tedious process that requires tooling and configuration. Generative AI is changing that process, and offers software engineers the ability to dramatically streamline the application development process, improve code quality, and deliver more functionality. To reap such benefits, they need to put the right guardrails in place.

According to the 2023 State of AI in Software Development report, code creation accounts for only 25% of a developer’s time. The rest is spent with prep work like setting up a development environment, collecting the necessary tools and libraries, establishing version control, and accounting for security issues.

A lot of this work is repetitive and follows set patterns. Humans tend to become distracted and sometimes make mistakes, leaving holes in the process. Generative AI can enhance the workflow in many ways. Integrating AI into software engineering creates more robust, more secure, higher quality software and does so much more quickly than traditional development. For instance, Generative AI software can examine a failed build, assess what went wrong, and provide possible solutions, reducing remediation time.

Establish Guardrails for AIAI is very helpful, but, like humans, imperfect and can overlook key items like security vulnerabilities. Such solutions can automatically build tests but this is not ideal for code that’s already created. Generative AI only tests what it wrote and does not perform sophisticated analysis, leaving the enhancement open to potential problems.

Therefore, dev teams must put checks in place, so they catch potential vulnerabilities before they make their way into production applications. Organizations need to scope out project parameters and then establish rules, best practices, and guardrails to mitigate risk and meet compliance requirements. This step is complicated and requires input from legal, compliance, and DevSecOps teams. Many companies are engaging in this area for the first time. As a result, they may need help. GitLab and its AI Transparency Center recently released valuable resources on building a transparency-first AI strategy.

After companies understand the potential risks, they need to talk with their AI provider and understand how the solution works. What AI models does it use? What data do the AI modules interact with? Which vector databases does the application access? How large are the language models (LLMs) that are being trained? How do they function? That analysis provides them with a good foundation for understanding where potential security holes may arise.

Another best practice is limiting how many distinct AI tools will be used throughout the software development lifecycle across the organization. The more tools in use; the more complexity introduced, potentially causing security risks, operational issues, and oversight challenges. The more numerous the tools, the greater the overhead. The more numerous the solutions, the more difficult it becomes to centrally manage what is occurring. The more numerous the tools, the more training that the tech staff needs.

Put Metrics in Place Sometimes, enhanced does not mean better. To truly understand AI’s impact, dev teams need to establish baselines and then measure areas like productivity. Typically, organizations would examine how quickly they move code into production, the four DORA metrics, or the time it takes to remediate bugs.

Those items provide snapshots and not a complete picture. A better option is building out standard workflow measurements inside groups and projects. As a result, metrics from teams to business units roll up automatically and managers analyze the outputs continuously.

Buy or Build? However, software engineers do not want to build such AI monitoring tools themselves. GitLab created an expanding AI DevSecOps platform and toolbox. It includes powerful generative AI models and cutting-edge technologies from hypercloud vendors. GitLab Duo delivers a range of features, like code assistants, conversational chat assistants, and a vulnerability explainer.

The solution’s benefits extend throughout the software development lifecycle.

Explain Code in Natural Language QA testers can use Code Explanation to quickly and easily understand code. For instance, if an MR includes code written in Rust and a complex set of methods, a QA tester can highlight the methods and deliver a natural language readout of what the change is trying to do. This feature enables a QA tester to write test cases more efficiently.

Write Merge Request Descriptions GitLab Duo automates the creation of comprehensive descriptions for merge requests and quickly and accurately captures the essence of an MR’s string of commits. The tool also identifies surface tasks that are missing.

Root Cause Analysis of Pipeline ErrorsIf something breaks, troubleshooting can be difficult. GitLab Duo identifies a possible root cause and a recommended action that can be copied and pasted directly back into a CI job.

Vulnerability ResolutionIn the rush to shift security left, engineering teams have had to quickly become security experts. Issues can arise that they’re not familiar with. With generative AI, engineers can access Duo Chat to learn what a vulnerability is, where it is in the code, and even open an automated MR with a possible fix. All of these actions occur within the development window, so no context-switching is needed, saving software engineers time.

Increase Security and ProductivityBy using a tool, like GitDuo, businesses increase software delivery velocity. They lower the time required to resolve vulnerabilities and validate merge requests and have the right reviewers and the right tests. So, the code review time diminishes, and quality increases.

They also gain visibility. Software engineers view each stage, including dependencies, and the delta it takes the development team to get through those stages. Dashboards illustrate what that speed looks like, so they can easily pivot, if needed. In essence, they also have a better handle on whether or not to release software into production.

When used consistently across the software development lifecycle, GitLab Duo can drive a 10x faster cycle time, helping organizations do more with less and allowing employees to spend their time on high value tasks.

The “Omdia Market Radar: AI-Assisted Software Development, 2023–24″ report highlighted GitLab Duo as one of the products the analyst firm considers “suitable for enterprise-grade application development,” noting that its “AI assistance is integrated throughout the SDLC pipeline”.

Software development moves faster and faster. DevSecOps teams sometimes have trouble keeping pace. Generative AI has the potential to automate different pieces of the development cycle. However, businesses need tools to ensure that unintended consequences don’t occur when processes are automated. GitLab Duo offers them a platform that lets them reap Generative AI’s potential benefits and avoid its pitfalls.

Full Disclosure: This blog post is sponsored by GitLab.

The post Weaving Generative AI into DevSecOps appeared first on Software Engineering Daily.

Autonomous vehicle engineering is a huge challenge and requires the integration of many different technologies. A self-driving car needs data from multiple sensors, ML models to process that data, engineering to couple software and mechanical systems, and much more.

Ian Williams is a Senior Staff Software Engineer at Cruise, and before that worked at Google, Lyft, and eBay. He joins the show to talk about the diverse engineering challenges and strategies associated with building self-driving cars.

This episode is hosted by Tyson Kunovsky. Tyson is the co-founder and CEO of AutoCloud, an infrastructure as code platform. He is originally from South Africa, and has a background in software engineering and cloud development. When he’s not busy designing new GitOps workflows, he enjoys skiing, riding motorcycles, and reading sci-fi books. Check the show notes for more information on Tyson’s work, and where to find him.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Bonus Episode: How to Build a Self-Driving Car with Ian Williams appeared first on Software Engineering Daily.

Ben Huber is a security engineer who has worked at companies including Crypto.com and Blackpanda. He joins the podcast to talk about his career, penetration or “pen” testing, attack vectors, security tools, and much more.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Security Engineering with Ben Huber appeared first on Software Engineering Daily.

George Mathew is a Managing Director at Insight Partners where he invested in Weights & Biases, Jasper, and others.

He has over 20 years of experience developing high-growth technology startups including most recently being CEO of Kespry.

George joins the podcast to talk about his path to becoming an investor, his data-first thesis about investment, the AI business landscape, his book recommendations, and more.

Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Startup Investing with George Mathew appeared first on Software Engineering Daily.

Event-driven architecture is a software design pattern where system components communicate through events that are generated by producers, and pushed to consumers. This design is often contrasted with a request-driven architecture, where components communicate with each other by sending requests and receiving responses.

Hookdeck is an event gateway for receiving, processing, and delivering asynchronous messages. It centralizes and streamlines communication between services, like a 3rd party API such as Shopify or Stripe, and internal endpoints or other APIs.

Alex Bouchard is the Co-founder of Hookdeck. He joins the podcast to talk about event-driven architecture, building event bridges, expanding Hookdeck beyond webhooks, and much more.

Full Disclosure: This episode is sponsored by Hookdeck.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Hookdeck and Building an Event Gateway with Alex Bouchard appeared first on Software Engineering Daily.

OAuth is an open standard for access delegation. It lets users grant websites or applications access to their information on other websites, but without giving away passwords.

OpenID Connect is an identity layer on top of OAuth. Even if you haven’t programmed using OAuth and OpenID Connect, you’ve certainly used them for authentication on Google, Facebook, Spotify, and countless other services.

Authlete is a service that provides a set of APIs to implement OAuth authorization servers, and OpenID Connect identity providers.

Justin Richer is the Principal Architect at Authlete and is part of the working group that developed OAuth 2.0. He joins the podcast to talk about the history of OAuth, OAuth as a delegation protocol, the Authlete API, and much more.

Check out https://events.authlete.com/sed to try out Authlete for an extended 90-day free trial.

Full Disclosure: This episode is sponsored by Authlete.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Authlete and Making OAuth Accessible with Justin Richer appeared first on Software Engineering Daily.

Today, you can access Netflix on virtually any device. For a Netflix user, this seamless experience can be easy to take for granted, but it requires an enormous engineering effort.

Jay Phelps is a Senior Software Engineer at Netflix where he works on Shared Client Foundations. He joins the show to talk about the start of his career, his work at Netflix, and much more.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Netflix Engineering with Jay Phelps appeared first on Software Engineering Daily.

Developing a modern video game can be a big technical and creative challenge. Game development involves bringing together skills in software engineering, game design, art and sound. Frequently, this is done in a collaborative setting which requires strong teamwork.

Software Engineering Daily has been expanding our coverage of game development to better understand how games are made. Joe Nash recently sat down with developers from Mintrocket, the team that developed the highly successful Dave the Diver in 2023.

Jaeho Hwang is the Game Director, and Minsuk Cha is the System Programmer, for Dave the Diver. They joined us to talk about engineering the myriad game systems, the choice of Unity as an engine, how they organized the dev team, and more.

Joe Nash: One of the first things the player notices about Dave the Diver is the game combines “traditional” 2D pixel art elements with low poly 3D models with smoother textures. For example Dave’s character is pixel art but the ocean is built of 3D models. What was the motivation to use this unique visual style?

Jaeho Hwang: We had a vision to create a game that brings the ocean and its unexplored creatures to life. Drawing on our team’s expertise in 2D game development, we thought it would be fun to play spearfishing as a 2D arcade action game; the idea took shape, and a prototype was born. Yet, we swiftly realized that the prototype lacked the necessary depth and spatial realism to truly embody the essence of the ocean. To address this shortfall, we decided to transition the background to 3D—a transition that, fortunately, proved less challenging than expected. However, the most formidable task lay in portraying the colossal fish in a natural manner within a 2D setting. Consequently, the decision was made to render these creatures in 3D; that being said, there was a creation of a 3D shark and a 2D character, Dave, locked in battles. Thankfully, after several research and development efforts, we were able to create them more naturally.

Joe Nash: The game has many different systems like fishing, cell phone apps, restaurant management, card collection (Marinca), and a rhythm game. Using this many systems seems like a daunting design challenge. What was it like integrating the different systems together?

Jaeho Hwang: Those systems are enjoyable on their own, but incorporating them into a cohesive narrative is a different story. To tackle this, we used characters. Characters were designed to seamlessly guide players by organically intertwining through the game systems within the game’s storyline. Some systems are necessary for user engagement, but we presented them as if they were essential in a storyline so that players could learn the systems naturally. This narrative-driven approach worked well and facilitated a seamless introduction to the game’s complexities.

Joe Nash: In the early stages of development, was there one system you started with?

Jaeho Hwang: Our initial focus was on implementing the harpooning mechanic and the tactile sensation of reeling in fish hunting. Recognizing that it’s a top priority to make this enjoyable, as it’s the core behavior that players will do, we dedicated considerable effort to perfecting the player experience in this aspect. Next, we worked on running a sushi bar after fishing. Bancho Sushi was originally intended to be automated once the menu was set up. However, in order to make the gameplay feel less loose, we later reworked this component to allow manual control of the characters and only run the sushi bar at night.

Joe Nash: Many of these systems build upon one another to significantly expand the scope of the game beyond the player’s initial expectations. The game starts out conceptually simple with catching fish to run a restaurant. It eventually introduces farming, bosses, and deep sea archaeology. How did you think about arranging the sequence of these elements?

Jaeho Hwang: We wanted to create a story that ultimately tackles significant challenges, but starts from small, everyday beginnings, in the same way that disaster movie narratives begin with humble beginnings. Mirroring this narrative arc, we considered designing the game’s systems to expand in a similar way, starting with fishing, then growing ingredients on the farm to serve a variety of customers, or crafting devices to explore the deep sea. As a result, players were able to use and interact with many systems in their gameplay experience.

Joe Nash: Were there any game systems you experimented with that didn’t make it into the final game?

Jaeho Hwang: There is a social media feature within the game, and we thought about making the number of likes (represented by “hearts” in the game) correspond to actual players. Currently, liking a post only affects the player’s local setting. Although there was even a notion of enabling players to upload the photos by themselves, we refrained due to the server-related complications and administrative overhead. However, this still remains a point of regret for our team.

Joe Nash: Given all these different systems, how did you organize the development team?

Jaeho Hwang: Uniquely, we use a specialized ‘area-focused’ approach. In this setup, we assign individual programmers for specific areas, such as the sushi bar, the marine section, and the farm area. Of course, we also do have plenty of collaborative tasks. The benefits of this are that it ensures clarity on who to consult for specific content and allows for the seamless integration of different content within each area in a responsible manner.

Joe Nash: What was the biggest challenge in developing the game?

Minsuk Cha: In game development, every aspect presents its own set of challenges. Particularly in my experience, a significant difficulty lay in establishing a solid technical codebase while efficiently meeting design requirements, identifying the fun factor promptly, and refining the game to perfection. Our team of programmers, myself included, focused on enhancing productivity by ensuring the modularity of each feature. This approach allowed for easy integration into different aspects of the game and increased the overall reusability of our work. We believed that maximizing these efficiencies was vital for a small team to generate a substantial amount of content. Admittedly, this strategy demanded meticulous attention to the fundamental software design of each feature, a task that required considerable dedication. Despite the complexities involved, our efforts proved fruitful as we successfully delivered a diverse range of content to the players.

Joe Nash: The game uses the Unity Engine. How did the team decide on Unity?

Minsuk Cha: The selection of a suitable game engine is pivotal in the initial development phase. In my view, crucial considerations encompass the expertise of the development team, the project’s timeline, and how well the engine’s features support the game’s overarching direction. Unity emerged as the optimal choice for us, ticking all the necessary boxes thanks to its versatility for multiplatform development. Furthermore, our established relationship with Unity and access to comprehensive technical support further solidified our decision. Throughout the development process, the robust technical assistance provided by Unity played a pivotal role in resolving numerous challenges that cropped up, contributing significantly to the successful completion of our project.

Joe Nash: One of the big advantages of Unity is that it’s cross-platform. However, were there any unexpected challenges when porting the game to multiple systems?

Minsuk Cha: When we took on the challenge of adapting our game for the Nintendo Switch platform, we faced various challenges. A striking memory that stands out is the considerable delay experienced in loading the initial title screen during the game’s premiere run on the Nintendo Switch console. Originating as a PC title distributed through Steam, Dave the Diver had initially exhibited seamless performance on systems meeting minimum requirements, alleviating concerns regarding performance during the porting endeavor. Despite this assurance, as we delved deeper into the porting process, we encountered a slew of issues ranging from frequent application crashes triggered by memory constraints to the failure of meeting our designated frame rate targets, not to mention prolonged loading intervals during scene transitions. To counter these challenges, extensive modifications were imperative throughout various facets of the game. Harnessing the capabilities of the Unity Profiler tool, we pinpointed performance bottlenecks and proceeded to address them methodically, necessitating a comprehensive review and enhancement of the content being developed by each team member to fine-tune performance levels.

Joe Nash: Dave the Diver also uses a tool called Cinemachine. Can you talk about what it does and why the team used it?

Minsuk Cha: In game development, cutscenes play a crucial role in shaping the narrative and elevating the overall entertainment quotient of the gameplay experience. Cutscenes are typically designed within a timeline using Cinemachine, which enables developers to design distinct scenes and effects autonomously. In our game development process, we heavily relied on Cinemachine due to its versatility. This approach allows our art team to concentrate on the creative aspects of the game, ensuring a captivating visual experience for players. Moreover, Cinemachine seamlessly integrates with the main camera, facilitating a smooth transition between cutscenes and in-game action, thereby maintaining a coherent and immersive gameplay experience.

Joe Nash: The game has impressive fish swarming behavior. What was challenging about developing that system?

Minsuk Cha: My fellow programmer Yongsun Lee, who spearheaded the development of the fish swarming behavior, articulates, “There are many different obstacles like a wall. Integrating the swarming system with the pathfinding system posed a significant technical hurdle, compounded by the need for optimizations to ensure smooth performance on the Nintendo Switch platform.” To address this, we came up with a solution where we treated each group of entities as a cohesive unit that shared a single pathfinding. This streamlined the process, requiring only a single pathfinding request per group, with the job system handling the determination of each fish’s position within the group. We opted for a more delayed response approach to maintain efficiency, prioritizing CPU load minimization as the fish entities learned and interacted within the game environment. However, this strategic implementation allowed us to strike a balance between performance and functionality, enhancing the gameplay experience on the Nintendo Switch.

Joe Nash: Can you tell us about the C# Job System and Burst compiler and how were they used for fish swarming?

Minsuk Cha: The Job System was introduced by Unity to allow developers to work within a multi-threaded environment that significantly improves performance, especially when paired with the Burst compiler. In the Unity ecosystem, where using C# threads can be challenging, the Job System maximizes the potential of CPU cores to efficiently handle concurrent tasks such as fish swarming. By taking advantage of this system, developers can extract optimal performance from parallel operations. The Burst Compiler assumes a pivotal role in refining code execution through an array of optimizations. In the context of fish swarming, the job system effectively manages the movement of groups of fish by consolidating pathfinding requests per group while also determining the precise position of each fish within the group. This approach streamlines processing, leveraging the capabilities of the job system to harmonize complex operations and elevate overall performance levels.

Joe Nash: Under the hood, the game uses a scene-loading system designed to organize and load scenes based on the current mission and probability of occurrence. It sounds like the game has a probabilistic model to predict what the player is likely to do next in their playthrough. This is an impressive engineering accomplish given the large number of possibilities later in the game after the player has unlocked a lot of content. Can you talk more about the scene-loading system?

Minsuk Cha: The underwater setup in our game is structured around multiple scenes, each featuring a unique selection of elements in the Bluehole that varies with every dive. The system is built to dynamically load the appropriate scene based on the player’s current gameplay status. This scene selection process is influenced by a myriad of factors, with mission progress standing out as the key determinant. Other considerations, such as game progression, probability of occurrence, and in-game time, are taken into account. These factors are meticulously evaluated, assigned weights, and prioritized to curate an ideal combination of scenes.

Joe Nash: What’s the future plan for Dave the Diver?

Minsuk Cha: Firstly, we are thrilled by the tremendous love our players have shown for the game, which has given us the opportunity to deliver additional content. Following the game’s release, we have continued to improve the quality of life, balance, and optimization, and we are now exploring the development of DLC to further enrich the story. At the same time, we are considering crossover content that we believe will captivate our players. Given the opportunity, we aspire to create games in varied genres featuring our characters before moving on to our next project.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

The post Developing Dave the Diver with Jaeho Hwang and Minsuk Cha appeared first on Software Engineering Daily.

Snowflake is one of the most prominent platforms for interacting with data and building data-intensive applications.

Dan Myers works in Developer Relations at Snowflake and he joins the show to talk about the future of application development, and building native data apps on the platform.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Data Applications on Snowflake with Daniel Myers appeared first on Software Engineering Daily.

Java is one of the most widely used programming languages, and a key contributor to its success is VMware Tanzu’s Spring, the most common framework for Java development. The Spring Framework is built on top of the Java Virtual Machine and provides a consistent programming and configuration model for application developers. From inception, it was designed with developer experience and modularity in mind.

The open-source application framework has been accelerating Java development times since its inception in 2004 (Happy 20th birthday). Since then, the platform has been expanding: growing 50% year over year during the last five years. In this blog we’re exploring what makes Spring important to Java, how the framework influenced the developer experience, and a look at the latest version of Spring, which introduces features to support AI integration.

Spring SuccessTwo decades ago, there were dozens of different ways to connect to a database, something just about every application has to do at some point. “At the time, the various approaches were very cumbersome: developers wrote a lot of code and gained very little functionality,” explained Mark Pollack, a Senior Staff Engineer, Tanzu Division, Broadcom. “Spring provided a lot of value by simplifying the process. Just getting a single app to talk to a database, present a web form, and do transactions correctly was a huge win. In that era, developers could spend weeks trying to create that function.”

Another reason for its success is its enterprise-focus. Most open source projects concentrate on the consumer market. However, large companies invest a lot of money building applications to run their businesses. “At the end of the day, large corporations’ largest expense is probably their developers,” explained Ryan Morgan, Senior Director of Engineering, Tanzu Division, Broadcom. Spring makes developer teams more efficient, which greatly enhances the bottom line.

Through the years, the ecosystem has grown. “There’s a large and vibrant community behind Spring,” noted Tanzu’s Morgan. Now, it has more than 200 different technology starters. These software building blocks make it simpler for software engineers to integrate their code with different third-party systems.

Get a Great StartDevelopment problems evolve over time, so various elements were added. Spring Initializer is a bootstrapping tool, a way for developers to create a new project. “Normally, software engineers started from a blank piece of paper and had to figure out what type of project it was and what type of libraries were needed,” said Tanzu’s Morgan. Then, they searched the web to find some place in the documentation that told them what library dependency was needed to add for different pieces of functionality. Then invariably, you cut and pasted from something that wasn’t consistent. You ended up with a mess.”

With Spring Initializer, software engineers go to a website which has clear instructions about what the options are and presents them in a typical web form. Then, they enter the Generate command and out comes a shell that they can use to start building their program. The solution does not generate any code but solves the problem of finding the right application dependencies. Developers start faster and are less frustrated than previous methods.

Under development is Spring CLI, which not only creates the shell of the app but also includes code. The advances have a significant impact because 1 million new projects are created each month.

Meet the Need for Development SpeedThe last 10 years have seen a major move to container deployment and Spring has aligned with this paradigm shift. “Really, when you think about all those cloud native patterns, a lot of those container functions are really baked into our projects already,” stated Tanzu’s Morgan. “If you want to do distributed configuration, we have a solution for that. You want to do service discovery; Spring has patterns and tools for that.”

Recently, a major change to Spring occurred. Rather than release new functions autonomously, they are gathered and bundled into Spring Boot. Version 3.0, which is based on Spring Framework 6.0, requires Java 17 or above. Previously, Spring supported Java 8, so the change is significant for some companies.

Better performance is one benefit from the change. “We’ve seen customers realize 15% performance improvements, just from doing the upgrade,” said Tanzu’s Morgan.

Java Supports Generative AIAI is being woven into many applications, especially with the emergence of Generative AI solutions. They represent a quantum leap in capabilities and overall intelligence compared to previous iterations of AI. One reason why today’s generative models are gaining so much attention is that they work with much larger volumes of information (hundreds of billions of words) and larger data models (hundreds of billions of parameters) compared to previous AI systems. They possess impressive and unprecedented power. Consequently, they can perform very sophisticated functions.

However as developers try to take advantage of the functionality, platform diversity again presents development challenges. “OpenAI has their API, Amazon Bedrock offers a different one, and so do other companies,” noted Tanzu’s Pollack.

A guiding focus and design principle in the Spring framework is simplifying such work by providing common abstractions over similar technologies and interfaces. Spring AI is quickly becoming the starting point when Java developers write AI applications. “Spring AI has the common patterns that Spring developers are used to,” noted Tanzu’s Pollack. It can abstract out models, clients, etc. in ways that are familiar to Spring users.”

Another crucial part of AI applications is using a vector database. Spring supports multiple vector databases, and its portable API simplifies changing implementations. So, Spring streamlines AI application development.

Java has been a popular programming language for enterprises for decades. Spring provides software engineers with tools that help them enhance the development process. The framework has reached its 20th year of empowering developers, and its years, engaged community is laying the groundwork for continued expansion in the coming decades. “Maybe one reason why Spring continues to do well is it constantly tries to improve itself and doesn’t just rest on its laurels,” concluded Tanzu’s Pollack.

The post Spring AI and Java in 2024 appeared first on Software Engineering Daily.

Kaze Emanuar is a ROM hacker who’s famous for the array of mods he’s made for Super Mario 64. He’s implemented remarkable optimization to the decompiled game code, even pushing Super Mario 64 to run at 60 frames per second.

Kaze joins the show to talk about his interest in Super Mario 64, ROM hacking, the N64 architecture, coding in assembly, and more. Be sure to check out Kaze’s YouTube channel to see his work, including breakdowns of how he accomplished his optimizations.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Bonus Episode: Optimizing Nintendo 64 Code with Kaze Emanuar appeared first on Software Engineering Daily.

Managing data and access to data is one of the biggest challenges that a company can face. It’s common for data to be siloed into independent sources that are difficult to access in a unified and integrated way.

One approach to solving this problem is to build a layer on top of the heterogenous data sources. This layer can serve as an interface for the data and provide governance and access control.

Cube is a semantic layer between the data source and data applications. Artyom Keydunov is the founder of Cube and he joins the show to talk about the approach Cube is taking.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post A Semantic Layer for Data with Artyom Keydunov appeared first on Software Engineering Daily.

Blasphemous and Blasphemous II are Metroidvania action-adventure games developed by the Spanish studio, The Game Kitchen. The games have a stunning, distinctive pixel art style and atmospheric world which is inspired by Spanish folklore and religious themes. They are known for their challenging combat and intricate level design.

David Erosa is the Lead Producer and Dani Márquez is a Senior Programmer on Blasphemous II. David and Dani join the show today to talk about designing the game systems, the game’s development framework, engineering character movement, console optimizations, and much more.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Watch the video episode here:

The post Developing Blasphemous II with David Erosa and Dani Márquez appeared first on Software Engineering Daily.

Python is famed for it’s intuitive syntax, powerful standard library, and rich ecosystem of packages and frameworks. It’s also dynamically typed which is part of what makes the language so accessible.

Paul Everitt is the Python and Web Developer Advocate at JetBrains. He joins the show to talk about developing the PyCharm IDE, the challenge of static analysis in Python, browser-based IDEs, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Building the PyCharm IDE at JetBrains with Paul Everitt appeared first on Software Engineering Daily.

All robust technology platforms require testing to ensure that features work as intended. In many cases, tests require data, but getting access to valid and high quality test data is a common challenge, especially when the technology runs on sensitive data. Realistically mimicking data that would normally contain sensitive financial or personal information is not easy.

Tonic.ai was started in 2018 to provide developer tools to transform production data into safe testing data. Andrew Colombi is the CTO and Adam Kamor is the Head of Engineering at Tonic. They join the show to talk about creating realistic synthetic data, data de-identification, validating LLM RAG output, Tonic’s subsetting engine, and much more.

Full Disclosure: This episode is sponsored by Tonic.**

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Tonic and Synthetic Data with Andrew Colombi and Adam Kamor appeared first on Software Engineering Daily.

Netlify is a popular hosting platform that provides build, deploy, and serverless backend services for web apps. The platform enables deployment directly from source files stored in a version control system like GitHub.

Erica Pisani is a Senior Software Engineer at Netlify. She joins the show to talk about how she got started at Netlify, edge computing, edge functions, private integrations, and more.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript.

Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Watch the video episode here:

The post Netlify and Edge Computing with Erica Pisani appeared first on Software Engineering Daily.

Language and compiler design are fundamental aspects of computer science. High-level languages are how most developers interact with computers, so it’s hard to overstate the significance of compiler engineering or the aesthetics of language syntax.

C# is a general-purpose high-level language that was created by Anders Hejlsberg at Microsoft in 2000, and was open-sourced in 2014.

Jared Parsons is the Principal Developer Lead on the C# Language Team at Microsoft, where he’s worked for 20 years. He joins the show to talk about how the C# compiler is developed, the compiler as an API, language creation as an art, the experience of open-sourcing C#, and much more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post C# Compiler and Language Design at Microsoft with Jared Parsons appeared first on Software Engineering Daily.

Sphinx Bio develops computational tools to accelerate scientific discovery. The company is focused on addressing the computational data analysis bottleneck by enabling scientists to do the analysis themselves.

Nicholas Larus-Stone is the founder of Sphinx. He joins the show to talk about being a computer scientist at the interface with biology, the data analysis bottleneck in biology, designing a software tool for scientists, their go-to-market strategy, and more.

Nicholas also started Bits in Bio which is a popular community for people building software for science. You can check out their upcoming meetups and hackathons at bitsinbio.org.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Watch the video episode here:

The post Biotech Special: Life Science Data Analysis with Nicholas Larus-Stone appeared first on Software Engineering Daily.

Java is one of the top programming languages used today and Java code is ubiquitous. A key factor to the overall success of Java is the Spring framework, which is the most common framework for Java development.

Spring is an open-source comprehensive application framework on top of the Java Virtual Machine that provides a consistent programming and configuration model.

Ryan Morgan is a Senior Director and Mark Pollack is a Senior Staff Engineer at VMware Tanzu, a division of Broadcom. They join the show to talk about the origins of Spring, the problems it solves, building APIs for AI models in Java, and much more.

Full Disclosure: This episode is sponsored by VMware.**

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post VMware’s Spring AI with Ryan Morgan and Mark Pollack appeared first on Software Engineering Daily.

Convex is a serverless backend platform to simplify fullstack application development. Its underlying database is written in Rust, and it uses TypeScript to integrate with reactive UI frameworks.

The platform is growing, which has presented new reasons to make the code open source, and Convex recently released the source code for a self-managed version of their platform.

The question of whether or not to open source is one that many companies consider. We were curious to explore the decision making landscape around open sourcing, and today are speaking with James Cowling, the Co-Founder and CTO at Convex.

James joins the show to talk about prioritizing developer experience, the choice to open-source, risks of open sourcing, software licenses, and much more.

Full Disclosure: This episode is sponsored by Convex.**

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.
Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Going Open Source at Convex with James Cowling appeared first on Software Engineering Daily.

Proteins are nanomachines inside cells and perform the incredible array of tasks required for cells to function. They are composed of a chain of hundreds to thousands of amino acid “building blocks”. Peptides are similar to proteins, but have only about 3 to 30 amino acids. Their smaller size gives them distinct properties that are useful in therapeutic applications.

Menten AI is using cutting-edge generative methods to engineer new peptide therapeutics, and are backed by Y Combinator, Khosla Ventures, and others.

Patrick Finneran is the Associate Director of Biochemistry at Menten. He joins the show to tell us about the drug development process, handling noisy biological data, building a hybrid team of software engineers and biologists, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Watch the video episode here:

The post Biotech Special: Computational Drug Discovery with Patrick Finneran appeared first on Software Engineering Daily.

GitHub Copilot is an AI tool to assist software developers by autocompleting code. It is no understatement to say it has already transformed how developers write code.

Adrián Mato Gondelle leads the Design team for GitHub Copilot. In this episode, recorded in 2023, Adrián joined the podcast to talk about his work.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Design at GitHub Copilot with Adrián Mato Gondelle appeared first on Software Engineering Daily.

Corbado is an authentication platform that provides APIs for developers to replace passwords with passkeys such as Face ID or Touch ID.

Vincent Delitz is a Co-Founder at Corbado and he joins the show to talk about the platform, the changing authentication landscape, the challenge of session management with passkeys, and more.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Biometric Authentication with Vincent Delitz appeared first on Software Engineering Daily.

The growing use of large datasets and ML in the life sciences has created new demand for data technologies. Snowflake is a cloud-based data warehousing company that provides a platform for storing and analyzing large volumes of data.

Harini Gopalakrishnan is the Field CTO of Life Sciences at Snowflake. She joins the show to talk about data challenges and solutions in biotech.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Watch the video episode here.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Biotech Special: Life Sciences at Snowflake with Harini Gopalakrishnan appeared first on Software Engineering Daily.

Apache Iceberg is an open source high-performance format for huge data tables. Iceberg enables the use of SQL tables for big data, while making it possible for engines like Spark and Hive to safely work with the same tables, at the same time.

Iceberg was started at Netflix by Ryan Blue and Dan Weeks, and was open-sourced and donated to the Apache Software Foundation in November 2018. It has now been adopted at many other companies including Airbnb, Apple, and Lyft.

Ryan Blue joins the podcast to describe the origins of Iceberg, how it works, the problems it solves, collaborating with Apple and others to open-source it, and more.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Iceberg at Netflix and Beyond with Ryan Blue appeared first on Software Engineering Daily.

Godot is a free, open-source game engine that’s growing rapidly in popularity. Ramatak is a new public benefit company founded by Godot engine veterans Hein-Pieter van Braam and Ariel Manzur. The goal of Ramatak is to help make Godot the number one choice for creating, deploying, and monetizing games on iOS and Android.

Hein-Pieter joins the show today to talk about his early career in Linux infrastructure, the shift to working on game engines, and his current work on Ramatak.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Watch the video episode here.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Bringing Godot to Mobile with Hein-Pieter van Braam appeared first on Software Engineering Daily.

NextFlow is a tool for managing scientific computation workflows. It’s increasingly popular for bioinformatics, computational biology, and other life science applications.

Evan Floden is the Co-Founder and CEO of Seqera Labs which develops NextFlow. He joins the show today to talk about his background as a scientist and engineer, the modular design of NextFlow pipelines, the unique challenges of genomic sequence data formats, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Watch the video episode here.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Biotech Special: Scientific Computing Pipelines with Evan Floden appeared first on Software Engineering Daily.

Stately is a web-based drag and drop editor for collaboratively developing code, diagrams, and documentation. Laura Kalbag is the Developer Advocate at Stately and she joins the show today to talk about Stately, state machines, building good documentation, and more.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Stately with Laura Kalbag appeared first on Software Engineering Daily.

The power of 3D graphics hardware and rendering technology is improving at an astonishing pace. To achieve high graphical fidelity, assets that compose 3D worlds must feature an ever-increasing level of detail.

Andrew Price is the founder of Poliigon, which is an asset production studio and store. Andrew also runs the highly popular Blender Guru YouTube channel where he teaches viewers how to use Blender.

Andrew joins the show to talk about how different virtual assets are made, building his company, the impact of AI on graphics production, whether graphics have achieved photorealism, and much more.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Watch the video episode here.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Graphical Photorealism with Andrew Price the Blender Guru appeared first on Software Engineering Daily.

Antibodies are a type of protein molecule produced by the immune system. They recognize and attach to other molecules with remarkable precision. Typically antibodies target foreign objects, like viruses, to mark them for destruction. However, they can also be engineered to treat diseases like cancer, and they are one of the fastest growing classes of drugs.

Recently, AI-driven antibody engineering has taken off, and BigHat Bio is one of the leaders of this revolution.

Eddie Abrams is the Chief Information Officer at BigHat. He joins the show to talk about protein engineering, what’s different about software development in biotech, how the engineering team is organized at BigHat, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Watch the video episode here

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Biotech Special: AI Protein Engineering with Eddie Abrams appeared first on Software Engineering Daily.

Portal is a 2007 game developed by Valve, where the player encounters puzzles that must be solved using the “portal gun”, a device that can create inter-spatial portals between surfaces.

Portal 64 is an open-source re-write of Portal that can be run on original Nintendo 64 hardware. The game was developed by James Lambert and gained enormous praise as a technical and creative achievement. The project was ended in 2024 at the request of Valve.

James joins the show to talk about the process of developing an N64 games, the toolchain for building Portal 64, its physics engine, the design of the N64 cartridge, and much more. Be sure to check out James’ YouTube channel to see his work and get updates about his future projects.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Watch the video episode here.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Portal 64 with James Lambert appeared first on Software Engineering Daily.

Waymo is an autonomous driving company that had its start as the Google Self-Driving Car Project.

David Margines is a Director of Product Management at Waymo and he joins the podcast to talk about Waymo today, the sensing technologies underpinning their cars, the huge impact of AI on their systems in recent years, and more.

This episode is hosted by Tyson Kunovsky. Tyson is the co-founder and CEO of AutoCloud, an infrastructure as code platform. He is originally from South Africa, and has a background in software engineering and cloud development. When he’s not busy designing new GitOps workflows, he enjoys skiing, riding motorcycles, and reading sci-fi books. Check the show notes for more information on Tyson’s work, and where to find him.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Waymo and Autonomous Driving with David Margines appeared first on Software Engineering Daily.

Recursion is at the leading edge of applying AI and ML to drug development. The company exemplifies a new wave of “techbio” companies, that tightly couple compute and robotics with biology and chemistry. The task of decoding biology requires vast amounts of biological data and innovative strategies to make use of that data. It also requires close coordination between experts across a wide range of domains – from software to cell biology.

Imran Haque is the SVP of AI and Digital Sciences and Jordan Christensen is the SVP of Technology at Recursion. They join the show today to talk about the unique data engineering challenges in biology, the growing importance of automation, reshaping the drug discovery funnel, their partnership with NVIDIA, and much more.Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Watch the video episode here.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Biotech Special: ML at Recursion with Jordan Christensen and Imran Haque appeared first on Software Engineering Daily.

Producing 3D films, games and simulations is a complex process, often involving multiple teams and tools. At Pixar, pipeline engineers needed to write lots of glue code to integrate different workflows and file formats, which was a big challenge, and led them to create the Universal Scene Description, or OpenUSD. OpenUSD implements abstract data models for producing 3D worlds, and is now an open-source project.

Making full use of OpenUSD required a software framework. This motivated NVIDIA to create Omniverse, which is a modular development platform that enables individuals and teams to develop OpenUSD-based 3D workflows and applications.

Aaron Luk is the Director of Product Management for Omniverse, and was previously a software engineer at Pixar where he helped create OpenUSD. Aaron joins the show to talk about the origins of the technology, how it works, digital twins, industry impacts, and more.

To learn more about OpenUSD, check out the resources below:

OpenUSD Day at NVIDIA GTC: https://nvda.ws/3wsFJMg

Alliance for OpenUSD: https://nvda.ws/47Ro26n

Full Disclosure: NVIDIA is a sponsor of Software Engineering Daily

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Watch the video episode here

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post NVIDIA and the Future of 3D Development with Aaron Luk appeared first on Software Engineering Daily.

SimpleWebAuthn is an open source TypeScript-centric pair of libraries – frontend and backend – that make it easier for devs to implement WebAuthn on the web.

Matthew Miller started the project in 2019 and it has grown in tandem with the popularization of WebAuthn. He joins the podcast today to talk about the history of the project starting from the first commit, the problems it solves, its design, and more.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post SimpleWebAuthn with Matthew Miller appeared first on Software Engineering Daily.

There is a revolution unfolding in biotech. The confluence of new biological methods like CRISPR, virtually unlimited computational capacity, and machine learning has fundamentally transformed our ability to engineer biology for wide-ranging applications.

Andreessen Horowitz, or a16z, is a venture capital firm that was founded by Marc Andreessen and Ben Horowitz.

Vijay Pande is a Founding General Partner at a16z, where he leads the firm’s investments focused on the cross section of biology and computer science, including areas such as digital therapeutics, cloud biology, and computational medicine.

He joins the podcast to talk about innovation in biotech and healthcare, the biotech startup landscape, the impact of AI, and much more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Watch the video episode here

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Biotech Special: a16z and the Biotech Revolution with Vijay Pande appeared first on Software Engineering Daily.

oneAPI is an open standard for a unified API to be used across different computing accelerator architectures. This including GPUs, AI accelerators, and FPGAs.

The goal of oneAPI is to eliminate the need for developers to maintain separate code bases, multiple programming languages, tools, and workflows for each architecture.

James Reinders is an engineer at Intel and has experience with parallel computing spanning four decades. He joins the show today to talk about oneAPI.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Building a Unified Hardware API at Intel with James Reinders appeared first on Software Engineering Daily.

A game engine is a system used to build and run games. Game engines let the programmer work at a high level of abstraction by providing interfaces for graphics, physics, and scripting.

Godot is an open source and free to use game engine, which makes it unusual. Its first release was in 2014 and its features and user base have grown steadily since then.

Emilio Coppola is the Executive Director at the Godot Foundation and he joins the show to talk about the design of Godot, its scripting language, open source in game development, and more.

To learn about the new features in the Godot 4.2 release check out this blog post.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Watch the video episode here

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post The Godot Game Engine with Emilio Coppola appeared first on Software Engineering Daily.

Starburst is a data lake analytics platform. It’s designed to help users work with structured data at scale, and is built on the open source platform, Trino.

Adam Ferrari is the SVP of Engineering at Starburst. He joins the show to talk about Starburst, data engineering, and what it takes to build a data lake.

Full Disclosure: Starburst is a sponsor of Software Engineering DailySean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Watch the video episode here

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Building a Data Lake with Adam Ferrari appeared first on Software Engineering Daily.

Chess.com started in 2007 and grew steadily in the years following. The platform exploded in popularity during the pandemic, to the point that their servers struggled with the traffic. It was a great problem to have. Chess.com was instrumental in helping to elevate chess to its current height of mainstream popularity.

But how did Chess.com come to be, and how was the platform created?

Our guest today is Jay Severson who co-founded Chess.com and was its CTO. He joins the podcast to talk about the origins of the site, its development, scaling the platform, and what he’s working on today.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Building Chess.com with Jay Severson appeared first on Software Engineering Daily.

Mastodon is an open source, decentralized social network. Eugen Rochko started building Mastodon in response to his dissatisfaction with centralized social networks like Facebook and Twitter. In the Mastodon model, users can run their own nodes, and other users can connect to them. You can follow users whose accounts reside in other nodes.

Eugen joins the show today to talk about the engineering behind the project, and more.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Mastodon with Eugen Rochko appeared first on Software Engineering Daily.

Vercel provides a cloud platform to rapidly deploy web projects, and they develop the highly successful Next.js framework. The company recently made headlines when they announced v0 which is a generative AI tool to create React code from text prompts. The generated code uses open-source tools like Tailwind CSS and shadcn/ui.

Lee Robinson is the VP of Product at Vercel. He helps lead the product teams and focuses on developer experience on the platform. He joins the show to talk about Vercel, their AI SDK to easily connect frontend code with LLMs, the v0 AI tool, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Watch the video episode here

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Vercel AI with Lee Robinson appeared first on Software Engineering Daily.

Blender is a free and open-source 3D graphics tool that was initially released in 1994 and just hit version 4.0. It’s one of the triumphs of open-source software development and is used for creating animated films, art, 3D games, and more.Sybren Stüvel is a Senior Software Developer at Blender. He joins the show today to talk about the history of Blender, its path tracing renderer, managing large scale render farms, the Blender data structure, Python scripting in Blender, and much more.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Check out Blender’s Resources here:

Flamenco in 5 minutes
Scripting for Artists
4.0 release notes
Blender conference playlist

Please click here to see the transcript of this episode.

Watch the video episode here

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Blender with Sybren Stüvel appeared first on Software Engineering Daily.

Netflix needs no introduction and is renowned for its engineering talent.Shaundai Person is a Senior Software Engineer at Netflix, blogger, and conference speaker. She joins the show today to talk about getting her position at Netflix, developing internal tools at the company, the value of TypeScript, what makes a great software engineering manager, and more.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

Please click here to see the transcript of this episode.

Watch the video episode here

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Netflix UIs at Scale with Shaundai Person appeared first on Software Engineering Daily.

Algolia is a platform that provides search as a service. The company was founded in 2012, was part of Y Combinator’s Winter 2014 class, and has become highly popular for integrating modern search functionality into web-facing services.Sean Mullaney is the CTO of Algolia and has worked at Google X, Stripe, and Zolando. He joins the show today to talk about Algolia, neural search, vector compression, search optimization, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Watch the video episode here

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Algolia with Sean Mullaney appeared first on Software Engineering Daily.

Software security is a critical issue for everyone, but it takes on an entirely different dimension when your life, or the lives of others, depend on it. Consider the security needs of an environmentalist whistle blower inside a chemical corporation, or a human rights activist in Iran. Hyper-secure and fully anonymous operating systems are vital for many legitimate use cases. They are a double-edge sword though, and also empower nefarious actors.

Tails is an operating system designed to protect against surveillance and censorship. Our guest today is a member of the Tails group. For privacy reasons we will refer to him as ludo.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Edward Snowden’s Operating System with REDACTED appeared first on Software Engineering Daily.

Security issues can often be traced back to small misconfigurations in a database or cloud service, or an innocent code commit. OpsHelm is a security platform that’s oriented around identifying and fixing these issues.

Kyle McCullough is the Co-Founder and CTO of OpsHelm and he has deep experience in backend and data engineering. He joins the show to talk about the challenges of security incident monitoring, prioritization, and response.

This episode is hosted by Tyson Kunovsky. Tyson is the co-founder and CEO of AutoCloud, an infrastructure as code platform. He is originally from South Africa, and has a background in software engineering and cloud development. When he’s not busy designing new GitOps workflows, he enjoys skiing, riding motorcycles, and reading sci-fi books. Check the show notes for more information on Tyson’s work, and where to find him.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post OpsHelm with Kyle McCullough appeared first on Software Engineering Daily.

Jodie Burchell is the Data Science Developer Advocate at JetBrains, which makes integrated development environments or, IDEs, for many major languages. After observing the rapid growth of the AI coding assistant landscape, the company recently announced integration of an AI assistant into their IDEs.Jodie joins the show today to talk about why the company decided to take this step, the design challenges of adding AI tools to software products, and the team’s particular interest in auto-generating code documentation. Jodie also talks about the different types of language AIs, how AI tools will impact software development, and more.Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here for the transcript of this episode.

Watch the video episode here

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post JetBrains AI with Jodie Burchell appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of AWS re:Invent 2023, which took place from November 27th through December 1st in Las Vegas.

In today’s interview, host Jordi Mon Companys speaks with Mike Miller who is the Director of AWS AI Devices.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post AWS re:Invent Special: PartyRock Generative AI Apps with Mike Miller appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of AWS re:Invent 2023, which took place from November 27th through December 1st in Las Vegas.

In today’s interview, host Jordi Mon Companys speaks with Kevin Kelly who is the Director of the AWS Cloud Institute.

Visit https://aws.amazon.com/training to learn more about AWS’s skills training offerings and learning resources.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post AWS re:Invent Special: The AWS Cloud Institute with Kevin Kelly appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of AWS re:Invent 2023, which took place from November 27th through December 1st in Las Vegas.

In today’s interview, host Jordi Mon Companys speaks with Rob Zuber who is the CTO at CircleCI.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post AWS re:Invent Special: CircleCI with Rob Zuber appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of AWS re:Invent 2023, which took place from November 27th through December 1st in Las Vegas.

In today’s interview, host Jordi Mon Companys speaks with Ankur Mehrotra who is the Director and GM of Amazon SageMaker.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post AWS re:Invent Special: Sagemaker with Ankur Mehrotra appeared first on Software Engineering Daily.

Hypnospace Outlaw is a hit indie game which features an internet and operating system simulator set in a surreal, alternate version of 1999. The game is entirely unique both in its design and aesthetic, and it has an impressive development history.

Jay Tholen is the developer of Hypnospace Outlaw and he joins the show to talk about the engineering of the game, the creation of an entire “fake internet”, and he gives an update on his next game, Dreamsettler.

Joe Nash is a developer, educator, and award-winning community builder, who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got his start in software development by creating mods and running servers for Garry’s Mod, and game development remains his favorite way to experience and explore new technologies and concepts.

Please click here to see the transcript of this episode.

Watch the video episode here

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Bonus Episode: Hypnospace Outlaw with Jay Tholen appeared first on Software Engineering Daily.

SQL databases were built for data consistency and vertical scalability. They did this very well for the long era of monolithic applications running in dedicated, single-server environments. However, their design presented a problem when the paradigm changed to distributed applications in the cloud. This shift eventually ushered in the rise of distributed SQL databases. One of the most prominent is CockroachDB, which uses a distributed architecture inspired by Google’s Spanner. But what were the engineering approaches that made this architecture possible?

Jordan Lewis is a Senior Director of Engineering at CockroachDB Cloud. He joins the show to talk about the design of CockroachDB and how it works under the hood.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post CockroachDB with Jordan Lewis appeared first on Software Engineering Daily.

Software supply chain security is a major challenge in the modern engineering environment. Many teams are working to establish best practices to proactively identify, fix, and prevent risks in their applications. Apiiro is a platform designed to solve this problem and gives risk visibility, prioritization, and remediation.

Yonatan Eldar is the Co-Founder and CTO at Apiiro and he joins the podcast to talk about the platform, application security posture management, and more.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Full Disclosure: 10KMedia is a sponsor of Software Engineering Daily.

The post Apiiro Security Posture Management with Yonatan Eldar appeared first on Software Engineering Daily.

An embedding is a concept in machine learning that refers to a particular representation of text, images, audio, or other information. Embeddings are designed to make data consumable by ML models.

However, storing embeddings presents a challenge to traditional databases. Vector databases are designed to solve this problem.

Pinecone has developed one of the most prominent vector databases that is widely used for ML and AI applications.Marek Galovic is a software engineer at Pinecone and works on the core database team. He joins the podcast today to talk about how vector embeddings are created, engineering a vector database, unsolved challenges in the space, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here to see the transcript of this episode.

Watch the video episode here

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Pinecone Vector Database with Marek Galovic appeared first on Software Engineering Daily.

Carlos Sanchez is a Principal Scientist at Adobe where he works on the Adobe Experience Manager. AEM is a content management system and provides a platform for site creation and content delivery. In addition to his work at Adobe, Carlos has a long history contributing to open source projects, including Apache Maven. He joins the show today to talk about his work at Adobe, open source, and more.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Please click here to see the transcript of this episode.

Sponsorship inquiries:sponsor@softwareengineeringdaily.com

The post Bonus Episode: Adobe Cloud Native with Carlos Sanchez appeared first on Software Engineering Daily.

Building scalable software applications can be complex and typically requires dozens of different tools. The engineering often involves handling many arcane tasks that are distant from actual application logic. In addition, a lack of a cohesive model for building applications can lead to substantial engineering costs.

Nathan Marz is the creator of Rama, which is a platform for building end-to-end, scalable backends. Previously Nathan led engineering at BackType which was acquired by Twitter in 2011. In addition, Nathan created the Apache Storm project and is the author of the book Big Data: Principles and best practices of scalable realtime data systems. Nathan joins the podcast today to talk about Rama, and how to build scalable software applications.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries:sponsor@softwareengineeringdaily.com

The post Rama with Nathan Marz appeared first on Software Engineering Daily.

Frédéric Harper is the Principal Developer Advocate at Kubefirst, which is an open source platform that integrates some of the most popular tools in the Kubernetes space. Frédéric has deep experience at major software companies having worked at npm, Mozilla, Microsoft, DigitalOcean, Fitbit, and others. He joins the show to talk about the challenges and solutions associated with working with Kubernetes.

Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne.

Please click here to see the transcript of this episode.

Sponsorship inquiries:sponsor@softwareengineeringdaily.com

The post Kubefirst with Frédéric Harper appeared first on Software Engineering Daily.

Vespa is a fully featured search engine and vector database, and it has integrated ML model inference. The project open sourced in 2017, and since then has grown to become a prominent platform for applying AI to big data sets at serving time.

Vespa began as a project to solve Yahoo’s use cases in search, recommendation, and ad serving. The company made headlines in October when they announced they’re spinning Vespa.ai out of Yahoo as a separate company.

Jon Bratseth is the CEO at Vespa and he joins the show to talk about large language models, retrieval augmented generation, or RAG, vector database engineering, and more.Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .
Please click here to see the transcript of this episode.

Sponsorship inquiries:sponsor@softwareengineeringdaily.com

The post Vespa.ai with Jon Bratseth appeared first on Software Engineering Daily.

SurrealDB is the result of a long-time collaboration between brothers Tobie and Jaime Morgan Hitchcock. The project has modest origins and started merely to support other projects the brothers were working on. However, over time the project grew and in 2021 they started working on it full-time. Since then the project has gained serious adoption.

What’s makes SurrealDB so special? Tobie Morgan Hitchcock is the CEO of SurrealDB and he joins the show to talk about his multimodal database, support for graph and time series data, why they rewrote the entire project in Rust, and more.

Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne.

Please click here to see the transcript of this episode.

Sponsorship inquiries:sponsor@softwareengineeringdaily.com

The post Bonus Episode: SurrealDB with Tobie Morgan Hitchcock appeared first on Software Engineering Daily.

Responsible AI is an approach to developing and deploying AI in a safe, trustworthy and ethical fashion. The concept has gained considerable attention with the rise of generative AI technologies.

Ezequiel Lanza is an AI Open Source Evangelist at Intel and he joins the show today to talk about responsible AI, and the practices and tools evolving around it.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm.

Please click here to see the transcript of this episode.

Sponsorship inquiries:sponsor@softwareengineeringdaily.com

The post Bonus Episode: Responsible AI with Ezequiel Lanza appeared first on Software Engineering Daily.

Ransomware attacks involve the deployment of malware that blocks access to a user’s or organization’s computer files by encrypting them. The attackers then demand a ransom payment in exchange for the decryption key that will restore access to the files. These attacks are often directed at governments and corporations, and can be costly.

Veeam is a data storage system that was designed specifically to provide protection against ransomware attacks. Object First is a system that works with Veeam to increase its safety and security.

Anthony Cusimano is the Director of Technical Marketing at Object First and he joins the podcast to talk about the growing sophistication of ransomware attacks and the emerging technologies to block them.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries:sponsor@softwareengineeringdaily.com

The post Blocking Ransomware Attacks with Anthony Cusimano appeared first on Software Engineering Daily.

Supabase is an open source backend-as-a-service platform and competes directly with Google’s Firebase. A key distinction between them is that Firebase is a document store, while Supabase uses Postgres, which is a SQL-based database management system.

Software Engineering Daily last covered Supabase in 2020 when its Founder Paul Copplestone came on the show, and a lot has changed since then. The platform has continued to grow its feature set and, importantly, this growth has come with an expanded number of security considerations.

Inian Parameshwaran is Head of Product and Engineering at Supabase and a former YCombinator Fellow. He started at Supabase when the platform was still in alpha, and has been there for three years. He joins the show today to talk about the current feature set of Supabase, and the various security challenges that he and his team work on, including how to handle crypto mining exploits, IP blocking, DDoS attacks, and more.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.

Please click here to see the transcript of this episode.

Sponsorship inquiries:sponsor@softwareengineeringdaily.com

The post Supabase Security with Inian Parameshwaran appeared first on Software Engineering Daily.

GitHub Copilot is an AI tool developed by GitHub and OpenAI to assist software developers by autocompleting code. Copilot kicked off a revolution in software engineering, and AI assistants are now considered essential tools to many developers.

Joseph Katsioloudes is a cyber security specialist and works at the GitHub Security Lab. He joins the show today to talk about Copilot, the future of software development in an AI world, using AI to improve security, and more.

Check out Joseph’s bio and the Secure Code Game which is an in-repo learning experience that Joseph created to teach how to secure vulnerable code.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information covisualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You ca connect with Sean on Twitter @seanfalconer .
Please click here to see the transcript of this episode.

Sponsorship inquiries:sponsor@softwareengineeringdaily.com

Watch the video episode here.

The post GitHub Copilot with Joseph Katsioloudes appeared first on Software Engineering Daily.

Build systems coordinate all the steps to transform source code into a production application. Bazel is a build system and testing tool that was first released in 2015 as a free and open-source port of Google’s internal build system called Blaze.

Historically, each language has its own build system which can create complexity when developing applications that use many languages. Bazel is special because it’s a polyglot system with unified support for many languages.

To handle build configuration, Bazel uses the Starlark language which has syntax inspired by Python. This is a key part of what contributes to Bazel’s growing popularity.

Julio Merino is a Senior Software Engineer at Snowflake, and before that worked at Google and Microsoft. He joins the podcast today to talk about Bazel.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Please click here to see the transcript of this episode.

Sponsorship inquiries:sponsor@softwareengineeringdaily.com

The post Bonus Episode: Bazel with Julio Merino appeared first on Software Engineering Daily.

On a recent trip to my hometown in Eastern Canada, my father picked me up at the airport. One of the first things he asked me was, “Is AI going to take everyone’s jobs?”.

When AI, generative AI, and large language models (LLM) have become topics of conversation within the senior citizen community of rural Canada, you know it’s on everyone’s minds. Generative AI, and especially the use of LLMs, is the “new new thing”. It dominates my X (i.e. Twitter) feed and nearly every conversation I have about technology.

There’s justifiably a ton of excitement about the power of generative AI, reminiscent of the introduction of the Internet or the first smartphone. Generative AI is poised to transform how we build products, design drugs, write content, and interact with technology. But as the utilization of AI grows, many governments and companies have raised concerns about the privacy and compliance issues that adopters of these technologies face.

The core challenge posed by generative AI right now is that unlike conventional applications, LLMs have no “delete” button. There’s no straightforward mechanism to “unlearn” specific information, no equivalent to deleting a row in your database’s user table. In a world where the “right to be forgotten” is central to many privacy regulations, using LLMs presents some difficult challenges.

So what does all this mean for businesses that are building new AI-powered applications or AI models?

In this post, we’ll explore this question and attempt to provide answers. We’ll examine the potential impact of generative AI, ongoing compliance hurdles, and a variety of privacy strategies. Finally, we’ll examine a novel approach grounded in the IEEE’s recommended architecture for securely storing, managing, and utilizing sensitive customer PII (Personally Identifiable Information)—the data privacy vault.

Generative AI’s privacy and compliance challengesImagine the following scenario: You’ve just copied and pasted sensitive contract details into an LLM to get some quick assistance with routine contract due diligence. The LLM serves its purpose, but here’s the catch: depending on how it’s configured, that confidential contract data might linger within the LLM, accessible to other users. Deleting it isn’t an option, predicting its future use—or misuse—becomes a daunting task, and retraining the LLM to “roll it back” to its state before you shared those sensitive contract details can be prohibitively expensive.

The only foolproof solution?

Keep sensitive data far away from LLMs.

Sensitive information, including internal company project names, core intellectual property, or personal data like birthdates, social security numbers, and healthcare records, can inadvertently find its way into LLMs in several ways:

Training data: LLMs are trained and honed on expansive datasets that often contain PII. Without robust anonymization or redaction measures in place, sensitive data becomes part of the model’s training dataset, meaning that this data can potentially resurface later.
Inference: LLMs generate text based on user inputs or prompts. Much like training data, a prompt containing sensitive data seeps into the model and can influence the generated content, potentially exposing this data.

Privacy lawsAI data privacy is a formidable challenge for any company interested in investing in generative AI technology. Recent temporary bans of ChatGPT in Italy and by companies like Samsung have pushed these concerns to the forefront for businesses looking to invest in generative AI.

Even outside of generative AI, there are increasing concerns about protecting data privacy. Meta was recently fined $1.3 billion by the European Union (EU) for its non-compliant transfers of sensitive data to the U.S. And this isn’t just an issue for companies doing business in the EU.

There are now more than 100 countries with some form of privacy regulation in place. Each country’s privacy regulations include unique and nuanced requirements that place a variety of restrictions on the use and handling of sensitive data. The most common restrictions relate to cross-border data transfers, where sensitive data can be stored, and to individual data subject rights such as the “right to be forgotten.”

One of the biggest shortcomings of LLMs is their inability to selectively delete or “unlearn” specific data points, such as an individual’s name or date of birth. This limitation presents significant risks for businesses leveraging these systems.

For example, privacy regulations in Europe, Argentina, and the Philippines (just to name a few) all support an individual’s “right to be forgotten.” This grants individuals the right to have their personal information removed or erased from a system. Without an LLM delete button, there’s no way for a business to address such a request without retraining their LLM from scratch.

Consider the European Union’s General Data Protection Regulation (GDPR), which grants individuals the right to access, rectify, and erase their personal data—a task that becomes daunting if that data is embedded within an LLM. GDPR also empowers individuals with the right to object to automated decision-making, further complicating compliance for companies that use LLMs.

Data localization requirements pose another challenge for users of LLMs. These requirements pertain to the physical location where customer data is stored. Different countries and regions have precise laws dictating how customer data should be handled, processed, stored, and safeguarded. This poses a significant challenge when using an LLM used for a company’s global customer base.

Data Subject Access Requests (DSARs) under GDPR and other laws add another layer of complexity. In the EU and California, individuals (i.e., “data subjects”) have the right to request access to their personal data, but complying with such requests proves challenging if that data has been processed by LLMs.

Considering the intricate privacy and compliance landscape and the complexity of LLMs, the most practical approach to maintaining compliance is to prevent sensitive data from entering the model altogether. By implementing stringent data handling practices, businesses can mitigate the privacy risks associated with LLMs, while also maintaining the utility of the model. Many companies have already decided that the risks are too high, so they’ve banned the use of ChatGPT, but this approach is shortsighted. Properly managed, these models can create a lot of value.

Privacy approaches for generative AITo address the privacy challenges associated with generative AI models, there have been a few proposals such as banning or controlling access, using synthetic data instead of real data, and running private LLMs.

Banning ChatGPT and other generative AI systems isn’t an effective long-term strategy, and these other “band aid” approaches are bound to fail as people can find easy workarounds. Using synthetic data replaces sensitive information with similar-looking but non-sensitive data and keeps PII out of the model, but at the cost of losing the value that motivated you to share sensitive data with the LLM in the first place. The model loses context, and there’s no referential integrity between the synthetically generated data and the original sensitive information.

The most popular approach to addressing AI data privacy, and the one that’s being promoted by cloud providers like Google, Microsoft, AWS, and Snowflake, is to run your LLM privately on their infrastructure.

For example, with Snowflake’s Snowpark Model Registry, you can take an open source LLM and run it within a container service in your Snowflake account. They state that this allows you to train the LLM using your proprietary data.

Snowpark Model Registry and Container Service (Source: Snowflake Blog)However, there are several drawbacks to using this approach.

Outside of privacy concerns, if you’re choosing to run an LLM privately rather than take advantage of an existing managed service, then you’re stuck with managing the updates, and possibly the infrastructure, yourself. It’s also going to be much more expensive to run an LLM privately. Taken together, these drawbacks mean running a private LLM likely doesn’t make sense for most companies.

But the bigger issue is that, from a privacy standpoint, private LLMs simply don’t provide effective data privacy. Private LLMs give you model isolation, but they don’t provide data governance in the form of fine-grained access controls: any user who can access the private LLM can access all of the data that it contains. Data privacy is about giving a user control over their data, but private LLMs still suffer from all of the intrinsic limitations around data deletion that are blocking the adoption of public LLMs.

What matters to a business—and individual data subjects—is who sees what, when, where, and for how long. Using a private LLM doesn’t give you the ability to make sure that Susie in accounting sees one type of LLM response based on her job title while Bob in customer support sees something else.

So how can we prevent PII and other sensitive data from entering an LLM, but also support data governance so we can control who can see what and support the need to delete sensitive data?

A new approach to PII managementIn the world of traditional data management, an increasingly popular approach to protecting the privacy of sensitive data is through the use of a data privacy vault. A data privacy vault isolates, protects, and governs sensitive customer data while facilitating region-specific compliance with laws like GDPR through data localization.

With a vault architecture, sensitive data is stored in your vault, isolated outside of your existing systems. Isolation helps ensure the integrity and security of sensitive data, and simplifies the regionalization of this data. De-identified data that serve as references to the sensitive data are stored in traditional cloud storage and downstream services.

De-identification happens through a tokenization process. This is not the same as LLM tokenization, that has to do with splitting texts into smaller units. With data de-identification, tokenization is a non-algorithmic approach to data obfuscation that swaps sensitive data for tokens. A token is a pointer that lets you reference something somewhere else while providing obfuscation.Traditional data management versus a data privacy vault architec

Traditional data management versus a data privacy vault architectureLet’s look at a simple example. In the workflow below a phone number is collected by a front end application. The phone number, along with any other PII, is stored securely in the vault, which is isolated outside of your company’s existing infrastructure. In exchange, the vault generates a de-identified representation of the phone number (e.g. ABC123). The de-identified (or tokenized) data has no mathematical connection with the original data, so it can’t be reverse engineered.

Any downstream services—application databases, data warehouse, analytics, any logs, etc.—store only a token representation of the data, and are removed from the scope of compliance:

Example of a data privacy vault in actionAdditionally, a data privacy vault can store sensitive data in a specific geographic location, and tightly control access to this data. Other systems, including LLMs, only have access to non-sensitive de-identified data.

The vault not only stores and generates de-identified data, but it tightly controls access to sensitive data through a zero trust model where no user account or process has access to data unless it’s granted by explicit access control policies. These policies are built from the bottom, granting access to specific columns and rows of PII. This allows you to control who sees what, when, where, for how long, and in what format.

For example, let’s say we have a vault containing customer records with columns defined for a customer’s name, social security number (SSN), date of birth (DOB), and email. In our application we want to support two types of users: support and marketing.

Support doesn’t need to know the exact details about a customer, they only need masked data so they can speak to the customer by name and verify their identity using the last four digits of the customer’s SSN. We can create a policy for the role support that grants access to only the limited view of the data.

ALLOW READ ON users.full_name, users.ssn, users.email WITH REDACTION = MASKEDALLOW READ ON users.dob WITH REDACTION = REDACTED Similarly, a marketing person needs someone’s name and email, but they don’t need the customer’s SSN or need to know how old someone is.

ALLOW READ ON users.full_name, users.email WITH REDACTION = PLAIN_TEXTALLOW READ ON users.dob WITH REDACTION = MASKEDALLOW READ ON users.ssn WITH REDACTION = REDACTED With roles and policies similar to ones above in place, the same de-identified data is exchanged with the vault. Based on the role and associated access control policies for the caller, different views of the same sensitive data can be supported.

Different views of sensitive data based on role.A privacy firewall for LLMsCompanies can address privacy and compliance concerns with LLMs with a similar application of the data privacy vault architectural pattern. A data privacy vault prevents the leakage of sensitive data into LLMs, addressing privacy concerns around LLM training and inference.

Because data privacy vaults use modern privacy-enhancing technologies like polymorphic encryption and tokenization, sensitive data can be de-identified in a way that preserves referential integrity. This means that responses from an LLM containing de-identified data can be re-identified based on zero trust policies defined in the vault that let you make sure that only the right information is shared with the LLM user. This lets you make sure Susie in accounting only sees what she should have access to (i.e., account numbers and invoice amounts) while Bob in customer support sees only what he needs to do his job.

Preserving privacy during model trainingTo preserve privacy during model training, the data privacy vault sits at the head of your training pipeline. Training data that might include sensitive and non-sensitive data goes to the data privacy vault first. The vault detects the sensitive data, stores it within the vault, and replaces it with de-identified data. The resulting dataset is de-identified and safe to share with an LLM.

Model Training Pipeline with a Data Privacy VaultAn LLM doesn’t care whether my name, Sean Falconer, is part of the training data or some consistently generated representation of my name (such as “dak5lhf9w”) is part of the training data. Eventually, it’s just a vector.

Preserving privacy during inferenceSensitive data may also enter a model during inference. In the example below, a prompt is created asking for a summary of a will. The vault detects the sensitive information, de-identifies it, and shares a non-sensitive version of the prompt with the LLM.

Since the LLM was trained on non-sensitive and de-identified data, inference can be carried out as normal.

Example of de-identifying inference data with an LLM and a data privacy vaultOn egress from the LLM, the response is passed through the data privacy vault for re-identification. Any de-identified data will be re-identified assuming the end-user has the right to see the information, according to explicit access control policies configured in the vault.

Privacy and complianceFrom a privacy and compliance standpoint, using a data privacy vault means that no sensitive data is ever shared with an LLM, so it remains outside of the scope of compliance. Data residency, DSARs, and delete requests are now the responsibility of a data privacy vault that’s designed to handle these requirements and workflows.

Incorporating the vault into the model training and inference pipelines allows you to combine the best of modern sensitive data management with any LLM stack, private, public, or proprietary.

Final thoughtsAs every company gradually morphs into an AI company, it’s critically important to face data privacy challenges head-on. Without a concrete solution to data privacy requirements, businesses risk remaining stuck indefinitely in the “demo” or “proof-of-concept” phase. The fusion of data privacy vaults and generative AI offers a promising path forward, freeing businesses to harness the power of AI without compromising on privacy.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.The post Privacy in the Age of Generative AI appeared first on Software Engineering Daily.

Containers make it possible to standardize the deployment of software to any compute environment. However, managing and orchestrating containers at scale is a major challenge. Kubernetes was originally created by Google and solves the problem of scaling container deployment.

Ben Elder is a Senior Software Engineer at Google, and an Elected Member of the Kubernetes Steering Committee. Ben joins the show to talk about why Kubernetes became the standard for container orchestration, Kubernetes control theory, how he runs his home infrastructure, and more.This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Kubernetes at Google with Ben Elder appeared first on Software Engineering Daily.

Deno is a free and open source JavaScript runtime built on Google’s V8 engine, Rust, and Tokio. The project was announced by Ryan Dahl in 2018 with the goal of addressing shortcomings of Node.js, which Ryan also created. Since then, the Deno project has grown tremendously in popularity, and they recently announced Deno KV which is a database built into Deno.

Luca Casonato is a Software Engineer on the Deno project and joins the show to talk about Deno’s design, its new database, and the future of the JavaScript ecosystem.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.

You can find Josh on: Bluesky, Fosstodon, Twitter, Twitch, YouTube, and joshuakgoldberg.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Deno with Luca Casonato appeared first on Software Engineering Daily.

Machine learning model research requires running expensive, long-running experiments where even a slight mis-calibration can cost millions of dollars in underutilized compute resources. Once trained, model deployment, production monitoring, and observability requirements all present unique operational challenges.

Chris Van Pelt is the Chief Information Officer of Weights and Biases, which is the industry standard in experiment monitoring and visualization, and has expanded that expertise into a comprehensive suite of ML Ops tooling including model management, deployment, and monitoring.

Chris joins us today to discuss the state of the machine learning ecosystem at large, as well as some of their more recent work around production LLM tracing and monitoring.Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Weights & Biases with Chris Van Pelt appeared first on Software Engineering Daily.

From the widely recognized GDPR in Europe to Brazil’s LGPD regulations, and the more recent introduction of India’s DPDP law, over 100 countries now have some form of privacy regulation in place. What’s common among many of these regulations is the concept of data residency – the physical location of your data. However, each region’s requirements bring their own unique nuances, encompassing restrictions on data transfer, data storage locations, and individual data rights.

Navigating this complex sphere of privacy regulations is a huge burden for many companies born in the cloud. Their data simply ends up everywhere, and tracking down the locations, adhering to local laws, and even storing and using it locally is enormously complex and expensive.

Over the past year, I’ve engaged with numerous companies eager to expand their businesses into new markets, such as Europe and Australia. However, they’ve encountered a significant roadblock – the absence of a robust technology solution to address the data residency requirements of these regions. As a result, they face the expensive and nightmarish scenario of duplicating their cloud infrastructure for each new region, which not only hampers operational efficiency but also limits their data analyst and scientists from running analytics globally.

In this blog post, I offer a solution to this pressing technology and business challenge by introducing a PII data privacy vault. This architectural approach to data privacy effectively removes the burden of data residency, compliance, and data security responsibilities from your infrastructure, providing a seamless path for global expansion and data management.

Let’s dive in.

Data Residency and Barriers to ExpansionTo grasp the intricacies of regulatory compliance in the context of global expansion, it’s important to understand a few key concepts.

ComplianceCompliance denotes a business’s adherence to the laws and regulations governing data privacy and protection. These regulations are contingent on the geographic location of the customer whose data is being collected. Ensuring compliance is imperative for legal reasons as it shields businesses from financial penalties, license revocations, and the erosion of customer trust.

Data ResidencyData residency pertains to the physical location where customer data is stored. For instance, a website may serve customers in the EU, but their data could be hosted on a server located in Chicago. Different countries and regions have precise laws dictating how customer data should be handled, processed, stored, and safeguarded, making data residency a critical consideration.

Varying RegulationsThe complexity surrounding data residency and compliance obligations primarily arises from the diversity of regulations worldwide. For instance, the European Union (EU) has GDPR, Brazil follows LGPD, and the United States enforces a patchwork of state-specific laws like CCPA in California and CTDPA in Connecticut. These regulations diverge significantly in terms of their stipulations and penalties.

Barriers to Global ExpansionThe disparities in regulations and compliance requirements often pose formidable obstacles for companies striving to attain a global presence. Navigating diverse regulatory frameworks demands significant time, resources, and expertise. The resulting complexity frequently dissuades businesses from venturing into new markets, thereby constraining opportunities for global expansion.

We’ve looked at the problem, now, let’s explore an approach to addressing these challenges.

What is a Data Privacy Vault?A data privacy vault isolates, protects, and governs access to sensitive customer data. Within the vault, confidential information is securely stored, while abstract and non-sensitive tokens, serving as references, are retained in conventional cloud storage. This means that only non-sensitive tokenized data is accessible to other systems, ensuring the utmost protection and compliance.

In a recent IEEE article, the authors made a case that this architectural approach to data privacy is the future of privacy engineering. Just as any modern system likely contains back end services, a database, and a warehouse, all modern systems need a data privacy vault to safely store, handle, and use of sensitive customer PII.

Traditional PII management versus a data privacy vault (source: IEEE).Let’s take a look at a specific example for a simple web application. In the image below, a phone number is being collected by a front-end application. For effective de-scoping, it’s ideal to initiate the de-identification process at the earliest stage in the data lifecycle. In this scenario, the phone number is stored directly within the vault during collection at the front end.

Example of vault architecture for collecting sensitive customer PII.Within the vault, the phone number, alongside any other personally identifiable information (PII), is stored within a robust and isolated environment, segregated from your organization’s existing infrastructure. All downstream services, ranging from application databases to data warehouses, analytics platforms, and logging systems, interact solely with tokenized (de-identified) representations of the data. Queries against the PII for specialized operations or algorithmic operations against PII execute directly within the vault.

Access to de-tokenize or re-identify data is controlled through a zero trust model. Policy-based rules control who sees what, when, where, and for how long on a row and column level.

Controlling access to vault data based on who is requesting the data.The vault combines the principle of isolation, zero trust, privacy-enhancing technologies, and governance controls to insulate your systems from ever having to touch PII directly. This places your AWS components beyond the scope of regulatory compliance, assuring a higher level of data protection and adherence to data residency requirements.

Your AWS Services Handle Only De-identified DataLet’s assume we have a simple application infrastructure as shown below with AWS Amplify providing the web server infrastructure, DynamoDB for application storage, and Redshift for warehousing.

Example web application infrastructure running on AWS.Without a vault in place, everything within our AWS account is under compliance and security scope.

By introducing the vault as shown below (in this example, the collection of PII is handled directly from the vault), we de-scope all our AWS services. The services are only ever handling de-identified data, including the warehouse.

Many analytical operations can be performed with de-identified data provided the data is consistently generated. A warehouse doesn’t need to have access to someone’s name, it only needs a consistently generated representation of the name in order to execute counts, group bys, and joins.

Example of de-scoping AWS services with a data privacy vault.Storing PII to Different Regionalized VaultsWith Skyflow, a data privacy vault company, you can host vaults in various global regions and route sensitive data to a specific regional vault for storage and use. For instance, consider how the following application architecture meets data residency requirements across multiple regions:

Using regional multiple vaults to comply with data residency requirements.1. Your company’s site collects customer PII during account creation. 2. On the client side, the website detects the customer’s location. 3. Detecting that the customer is in the EU, the client-side code uses Skyflow’s SDK to collect the PII data and store it in your company’s data privacy vault in Frankfurt, Germany. Note: For customers based in the US, the PII data is instead routed to the data privacy vault in the US (in this case, Virginia). 4. The EU-based customer’s sensitive PII is stored in the EU-based data privacy vault, and Skyflow responds with de-identified data. 5. The client-side code sends the account request, now with de-identified data, to the server. 6. The server processes the request, storing the data (now de-identified and tokenized) in cloud storage in the “Oregon, US” region. 7. At the end of the week, your company’s Redshift instance in Tokyo, Japan, loads the data (already de-identified and tokenized) from cloud storage to perform analytics.

Deploying multiple vaults situated in different regions streamlines the management of your sensitive data, ensuring compliance with data residency requirements across all your markets.

The data privacy vault architecture significantly simplifies the complexities associated with data residency and compliance. Furthermore, by exempting Redshift (or any warehouse) from the compliance responsibilities tied to data residency, global analytics operations continue seamlessly within a single warehouse instance.

Final ThoughtsCompliance regulations, with their stringent data residency stipulations, necessitate businesses to maintain rigorous standards for data localization, protection, privacy, and security. Adhering to these regulations is essential to mitigating the risks associated with breaches, penalties, and potential damage to reputation. However, enterprises operating in various global regions, serving diverse customer bases, are left to deal with the complex task of navigating multiple regulatory landscapes.

Using data privacy vaults as your core infrastructure for customer PII offers a streamlined solution to simplify global compliance, particularly concerning AWS services and cloud storage.

With a data privacy vault, organizations gain the ability to centralize the security of all sensitive data, effectively removing AWS and cloud storage from their compliance scope. By deploying data privacy vaults in various regions, companies can ensure that sensitive data storage and transmission align with the specific laws and regulations of each operational jurisdiction, thereby enhancing their overall compliance and security posture.

If you have thoughts on this or questions about this approach, please reach out to me on LinkedIn.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.The post De-scoping Your AWS Services from Data Residency Requirements appeared first on Software Engineering Daily.

Maritime logistics is the process organizing the movement of goods across the ocean. Historically, this has been a challenging problem because of the multinational nature of shipping, as well as piracy, smuggling, and legacy technology. It’s also profoundly important for security reasons, and because 90% of what we buy travels over the oceans. Ocean vessels produce a lot of CO2, which adds climate change and energy dimensions to maritime logistics.

Windward AI is a maritime logistics platform that was started 13 years ago by two ex-Israeli naval officers. The idea for the company came from the observation that, at that time, it was hard or impossible to know what’s happening on the deep sea.

Benny Keinan is the VP of R&D and Lior Resisi is the Data Platforms Group Lead at Windward AI. They join the podcast today to talk about the technical and practical challenges of maritime logistics, why Rockset was the right database for their unique datasets, the impact of the Ukraine war, and more.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Tracking Drug Smugglers and Migrating Databases with Benny Keinan and Lior Resisi appeared first on Software Engineering Daily.

One of the most famous software exploits in recent years was the SolarWinds attack in 2020. In this attack, Russian hackers inserted malicious code into the SolarWinds Orion system, allowing them to infiltrate the systems of numerous corporations and government agencies, including the U.S. executive branch, military, and intelligence services.

This was an example of a software supply chain attack, which exploits interdependencies within software ecosystems. Software supply chain security is a growing issue, and is particularly important for companies that rely on large numbers of open source dependencies.

Michael Lieberman is the Co-Founder and CTO of Kusari and has an extensive background in software security from his time at Citi Bank, MUFG and Bridgewater. He’s also active in the open source and security communities, including the Open Source Security Foundation and Cloud Native Computing Foundation. Michael joins the show today to talk about challenges and strategies in software supply chain security.

Gregor Vand is a security-focused technologist, and is the founder and CTO of Mailpass. Previously, Gregor was a CTO across cybersecurity, cyber insurance and general software engineering companies. He has been based in Asia Pacific for almost a decade and can be found via his profile at vand.hk.Please click here for the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Software Supply Chain Security with Michael Lieberman appeared first on Software Engineering Daily.

Hugging Face was founded in 2016 and has grown to become one of the most prominent ML platforms. It’s commonly used to develop and disseminate state-of-the-art ML models and is a central hub for researchers and developers.

Sayak Paul is a Machine Learning Engineer at Hugging Face and a Google Developer Expert. He joins the show today to talk about how he entered the ML field, diffusion model training, the transformer-based architecture, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Hugging Face with Sayak Paul appeared first on Software Engineering Daily.

In any given week, if you search the news for “data breach”, you’ll see headlines like the ones below.

Companies like MGM and Caesars spend millions of dollars on firewalls, SIEMs, HSMs, and a whole smorgasbord of cybersecurity tools and yet, they can’t protect your social security number.

From hotels and casinos to some of the most innovative technology companies in the world, why is it that companies with seemingly endless financial and talent resources can’t get a handle on their data security challenges?

I believe this is due to a fundamental misunderstanding about the nature of data that started over 40 years ago.

Back in the 1980s, as computers found their way more and more into businesses, we lived in a disconnected world. To steal someone’s data, you had to physically steal the box the data lived on. As a consequence, we assumed that all data is created equal, that all data is simply ones and zeros, but this is wrong. All data isn’t created equal, some data is special, and needs to be treated that way.

In this blog post, I share my thoughts on what I refer to as the “Cheese and Diamond Problem” and how this has led to the data security challenges companies face today. I also offer explore an alternative approach, a new way of thinking, a privacy by engineering approach that helps us move towards a world where security is the default, and not bolted on.

The Cheese and Diamond ProblemImagine that in my house I have cheese and I have diamonds. As a gracious host, I want guests of my home to be able to access my cheese. They should be able to freely go into the refrigerator and help themselves to some delicious cheese and perhaps a cracker.

However, I don’t want just anyone to touch my diamonds. Perhaps my diamonds even have sentimental value because it’s a diamond ring that’s been passed down through many generations in my family. Clearly the diamond is special.

Yet, if I store my diamonds in the refrigerator next to my cheese, it makes controlling access to the diamonds much more challenging. By co-locating these very different objects, my refrigerator alone isn’t enough to make sure my wife has access to the diamonds and cheese, but my guests only have access to my cheese.

The rules of engagement for something like diamonds are completely different than the rules of engagement for cheese. We all understand this distinction when it comes to physical objects.

This is exactly why my passport and my children’s birth certificates aren’t in the junk drawer in my kitchen with my batteries and my flashlights. If someone breaks into my home and steals my batteries, it’s not that big a deal, but if someone steals my daughter’s birth certificate, then I not only feel like I’ve failed as a parent, but the information on her birth certificate is also now compromised forever. I can’t simply replace her date of birth.

Despite all of us intuitively understanding that some physical objects are different, that they’re special, we somehow miss this point when we work with data. We don’t apply this thinking to Personally Identifiable Information (PII). We treat it like any other form of transactional or application data. We stuff it in a database, pass it around, make a million copies, and this leads to a whole host of problems.

The PII Replication ProblemLet’s consider a simple example.

In the diagram below, which represents an abstraction of a modern system, a phone number is being collected in the front end of the application, perhaps during account creation. That phone number ends up being passed downstream through each node and edge of the graph and at each node, we potentially end up with a copy of the phone number.

We store it in our database, in the warehouse, but we may also end up with a copy in our log files and the backups of all these systems. Instead of just having one copy of the phone number, we now have many copies and we need to protect all those locations and control access consistently wherever the data is stored.

Imagine that instead of having one copy of your passport that you keep in a secure location, you made 10,000 copies and then distributed them all over the world. Suddenly keeping your passport safe becomes a much harder problem in all 10,000 locations than if you have one copy secure in your home.

But this is exactly what we do with data.

We copy it everywhere and then attempt to lock down the hatches across all these systems and keep the policies and controls in sync about who can see what, when, and where. Additionally, because of the Cheese and Diamond Problem, we can’t adequately govern access to the data because the intermixing of our data conflates the rules of engagement about who has access. This quickly becomes an intractable problem because businesses don’t know what they’re storing or where it is, leading to the world we live in now where major corporations have data breaches on a regular basis.

Not All Data is EqualBusinesses are collecting and processing more data than ever. With the explosion of generative AI, as much as we are in an AI revolution, we are also in a data revolution. We can’t have powerful LLMs without access to massive data.

Companies leverage their data to drive business decisions, product direction, help serve customers better, and even create new types of consumer experiences. However, as discussed, not all data is created equal, some data, like PII, is special.

Over time, we’ve recognized that other forms of data like encryption keys, secrets, and identity are special and need to be treated that way. There was a time when we stored secrets in our application code or database. We eventually realized that was a bad idea and moved them into secret managers.

Approaches to managing different types of sensitive data.Despite this progress, we are still left without an accepted standard for the storage and management of sensitive PII data. PII deserves the same type of special handling. You shouldn’t be contaminating your database with customer PII.

Luckily there’s a solution to this problem originally pioneered by companies like Netflix, Google, Apple, and Goldman Sachs and now touted by the IEEE as the future of privacy engineering, the PII Data Privacy Vault.

The PII Data Privacy VaultA data privacy vault isolates, protects, and governs access to sensitive customer data (i.e. PII) while also keeping it usable. With a vault approach, you remove PII from your existing infrastructure, effectively de-scoping it from the responsibility of compliance and data security.

A vault is a first principles architectural approach to data privacy and security, facilitating workflows like:

PII storage and management for regulated industries
PCI storage and payment orchestration
Data residency compliance
Privacy-preserving analytics
Privacy-preserving AI

Let’s go back to our example from earlier where we were collecting a phone number from the front end of an application.

In the vault world, the phone number is sent directly to the vault from the front end. From a security perspective, we ideally want to de-identify sensitive data as early in the life cycle as possible. The real phone number will only exist within the vault, it acts as a single source of truth that’s isolated and protected outside of the existing systems.

Example of using a data privacy vault to de-scope an application.The vault securely stores the phone number and generates a de-identified reference in the form of a token that gets passed back to the front end. The token has no mathematical connection to the original data, so it can’t be reverse engineered to reveal the original value.

This way, even if someone steals the data, as what happened with the Capital One data breach, the tokenized data carries no value. In fact, Capital One was fined only because they failed to tokenize all regulated data, some records were purely encrypted and those records were compromised.

Revealing Sensitive DataWhile it’s great to securely store sensitive data, if we simply lock it up and throw away the key, it’s not super useful. We store all this customer PII so we can use it.

For example, we may need to reveal some of the data to a customer support agent, an IT administrator, a data analyst, or to the owner of the data. In this case, if we absolutely need to reveal some of the data, we want to re-identify it as late as possible, for example during render. We also want to limit what a user has access to based on the operations they need to perform with the data. While I might be able to see my full phone number, a customer support agent likely only needs the last four digits of my phone number and an analyst maybe only needs the area code for executing geo-based analytics.

The vault facilitates all of these use cases through a zero trust model where no one and no thing has access to data without explicit policies in place. The policies are built bottoms up, granting access to specific columns and rows of PII. This allows you to control who sees what, when, where, for how long, and in what format.

Let’s consider the situation where we have a user logging into an application and navigating to their account page. On the account page, we want to show the user their name, email, phone number, and home address based on the information they registered with us.

In the application database, we’ll have a table similar to the one shown below where the actual PII has been replaced by de-identified tokens.

Example of users table within the application database.As in the non-vault world, the application will query the application database for the user record associated with the logged in user. The record will be passed to the front end application and the front end will exchange the tokens for a representation of the original values depending on the policies in place.

In the image below, the front end already has the tokenized data but needs to authenticate with the vault attaching the identity of the logged in user so that access is restricted based on the contextual information of the user’s identity. This is known as context-aware authorization.

Once authenticated and authorized, the front end can directly call the data privacy vault to reveal the true values of the user’s account information. But the front end only has access to this singular row of data and it’s limited to the few columns needed to render the information on the account page.

Example of revealing sensitive data for a single record.Sharing Sensitive DataNo modern application exists in a silo. Most applications need to share customer PII with third party services to send emails, SMS, issue a payment, or some other type of workflow. This is also supported by the vault architecture by using the vault as a proxy to the third party service.

In this case, instead of calling a third party API directly, you call the data privacy vault with the de-identified data. The vault knows how to re-identify the PII securely within its environment, and then securely share that with the third party service.

An example of this flow for sending HIPAA compliant forms of communication is shown below. The backend server calls the vault directly with tokenized data and the vault then shares the actual sensitive data with the third party communication service.

Example of using a vault to send HIPAA compliant communication.Final ThoughtsWe’ve come a long way since building business applications in the 1980s, but we’ve failed to evolve our thinking regarding how we secure and manage customer PII. Point solutions like firewalls, encryption, and tokenization alone aren’t enough to address the fundamental problem. We need a new approach to cut to the root of the Cheese and Diamond Problem.

Not all data is the same, PII belongs in a data privacy vault.The data privacy vault provides such an approach.

It’s an architectural approach to data privacy where security is the default. Multiple techniques like polymorphic encryption, confidential computing, tokenization, data governance, and others combine with the principle of isolation and zero trust to give you all the tools you need to store and use PII securely without exposing your systems to the underlying data.

If you have comments or questions about this approach, please connect with me on LinkedIn. Thanks for reading!

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.The post The Data Cloud’s Cheese and Diamond Problem appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of KubeCon 2023, which took place from November 6th through 9th in Chicago.

In today’s interview, host Jordi Mon Companys speaks with Justin Cormack who is the CTO at Docker.Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Please click here to see the transcript for this episode.Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post KubeCon Special: Docker with Justin Cormack appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of KubeCon 2023, which took place from November 6th through 9th in Chicago.

In today’s interview, host Jordi Mon Companys speaks with Santiago Torres-Arias who is a contributor to Sigstore, which is a system to register software supply chain actors using federated identity management.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Please click here to see the transcript for this episode.Sponsorship inquiries: sponsor@softwareengineeringdaily.com
The post KubeCon Special: Sigstore with Santiago Torres-Arias appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of KubeCon 2023, which took place from November 6th through 9th in Chicago.In today’s interview, host Jordi Mon Companys speaks with Darren Shepherd who is the Chief Architect and Co-Founder at Acorn Labs.Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Please click here for the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post KubeCon Special: Acorn with Darren Shepherd appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of KubeCon 2023, which took place from November 6th through 9th in Chicago.In today’s interview, host Jordi Mon Companys speaks with David DeSanto who is the Chief Product Officer at GitLab.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Please click here for the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post KubeCon Special: GitLab’s AI Vision with David DeSanto appeared first on Software Engineering Daily.

Josh Prismon is a veteran software architect, having worked at FICO for 17 years before shifting to Index Exchange in 2022.

In this episode, Josh joins the podcast to speak with host Lee Atchison, who also has deep experience in software architecture from his time at Amazon, New Relic, and other companies.

Josh and Lee discuss different paradigms for thinking about software architecture, including the importance of storytelling to build a shared vision for architecture across a team. They also talk about the tension between “emergent” and “intentional” design, how to allow architecture to evolve dynamically without sacrificing structure, and more.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here for the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Software Architecture with Josh Prismon appeared first on Software Engineering Daily.

Data breaches at major companies are so now common that they hardly make the news. The Wikipedia page on data breaches lists over 350 between 2004 and 2023. The Equifax breach in 2017 was especially notable because over 160 million records were leaked, and much of the data was acquired by Equifax without individuals’ knowledge or consent. Data breaches are increasingly costly to companies and to affected users who must deal with the ensuing identity theft.In 2018 the European Union implemented the General Data Protection Regulation, or GDPR. Despite its mild name, the GDPR had major consequences for individuals’ rights to control their data, and for companies that operate in the EU. Among other things, it gives the right to have personal data removed from a company’s records. This is the so-called “right to be forgotten”.

Gal Ringel is the Co-Founder and CEO of Mine, which allows users to identify which companies have their data, and it automates the process of removing data on behalf of its users. In this way, Mine aims to reduce online exposure and minimize risk for anyone using online services. Gal joins the show today to talk about his company, the impact of GDPR, and how his experience in military intelligence, venture capital, and tech led him to co-found the company.Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Please click here to see the transcript of this episode.Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post The Right to Be Forgotten with Gal Ringel appeared first on Software Engineering Daily.

If you’re a sports fan and like to track sports statistics and results, you’ve probably heard of Sofascore. The website started in 2010 and ran on a modest single server. It now has 25 million monthly active users, covers 20 different sports, 11,000 leagues and tournaments, and is available in over 30 languages.Josip Stuhli has been with Sofascore for 13 years. He started there as an engineer and is currently CTO. Josip joins the show today to talk about the challenges Sofascore encountered over the years, and how the team solved them. He discusses dealing with traffic spikes from game days, structuring and restructuring the codebase, organizing the frontend and backend, and much more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please clicke here for the full transcript of this episodeSponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Sofascore with Josip Stuhli appeared first on Software Engineering Daily.

As Large Language Models (LLMs) and generative AI continue to grow more sophisticated and available, many organizations are starting to build, fine-tune, and customize LLMs based on their internal data and documents. This can bring incredible efficiency and reliability to data-driven decision-making processes. However, this practice comes with its share of challenges, primarily around data privacy, protection, and governance.

Let’s consider the construction of the LLM itself, which is trained on a massive amount of data collected from public and private sources. Without careful anonymization and filtering, sensitive data — such as PII or intellectual property — may be inadvertently included in the training set, potentially leading to a privacy breach.

Furthermore, privacy concerns are introduced when interacting with LLMs, as users might input sensitive data, such as names, addresses, or even confidential business information. If these inputs aren’t handled properly, the misuse or exposure of this information is a genuine risk.

In this post, we’ll explore how to work with LLMs in a privacy-preserving way when building an LLM-based chatbot. As we walk through the technology from end-to-end, we’ll highlight the most acute data privacy concerns and we’ll show how using a data privacy vault addresses those concerns.

Let’s start by taking a closer look at the problem we need to solve.

The problem: Protecting sensitive information from exposure by a chatbotConsider a company that has uses an LLM-based chatbot for its internal operations. The LLM for the chatbot was built by modifying a pre-existing base model with embeddings created from internal company documents. The chatbot provides an easy-to-use interface that lets non-technical users within the company access information from internal data and documents.

The company has a sensitive internal project called “Project Titan.” Project Titan is so important and so sensitive that only people working on Project Titan know about it. In fact, the team often says: the first rule of Project Titan is don’t talk about Project Titan. Naturally, the team wants to take advantage of the internal chatbot and also include Project Titan specific information to speed up creation of design documents, documentation, and press releases. However, they need to control who can see details about this sensitive project.

What we have is a tangible and pressing privacy concern that sits at the intersection of AI and data. These challenges appear extremely difficult to solve in a scalable and production-ready way. Simply having a private version of the LLM doesn’t address the core issue of data access.

The proposed solution: Sensitive data de-identification and fine-grained access controlUltimately, we need to identify the key points where sensitive data must be de-identified during the process of building (or fine-tuning) the LLM and the end user’s interaction with the LLM-based chatbot. After careful analysis, we’ve identified that there are two key points in the process where we need to de-identify (and later re-identify) sensitive data:

Before ingestion: When documents from Project Titan are used to create embeddings, the project name, any PII, and anything else sensitive to the project must be de-identified. This de-identification should occur as part of the ETL pipeline prior to data ingestion into the LLM.
During use: When a user inputs data to the chatbot, any sensitive data included in that input must also be de-identified.

You can de-identify sensitive data using Skyflow’s polymorphic encryption and tokenization engine that’s included within Skyflow Data Privacy Vault. This includes detection of PII but also terms you define within a sensitive data dictionary, like intellectual property (i.e. Project Titan).

Of course, only Project Titan team members who use the chatbot should be able to access the sensitive project data. Therefore, when the chatbot forms a response, we’ll rely on Skyflow’s governance engine (which provides fine-grained access control) and detokenization API to retrieve the sensitive data from the data privacy vault, making it available only to authorized end users.

Before we dive into the technical implementation, let’s go through a brief overview of foundational LLM concepts. If you’re already familiar with these concepts, you can skip the next section.

A brief primer on LLMsLLMs are sophisticated artificial intelligence (AI) systems designed to analyze, generate, and work with human language. Built on advanced machine learning architectures, they are trained on vast quantities of text data, enabling them to generate text that is convincingly human-like in its coherence and relevance.

LLMs leverage a technology called transformers — one example is GPT, which stands for Generative Pre-Trained Transformer — to predict or generate a piece of text when given input or context. LLMs learn from patterns in the data they are trained on and then apply these learnings to understand newly given content or to generate new content.

Despite their benefits, LLMs pose potential challenges in terms of privacy, data security, and ethical considerations. This is because LLMs can inadvertently memorize sensitive information from their training data or generate inappropriate content if not properly regulated or supervised. Therefore, the use of LLMs necessitates effective strategies for data handling, governance, and preserving user privacy.

A technical overview of the solutionWhen embarking on any LLM project, we need to start with a model. Many open-source LLMs have been released in recent months, each with its specific area of focus. Instead of building an entire LLM model from scratch, many developers choose a pre-built model and then adjust the model with vector embeddings generated from domain-specific data.

Vector embeddings encapsulate the semantic relationship between words and help algorithms understand context. The embeddings act as an additional contextual knowledge base to help augment the facts known by the base model.

In our case, we’ll begin with an existing model from Hugging Face, and then customize it with embeddings. Hugging Face provides ML infrastructure services as well as open-source models and datasets.

In addition to the Hugging Face model, we’ll use the following additional tools to build out our privacy-preserving LLM-based ETL pipeline and chatbot:

LangChain an open-source Python library that chains together components typically used for building applications (such as chatbots) powered by LLMs
Snowflake, which we’ll use for internal document and data storage
Snowpipe, which we’ll use with Snowflake for automated data loading
Chroma, an AI-native, open-source database for vector embeddings
Streamlit, an open-source framework for building AI/ML-related applications using Python
RetrievalQA, a question-answering chain in LangChain which gets documents from a Retriever and then uses a QA chain to answer questions from those documents

The following diagram shows the high-level ETL and embeddings data flow:

Example of the ETL and embeddings data flow.The ETL and embeddings flows from end to end are:

ETL

Start with source data, which may contain sensitive data.
Send data to Skyflow Data Privacy Vault for de-identification.
Use Snowpipe to load clean data into Snowflake.

Create vector embeddings

Load documents from Snowflake into LangChain.
Create vector embeddings with LangChain.
Store embeddings in Chroma.

Once the model has been customized with the Project Titan information, the user interaction and inference flow is as follows:

User interaction and inference information flow

Chat UI input
Accept user input via Streamlit’s chat UI.
Send user input to Skyflow for de-identification.

2. Retrieve embeddings

Get the embeddings from Chroma and attach to RetrievalQA.

3. Inference

Send clean data to RetrievalQA.
Use QA chain in RetrievalQA to answer the user’s question.

4. Chat UI response

Send RetrievalQA’s response to Skyflow for detokenization.
Send re-identified data to Streamlit for display to the end user.

Now that we’re clear on the high-level process, let’s dive in and take a closer look at each step.

ETL: Cleaning the source dataCleaning the source data with Skyflow Data Privacy Vault is fairly straightforward and I’ve covered some of this in a prior post. In this case, we need to process all the source documents for Project Titan available in an AWS S3 bucket.

Skyflow will store the raw files, de-identify PII and IP, and save the clean files to another S3 bucket.

import boto3from skyflow.vault import ConnectionConfig, Configuration, RequestMethod# Authentication to Skyflow APIbearerToken = ''def tokenProvider(): global bearerToken if is\_expired(bearerToken): return bearerToken bearerToken, \_ = generate\_bearer\_token('<YOUR\_CREDENTIALS\_FILE\_PATH>') return bearerTokendef processTrainingData(trainingData): try: # Vault connection configuration config = Configuration('<YOUR\_VAULT\_ID>', '<YOUR\_VAULT\_URL>', tokenProvider) # Define the connection API endpoint connectionConfig = ConnectionConfig('<YOUR\_CONNECTION\_URL>', RequestMethod.POST, requestHeader = { 'Content-Type': 'application/json', 'Authorization': '<YOUR\_CONNECTION\_BASIC\_AUTH>' } requestBody = { 'trainingData': trainingData } # Connect to the vault client = Client(config) # Call the Skyflow API to de-identify the training data response = client.invoke\_connection(connectionConfig) # Define the S3 bucket name and key for the file bucketName = "clean-data-bucket" fileKey = "{timestamp}-{generated-uuid}" # Write the data to a file in memory fileContents = bytes(response.training\_data.encode("UTF-8")) # Upload the file to S3 s3 = boto3.client("s3") s3.put\_object(Bucket=bucketName, Key=fileKey, Body=fileContents) except SkyflowError as e: print('Error Occurred:', e) Next, we’ll configure Snowpipe to detect new documents in our S3 bucket and load that data into Snowflake. To do this, we’ll need to create the following in Snowflake:

A new table
A file format
A stage
A new pipe

CREATE OR REPLACE TABLE custom\_training\_data ( training\_text BINARY );CREATE OR REPLACE FILE FORMAT training\_data\_json\_format TYPE = JSON;CREATE OR REPLACE TEMPORARY STAGE training\_data\_stage FILE\_FORMAT = training\_data\_json\_format;CREATE PIPE custom\_training\_data AUTO\_INGEST = TRUE AS COPY INTO custom\_training\_data FROM (SELECT $1:records.fields.training\_text FROM @ training\_data\_stage t) ON\_ERROR = 'continue'; With that, we have raw data that goes through a de-identification process, and then we store the plaintext sensitive data in Snowflake. Any sensitive data related to Project Titan is now obscured in the LLM, but because of Skyflow’s polymorphic encryption and tokenization, the de-identified data has referential integrity, meaning we can return the data to its original form when interacting with the chatbot.

Creating vector embeddings: Customizing our LLMNow that we have our de-identified text data stored in Snowflake, we’re confident that all information related to Project Titan has been properly concealed. The next step is to create embeddings of these documents.

We’ll use the Instructor model provided by Hugging Face as our embedding model. We store our embeddings in Chroma, a vector database built expressly for this purpose. This will allow for the downstream retrieval and search support of the textual data stored in our vector database.

The code below loads the base model, embedding model, and storage context.

from langchain.chat\_models import ChatOpenAIfrom langchain.embeddings import HuggingFaceEmbeddingsfrom langchain.embeddings.openai import OpenAIEmbeddingsmodel\_id = "hkunlp/instructor-large"embed\_model = HuggingFaceEmbeddings(model\_name=model\_id)vectorstore = Chroma("langchain\_store", embed\_model) Next, we need to load all documents and add them to the vector store. For this, we use the Snowflake document loader in LangChain.

from snowflakeLoader import SnowflakeLoaderimport settings as sQUERY = "select training\_text as source from custom\_training\_data"snowflake\_loader = SnowflakeLoader( query=QUERY, user=s.SNOWFLAKE\_USER, password=s.SNOWFLAKE\_PASS, account=s.SNOWFLAKE\_ACCOUNT, warehouse=s.SNOWFLAKE\_WAREHOUSE, role=s.SNOWFLAKE\_ROLE, database=s.SNOWFLAKE\_DATABASE, schema=s.SNOWFLAKE\_SCHEMA, metadata\_columns=["source"],)training\_documents = snowflake\_loader.load()vector\_store.add\_documents(training\_documents) With the training document and vector store created, we create the question-answering chain.

qa = RetrievalQA.from\_chain\_type(llm=ChatOpenAI(temperature=0.2,model\_name='gpt-3.5-turbo'), chain\_type="stuff", retriever=vector\_store.as\_retriever())result = qa.run("What is Project Titan?") This question (“What is Project Titan?”) will fail because the model doesn’t actually know about Project Titan, it knows about a de-identified version of the string “Project Titan”.

To issue a query like this, the query needs to be first sent through Skyflow to de-identify the string and then the de-identified version is passed to the model. We’ll tackle this next as we start to put the pieces together for our chat UI.

Chat UI Input: Preserving privacy of user-supplied dataWe’re ready to focus on the chatbot UI aspect of our project, dealing with accepting and processing user input as well as returning results with Project Titan data detokenized when needed.

For this portion of the project, we will use Streamlit for our UI. The code below creates a simple chatbot UI with Streamlit.

import openaiimport streamlit as stst.title("Acme Corp Assistant")# Initialize the chat messages historyif "messages" not in st.session\_state.keys(): st.session\_state.messages = [ {"role": "assistant", "content": "Hello ! \nHow can I help?"} ]# Prompt for user input and saveif prompt := st.chat\_input(): st.session\_state.messages.append({"role": "user", "content": prompt})# display the prior chat messagesfor message in st.session\_state.messages: with st.chat\_message(message["role"]): st.write(message["content"])# If last message is not from assistant, we need to generate a new responseif st.session\_state.messages[-1]["role"] != "assistant": # Generate a response with st.chat\_message("assistant"): with st.spinner("Thinking..."): response = "TODO" message = {"role": "assistant", "content": response} st.session\_state.messages.append(message) Our simple chat UI looks like this:

As you can see, the UI accepts a user input, but doesn’t currently integrate with our LLM. Next, we need to send the user input to Skyflow for de-identification before we use RetrievalQA to answer the user’s question. Let’s start with accepting and processing our input data.

To detect and de-identify plaintext sensitive data with Skyflow, we can use the detect API endpoint with code similar to the following:

def deIdentifyText(input): data = { "text": [ { "message": input } ], "deidentify\_option": "tokenize" } response = client.detect(data) return response[0].processed\_text Now that we’ve de-identified the user input data, we can send the question to RetrievalQA, which will then use a QA chain to answer the question from our documents.

def performCompletion(input): clean\_input = deIdentifyText(input) qa = RetrievalQA.from\_chain\_type(llm=ChatOpenAI(temperature=0.2,model\_name='gpt-3.5-turbo'), chain\_type="stuff", retriever=vector\_store.as\_retriever()) return qa.run(clean\_input) We now have our response from RetrievalQA. However, we need to take one additional step before we can send it back to our user: detokenize (re-identify) our response through Skyflow’s detokenization API. This is fairly straightforward, similar to previous API calls to Skyflow.

Everything we need is encapsulated by the function performInference, which calls a function to reIdentifyText after the completion is returned.

Who can see what and in which format is controlled by Skyflow’s governance engine. There’s too much to cover here, but if you want to learn more, see Introducing the Skyflow Data Governance Engine.

def performInference(input): response = performCompletion(input) return reIdentifyText(response) These final steps connect our entire application from end-to-end. Now, we need to update our UI code from above so that the response is correctly set.

```

If last message is not from assistant, we need to generate a new responseif st.session_state.messages[-1]["role"] != "assistant": # Generate a response with st.chat_message("assistant"): with st.spinner("Thinking..."): response = performInference(m["content"])

``` With these pieces in place, here’s a quick demo of our privacy-preserving LLM-based chatbot in action:

Example of the privacy-preserving bot in action.Tying it all togetherIn this article, we walked through the general steps to construct a privacy-preserving LLM-based chatbot. With organizations increasingly using LLM-based applications in their businesses and operations, the need to preserve data privacy has become acute. Concerns about protecting the privacy and security of sensitive data are the biggest adoption blocker that prevents many companies from making full use of AI with their datasets.

Solving this problem requires identifying the key points where sensitive data might enter your system and need to be de-identified. When working with LLMs, those points occur during model training — both when building an LLM or customizing one — and at the user input stage. You can use Skyflow Data Privacy Vault to implement effective de-identification and data governance for LLM-based AI tools like chatbots.

Building an LLM-based chatbot requires the use of several tools to ensure that data is handled in a manner that preserves privacy. Taking privacy-preserving measures is critical to prevent the misuse or exposure of sensitive information. By using the tools and methods we’ve demonstrated here, companies can leverage AI’s benefits and promote efficient data-driven decision-making while prioritizing data privacy and protection.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.The post Building a Privacy-Preserving LLM-Based Chatbot appeared first on Software Engineering Daily.

Cloud-based software development platforms such as GitHub Codespaces continue to grow in popularity. These platforms are attractive to enterprise organizations because they can be managed centrally with security controls. However, many, if not most, developers prefer a local IDE.Daytona is aiming to bridge that gap. It’s a layer between a local IDE and a backend server, so developers can work locally while interfacing invisibly with a remote environment. Ivan Burazin is the CEO and Co-Founder at Daytona, and he joins the show today to talk about how Daytona works, Spotify as an inspiration for his product, and more.Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Please click here to see the transcript of this episode.Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Daytona with Ivan Burazin appeared first on Software Engineering Daily.

Knowledge graphs are an intuitive way to define relationships between objects, events, situations, and concepts. Their ability to encode this information makes them an attractive database paradigm.

Hume is a graph-based analysis solution developed by GraphAware. It represents data as a network of interconnected entities and provides analysis capabilities to extract insights from the data. Luanne Misquitta is VP of Engineering at GraphAware and she joins the show today to talk about graph databases, and the engineering of Hume.Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne.

Please click here to view this show’s transcript.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post GraphAware with Luanne Misquitta appeared first on Software Engineering Daily.

Shopify is an e-commerce platform focused on enabling small businesses to sell online. The company was founded in 2006 and since then has become a core technology of online business infrastructure.

Mike Shaver is a Distinguished Engineer at Shopify and previously worked at Facebook, Mozilla, Oracle and others. At Shopify he works on the core team, which handles store fronts, merchant experience, and the commerce engine. He joins the show to talk about Shopify’s bet on Rust, their shift to Google Cloud, and the experience of changing from a management role to a high-level individual contributor role.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.

Please click here for the full transcript of this episode.Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Shopify with Mike Shaver appeared first on Software Engineering Daily.

Data residency requirements, which govern where sensitive data can be stored or processed in the cloud (or in an on-prem server) are a common feature of many modern data protection laws. Because of data residency requirements, the location of sensitive data has significant regulatory compliance implications in countries and regions around the world.

In this post, we’ll look at the challenges of managing data residency with Snowflake. We’ll start by examining how Snowflake Cloud Regions address data residency challenges, and consider the compliance implications of this approach — especially when loading data from cloud storage. Then, we’ll look at how to simplify data residency compliance using one or more regional data privacy vaults.

Let’s begin with a deeper dive into data residency, and how it impacts compliance.

The implications of data residency on complianceWhen you work with personally identifiable information (PII), where you store and process this information has a direct impact on your legal compliance requirements. Some jurisdictions have regulations that govern the protection and privacy of their residents’ PII, restricting how and where it’s used by businesses and other organizations.

For example, the personal data (i.e., PII) of European Union residents cannot be transferred outside the EU without appropriate safeguards.

The laws of each jurisdiction impact how you transmit, manage, process, and store sensitive data in that jurisdiction. Because data residency dictates where (geographically ) data is stored in the cloud, data residency becomes a critical concern in cloud environments that handle sensitive data.

Choose your cloud region carefullyCloud service providers have data centers located in multiple regions around the world. When businesses sign up for cloud services and configure storage regions and other tooling, they select specific regions where their data is stored.

For many businesses, the selection of regions and locations for data storage is an afterthought.

But, treating this decision as an afterthought is a costly mistake that can come back to haunt you if you’re handling sensitive data. That’s because choosing storage regions is a weighty decision that can have a long-term impact on compliance, and on your business operations.

Snowflake Cloud Regions: a data residency solution?Snowflake Cloud Regions let you choose the geographic location where your Snowflake data is stored across the data centers provided by the Snowflake-supported public cloud providers — AWS, GCP, and Azure. Each cloud provider offers a set of regions across the globe, with specific geographic data center locations in each cloud provider region.

Source: Snowflake Documentation Supported Cloud RegionsIf your company uses Snowflake Cloud Regions, you have your choice of providers, as well as regions where your data can be stored. When you create an account to deploy and set up Snowflake, whichever region you select becomes the primary location for data storage and for data processing resources.

At first glance, it might seem like Snowflake Cloud Regions provides a simple, effective solution to your data residency and compliance concerns. But for global companies who need global analytics, it isn’t that simple. That’s because, as noted in the Snowflake Cloud Regions documentation:

Each Snowflake account is hosted in a single region. If you wish to use Snowflake across multiple regions, you must maintain a Snowflake account in each of the desired regions.

This means that for each region where your business operates that has data residency requirements, you’ll need a different Snowflake account hosted in that region. Compliance becomes increasingly complex as you scale globally to more and more regions around the world. With this approach, running global analytics operations across different accounts to get a comprehensive view of your business can be a massive and ongoing challenge.

Instead of managing multiple Snowflake accounts with multiple Snowflake instances distributed in various regions around the world, you’d rather maintain a Snowflake instance in a single region to support global data operations. However, you still need to consider the need to honor data residency requirements for sensitive data so you can uphold your compliance obligations and safeguard customer trust.

For example, if you collect the personal data (PII) of customers located in the EU, but your Snowflake instance is located somewhere else, then you need to think through the privacy and compliance impact of storing and processing that data.

Loading data from cloud storage into SnowflakeSnowflake also lets businesses load data from cloud storage services like AWS S3, Google Cloud Storage, Microsoft Azure — regardless of which cloud platform hosts the businesses’ Snowflake account. This can present additional challenges when working to ensure data residency compliance.

For example, let’s say that your company collects PII from both US and EU customers using its website. And, let’s say that this sensitive data is then stored in a Google Cloud Storage bucket that’s located in the AUSTRALIA-SOUTHEAST1 (Sydney) region.

How does transmitting this PII data to Australia, and then storing it in Australia, affect your compliance with regulations like the EU’s GDPR?

The answer is: doing this likely puts you out of compliance with GDPR. This is just one example of how the location where sensitive data is stored — and where it’s processed and replicated — complicates the compliance requirements faced by businesses that handle sensitive PII.

Businesses that handle PII must ensure regulatory compliance by aligning their choice of cloud storage regions with the data residency requirements of markets where they operate.

And beyond compliance issues, businesses should also consider data transfer costs. Transferring data between cloud storage regions can incur significant additional costs, especially if your company is frequently transferring large volumes of data. So, we not only have compliance concerns with cross-border transfers of PII, we also have a cost concern.

So, to briefly recap our problem:

Countries and regions have their own laws and regulations that govern how to handle their residents’ sensitive data (PII).
The geographic location where your business stores and processes sensitive data impacts whether you’re compliant with the data residency requirements of the markets where you operate.
If you use Snowflake to perform analytics on PII, then the complexity of meeting your compliance obligations will depend on the location of your Snowflake account.
If you load PII data into Snowflake from cloud storage, then your compliance obligations are also impacted by the location of your cloud storage.

So, how can we meet data residency requirements, support global analytics operations, and remove the operational overhead of managing multiple Snowflake accounts and instances?

We can solve our data residency problems and protect sensitive data with one or more data privacy vaults.

How a data privacy vault simplifies data privacyA data privacy vault isolates, protects, and governs access to sensitive customer data. Sensitive data is stored in the vault, while opaque tokens that serve as references to this data are stored in traditional cloud storage or used in data warehouses. A data privacy vault can store sensitive data in a specific geographic location, and tightly controls access to this data. Other systems only have access to non-sensitive tokenized data.

In the example architecture shown below, a phone number is collected by a front end application. Ideally, we should de-identify (i.e., tokenize) this sensitive information as early in the data lifecycle as possible. A data privacy vault lets us do just that.

This phone number, along with any other PII, is stored securely in the vault, which is isolated outside of your company’s existing infrastructure. Any downstream services — the application database, data warehouse, analytics, any logs, etc. — store only a token representation of the data, and are removed from the scope of compliance:

Example of reducing compliance scope with a data privacy vaultSnowflake handles only de-identified dataBecause no sensitive data is stored outside the data privacy vault, your compliance scope is restricted to just the vault. This removes the compliance burden from your Snowflake instance.

Example pipeline where sensitive data is isolated and protected within a data privacy vaultTo satisfy data residency requirements, we can extend this approach by using multiple regional data privacy vaults placed near customers whose data is subject to these requirements. With sensitive data stored in these data privacy vaults, Snowflake contains only de-identified, tokenized data. It no longer matters if you operate a single global instance of Snowflake or multiple Snowflake accounts across different regions because data residency concerns no longer apply to your Snowflake instances.

Compliance with data residency requirements now depends solely on where your data privacy vaults are located. You no longer need to worry about data residency for all the different parts of your data tech stack, including cloud storage and Snowflake. All sensitive data goes into your data privacy vaults, and these vaults become the only component of your architecture subject to data residency requirements.

Store PII in a data privacy vault in a specific regionWith Skyflow Data Privacy Vault you can host your vaults in a wide variety of regions around the world. You can also route sensitive data to a data privacy vault located in a specific region for storage.

For example, consider how the application architecture shown below supports data residency requirements from multiple regions:

Using vaults to satisfy multiple data residency requirements for one Snowflake instance1. Your company’s e-commerce site collects customer PII whenever a customer places an order. 2. On the client side, the website detects the customers’ location. 3. Detecting that the customer is in the EU, the client-side code uses Skyflow’s API to send the PII data to your company’s data privacy vault in Frankfurt, Germany.
Note: For customers based in the US, the PII data is instead routed to the data privacy vault in the US (in this case, Virginia). 4. This EU-based customer’s sensitive PII is stored in the EU-based data privacy vault, and Skyflow’s API responds with tokenized data. 5. The client-side code sends the customer order request, now with tokenized data, to the server. 6. The server processes the order, storing the data (now de-identified and tokenized) in cloud storage in the “Oregon, US” region. 7. At the end of the week, your company’s Snowflake instance in Tokyo, Japan, loads the data (already de-identified and tokenized) from cloud storage to perform analytics.

By using multiple vaults located in different regions around the world, you can easily manage all of your sensitive data to meet various data residency compliance obligations across each of your global markets.

The data privacy vault architectural pattern vastly simplifies the challenges of data residency and compliance. Additionally, by de-scoping Snowflake from the compliance burden of data residency, global analytics executes as normal — within a single Snowflake instance.

Final thoughtsCompliance regulations and their data residency requirements require that businesses uphold stringent standards for data localization, protection, privacy, and security to reduce their risk of breaches, penalties, and reputational damage. However, businesses with customers (and data) located in a variety of global regions face the added challenge of managing multiple regulations across jurisdictions.

Using data privacy vaults lets businesses simplify their global compliance obligations around data residency as they relate to Snowflake and cloud storage.

Using a data privacy vault, companies can isolate and secure all sensitive data in one or more data privacy vaults, removing Snowflake and cloud storage from their compliance footprint. At the same time, by leveraging data privacy vaults in different regions, companies can help ensure that sensitive data is stored and transmitted according to the laws and regulations of each specific region where they operate.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.The post One Snowflake, Multiple Vaults: A Solution to Data Residency appeared first on Software Engineering Daily.

When Adam Berger was at Uber, his team was responsible for ensuring that Uber Eats merchants correctly receive and fulfill orders. This required them to think hard about engineering workflows and state management systems. Six years of experience at Uber motivated Adam to create State Backed, which is an open-source backend system written in Typescript. The platform is oriented around using state machines to model application logic, and automatically handles the associated persistence, infrastructure, and consistency.

Adam joins the show to talk about state machines, why they’re the right paradigm to manage global application state, and what are the practical advantages of using state machines in a backend platform.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Building a State Machine Backend with Adam Berger appeared first on Software Engineering Daily.

One of the key challenges that teams encounter is how to smoothly collaborate on converting a design into code. For example, if a designer designs a web component, how can it be most efficiently implemented by a developer? What happens if the designer needs to adjust the design and communicate this change to the developer? These sorts of issues can often lead to inefficiencies and frustrations on a team.

Figma recently announced Dev Mode which aims to smooth design and developer collaboration. The idea is to tie the visual language of designers to the actual component implementation of the developer.

Marcel Weekes is VP of Product Engineering at Figma, and before that he spent 6 years at Slack. Marcel joins the show today to talk about Dev Mode, how it will boost collaboration between designers and devs, and the new Figma VS Code plugin that brings design into the IDE. Marcel also talks about the concept of the “new manager death spiral”, and how individual contributors, or ICs, can smoothly transition to management roles.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript.

Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.You can find Josh on: Bluesky, Fosstodon, Twitter, Twitch, YouTube, and joshuakgoldberg.com.

Please click here to see the transcript for this episode.Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Figma Dev Mode with Marcel Weekes appeared first on Software Engineering Daily.

Frontend web frameworks are software toolkits that handle many of the low-level and repetitive aspects of building a website. These frameworks have made it easier than ever to build a modern website. The open-source Astro framework was created in 2021 for the purpose of creating simple static sites that load quickly. A key factor to its high performance is that, by default, webpages don’t include any JavaScript. However, Astro gives users the ability to opt into using JavaScript wherever it makes sense in their projects. Astro also has the file based routing and server-side rendering capabilities found in frameworks like NextJS.

James Q. Quick has worked at Microsoft, PlanetScale, and Auth0 and is a popular technical content creator. He’s an advocate of the Astro framework and joins the show to talk about what distinguishes it from NextJS, Sveltekit, and other major frameworks.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Please click here for the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post The Astro Framework with James Quick appeared first on Software Engineering Daily.

Observability software helps teams to actively monitor and debug their systems, and these tools are increasingly vital in DevOps. However, it’s not uncommon for the volume of observability data to exceed the amount of actual business data. This creates two challenges – how to analyze the large stream of observability data, and how to keep down the compute and storage costs for that data.

Chronosphere is a popular observability platform that works by identifying the data that’s actually being used to power dashboards and metrics. It then shows the cost for each segment of data, and allows users to decide if a metric is worth that cost. In this way, technical teams can manage costs by dynamically adjusting which data is analyzed and stored. Martin Mao is the Co-founder and CEO of Chronosphere and he joins the podcast today to talk about the growing challenge of managing observability data, and the design of Chronosphere.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.Please click here to see the transcript for this episode.Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Chronosphere with Martin Mao appeared first on Software Engineering Daily.

The open source coding philosophy has enormous appeal to many software engineers, and with good reason. Open source libraries, applications, and operating systems are now essential to the overall technology ecosystem. And the number of open source projects is only increasing. But many developers don’t know how to get involved in open source. Or, they may have even faced resistance when trying to make a pull request to their favorite open source codebase.

Open Sauced is a platform to help developers get involved in open source development. While the number of GitHub stars on a project is often seen as a metric of success for a code base, Open Sauced focuses on the number of new contributors on a project. This number serves as a signal to help drive the platform’s recommendation system, which pairs its users with open source projects in need of developers. Brian Douglas is a former Developer Experience Lead at Netlify and he was the Director of Developer Advocacy at GitHub. He is also the founder and CEO of Open Sauced and he is our guest in this episode.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript. Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.You can find Josh on: Bluesky, Fosstodon, Twitter, Twitch, YouTube, and joshuakgoldberg.com.Please click here to see the transcript for this episode.Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Open Source Contributing with Brian Douglas appeared first on Software Engineering Daily.

There are countless real world scenarios where a workflow or process has multiple steps, and some steps must be completed before others can be started. Think of something as simple as cooking dinner. First you look up a recipe, then you write down the ingredients you need, you go shopping, and then you cook. These steps must be run in a certain order, and the state of the workflow must be tracked throughout. Workflow management is everywhere in the software world, and today it’s common for teams to engineer custom solutions. This makes sense, because creating a general-purpose solution for workflow management is a hard conceptual problem, and perhaps an even harder engineering challenge.

Maxim Fateev has a deep background engineering distributed systems and workflow management services at Google, Amazon, and Microsoft. In 2015, he joined Uber and helped create the open-source project, Cadence, which is an orchestration engine to execute asynchronous long-running business logic. The success of Cadence led Max to co-found Temporal, which is an open-source programming package for workflow execution. Max joins the show today to talk about the engineering challenges at Temporal, the concept of “durable execution”, how he organizes his engineering teams, and more.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.Please click here to see the transcript for this episode.Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Temporal with Max Fateev appeared first on Software Engineering Daily.

Speech technology has been around for a long time, but in the last 12 months it’s undergone a quantum leap. New speech synthesis models are able to produce speech that’s often indistinguishable from real speech. I’m sure many listeners have heard deep fakes where computer speech perfectly mimics the voice of famous actors or public figures. A major factor in driving the ongoing advances is generative AI.

Speechlab is at the forefront of using new AI techniques for realtime dubbing, which is the process of converting speech from one language into another. For the interested listener, we recommend hearing the examples with President Obama speaking Spanish or Elon Musk speaking Japanese in this YouTube video. Ivan Galea is the Co-founder and President at Speechlab and he joins the show to talk about how we’re on the cusp of reaching the holy grail of speech technology – real time dubbing – and how this will erase barriers to communication and likely transform the world.This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.Please click here to view this show’s transcript.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Speechlab and Realtime Translation with Ivan Galea appeared first on Software Engineering Daily.

If you’re a developer, there’s a good chance you’ve experimented with coding assistants like GitHub Copilot. Many developers have even fully integrated these tools into their workflows. One way these tools accelerate development is by autocompleting entire blocks of code. The AI achieves this by having awareness of the surrounding code. It understands context. However, in many cases the context available to an AI is limited. This restricts the AI’s ability to suggest more sweeping changes to a codebase, or even to refactor an entire application.

Quinn Slack is the CEO of Sourcegraph. He is now hard at work on the challenge of giving more context to AI – to make it aware of entire codebases, dependencies, error logs, and other data. Quinn joins the show today to talk about what it takes to move beyond code autocomplete, how to develop the next generation of coding AI, and what the future looks like for software engineers and programming languages.

Josh Goldberg is an independent full time open source developer in the TypeScript ecosystem. He works on projects that help developers write better TypeScript more easily, most notably on typescript-eslint: the tooling that enables ESLint and Prettier to run on TypeScript code. Josh regularly contributes to open source projects in the ecosystem such as ESLint and TypeScript.Josh is a Microsoft MVP for developer technologies and the author of the acclaimed Learning TypeScript (O’Reilly), a cherished resource for any developer seeking to learn TypeScript without any prior experience outside of JavaScript. Josh regularly presents talks and workshops at bootcamps, conferences, and meetups to share knowledge on TypeScript, static analysis, open source, and general frontend and web development.You can find Josh on: Bluesky, Fosstodon, Twitter, Twitch, YouTube, and joshuakgoldberg.com.Please click here to view this show’s transcript.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Sourcegraph with Quinn Slack appeared first on Software Engineering Daily.

Documentation is something that everyone knows is important but it’s often difficult to get right. On software teams, good documentation can help to onboard new people, improve communication across teams, and troubleshoot technical issues. When an application, API, or library is a commercial product, the quality of its documentation can determine whether it attracts users and succeeds on the market.

GitBook is a popular documentation platform built with TypeScript and Node. It’s found particular use among software teams for creating technical documentation. Addison Schultz is the Developer Relations Lead at GitBook and he joins the show to talk about GitBook’s development, software stack, and how it’s adding new integrations for VS Code, Slack, and other tools.

Full disclosure: GitBook is a sponsor of Software Engineering Daily.

Please click here to view this show’s transcript.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post GitBook with Addison Schultz appeared first on Software Engineering Daily.

Since its release by Facebook in 2013, React has etched itself into the canvas of web development. Fast-forward a decade, and it’s no exaggeration to say that React has become the most popular web framework, globally. The majority of the world’s web apps now have a consistent and composable approach for managing state in which

The post Time for backend architectures to…React appeared first on Software Engineering Daily.

Cloud applications continue to grow in popularity, but ensuring the security of these applications often presents a formidable engineering challenge. This challenge motivated the creation of Jit. Jit is a continuous security platform for developers, and seeks to enable every cloud app to start with minimum viable security, or MVS, without slowing development velocity. David

The post Minimum Viable Security for Cloud Apps with David Melamed appeared first on Software Engineering Daily.

ScyllaDB is a fast and highly scalable NoSQL database designed to provide predictable performance at a massive cloud scale. It can handle millions of operations per second at a scale of gigabytes or petabytes. It’s also designed to be compatible with Cassandra and DynamoDB APIs. Scylla is used by Zillow, Comcast, and for Discord’s 350M+ users, and it continues to gain popularity in other cloud-native and highly scalable cloud-centric workloads.

Dor Laor is the Co-founder and CEO of Scylla, and he’s our guest today.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Please click here to view this show’s transcript.

The post Highly Scalable NoSQL with Dor Laor appeared first on Software Engineering Daily.

React is an immensely popular JavaScript library that is used to build website user interfaces. A key feature of React is that it uses a virtual Document Object Model, or DOM, to selectively update the desired regions of the web page, which provides major performance advantages. Million.js is an open source project that provides an optimized virtual DOM. Remarkably, these optimizations make React up to 70% faster and the code weighs in at less than 4 kilobytes in size.Aiden Bai is the creator of Million.js and he joins us in this episode.Mike Bifulco is CTO and co-founder of Craftwork. He’s also a developer advocate, writer, podcaster and serial startup founder. In past lives, Mike worked for Google, Stripe, Microsoft, and Gymnasium. Mike is also co-founder of APIs You Won’t Hate, a community for API Developers on the web. Mike’s publishes a weekly newsletter for product builders called Tiny Improvements at mikebifulco.com. Mike is on Mastodon at https://hachyderm.io/@irreverentmike

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Please click here to view this show’s transcript.

The post Making React 70% faster with Aiden Bai of Million.js appeared first on Software Engineering Daily.

Cox Automotive has been at the frontier of digitizing the automotive space with popular products such as AutoTrader, Kelly Blue Book, and Dealertrack. To deliver these products the company manages large quantities of data and diverse engineering teams. Scaling its operations required an engineering transformation of the company. Chris Dillon is the VP of Architecture

The post Engineering Transformation at Scale with Chris Dillon appeared first on Software Engineering Daily.

Serverless backend platforms are cloud services that simplify the process of building a backend. These platforms are growing rapidly in popularity because they can greatly accelerate application development, and improve the developer experience. Convex is a real-time backend platform that uses 100% TypeScript and is designed with reactive UI frameworks in mind. The team behind

The post Building a full cloud backend with James Cowling appeared first on Software Engineering Daily.

Exafunction is a leader in deploying deep learning models at scale. One of their products is Codeium, a coding assistant for software developers based on Exafunction’s deep learning technology. Codeium provides AI-assisted autocompletion in your IDE, making it easier for you to incorporate deep learning technology in your software development workflow. Varun Mohan is the

The post Modern Coding Superpowers with Varun Mohan appeared first on Software Engineering Daily.

Tom Preston-Werner is a renowned software developer, inventor and entrepreneur. He co-founded GitHub and is the creator of the avatar service Gravatar, the TOML configuration file format, and the static site generator software Jekyll. Tom is currently working on the full-stack web framework, RedwoodJS. He joins us today to tell us the latest about RedwoodJS,

The post The Latest on RedwoodJS with Tom Preston-Werner appeared first on Software Engineering Daily.

Incident management is the process of managing and resolving unexpected disruptions or issues in software systems, especially those that are customer-facing or critical to business operations. Implementing a robust incident management system is often a key challenge in technical environments. Rootly is a platform to handle incident management directly from Slack, and is used by

The post Cross-functional Incident Management with Ashley Sawatsky and Niall Murphy appeared first on Software Engineering Daily.

AI-assisted software delivery refers to the utilization of artificial intelligence to assist, enhance, or automate various phases of the software development lifecycle. AI can be used in numerous aspects of software development, from requirements gathering to code generation to testing and monitoring. The overarching aim is to streamline software delivery, reduce errors and, ideally, reduce

The post AI for Software Delivery with Birgitta Böckeler appeared first on Software Engineering Daily.

The Buy Now, Pay Later model, or BNPL, is traditionally a business-to-consumer model that is gaining traction in the business-to-business domain. Adoption of BNPL in business-to-business scenarios offers benefits like improved cash flow management for buyers, and increased sales for suppliers, often leading to higher order value. However, this model introduces complexities, including the need

The post Expanding B2B payments with Sohil Pandya appeared first on Software Engineering Daily.

Database caching is a fundamental challenge in database management and there are hundreds of techniques to satisfy different caching scenarios. PolyScale is a fully automated database cache. It offers an innovative approach to database caching, leveraging AI and automated configuration to simplify the process of determining what should and should not be cached. Ben Hagen

The post Database Caching with Ben Hagen appeared first on Software Engineering Daily.

In memory of Software Engineering Daily Founder, Jeff Meyerson. 1988 – 2022

The post An update about SED from Jeff’s family appeared first on Software Engineering Daily.

In the realm of modern software development, clear and comprehensive documentation is essential for guiding users and contributors alike. Developers often struggle to create, manage, and maintain documentation that is both user-friendly and visually appealing. In this episode, we sit down with Sebastian Lorber, the lead maintainer of Docusaurus, a widely acclaimed open-source static site

The post Simplifying Documentation with Sébastien Lorber appeared first on Software Engineering Daily.

Nearly all developers are interested in developing APIs to streamline the connection process for other companies to their services. APIs are ubiquitous and critical to building modern software. However, making an API that provides a great developer experience can be a time-consuming endeavor. As a result, API teams often leave the “final” mile of integration

The post SDKs for your API with Sagar Batchu appeared first on Software Engineering Daily.

Generative pre-trained transformer models, or GPT models, have countless applications and are being rapidly deployed across a wide range of domains. However, using GPT models without appropriate safeguards can lead to leakage of sensitive data. This concern underscores the critical need for privacy and data protection. Skyflow GPT Privacy Vault prevents sensitive data from reaching

The post Ethical GPTs with Amruta Moktali appeared first on Software Engineering Daily.

The CAP theorem, also known as Brewer’s theorem, is a fundamental principle in distributed systems that states that it is impossible to simultaneously achieve three desirable properties in a distributed data system: Consistency, Availability, and Partition tolerance. Eric Brewer is the VP of Infrastructure & Google Fellow at Google and he joins us today. This

The post CAP Theorem 23 Years Later with Eric Brewer appeared first on Software Engineering Daily.

Companies have high hopes for Machine learning and AI to support real-time product offerings, prevent fraud and drive innovation. But there was a catch – training models require labeled data that machines can digest. As data volumes increase, the opportunity to get great ML results rises, but so does the problem of labeling all the

The post Data-Centric AI with Alex Ratner appeared first on Software Engineering Daily.

Elasticsearch is the most established solution today to search and analyze large amounts of logs. However, it can be costly and complex to manage. Quickwit searches large amounts of append only cloud data like logs or ledgers in a fraction time with significantly less cost than Elasticsearch. In this episode, we interview Paul Masurel, one

The post Cloud-native Search with Paul Masurel appeared first on Software Engineering Daily.

As Companies scale and their codebase becomes large, it can become difficult to measure team effectiveness. With so many moving pieces, and an increased lack of visibility, Engineering Leaders mostly rely on gut feelings to guide decisions and determine impact. Uplevel pulls meta-data from programs like Git, Slack, Jira and public calendar events and uses

The post Engineering Insights with Christina Forney appeared first on Software Engineering Daily.

Rudderstack is a warehouse-native customer data platform (CDP) that helps businesses collect, unify, and activate customer data from all their different sources. In today’s episode, we’re talking to Soumyadeb Mitra, the founder and CEO of Rudderstack. We discuss the importance of activating all your data, how Rudderstack can help you activate your data, the challenges

The post Making Data-Driven Decisions with Soumyadeb Mitra appeared first on Software Engineering Daily.

Hyperscalers refer to expansive cloud service providers capable of delivering enterprise-scale computing and storage services. Companies like Google, Amazon, Facebook that have huge data centers and are either running their own software or renting out this infrastructure can be called Hyperscalers. These Hyperscalers realized a long time back that the traditional Network, Storage & Compute

The post Shipping Oxide with Bryan Cantrill appeared first on Software Engineering Daily.

Available as a cloud service, Red Hat Trusted Software Supply Chain provides a DevSecOps framework to create applications more securely. Vincent Danen is the VP of Product Security at Red Hat and joins us in this episode. Red Hat has been a Secure Open Source software provider for very long. We discuss how the Red

The post Trusted Software Supply Chain with Vincent Danen appeared first on Software Engineering Daily.

The state of Data inside most companies is chaotic. It takes significant time and investment to tame this chaos. When you are a platform provider you are gathering tons of data from the developers using your platform. These developers building products on your platform need insight into that data to better understand how their application

The post Customer-facing Analytics with Tyler Wells appeared first on Software Engineering Daily.

The Internet of Things (IoT) has the potential to revolutionize many industries, but its full potential has yet to be realized. One reason is that the focus has been on the sensors rather than the data. IoT devices collect data from various sources, including machinery, customer behavior, and environmental conditions. This data can be used

The post The Internet of Data, Not Things appeared first on Software Engineering Daily.

The company Continue has a new take on the resale marketplace. By partnering with brands and pulling post-purchase data onto their platform, this innovative solution simplifies and enhances the resale process, empowering consumers to participate in a more sustainable and circular economy. Richie Ganney and Harry Riley are the Co-founders of Continue and they join

The post Resale Revolution with Richie Ganney and Harry Riley appeared first on Software Engineering Daily.

CRaC (Coordinated Restore at Checkpoint) is a new technology that can improve startup and warmup times by orders of magnitude. It is a project of OpenJDK that was proposed and led by Azul. The CRaC Project defines public Java APIs that allow for the coordination of resources during checkpoint and restore operations. With CRaC, a

The post Superfast JVM Startup with Gerrit Grunwald appeared first on Software Engineering Daily.

WebAssembly is a low-level binary format for the web that is compiled from other languages to offer maximized performance and is meant to augment the places where JavaScript isn’t sufficient. High Performance applications like AutoCAD, Figma and Photoshop are now leveraging the capabilities of WebAssembly to provide native experiences on the web. Matt Butcher is

The post WebAssembly with Matt Butcher appeared first on Software Engineering Daily.

In this episode, we are talking to Robert Cooke, founder and CTO of 3forge. He has spent the last decade creating a full stack software platform that revolutionized enterprise real-time data management, visualization, and workflows through its inventive “high impact code” concept. With offices in New York, London, and Singapore, 3forge has been serving a

The post Building Enterprise Applications with Robert Cooke appeared first on Software Engineering Daily.

Usage-Based Pricing is becoming more and more popular. Led by the wild popularity of cloud service providers such as AWS, customers are demanding more visibility into usage information and more accountability for the dollars they spend on a service. But historically service oriented products have shied away from usage based pricing because of the complexity

The post Solving Usage-Based Pricing with Puneet Gupta appeared first on Software Engineering Daily.

Feature flags also known as feature toggles, release toggles or feature flippers are a way to enable or disable a particular feature from your app without making any changes to the source code. You can turn on or off a particular functionality without deploying new code. Feature flags can also be used to serve different

The post Shipping Features with Ben Rometsch appeared first on Software Engineering Daily.

As companies depend more on data to improve digital products and make informed decisions, it’s crucial that the data they use be accurate and reliable. MonteCarlo, the data reliability company, is the creator of the industry’s first end-to-end data observability platform. Barr Moses and Lior Gavish are the founders of Monte Carlo and they join

The post Data Reliability with Barr Moses and Lior Gavish appeared first on Software Engineering Daily.

Programming robotics software has traditionally been a specialized field. The software industry has seen rapid progress, the operating system that provides the foundation for our software applications is taken care of by companies like Google, Microsoft, and other players like Canonical, Amazon, etc. The robotics industry still needs that OS layer that handles the complexities

The post Modern Robotics Platform with Eliot Horowitz appeared first on Software Engineering Daily.

Mach5 Search is a slide-in, cloud-native replacement for Elasticsearch and OpenSearch that immediately saves up to 90% in operating cost. Mach5 Search can run on top of Google BigQuery, Snowflake, and Databricks, or natively on Object Stores in all the major clouds. Vinayak Borkar is the CEO and Co-Founder of Mach5 Software and he joins

The post Cloud Native Search with Vinayak Borkar appeared first on Software Engineering Daily.

Metrist is an observability platform designed to commoditize application observability. Jeff Martens is the CEO of Metrist, and he joins us today.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His most recent book, Architecting for Scale (O’Reilly Media) is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Subscribe at mdb.fm. Follow Lee, at leeatchison.com.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Observability for Your Cloud Dependencies with Jeff Martens appeared first on Software Engineering Daily.

In this podcast episode, we take a look at the intricacies of low-code data pipelines with Raj Bains, the founder of Prophecy.io. Raj shares valuable insights into how performant low-codedata pipelines are revolutionizing industries and transforming everyday operations. Raj discusses the founding story of Prophecy.io, the company’s mission, and its approach to democratizing the creation of efficient data pipeline solutions visual design and code generation. We also discuss technical concepts/conundrums such as data lineage, schema evolution, and metadata management, which are critical in addressing the challenges faced by data pipeline developers and businesses. The episode concludes with Raj’s thoughts on the future of low-code data pipelines, the Prophecy.io roadmap and its potential impact on various industries, from healthcare to finance.

Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Low-Code SQL on dbt Core with Raj Bains from Prophecy appeared first on Software Engineering Daily.

Crossplane is an innovative open source control plane framework that helps companies provide managed access to cloud native control planes. Upbound provides a single global platform to build, deploy, and operate these internally managed control planes that are powered by cross plane.

Bassam Tabbara is the CEO of Upbound, and he joins us today.

Free Upbound trial: https://www.upbound.io/registerAbout Crossplane: https://www.crossplane.io/This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His most recent book, Architecting for Scale (O’Reilly Media) is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Subscribe at mdb.fm. Follow Lee, at leeatchison.com.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Cloud-native Control Planes with Bassam Tabbara appeared first on Software Engineering Daily.

When your application fails, finding the reason quickly is essential for limiting downtime. Often, most of the time it takes to repair a problem is taken by figuring out what exactly went wrong. PromptOps is a service designed to make tracing the answer to why something happened back to the original cause.

Dev Nag is the CEO of PromptOps, and he joins us today.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His most recent book, Architecting for Scale (O’Reilly Media) is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Subscribe at mdb.fm. Follow Lee, at leeatchison.com.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post PromptOps with Dev Nag appeared first on Software Engineering Daily.

In the Software Industry, Conferences are crucial for developers, offering learning, networking, and collaboration opportunities. Attendees stay updated on the latest technologies, gain new skills, and connect with peers. These events expose developers to industry trends, inspire innovation, and foster professional growth. Overall, Conferences play a vital role in advancing knowledge, building connections, and driving success in the developer community.
In this episode, we discuss what makes a great Conference and how Shift Conference became of the largest events in Europe.
Ivan Burazin is the Chief Developer Experience Officer at Infobip and the CEO of Codeanywhere, and he joins us today.

Get your Tickets for Shift Miami on May 23rd and Shift Croatia, which is later this year.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer .

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Shift Conference with Ivan Burazin appeared first on Software Engineering Daily.

The software supply chain refers to the process of creating and distributing software products. This includes all of the steps involved in creating, testing, packaging, and delivering software to end-users or customers.

Socket is a new security company that can protect your most critical apps from supply chain attacks. They are taking an entirely new approach to one of the hardest problems in security in a stagnant part of the industry that has historically been obsessed with just reporting on known vulnerabilities.

Feross is the Founder and CEO of Socket Security, and he joins us today.

This Interview was recorded at Open Source Summit North America and wouldn’t have been possible without the generous support fromThe Linux Foundation.ᐧJordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Software Supply Chain with Feross Aboukhadijeh appeared first on Software Engineering Daily.

Chaos engineering is a discipline within the field of software engineering that focuses on testing and improving the resilience and stability of a system by intentionally introducing controlled instances of chaos and failure. The primary goal of chaos engineering is to identify and address potential weaknesses and vulnerabilities in a system, ultimately making it more robust and reliable.
Uma Mukkara is the Head of Chaos Engineering at Harness and he joins us today.

This Interview was recorded at KubeCon Europe and wouldn’t have been possible without the generous support from The Linux Foundation.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Chaos Engineering with Uma Mukkara appeared first on Software Engineering Daily.

The CAP theorem, also known as Brewer’s theorem, is a fundamental principle in distributed systems that states that it is impossible to simultaneously achieve three desirable properties in a distributed data system: Consistency, Availability, and Partition tolerance.Eric Brewer is the VP of Infrastructure & Google Fellow at Google and he joins us today.This Interview was recorded at Open Source Summit North America and wouldn’t have been possible without the generous support fromThe Linux Foundation.ᐧ Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post CAP Theorem 23 Years Later with Eric Brewer appeared first on Software Engineering Daily.

Astasia is a Partner on Quiet Capital’s enterprise team. Prior to joining Quiet, she was an investor on Redpoint Venture’s early stage enterprise team. Astasia joins the show to talk about Early Stage Venture Capital.Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Early Stage Venture Capital with Astasia Myers appeared first on Software Engineering Daily.

Ian Coldwater is a DevSecOps engineer turned red teamer who specializes in breaking and hardening Kubernetes, containers, and cloud native infrastructure. In their spare time, they like to go on cross-country road trips, capture flags, and eat a lot of pie. Ian lives in Minneapolis and tweets as @IanColdwater.This Interview was recorded at KubeCon Europe and wouldn’t have been possible without the generous support from The Linux Foundation.Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Kubernetes Security with Ian Coldwater appeared first on Software Engineering Daily.

Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.

These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil.

Chris Aniszczyk is the CTO at Cloud Native Computing Foundation and he joins us today.Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Cloud Native in 2023 with Chris Aniszczyk appeared first on Software Engineering Daily.

Kubernetes is one of the most used open source platforms that orchestrates and automates operations. Although it improves the management and scaling of infrastructure and applications, Kubernetes frequently has challenges managing the complexity of releasing applications.

Git is the most widely used version-control system in the software industry today.
GitOps is a set of procedures that uses the power of Git to provide both revision and
change control within the Kubernetes platform.

Flux is an open source project that implements GitOps-driven continuous deployment for Kubernetes.

In this episode we speak with Priyanka Ravi and Stefan Prodan from Weaveworks.

The post GitOps for Kubernetes with Priyanka Ravi and Stefan Prodan appeared first on Software Engineering Daily.

Bug reporting hasn’t changed since the 1990’s. Despite all the technological advancements we’ve made in the rest of software development, the way we handle bugs has stayed the same.

It is common practice for non technical teams to provide bug reports that are missing vital information for developers to identify and quickly fix code, such as: network requests, console logs, HAR file, device information, video or screenshot replication, and more.

Providing a comprehensive bug report requires logging into many tools and time spent context switching, creating an environment where people are less likely to report bugs, which is not an ideal scenario for any founder or builder.

Dani & Irtefa, co-founders at Jam.dev set out to build a tool that will enable anyone, no matter their technical background, to capture rich contextual technical data about bugs, so that developers can quickly identify and resolve issues.

Jam is a browser extension that will instantly replay a bug while automatically attaching diagnostics: network requests and speed, console logs, device, and more – all in one easy click. You don’t have to log into a new tool or disrupt workflows, as Jam has direct integrations with all the first class bug reporting tools to seamlessly drop diagnostics into a ticket.

In this episode, Dani & Irtefa share their experience building greenfield products for Cloudflare’s Emerging Technology and Incubation team, how bug reporting was the common pain-point that impacted their product velocity, and the technology decisions they made when building Jam.

Test it out for free at: Jam.dev

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Bug Reporting is Broken and how Jam is Fixing that with Dani Grant & Mohd Irtefa appeared first on Software Engineering Daily.

In this episode we have two guests Ramiro Berrelleza is the founder and CEO of Okteto, a developer tool that helps you create production like dev environments in the cloud for every developer. Ramiro will tell us how to easily make replica environments for your day to day development , but computing related infrastructure is only half the battle. It also helps to have realistic data when you’re developing.
John Craft is the CTO at Privacy Dynamics, a company that helps you create anonymized data. You can use Privacy Dynamics to create realistic data from your production database without compromising your customer information.

Alex is an AWS Data Hero, an independent consultant, and the author of The DynamoDB Book, the comprehensive guide to data modeling with DynamoDB. He was an early employee at Serverless, Inc., creators of the Serverless Framework, and was an early community member in the serverless space. His consulting and training work focuses on serverless architectures and database optimization. You can find him on Twitter as @alexbdebrie or on his site, alexdebrie.com.

Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Pre-configured Development Environments with John Craft and Ramiro Berrelleza appeared first on Software Engineering Daily.

The cloud has become an all-encompassing platform for running diverse applications and enabling individuals and teams to add value by utilizing services and infrastructure that streamline the process of software building and operation. Nonetheless, the cloud has presented new hurdles for developers as it is intricate, and application development demands comprehension of cloud service intricacies. Moreover, creating portable applications across different cloud providers, as well as local testing and debugging, have proven to be difficult. Additionally, leaky abstractions and inadequate tooling have made it challenging to repurpose cloud architectures into reusable components.

To address these challenges, a cloud-centric programming language called Wing has been introduced. Wing enables developers to create distributed systems that fully capitalize on the cloud’s capabilities without the need to concern themselves with the underlying infrastructure.

Jeff is a DevSecOps engineer with strong experience in Security, The Software Development Life Cycle and Cloud Technologies. His advanced expertise in HashiCorp technologies places him as one of the most sought after trainers in the EMEA region and beyond. Jeff has worked for a range of different companies – from small startups to some of the biggest financial institutions. He now successfully runs his own consultancy that provides services in DevSecOps, Cloud and Security. Check it out on hemmen.LU and follow Jeff on LinkedIn @jeffhemmen

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Cloud-Oriented Programming (Part 2) with Elad Ben-Israel appeared first on Software Engineering Daily.

The goal of having a single pane of glass that allows us to see what is happening with our organization’s IT operations has been a long-standing goal for many organizations. The goal makes a lot of sense. Without a clear end-to-end picture, it is hard to determine where your problems are if you can’t determine whether something happening upstream is creating significant knock-on effects.

When we have these high-level views, we are, of course, aggregating and abstracting details. So the ability to drill into the detail from a single view is an inherent requirement. The problem comes when we have distributed our solutions across multiple data centers, cloud regions, or even regions with multiple vendors.

The core of the challenge is that our monitoring through logs, metrics, and traces accounts for a significant amount of data, particularly when it isn’t compressed. An application that is chatty with its logs or hasn’t tuned its logging configuration can easily generate more log content than the actual transactional data. The only reason we don’t notice it is that logs are generally not consolidated, and log data is purged.

When it comes to handling the monitoring in a distributed arrangement, if we want to consolidate our logs, we’re potentially egressing a lot of traffic from a data center or cloud provider, and that costs. Cloud providers typically don’t charge for inbound data, but depending upon the provider, it can be expensive for data egress; it can even cost to transmit data between regions with some providers. Even for private data centers, the cost exists in the form of bandwidth of connectivity to the internet backbone and/or the use of leased lines. The numbers can also vary around the world as well.

The following diagram provides some indicative figures from the last time I surveyed the published prices of the leading hyper scalers, and the on-premises costs are derived from leased line pricing.

This raises the question of how on earth do you create a centralized single pane of glass for your monitoring without risking potentially significant data costs. Where should I consolidate my data to? What does this mean if I use SaaS monitoring solutions such as DataDog?

There are several things we can do to improve the situation. Firstly, let’s look at the logs and traces being generated. They may help during development and testing, but do we need all of it? If we’re using logging frameworks, are the logs correctly classified as Trace, Debug, and so on? When logging frameworks are being used by applications, we can tune the logging configuration to deal with the situation when one module is particularly noisy. But for those systems that are brittle, people who are nervous about modifying any configuration or a 3rd party support organization will void any agreements if you modify any configuration. The following line of control is to take advantage of tools such as Fluentd, Logstash, or Fluentbit, which brings with it full support for OpenTelemetry. We can introduce these tools into the environment near the data source so that they can capture and filter the logs, traces, and metrics data.

The way these tools work means they can consume, transform and send logs, traces, and metrics to the final destination in a format that most systems can support. Further, Fluentd and Fluentbit can easily be deployed to fan out and fan in workloads – so scaling to sort out the data comprehensively can be done easily. We can also use them as a relay capability so we can funnel the data through specific points in a network for added security.

As you can see in the following diagram, we’re mixing Fluentd and Fluentbit to concentrate data flow before allowing it to egress. In doing so, we can reduce the number of points of network exposure to the internet. A strategy that shouldn’t be used as the only mechanism to secure data transmission, but certainly one that can be part of an arsenal of security considerations. It can also be used as a point of failsafe in the event of connectivity issues.

As well as filtering and channeling the data flow, these tools can also direct data to multiple destinations. So rather than throwing away data that we don’t want centrally, we can consolidate the data into an efficient time-series data store within the same data center/cloud and send on the data that has been identified as high value. This then gives us two options; in the event of investigating an issue, we can do a couple of things:

Identify the additional data needed to enrich the central aggregated analysis and ingest just that additional data (and possibly further refine the filtration for the future) needed.
Implement localized analysis and incorporate the resultant views into our dashboards.

Either way, you have access to additional information. I would opt for the former. I’ve seen situations where the local data stores have been purged too quickly by local operational teams, and data like traces and logs compress well in greater volume. But remember, if the logs include data that may be sensitive to location, pulling them to the center can raise additional challenges.

While in the diagram, we’ve shown the monitoring center to be on-premise, this could equally be a SaaS product or one of the clouds. The key to where the center is comes down to three key criteria:

Any data constraints in terms of the ISO 27001 view of security (integrity, confidentiality, and availability).
Connectivity and connectivity costs. This will tend to bias the location for monitoring to where the largest volume of monitoring data is generated.
Monitoring capability and capacity – both functional (visualize and analyze data) and non-functional factors, such as how quickly inbound monitoring data can be ingested and processed.

Adopting a GitOps strategy to help ensure that we have consistency in configuration and, therefore, data flow from software that may well be deployed across data centers or cloud regions and possibly even multiple cloud vendors can be kept consistent because the monitoring sources are consistent in configuration If we identify changes to the filters (to remove or include) data coming to the center.

Incidentally, most stores of log data, be that compressed flat files, databases can be processed with tools like Fluentd not only as a data sink but also as a data source. So it is possible through GitOps to distribute out temporary configurations for your Fluentd/Fluentbit nodes which can harvest and bulk move any newly required data for the center from those regionalized staging stores rather than manually accessing and searching them. But if you adopt this approach, we recommend creating templates for such actions in advance and use as part of a tested operational process. If such a strategy were to be adopted at short notice as part of a problem remediation activity, you could accidentally try and harvest too much data or impact current active operations. It needs to be done with awareness about how it can impact what is live.

Hopefully, this will help offer some inspiration for cost-efficiently handling hybrid and multi-cloud operational monitoring.

The post Challenges of Multi-Cloud and Hybrid Monitoring appeared first on Software Engineering Daily.

Chroma is an open source embedding database that is designed to make it easy to build large language model applications by making knowledge, facts and skills pluggable.Anton Troynikov is the co-founder of Chroma and he is our guest today.This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His most recent book, Architecting for Scale (O’Reilly Media) is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Subscribe at mdb.fm. Follow Lee, at leeatchison.com.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Open-Source Embedding Database appeared first on Software Engineering Daily.

The cloud has become an all-encompassing platform for running diverse applications and enabling individuals and teams to add value by utilizing services and infrastructure that streamline the process of software building and operation. Nonetheless, the cloud has presented new hurdles for developers as it is intricate, and application development demands comprehension of cloud service intricacies. Moreover, creating portable applications across different cloud providers, as well as local testing and debugging, have proven to be difficult. Additionally, leaky abstractions and inadequate tooling have made it challenging to repurpose cloud architectures into reusable components.

To address these challenges, a cloud-centric programming language called Wing has been introduced. Wing enables developers to create distributed systems that fully capitalize on the cloud’s capabilities without the need to concern themselves with the underlying infrastructure.

Jeff is a DevSecOps engineer with strong experience in Security, The Software Development Life Cycle and Cloud Technologies. His advanced expertise in HashiCorp technologies places him as one of the most sought after trainers in the EMEA region and beyond. Jeff has worked for a range of different companies – from small startups to some of the biggest financial institutions. He now successfully runs his own consultancy that provides services in DevSecOps, Cloud and Security. Check it out on hemmen.LU and follow Jeff on LinkedIn @jeffhemmen

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Cloud-Oriented Programming with Elad Ben-Israel appeared first on Software Engineering Daily.

Code editors are complex and demanding pieces of software that are both essential tools and constant companions for developers. Today, we are interviewing Nathan Sobo to hear about Zed, a code editor focused on “multiplayer” experiences, performance and a streamlined, minimalist design.

Nathan Sobo was a member of the Atom editor team at GitHub, which worked on the (now-deprecated) Atom code editor. He will share his thoughts on Rust, the impact of very low-latency collaborative coding on happiness, and how to turn an authoring environment into a first-class concern.

Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securit.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on Linkedinor Twitter @jocelynbyrne

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Multi-player coding with Nathan Sobo appeared first on Software Engineering Daily.

Most companies aren’t in the building business yet they need to deal with pricing and billing as a core piece of technology whether they want to or not. Stigg is an easy to implement headless pricing and packaging platform that takes a lot of the hassle out of pricing and billing for your SaaS application. Their APIs help developers launch new pricing plans faster that lead to better customer buying experiences.

Anton Zagrebelny is the CTO of Stigg and he is our guest today.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His most recent book, Architecting for Scale (O’Reilly Media) is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Subscribe at mdb.fm. Follow Lee, at leeatchison.com.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Pricing and Packaging Orchestration with Anton Zagrebelny appeared first on Software Engineering Daily.

Finout helps FinOps, DevOps, and Finance to manage & reduce cloud spend and improve the company’s profitability without adding code or changing existing tags.

Roi Ravhon is the Co-founder and CEO at Finout and he joins us today.

Full disclosure: Fineout is a sponsor of Software Engineering Daily.

Alex is an AWS Data Hero, an independent consultant, and the author of The DynamoDB Book, the comprehensive guide to data modeling with DynamoDB. He was an early employee at Serverless, Inc., creators of the Serverless Framework, and was an early community member in the serverless space. His consulting and training work focuses on serverless architectures and database optimization. You can find him on Twitter as @alexbdebrie or on his site, alexdebrie.com.

Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Cloud Cost Management with Roi Ravhon appeared first on Software Engineering Daily.

Data Activation is the method of unlocking the knowledge sorted within your data warehouse, and making it actionable by your business users in the end tools that they use every day. In doing so, Data Activation helps bring data people toward the center of the business, directly tying their work to business outcomes.

Hightouch is the simplest and fastest way to get started with Data Activation. As a Data Activation Platform, Hightouch uses Reverse ETL to sync data from the warehouse to 100+ different integrations. With Hightouch companies can leverage their existing data models and easily view and monitor all of their data syncs in a single platform. Better yet, Hightouch offers a visual audience builder that makes it easy for non-technical users to create custom audiences at moment’s notice.

Tejas Manohar is the CEO at Hightouch and he joins us today.

Full disclosure: Hightouch is a sponsor of Software Engineering Daily.

Alex is an AWS Data Hero, an independent consultant, and the author of The DynamoDB Book, the comprehensive guide to data modeling with DynamoDB. He was an early employee at Serverless, Inc., creators of the Serverless Framework, and was an early community member in the serverless space. His consulting and training work focuses on serverless architectures and database optimization. You can find him on Twitter as @alexbdebrie or on his site, alexdebrie.com.

Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Data Activation with Tejas Manohar appeared first on Software Engineering Daily.

Enabling data engineers to create data pipelines easily while delivering data streams that meet low-latency, production requirements is a difficult balancing act. David Yaffe and Johnny Gaettinger join us today to share how they have created that balance at Estuary.
Estuary is a data operations platform that synchronizes data across the systems where data lives and delivers it where you want it to live with sub second low latency. Dave and Johnny will share the technical choices that support sub second performance, use cases for batch and streaming data applications and some of their business perspective as an early stage start-up.Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securit.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on Linkedinor Twitter @jocelynbyrne

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Unified DataOps for Teams and Enterprise with Estuary.dev appeared first on Software Engineering Daily.

A data catalog provides an index into the data sets and schemas of a company.Data teams are growing in size, and more companies than ever have a data team, so the market for data catalog is larger than ever.

Mark is the CEO of Stemma and the co-creator of Amundsen, a data catalog that came out of Lyft. In today’s show Mark shares how his history as a software developer, data engineer and product leader at Lyft have helped him create a vision for a modern data catalog.

Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securit.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on Linkedinor Twitter @jocelynbyrne

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Self-Service Data Culture with Stemma’s Mark Grover appeared first on Software Engineering Daily.

Streaming analytics refers to the process of analyzing real-time data that is generated continuously and rapidly from various sources, such as sensors, applications, social media, and other internet-connected devices. Streaming analytics platforms enable organizations to extract business value from data in motion, similar to how traditional analytics tools derive insights from data at rest. DeltaStream is a unified serverless stream processing platform to manage, secure and process all your event streams and is based on Apache Flink.
Hojjat Jafarpour is the Founder and CEO at DeltaStream and he joins us today. Before joining DeltaStream, Hojjat was at Confluent, the company behind Apache Kafka, he built a product called ksqlDB, ksqlDB is a database built to do Stream processing on top of Apache Kafka.Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securit.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capitol One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on Linkedinor Twitter @jocelynbyrneSponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Streaming Analytics with Hojjat Jafarpour appeared first on Software Engineering Daily.

Observability refers to the capability of comprehending the internal functioning of a complex system by observing its external outputs. Observability tools help developers understand the entire scenario of an application by digging into the data that is being used and recorded. And can help pipe data across various solutions used within enterprises to read and discover what’s going on from analytics.
Rookout is an observability platform with a developer-first approach that empowers you to collect any data point, even from the most intricate levels of live code, in real time and as needed.
Liran Haimovitch is the CTO at Rookout, and he joins us today. We discuss about Rookouts recent offering called Snapshots. Rookout calls this the fourth pillar of Observability. With Snapshots, developers can accurately and quickly capture application state with outstanding performance and built-in security.

Mike Bifulco is CTO and co-founder of Craftwork. He’s also a developer advocate, writer, podcaster and serial startup founder. In past lives, Mike worked for Google, Stripe, Microsoft, and Gymnasium. Mike is also co-founder of APIs You Won’t Hate, a community for API Developers on the web. Mike’s publishes a weekly newsletter for product builders called Tiny Improvements at mikebifulco.com. Mike is on Mastodon at https://hachyderm.io/@irreverentmike

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Observability via Snapshots with Liran Haimovitch appeared first on Software Engineering Daily.

Distributed databases are necessary for storing and managing data across multiple nodes in a network. They provide scalability, fault tolerance, improved performance, and cost savings. By distributing data across nodes, they allow for efficient processing of large amounts of data and redundancy against failures. They can also be used to store data across multiple locations for faster access and better performance.

Turso is an edge-hosted, distributed database based on libSQL, an open-source and open-contribution fork of SQLite. It was designed to minimize query latency for applications where queries come from anywhere in the world. In particular, it works well with edge functions provided by cloud platforms such as CloudFlare, Netlify, and Vercel, by putting your data geographically close to the code that accesses it.

Glauber Costa is the Founder and CEO of ChiselStrike the company behind Turso, and he joins us today.

Alex is an AWS Data Hero, an independent consultant, and the author of The DynamoDB Book, the comprehensive guide to data modeling with DynamoDB. He was an early employee at Serverless, Inc., creators of the Serverless Framework, and was an early community member in the serverless space. His consulting and training work focuses on serverless architectures and database optimization. You can find him on Twitter as @alexbdebrie or on his site, alexdebrie.com.

Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Turso: Globally Replicated SQLite with Glauber Costa appeared first on Software Engineering Daily.

The increasing complexity of modern cloud-native architectures has led to the emergence Platform Engineering. This practice involves the development and upkeep of an integrated product, known as an “Internal Developer Platform,” which serves as a flexible and supported abstraction layer between application developers and the underlying technologies. Luca Galante leads Product at Humanitec and he joins us today.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Platform Engineering with Luca Galante appeared first on Software Engineering Daily.

C++ is a powerful programming language that has been in use for several decades. Its importance lies in its versatility and efficiency, making it a popular choice for developing software and systems across different domains. The impact of C++ is significant, as it has been used to create numerous high-performance applications, including operating systems, browsers, gaming engines, and financial systems. It has also played a crucial role in the development of the internet and the growth of modern computing.

In a recent development the NSA indirectly asked developers to use memory safe languages whenever possible because hackers frequently exploit memory issues. In this episode, we’ll discuss these safety issues with the creator of C++ Bjarne Stroupstrup.

Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Hardening C++ with Bjarne Stroustrup appeared first on Software Engineering Daily.

A SIEM platform provides organizations with a powerful tool for improving their security posture, by providing insights into potential security threats and enabling proactive security measures.
Panther is a Cloud based security monitoring platform that helps teams detect and respond to security breaches quickly and intelligently.
Jack Naglieri is the Founder and CEO of Panther and he joins us today.Jeff is a DevSecOps engineer with strong experience in Security, The Software Development Life Cycle and Cloud Technologies. He’s an expert in HashiCorp Technologies and is one the most sought after trainers in EMEA and beyond. Jeff has worked for companies ranging from small startups to the biggest financial institutions, he runs his own consultancy that provides services in DevOps, Cloud and Security. Follow Jeff on LinkedIn @jeffhemmen

The post Seamless SecOps with Jack Naglieri appeared first on Software Engineering Daily.

DataSet is a log analytics platform provided by Sentinel One that helps DevOps, IT engineering, and security teams get answers from their data across all time periods, both live streaming and historical. It’s powered by a unique architecture that uses a massively parallel query engine to provide actionable insights from the data available.

John Hart is a distinguished engineer leading the Event DB team, where he’s responsible for the time series database that powers the Dataset product. John is our guest here today.

Full disclosure: SentinelOne is a sponsor of Software Engineering Daily.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His most recent book, Architecting for Scale (O’Reilly Media) is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.
Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Subscribe at mdb.fm. Follow Lee, at leeatchison.com.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Observability Trends with John Hart appeared first on Software Engineering Daily.

Vercel is the platform for frontend developers, providing the speed and reliability innovators need to create at the moment of inspiration. Vercel’s Edge Functions enable you to deliver dynamic, personalized content with the lightweight Edge Runtime.

Guillermo Rauch is the CEO at Vercel and he joins us today.

This episode is hosted by Mike Bifulco. To learn more about Mike visit mikebifulco.com

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Building Delightful User Experiences with Guillermo Rauch appeared first on Software Engineering Daily.

What are the reasons why we need to improve performance? The JVM is a powerful piece of software – this is the reason why Java has maintained its popularity over the past 26+ years. But… there are some ways that the JVM works that can impact the performance of your applications.

One of the biggest challenges is latency – how quickly can we respond to a request from a user to access your application. That comes down to garbage collection – which leads to pauses and a bad customer experience. Garbage collection pauses can vary. But they impact application performance. The bigger the heap, the longer your pauses…but how do you optimize? What about warm-up time? That’s also a big challenge. We’ve addressed these problems without re-writing code from scratch – and that leads to better Java performance without any code changes.

This episode explores how you can use a performance JVM to make applications start fast, go fast, and stay fast.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Improved Java Performance Sans Code Changes with Simon Ritter appeared first on Software Engineering Daily.

There are many types of early stage funding available from friends and family to seed to series A. Some firms invest across a wide set of technologies and seek only to provide capital. Others are in it for the long haul – they focus on specific areas of technology and develop both long term relationships and deep expertise over time.

Today, we are interviewing Matt Turck of First Mark Capital, who is in it for the long haul and whose portfolio companies include Dataiku, Crossbeam, Ada, Cockroach Labs, Clickhouse and more. Today we will talk about Matt’s career, investment point of view, founding the Data-driven NYC community and the recent release of the 20234 MAD – an industry resource for understanding the Machine Learning, AI and Data Landscape

Be sure to check out the show notes for links to the MAD

This epsiode is hosted by Jocelyn Houle. Follow Jocelyn on Linked or on Twitter @jocelynbyrne.

Show notes –

In today’s show we referenced a couple things you may want to check out.

Matt’s blog and MAD Landscape

The interactive MAD Landscape

The picture in Matt’s Office was The Son of Man by Rene Magritte

Matt’s full bio

FirstMark Capital Site

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Data Investing and the MAD with Matt Turck appeared first on Software Engineering Daily.

eBPF stands for “extended Berkeley Packet Filter” and is a technology that allows developers to write and run highly efficient and secure programs that can analyze and modify data packets as they move through a system. eBPF programs can be used for a wide range of purposes, including network monitoring, security, and performance optimization.

Thomas Graf is the CTO & Co-Founder at Isovalent and he joins us today.

The post eBPF with Thomas Graf appeared first on Software Engineering Daily.

Netlify is a cloud-based platform that provides web developers with an all-in-one workflow to build, deploy, and manage modern web projects.
Matt Biilmann is the CEO of Netlify and he joins us today.

This episode is hosted by Mike Bifulco. To learn more about Mike visit mikebifulco.com

The post Netlify with Mathias Biilmann Christensen appeared first on Software Engineering Daily.

ChatGPT is an artificial intelligence language model developed by OpenAI. It is part of the GPT (Generative Pre-trained Transformer) family of models, which are designed to generate human-like text based on input prompts. ChatGPT is specifically trained to carry out conversational tasks, such as answering questions, completing sentences, and engaging in dialogue. It has been pre-trained on a large corpus of text data and fine-tuned on specific tasks to improve its performance. As a result, ChatGPT can generate responses that are often coherent, relevant, and natural-sounding.

Christian Hubicki is an Assistant Professor in the Robotics Department at Florida State University. He joins us today to discuss ChatGPT and its implications. We also discussed the future of Artificial Intelligence in general.

This show is hosted by Sean Falconer. Sean is the Head of Developer Relations and Marketing @Skyflow. Follow Sean at @seanfalconer

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Surviving ChatGPT with Christian Hubicki appeared first on Software Engineering Daily.

Lee Edwards is a Partner at Root Ventures, a venture capital firm that invests in early-stage startups that are focused on solving hard engineering problems. Lee is an investor in Privacy Dynamics, a company that offers data anonymisation as a service. You can listen to our interview with the CTO of Privacy Dynamics, John Craft here.

Lee is interested in investing in startups founded by engineers who are committed to addressing complex software challenges. This includes areas such as developer tools and services, software infrastructure, applied artificial intelligence and machine learning, tooling for data scientists and engineers, computer vision, and any other endeavor that requires a highly skilled technical team to develop.

This epsiode is hosted by Jocelyn Houle. Jocelyn is focused on data, ML and enterprise software. She has experience as a founder, investor, and product leader and has worked with both start-ups and large financial services companies. Jocelyn is currently a Senior Director of Product management for Securiti, a unified data controls company. Follow Jocelyn on Linked or on Twitter @jocelynbyrne.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Seeding Bold Engineers with Lee Edwards appeared first on Software Engineering Daily.

API mocking is a technique used to simulate the behavior of an API without actually connecting to the real API. It is useful for various reasons, including testing, isolation, development, and cost reduction. By using API mocking, developers can test their code without relying on the availability of the real API, isolate the code being tested from the behavior of the API, work on their code even when the API is not yet available, and reduce the number of requests they make to the real API, thus saving money.
WireMock is a flexible tool for building mock APIs and can operate as either a standalone server or as a managed service through WireMock Cloud.

Tom Akehurst is the CTO & Co-founder at WireMock and he joins us today.

Show Host: Sean Falconer

Full disclosure: Wiremock is a sponsor of Software Engineering Daily.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Creating Mock APIs with Tom Akehurst appeared first on Software Engineering Daily.

Keith Adams has worked previously at Facebook and has also been a Chief Architect at Slack. He joins us today to discuss technology, startups, investing, and like all conversations that exist today, ChatGPT.

The post Pebblebed with Keith Adams appeared first on Software Engineering Daily.

As companies move more and more workloads to the cloud, cost management in the cloud is becoming critically important. Modern application development methodology increasingly involves moving to service and microservice architectures enabled by the cloud and kubernetes is the backbone of this modern infrastructure trend. Managing costs of kubernetes clusters is therefore becoming increasingly important. Kubecost is the company focused on giving visibility into kubernetes resources, allowing you to reduce your overall kubernetes based infrastructure spend. OpenCost is an open-source project by Kubecost and supported by a few other partner companies. Matt Ray is Senior Community Manager at OpenCost and he joins us today.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Kubernetes Cost Management with Matt Ray appeared first on Software Engineering Daily.

Jeli.io is an end to end incident management tool. Jeli helps you understand why an incident took place, how it could’ve been prevented, and allows you to dive deep into uncovered themes, patterns, or problem areas!
Nora Jones is the Founder and CEO at Jeli.io and she joins us today.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Learning From Incidents with Nora Jones appeared first on Software Engineering Daily.

Data observability is the idea of creating a unified view of all the data so you can head off problems early, identify outages, and address any conflicts in real time. Today we’re interviewing Streamdal’s founders Ustin Zarubin and Dan Selans. They’re gonna share with us how they’re using an application monitoring approach to address the challenges of getting that unified view in today’s data-centric streaming enviroment.

Show host: Jocelynbyrne

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Streamdal: Streaming Data Performance Monitoring with Ustin Zarubin and Dan Selans appeared first on Software Engineering Daily.

Let’s say you have a set of microservices running on a Kubernetes cluster. In the past, developers used to program features like service discovery, observability, who’s allowed to talk to whom and other security related features directly into the application code.This slowed down the dev cycle and it made these microservices bigger and just generally made everything less flexible.
The service mesh is a tool responsible for handling service-to-service communication. You keep your application small and business focused and instead you dynamically program the intelligence into the network.
The Istio ambient mesh is an alternative to the standard Istio architecture, it allows you to not have to run sidecars per application.

The company Solo.io was founded by Idit Levine a few years ago. Interestingly the name Solo comes from her being the Solo founder when she started the company and the io domain name was a cool thing to have back then.
Brian Gracely is the Head of Marketing at Solo.io and he joins us today.

Show host: Jordi Mon Companys

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Istio Ambient Mesh with Brian Gracely appeared first on Software Engineering Daily.

Kubernetes is an open-source platform for automating the deployment, scaling, and management of containerized applications. The company Komodor started as a Kubernetes diagnostics platform focusing on Kubernetes troubleshooting for the entire Kubernetes stack. More recently, Komodor is taking a step towards becoming a single-pane-of-glass to simplify Kubernetes for developers. In the past, they were more DevOps-focused, and they’ve made some significant changes in the product that will appeal more to the developer. Komodor is also entering into the cluster management space to compete with Lens.

Itiel Shwartz is the co-founder and CTO of Komodor, and he joins us today.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Dev-First Kubernetes Operations Platform with Itiel Shwartz appeared first on Software Engineering Daily.

With the rise of big data and artificial intelligence, organizations are scrambling to find talent who can analyze large amounts of data and make informed decisions. In this article, we’ll explore what it takes to pursue a career in data science — from honing your skillset to understanding the job market and more.

What is Data Science?The term “data science” refers to the field of study that uses today’s current technology and methodologies to analyze large amounts of data in order to find previously unknown patterns, extract useful information, and make business decisions based on this information. Data scientists employ complex machine learning algorithms to construct predictive models used to make predictions.

Data science is an ever-evolving field that is all about using data to solve problems for organizations. Its purpose is to help companies make better decisions, and it can be applied in a variety of sectors, from healthcare to finance to manufacturing.

Data Science involves many different components including statistics, machine learning, and programming that can be used to examine large datasets and gain meaningful insights.

Pursuing a Career in Data SciencePursuing a career in Data Science requires a firm understanding of many different components such as statistics, machine learning, and programming. If you’re interested in pursuing a career in this sphere, it’s vital to understand what the field entails and what specializations you may wish to pursue.

Once you have a good grasp on this subject, you can then begin looking at the different career options available to you. With this knowledge, a data scientist is able to analyze large datasets and discover insights that can be used to make decisions and solve business problems.

What skills are needed?There is no single solution when it comes to carving a career in data science; the skillset will depend on the position you are aiming for. Regardless, strong analytical and mathematical aptitudes, coding talents, and communication capabilities are essential for any role. Having familiarity with huge datasets would be advantageous and a strong background in machine learning and artificial intelligence will help any data scientist make sense of the data they are exploring.

Additionally, knowledge of working with vast datasets and employing statistical tools will be beneficial. Moreover, having an understanding of machine learning algorithms is becoming increasingly required. If you’re aspiring to become a data scientist, you’ll need to hone your analytical and critical thinking abilities along with impressive mathematical and statistical proficiencies.

Any aspiring data scientist must have a firm grasp of programming languages such as Python and R, as well as an understanding of the statistical fundamentals of data science. Additionally, it is important to have an in-depth knowledge of how to use big data platforms and databases to store, process, and analyze large datasets.

Networking SkillsNetworking with specialists provides opportunities for learning about fresh approaches. Moreover, staying updated on innovative methods and processes in the field of data science will be vital; participating in conferences, seminars and online classes can help a lot.

Networking is an important step to take when looking to pursue a career in data science. Make sure you attend industry events and seminars, join professional associations or groups, and build relationships with those already working in the field.

This will give you an opportunity to learn more about the field, gain insight into new job openings and offers, and grow your network of contacts in the data science industry. Spending time talking to professionals in the field and attending events or seminars can also help you develop the skills and knowledge necessary to become a successful data scientist.

Networking can open doors to internships and future job opportunities so it’s important to be active in professional associations or groups related to data science. Networking can also be done online through social media outlets such as LinkedIn, where you can stay up-to-date on job opportunities and connect with other data scientists.

The difference types of Data Science JobsData science can encompass a range of job titles, from data analysts and engineers to business intelligence analysts and big data engineers. Data analysts are tasked with extracting, cleaning, and examining data to find patterns and insights, while data engineers build the infrastructure necessary for their work.

While the business intelligence analysts use data for decision-making, big data engineers work on massive datasets, creating systems to process them. As demand for data scientists increases, new job titles are expected to be created in the near future.

Training and EducationA great way to start training in data science is by taking a course with a reputable online school – there are many options such as Coursera which can provide an excellent foundation for learning data science.

Many of the courses focus on the process of data science, from collecting and cleaning data all the way to building predictive models and other forms of analysis. These courses provide a great way to get your feet wet in data science and help build your skillset while also giving you the chance to explore different aspects of it.

A number of internships are available which allow students to gain real world experience in the field and give them a competitive edge when applying for jobs. Additionally, there are many free data science resources available online – from tutorials and blog posts to open-source datasets and even project ideas – all of which can help you develop your skills in the area of data science.

Attending conferences and bootcamps is an excellent way to learn more about data science and to network with professionals who can provide valuable knowledge about pursuing a career in data science. Often, data science events are held every few months and will provide participants with opportunities to gain hands-on experience and network with professionals who will be able to offer valuable insight into this field.

Data Science vs Data EngineeringData science and data engineering both involve working with massive amounts of data and utilising it to tackle issues. Data engineers create, build, test, and maintain databases as well as big-data processing systems that are used in large organizations. On the other hand, a data scientist deals with massive amounts of data that need to be cleaned, manipulated, and organized.

Data scientists typically come from scientific fields such as maths, statistics or computer science, often creating algorithms and models for problem-solving purposes, whereas data engineers need a strong knowledge of engineering principles plus the ability to work across multiple software tools.

Job opportunities in Data ScienceData Science is a highly sought-after and constantly growing field – with numerous businesses relying on data to inform their decisions, the need for data scientists is soaring. Beyond attractive salaries, those in the industry can benefit from job security and plenty of possibilities for growth.

If Data Science is an area you are considering as a career, then math, programming and communication skills are essential prerequisites. Plus, handling large amounts of data must be comfortable to you. With the right qualifications and attitude, it’s very T rewarding to join this exciting field.

The future of data scienceAs big data has become more prominent, so has the need for those who can analyze and interpret it. Data science is still relatively new, but consistently evolving to meet the ever-changing needs of businesses and organizations.

We can anticipate that data science will become a key part in how companies operate. As data grows increasingly intricate, data scientists must come up with inventive techniques to examine it. Therefore, the future of data science is an intriguing one, full of possibilities and growth as we learn more about what we can do with this information to enhance our lives.

ConclusionIn a world where organizations are increasingly relying on data analysis to make decisions, data scientists are in high demand. While it can be intimidating at first, the field of data science is growing exponentially and provides great opportunities for those who are willing to work hard and stay ahead of the curve. With the right attitude and dedication, anyone can become a successful data scientist.

Pursuing a career in data science is an excellent option for those who are interested in using technology and analytics to solve challenging problems. By taking courses, obtaining certifications, and honing one’s skills through hands-on experience, anyone can successfully pursue this rapidly growing field.

The post Pursuing a Career in Data Science appeared first on Software Engineering Daily.

Permissions are Hard! And they are becoming harder as we move more into the Cloud-native ecosystem. If we go back in time to the point where it was just a single monolith that you were building on your own. You’ll probably have a framework to manage the permissions for you. But when you are working with distributed microservices, especially if you are a polyglot, you can’t use those solutions anymore. So you end up having to sprinkle a bit of access control into every little microservice and component that you build. In addition, with the scale of modern applications, it’s no longer just your services, there are a lot of third-party services that you have to connect to. Think about things like authentication, billing, analytics and other stuff that you combine from eternal services into what you are building.

Permit.io empowers developers to bake in permissions and access control into any product in minutes and takes away the pain of constantly rebuilding them. Or Weis is the co-founder and CEO of Permit.io and joins us today.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Permit Elements and Fullstack Authorization with Or Weis appeared first on Software Engineering Daily.

Data catalogs are one way to address the tension between wanting to use all the data for business advantage and needing to govern all the data for compliance. Today, Zeashan Pappa, a Databricks Product Specialist for the Unity Catalog, joins us to discuss how catalogs can help companies capture the full value of their data. Zeashan Pappa is a technology executive and architect who has 20+ years of enterprise software architecture, engineering, consulting, and project management expertise.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Databricks Unity Catalog with Zeashan Pappa appeared first on Software Engineering Daily.

Webhooks are used in connecting two different online applications. Webhooks allow one program to send data to another as soon as a certain event takes place.And because they are event-driven, webhooks are ideal for things like real-time notifications and data updates. The company Hookdeck helps build webhook integrations at scale.

In this episode, we talk with Alexandre Bouchard, Co-founder of Hookdeck.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Webhooks at Scale with Alexandre Bouchard appeared first on Software Engineering Daily.

Twitter is a social media platform that does some incredibly complex stuff when it comes to distributed systems engineering to keep the website up and running. Twitter has open sourced a lot of projects for others to use. Twitter created a fork of Memcached called Twemcache and also a fork of Redis to handle the caching issues. In this episode we talk to Yao Yue. Yao Yue is an expert in distributed systems and performance. She led the cache team at Twitter. Her most notable open-source project is Pelikan, a modular caching framework with the best performance and operator ergonomics in the domain of distributed caching. Pelikan is ideal for large scale deployment as well as cutting-edge research.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Caching at Twitter with Yao Yue appeared first on Software Engineering Daily.

Java is a write once run anywhere programming language. The way you do that is you use the javac compiler and compile the source files down to the class files. Those class files can run on any system in interpreted mode. But those class files that are run in interpreted mode, they are not machine code specific. To get that code to run fast you need to turn that into optimized machine code and that’s what the JIT compiler does. The open JDK contains its JIT compiler called Hotspot.

Azul is a company that specializes in Java for the enterprise. It has the largest Java engineering team after Oracle. John Ceccarelli is the Senior Director of Product Management at Azul Systems responsible for Azul Platform Prime which is their hyper optimized build of OpenJDK. In this episode we explore the fundamentals of compilers. We also discuss Azul’s Cloud Native Compiler, the programs that are suitable for the Cloud Native Compiler and the Deployment model of CNC.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Cloud Native Compiler with John Ceccarelli appeared first on Software Engineering Daily.

Although we like to think about ML workflows as straight-line narratives from experiment to training to production, and then finally monitoring; the reality for large companies is that all the steps are happening at one time in concert with other models, with shifting data, and, sometimes, misaligned key feature inputs.Moreover, regulated firms are required to track all the models, the changes, and the impacts of those changes for compliance. Enter explainability supported by model monitoring. Far from the old, single task process of only monitoring changes and anomalies, today’s ML monitoring does much more. It uses AI to deliver full featured performance management that can identify changes, alert the right people, and pop models back into production in real time with proper governance.

FiddlerAI is a startup focused on enterprise model performance management. They are tackling the unique challenges of building in-house stable and secure MLOps systems at scale. Today we are interviewing Krishna Gade about trusting AI, the technical challenges of ML monitoring and the real world problem statements beyond compliance that explainability can address.Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post MLOps Systems at Scale with Krishna Gade appeared first on Software Engineering Daily.

Just like everything else in software development, the idea of observability is not new – it emerged alongside the advent of information systems. Observability is a critical part of SDLC and helps developers and operations teams monitor their applications and environments, identify issues before they impact customers, and improve the performance of their software products.

This article will discuss the following points:

What is Observability?
What problems does it solve?
- Releases are faster
- Incidents become easier to fix
What are the challenges of observability?
Observability vs Monitoring
The Three Pillars of Observability
- Metrics
- Logs
- Traces
How do you implement Observability?
Choosing an Observability Platform
Best Practices of Observability
Conclusion

What is Observability?Observability helps developers and operations teams monitor their applications and environments, identify issues before they impact customers, and improve the performance of their software products.

Observability encompasses the monitoring of application metrics (usually via instrumentation), logs and exceptions, tracing data, and many other aspects of software applications. You can leverage observability to diagnose problems in real time or after they have occurred so that they don’t occur again.

Observability is the art of observing and understanding your system in order to make better decisions. Observability is generally understood as the ability to observe, understand and act upon events that occur within software systems or their components.

The observation part is straightforward – we have tools that can collect data about what has happened inside our application and correlate those observations.

What problems does it solve?Here are some of the key benefits of observability:

Gain insights into the infrastructure as a whole
Promote faster releases
Resolve issues easily and quickly
Reduce costs
Enhance developer productivity

The Three Pillars of ObservabilityThe three pillars of observability are metrics, logs, and traces.

MetricsMetrics provide quantitative data points about what’s happening within your system at any given point in time. This may take the form of CPU utilization or memory usage over time, counts on individual requests being served by an API gateway, etc., but they’re typically aggregated across multiple instances of your application (e.g., per cluster node). They can also include derived values such as averages or percentiles; for example: “the average CPU utilization across all nodes was 20% today.”

LogsLogs are structured messages that provide context about what’s happening within your system. They often include information such as request IDs, timestamps, and payloads for individual requests being served by an API gateway. As with metrics, these logs can be aggregated across multiple instances of your application (e.g., per cluster node).

TracesTraces are unstructured streams of events emitted by your software. They’re typically emitted at a high rate (e.g., thousands per second) and include data such as the time at which each event occurred, what kind of event it was (e.g., HTTP request, database query), and any additional parameters that were passed along with it (e.g., query parameters for an HTTP request).

Observability vs MonitoringMonitoring and Observability are related concepts, they complement each other. In other words, the two terms “monitoring” and “observability” are often used interchangeably. However, there are subtle differences between the two.

The key difference here is that while monitoring is reactive (i.e., it responds after an event has occurred), observability allows you to detect problems before they occur or even know when they occur in the first place (i.e., it is proactive).

Monitoring refers to the process of collecting, storing, and analyzing data. Observability provides valuable insights into how an application behaves at runtime. So, observability provides visibility into how your application has been behaving in a production environment.

Monitoring is the act of tracking and measuring the performance of a system. This can be achieved by using tools such as New Relic, which track application performance metrics like response times, error rates, and concurrency issues. Observability refers to the capability of observing and understanding the state of a system. With it, you can detect problems before they occur or even determine when they are likely to occur.

Both monitoring and observability tools are used to collect data from systems in order to help identify issues and understand behaviour. The key difference between the two is that observability provides more complete data collection and analysis, while monitoring may provide more limited data collection and analysis.

To be able to monitor something, there must be some level of observation involved. Observability takes advantage of instrumentation to provide insights that help with monitoring. The extent of observability depends on the ability to discover unknown qualities and patterns.

Observability and monitoring solutions provide a comprehensive overview of the health of your IT infrastructure, allowing for better decision-making. While monitoring warns the team of a possible problem, observability assists the team in determining and resolving the underlying cause of the problem.

How do you implement Observability?In order to achieve observability, you need to instrument your code so that you can collect data at every point in the system from the data sources themselves. This data can include everything from application and database logs to network traffic and performance metrics.

Choosing an Observability PlatformThere are certain factors you should consider before choosing an observability platform.

Ease of useYou should pick an observability platform that is easy to use. There is no point in selecting an observability platform if you’re going to struggle with it or get frustrated by its complexity. You need a tool that makes sense to you and your team, so choose one that has good documentation, guides and tutorials for new users, and a community forum where you can ask questions when things aren’t clear.

Community SupportYou should choose an observability platform that has a community behind it. It’s important for your chosen tool to have good support from its developers as well as other users who are using it in production environments like yours—so look for options with active communities on social media sites such as Twitter or Reddit, etc.

FlexibleYou should select an observability platform that can be used in multiple use cases. Even though some monitoring tools specialize in certain functions such as tracing, most of them are designed with flexibility in mind so they can be used across different teams within organizations—and even combined with other tools like log management solutions if needed.

Best Practices of ObservabilityWhen configuring observability for your application, you should adhere to a few recommended practices.

Make sure your observability tool is compatible with your existing tools, like monitoring dashboards, CI/CD pipelines, etc. Use tools that can help you interpret the data and easily identify anomalies.
Make sure it’s easy for everyone on your team to use so that no one gets left behind in the adoption process.
Keep an eye out for new features that might make it easier for you to see what’s happening with your systems, like alerts or notifications when something goes wrong—it makes it easier for everyone to stay on top of issues before they turn into problems.
Instrumenting your system with monitoring tools will allow you to see the data that is collected by those tools, and it can help you determine issues with your code or infrastructure.
Having alerts set up that let you know when something goes wrong is an important part of any observability strategy. These alerts will also tell you when things are going well, which means that they can be used as a baseline for comparison when troubleshooting issues.
You should instrument as much data as you can. You can obtain such data from several sources, such as application and server logs, performance counters, and network traffic data. When you have more data, you can gain better insights and identify problems in your application more efficiently.
You should ensure that you have the necessary tools to gather and evaluate this data. There are many alternatives available; choose the one that works best for you. Once you have the data, you must be able to visualize it and detect patterns quickly.
You should also set thresholds for each metric you’re tracking. This will assist you in determining when something is wrong. For example, if your system’s response time grows dramatically, this might signal a problem. Setting criteria in advance allows for detecting potential issues before they become severe disruptions.

ConclusionObservability can help you understand the behaviour of your application at runtime and identify issues as they happen. By tracking the right metrics and logging the appropriate data, you can gain invaluable insights into your system’s performance and optimize its stability.

With the right observability strategy in place, you can avoid outages, diagnose problems quickly, and ensure that your system runs smoothly.

The post An Introduction to Observability appeared first on Software Engineering Daily.

Mingsheng Hong, is CEO and co-founder (along with Zheng Shao) of Bluesky. He is a thought leader for machine learning and has built data infrastructure to support some of the world’s most challenging workloads. Most recently, Mingsheng was at Google, where he led teams to build storage and querying stacks that power Google’s $100 billion ads business and a new TensorFlow backend to improve performance for AI workloads.Jocelyn Byrne Houle is a product leader, operator and investor focused on data and machine learning. Today, Jocelyn is a Sr. Director of Product Management for Securiti.ai, a Unified Data Controls platform. She has experience working at startups and at large companies like Capital One, Fannie Mae and Microsoft. Any opinions expressed by Jocelyn are her own and do not reflect the opinions or her employers.

Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Optimizing Cloud Data Platforms with Mingsheng Hong appeared first on Software Engineering Daily.

Show host: Jordi Mon Companys

Guest: Dan Lorenc

The post Software Supply Chain with Dan Lorenc appeared first on Software Engineering Daily.

In this episode we talk with Varun Badhwar, Founder and CEO of Endor Labs.

Show host: Sean Falconer

The post Open Source Software Lifecycle Management and Security with Varun Badhwar appeared first on Software Engineering Daily.

Show Host: Jordi Mon Companys

Guest: Sam Alba

Transcript

The post Pipelines as Code with Sam Alba appeared first on Software Engineering Daily.

As applications grow in size and complexity, and as they increasingly move to microservice architectures, it becomes harder for individual developers to perform end-to-end tests of an entire application stack. Connecting development services to production services is off limits. Test environments are limited to automated tests.
Staging environments are woefully inadequate with the amount of data available for their use. Testing large and complex applications is becoming harder and harder. Speedscale provides developers a solution to this problem. Speedscale assists in developing and testing applications by recreating real world traffic loads in test and development environments, essentially bringing the data quality of production into the exploratory world of development. Matthew LeRay is the CTO and co-founder of Speedscale, and he’s our guest today.

Transcript

Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post API Testing in Kubernetes with Matthew LeRay appeared first on Software Engineering Daily.

Show Host: Jordi Mon Companys

Heather Meeker

Scott Jenson

Pablo Ruiz-Múzquiz

The post Open Source Design Collaboration appeared first on Software Engineering Daily.

The post API Observability with Jean Yang appeared first on Software Engineering Daily.

How log management is undertaken for many hasn’t progressed in approach for more than twenty years. At the same time, we’ve seen improvements in storing and searching semi-structured data. These improvements allow us to have better analytical processes that can be applied to log content once aggregated. I believe we’re often missing some great opportunities with how we handle the logs between their creation and putting them into some store.

This illustrates how more traditional non-microservice thinking with logging and analytics is.

Yes, Grafana, Prometheus, and observability have come along, but their adoption has focused more on tracing and metrics, not extracting value from general logging. In addition, adopting these tools has been focussed on the container-based (micro)service ecosystems. Likewise, the ideas of Google’s Four Golden Signals emphasize metrics. Yet vast amounts of existing production software (often legacy in nature) are geared towards generating logs and aren’t necessarily running in containerized environments.

The opportunities I believe we’re overlooking relate to the ability to examine logs as they’re created to spot the warning signs of bigger issues or at least be able to get remediation processes going the moment things start to go wrong. Put simply, becoming rapidly reactive, if not becoming pre-emptive, in problem management. But before we delve more into why and how we can do this, let’s take stock of what the 12 Factor Apps document says about this.

When the 12 Factor App principles were written, they addressed some guidelines for logs. The seeds of potential with Logs were hinted at but weren’t elaborated upon. In some respects, the same document also influences thinking in the direction of the traditional approach of gathering, storing, and analyzing logs retrospectively. The 12 Factor App statement about logging has, I think, a couple of key points, both right and, I’d argue if taken literally wrong. These are:

logs are streams of events
we should send logs to stdout’ and let the infrastructure sort out handling the logs
The description of how logs are handled either reviewed as they go to stdout or examined in a database such as OpenSearch using tools such as Fluentd.

We’ll return to these points in a moment, but we need to be mindful of how microservices development practices move the possibilities of log handling. Development here has driven the development and adoption of the idea of tracing. Tracing works by associating with an event a unique Id. As that unique Id flows through the different services. The end-to-end execution could be described as a transaction, which then when may make use of new ‘transactions’ (literal in terms of database persistence’ or conceptual in terms of the scope of functionality. Either way, these sub-transactions will also get their trace Id linked to the parent trace Id (sometimes called a context). These transactions of more called spans and sub-spans. The span information is typically carried with the HTTP header as the execution traverses through the services (but there are techniques) for carrying the information using asynchronous communications such as Kafka. With the trace Ids, we can then associate log entries. All of this can be supported with frameworks such as Zipkin and OpenTracing. What is more forward-thinking is OpenTelemetry which is working towards providing an implementation and industry stand specification, which brings the ideas of OpenCensus (an effort to standardize metrics), OpenTracing, and the ideas of log management from Fluentd.

OpenTelemetry’s efforts to bring together the three axes of solution observability hopefully create some consistency and maximize the opportunities of making it easier to link behaviors shown through the visualized metrics easier to link with traces and logs that describe what software is doing. While OpenTelemetry is under the stewardship of the CNCF, we should not assume it can’t be adopted outside of cloud-native/containerized solutions. OpenTelemetry addresses issues seen with software that have disturbed characteristics. Even traditional monolithic applications with a separate database have distributed characteristics.

The 12 Factor App and why should we be looking for evolution?The reason for looking for evolution is mentioned briefly in the 12 Factor App. Logs represent a stream of events. Each event is typically constructed from some semi of fully-structured data (either general descriptive text and/or structured content reflecting the data values being processed). Every event has some universal characteristics, as a minimum, a timestamp. Ideally, the event has other metadata to help, such as the application runtime, thread, code path, server, etc. If logs are a stream of events, then why not bring the ideas from stream analytics to the equation, particularly that we can perform analytical processes and decisions as events occur? The technologies and ideas around stream processing and stream analytics have evolved, particularly in the last 5-10 years. So why not exploit them better as we pass the stream of logs to our longer-term store?

Evaluating log events when they are still streaming through our software environment means we stand a chance of observing warning signs of a problem and enabling actions to be applied before the warning signs become a problem. Prevention is better than a cure. The cost of prevention is far lower than the cost of the cure. The problem is that we perceive preventative actions as expensive as the investment may never have a return. Put another way, are we trying to prevent something that we don’t believe will ever happen? Humans are predisposed to risk-taking and assuming that things won’t happen.

If we consider the fact that compute power continues to accelerate, and with it, our ability to crunch through more data in a shorter period. This means that when something goes wrong, a lot more disruption can occur before we intervene when we don’t work on a proactive model. To use an analogy, if our compute power is a car and the volume and value of the data are related to the car’s value. If our car could travel at 30mph ten years ago, crashing into a brick wall would be painful and messy, and repairing the car is going to cost and take time – not great, but unlikely to put us out of business. Now it can do 300mph; hitting the same wall will be catastrophic and fatal. Not to mention whoever had to clear up the fallout has got to replace the car, the impact with have destroyed the wall, and the energy involved would mean debris flung for 100s of meters – so much more cost and effort it could now put us out of business.

Take the analogy further; car manufacturers recognize accidents as much as we try to prevent them with legislation on speed, enforcement with cameras, and contractual restrictions with car insurance such as classes excluding racing, etc., accidents still happen. So, we try to mitigate or prevent them with better braking with ABS. Vehicle proximity and lane drift alarms. We are mitigating the severity of the impact through crumple zones, airbags, and even seat belts and their pretensions. In our world of data, we also have legislation and contracts, and accidents still happen. But we haven’t moved on much with our efforts to prevent or mitigate.

Compute power has had secondary indirect impacts as well. As we can process more data, we can gather more data to do more things. As a result, there can be more consequences when things go wrong, particularly regarding data breaches. Back to our analogy, we’re now crashing hypercars.

One reaction to the higher risks and impacts of accidents with cars or data is often more legislation and compliance demands on handling data. It is easy to accept more legislation – as it impacts everyone. But that impact is not consistent. It would be easy to look at logs and say they aren’t impacted. It’s the noise we must have as part of processing data. How often, when developing and debugging code, do we log the data we’re handling – it’s common from my experience, and in non-production environments, so what? Our data is synthetic, so even if the data was sensitive in nature logging, it isn’t going to harm. But along, suddenly, something starts going wrong in production; a quick way to try to understand what is happening is to turn up our logging. Suddenly, we’ve got sensitive data in our logs which we’ve always treated as not needing secure treatment.

Returning to the 12 Factor App and its recommendation on the use of stdout. The underlying goal is to minimize the amount of work our application has to perform regarding log management. It is correct that we should not burden our application with unnecessary logic. But resorting simply to stdout creates a few issues. Firstly, we can’t tune our logging to reflect whether we’re debugging, testing, or operating in production without introducing our own switches in the code. Something that becomes implicitly handled by most logging frameworks for us. More code means more chances of bugs. Particularly when code has not been subject to extended and repeated use as a shared library. In addition to elevated bug risk, the chances of sensitive data being logged also go up, as we’re more likely to leave stdout log messages than remove them. If the potential for logs goes up for production, so does the chance of it including sensitive data.

Firstly if we avoid the literal interpretation of the 12 Factor App of using stdout but look more at from the idea that our application logic should not be burdened with code for log management but utilizing a standard framework to sort that out, then we can keep our logic free of reams of code sorting out the mundane tasks. At the same time, maximizing consistency and log structure then, our tools can easily be configured to watch the stream as it passes the events to the right place(s). If we can identify semi or fully-structured log events, it becomes easy to raise the flag immediately that something is wrong.

The next issue is that stdout involves our I/O and more compute cycles. I’ve already made the point about ever-increasing compute performance. But performance investment in non-functional spaces always draws concerns, and we’re still chasing the performance issues to keep solution costs down.

We can see this with the effort to make containers start faster and tighten footprints of interpreted and byte code languages with things like GraalVM and Quarkus generating hardware-specific native binaries. Not only that, I pointed to the fact that to get value from logs, we need to have meaning. What is worse, a small element of logging logic in our applications so we can efficiently hand off logs and the receiver has an implicit or explicit understanding of the structure, or we have to run more logic to derive meaning from the log entries from scratch, using more compute effort, more logic, and more error-prone? It is correct that the main application should not be subject to performance issues that a logging mechanism might have and any back pressure impacting the application. But the compromises should never be to introduce greater data risks. To my mind using a logging framework to pass the log events off to another application is an acceptable cost (as long as we don’t stuff the logging framework with rafts of complex rules duplicating logs to different outputs etc.).

If we accept the question –isn’t it time to make some changes to up the game with our use of logging, then what is the answer?

What’s the answer?The immediate reaction to this is to look at the latest, most innovative thinking in the operational monitoring space, such as AI Ops – the idea of AI detecting and driving problem resolution autonomously. For those of us who are fortunate to work for an organization that embraces the latest and greatest and isn’t afraid of the risks and challenges of working on the bleeding edge – that’s fantastic. But you fortunate souls are the minority. Many organizations are not built for the risks and costs of that approach; to be honest, only some developers will be comfortable with such demands. The worst that can happen here is that the conversation to try to improve things gets shut down and can’t be re-examined.

We should consider a log event life more like this:

This view shows a more forward-thinking approach. ~While it looks complex, using tools like Fluentd means it is relatively easy to achieve. The complex parts are finding the patterns and correlations indicative of a problem before it occurs.

Returning to the 12 Factor App again. Its recommendation for using services like Fluentd and thinking of logging as a stream can take us to a more pragmatic place. Fluentd (and other tools) are more than just automated text shovels taking logs from one place and chucking it into a big black hole of a repository.

With tools like Fluentd, we can stream the events away from the ‘frontline’ compute and process the events with filters, route events to analytics tools and modern user interfaces or even trigger APIs that could execute auto-remediation for simple issues such as predefined archiving actions to move or compact data. At the simplest – a mature organization will develop and maintain a catalog of application error codes. That catalog will reflect likely problem causes and remediation steps. If an organization has got that far, there will be an understanding of which codes are critical and which need attention, but the system won’t crash in the next 5 minutes. If that information is known, it is a simple step to incorporate into an event stream processing the checks for those critical error codes and, when detected, use an efficient alerting mechanism. The next potential step would be to look for patterns of issues that together indicate something serious. Tools like Fluentd are not sophisticated real-time analytics engines. But in terms of simplicity, turning specific logs events into signals that can be processed with Prometheus can handle, and without introducing any heavy data science, we have the potential to handle situations such as how many times do we get a particular warning? Intermittent warnings may not be an issue as the application or another service could sort the issue out as part of standard housekeeping, but if they come frequently, then intervention may be needed.

Using tools like Fluentd won’t preclude the use of the slower bulk analytics processing, and as Fluentd integrates with such tools, we can keep those processes going and introduce more responsive answers.

We have seen a lot of advancement with AI. A subject that has been discussed as delivering potential value since the 80s. But in the last half-decade, we’ve seen changes that have meant AI can help in the mainstream. While we have seen mentions of AIOps in the press –. AI can help in very straightforward, practical means of extracting and processing written language (logs are, after all, written messages from the developer). The associated machine learning helps us build models to find patterns of events that can be identified as significant markers of something important, like a system failure. AIOps may be the major long-term evolution, but for the mainstream organization – that is still a long way downstream, but simple use cases for detecting the outlier events (supported by services such as Oracle Anomaly Detection) aren’t too technically challenging, and using AI’s language processing to help better process the text of log messages.

Finally, the nature of tools like Fluentd means we don’t have to implement everything from the outset. It is straightforward to progressively extend the configuration and continuously refine and improve what is being done, all of which can be achieved without adversely impacting our applications. Our previous diagram helps indicate a path that could reflect progressive/iterative improvement.

ConclusionI hope this has given pause for thought and highlighted the risks of the status quo, and things could advance.

The post Time to improve on how we handle our logs? appeared first on Software Engineering Daily.

Silicon Valley has many investors and Venture Capital firms, but most are not trying to solve hard technical problems. Engineering Capital partners with companies that are taking a technical risk. These are the companies who have some innovation where there is a doubt on whether it can even be built. Ideally, the market for this innovation exists, but nobody has bothered to build it yet, or nobody has been able to build it yet.
In this episode, We sat down with Ashmeet Sidana, Founder and Chief Engineer of Engineering Capital.TranscriptSponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Startup Investing with Ashmeet Sidana appeared first on Software Engineering Daily.

We do it every day. We search on the internet for some information. Many ask, why is that easy? And yet doing the same thing at my company is hard sometimes, not even possible. And when you do get the data, it’s unclear where it’s from or the degree you can trust it or use it. In a highly regulated firm, there’s even more pressure to select the best approved version of the information.

Enter the data catalog. Enterprise Data Catalogs are essential for searching for data in an organization. Moreover, their creation, maintenance, and design tap into the most fundamental theories of the philosophy of language and the nature of creating knowledge. Today we are interviewing architect and author Ole Olesen-Bagneux, who is finishing his first O’Reilly published book called Enterprise Data Catalogs. Today, he will explain what a data catalog is, the different ways computer scientists and information scientists think about searching for data versus searching in data, and how thinking about archiving data for a hundred years can help you create the best data catalog for your business today.

Follow Jocelyn : @jocelynbyrne

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post The Enterprise Data Catalog with Ole Olesen-Bagneux appeared first on Software Engineering Daily.

Data anonymisation is an ever more important problem with many pitfalls, and the legal context requires more and more companies to find a solution to it.
In this episode, we talk to John Craft from Privacy Dynamics, who offer data anonymisation as a service.
Privacy Dynamics is powering a world where ethics, data insights and personalized technology can all co-exist to support business growth and customer privacy simultaneously.
John talks about the different use-cases, technologies, requirements, and challenges he and his team faced throughout Privacy Dynamics’ journey so far, and what might be to come.

Show Host – Jeff Hemmen

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Minimizing PII in Dev Environments with John Craft appeared first on Software Engineering Daily.

Fintechs and traditional finance firms want to help their small business customers replace disorganized competing manual processes and multiple data sources with a single unified set of services and data. They could do that. They could more easily offer quicker decisions, better customer experience, and even get more first time customers founded in 2017.Codat wants to be that universal API that can connect financial firms to the platforms their small business customers use. Codat seeks to create a single flow and view for the bus. Small businesses, many accounting, banking, and commerce systems. That goal requires specialized approaches to authorization, data normalization, and availability.It also unlocks capabilities that small businesses can’t usually access, like real time risk scoring and predictive analytics. Today we are interviewing Jason Dryhurst-Smith, head of engineering and employee one at Codat, he’s an engineer who started in firmware and embedded systems. He has, uh, years of general development experiences, but now focuses primarily on platform engineering and building codat’s full engineering team.Sponsorship inquiries: sponsor@softwareengineeringdaily.comThe post Financial Data Aggregation for SMB with Jason Dryhurst-Smith appeared first on Software Engineering Daily.

OpenSSL is a free, open-source cryptographic library that provides secure communications over computer networks. It’s widely used to implement the secure socket layer (SSL) and transport layer security (TLS) protocols, which are the basis for secure, encrypted connections on the internet.
On Oct 25th, the OpenSSL project informed its users of a critical vulnerability that affects the 3.0 and later versions of the OpenSSL component. In a twist to the usual formula, the project gave the world a week’s advance notice of the upcoming update, and various stakeholders prepared for this accordingly. In this episode, we sat down with Ilkka Turunen, Sonatype’s Field CTO; we discussed a wide range of topics, including the OpenSSL vulnerability, Shodan, SBOMs, Software Supply Chain and others.

Show Host – Jeff Hemmen

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post OpenSSL Vulnerability with Ilkka Turunen appeared first on Software Engineering Daily.

Comma is a startup aimed at solving self-driving cars. A lot of the new cars in the market have built-in stock Advanced driver assistance systems. Comma takes this system to the next level with Openpilot. Openpilot is an open-source driver assistance system. Currently, with features like Adaptive Cruise Control (ACC), Automated Lane Centering (ALC), Forward Collision Warning (FCW), and Lane Departure Warning (LDW), openpilot is one of the most state-of-the-art solutions in the self-driving space. The Comma Three device is designed to live in your car, and purpose built to run openpilot.
In this episode we speak to George Hotz, President of Comma and now an intern at Twitter.

Watch the video on Youtube: https://www.youtube.com/watch?v=Yi6kwUZDtkw

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Special Episode with George Hotz appeared first on Software Engineering Daily.

Monolithic relational databases are the traditional foundation of financial core ledger systems. Nevertheless, the process of building and operating mission critical financial ledgers on these databases, and implementing homegrown accounting models, is a journey fraught with engineering challenges.

Twisp has set out to rethink the underlying technology for financial ledger systems by combining the operational and scaling characteristics of a distributed database, the correctness guarantees offered by relational databases, with pre-built accounting primitives, while fully leveraging the modern cloud.

In this episode, we speak to Michael Parsons, co-founder & CTO of Twisp.

To learn more about Twisp and get access to a sandbox ledger, go to: www.twisp.com/

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Twisp: Reinventing Accounting Systems with Michael Parsons appeared first on Software Engineering Daily.

Java Applications became the go-to preference of most developers because of the write-once-run-anywhere advantage it gave over other languages. And it didn’t take much time for Java to become the language for the enterprise. Consequently, most enterprises are still running legacy Java Monoliths on their infrastructure.

Breaking up a monolith is not an easy process–nor is it something that every company should do just because they have a monolith. In some cases, a monolith is just fine. But sometimes, you do need to decompose a monolith as the complexity of the monolith grows and leads to longer release cycles or scalability issues. Breaking down into microservices is a natural way to shift legacy applications to the cloud.

vFunction is an artificial intelligence platform that assesses, analyzes, helps you design microservices and then automatically creates those microservices for you. So it’s an end-to-end platform from analysis to the actual creation of the code of those microservices with their respective APIs. While vFunction started with Java, they are expanding these capabilities to other platforms as well. Moti Rafalin who is the CEO of vFunction and Amir Rapson who is the CTO of vFunction join us today.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Modernizing the Monolith with Moti Rafalin and Amir Rapson appeared first on Software Engineering Daily.

When Web Assembly was created it was supposed to be a compile target, where you could compile your favorite programming language and then execute it inside of a web browser. This made it possible for developers to choose a programming language like C++ for compute intensive applications. Fermyon is taking Web Assembly to the cloud.

The post Cloud-native WebAssembly with Matt Butcher appeared first on Software Engineering Daily.

If you are a company with a large pool of physical servers, and compute resources sitting in the data center, and you want to use them efficiently, one way to do that would be to virtualize them. Then you can use those smaller virtual compute resources to run whatever workloads you need. VMware is a

The post VMware Tanzu with Betty Junod appeared first on Software Engineering Daily.

These days, it’s easy to relegate “analytics” to the world of reporting and business intelligence, where batch ETL, complex queries, and scheduled reports are powered by the dominant cloud data warehouses. But a new trend is emerging. “Realtime analytics” is a new category of solution to a growing class of problem: Product companies want to

The post Tinybird is making OLAP feel like Postgres appeared first on Software Engineering Daily.

With an increasing number of data breaches impacting customer trust, prioritizing data privacy and security is more important than ever. However, as you layer on security and privacy, the overall complexity of a system grows and the data gets harder to use. There’s a constant push and pull between securing and locking down the data

The post Secure Workflows on Sensitive Data with Amruta Moktali appeared first on Software Engineering Daily.

The complexity of the software infrastructure has been increasing as companies have migrated towards kubernetes, containers, microservices and other distributed systems. However the tools around observability and monitoring have not seen much improvement. These tools are usually managed by teams distributed across different locations and time zones, which results in siloing of knowledge of your

The post Collaborative Notebooks for DevOps and SRE with Micha Hernandez appeared first on Software Engineering Daily.

The Presto/Trino project makes distributed querying easier across a variety of data sources. As the need for machine learning and other high volume data applications has increased, the need for support, tooling, and cloud infrastructure for Presto/Trino has increased with it. Starburst helps your teams run fast queries on any data source. With Starburst you

The post Accessing Data at Scale with Justin Borgman appeared first on Software Engineering Daily.

Building and managing data-intensive applications has traditionally been costly and complex, and has placed an operational burden on developers to maintain as their organization scales. Todays’ developers, data scientists, and data engineers need a streamlined, single cloud data platform for building applications, pipelines, and machine learning models — without having to move or copy their

The post Building on the Data Cloud with Torsten Grabs appeared first on Software Engineering Daily.

Companies are rapidly moving to the cloud, and modern businesses are operating faster with a myriad of SaaS tools in their day-to-day operations. Provisioning resources has been easier than ever. With a few clicks you can spin up resources in any part of the world. While we all love the cloud because of the scalability

The post Cloud Asset Management with Serhat Can appeared first on Software Engineering Daily.

Mikko Hypponen is the CRO of WithSecure and a principal research Officer at F-Secure. In this interview with our host, Jordy Monn, he discusses the eventful and rocky early start of his software engineering career, hybrid warfare, reverse engineering, recent exploits like Log for Shell and heart bleed and more. This episode is packed with

The post The Internet with Mikko Hypponen appeared first on Software Engineering Daily.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Tao of React with Alex Kondov appeared first on Software Engineering Daily.

Observability is a critical aspect of modern digital applications. You can’t operate an application at scale that satisfies your customer needs without understanding how the application is currently performing, whether it’s understanding the current operating needs of the application, adjusting resource usage, detecting issues before they become serious or solving an ongoing technical issue as

The post Modern Application Observability with Berkay Mollamustafaoglu appeared first on Software Engineering Daily.

The classical computing power has doubled every two years – a pattern known as Moore’s law. However, the ability to fabricate more and more transistors in a computer chip is approaching saturation as we are approaching atomic dimensions. Quantum Computing is a promising technology to take us beyond this. A quantum Computer uses qubits to

The post The Quantum Leap with Dmitri Maslov appeared first on Software Engineering Daily.

The Java Virtual Machine is an abstract machine that makes it possible for you to write Java code once and run it across multiple devices and operating system types. While you can use the OpenJDK it comes with various issues like Security Vulnerabilities and compliance. Azul is a company that specializes in Java for the

The post Azul with John Ceccarelli appeared first on Software Engineering Daily.

Passwordless Authentication is a technique in which users are given access to an environment without entering a password or answering a security question. This allows users to access an environment securely and protects organizations against attack vectors like Keylogging, Brute force methods, and phishing. The company SuperTokens provides secure login and session management for your

The post Open Source Authentication with Advait Ruia and Rishabh Poddar appeared first on Software Engineering Daily.

How we perceive those that have authored technical books is interesting. The opportunities for anyone to publish content are never better. Anyone can blog; all you need is an email address and access to the internet. Produce enough related content, and you could self-publish a book easily through a platform, even one as big as

The post What does it take to be a technical author? appeared first on Software Engineering Daily.

Monolithic relational databases are the traditional foundation of financial core ledger systems. Nevertheless, the process of building and operating mission-critical ledgers to track and reconcile payments and money movement is complex. Twisp is rethinking core accounting and financial orchestration on a cloud-native ledger system, so developers can stop re-inventing the ledger and focus on building

The post Twisp: Reinventing the Ledger with Jarred Ward appeared first on Software Engineering Daily.

PostgreSQL is a free and open-source relational database management system. Postgres-based databases are widespread and are used by a variety of organizations, from Reddit to the International Space Station, and Postgres databases are a common offering from cloud providers such as AWS, Alibaba Cloud, and Heroku. Neon is a serverless open-source alternative to AWS Aurora

The post Open-source Serverless Postgres with Nikita Shamgunov appeared first on Software Engineering Daily.

Today, we spoke with Daniel Situnayake of Edge Impulse. We discussed Cloud-based dev environments, Cloud-based IDEs, Infrastructure as code, Dev containers and Live collaboration. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Edge Impulse with Daniel Situnayake appeared first on Software Engineering Daily.

Vedran Jukic Vedran Jukic This episode is hosted by Sean Falconer. Sean’s been an academic, founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Developer Relations and Product Marketing at Skyflow and host of the podcast Partially Redacted, a podcast

The post Code Anywhere with Toma Pujlak and Vedran Jukic appeared first on Software Engineering Daily.

The default configuration in most databases is meant for broad compatibility rather than performance. Database tuning is a process in which the configurations of a database are modified to achieve optimal performance. Databases have hundreds of configuration knobs that control various factors, such as the amount of memory to use for caches or how often

The post Automatic Database Tuning with Andy Pavlo appeared first on Software Engineering Daily.

Originally published on January 1, 2022. Charlie Gerard is an incredibly productive developer. In addition to being the author of Practical Machine Learning in JavaScript, her website charliegerard.dev has a long list of really interesting side projects exploring the intersection of human computer interaction, computer vision, interactivity, and art. In this episode we touch on

The post Practical Machine Learning in JavaScript with Charlie Gerard appeared first on Software Engineering Daily.

Software engineering can be a surprisingly grueling career. It is both physically and mentally demanding to sit in front of a screen for hours on end producing code. Andy Johns is our guest in this episode. Andy previously worked on growth at Facebook, Twitter, and Quora as an early employee. He’s also an investor in

The post Mental Health in Tech with Andy Johns appeared first on Software Engineering Daily.

Data analytics technology and tools have seen significant improvements in the past decade. But, it can still take weeks to prototype, build and deploy new transformations and deployments, usually requiring considerable engineering resources. Plus, most data isn’t real-time. Instead, most of it is still batch-processed. Tinybird Analytics provides an easy way to ingest and query

The post Data Lake for Developers with Jorge Sancha appeared first on Software Engineering Daily.

Originally published on August 23, 2021. Application security is usually done with a set of tools and services known as SIEM – Security Information and Event Management. SIEM tools usually try to provide visibility into an organization’s security systems, as well as event log management and security event notifications. The company Panther takes traditional SIEM

The post Panther: Security as Code with Jack Naglieri appeared first on Software Engineering Daily.

Cloud computing provides tools, storage, servers, and software products through the internet. Securing these resources is a constant process for companies deploying new code to their cloud environments. It’s easy to overlook security flaws because company applications are very complex and many people work together to develop them. Wyze Labs, for example, had millions of

The post Bridgecrew: Cloud Security with Guy Eisenkot appeared first on Software Engineering Daily.

Maintaining availability in a modern digital application is critical to keeping your application operating and available and to keep meeting your customers growing demands. There are many observability platforms out there and certainly Prometheus is a popular open source solution for cloud native companies yet operating an observability platform, costs money, and all of the

The post Cloud-native Observability with Martin Mao appeared first on Software Engineering Daily.

As software engineering teams start to build products that become more and more mature, it becomes necessary to be able to debug complex issues with tooling that enables understanding of the full scenario. This can come from application scenarios where APIs are running multiple versions, where users are using your app from multiple devices, device

The post Developer-first Observability with Liran Haimovitch appeared first on Software Engineering Daily.

As software engineering teams start to build products that become more and more mature. It becomes necessary to be able to debug complex issues with tooling that enables understanding of the full scenario. This can come from application scenarios where APIs are running multiple versions, where users are using your app from multiple devices, device

The post Developer-first Observability with Liran Haimovitch appeared first on Software Engineering Daily.

Web infrastructure has evolved from individual servers to shared hosting services to virtual machines and virtual functions. The future of the internet however is looking toward a much more distributed computation model . Blockchain technology is central to the future of this modern internet . Blockchains are still in their infancy. And the most

The post Modular Blockchain Architecture with Nader Dabit appeared first on Software Engineering Daily.

As companies move to Spark and a Lakehouse architecture, they are realizing that the data tools are lagging way behind. You need to be a programmer to effectively use Spark and Airflow. There are some low-code ETL tools, but is that enough? Companies want to treat their data pipelines like mission-critical apps. They want DevOps

The post Lakehouse Data Stack with Raj Bains appeared first on Software Engineering Daily.

Originally published on February 12, 2022. Lee Atchison spent seven years at Amazon working in retail, software distribution, and Amazon Web Services. He then moved to New Relic, where he has spent four years scaling the company’s internal architecture. From his decade of experience at fast-growing web technology companies, Lee has written the book Architecting

The post Architecting for Scale with Lee Atchison appeared first on Software Engineering Daily.

Data is becoming a bank’s biggest asset. These complex enterprises have a huge opportunity ahead – to transform themselves to become a trusted hub of a much broader data ecosystem that goes beyond the financial industry and helps to form a new class of cross-industry experience architectures that are scalable and transparent. The data physics

The post Data Infrastructure for Finance appeared first on Software Engineering Daily.

Cloud native applications utilizing microservice architectures has grown into one of the most popular application architectural patterns in recent years. The value of leveraging dynamic cloud resources, along with the flexibility and scalability of microservice architectures, creates a strong paradigm that’s hard to miss. The strong adoption of Kubernetes has strengthened the pattern enormously. The

The post Kubernetes Troubleshooting with Itiel Shwartz and Lee Atchison appeared first on Software Engineering Daily.

Permissions are Hard! And they are becoming harder as we move more into the Cloud-native ecosystem. If we go back in time to the point where it was just a single monolith that you were building on your own. You’ll probably have a framework to manage the permissions for you. But when you are working

The post Permissions and Access-control with Or Weis appeared first on Software Engineering Daily.

Remember Frogger? That pixelated bit of arcade magic where you navigate your amphibious avatar across roads and rivers without getting smushed by passing cars or swept away by rushing water? Go Frogger go! As it turns out, Frogger is a perfect metaphor for Realtime Analytics. If you never played it, then grab some virtual quarters and spend 1 minute

The post What is Realtime Analytics, anyway? appeared first on Software Engineering Daily.

Enabling authorization policies across disparate cloud-native environments such as containers, microservices and modern application delivery infrastructure is complex and can be a roadblock for software engineering teams. Open Policy Agent, or OPA, is an open, declarative, policy-as-code approach to authorization that reduces security and compliance burden for engineering teams. Business context is translated into declarative

The post Cloud-native Authorization with Tim Hinrichs appeared first on Software Engineering Daily.

Ian Coe CEO Adam Kamor Head of Engineering Companies that gather data about their users have an ethical obligation and legal responsibility to protect the personally identifiable information in their dataset. Ideally, developers working on a software application wouldn’t need access to production data. Yet without high-quality example data, many technology groups stumble on avoidable

The post Faking Data Using Tonic.ai with Ian Coe and Adam Kamor appeared first on Software Engineering Daily.

Ian Coe CEO Kamor Adam Head of Engineering Companies that gather data about their users have an ethical obligation and legal responsibility to protect the personally identifiable information in their dataset. Ideally, developers working on a software application wouldn’t need access to production data. Yet without high-quality example data, many technology groups stumble on avoidable

The post Faking Data Using Tonic.ai with Ian Coe and Adam Kamor appeared first on Software Engineering Daily.

Cloud native applications utilizing microservice architectures have grown into one of the most popular application architectural patterns in recent years. The value of leveraging dynamic cloud resources, along with the flexibility and scalability of microservice architectures, creates a strong paradigm that’s hard to miss. The strong adoption of Kubernetes has strengthened the pattern enormously. The

The post Kubernetes Troubleshooting with Itiel Shwartz and Lee Atchison appeared first on Software Engineering Daily.

Originally published January 4, 2019 When a user makes a request to a product like The New York Times, that request hits an API gateway. An API gateway is the entry point for an external request. An API gateway serves several purposes: authentication, security, routing, load balancing, and logging. API gateways have grown in popularity

The post Kong API Platform with Marco Palladino Holiday Repeat appeared first on Software Engineering Daily.

Couchbase is a distributed NoSQL cloud database. Since its creation, Couchbase has expanded into edge computing, application services, and most recently, a database-as-a-service called Capella. Couchbase started as an in-memory cache and needed to be rearchitected to be a persistent storage system. In this episode, We interviewed Ravi Mayuram, SVP Products, and Engineering at Couchbase.

The post Couchbase Architecture with Ravi Mayuram appeared first on Software Engineering Daily.

Apache Arrow defines a language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware like CPUs and GPUs. The Arrow memory format also supports zero-copy reads for lightning-fast data access without serialization overhead. Wes McKinney is the CEO of Ursa Computing, a new startup working on accelerated computing

The post Arrow Infrastructure with Wes McKinney appeared first on Software Engineering Daily.

Originally published on July 11, 2016. Negotiation is an important skill for software engineers. The salary you negotiate at the beginning of your job could be a difference of tens of thousands of dollars over the course of an engineer’s career, but intimidating recruiters and exploding offers scare many engineers from negotiating at all. Today,

The post Salary Negotiation with Haseeb Qureshi appeared first on Software Engineering Daily.

Remix is a full stack web framework that lets you focus on the user interface and work back through web fundamentals to deliver a fast, slick, and resilient user experience that deploys to any Node.js server and even non-Node.js environments at the edge like Cloudflare Workers. In this episode, we interviewed Ryan Florence, co-founder at

The post Remix with Ryan Florence appeared first on Software Engineering Daily.

As engineers, we need to secure and protect sensitive data to prevent its misuse and comply with various data protection laws, while still supporting data-driven workflows like machine learning, analytics, or data sharing. Luckily there’s amazing innovation going on in the privacy and security space that can help us not only maintain our customers’ data

The post 10 Privacy-enhancing Technologies Every Engineer Should Know About appeared first on Software Engineering Daily.

Profiling has been a part of the software engineering toolbox since the 1970s, but it was long regarded to be too high in overhead, so it was only done ad hoc. The problem with profiling that way is that you never catch the moment in time where for example an incident happened. Continuous profiling is

The post Continuous Profiling using eBPF with Frederic Branczyk appeared first on Software Engineering Daily.

Zigi is an AI-powered personal assistant for developers. By managing your dev workflow and handling all your mundane, non-programming tasks across multiple apps directly from Slack, Zigi helps you focus on code creation and innovation. Zigi connects to all your cloud tools, such as Jira, Github, and your calendar, in order to get a 360°

The post Zigi with Nir Benson appeared first on Software Engineering Daily.

Whether a machine learning engineer, data scientist, business analyst – or honestly, anyone else at this point – you’ve probably received, used, imported or opened a CSV (comma-separated value) file . A well-known, oft-used text-based file, CSV is used to store large data records in a format that is easily transferable, and human readable. The simplicity

The post Importing CSV: Meeting the Challenge Head-on appeared first on Software Engineering Daily.

Enterprise-grade authentication is often an essential ingredient to virtually all applications in today’s world. However, companies often have a hard time understanding the value of that authentication especially during the early stages of product development. And hardening of an application is often left as an afterthought. Add enterprise-level requirements such as single sign-on and two-factor

The post WorkOS with Michael Grinich appeared first on Software Engineering Daily.

Sherlock Holmes once said, “It is a capital mistake to theorize before one has data”. Though from a fictional character in the early 1900s, the adage rings true today. Companies are increasingly leveraging data to make and guide decisions. As they should, data-driven decisions are proven to be more consistent, more reliable, and more accurate.

The post Software Engineering Best Practices Applied to Data appeared first on Software Engineering Daily.

Build automation tools automate the process of building code, including steps such as compiling, packaging binary code, and running automated tests. Because of this, build automation tools are considered a key part of a continuous delivery pipeline. Build automation tools read build scripts to define how they should perform a build. Common build scripts include

The post Earthly Builds with Adam Gordon Bell appeared first on Software Engineering Daily.

Creating great creative tools is extremely difficult. There are thousands of paths a user could take, and every aspect of the user experience has to be carefully considered and optimized for performance. And when you try to add real-time collaboration on top of that, you can be sure that you’re left with almost no time

The post LiveBlocks with Guillaume Salles appeared first on Software Engineering Daily.

Most technology companies rely on open source software projects. But open source software projects are often maintained by a group of people that is not affiliated with any particular company. When an open source project develops too much technical debt, it can become a tragedy of the commons. Who is responsible for maintaining these open

The post Gitcoin Ecosystem with Gloria Kimbwala appeared first on Software Engineering Daily.

Mailchimp is an all-in-one marketing platform for growing businesses, empowering millions of customers around the world to launch, build, and grow their businesses with world-class marketing technology, award-winning customer support, and inspiring content. Eric Muntz is Mailchimp’s CTO, responsible for the engineering teams that design, implement, and maintain Mailchimp’s products and infrastructure. He joins the

The post Mailchimp Engineering with Eric Muntz appeared first on Software Engineering Daily.

Venture capital investment has continued to flow into technology startups. No one builds technology from scratch. There are cloud services, software libraries, 3rd party services, and software platforms that modern entrepreneurs must adopt to build their products efficiently and quickly. These layers of infrastructure are a key area for many investors. Tim is a partner

The post Technical Venture Capital with Tim Tully appeared first on Software Engineering Daily.

Between hyper financialisation and the growth of automation in the world, trade and investments have become an ever more prevalent and important tool for the mass market. But the relative knowledge of professional and hobbyist investors is different, and so is the access to opportunities. Coinrule empowers regular people to compete with professional traders by

The post Coinrule Automated Crypto Trading with Zdenek Hofler appeared first on Software Engineering Daily.

Breakthroughs from labs that are exclusively or mostly focused on research tend to stay buried as papers. Sometimes the lag is natural, the research is far ahead of its broad applicability. But we find that there are hundreds of breakthroughs useful long before they reach users. It can take many years, even decades, before breakthroughs

The post Protocol Labs Research with Jeromy Johnson appeared first on Software Engineering Daily.

As crypto gains worldwide attention, the most scalable, resilient and performant systems are required for the industry to thrive. Products must be designed for a mix of technical and non-technical users. Blockchain.com’s VP of Engineering Lewis Tuff takes us behind the scenes for a look at the architecture, programming languages and database choices required to

The post Building for Crypto with Lewis Tuff appeared first on Software Engineering Daily.

Blockchains are the distributed ledger technology underlying bitcoin and other cryptocurrencies. More broadly, a blockchain is a mechanism for updating truth states in distributed network computing, producing consensus trust and serving as a new form of general computational substrate. Lewis Tuff is the VP of Engineering at Blockchain.com and joins the show to discuss the

The post Blockchain.com Infrastructure with Lewis Tuff appeared first on Software Engineering Daily.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His most recent book, Architecting for Scale (O’Reilly Media) is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host

The post Kubernetes Spend with Webb Brown appeared first on Software Engineering Daily.

Web3 promises to change the way we interact and transact online, but today’s dapps and wallets are hard to use. They are typically optimized for desktop machines and domain experts. In particular, it can be hard to bring value on-chain, and hard to know what to do once you’ve onboarded. Charlie Andrews-Jubelt is a software

The post Valora Decentralized Venmo with Charlie Andrews-Jubelt appeared first on Software Engineering Daily.

While Kubernetes has many benefits, there is often a need for teams to deploy a monitoring and observability stack to troubleshoot issues that happen within the cluster and the applications themselves. ContainIQ, an out-of-the-box solution, allows engineers to monitor the health of their cluster and troubleshoot issues faster. ContainIQ is unique in its approach in

The post ContainIQ with Matthew Lenhard appeared first on Software Engineering Daily.

Nearly all new tech companies build in a public cloud and established companies are rapidly migrating to the cloud from their on-prem data centers. But this move to the cloud can lead to a visibility problem. Cloud Providers offer not only Compute instances but also manage services like databases, blob storage, queues and more. It

The post Open-Source Cloud Asset Management with Yevgeny Pats appeared first on Software Engineering Daily.

How to build a rocket ship while flying it? In this episode, Cos returns to the show to discuss the technologies and approach Brex chose to create an engineering organization and innovation engine for the long term, while navigating shifts as the company continued to scale at a rapid pace. Cos and Jeff also cover

The post Upgrading the rocket in the air – CTO Decisions with Cos Nicolaescu appeared first on Software Engineering Daily.

Most engineering organizations are full of highly analytical people with STEM degrees. This is why it’s not at all surprising that the most data-driven organizations in any company are … Finance, Sales, and Marketing. Right? No, but seriously, when was the last time your engineering organization used data to make a decision? When we were

The post Towards EngOps: Scaling Engineering Orgs With Data appeared first on Software Engineering Daily.

Convex makes storing your application’s data as easy as using React state management. If you can use React hooks, you can also manage your backend data using Convex. James Cowling is a former Dropbox infrastructure engineer turned startup founder. James joins the show to discuss how Convex offers a simpler full-stack developer experience than the

The post Global State Management with James Cowling appeared first on Software Engineering Daily.

Developers looking for read or write access to Ethereum, Polygon, IPFS or other Web3 networks in order to get their idea in the hands of users need reliable RPC endpoints they can count on, whether they are working on a hackathon project or running hybrid architecture for a production application. Infura provides a platform for

The post Infura Ethereum Infrastructure with E.G. Galano appeared first on Software Engineering Daily.

Streaming data platforms like Kafka, Pulsar, and Kinesis are now common in mainstream enterprise architectures, providing low-latency real-time messaging for analytics and applications. However, stream processing – the act of filtering, transforming, or analyzing the data inside the messages – is still an exercise left to the receiving microservice or datastore, a custom programming exercise

The post Decodable Streaming with Eric Sammer appeared first on Software Engineering Daily.

The Kubernetes ecosystem has drastically changed how development teams ship software. While Kubernetes has provided many advancements in cloud infrastructure, it has also left organizations with massive security blindspots. KSOC was created to give developers and security teams a single control plane to harden multi-cluster Kubernetes environments through event-driven analysis, least privilege enforcement, and remediation-as-code.

The post Kubernetes Security Compliance with Jimmy Mesta appeared first on Software Engineering Daily.

In software engineering, telemetry is the data that is collected about your applications. Unlike logging, which is used in the development of apps to pinpoint errors and code flows, telemetry data includes all operational data including logs, metrics, events, traces, usage, and other analytical data. Companies usually visualize this information to troubleshoot problems and understand

The post New Relic Architecture with Nic Benders appeared first on Software Engineering Daily.

One of the challenges with Microservices architecture is how you manage dependencies between your services when implementing workflows. Conductor is an open-source microservices and workflow orchestration platform. Boney Sekh co-founded Orkes Inc – a company focused on offering Conductor as a service. Boney joins the show to discuss how engineers leverage Conductor to build highly

The post Conductor Orchestration with Boney Sekh appeared first on Software Engineering Daily.

Notifications have typically been an area of a product where building in-house has been the only option. However, building a best-in-class notifications system that delivers a great customer experience requires a significant investment when you start to layer in complexities like batching, cross channel orchestration, and user preferences. Chris Bell of Knock joins the show

The post Notifications Infrastructure with Chris Bell appeared first on Software Engineering Daily.

The advent of the cloud introduced a new form of technical debt in which organizations can lose track of what infrastructure they have and how it relates to the business. While the cloud’s native APIs offer some transparency into your infrastructure, these offerings are often described as necessary but not sufficient. When companies have a

The post CloudGraph with Tyson Kunovsky appeared first on Software Engineering Daily.

Highlight is a tool that helps teams reproduce end-to-end user sessions to troubleshoot their applications faster, more efficiently, and with all the context they need. With Highlight, engineering teams can replay errors with high precision, which includes complete session replay, outgoing network requests, dense stack traces and insight into the app’s state management system. Jay

The post Browser Observability With Jay Khatri appeared first on Software Engineering Daily.

Lee Atchison spent seven years at Amazon working in retail, software distribution and Amazon Web Services. He then moved to New Relic, where he spent eight years scaling the company’s internal architecture. Lee has deep expertise in building and managing fast growing web applications, and he used this knowledge to write architecting for scale from

The post Technical Debt With Lee Atchison appeared first on Software Engineering Daily.

Benefits of fostering a culture of joint accountability. Hybrid, multi-cloud, and edge environments are becoming the standard for organizations, but securing cloud environments is substantially more complex than securing on-premise environments. In addition to accelerating security concerns, most organizations must comply with at least 13 specific regulations. These regulatory frameworks and associated policies are often

The post Lower organizational risk by embracing a culture of compliance. appeared first on Software Engineering Daily.

There’s an increasing regulatory and consumer pressure on companies to do a better job protecting sensitive customer data. Yet, despite this pressure, data breaches and compliance issues continue to plague the tech industry. Companies like Apple, Netflix, and Google have solved these challenges by pioneering a new type of technology, the data privacy vault. Skyflow

The post Skyflow Privacy and Compliance with Sean Falconer appeared first on Software Engineering Daily.

Data-as-a-service is a company category type that is not as common as API-as-a-service, software-as-a-service, or platform-as-a-service. In order to vend data, a data-as-a-service provider needs to define how that data will be priced, stored, and delivered to users: streaming over an API or served via static files. Naqeeb Memon of Safegraph joins the show

The post Data Delivery with Naqeeb Memon appeared first on Software Engineering Daily.

Blockchains were the first systems to allow guaranteed permanent storage of public data. As cryptocurrency technology has advanced, a rich ecosystem of permanent storage and compute has developed as well. One of these is Arweave, a system for permanent information storage. Sam Williams is a founder of Arweave and joins the show to talk through

The post Arweave with Sam Williams appeared first on Software Engineering Daily.

Data labeling allows machine learning algorithms to find patterns among the data. There are a variety of data labeling platforms that enable humans to apply labels to this data and ready it for algorithms. Heartex is a data labeling platform with an open source core. Michael Malyuk joins the show to talk through the platform

The post Data Labeling with Michael Malyuk appeared first on Software Engineering Daily.

Real-time analytics are difficult to achieve because large amounts of data must be integrated into a data set as that data streams in. As the world moved from batch analytics powered by Hadoop into a norm of “real-time” analytics, a variety of open source systems emerged. One of these was Apache Pinot. StarTree is a

The post Pinot and StarTree with Chinmay Soman appeared first on Software Engineering Daily.

Web3 is powerful but difficult to work with. Deploying blockchain nodes, accessing data, and performing staking operations are non-trivial engineering actions. To simplify web3, Ankr hosts APIs for node deployment, RPC, and staking. Josh Neuroth from Ankr joins the show to talk about modern web3 infrastructure. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Web3 Infrastructure with Josh Neuroth appeared first on Software Engineering Daily.

Brendan Falk Matt Schrage The terminal is a necessary tool for any software engineer. In order to work quickly, developers have always customized their terminals to work for their specific application workflow. Fig is a reimagined terminal product that adds autocomplete and an entire app ecosystem to the existing terminal you are familiar with. In

The post Fig Engineering with Brendan Falk and Matt Schrage appeared first on Software Engineering Daily.

WP Engine is a domain specific cloud provider that hosts high performance WordPress infrastructure. This website, Software Engineering Daily, runs on WP Engine. Scaling a domain specific cloud provider for WordPress includes complexities at the level of the database, application, load balancer, and other areas. Brandon DuRutte from WP Engine joins the show to talk

The post Scaling WordPress with Brandon DuRette appeared first on Software Engineering Daily.

Data loss can occur when large data sources such as Slack or Google Drive get leaked. In order to detect and avoid leaks, a data asset graph can be built to understand the risks of a company environment. Polymer is a data loss prevention product that helps companies avoid problematic data leaks. Yasir Ali is

The post Data Loss Prevention with Yasir Ali appeared first on Software Engineering Daily.

The Presto/Trino project makes distributed querying easier across a variety of data sources. As the need for machine learning and other high volume data applications has increased, the need for support, tooling, and cloud infrastructure for Presto/Trino has increased with it. Justin Borgman is the CEO of Starburst, and joins the show to talk about

The post Starburst Infrastructure with Justin Borgman appeared first on Software Engineering Daily.

Data integration infrastructure is not easy to build. Moving large amounts of data from one place to another has historically required developers to build ad hoc integration points to move data between SaaS services, data lakes, and data warehouses. Today, there are dedicated systems and services for moving these large batches of data. Airbyte builds

The post Airbyte Engineering with Michel Tricot appeared first on Software Engineering Daily.

Modern organizations eventually face data governance challenges. Keeping track of where data came from, what systems update it, in what ways updates can be made are just some of the issues to be tackled. Large organizations face additional challenges like training, onboarding, and capturing the institutional knowledge that leaves with the departure of key team

The post Select Star with Shinji Kim appeared first on Software Engineering Daily.

The JavaScript supply chain includes numerous vulnerabilities due to its expansive nature and the long dependency chains. Socket is a new security company that can protect your most critical apps from supply chain attacks. They are taking an entirely new approach to one of the hardest problems in security in a stagnant part of the

The post JavaScript Supply Chain with Feross Aboukhadijeh appeared first on Software Engineering Daily.

Loft is a platform for Kubernetes self-service and multi-tenancy. Loft allows you to control Kubernetes clusters with added multi-tenancy and self-service capabilities to get more value out of Kubernetes beyond simply cluster management. It allows for cost optimization, more efficient provisioning, and other features. Lukas Gentele joins the show to talk about Kubernetes multi-tenancy and

The post Loft Kubernetes Namespaces with Lukas Gentele appeared first on Software Engineering Daily.

Ben Sigelman Alex Kehlenbeck Observability consists of metrics, logs, and traces. Lightstep is a company that builds distributed tracing infrastructure, which requires them to store and serve high volumes of trace data. There are numerous architecture challenges that come with managing this data. Ben Sigelman and Alex Kehlenbeck join the show to discuss the implementation

The post Distributed Tracing Infrastructure with Ben Sigelman and Alex Kehlenbeck appeared first on Software Engineering Daily.

The solution many turn to for capturing their streaming data is InfluxDB. In this episode, I interview Brian Gilmore, Director of Product Management at InfluxData, about how real time applications achieve success built on top of InfluxDB. When most people hear the phrase Internet of Things, it typically evokes an image of connected devices we

The post Time Series IoT on InfluxDB with Brian Gilmore appeared first on Software Engineering Daily.

As companies move to Spark and a Lakehouse architecture, they are realizing that the data tools are lagging way behind. You need to be a programmer to effectively use Spark and Airflow. There are some low-code ETL tools, but is that enough? Companies want to treat their data pipelines like mission-critical apps. They want DevOps

The post Lakehouse Data Stack with Raj Bains appeared first on Software Engineering Daily.

Vercel’s 4KB environment variable limit stems from using AWS Lambda as their infrastructure provider. While Vercel has a documented solution for working around this limitation, it came with the drawback of large application code changes due to not using environment variables for secrets anymore plus the introduction of a new API endpoint. Without a quick and easy

The post Working around Vercel’s 4KB Environment Variables Limit for Node.js appeared first on Software Engineering Daily.

Terminals are a gateway to building and running software, but they have not been reimagined since their initial creation. Warp is a new kind of terminal with visual aids, autocomplete, documentation, customization, and other features. It includes GPU acceleration, its own UI framework, and other engineering systems to increase speed and performance. Zach Lloyd joins

The post Warp Terminal with Zach Lloyd appeared first on Software Engineering Daily.

Lior Gavish James Densmore Data infrastructure is a fast-moving sector of the software market. As the volume of data has increased, so too has the quality of tooling to support data management and data engineering. In today’s show, we have a guest from a data intensive company as well as a company that builds a

The post Data Engineering Trends with Lior Gavish and James Densmore appeared first on Software Engineering Daily.

Gabe Greenberg Michelle Bakels Software engineering can be a surprisingly grueling career. It is both physically and mentally demanding to sit in front of a screen for hours on end producing code. During the pandemic, the strains of being an engineer became even more acute, as we were secluded in our homes, in some cases

The post Developer Health with Gabe Greenberg and Michelle Bakels appeared first on Software Engineering Daily.

Crypto companies have cemented themselves as a company category that is not going away. Bitski is a company that makes web3 tooling and infrastructure, including a wallet and a collection of tools for creating and selling NFTs. Patrick Tescher joins the show to talk through the engineering behind crypto infrastructure and APIs. Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Bitski: Web3 Infrastructure with Patrick Tescher appeared first on Software Engineering Daily.

Running a database company requires expertise in both technical and managerial skills. There are deeply technical engineering questions around query paths, scalability, and distributed systems. And there are complex managerial questions around developer productivity and task allocation. Sam Lambert is the CEO of PlanetScale, which is building modern relational database infrastructure. Before PlanetScale he spent

The post PlanetScale Management with Sam Lambert appeared first on Software Engineering Daily.

Alexis Richardson Mohamed Ahmed GitOps is a deployment and infrastructure management strategy based around continuous delivery and Kubernetes, with Git at the center of deployment workflows. Policy management can be used to define permissions and rules around who can deploy and what constitutes safe deployment. There is a synergy between GitOps tooling and policy management

The post GitOps and Policy Management with Alexis Richardson and Mohamed Ahmed appeared first on Software Engineering Daily.

A company’s response to an incident defines how that company responds to an adverse, unexpected scenario. Kintaba automates the incident management process so teams can quickly report, respond, resolve, and reflect on major incidents collaboratively. Kintaba’s incident response platforms gives you instant access to incident management processes based on the practices of companies like Facebook,

The post Kintaba Incident Response with John Egan appeared first on Software Engineering Daily.

SingleStore is a multi-use, multi-model database designed for transactional and analytic workloads, as well as search and other domain specific applications. SingleStore is the evolution of the database company MemSQL, which sought to bring fast, in-memory SQL database technology to market. Jordan Tigani is Chief Product Officer of SingleStore and joins the show to talk

The post SingleStore with Jordan Tigani appeared first on Software Engineering Daily.

Compute resources continue to trend towards being cheaper, easier to use, and faster. Despite these positives, more compute demands more energy and therefore an increasing carbon footprint. With many companies committing to controlling their net carbon emissions, tools are required for engineers to not only measure their cloud infrastructure, but to make informed choices about

The post Cloud Carbon Footprint with Steren Giannini appeared first on Software Engineering Daily.

Vantage is a system for optimizing cloud costs. It provides tools and interfaces for developers to analyze how they are spending on AWS resources, and has recently expanded into GCP as well. Vantage users gain an easy interface into their costs that would otherwise be hard to analyze via the raw AWS console. Ben Schaechter

The post Vantage Engineering with Ben Schaechter appeared first on Software Engineering Daily.

At Lyft, Ketan Umare worked on Flyte, an orchestration system for machine learning. Flyte provides reliability and APIs for machine learning workflows, and is used at companies outside of Lyft such as Spotify. Since leaving Lyft, Ketan founded Union.ai, a company focused on productionizing Flyte as a service. He joins the show to talk about

The post Union.ai with Ketan Umare appeared first on Software Engineering Daily.

DuckDB is a relational database management system with no external dependencies, with a simple system for deployment and integration into build processes. It enables complex queries in SQL with a large function library, and provides transactional guarantees through multi-version concurrency control. Hannes Mühleisen works on DuckDB and joins the show to talk about query engines

The post DuckDB with Hannes Muleisen appeared first on Software Engineering Daily.

Substack is a platform for subscription based content, covering newsletters, podcasts, and soon video. The design and speed of the platform are notable–with a minimalistic format that simply gets creators paid for its work, Substack has gained massive popularity. Chris Best is the CEO of Substack and joins the show to talk about the platform,

The post Substack with Chris Best appeared first on Software Engineering Daily.

Customer data pipelines power the backend of many successful web platforms. In a customer data pipeline, data is collected from sources such as mobile apps and cloud SaaS tools, transformed and munged using data engineering, stored in data warehouses, and piped to analytics, advertising platforms, and data infrastructure. RudderStack is an open source customer data

The post RudderStack Engineering with Soumaydeb Mitra appeared first on Software Engineering Daily.

Managing Kubernetes nodes leads to operational complexity, security issues, and nodes that are perhaps more expensive to run than necessary. Deferring the node management to an underlying platform abstracts away these problems and can improve operations. Madhuri Yechuri runs Elotl, a nodeless Kubernetes platform. She joins the show to talk about the architecture and purpose

The post Nodeless Kubernetes with Madhuri Yechuri appeared first on Software Engineering Daily.

Retool is a company that allows customers to build complex internal tools using a high level GUI. Users configure the relationships between these different tools, giving them the ability to build applications even without much background in engineering. Of course, having some engineering expertise helps, as users can write JavaScript to interact with the higher

The post Retool Engineering with Snir Kodesh appeared first on Software Engineering Daily.

For over 20 years, our unquenchable appetite for data has led to us inventing and investing in a myriad of technologies, like SQL and NoSQL databases, streaming services, data lakes and warehouses, machine learning, and many more. The demand for data isn’t slowing down, with 97.2% of organizations investing in big data and AI. However,

The post Why Everyone Needs a Data Privacy Vault appeared first on Software Engineering Daily.

The software supply chain consists of packages, imports, dependencies, containers, and APIs. These different components each have unique security risks. To ensure the security of their software supply chain, many developers use tools to analyze and scan their infrastructure for vulnerabilities. Barak Schoster works at Bridgecrew, a DevSecOps cloud security platform. He joins the show

The post Software Supply Chain with Barak Schoster appeared first on Software Engineering Daily.

The data lake architecture has become broadly adopted in a relatively short period of time. In a nutshell, that means data in it’s raw format stored in cloud object storage. Modern software and data engineers have no shortage of options for accessing their data lake, but that list shrinks quickly if you care about features

The post Apache Hudi with Vinoth Chandar appeared first on Software Engineering Daily.

National Instruments develops software and hardware for engineering in a wide variety of domains, from aerospace to government technology to application testing. The interface between hardware and software presents a variety of difficult engineering challenges. Luke Schreier is a Senior Vice President at National Instruments and joins the show to discuss the engineering and management

The post National Instruments with Luke Schreier appeared first on Software Engineering Daily.

Automating video search requires a data pipeline that extracts metadata from videos and allows users to annotate the video with information that correlates with that metadata. The video needs to be segmented into intervals that define the search space, and the search space needs to be queryable by a user. Sieve Data is an automated

The post Video Search with Mokshith Voodarla appeared first on Software Engineering Daily.

Increasingly, technology groups need to be strategic about the cloud services they adopt to ensure their vendor’s pricing is both fair and measured exactly right for the application’s unique access pattern. Definitely answering questions like these can take a significant amount of time and energy from your most valuable engineers. Taloflow is a company trying

The post Taloflow with LV Jadavji appeared first on Software Engineering Daily.

Developers struggle with mental health, and this struggle has only gotten more acute during the pandemic. Sitting in front of a computer all day, engulfed by social media and code can cause us to lose our sanity. I personally had some issues that were hard to grapple with during the COVID lockdowns. In today’s show,

The post Mental Health with Kelsey Hightower appeared first on Software Engineering Daily.

A data catalog provides an index into the data sets and schemas of a company. Data teams are growing in size, and more companies than ever have a data team, so the market for data catalog is larger than ever. Mark is the CEO of Stemma and the co-creator of Amundsen, a data catalog that came out of

The post Data Catalog in Practice with Mark Grover appeared first on Software Engineering Daily.

Splunk is a monitoring and logging platform that has evolved over its 18 years of existence. In its modern focus on observability it is focused on open source and AIOps. Observability has evolved with the growth of Kubernetes, and Splunk’s work around OpenTelemetry has kept parity with the open source community of Kubernetes. Spiros Xanthos

The post Splunk Platform with Spiros Xanthos appeared first on Software Engineering Daily.

Francesco Cesarini founded Erlang Solutions in 1999 with a mission to help companies adopt Erlang. In this interview, I speak with Francesco and Gabor Olah from Erlang Solutions. We discuss the Erlang language, it’s ecosystem, and features like concurrency, resilience, and scalability that motivate adoption. We use Java and the Java Virtual Machine as a

The post Erlang Deep Dive with the Erlang Solutions team appeared first on Software Engineering Daily.

Serious software projects require several environments. Your production environment is obviously mission critical. A staging environment is also necessary to perform validation and regression testing before taking the risk of pushing an update to production. Best practices and approaches for managing these and other environments vary from organization to organization. In some sense, different software

The post The Staging Dichotomy with Senthil Padmanabhan appeared first on Software Engineering Daily.

Barry McCardel Co-Founder and CEO at Hex Caitlin Colgrove Co-Founder and CTO at Hex In contrast to other IDEs, the notebook interface offers software developers a unique environment idealized for data professionals. Despite the growth in popularity, a surprising learning curve still exists for setup and configuration. A siloed notebook offers no native collaboration tools.

The post Hex Collaborative Data Workspace with Barry McCardel and Caitlin Colgrove appeared first on Software Engineering Daily.

When writing code, test driven development is a common accepted methodology to ensure the development of high quality software. Your organization’s data, on the other hand, is an entirely different challenge. Data can be missing due to human error, a failure with a 3rd party provider, a botched release, or dozens of other issues. When

The post Data Quality Using Anomalo with Jeremy Stanley appeared first on Software Engineering Daily.

When you visit a web page, the creator’s intent is to present you a seamless experience that fills your browser window. That web page or web application is generally divided up in some meaningful way across navigation elements, content, ads, header, footer, and other components. Those components may represent the work of independent teams. Typically

The post Micro-Frontends with Luca Mezzalira appeared first on Software Engineering Daily.

Lee Atchison spent seven years at Amazon working in retail, software distribution, and Amazon Web Services. He then moved to New Relic, where he has spent four years scaling the company’s internal architecture. From his decade of experience at fast-growing web technology companies, Lee has written the book Architecting for Scale, from O’Reilly. As an

The post Architecting for Scale with Lee Atchison appeared first on Software Engineering Daily.

The creator economy has seen rapid growth, thanks largely to software solutions like Patreon that are enabling creators. As the creator economy grows, providers must be prepared for scalability issues and the challenges of maintaining and growing a software infrastructure and the team that evolves it. In this episode, I interview Utkarash Srivastava, SVP Engineering

The post Patreon Engineering with Utkarsh Srivastava appeared first on Software Engineering Daily.

Snyk is a platform for security that started with open source scanning and has expanded into container security, infrastructure as code, and other products. Snyk is a simple product to use, but has hidden complexities that build large data structures to manage and scan code dynamically. In a previous episode we discussed the core Snyk

The post Snyk Engineering with Guy Podjarny appeared first on Software Engineering Daily.

Linkerd is a service mesh that runs efficiently with a low memory footprint. We have covered the details of Linkerd in previous episodes. Buoyant is the company that sells Linkerd as a service, and today’s show focuses on the engineering details of the company, and how Linkerd is architected in 2022. William Morgan is the

The post Buoyant Cloud with William Morgan appeared first on Software Engineering Daily.

Online poker was deemed illegal in the United States ten years ago. Since then, poker has decreased in popularity, then found new invigoration thanks to live streaming and a large volume of bored gamers looking for something to do during the pandemic. Poker is a strategy game that can be played even without the financial

The post Clubs Poker with Taylor Crane appeared first on Software Engineering Daily.

Database product companies typically have a few phases. First, the company will develop a technology with some kind of innovation such as speed, scalability, or durability. The company will offer support contracts around that technology for a period of time, before eventually building a managed, hosted offering. PlanetScale is a database company built around the

The post Scaling PlanetScale with Sugu Sougoumarane appeared first on Software Engineering Daily.

Couchbase is a distributed NoSQL cloud database. Since its creation, Couchbase has expanded into edge computing, application services, and most recently a database-as-a-service called Capella. Couchbase started as an in-memory cache and needed to be rearchitected to be a persistent storage system. In this episode, I interview Ravi Mayuram, SVP Products and Engineering at Couchbase

The post Couchbase Architecture with Ravi Mayuram appeared first on Software Engineering Daily.

Rackspace is a multi cloud solutions provider that has evolved beyond its cloud computing origins into a diverse set of services and support offerings. Customers work with Rackspace to adopt cloud application deployments, modern data analytics, and all the other opportunities offered by cloud computing. Much of this occurs through partnerships where Rackspace provides teams

The post Rackspace with Jeff DeVerter appeared first on Software Engineering Daily.

Historically, search engines made money by showing sponsored ads alongside organic results. As the idiom goes, if you’re not paying for something, you are the product. Neeva is a new take on search engines. When you search at neeva.com, you get the type of result you’d expect from a search engine minus any advertising. In

The post Ad-free Search on Neeva with Darin Fisher appeared first on Software Engineering Daily.

Tabnine is an AI assistant that helps software engineers write more efficient code. It’s been trained on a large corpus of source code or can be trained based on your specific codebase. Either way, the resulting model offers predictive completion of code that can make programmers more productive. In this episode, I interview Eran Yahav,

The post Tabnine with Eran Yahav appeared first on Software Engineering Daily.

Protecting your customers begins with best practices for securely capturing, storing, and protecting the data you collect for or about them. When an organization has a large enough dataset, needs typically arise for doing analytical workloads or training machine learning models on this data. If you use random or mock data to generate a report

The post Privacy Engineering with Alex Watson appeared first on Software Engineering Daily.

When businesses share a common need such as payroll, commercial offerings can compete for market share with software solutions that easily adapt to a variety of businesses. Not all tasks can be easily commoditized or standardized. Take content moderation as an example. Every site that accepts user generated content is likely to have unique and

The post Flowdash with Nick Gervasi appeared first on Software Engineering Daily.

If you’re working on a proof of concept which you hope will help you raise funding, it’s fine to take a few shortcuts. Use the tech stack you know the best, don’t fall in love with your code, and when you start to experience growing pains, hopefully you’ll have the time to thoughtfully and carefully

The post Uber State Machine with Uday Kiran Medisetty appeared first on Software Engineering Daily.

Writing software is an absolute joy. Getting software to build is a chore. Thus, build systems emerged as a solution to automate this chore. At some point software engineers either use or hear legends about make and makefiles. While perhaps being the historically known tool, a great deal of thought has gone into approaches to

The post Build Tools with Benjy Weinberger appeared first on Software Engineering Daily.

Data onboarding – the process of importing a customers’ required data into a software product – is often a fairly manual process and the lack of automation means data onboarding falls squarely into the lap of the organization’s software development team. In smaller companies, this translates to one or two individuals focused on importing customer

The post 6 Data Import Challenges Facing Software Developers appeared first on Software Engineering Daily.

Infrastructure as code is a concept that has delighted software engineers, dev ops, and engineering management across the board. It’s neither fun nor efficient to configure the infrastructure and environments software teams require. Operating software at scale on a cloud, on-prem, or hybrid model is a problem of modernity that many enterprises find surprisingly challenging.

The post Infrastructure as Code with Rob Hirschfeld appeared first on Software Engineering Daily.

Pinterest is a visual discovery engine people use to find ideas in home, food, style, beauty, and more. The service grew quickly after its founding in 2010 and the company has grown to be a global team of thousands of professionals spanning time zones and continents. To deliver the smooth experience Pinterest users have come

The post Pinterest Engineering appeared first on Software Engineering Daily.

If you were to poll most organizations, the majority of people within them aren’t particularly fond of incidents. They are disruptive, sometimes damaging, and almost always have a negative connotation. The natural reaction is to simply want them to stop as quickly as possible. But as an industry, we need to evolve to

The post More, more, more! Why the most resilient companies want more incidents appeared first on Software Engineering Daily.

Charlie Gerard is an incredibly productive developer. In addition to being the author of Practical Machine Learning in JavaScript, her website charliegerard.dev has a long list of really interesting side projects exploring the intersection of human computer interaction, computer vision, interactivity, and art. In this episode we touch on some of these projects and broadly

The post Practical Machine Learning in JavaScript with Charlie Gerard appeared first on Software Engineering Daily.

TechLit Africa is a non-profit on a mission to lessen African poverty by leveraging the internet. Rural Africans lack digital skills and computers to gain from the digital economy, even though developed countries have an abundance of used computers. That’s where TechLit Africa comes in. They accept used computers, refurbish them with custom classroom code,

The post TechLit Africa with Nelly Cheboi appeared first on Software Engineering Daily.

Getting a computer program to run the same in different environments has been a recurring problem since the earliest days of software systems. Software versioning, versions of dependencies, hardware configurations, and CPU instruction set differences are just a few examples of challenges engineers have faced to get their software to run in different settings. A

The post Codename One with Steve Hannah appeared first on Software Engineering Daily.

With the push for stronger security, new employees are often frustrated during an initial couple of days of their employment, just waiting to get access to all the resources they’ll need as IT staff frequently has to manually go through each and every system and generate tokens for the new hire. Given the current industry

The post Automate your Infrastructure Access Workflows appeared first on Software Engineering Daily.

If you haven’t encountered a data quality problem, then you haven’t yet worked on a large enough project. Invariably, a gap exists between the state of raw data and what an analyst or machine learning engineer needs to solve their problem. Many organizations needing to automate data preparation workflows look to Trifacta as a solution.

The post Trifacta with Joe Hellerstein appeared first on Software Engineering Daily.

As the internet has grown, increasingly, we are consumers of services provided by corporations rather than owners and operators of our own systems. To many, this trend towards centralization is antithetical to the spirit of a free and open internet. Urbit is a new operating system and peer-to-peer network. There are several layers of novel

The post Urbit with Galen Wolfe-Pauly appeared first on Software Engineering Daily.

InfluxDB is an open-source time-series database. It’s maintained by InfuxData who offers a suite of products that help organizations gain insights from time-series data. In this episode, I interview Zoe Steinkamp, Software Engineering and Developer Advocate at InfluxData. We explore some of the common use cases for time-series databases such as IoT and some recent

The post InfluxData with Zoe Steinkamp appeared first on Software Engineering Daily.

Robotic process automation or RPA refers to software robots constructed to automate some business process. Perhaps the most ubiquitous example is adding filters to your email inbox. I’ve worked with a lot of salespeople that configure complex email follow-up campaigns when inbound emails come in, but even that’s a fairly basic example compared to what’s

The post UiPath with Boris Krumrey appeared first on Software Engineering Daily.

Everyone is becoming increasingly aware of supply chains for physical goods. Software has its own supply chain. A supply of open source solutions exists as does a demand for these solutions by industry. Both have surely grown, but it would be nice to have a way of measuring by how much. The State of Software

The post The State of Software Supply Chain 2021 with Ilkka Turunen appeared first on Software Engineering Daily.

There are a variety of new and upcoming security trends that will have a direct impact on software engineering workflows in 2022. In this article we’ll look at three emerging security approaches, and how new technologies–such as strongDM– can potentially help organizations become more secure. 1. Zero Trust The Zero Trust model was

The post The 3 Security Trends Impacting Engineering Workflows appeared first on Software Engineering Daily.

Relational databases have been a fixture of software applications for decades. They are highly tuned for performance and typically offer explicit guarantees like transactional consistency. More recently, there’s been a figurative cambrian explosion of other-than-relational databases. Simple key value stores or counters were an early win in this space. Managing a graph data structure is

The post MemGraph with Dominik Tomicevic appeared first on Software Engineering Daily.

The lifeblood of most companies is their sales departments. When you’re selling something other than a commodity, it’s typically necessary to carefully groom the onboarding experience for inbound future customers. Historically, companies approached this in a one-size-fits-all manner, giving all customers a common experience. In today’s data-driven age, a better experience can be provided that

The post Amplemarket with João Batalha appeared first on Software Engineering Daily.

As cloud providers enable greater levels of specificity and control, they empower compliance-driven enterprise companies. This level of parameterization is downright inhospitable to a new software engineer and can be a cognitive barrier to entry for a senior professional with a great idea but limited time. Developers want to focus on their code, algorithms, front

The post Render with Anurag Goel appeared first on Software Engineering Daily.

Writing your application’s code is only half the battle. Getting it to run on your machine is a milestone, but it’s far from your code running in a production environment. There are an increasing set of options application designers have for helping to manage deployment, environments, and CI/CD. Encore is a backend engine for the

The post Building Go Apps Using Encore with André Eriksson appeared first on Software Engineering Daily.

Kubernetes is here to stay. The benefits the container-orchestration systems provide to cloud-native applications are incalculable. It has become the de facto standard for automating application deployment, scaling, and management. However as the ecosystem matures, several roadblocks have emerged on the way to a production-ready system. Several issues like high availability and monitoring have been

The post Simplifying Access to Kubernetes appeared first on Software Engineering Daily.

Microservice architecture has become a ubiquitous design choice. Application developers typically have neither the training nor the interest in implementing low-level security features into their software. For this and many other reasons, the notion of a service mesh has been introduced to provide a framework for service-to-service communication. Today’s guest is Zack Butcher. While working

The post Tetrate Service Bridge with Zack Butcher appeared first on Software Engineering Daily.

Many software projects run the risk of evolving over time to a complex state that is inhospitable for new contributors to join. This is a dangerous place for a company to be. Either software needs to remain more accessible, or faster paths must be created to help them get on board. Today’s interview is with

The post Understandable Software with Kartik Agaram appeared first on Software Engineering Daily.

Climate modeling is increasingly important as supply chains, emergency management, and dozens of other efforts need to make predictions about future conditions and how they will impact business. Analyzing climate data requires geospatial systems, and those systems need a full-stack geospatial technology solution. Gopal Erinjippurath serves as CTO and Head of Product at Sust Global,

The post The Software of Climate Adaptation with Gopal Erinjippurath appeared first on Software Engineering Daily.

Once a machine learning model is trained and validated, it often feels like a major milestone has been achieved. In reality, it’s more like the first lap in a relay race. Deploying ML to production bears many similarities to a typical software release process, but brings several novel challenges like failing to generalize as expected

The post Responsibly Deploy AI in Production with Anupam Datta appeared first on Software Engineering Daily.

Internships can be an incredibly valuable resource to new professionals and are often the first professional work experience for many participants. It’s often the case that internship programs are suboptimal. Employers don’t always provide a clear path to success for the intern. Interns in turn don’t always have a resource to reach out for help

The post Internship Management Solutions with Nikita Gupta from Symba.io appeared first on Software Engineering Daily.

When creating a website, there’s no shortage of choices for how to do it. Builders must make strategic decisions about the language or framework they want to adopt. An important first consideration for many is selecting a web application framework like React or Vue. Motivated by a low page response time and good user experience,

The post Nuxt.js with Alexander Lichter appeared first on Software Engineering Daily.

Application observability is a fairly mature area. Engineering teams have a wide selection of tools they can choose to adopt and a significant amount of thought leadership and philosophy already exists giving guidance for managing your application. That application is going to persist data. As you scale up, your system is invariably going to experience

The post Metaplane with Kevin Hu appeared first on Software Engineering Daily.

Consumers are increasingly becoming aware of how detrimental it can be when companies mismanage data. This demand has fueled regulations, defined standards, and applied pressure to companies. Modern enterprises need to consider corporate risk management and regulatory compliance. In this interview, I speak with Terry O’Daniel, Director of Engineering (Risk & Compliance) at Instacart. Sponsorship

The post Risk and Compliance with Terry O’Daniel appeared first on Software Engineering Daily.

Thanks to the amazing books, blogs, videos, quickstarts, frameworks, and other software-related resources, getting started as a software engineer is easier than ever. Although you can get started in a day, it can take years to become a master of the craft and most practitioners describe it as a profession of lifelong learning. Titus Winters

The post Software Engineering at Google with Titus Winters appeared first on Software Engineering Daily.

With a few impressive exceptions, software is rarely written by one person. It takes a team and as that team outgrows a single shared office, coordination and communication become emergent problems. There are lots of lessons to be learned from companies that have already found approaches that scale. In this episode, I interview Tramale Turner,

The post Building Engineering Teams with Tramale Turner appeared first on Software Engineering Daily.

Neural networks, in particular, deep neural networks have revolutionized machine learning. Researchers and companies have pushed on the efficiency of every aspect of the machine learning lifecycle. The impact of the trained models is particularly significant for computer vision and in turn for autonomous driving and security systems. In this episode, I interview Forrest Iandola,

The post Deploying Computer Vision to the Edge at Anduril Industries with Forrest Iandola appeared first on Software Engineering Daily.

Modern businesses run on the cloud and increasingly so they run on multi-cloud infrastructure. As any growing company can tell you, cloud costs can easily run far out of control. Today’s enterprises are trying to deliver new products and services at a fast pace. That needs to be done in a cost-effective, ideally cloud-agnostic way.

The post Yotascale with Jake Reichert appeared first on Software Engineering Daily.

To many people’s surprise tech sales is not much of an art. It’s actually a regimented science where reps have clear step-by-step processes to bring in new business. Each stage takes the customer closer to the end of the deal and consists of learning more about the customer’s needs. A CRM is a database reps

The post AI in Sales with Ohad Parush appeared first on Software Engineering Daily.

The banking industry uses technology that some modern software engineers may regard as out of date or old-fashioned. Entrepreneurs wanting to create products in the banking space historically faced a steep curve to build software that could integrate with established banking systems. Christopher Dean seeks to change that. He founded Treasury Prime, a company that

The post Treasury Prime with Christopher Dean appeared first on Software Engineering Daily.

The internet is a layer cake of technologies and protocols. At a fundamental level, the internet runs on the TCP/IP protocol. It’s a packet based system. When your browser requests a file from a web server, that server chops up the file into tiny pieces known as packets and puts them on the network labeled

The post Scalable Streaming Video with Amit Mishra appeared first on Software Engineering Daily.

Machine learning models must first be trained. That training results in a model which must be serialized or packaged up in some way as a deployment artifact. A popular deployment path is using Tensorflow.js to take advantage of the portability of JavaScript, allowing your model to be run on a web server or client. Gant

The post Learning Tensorflow.js with Gant Laborde appeared first on Software Engineering Daily.

It does not matter if it runs on your machine. Your code must run in the production environment and it must do so performantly. For that, you need tooling to better understand your application’s behavior under different circumstances. In the earliest days of software development, all we had were logs, which are still around and

The post Observability Using Honeycomb.io with Christine Yen appeared first on Software Engineering Daily.

Angular is a free and open-source web application framework. It’s maintained by the Angular team at Google. It’s used by millions of web applications and has a strong ecosystem of core contributors and library builders. In this episode, I interview Minko Gechev, Developer Relations Lead at Google. We explore several aspects of open-source software development,

The post Angular Dev Tools with Minko Gechev appeared first on Software Engineering Daily.

It wasn’t that long ago that companies scheduled downtime in order to release an updated version of the software running their website. That’s rare today. Most developers want continuous testing, integration, and deployment. While that comes with many benefits, it also places greater demands on quality engineers who can no longer gate all updates into

The post The Future of Quality Engineering with Liliya Frye appeared first on Software Engineering Daily.

The manner in which users interact with technology has rapidly switched to mobile consumption. The devices almost all of us carry with us at all times open endless opportunities for developers to create location-based experiences. Foursquare became a household name when the introduced social check-ins. Today they’re a location data platform. Ankit Patel is the

The post Location-Based Experiences Using Foursquare with Ankit Patel appeared first on Software Engineering Daily.

The React Framework has seen continuous growth of adoption since its launch. There are many reasons for that, but one reason is how relatively painless it is to use react-create-app or copy some boilerplate code and have a functioning, hot reloading, live demo up and running in minutes. There is, however, a long way to do between proof of concept and enterprise app.

Paige Niedringhaus is a software engineer and creator of The Newline Guide to Modernizing an Enterprise React App. In this episode, we discuss best practices and strategies for structuring your React project.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Enterprise React Apps with Paige Niedringhaus appeared first on Software Engineering Daily.

Welcome to Software Engineering Daily; I’m your guest host, Joey Baruch. I’m the CTO at Alvarez and Marsal Data Intelligence Gateway (A&M DIG), prior to which I co-founded and was CTO of HuMoov, a vertical SaaS. I’ve been a software engineer at PayPal, IBM Research Labs, and Qualcomm via the acquisition of Wilocity.

Joining me is Fernando Gómez-Baquero, the Director of the Runway and Spinouts Program at the Jacobs Technion-Cornell Institute at Cornell Tech, which is the technology campus of Cornell University located on Roosevelt Island in New York City. Fernando is a nanomaterials scientist and serial entrepreneur. He’s built companies around composite aerospace materials, technologies for energy storage and has mentored 100s of companies – mostly with deep tech innovations at their core. He also manages a portfolio of 82 companies created at Cornell Tech and the Jacobs Institute, with a combined enterprise value of over half a billion USD.

Fernando joins us to talk about his experience, speaking to the engineer with the entrepreneurial spirit, encouraging the journey, and laying out the core attributes of a process that can be taught, which will help the entrepreneur build a great company. He reminds us of what’s mission-critical and what can be left for another day.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Show Notes Cornell Tech: https://tech.cornell.edu/

Runway Program at the Jacobs Technion-Cornell Institute: https://tech.cornell.edu/programs/phd/startup-postdocs/

Lunchclub: https://lunchclub.com/

NewLab: https://newlab.com/

Company: https://company.co/

Activate: https://www.activate.org/

Tech.NYC: https://www.technyc.org/

Aren.ai : https://aren.ai/

Nanit: https://www.nanit.com/

The post Building Startups Around Deep Tech Innovations with Fernando Gómez-Baquero appeared first on Software Engineering Daily.

According to builtwith.com, more than 10 million websites are powered by React framework. Of the top 10k sites by traffic, 44.7% of those are built with React. This javascript framework is capable of powering a wide array of modern applications and remains fairly beloved by developers that use it.

In this episode, I interview Kent C. Dodds, Software Engineer Educator. We discuss Kent’s journey learning React and keeping up with the changes it’s taken on as the framework evolves and how those lessons culminated into his Epic React course.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Course Link: https://epicreact.dev/

The post Learning React with Kent C. Dodds appeared first on Software Engineering Daily.

Modern business applications are complex. It’s not enough to have raw logs or some basic telemetry. Today’s enterprise organizations require an application performance monitoring solution or APM. Today’s applications are complex distributed systems whose performance depends on a wide variety of factors. Every single line of code can affect production and teams need insights into the health of the system and how to improve them.

In this episode, I interview Omri Sass and Hugo Kaczmarek from Datadog, a provider of cloud monitoring as a service. We discuss the APM space and the challenges faced by modern enterprise teams. We also get into some detail about their new live search option and why that solution was an important addition to their product suite.

Full disclosure: Datadog is a sponsor of Software Engineering Daily.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Datadog with Omri Sass and Hugo Kaczmarek appeared first on Software Engineering Daily.

One of the most painful parts of getting started on a new development team is getting one’s environment set up. Whether it’s undocumented steps, overly complex setups, or simply the challenges of understanding how the pieces fit together, getting a dev environment up often feels like a chore to be suffered through in order to do what I want: contribute to the codebase.

Gitpod seeks to solve these and other common challenges. With Gitpod, you can spin up fresh, automated dev environments, in the cloud, in seconds. In this episode, I interview Sven Efftinge, CEO of Gitpod.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Remote Development in the Cloud with Gitpod & OpenVSCode Server with Sven Efftinge appeared first on Software Engineering Daily.

The notebook paradigm of coding is relatively new in comparison to REPLs and IDEs. Notebooks run in your browser and give you discrete cells for running segments of code. All the code in a single cell runs at once, but cells run independently. Cells can be re-run, which is a blessing and a curse. The ability to run cells out of order can make it difficult for users to have a clear understanding of what else they might want to re-compute. The NB Safety project is an easy-to-install tool for automated management of notebook states which can help you catch bugs early.

Stephen Mackey is a Ph.D. student in the Data and Information Systems laboratory at UIUC. In this episode, we discuss Jupyter Notebooks, the development of a custom kernel, and how NB Safety can help notebook users.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post NBSafety for Jupyter Notebooks with Stephen Macke appeared first on Software Engineering Daily.

Virtual meetings were growing in popularity before the need accelerated as a result of the pandemic. Gather is a place where you can create a space for your community today. Users who join find themselves in a shared virtual space that offers the ability to interact with other users as well as interact with the environment itself.

In this episode, I interview Phillip Wang, CEO at Gather about the platform, online communities, and bridging the gap between people, technology, and connecting with each other.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Online Communities on Gather with Phillip Wang appeared first on Software Engineering Daily.

Imagine a world where you own some sort of building whether that’s a grocery store, a restaurant, a factory… and you want to know how many people reside in each section of the store, or maybe how long did the average person wait to be seated or how long did it take the average factory worker to complete their assembly task.

Currently today these systems are either not using AI and instead use a mix of sensors and buttons to track certain actions or they do use AI but in a way that’s highly specific to their use case and hard to easily modify for new use cases that come down the line.

This is where BrainFrame comes in. BrainFrame is a tool that connects to all your on-prem cameras and lets you easily leverage AI models and business logic. Alex Thiele is the CTO of Aotu the company that makes BrainFrame and he joins me today to talk about BrainFrame and the vision for a future where computer vision can be run by anyone.

This episode is hosted by David Cohen. David is a Software Engineering Lead at LinkedIn where he works on backend applications and APIs that power their enterprise data systems. In his free time, he is an AI enthusiast and enjoys talking about all things Software. You can contact him on LinkedIn or Twitter.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Show Notes: * BrainFrame * OpenCV BrainFrame Capsules

The post No Code AI for Video Analytics with Alex Thiele appeared first on Software Engineering Daily.

The gig economy involves independent contractors engaging in flexible jobs. Today gig workers often get work from centralized platforms that facilitate the process of connecting workers with employers in exchange for a fee. Some workers find the relationship between worker and platform to be adversarial in nature since the platform can establish and enforce rules at its own discretion.

In this episode, I interview Adam Jackson, Founder & CEO of Freelance Labs, builders of Braintrust. We discuss the state of the gig economy and his vision for how Braintrust can create a new kind of marketplace.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post The Future of Gig Work with Adam Jackson appeared first on Software Engineering Daily.

Venture capital investment has continued to flow into technology startups. No one builds technology from scratch. There are cloud services, software libraries, 3rd party services, and software platforms that modern entrepreneurs must adopt to build their products efficiently and quickly. These layers of infrastructure are a key area for many investors.

In this episode, I interview Tim Chen, managing partner of Essence VC, a venture fund on a mission to help highly technical founders go from zero to one. We discuss his approach to investment in infrastructure companies, developer tools, and similar areas for early-stage investments.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Investing in Infrastructure and Dev Tools with Timothy Chen appeared first on Software Engineering Daily.

Whether you love them or hate them, share them or ignore them, you encounter memes all over the internet. Those that are popular can often take off and spawn a long history of remixes, variants, derivatives, and inspired works.

In this episode, we interview Johan Unger, the founder of meme.com. They’re creating a platform for Meme Explorers to track these Memes and earn rewards along the way.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Meme.com with Johan Unger appeared first on Software Engineering Daily.

The last 15 years have seen the emergence of cloud-based developer APIs and services as dominant components of the developer toolchain. As a result, there has never been more power at developers’ fingertips. But making that power usable and accessible is a challenge that is shared between the providers and the consumers of these services.

Google’s Developer Relations team has spent the past 15 years bridging the gap between Google’s internal engineering teams and the broader developer community that consumes their APIs and services. Luke Mahé is a long-time engineering manager at Google DevRel. We talk about the history of Google’s developer offering and the critical role DevRel plays in shaping and promoting that offering.

This episode is hosted by Yaniv Bernstein. Yaniv Bernstein was until recently COO of Airtasker and is now a startup coach, adviser, and public speaker. He is the author of the People Engineering newsletter and is active on LinkedIn and Twitter.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Developer Relations at Google with Luke Mahe appeared first on Software Engineering Daily.

Decision-making around software projects has always been a contentious issue, should one move in an extremely thoughtful – planning all the way through, OR should teams opt for a quick approach, iterating through ideas, even if it means the approach might not be picture perfect. While both approaches lead to some trade-off, the former leading to time being consumed (where competitors can launch products, the idea can become obsolete, etc) and the latter causing unintended bugs (possibly making it all the way to production).

Google often is regarded as the organization founded by the PhDs who wish to take things slow ensuring that everything’s done in as much of a picture-perfect manner as possible, whereas Facebook (at least in the early days) was a company full of college-going engineers who chose to move fast and get something out there, even if that meant a few things would most likely break.

This article will summarize an interview with Jeff Meyerson, author of Move Fast: How Facebook Builds Software. Daliana Liu, the host for this episode raises a good point, “All the tech companies, they have to move fast to win competition, launch products. So what’s specific about Facebook that they move fast in a way that’s different from other companies?”

Jeff’s response to the question summarizes Facebook’s strategy in the early days: “Facebook basically said, ‘Look, we can’t compete with that. We have to do something different.’ And so they said, ‘Okay, we’re just going to step on the gas as fast as we can, and just build as fast as we can, and put some loose numbers around what success means and just drive towards those numbers like really aggressively, very aggressively.’ And that’s move fast.”

The interview goes on to talk about another rather interesting (and albeit controversial) topic covered in the book Move Fast, and that is the 10x engineer and the influential engineer.

However, Jeff is very much aware of the fact that arbitrarily judging someone to be “10x” is a naive thing to do. He puts forth the argument that while he does believe in 10x engineers, he’d like that there be some established KPI against which engineers can be compared, which would eventually lead to finding a few engineers who end up banking 10x more points on the metric than the average.

For the influencer engineer, the water gets a bit murky. The general philosophy here is code wins arguments. As a result, an influencer engineer would be the one who ends up having a (sizeable) impact on the organization by the code that they’ve written. Jeff goes on to take Nick Schrock here as an example based on the work he has done in writing and architecting GraphQL, something that had a great impact on Facebook.

“You want to move fast both to evade your competitors and to reach your goals as quickly as possible so that you can define new goals.” – Jeff Meyerson

The second half of the interview goes on to talk about ethical concerns with some of the recent mishaps at Facebook, to which Jeff succinctly and rather correctly points out that no one is forced to be on Facebook – and that he (Jeff) does support scrutiny for all firms!

In fact, Jeff even points out Facebook’s biggest enemy – itself. Facebook’s been trending in this rabbit hole of zero-sum hyper-competitive mode where it makes its competitor’s products perform poorly inside its own products (thus discouraging their use). Youtube can be seen as an example here, and how poorly it performs when being used inside Facebook Messenger. Imagine if these products worked in synergy, leading to a positive-sum output and a win-win for both customers and the companies involved!

Fun fact: uploading the audio version of Move Fast on SEDaily rather than a platform like Audible was actually Jeff himself practicing moving fast. Since Audible took too long to publish the audiobook, Jeff decided that it made more sense to have something out there, rather than wait for Audible indefinitely!

While there’s a lot more nuance to the topics covered here and a lot of topics that aren’t touched upon at all in this article, I’d highly recommend checking out Move Fast – it’s on Amazon and the audio version is in fact, free available on the SEDaily website.

Click here to listen to the full episode, or watch it on our YouTube channel.

The post Episode Summary: Move Fast with Jeff Meyerson appeared first on Software Engineering Daily.

Move Fast: How Facebook Builds Software

Coinbase is a popular and well-trusted cryptocurrency platform for buying and selling digital currencies. With Coinbase, your portfolio for crypto investments is managed in a central location. Originally just a web application, Coinbase has now grown to multiple different products across multiple platforms such as web, iOS and Android. With that increased scale came increased complexity as the team began maintaining multiple codebases and hiring individuals to fill the same role for multiple platforms.

Around two years ago, the development team at Coinbase came up with the theory that by using React Native and its ability to develop for multiple platforms with the same codebase, they could reduce the code complexity of maintaining multiple platforms while maintaining the same high quality that people expect from native code. Traditionally iOS and Android development has happened in silos. Each team had its own development team, design team and product management. In addition, since both apps accessed the same backend data models, inconsistencies and headaches would arise with how each app accessed and used the same backend data. React Native promised to solve this problem and at least present a unified mobile platform. However, the journey would take around two years and present its own challenges and headaches.

Coinbase wanted to have an incremental approach where they could test and validate assumptions while maintaining the availability of their core app. They started by rewriting their main app for Android. At the time it was clear that React Native was much less mature, much more buggy and more difficult to make performant. If they could reach the quality bar they wanted to, Android then transitioning the iOS app would be simpler and faster. This assumption proved true as the Android app took over a year and half to transition over and the iOS app took less than six months. However, a transition from native Katlin and Swift apps was not without its own problems.

One of the biggest problems that the Coinbase team faced was people management. Coinbase had already developed two separate mobile development teams that specialize in iOS and Android development and convincing them to move over to React Native was an arduous process. Coinbase set up a React Native school where they had an expert teacher to select individuals. Those individuals then paired up with individual team members to pass on knowledge and give a tour of the features and capabilities. Only when everyone had their hands dirty writing React Native code did they begin the transition of the codebase. It required a large initial investment in time and money and continuous investment to improve the team and the technical expertise. To this day, Coinbase has a team of React Native consultants to ensure that the team is always taking advantage of new features and developing with best practices in mind.

Another big problem that Coinbase had was recognizing the limitations of React Native. There is this stigma that still exists to this day that React Native code cant be performant or as high quality as native code. Airbnb tried to make the transition to React Native but abandoned the project due to this very reason. However, this is less to do with the actual capabilities of the platform and more to do with the applications being developed. Coinbase made a concerted effort to make sure the performance and quality were as high as possible. This led to several concrete actions. They developed some custom tooling as the React Native debugger was and is still fairly obscure. They had a unified design team that made sure the user experience was consistent and exceptional across platforms and products. They also made sure to retain some of the mobile platform experts and roles as some changes needed to be made per platform experience to ensure an excellent experience. Native experts are important to making React Native look and feel really, really good on their respective platforms. Things like headers growing in IOS when you scroll over, or different rubber banding in certain places and other tiny details disguise the abstraction of Typescript so that end-users feel like they are using a Native Platform.

Part of this success at this rewrite was due to Coinbase’s unique culture of debate, decide, deliver. You make sure to take everyone’s opinion but once a decision has been made, everyone rows in the same direction. That culture led the mobile engineers to trust, work hard, and dedicate themselves to the rewrite. By honestly discussing the migration, the journey, the metrics, and the decision points and documenting them with a heavy paper trail, Coinbase got buy-in from everyone and that was an important part of their success.

Looking back, the transition from native development to React Native was not so clear for Coinbase. They had to pursue a “brownfield” approach to ensure that their product was always running. React Native was the only real solution out there for multi-platform development as alternatives like Flutter and Kotlin multi-platform were in their infancy. There were problems with efficiency and performance. However, nowadays the decision is much more simple, especially in a “greenfield” product. React Native is gaining performance by removing things from the Javascript bridge and new libraries like Reanimated 2 ensure that React Native feels like native code. Alternatives like Flutter and Kotlin are much more mature. Nowadays if you are building a product from scratch, there is no reason to not use one of these libraries. However, the technology stack is much less important than the engineer’s attention to detail and the bar of quality you set for the product.

Looking to the future, Coinbase’s development team sees three waves. Wave number one was creating a unified Javascript platform across web and mobile. That was achieved when they launched React Native application on iOS earlier this year. The second wave is unifying that UI, data and tooling layer. That way when you improve build times by five seconds, it across hundreds of engineers. The third wave is focusing on developer effectiveness, velocity and quality so you can continually iterate and improve those over time and drive key business metrics.

Coinbase is an impactful company. It’s this generation’s Facebook or Amazon. Crypto will only continue to grow and as it does Coinbase will be right beside it providing secure, highly scalable products.

This summary is based on an interview with Brent Walter, a Senior Engineering Manager at Coinbase and Jacob Thornton, a Senior Staff Engineer at Coinbase. To listen to the full interview, click here

The post Episode Summary: Coinbase React Native: Building the Cryptocurrency Ecosystem appeared first on Software Engineering Daily.

Introduction

How should you choose which automatic speech recognition (ASR) solution to invest in? With so many options available, it can be a challenging decision. That’s why Rev.ai has prepared a guide that outlines the key questions to ask when deciding on an ASR vendor.

By considering these questions, you’ll have a better idea of the strengths and weaknesses of each ASR product. How much value you place on each answer will depend on your own needs. So it’s also important to understand what exactly you need ASR for, and if there are any particular must-haves for your products or the pain points you’re trying to solve.

How much uptime can you ensure?

Uptime is one of the most important aspects of any ASR solution, so it follows that this should be the first question you ask of potential ASR vendors. Each ASR vendor will have a guaranteed percentage of uptime for their customers, as laid out in their service level agreement (SLA).

Generally, higher uptimes come at higher costs. But if speech recognition is critical to your work, a high level of uptime is priceless.

Rev.ai guarantees 99.9% uptime in their SLA, which corresponds to just 8 hours and 45 minutes of downtime in a year.

How fast is it?

There are two scenarios to think about when asking for the speed at which an ASR can turn speech into text. The first is real-time streaming applications, such as live-captioning meetings, lectures, or broadcasts. Here, the key measure of speed is latency: the delay between the time something is spoken and the time the corresponding text is returned to the user. You can expect a good ASR to have a latency of under one second.

The second scenario is asynchronous applications, such as generating transcripts from recordings. The key measure here is overall turnaround time. This is usually less critical than latency, as you aren’t relying on your ASR in real-time. However, overall turnaround time can still be an important consideration to take into account—waiting for prolonged periods for a transcript can be frustrating.

How accurate is it?

Accuracy is an important consideration when choosing an ASR vendor. Inaccurate speech recognition software can potentially require many time-consuming corrections.

Different ASR vendors will use different benchmarking approaches to measure accuracy. The primary tool Rev uses, which they suggest you use too, is the Word Error Rate (WER). This gives a percentage of how many words the ASR got wrong. Failures include omitting the right word, inserting the wrong word, and incorrectly substituting one word in the place of another.

Another consideration to make here is whether the ASR vendor includes verbatim words. These include filler words, false starts, and self-corrections. Verbatim words are verbal cues that provide helpful context and set the scene of recording. Rev is currently the only ASR vendor that can offer fully verbatim transcripts. This is thanks to the millions of hours of Rev’s unique verbatim training data from professional transcriptionists.

As ASR vendors will tell you their WER numbers from their own benchmarking processes, Rev recommends running some of your own audio through several ASR products. To compare the results, Rev offers a free, open-source command-line tool, called FSTAlign. You provide FSTAlign with the output text file from an ASR along with a ground-truth transcript and it reports back the WER. FSTAlign is available on GitHub.

When Rev ran a publicly available, 39-hour unedited long-form audio dataset called Earnings-21 through FSTAlign, they found they had the lowest WER value compared to Google and Amazon, among other competitors.

How quickly can I implement it?

There are two main components to consider when it comes to the implementation speed of an ASR. The first is the API and its documentation, and the second is how much fine-tuning the ASR machine learning model will require for your purposes.

There are several factors that will affect how long it will take to achieve the first successful API call-in production. Well-designed APIs with comprehensive and clear documentation can significantly shorten the development life cycle. Some APIs are ready to use straight away, whereas others will require a conversation with a representative from the vendor organization to get production keys. It is also worth finding out whether API error codes are clearly defined and which programming languages software developer kits are available for.

Implementation speed can also be greatly affected by how much adjustment the ASR machine learning model requires before it’s fully operational. Will you have to fully train a model from scratch or can you tweak a prebuilt model for your own use cases? How much of your own data will you need to provide before the model operates at suitable performance levels? Will you need data science expertise in your team?

Further questions to ask potential ASR vendors

Rev encourages you to get in touch with multiple vendors, ask for the details above, and compare the results. Other questions you may want to ask include:

How easy is it to customize?
How does it perform across multiple demographics?
Is there support for multiple languages?
How well does it, separate speakers?
Does it include automatic punctuation?
What level of developer support is offered?
Who owns the data and where is it stored?
How much does it cost and what’s the contract length?
Can I try before I buy?
What does a true partnership look like?

Many of these considerations are nuanced, especially when it comes to culture. For more details, read Rev’s guide.

The post Questions to Ask When Searching for an ASR Vendor appeared first on Software Engineering Daily.

The expression firing on all cylinders dates back to the early 1900s and refers to a function of the internal combustion engine. This expression poetically applies to successful businesses as well. Each department must operate at peak performance and the couplings between departments need optimization as well.

In this episode, I interview business coach Jon Dwoskin about a variety of topics related to entrepreneurship and how to engineer a successful organization.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Getting Businesses Unstuck with Jon Dwoskin appeared first on Software Engineering Daily.

Infrastructure as Code is an approach to machine provisioning and setup in which a programmer describes the underlying services they need for their projects. However, this infrastructure code doesn’t compile a binary artifact like traditional source code. The successful completion of running the code signals that the servers and other components described in the configuration file have been created automatically by the tools being used.

In this episode, I speak with Christian Tragesser, DevOps Consultant with World Wide Technology. We discuss the tools and processes that are helping modern development groups solve technical problems.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Infrastructure as Code with Christian Tragesser appeared first on Software Engineering Daily.

The internet was designed to move data worldwide, despite technological advancements, global catastrophes, or your geography.

It’s meant to be reliable, no matter any of those conditions.

But the evolution of the internet has seen an increased pressure to deliver a larger volume of data while maintaining the same level of reliability.

For example, real-time applications have begun to take over the internet. While they don’t necessarily rely on transferring large amounts of data, they do require stable connections and the lowest possible latency to avoid minimal delays or interruptions.

Enter WebRTC.

Google first announced WebRTC as an open-source package to facilitate real-time audio and video streaming in a browser-agnostic fashion. Fast forward roughly ten years since the release in 2011, and we have reached a point where we can’t imagine life without using these real-time communication applications.

While the support of WebRTC is still a work in progress, to say the least (Safari being the most notorious offender of all major browsers), both WebRTC and the applications employing WebRTC have come a long way, to the point where for the most part our online communication feels as good as real-time with a decent frame rate and overall video/audio quality.

A vast majority of the applications employing WebRTC rely heavily upon an underlying API present in most modern-day browsers called “getUserMedia.” Here’s where we can draw a correlation-causation analogy. getUserMedia is used to fetch either the audio or video streams from the user’s microphone and camera. Since most WebRTC based applications rely on either voice or video or both, we can make an informed guess that an upward trend in microphone and camera use would imply an increased usage of WebRTC based products, too.

We can glance over the Chrome platform status; for this API (getUserMedia), you can see that the trend keeps going upwards! The slight downward trend can be attributed to the fact that offices started opening up – this isn’t necessarily indicative of the fact that WebRTC’s usage has gone down – remember, we are making a correlation-causation analogy here!

One can quickly realize that this leads to a central point of failure – if a malicious actor understands how to game the WebRTC protocol, they can essentially apply the same formula to most communication tools to overtake them. Since the pandemic caused in-person meetings to be a thing of the past, all talks, be it random jibber-jabber or highly secretive/sensitive business topics, happen over these applications. Security needs to be one of the top concerns of a software group deploying WebRTC.

What exactly would a company that relies heavily on WebRTC in their application want to achieve? Given the current state of WebRTC, there are three things to consider:

Latency
Security of communications
Maintenance cost of infrastructure

Latency To understand what affects latency, we need to know how WebRTC functions. The diagram below describes the handshaking and eventual secure communication between Peer A and Peer B.

A quick takeaway from looking at this picture is that Peer A and Peer B cannot somehow magically communicate with each other. WebRTC needs an intermediary infrastructure in place to somehow connect these two peers.

Suppose Peer A wishes to speak with B, then A generates something called an offer, and the task of the WebRTC protocol is to relay the offer to B. Hiding the underlying intricacies, Peer A’s “offer” includes information that can help Peer B link back to Peer A, the protocol facilitating this transfer is called Session Description Protocol & the act of exchanging the offer is managed by signaling methods which can either be SIP (Session Initiation Protocol) over a WebSocket or a JSON over WebSocket. Once this initial offer/answer combination is exchanged, ICE (Interactive Connectivity Establishment) swoops in and processes the inbound SDP (Session Description Protocol) to then determine the optimal path to connect through – here’s where STUN, TURN & Signalling servers come into the picture.

A STUN server is required by the party initiating the offer (in our case Peer A) to generate a list of ICE candidates. Once that’s done, using the signaling servers, we transmit the offer to the intended Peer (Peer B in our example).

Skipping a few steps ahead, Peer B would now employ the signaling servers again to send an answer to the offer generated by Peer A. Then, based on the answer, the two peers may begin communicating.

You might wonder how and why TURN servers never came into the picture in this entire description. And you’re right – TURN servers are a sort of fallback. In cases where your clients cannot generate a direct Peer-to-Peer connection, TURN (Traversal Using Relay NAT) servers come into the picture and facilitate your peers to connect to each other.

TURN is a must-have for any application that is heavily reliant on WebRTC since not all clients can have a P2P connection. However, these are notorious for introducing latency into your network.

An application like Subspace provides a managed solution to this problem. GlobalTURN is a way to achieve managed TURN servers providing the least latency! The graph below indicates the latency savings achieved when using Subspace instead of normal internet routing.

Think about it – you’d ideally have to figure out the best server pairing by investing development time behind this. Even after investing all this time, as your application expands and your user base grows, this data becomes irrelevant since users from different parts of the world start connecting. By eliminating the act of self-managing, you free up time that can be invested behind actual business logic and let a network experienced in managing these to the most optimal degree do it for you – ensuring your users get the best experience possible. Traffic bottlenecks are one of the significant sources of delay in RTC. GlobalTURN avoids this altogether, making even the TURN-based connections feel as if they were genuinely peer-to-peer.

Security Building on top of what we’ve established so far – we can realize there are many points of failure in this entire communication channel, leaving the peers and subsequently the whole network-wide open to attacks.

A bunch of possibilities includes:

Taking over the initial offer to forge the identity of a malicious actor as Peer A
Taking over the STUN server(s) to make Peer A believe that the malicious actor is in fact Peer B
Taking over the communication channel to listen to the actual data being transmitted between A and B
Taking over the TURN servers and listening to the data being relayed via them.

The worst offender of all would be a Distributed Denial of Service (DDOS) attack on your infrastructure that would overload both the STUN and TURN servers, causing users to not connect in the first place.

Subspace comes to the rescue in this scenario too – providing solid security in every managed solution it provides (PacketAccelerator, GlobalTURN, or SIPTeleport). It does so by masking the IP Addresses of the peers involved by passing their messages through a proxy to prevent the malicious actors from tracing back packets to original machines.

Alongside this, it also performs packet analysis using anomaly detection algorithms to get a hold of unusual activity before it causes significant damage. Not to forget, all of this security generally would end up coming at a price – a price paid by latency, but since Subspace has optimized routing algorithms, the additional security doesn’t come with a majorly noticeable performance hit.

Subspace achieves this level of latency by providing optimization on all seven layers of the OSI model, i.e. from the physical infrastructure to the application layer.

Maintenance cost of infrastructure (TURN, STUN, etc) All the servers mentioned above need to be configured during setup and need to be expanded as your organization needs increased capacity. As mentioned previously, figuring out the right balance & placement of these servers is quintessential to having a highly performant application; however, figuring out the balance is not easy.

With the use of PacketAccelerator, GlobalTURN & SIPTeleport you offload the work of upkeep away from your team and get the peace of mind that your application is running in the most optimized fashion possible.

SIPTeleport comes packed with proprietary algorithms to ensure the least packet drops, reducing the jittery/laggy feeling we often associate with low-quality RTCApplications. It helps in establishing the connections much faster, providing customers with instant access/responses.

PacketAccelerator provides enhancements in layers 3 and 4 of the TCP & UDP packet delivery phase, providing improvements of up to 80% reduced jitter & 99% reduced packet loss.

GlobalTURN, as described previously, provides a managed TURN solution without having to deal with the hassle of maintenance and acquiring personnel with the required networking knowledge. Global TURN essentially allows connecting to a single TURN IP address – simplifying and accelerating your ICE negotiations.

Subspace provides a managed solution for handling pretty much every communication need for a product that relies on real-time communications – be it scaling up in real-time or maintaining low latency while providing excellent security. A solution like Subspace alleviates the stress of managing these services, allowing your team to focus on your business logic!

Go to softwareengineeringdaily.com/subspace to learn more.

The post How Your Network Affects User Interactivity appeared first on Software Engineering Daily.

As our guest today points out, most enterprise software applications are essentially forms for collecting data. The

tag and related components started appearing in HTML fairly early and those same concepts are still in use with modern web browsers. However, the technology for capturing state, validating input, and providing other common services for the management of form data has continued to evolve in many languages and frameworks.

Erik Rasmussen is the author of many popular open-source libraries including Redux Form and React Final Form. In this interview, we discuss the need for those tools to fill in the market and some of the modern approaches for form state management.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Show Notes: * Express, Webpack, SSR, Redux example repo * Redux Form * Final Form

The post React Final Form with Erik Rasmussen appeared first on Software Engineering Daily.

Amazon Web Services changed how software engineers work. Before AWS, it was common for startups to purchase their physical servers. AWS made server resources as feasible as an API request and has gone on to create higher-level abstractions for building applications. In this episode of Software Engineering Daily, we talked to Corey Quinn, Chief Cloud Economist at The Duckbill Group.

Making technical choices can be overwhelming. As software developers, we have to make many choices from what seems like unlimited options. We choose the programming language, libraries, compute, database, schema and many other things. AWS alone offers about 200 services from its data centers across the globe.

When you start with AWS, it seems simple. But as your infrastructure gets huge, visibility becomes a thing that you need to take very seriously.

“Large companies are generally used to the historical data centre model, where you would wind up building things on a capital expense basis. You would plan out your data centre build-outs, and it’s super hard for a single engineer to accidentally order $6 million worth of hardware without getting fired or arrested. The new model though is that someone can inadvertently spin up that level of resource and not only not be aware of it, but no one is aware of that, for in some cases, years at a time. It is not at all transparent what’s happening in your environment.”, says Corey.

One of the other ways in which organizations lose money is by not purchasing reserved instances, because they are convinced they are going to turn off that cluster next week and then months go by and that never gets turned off.

Organizational Layout

AWS is effectively a microservices-driven company, which means that they have very small teams, which they call Two-pizza Teams, each working on individual projects. Looking at the organizational layout of AWS Corey points out, “The feeling I guess is closest to a bunch of internal startups that are competing for funding, for mind share, and they go through iterative rounds until something winds up getting released. At the end of an entire laborious process of iteration going through series of fundings, their “exit” is when someone at AWS gives the service a stupid name and launches it to the public.”

Open Source is not a business model

What does an open-source project have to do in order to succeed as a product company that might be competing with Amazon’s much cheaper, easier-to-sell hosted product?

“I would say give up because an open-source project is not a business model. It’s a means of development. It’s a means of community engagement. It’s a way of solving technical challenges, but there’s an enormous difference between that and having a viable, functional, healthy business.” says Corey.

Primitive AWS Services

If you manage or use AWS systems, you likely need to know at least a little about all of these. Even if you don’t use them, you should know enough to make that choice intelligently.

IAM: Identity and Access Management (IAM) enables you to manage access to AWS services and resources securely.

EC2: Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud.

AMIs: Amazon Machine Images

CLBs and ALBs: Classic Load Balancer is intended for applications that are built within the EC2-Classic network. We recommend Application Load Balancer for Layer 7 traffic and Network Load Balancer for Layer 4 traffic when using Virtual Private Cloud (VPC).

Autoscaling: Auto Scaling monitors your applications and automatically adjusts capacity to maintain steady, predictable performance at the lowest possible cost.

EBS: Elastic Block Store (EBS) is an easy-to-use, high-performance, block-storage service.

Elastic IPs: Elastic IP address is a static IPv4 address designed for dynamic cloud computing.

S3: It provides access to reliable, fast, and inexpensive data storage infrastructure.

Route 53: DNS and domain registration

VPC: Virtual networking, network security, and co-location; you automatically use

CloudFront: content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment.

CloudWatch: provides you with data and actionable insights to monitor your applications, respond to system-wide performance changes, optimize resource utilization, and get a unified view of operational health.

Amazon Linux

One of the choices you need to mitigate while spinning up a server is the right operating system. There are various Linux distros, including Amazon Linux. Should you deploy on Amazon Linux? AWS advocates their own Amazon Linux, which emerged from Red Hat Enterprise Linux (RHEL) and CentOS. It is used by many, is heavily tested, and is better supported in the unlikely event you have deeper concerns with OS and virtualization on EC2. But overall, many companies do just fine using a standard, non-Amazon Linux distribution, such as Ubuntu or CentOS. Using a standard Linux distribution means you have a replicable environment should you use another hosting provider instead of (or in addition to) AWS.

Last Week in AWS

Corey also publishes the Last Week in AWS newsletter. “Posts about AWS come out over sixty times a day. The signal to noise ratio is abysmal. I filter through it all to find the hidden gems and the stuff worth reading, and share it with you – minus the nonsense.” says Corey on his website.

Listen to the full Software Engineering Daily conversation: AWS Analysis with Corey Quinn.

The post Episode Summary: AWS Analysis with Corey Quinn appeared first on Software Engineering Daily.

By most accounts, demand for software engineers exceeds supply. Not just anyone can develop this skill set to the level required to deliver enterprise-grade production code. For those that can, companies are incentivized to take extra measures to ensure software engineers are as productive as possible. The pace of business is often throttled by the pace of software releases.

In today’s episode, we speak with Utsav Shah about developer productivity in the context of the monolith, CI/CD, and best practices for growing teams.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Developer Productivity with Utsav Shah appeared first on Software Engineering Daily.

Spacemesh is an innovative blockchain startup aiming to solve the problem of running smart contracts at scale. A second generation blockchain startup that aims to learn the lessons of past blockchains, Spacemesh aims to make the blockchain better, more usable, more robust, and of course more popular among users. Current blockchains are not scalable both in terms of transaction volume and also the hardware considerations of the blockchains themselves. Bitcoin and other proof of work cryptocurrencies currently consume around two percent of the world’s energy consumption and that figure is only growing. ASICS and CMP graphics cards are effectively e-waste that only serve one purpose and quickly become obsolete once new versions come out. Spacemesh serves to solve this main problem of scalability both from a consensus level and a blockchain architecture level to make Spacemesh the preeminent blockchain for running smart contracts at scale.

At the core of Spacemesh is a new consensus protocol that aims to solve the problems presented by the two incumbent protocols proof-of-work and proof-of-stake.

Proof-of-work in a nutshell requires individuals to secure the blockchain by guessing random numbers and then performing complex calculations that check whether the number is valid or not. Individuals, or miners, race to find a valid number first and once someone does, the consensus resets making all previous calculations invalid and miners race to find a valid number again. This presents two primary problems. The first is that performing these complex calculations takes a considerable amount of energy and as the blockchain grows this energy cost only grows.

The second is like any computer process, specialized hardware like ASICS or GPUs, can be used to perform these calculations faster. Not only does this create ewaste that’s only use is to perform these basically useless calculations, but also centralizes mining power in the hands of those who can afford to invest money in huge server farms which is the antithesis of blockchain’s mission to create decentralized trust. Proof-of-stake tries to solve the energy consumption aspect of proof-of-stake by removing the requirement to perform calculations and instead using large amounts of cryptocurrencies. Individuals, or stakers, lock up large amounts of cryptocurrencies for a certain amount of time in order to provide security to the network. However this fails to solve the centralization problem of proof-of-work and in some aspects only makes it worse as only individuals with a large amount of crypto can contribute to the security of the network.

Spacemesh introduces a new consensus protocol called proof-of-space-and-time that requires individuals to reserve a certain amount of space on hard drives to secure the blockchain. This requires less energy than proof-of-stake and solves the centralization problem of both proof-of-stake and proof-of-works as storage is cheap and there are various mediums such as hard drives and SSD that can provide storage at differing price points. This protocol even has some advantages of other proof-of-stake protocols like Chia as it only requires data to be written and read occasionally, reducing wear on the storage medium unlike Chia which requires frequent reads and writes.

Spacemesh is also experimenting with a blockweave or blockmesh to solve the scalability issue of blockchains. Traditional blockchains can only add one block of transactions at a time limiting scalability. Scalability solutions, most commonly referred to as Layer 2 solutions, are being developed for traditional blockchains like Etherum and Bitcoin but they are oftentimes complex and are still a while away from mass adoption. Spacemesh builds scalability into the blockchain architecture itself by allowing multiple blocks to be worked on and validated at once.

Even the company itself is built with scalability in mind. Traditional blockchains need to raise funds and bootstrap the network. The most common way for them to do so was an initial coin offering or ICOs. In 2017 and 2018, there was an ICO boom that had the unfortunate side effect of attracting scammers that would take the funds and run. Spacemesh has a contract that distributes the funds over time ensuring that the company has an incentive to continue development and support the network. Also they bootstrap the network with their own servers and plan to slowly wean over their servers as the blockchain gains more and more participants.

Spacemesh does not see their cryptocurrency called Smesh as becoming the dominant blockchain in the future. Rather they see a variety of blockchains that serve different purposes. Of course some current project will die but there is space for multiple blockchains to exist. Some will focus on privacy, some will focus on speed, some will focus on ledger size. However many there are though, Spacemesh will be there powering the smart contracts of the future.

This summary is based on an interview with Anton Learner, Core Team Lead at Spacemesh. To listen to the full interview, click here.

The post Episode Summary: Spacemesh: The Proof of Space and Time appeared first on Software Engineering Daily.

The first industrial deployments of machine learning and artificial intelligence solutions were bespoke by definition and often had brittle operating characteristics. Almost no one builds custom databases, web servers, or email clients. Yet technology groups today often consider developing homegrown ML and data solutions in order to solve their unique use cases. Today’s modern data stack is a patchwork of interconnecting tools built to suit a variety of personas that need to interact with the data in notably different ways.

In this episode, I speak with Leigh Marie Braswell, an Investor with Founders Fund. We have a wide-ranging discussion about the technology landscape of data and machine learning solutions and the modern enterprise data stack.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Modern Data Infrastructure and Tools with Leigh Marie Braswell appeared first on Software Engineering Daily.

NFTs are the current blockchain craze. Graphics designer “Beeple”, or Mike Winkelmen, sold his artwork as a NFT for $69 million in March. At the same time. Jack Dorsey, CEO of Twitter, sold a NFT of his first tweet for $2.9 million dollars . Though the hype has died down since then, it has driven development and adoption of NFTs and the ecosystem surrounding NFTs continues to thrive and grow. At the center of this ecosystem is Showtime, the Instagram for NFTs.

To give some background NFTs, or non-fungible tokens, is a unique token that can be used to represent an asset on a blockchain. This token is non-reproducible and non-fungible, i.e individual tokens and not interchangeable. NFTs have a wide use case from art and paintings to in-game items. Even physical assets like property can be represented with a NFT. NFTs have the benefits of being on a blockchain, meaning it is easy to trade and sell it, as well as having an immutable record of who owned it and it’s transaction history. Similar to a painting’s provenance, this record can be used to determine ownership of a NFT and whether or not it is authentic. NFts can even encode contracts such that every time a NFT is sold the original author gets a small fee or cut.

NFTs first took of in 2017 with Cryptokitties, a blockchain based collectable, where individuals could purchase, collect and sell virtual cats as NFTs. There were previous attempts at using NFTs like Cryptopunks in 2016 but Cryptokitties was the first to take off and gain mainstream adoption.

After Crypto Kitties, the hype died down for a bit and important infrastructure was built surrounding NFTs that was not as consumer facing. ERC-721 was released in 2018 which formalized the technical definition of a NFT so that all NFTs followed the same standard. OpenSea was created in 2017 which was a search engine that allowed individuals to trade and search differing NFTs. In July of 2020, trading volume of NFTs exploded and though it has fallen from its peak, it is still growing.

Showtime is a social network built on top of NFTs. While you can’t trade or create NFTs right now, you can display all your NFTs in one place and have followers view your NFTs. Built with a React front end, Django backend, and leveraging OpenSea API for blockchain data, Show time serves as an Instagram for NFTs. A centralized location where collectors can display all their art and gain followers. After all, what use is buying art if people can’t see it. Currently only on web but looking to expand to mobile, Showtime isn’t necessarily a crypto application, rather it’s a web app focused on crypto. However it’s an important part of the NFT ecosystem as it raises awareness and increases mainstream adoption.

Looking to the future, NFTs still have a ways to go and innovation is still ongoing. Companies like NIFTEX, NFTfi, NFX are providing loans against your NFT. Airbns created money out of idle homes, why not create money out of idle NFTs. Eventually any kind of video, photo or meme on social media will be a NFT. This is especially important with virtual reality and as the world transitions online. A virtual gallery for instance could be created with NFTs and you could charge people admission to view the gallery. Creator coins like those on BitClout or Rally enable individuals to quite literally sell themselves and their unique brand giving rise to a new influence economy and enable individuals to raise funds. All of these things are possible because of NFTs and the blockchain. The current NFT explosion is probably a fad due to the increased price of digital art, but the use cases for NFTs are so numerous that as long as blockchain is there, NFTs will be right alongside it.

This summary is based on an interview with Alex Masmej, co-founder and CEO at Showtime. To listen to the full interview, click here

The post Episode Summary: Showtime: Crypto Art and NFTs appeared first on Software Engineering Daily.

Abstract

Software Daily is a place to create software.

Introduction

SoftwareDaily.com is a social network that allows people from all over the world to come together and create software.

Inspiration from Amazon

Working at Amazon taught me that we can build anything.

Inspiration from Facebook

Writing “Move Fast” taught me that social networking will allow us to expand our species and celebrate an increased breadth of individuality while also deepening our shared values and experiences.

Faceazon

Faceazon is a social network like Facebook combined with a universal ecommerce experience like Amazon.

Notes on Podcasting

Podcasting has taught me empathy, both technical and emotional.

Software Engineering Daily

SE Daily has proven my credibility as the most knowledgeable software engineer in the world. Anyone who disagrees is welcome to bet me $1M+ that they can out-earn me over the next decade.

Notes on Digital Advertising

Digital advertising in its current form allows for blatant, repeated theft by adtech companies including many with $1B+ market cap.

Adforprize

Adforprize is our video social network and creative advertising platform that will integrate seamlessly with YouTube, TikTok, Instagram, Stripe, and OnlyFans to provide you with a full service user generated and industrially maintained advertising ecosystem.

Podsheets

Podsheets is a combination open source podcasting and video publishing and consumption platform.

Notes on Hosting Businesses

Hosting businesses with proprietary UIs or APIs have the best, most defensible margins (even better than payments). Heroku remains one of the most underestimated portfolio sleepers in business, and explains Salesforce’s eventual potential to rival even Amazon as the most rapidly growing behemoth in the world.

Notes on Collaboration Businesses

Super hard to get enough people using a collaboration business, and once they do they are still willing to switch because collaboration is just a little productivity thing that can usually be swapped out (yes, even you Slack will one day be swapped out).

FindCollabs

I should never have shut down FindCollabs and if you want to help me reboot it please email me.

Software Daily

Check out SoftwareDaily.com to see all of my plans, including an Uber-like services company and a new mobile operating system.

Supercompute

Check out supercompute.games to learn the software business while earning botcoin!

Notes on Gaming Businesses

Gaming businesses are high margin, high risk, and super fun to build.

Notes on Payments Businesses

Payments businesses are as sticky as surgical mesh and Stripe is as admirable as AWS and Oracle–which is to say I am deeply envious of Stripe and wish I was a good enough human being and effective altruist to change into a Collison.

Rectangle

Rectangle is an open source, open operations payments systems business with a minimum goal of offering open source alternatives to Stripe.

Notes on Censorship

Companies are forced to censor their content by an outdated, overly religious and warmongering government that more closely resembles the Deep State than The West Wing.

Sqripe

Sqripe is an uncensorable internet built on a data structure I invented called a cloud blockchain.

Notes on Virtue Capital

Venture capitalists are mostly cowards with no real opinions about anything other than which vest to wear given their daily set of potential microclimates.

Investment Discussions

Investment Discussions is a Slack group where I give away investment liquidity in exchange for knowledge from entrepreneurs and investors.

Notes on Portfolio Synergies

There are portfolio synergies across every portfolio. a16z, for example gets portfolio synergy from their wide variety of data transfer, ETL, data management, and data engineering businesses, who can share notes and form loose cabals of success. Sequoia Capital gets portfolio synergy by owning the investment communities of both China and the United States. Facebook would get portfolio synergy if it owned both Twitter and Google, hypothetically. See, this is the kind of paranoia I have, which is why I’m planning to run for president.

Notes on Healthcare

Our healthcare system is hurting everyone in the United States, and I have no idea how to fix it. It makes me very sad. I think that instead of focusing on healthcare, we should focus on homelessness, which has more measurable outcomes.Auditable Hospitals

We need a hospital system where every single patient has access to the actions of every single care provider, allowing patients to uncover conspiracies and antitrustworthy market actions.Notes on Fear

You should never let fear stop you from doing what you know you are capable of.

The post Software Daily Prospectus appeared first on Software Engineering Daily.

Move Fast: How Facebook Builds Software

Phishing attacks, malware, and ransomware are just some of the major threats everyone connected to the internet faces. For companies, the stakes are especially high. Setting up a secure infrastructure is difficult. Your adversary only needs to find one flaw to get in.

Vancord is a private cybersecurity company, based in Connecticut, that was founded and built by security engineers to specialize in incident resilience and response. In this episode, I interview Jason Pufahl and Russell Jancewicz from Vancord.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Cybersecurity Threats with Jason Pufahl and Russell Jancewicz appeared first on Software Engineering Daily.

In a version control system, a Monorepo is a version control management strategy in which all your code is contained in one potentially large but complete repository. The monorepo is in stark contrast to an alternative approach in which software teams independently manage microservices or deliver software as libraries to be imported in other projects. The monorepo strategy has been followed by noteworthy companies such as Google, Facebook, and Microsoft.

Derrick Stolee is a software engineer working at GitHub. He joins us today to talk about strategies for monorepos and innovations to the way git works in order to better support this style of the repository.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Git Scales for Monorepos with Derrick Stolee appeared first on Software Engineering Daily.

Github is the largest source code host on the internet, reporting over 40 million users and more than 190 million repositories according to Wikipedia.

The home to hobbyist projects, large-scale open source applications, and confidential private company code, Github has become the defacto SaaS provider for source code management and version control. Every developer has probably used it at one point in their career. However any hard reliance on a third party, even one as big as Github and backed by Microsoft, presents glaring security and operational risk. Github is not a backup service and companies that rely on the hosted product should have a backup solution in place.

Github, like most SaaS platforms, follows the Shared Responsibility Model in which responsibilities are divided between the platform and the user.

Just like on-prem solutions, a Saas requires the user to retain responsibilities for the information and data stored on the platform. In fact, Github’s terms of service explicitly state that they are “not liable to you or any third party for any loss of profits, use, goodwill, or data.” Github is a hosted source control service. You need to bring your own security policy, access policy, and plan for how your team should leverage this tool.

Github provides more than just storage and versioning of the actual code. You can also use it to store and track essential metadata like bug tracking and pull requests (PRs).

Due to its ubiquity, many organizations take for granted that Github will be reliable and trustworthy. Many professionals underestimate the level of risk they can expose themselves to without a backup for Github and the critical operational data stored in it.

The most common cause of data loss is human error or rogue employees. Applications can fail for a variety of reasons but the most common cause of unplanned downtime is human error, both malicious and unintentional. Employees who have recently been let go or fired may act vindictively towards their former employers. In 2020, a former Cisco employee maliciously began deleting systems after they were fired and this led to a $2.4 million dollar loss for the company. In 2021, a fired credit union employee wiped 21 GB of files in retaliation for being fired. Data loss whether it be code or just general data is a big concern for organizations and can lead to huge real-world losses. Data loss can even occur if the individual is not malicious. Force pushes for example can cause you to lose massive amounts of data.

Even if a company is not worried about human targeting, malware is still a concern. Malware is growing more and more complex every day and these days there is malware specifically targeted at gaining access to Github or other source control repositories. These can be fairly complex pieces of malware that try to inject code in build files or something as basic as a phishing attack to gain access to Git credentials. In 2018, a piece of malware called Octopus Scanner began appearing in Github repositories. The malware infected a computer, looked for projects that used Git and Github, and then uploaded infected build and source files to Github. This malware was particularly difficult to manage because it infected real projects that could not be blanket banned or blacklisted. In 2020, a widespread phishing attack targeting Github users, nicknamed Sawfish was discovered.

This phishing attack mimicked the Github login page and relayed both passwords and TOTP two-factor authentication codes. These can lead to compromised accounts that can wreak havoc by deleting code and creating issues. If you need an example of what type of damage a compromised Github account can cause, look to the 2019 incident where Canonical lost control of a Github account. Even ransomware can be repurposed or modified to target Github. In May 2019, it was reported that hackers were holding source code hostage and if developers did not post the ransom they would release the code to the public. All of these prove that malicious software can and oftentimes does target the data stored on Github.

Finally, service downtime is a real concern for organizations that rely quite heavily on Github. Github, like any cloud or SaaS offering, is not immune to downtime or unexpected outages.

To many organizations, Github has become a key part of their developmental workflow and any interruptions to the site result in delays that can cost thousands or even millions of dollars. In June 2020, Github went down for two hours costing thousands of developers thousands of man-hours. Some developers were completely locked out of their workflow because of how closely Github was integrated into their daily tasks.

All of these problems and more can be solved with a Github backup solution. A backup strategy is essential for all types of data, so why not backup arguably the most important part of your application. While building a custom backup solution is often possible it negates much of the benefits of using a SaaS source and often requires a large upfront cost in development hours but also has associated ongoing labor and maintenance expenses. Going with a third-party solution like BackHub by Rewind, organizations can conduct daily backups and restore their data in a few clicks saving developmental time. In addition, it is compliant with SOC2 and similar standards.

Regardless of if data is stored on-premise or on the cloud, it is vulnerable. Data can be lost in accidental deletions, malware attacks, or any number of security threats. Even Github, a large cloud-based service, is not excluded from these threats. Some would even argue due to its size and prominence it is more vulnerable. Securing data in the cloud is a shared responsibility between the developer and the platform and it has become clear what the roles are.

An effective repository backup and recovery solution is the first step in protecting code in the cloud. Check out Rewind with a 14-day free trial here.

The post Three Reasons Why You Should Have GitHub Backups appeared first on Software Engineering Daily.

There are wrong ways to fake your data. Whether you’re bootstrapping a dev environment, automating integration tests, or capacity testing in staging, we all need high-quality fake data. Regardless of your use-case, common data generation pitfalls can break your testing or, worse, leak sensitive data into unsecured environments.

There are ways to generate synthetic test data that achieve the realism required for effective testing along with the security needed to protect your company and customers. The secret lies in identifying potential anti-patterns and then preventing them from ever forming in the first place.

In the context of test data generation, we see anti-patterns emerging in one of three key ways.

The process fails to mimic the complexity and requirements of real-world situations.
The process fails to protect the privacy of the individuals behind the numbers.
The process fails to work effectively for all data types and sources in the data ecosystem being mimicked.

The outcome of these failures? Broken data, worthless tests, bugs in production, and in the worst cases, a data security crisis.

A strong data generation infrastructure should have built-in tools to enable its users to generate the patterns they need as opposed to random data. Here, we’ll explore the bad (anti-patterns) to understand what it takes to enable the good (patterns).

1. A series of impossible events Solution: Defined time series rules From healthcare records to financial transactions, to student progress reports, data across industries and platforms is rich with event pipelines. Events can trigger actions in your product and reveal the success of your user journey. They’re a fundamental part of the user experience and the data that experience creates. For accurate testing, they need to be realistic.

Event pipelines generated at random inevitably create impossible time series. A quality solution allows you to define the relationships between events in your data by linking related fields and dictating the order in which they occur. For the highest degree of accuracy, an event generator should also be designed to mirror the distribution of the dates in your original dataset. It’s a combination of complex algorithms on the back-end and customizable rules on the front-end.

2. Random categorical shuffling Solution: Shuffling with defined ratios A frequently used way to obfuscate real data is by shuffling categorical data, for example, the job titles of employees within an organization. The risk in shuffling this data is that it can wipe out the integrity of the data if the ratios, and their relationships to other fields within your dataset, aren’t preserved. For example, imagine you’re generating a synthetic workforce. Random generation might come up with 20 assistants for a single manager or 20 managers with a single assistant.

The ratios and relationships between categories make all the difference in whether the data you generate will be able to simulate real-world situations. A well-designed algorithm for categorical shuffling must take this into account to generate distributions of categorical data that mirrors the reality in your original data.

3. Unmapped relationships Solution: Column linking The vast majority of data involves logical relationships that any human would immediately recognize, but random generators will not draw these relationships unless they are given rules to do so. When the underlying data does not reflect real-world relationships, your testing cannot reflect real-world usage. Not linking columns with a defined relationship during generation can lead to the formation of anti-patterns where you least expect them, both in your testing and in your product.

The tool for avoiding this hazard is the capability of linking as many columns as you need to ensure that dependencies are captured and the stories in your data ring true. So, for example, in a table of payroll data, bonuses become a function of salaries which are tied to job titles partitioned by office location.

4. Inconsistent transformations Solution: Input-to-output consistency Even when anonymizing data, it’s often important to anonymize certain values in the same way throughout your dataset. Performing inconsistent data transformations can easily break your data to the point that it’s no longer usable. De-identifying data consistently is the pattern you need. It means that the same input will always map to the same output, throughout your database, allowing you to preserve the cardinality of a column, match duplicate data across databases, or fully anonymize a field and still use it in a join.

Perhaps you have a user database that contains a username in both a column and a JSON blob as well as another database that contains their website activity. Consistency enables you to safely anonymize the username, but still, have that identifier be the same in all locations.

5. Sensitive data leakage Solution: Identify and flag PII/PHI When you’re dealing with personally identifiable information (PII) or protected health information (PHI), your company has a legal obligation to maintain data privacy. The first step in de-identifying PII is identifying columns containing sensitive information and flagging them as needing protection throughout your database. An algorithm can do this quickly and at scale, but it must be carefully built. Imagine a column of birthdates named student_BD instead of birthdate or DOB. A de-identification system that only relies on column names to find PII may not flag that column as sensitive, and a data privacy anti-pattern is born.

An effective de-identification system uses machine learning to examine both column names and the data within those columns to determine what may or may not be PII. And once the PII is identified, it must be flagged by the system in a way that ensures it will be protected without slipping through.

6. Unaccounted-for schema changes Solution: Flagging schema changes and refreshing test data on demand Schema changes are the only constant in modern data ecosystems. Failing to account for these changes, even seemingly minor ones, can lead to failures in your automated testing and, equally as important, risky data leaks. In the best-case scenario, your test data may simply no longer work. In the worst, you’ve now got sensitive data in your lower environments.

The pattern you need here is a tool built into your data generation pipeline that alerts you to any schema changes as they come through. Better yet, it should require you to update your generation model before pulling new data into staging. An ideal system will also allow you to refresh your data on demand, multiple times a day, so your data truly represents a mirror production at all times, schema included.

7. Outliers revealing TMI Solution: Adding noise with differential privacy All data has its outliers. The more precise your data anonymization methods are, the more likely they are to pull those outliers through—outliers that could be used to re-identify individuals if the anonymized data is combined with other available resources. When outliers aren’t taken into consideration, they serve as bold clues to revealing what synthetic data is designed to protect.

The solution here is differential privacy, which adds noise to the data to create a more tempered pattern that obscures outliers. Differential privacy is a property that can be applied to data generation algorithms to guarantee a higher level of privacy in your output data. The more algorithms within your data generation process that can be made differentially private, the safer your outliers will be.

8. Insufficient integration Solution: Cross-functional APIs and seamless integration into CI/CD pipelines Given the nature of tech stacks today, integration should be a key feature of any system you put in place. Whether a data automation tool has an API shouldn’t even be a question you have to ask.

When it comes to building a data generation tool in-house, the process almost always involves writing scripts—scripts that are consistently prone to failure. Building a solution in-house isn’t just a matter of the initial lift; it also requires continuous maintenance to keep the system up and running.

Your data de-identification infrastructure should enable developers to move faster, not weigh them down with double work. A data mimicking tool that has an API, can connect to any data source, integrates seamlessly into your existing systems, and works with your data no matter how it changes over time equips your developers to do their best work. As your data needs evolve, so should the systems that support them.

9. Vendor lock-in Solution: Support for all data sources Relational databases still dominate the world of big data, but NoSQL databases like MongoDB and Cassandra are gaining fast. Even PostgreSQL can easily work with NoSQL code and store JSON files. The future is hybrid. Your data de-identification infrastructure needs to support that.

When it comes to building this infrastructure in-house, you may find yourself dedicating significant resources to creating a process that works with PostgreSQL, only to end up back at square one when your company adds Redshift to the stack. And if you’re using Mongo, you’re going to require an entirely different approach. What’s more, your data may live in separate database types, but that doesn’t mean it isn’t interrelated. Not only will your solution have to work for multiple databases, but it also has to work between them as well.

Given today’s ever-expanding data ecosystems, it simply doesn’t make sense to build a system that only works with one database type. Your data generation solution should work as seamlessly with Postgres as it does with Redshift, Databricks, DB2, and MongoDB. Anything less is just a roadblock to data management. Seek out a tool that will work with your data wherever you keep it, now or in the future.

Top Takeaways Building a high-quality data mimicking and de-identification solution that satisfies all of the above is a major investment of resources, and with both data privacy and data utility on the line, the stakes are incredibly high. The ultimate anti-pattern may very well be burdening your team with all of these requirements in-house or settling for a solution that fails to deliver in any of these areas. The ultimate pattern? Use the above nine sections as a guide to building your own checklist, then seek out a proven platform that is ready to equip your team with all your fake data needs.

The post 9 Fake Data Anti-patterns and How to Avoid Them appeared first on Software Engineering Daily.

Companies that gather data about their users have an ethical obligation and legal responsibility to protect the personally identifiable information in their dataset. Ideally, developers working on a software application wouldn’t need access to production data. Yet without high-quality example data, many technology groups stumble on avoidable problems. Organizations need a solution to protect privacy while simultaneously preserving aspects of the data which are important.

Tonic is automating data synthesis to advance data privacy. Their solution gives your production-like data for development and analytical purposes without compromising on data quality or privacy. In this episode, I interview Tonic’s CEO Ian Coe, and Head of Engineering Adam Kamor.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Faking Data Using Tonic.ai with Ian Coe and Adam Kamor appeared first on Software Engineering Daily.

Applications write data to persistent storage like a database. The most popular database query language is SQL which has many similar dialects. SQL is expressive and powerful for describing what data you want. What you do with that data requires a solution in the form of a data pipeline. Ideally, these analytical workflows can follow similar best practices to those handled in application code.

DBT is a transformation workflow that lets teams deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation. Users who know SQL can build production-grade data pipelines. In this episode, I interview Tristan Handy, CEO, and founder of DBT Labs.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post DBT: Data Build Tool with Tristan Handy appeared first on Software Engineering Daily.

The way we write, compile, and run software has continued to evolve since computer programming began. The cloud, serverless, no-code, and CI/CD are all contemporary ideas introduced to help software engineers spend more time on their application and less time on the chores of running it.

Darklang is a new way of building serverless backends. It’s a cloud native programming language with several novel features including being not just serverless but deployless as well. In this episode, I interview Paul Biggar, founder and CEO of Dark.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Darklang Deployless Applications with Paul Biggar appeared first on Software Engineering Daily.

Tedious, repetitive tasks are better handled by machines. Unless these tasks truly require human intelligence, repetitive tasks are often good candidates for automation. Implementing process automation can be challenging and technical. Increasingly, engineers are seeking out tools and platforms to facilitate faster, more reliable automation.

In this episode I talk to Yaseer Sheriff, Co-Founder and CEO at Axiom about no-code solutions, process automation, and some of the challenges in developing the software powering those services.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post No Code Process Automation at Axiom with Yaseer Sheriff appeared first on Software Engineering Daily.

Financial technology or fintech has always been a hot topic. This is increasingly true in recent years as disruptive companies enter the market to give better alternatives and solutions to consumers.

Current is focused on creating better financial outcomes. In addition to providing banking services, their app has many tools and reminders to help users learn and execute better money management strategies. In this episode, I interview Trevor Marshall, CTO of Current about starting a fintech company, how banking software is written, and what opportunities exist for disruption.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Reimagining Banking at Current with Trevor Marshall appeared first on Software Engineering Daily.

As Bitcoin and other cryptocurrencies gained value in 2010, exchanges popped up so that people could extract value from these blockchain based currencies. However, the failure of Mt. Gox in 2014 and several other prominent exchanges illustrated the fallacies and weakness of centralized exchanges.

The fact that large corporations had custody of individuals’ wallets and therefore their bitcoin presented not only large security concerns but also was antithetical to the decentralized, trustless philosophy that blockchains were trying to promote. The failure of centralized exchanges combined with the proliferation of ERC-20 standard and the subsequent tokens cemented the need for a blockchain-based exchange that gave users the ability to exchange various tokens.

Decentralized exchanges, or DEXs, solved this need. Early DEX’s were crude smart contracts that operated on a simple order book principle, where buyers and sellers manually submitted their orders and a smart contract kept track of them.

This had two main problems. The first was that it was extremely costly in terms of gas. You had to pay gas in order to submit a trade that may never get filled and pay again when the trade was executed. This was eventually solved by moving order creation off-chain. The second problem was the reliance on market-makers to add liquidity to a market. This problem was eventually solved through the use of automated market makers, or AMM. However both of these improvements did not solve the fundamental issue that the more volume you wanted to trade, the worse price you would get. This price disparity, or slippage, was becoming more and more of a problem as the ecosystem grew and began to attract a larger amount of capital. 1inch and other DEX aggregators solve this problem.

1inch was created at Hackathon in 2019 as a simple information aggregator. It wanted to aggregate DEX prices by indexing available DEX to provide a price comparison to users looking to trade a specific set of tokens. This was enabled by the permissionless nature of blockchain. Unlike the traditional web where sites like Facebook and Twitter can restrict indexers from Google and other aggregators, data on the blockchain is viewable by everyone. The UI or front end of the DEXs could be abstracted away but the brains of the operation, the smart contract, can be viewed by everyone including the initial 1inch developers. In that same hackathon, 1inch added the capability to split order volume between multiple DEX’s. By splitting one large trade on a single exchange to multiple smaller transactions across multiple exchanges, 1inch allowed traders to make the trade off between lower slippage and increased total gas price for multiple transactions. For really large trades, this tradeoff was worth it.

Today 1inch is the leading DEX aggregator supporting over 49 different DEXs on Ethereum and 12 different DEX on Binance. At its core it is still an informational service. It is not a broker, or middleware. However it has grown beyond just comparing pricing on different exchanges, rather it provides information on how to compose your swap to achieve the best possible rate. This could be as simple as choosing the DEX with the lowest rate or as complex as transacting with multiple exchanges across multiple currencies. With a front end hosted on IPFS and a governance token, 1inch is fully decentralized erasing the worries associated with centralized exchanges. It adds new DEXs in a few hours or at worst a few days. It provides better gas efficiency than even the Uniswap router. Tokens and DEXs will rise and fall but 1inch will be here to stay giving users the best price for their trade.

This article is based on an interview with Anton Bukov about 1inch. Check out that interview and others at softwareengineeringdaily.com

The post Episode Summary: 1inch: The Super Liquider DEX Aggregator appeared first on Software Engineering Daily.

The dream of machines with artificial general intelligence is entirely plausible in the future, yet well beyond the reach of today’s cutting edge technology. However, a virtual agent need not win in Alan Turing’s Imitation Game to be useful. Modern technology can deliver on some of the promises of narrow intelligence for accomplishing specific tasks.

PeopleReign has created a virtual agent for IT and HR employee service. This agent’s goal is not to replace a human agent but to augment them by handling some requests and elegantly handing off to a human in other cases. In this episode, I speak with Dan Turchin, CEO of PeopleReign about their virtual agent and the future of work.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Virtual Agents for IT and HR with Dan Turchin appeared first on Software Engineering Daily.

Kubernetes has dominated the cloud application development landscape. First developed in 2015, Kubernetes won the container orchestration wars against rivals such as Mesosphere and Docker Swarm to become the de-facto standard for container orchestration. However as cloud computing has become the de facto standard for container orchestration, improvements in debugging and fixing issues in native Kubernetes applications are lacking. When developers encounter similar issues with known solutions across multiple pods, they have two primary solutions. One, go pod-by-pod and find or fix the problem, similar to how developers of lore used to SSH into individual machines or two, try to automate that process by investing significant time and manual labor. Shoreline provides a real-time debugging and automated repair tool that enables developers to efficiently find and debug errors across thousands of pods and develop automation so that those errors can be fixed automatically in the future.

As Kubernetes has gained popularity, more and more tooling has been developed to create and manage production-level Kubernetes clusters and extend their functionality. However, these tools often focus on preventive measures for a specific area of expertise. Deployment tools like Helm serve to create an IaaC, or Infrastructure as Code, type environment where creating and managing application commits on Kubernetes is repeatable and debuggable. Visualization tools like Lens and K9s serve to increase the observability of your cluster.

However, none of these focus on incident management and resolutions. Shoreline focuses on incident management and resolution, or “Day 2” operations by solving two problems.

The first problem Shoreline solves is creating an intuitive and powerful Kubernetes native environment that enables developers to monitor and maintain specific pods in an aggregated manner to diagnose and solve issues.

The second problem it tries to solve is to enable developers to easily create automation such that if the issues reappear or require ongoing maintenance, it can be effectively automated.

Shoreline had to solve several hard technical challenges. Notably, there are five main problems that give Shoreline its uniqueness.

Data has to be trusted. Existing tools like Splunk collect data through batch processing which results in time delays and duplicate logs. By collecting data in real-time Shoreline solves these problems and provides more value. Shoreline leverages the Prometheus exporter ecosystem that is both open-source and best-in-class in combination with custom integration code in order to get accurate, per-second metrics with no lag.
The second problem that Shoreline solves is creating an easy way to define queries and automate repairs. This is done primarily through the Shoreline Op language which will be touched upon later.
It needs to be fast. With large Kubernetes clusters having thousands of nodes and even more pods, Shoreline has been designed to take automation and turn them into a distributed execution graph that runs in parallel.
It needs to work even when other things are broken. Shoreline has been designed from the bottom up to be fault-tolerant and even work locally when you lose connectivity.
It needs to be safe, Shoreline provides controls to limit the scope of both manual and automated commands.

Shoreline has created a DSL, or domain-specific language called Op. Op attempts to fluently integrate three primitives: resources, Linux commands, and metrics.

Resources are simply the pods you want to act upon. They can be defined by namespaces, regex commands, or Kubernetes tags. Linux commands are Linux commands like a top, Kubectl, Netstat, or any other command you might find useful in debugging a pod. Metrics take the output of many different outputs of pods and combine them into a single number. This can be as simple as averaging together numeric outputs or something more exotic. A fairly simple example that many enterprises may find useful is selecting all application pods by tag, running top on them, stripping the output, and monitoring to see if the NodeJs process is taking more than 90% CPU utilization. Though this is a simple example, it gives you an idea of how Ops can be used. By using familiar Linux commands, the Op DSL is intuitive yet powerful. Once you have constructed a query, you can then trigger a “bot” when that query condition is met. This “bot: then executes an action also defined in Op to fix the issues. All of this together creates a simple and powerful remediation loop that checks for issues, collects diagnoses, and automatically applies repairs. This enables Ops teams to meaningfully solve repetitive incidents once an issue is fixed, it can be fixed forever.

Shoreline is continuing to grow and develop features. Their goal is to radically increase system availability and reduce operations toil through incident automation. They currently support AWS with Azure and GCP support to come. They have a large library of “Op Packs” that solve commonplace issues seen by Kubernetes operators.

All things break. The question is how do you get back to normal as quickly as possible. For those looking for an integrated solution that allows you to quickly restore critical functions at scale, look no further than Shoreline. If you’re an up-and-coming SaaS company that has ticket pain and that ticket pain is growing and growing fast, take a look at Shoreline at shoreline.io. Right now Shoreline is in the early GA stage, so they can become a partner with your company, taking in feedback, and together you can design a product that reduces your ops pain.

This sponsored article is based on an interview with Anurag Gupta, Founder and CEO of Shoreline. If you are interested in the full interview, you can find it here.

The post Shoreline: Pod Debugging and Management Automation appeared first on Software Engineering Daily.

A developer’s core deliverables are individual commits and the pull requests they aggregate into. While the number of lines of code written alone may not be very informative, in total, the code and metadata about the code found in tracking systems present a rich dataset with great promise for analysis and productivity optimization insights.

LinearB is a systematic approach to engineering improvement. Their WorkerB Slack Bot connects with teams on an individual level to help with productivity and collaboration. In this episode, I speak with Dan Lines, Co-Founder, and COO.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Show Notes: * LinearB.io * Learn more about WorkerB * Check out the Dev Interrupted Podcast * Save your spot for the INTERACT conference

The post LinearB with Dan Lines appeared first on Software Engineering Daily.

“All jobs are now becoming data jobs,” notes Srini Kadamati, senior data scientist and developer advocate at Preset. The unofficial motto of the company, he explains, is that every member of an organization should interact with data, creating charts and dashboards, and asking and answering their own data-driven questions.

Sales managers, for example, can interrogate dashboards to discover where they should invest marketing dollars, whether that’s paid advertising, events, and webinars or writing blog posts. Executives from all teams, in all business units, can use dashboards in meetings, in OKR reviews, or as indicators of progress towards goals.

But traditional business insights tools, such as Tableau, are often priced at a level that makes company-wide use prohibitively expensive. This is where Preset comes in. The company will offer a business insights platform with pricing at a level similar to the viewer-only tiers from other BI platforms, but with every user capable of having the same extensive permissions.

Preset offers a cloud-hosted version of Apache Superset, an open-source application for data exploration and visualization. Maxime Beauchemin started working on Superset at a hackathon in 2017. At the time Beauchemin was working for Airbnb and was frustrated to find that business intelligence platforms weren’t compatible with the modern data stack. His team wanted to play around with big data tools like Druid, Presto, and Trino, for example, and found that wasn’t possible with Tableau.

Beauchemin saw how web scale technologies could cheaply and efficiently deliver value to millions of users — but in the business intelligence space, companies were still selling based on a cost-per-core basis. Superset uses React and TypeScript in the frontend and Python in the backend, and the company has embraced type hinting, a new Python feature to help with scalable code, and SQLAlchemy, an open-source SQL toolkit for the language.

“Open-source ultimately gives us and gives end users that want to use cutting edge features just a lot more speed,” Kadamati explains. “By betting on these open standards, you know that as your data stack evolves and changes, your BI layer will also kind of just keep up, that you’re not waiting for a vendor to hopefully support and update their functionality one day.”

Preset is currently in closed beta testing, with plans for a general release within the next few months. The platform will be entering a competitive and crowded business intelligence market: in 2019, Salesforce acquired Tableau for just under $15.7 billion. At such a cost, Tableau might be expected to make up a hefty share of the business intelligence market, but in reality, it currently sits at around 11%.

After the release, Kadamati is particularly excited about the potential to add support for notebooks into the platform. Superset currently has a no-code chart builder as well as a SQL Lab, but the added functionality would mean users could also do processing, analysis, and visualization in Python or Julia. “I think adding notebooks will really kind of complete the platform,” Kadamati says, “And I think will make it super-compelling.”

This summary is based on an interview with Srini Kadamati, senior data scientist and developer advocate at Preset. To listen to the full interview, click here.

The post Episode Summary: Preset: Visualizing Big Data with Srini Kadamati appeared first on Software Engineering Daily.

Infamous for their role in the 2008 financial crisis, synthetic assets are a wrongfully maligned financial instrument that actually provide real world value. Simply put a synthetic asset is simply a combination of assets, usually some mix of options, futures and swaps, that attempt to track the value of another asset, typically not available to trade through traditional means. A fairly common synthetic asset is an inverse ETF, an asset that attempts to have an inverse relationship with another ETF. You can’t really buy negative shares, so instead you buy an asset that has attempts to mimic the inverse performance of an ETF.

This type of synthetic asset allows people to create more value from a specific asset’s performance than is traditionally available. This is especially valuable in the DeFi space as the availability of synthetic assets on Ethereum enables individuals to create complex financial instruments and contracts that depend on assets that may not be traditionally available on the blockchain. A fairly simple example is you can have a synthetic Litecoin on the Ethereum blockchain that you can trade for and interact with on Ethereum instead of having to go to an exchange, deposit Etherum, create a Litecoin wallet, and make the trade. The development of synthetic assets on Ethereum cements Ethereum as the DeFi blockchain of choice and enables the development of a more mature financial market that more closely resembles traditional markets.

Built from the backbone of Havven, a decentralized stablecoin, Synthetix gives people the ability to trade assets traditionally on the Ethereum blockchain. This can be everything from other cryptocurrencies such as Bitcoin and Dash or something not even available on any blockchain like USD or gold. There are two main users to Synthetix, stakers and traders. Traders are any type of physical user or smart contract that wants to use the exchange functionality of Synthetix. They need to trade Ethereum for some sort of synthetic assets like BTC and gold. They approach a trading contract, or more likely some sort of exchange aggregator like 1inch and exchange Ethereum for their synthetic assets like sUSD or sBTC. In exchange for providing this functionality Synthetix takes a fee. If it’s an actual individual, then the transaction is done. If it’s a smart contract, they can then leverage the exchange as underlying functionality in an app or a more complex instrument.

On the other hand we have the stakers. Stakers approach the Staking contract and stake SNX, Synthetix native token, in exchange sUSD or some other synthetic asset. In exchange for staking they get a proportion of the exchange fees and other liquidity rewards. Through decentralized oracles, Synthetix keeps track of the price of the actual assets and if the price rises or falls, it releases or demands more SNX from individuals who staked, i.e the debt pool, proportion to how much of that synthetic asset there is to begin with. In other words, all the stakers collectively are responsible for providing more or less SNX if the price of one of the assets that backs a Synthetic asset rises or falls. This is better than individually backing assets as it distributes risk. However since the debt pool is shared, creating new synths is not a trivial process. There are a number of methodologies for new synthetic assets but all of them are community driven and must go through rigorous evaluation processes.

Synthetix is constructed on the Ethereum blockchain and has various differing ERC20 tokens and smart contracts. At a high level the base components of Synthetix is the base contract, ERC20 Synths and the various contracts surrounding them, arbitrage contracts and Uniswap, the inflationary pool, and the fee pool.

High Level Architecture Diagram: Synthetic Docs Part of the reason why the logic and state management is divided into so many contracts is to manage smart contract size. Ethereum limits smart contract size to 24 KB, yet each external contract call costs gas. So implementing complex behavior like managing state and logic on the blockchain is a balancing act, weighing gas costs and complexity against the smart contract size. Another consideration when building Synthetix was how to integrate into the extensive developer tooling that has fairly recently become mainstream in the Ethereum ecosystem. Though lacking when compared to traditional web development or software engineering, Ethereum development has the best developer ecosystem and community of the various blockchains. Companies like Hardhart are always working to increase developer productivity and provide a development experience like traditional software development.

Synthetix is rapidly becoming a staple in the DeFi community. Exchanges and liquidity pools like Curve are using it to reduce slippage when then exchange between multiple assets. Crypto hedge funds like dHedge are using it to reduce risk through exposure to inverse Eth and various other Syths and of course individuals are using it to gain exposure to assets not natively available on the blockchain. The company surrounding Synthetix, or more accurately the DAO surrounding Synthetix, is committed to building the protocol even further and integrating with future features of the Ethereum blockchain, like layer two scaling solution.

To listen to the full interview on Synthetix with Justin Moses, click here.

The post Episode Summary: Synthetix: The Derivative Liquidity Protocol appeared first on Software Engineering Daily.

Interest in autonomous vehicles dates back to the 1920s. It wasn’t until the 1980s that the first truly autonomous vehicle prototypes began to appear. The first DARPA Grand Challenge took place in 2004 offering competitors $1 million dollars to complete a 150-mile course through the Mojave desert. The prize was not claimed.

Since then, rapid progress has begun in autonomous driving fueled by advances in sensor technology, software, and the hardware which runs it. Infrastructure has become a serious consideration for autonomous vehicle companies. In this episode, I speak with Vinoj Kumar, VP of Infrastructure at Cruise, the San Francisco company building an all-electric self-driving rideshare and delivery service. They’re tackling the infinite longtail scenarios of city driving and helping Walmart with self-driving grocery delivery.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Autonomous Driving Infrastructure with Vinoj Kumar appeared first on Software Engineering Daily.

Flutter is a UI toolkit developed by Google that helps developers build natively compiled applications for mobile, web, desktop, and embedded device under of Plangora, a web and mobile development company that specializes in PHP, iOS with Swift, Android with Java, WordPress, and Flutter. Allen is also an Elixir Mix panelist at Devchat.tv where he interviews members of the Elixir community. He is a Flutter expert and has a podcast Flying High with Flutter. We discussed native app development and developing with Flutter.

With Flutter, developers can build apps for iOS, Android, Xbox, and web apps and all in production quality. There is also desktop support for Windows, Mac, and Linux. One of the features of Flutter is null checking. It helps to catch a lot of errors that usually users run into. Removing the idea of nulls also reduces the code that users compile. Hence, users can reduce their app size just by turning on the null safety feature. They will also get the benefits of having safer code and fewer crashes in their app.

Flutter runs mostly in the Dart programming language. Dart is a mixture of JavaScript and Java. The advantage of Flutter is that numerous new features are coming into Dart continuously. According to one of the creators of Dart, the technology that helps Flutter code gets transpiled cross-platform actually came from Google Chrome. The user interface of Flutter is also mostly based on Google Chrome. One of the nice parts of the development process by using Flutter is an app feels about the same across, although there are also minor differences between platforms. For instance, on Android users can have some additional options like a list view as they call it, a long scrollable list. Although users can make them change per platform, the apps are the same in general.

Cross-platform compatibility and a similar interface help to cut down on a couple of things. Firstly, general testing is done straightforwardly, and the user makes sure the application looks and feels nice. Since when users run this app on Android or iOS, they are going to get the same feeling. Flutter helps developers to build apps that are not only functional but also good-looking. There are never any worries about visuals in Flutter. The powerful part of Flutter is that users get a consistent feeling across platforms, with some minor differences between every platform. The advantage of cross-platform working is that once users reach an error-free program, they get a very consistent app across both platforms. WeChat, Tencent, BMW, eBay are some of the companies that take advantage of Flutter.

Flutter is easy to learn as well. Allen Wyma says one of his employees who is from a physics background and no coding experience earlier, started developing apps after spending a weekend learning Flutter. He explains there is an explosion of development by using Flutter in Asia, particularly, in Pakistan, Iraq, Indonesia, and the Philippines. In these areas, people download the Flutter SDK, install it and directly start building applications.

Allen Wyma says, “My personal opinion is that Flutter is extremely easy to get up to speed, no matter if you’ve had a background in coding or not. Because a lot of people have no background at all, and they’re building apps. Now, they may not be bug-free, but they’re building apps and some of them look really, really nice. “ In some workshops in Hong Kong, he found that a lot of designers who’ve never even coded before were able to pick up Flutter quite quickly and make some interesting-looking apps. The real power of Flutter is that users can make some very beautiful-looking apps just by using a couple of widgets and components. Since Flutter mostly focuses on the user interface side, it is also important that packages work properly and suit well for a particular application.

Regarding in comparison of React Native and Flutter, Allen Wyma says, “I think, the positives in my experience so far have proven to me and given me the confidence that I don’t ever see a reason to go back to native, especially since I could make whatever I want.”

Allen Wyma says Flutter is not so popular because people do not really understand it. People assume it as a cross-platform application that uses a similar background with other technologies based on HTML, CSS or JavaScript. Secondly, there is the perception that when the developer says having cross-platform, people think users are not going to get very good performance and quality out of it. People think that developers sacrifice the quality of native performance by going cross-platform. However, that is not true at all when users use some application that is based on Flutter. There’s a lot of Flutter apps that most people do not realize are actually based on Flutter.

The Flutter community and development environment are going bigger day by day. Engineers of Flutter add more platforms and features constantly. Flutter becomes a more smooth working application as well. They try their best to make things better. Allen Wyma recommends users go to the dartpad.dev website and try it themselves. Considering the advantages like fast development, expressive design architecture, well-built widgets and cross-platform compatibility, it seems that Flutter will be much more popular in the future. For more information, you can visit the Flutter website flutter.dev.

This summary is based on an interview with Allen Wyma, Founder of Plangora. To listen to the full interview, click here.

The post Episode Summary for Flutter: Native Web and Mobile App Development with Allen Wyma appeared first on Software Engineering Daily.

Money laundering is not a new crime. However, the growth of digital communications has greatly expanded the opportunity for money launderers to find innovative new ways to hide their true intent. Some estimates suggest it could be as high as 2-5% of the world’s GDP.

Unit21 is a customizable no-code platform for risk and compliance operations. They offer a simple API and dashboard for detecting and managing money laundering and fraud. Today on the show, I speak with Clearance Chio, co-founder and CTO of Unit21.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Detecting Money Laundering with Clarence Chio appeared first on Software Engineering Daily.

Web applications often have some sort of login system, and once a user creates an account, they have access to features anonymous users can’t see. In time, application designers will often add an admin level of access for special users. This is often a slow trickle of technical debt. Proper execution of a programmatic authorization system requires concepts like roles, resources, departments, and organizations.

OSO describes itself as batteries included authorization. It’s an open source library used by companies like Intercom and Wayfair whicseh allows them to manage authorization in a robust and standardized framework without reinventing the wheel. In this episode we speak with Sam Scott, CTO at OSO.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Authorization with Sam Scott appeared first on Software Engineering Daily.

As developers hone their craft, becoming more productive often means learning utilities and tools at the command line. The right combination of various parsing commands chained together through pipes can enable engineers to quickly and efficiently automate many adhoc data processing tasks.

In this episode I speak with Adam Gordon Bell about some of his favorite command line tools. We also discuss his role as a developer advocate for Earthly, a powerful tool for building software in a repeatable and understandable way. Adam is also the host of the Co-Recursive podcast.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Earthly and CLI Productivity with Adam Gordon Bell appeared first on Software Engineering Daily.

Modern companies leverage dozens or even hundreds of software solutions to solve specific needs of the business. Organizations need to collect all these disparate data sources into a data warehouse in order to add value. The raw data typically needs transformation before it can be analyzed. In many cases, companies develop homegrown solutions, thus reinventing the wheel and possibly planting deep rooted seeds of technical debt.

Mozart Data helps you collect all of your data sources in under an hour. They provide managed data pipelines, data warehousing, and transformation automation. In this episode, I interview CEO Peter Fishman and CTO Dan Silberman about the modern data stack.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Modern Data Stacks Optimized by Mozart Data with Peter Fishman and Dan Silberman appeared first on Software Engineering Daily.

Arun Kumar is an Assistant Professor in the Department of Computer Science and Engineering and the Halicioglu Data Science Institute at the University of California, San Diego. His primary research interests are in data management and systems for machine learning/artificial intelligence-based data analytics.

His Advanced Data Analytics Lab (ADALab) focuses on basically improving machine learning and artificial intelligence-based data analytics from a holistic standpoint of concerns such as scalability, usability, ease of deployment, ease of development. He calls all of this democratization of machine learning-based data analytics. He is building software systems, abstractions, tools, devising new algorithms, and also studying the theory behind empiricism. The experimental analysis behind this whole process of making machine learning-based analytics easier, and faster, and cheaper for people.

The lifecycle of advanced Data Analytics consists of three phases which are sourcing, building and deploying. The sourcing stage consists of converting raw data in data lakes and data warehouses into ML-ready data. Data preparation is basically reformatting the data. In the second step, the build stage is converting that into a prediction pipeline. MLflow is one of the tools that people use for standardized data. The building stage is where a lot of machine learning and data mining classes focus their efforts on things like how to prepare features. For data scientists, the first thing they need to worry about is defining the application goal. Finding out how data is going to achieve something for the application is the first step towards finding the application goal. The objectives themselves are a big bottleneck, for a lot of users of machine learning.

Data building involves applying the knowledge to the algorithms and the theory of machine learning and connecting them with the application goals that users have. Fairness, inference latency, training cost, interpretability and explainability of data are things that matter significantly. Finally, the deploying step is integrating a prediction pipeline with an application and monitor and oversee the life cycle as the data and application evolve.

Over the last couple of decades, the complexity of the model deployment landscape has exploded due to cloud computing, the internet of things (IoT), heterogeneity of hardware and models. All of these things have caused an explosion of complexity. In the web companies, app companies and the IoT world, deployment is very heterogeneous, and the machine learning environment is no exception. Usage of MLflow, TensorFlow Extended (TFX) and TensorFlow Serving makes it easier for people who do model building and data sourcing. Hence, users can build prediction pipelines.

In these three stages, the amount of scientific inquiry scientists have is still not at the level that they want it to be. A big portion of the computing research in his lab is fundamentally answering questions like, What are practitioners doing accurate enough? Can we reduce the cost and improve productivity? Arun Kumar says he has to interact with practitioners to understand where the bottlenecks are.

One of the projects in Arun’s Data Analytics lab focuses on Project Cerebro. In the Cerebro project, they propose a new system for Artificial neural network (ANN) model selection that raises model selection throughput without raising resource costs. Their target setting is small clusters (10s of nodes), which covers a vast majority (almost 90%) of parallel ML workloads in practice. We have four key concerns: scalability, statistical convergence efficiency, reproducibility, and system generality. To satisfy all these, they develop a novel parallel execution strategy.

There are many tradeoffs that are inherent in a real-world application of ML. For instance, correct model selection is a key bottleneck in raising the accuracy of the data. The Cerebro project aimed at making the process of model selection easier. Arun and his coworkers want to make a sort of high-throughput exploration process so that they will reduce resource costs and reduce runtimes. Therefore, people will be more willing to do more of the sort of exploration and reach better accuracy and better metrics. And that was what they observed in the Cerebro project. He says that is what motivated them to build out the Cerebro system. The Cerebro system was designed in a way that scientists planned to use the tools that they are comfortable with. So they don’t have to change their TensorFlow, PyTorch or Kerris implementation.

The Data stored on data lakes, data warehouses and database systems. Colleagues of Arun Kumar at VMware would like to do machine learning tasks like sentiment mining and named entity recognition. Hence, they could just specify the off-the-shelf transformer architectures, doing hyperparameters for fine-tuning on the data sets, and then give that to the Cerebro API’s, and then they could build it at a higher throughput. This is an example of an application that uses Cerebro as a tool to improve model selection.

Arun Kumar says, “Any prediction task where you would need to build a model and need to tune hyperparameters, need to tune the architecture, Cerebro is useful there. It’s really a very general system for all these sorts of model selection tasks, where you’re building a neural computational graph.”

Neural computational graph frameworks that they currently support are TensorFlow and PyTorch. They are the most commonly used frameworks for specifying neural computation graphs, and executing them and training them with Stochastic gradient descent (SGD). In the future, it can be extended to other frameworks as well.

Cerebro project born out of academic needs. Big corporate companies like Google, Amazon or Facebook could try to solve a problem with 100 GPUs. However, this is not the case in academia. In academia, people need to share their total computation power properly so that everyone on the campus could benefit from it. The technique of model parallelism in Cerebro mitigates the cons of this kind of shared usage. Arun and his team invented Cerebro because they find it necessary for academicians like themselves and also for a lot of domain scientists, academics, enterprises and tech companies.

Regarding excessive usage of resources in the big technology corporations, Arun Kumar says, “They think they could just throw machines at everything and get away with it. And in turn, it comes back to bite people. Like if you’re in the cloud, and you’re just throwing machines at it, who is going to benefit from it? It’s the cloud vendors because they can pocket more money from the enterprises.”

The importance of resource efficiency is one of the reasons why the Cerebro project took this course. VMware is also a big tech company. But their philosophy is very different from all these other companies. Arun believes big tech companies and cloud providers will eventually come around to this worldview because if they’re going to sell products to enterprises, enterprises are going to demand bills that are lower than what they have today.

For the future of data science, Arun thinks, problems are getting wilder and less well-defined. Hence, there will be a lot of research attention that is going to be needed. Scientists need a lot more work on data preparation and data sourcing part of the lifecycle from the scientific and research standpoint. There is also a need for more industry-academia partnerships on the deployment side.

This summary is based on an interview with Arun Kumar, Assistant Professor in the Department of Computer Science and Engineering and the Halicioglu Data Science Institute at the University of California, San Diego. To listen to the full interview, click here.

The post Episode Summary: Data Management Systems and Artificial Intelligence with Arun Kumar appeared first on Software Engineering Daily.

An application network is a way to connect applications, data and devices through APIs that expose some or all of their assets and data on the network. That network allows other consumers from other parts of the business to come in and discover and use those assets (mulesoft.com).

The company Tetrate provides the tools necessary for a highly efficient application (aware) network. Tetrate helps connect and manage applications across clusters, clouds, and data centers. It supports integrating traditional workloads into your cloud-native application infrastructure, defining access control and editing rights or teams on shared infrastructure, and out of the box conformance with NIST standards for microservices security.

In this episode we talk to Varun Talwar, Co-Founder of Tetrate.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Tetrate: Application Aware Networking with Varun Talwar appeared first on Software Engineering Daily.