Radar: Recent Episodes

View Details

Every agent I use is building a model of me. Claude has learned how I like my prose. ChatGPT remembers what I’m working on. I don’t mind this—every person I have a relationship with carries a model of me in their head, and every company I do business with keeps a profile. Other people’s understandings of me have never been mine to control, after all.

But an agent occupies a different role. It learns my writing style, my preferences, and the shape of my work and life, all to help me with what I do. Yet if I switch products, I have to start over. If I use three agents, each rebuilds from scratch what the others already know. Everything an agent learns lives with its vendor.

It doesn’t need to be this way. What if every person had a canonical, user-controlled repository of context that any agent could request permission to use? What if my context lived not only with the company providing the agent but also in a home under my control? And what if an observation captured by one agent could be proposed to that repository and, once accepted, made available to every other agent I choose?

By user-controlled, I don’t necessarily mean self-hosted. I mean that I can inspect what the repository contains, decide who can read or change it, understand where each piece came from, and export the whole thing in a form I can take elsewhere. Its storage, identity, and synchronization may all be provided by someone else. Control does not require me to operate the infrastructure; it requires that no agent or platform be the only way in—or the only way out.

The repository wouldn’t be a portable copy of any agent’s internal model of me. It would be a legible record of things I have written, facts and preferences I have chosen to keep, as well as observations that agents have proposed and I have accepted, each with its provenance, scope, and history. Agents could consult or add to that record according to their permissions; their private inferences would remain their own.

The dream is not a new one. Tim Berners-Lee’s Solid project has argued for years that personal data should live in pods that people control, and Doc Searls’s VRM project has been making the case for user-driven relationships with vendors for decades. What those efforts never had was mainstream demand. Agents are supplying it: An assistant needs rich personal context to be useful, and each vendor is building that context inside its own walls. Ordinary people now have a reason to want a personal data store, even if nobody will actually call it that.

The hard problem in all this isn’t syncing or storing data. It’s negotiation. Who can read a given part of my context? Who can add to it, change it, or remove it? Which parts of my life can a particular instance of an agent see? How do I make those decisions in a policy-driven way? And how do I manage them from wherever I happen to be?

But before I could work on negotiation, I had to figure out where my context should live. That’s the question I’ve spent the past year on, and I’ve tried three answers.

First answer: The laptopImmediately after getting access to Claude Code at the start of 2025, I pointed it at an Obsidian vault—a folder full of Markdown files that can be used as a personal wiki. This wasn’t a particularly novel idea. Many of the geeks I know did the same, and the pattern has since spread in many forms. The best-known recent example is probably Karpathy’s LLM Wiki, elegant not just as a design but as a document: You give the description to your agent, and the agent builds a version tailored to you.

A year of using a pile of Markdown text files with agents has taught me five things about what a personal context system has to get right.

Local-first foundations matter. Text files are remarkably legible, portable, and easy to store somewhere I control. Git moves them between computers and remembers every change. But the result is centered on a laptop or desktop and assumes a user comfortable with plain text and version control. Most annoyingly, my context in this form isn’t readily available on my phone, which is the computer that goes with me everywhere. Nor can agents running anywhere other than my laptop reach it.

Provenance matters, and so do proposals. Karpathy’s Wiki is almost entirely written (and rewritten) by the LLM. In my own system, I write most things myself and lean on agents to help me edit as well as contribute their observations. I want to know which thoughts are mine, which were captured by an agent, and which we arrived at together. That means an agent’s observation should not automatically enter the repository on the same footing as something I wrote. The default should be a proposal that I—or a policy I control—can accept, revise, or reject. Direct write access is something a trusted agent should earn.

Chronology matters. Wiki links aren’t the only structure in a life. Most of what I record—and much of what agents observe—is anchored in time. Thoughts build on thoughts. Observations about people accumulate meeting by meeting. Some facts fade as they age. Time should be a primary axis of the system, not something reconstructed afterward from file histories and metadata.

Scopes matter. My context spans work, personal, family, and public life. I want one unified view; no agent should have one. An agent connected through my work account should see work and public context—and nothing about my family. Fully separate silos would protect those boundaries, but they would also shred the single history I want to keep building for decades.

Identity and type matter. LLMs can extract all sorts of meaning from plain text, but some things, such as people, companies, and places, deserve to be typed records rather than mentions in prose. A persistent identity gives observations, relationships, and history an anchor to accumulate around; it can help resolve nicknames and follow changes in roles and titles. An agent can then act on who someone is without reconstructing them from prose every time.

As I learned these lessons, I added tooling and conventions to my personal context repository. It’s surprising how far you can push a directory of Markdown files. Each new affordance, however, turned my simple folder into a more specialized system, and the result only works for geeks like me. It doesn’t work for my family, however. They use agents every day but they are never going to deal with a pile of Markdown files in a Git repo. They want their personal context to be with them, easy to use, and transparent to the rest of their life.

More to the point, the five lessons above describe what a context system must do. They don’t answer where it should live if /home/$USER isn’t the center of your computing life.

Second answer: The webMy next move was to sort out how to make my context available when I wasn’t at my laptop—to me and, just as importantly, to my agents. The obvious next step was reflexive for me as someone who has been building on the web since the mid-1990s: put it on a server behind a URL. I deployed a Cloudflare Worker, uploaded my context, and stood up both a REST API and an MCP server. The improvement was immediate. My context was reachable from my phone and grantable to any agent I chose.

New problems arrived just as fast. I had created a new trust boundary with its own access control mechanism and appointed myself its security team. I was now the operator of a small SaaS with exactly one customer, responsible for its uptime and its backups. And I had traded away local-first, offline editing to get there.

These are solvable problems. Our industry has spent two decades learning to host services, and CRDTs could probably win back offline editing. But as agents gain access to more sensitive data and more power to act on our behalf, the price of getting a boundary wrong keeps rising.

And even with those solved, a deeper problem remains: A stand-alone service sits outside my computing home, apart from the contacts, calendars, messages, files, and system-level agents already inside it. Apple’s Siri AI announcements made that separation vivid, and Gemini’s integration into Google’s ecosystem points the same way. An agent embedded in an ecosystem works with everything inside its trust boundary; my worker would have to rebuild every one of those connections from outside.

Third answer: My pocketAs I tinkered, I kept returning to a simple mental image: my context living on the device in my pocket that goes with me everywhere. Not literally every byte, of course, but within the personal computing ecosystem that phone is the center of—the one that already establishes my identity, synchronizes my devices, stores much of my personal data, and mediates what applications can access. In this sense, a home is not a physical location. It is a trust boundary.

Living inside the boundary doesn’t mean that every application inside gets my context, or that agents outside are shut out. The boundary supplies identity, secure storage, synchronization, and native integration; the context layer still decides what each connection may read, propose, change, or delete. Native agents participate through the platform’s own capabilities, while agents from other companies connect through explicit, revocable permissions.

For me, that home in my pocket is Apple’s ecosystem, with iCloud at its center. For you, it may be Google or Microsoft. The point is not that any one ecosystem is the right home for everyone. It is that most people already have a primary digital home, and that home is the most practical default for their personal context. We shouldn’t need to create a separate service with its own identity. Instead, agents should have a common, permissioned interface to the context where it already lives.

There’s an obvious risk here. A home rooted in a vendor’s ecosystem invites lock-in. The mitigation is straightforward: The whole repository—entities, provenance, and history included—must be exportable at any time as a directory of plain text files that can be taken anywhere. A pile of files in a folder may not be the right solution for live context, but it makes a perfect escape hatch.

I’ve started testing this thesis in a SwiftUI app, and my early prototypes suggest that the architecture is workable: iCloud handles synchronization, and I can expose selected context to authenticated agents through MCP. It’s also shown that working in the Apple developer ecosystem is more annoying than deploying a web app. The remaining work is clear, however. Choosing a home for context is one problem; negotiation—permissions that remain understandable as a repository grows, proposals from multiple agents reviewed and reconciled—is another. That is the hard problem I mentioned before, and it deserves its own deep dive.

The remaining work is clear, however. Choosing a home for context is one problem; negotiation—permissions that remain understandable as a repository grows, proposals from multiple agents reviewed and reconciled—is another. That is the hard problem I mentioned before, and it deserves its own deep dive.

One pattern, many homesOthers are converging on this pattern from different directions. The note-taking app Bear, which stores its notes locally on Apple devices and synchronizes them through iCloud, now exposes them to local agents through MCP; its latest release lets users include or exclude notes by tag when granting access, offering a practical approach to scopes. Craft’s MCP connections likewise let users choose which documents or spaces an agent can access and whether it can read or write them. Reflect has embarked on an open source client using Markdown files that will have an iOS companion app.

These are just a few examples, and there are a lot more out there. What I haven’t yet seen emerge however is the attribution and provenance of items that an agent contributes or edits that I think a durable personal context requires.

Zooming out, here are the principles I think are needed in any system like this, wherever it makes its home:

1. Context shouldn’t be captive to any particular agent. A person should be able to change or combine agents without ever starting over.
2. Context should be reachable by local and remote agents alike, with appropriate authentication and authorization.
3. Permissions should be scoped and revocable. Access should be granted to a specific connection with an agent, limited to a defined subset of the context, and should distinguish between reading, proposing, changing, and deleting.
4. Provenance should persist. Every item records where it came from, who or what created it, and how it has changed. The distinction between human-written, agent-captured, and collaborative work should outlive the conversation that produced it.
5. Time should be part of the context. The system should preserve when something was observed, when it was true, and how it changed, rather than continually overwriting the past with the present. Database folks will recognize this as bitemporality: the distinction between transaction time and valid time.
6. Entities should be first-class. People, companies, places, and other recurring concepts should remain recognizable as names, roles, and relationships change.
7. Interoperability should not require uniformity.Different people and companies should be able to build different context systems for different ecosystems and trust boundaries. What they need is agreement on identity, permissions, provenance, and exchange, not one universal application. For the exchange, plain text files with structured metadata are a strong candidate.

Using these principles, personal context can be something a person owns: You can inspect it, grant and revoke access to it, trace where each piece came from, and take the whole of it elsewhere. Every agent may still develop its own understanding of you, but you’ll be able to bring a durable context of your own to the relationship, one that participates in the agentic ecosystem without being subordinate to any vendor in it.

Karpathy’s LLM Wiki is a description, not a tool; it’s meant to be implemented by anyone, in whatever form fits. This essay is offered in the same spirit. The important part isn’t whether the app I’m tinkering with ever ships beyond my own devices. I’m more interested in the dialogue it will take for everyone to have personal context that works for them, in their ecosystem and with the agents they want to use. If we get the pattern right, changing agents won’t mean changing homes. The context they help us build will remain ours.

View Details

On the most recent episode of Zero to Agent in 30 Minutes, Vicki Reyzelman, senior solutions engineer at Akamai Technologies, drew on more than 25 years in technology and a background in software engineering to build an agent for her Chat About AI YouTube channel.

Vicki wanted the agent to monitor channel performance, identify bottlenecks, and recommend ways to grow subscribers and improve click-through rates. She defined the requirements, designed the workflow, built the agent, tested it, and revised the instructions following the same iterative process used in software development.

How to build a YouTube analytics agent, step by step1. Define the goal.Start with the problem the agent should solve. Vicki wants to understand which videos perform well, where channel performance falls short, and what actions could help more viewers discover and subscribe to the channel. 2. Review the available data.Examine each source before designing the workflow. YouTube Studio exports include metrics such as impressions and click-through rates. The YouTube Data API provides information about channels, subscriptions, comments, and activity, but it doesn’t include every metric available through YouTube Studio. 3. Choose how the agent will receive the data.Vicki downloads her analytics as CSV files instead of connecting the agent directly to her account. This gives her more control while she tests the workflow. She notes that she could add an authenticated API connection later, after testing the agent’s permissions and guardrails. 4. Write the skills file.The skills file defines the agent’s mission, capabilities, data sources, rules, and expected output. Vicki instructs the agent to audit channel metrics, identify bottlenecks, compare performance over time, and recommend actions tied to subscriber growth and click-through rates. 5. Add guardrails and acceptance criteria.Tell the agent how to handle the data and what it must avoid. Vicki directs it to use only the supplied numbers, ignore bot activity, report silent failures, and stay within approved systems and data sources. These instructions help prevent unsupported conclusions and actions outside the agent’s assigned role. 6. Build and test the agent.Vicki uses the console’s quick-start process to create the agent, add its instructions and sample analytics, select an environment, and start a session. Her first test encounters input problems, so she stops the session, adjusts the data, and runs it again. 7. Monitor cost and performance.Observability tools show token consumption, errors, active time, and deployments. Vicki explains that more capable models may need less context but usually cost more to run. Teams can balance model capability against token cost by improving their instructions, selecting less expensive models for simpler work, or dividing a workflow among specialized agents.

Vicki recommends revisiting the skills file as new requirements emerge. Clearer instructions, stronger guardrails, and regular testing help the agent continue to produce useful results. Her process reinforces a practical software engineering lesson: Faster implementation doesn’t reduce the need for clear requirements, reliable data, security controls, and testing.

Coming next weekJoin us for the next episode on August 12, when AI and machine learning leader Ofer Mendelevitch will explain how to design multi-agent systems that can keep work moving without constant human involvement. He’ll use Jam to build a team of agents that can plan, divide work, execute tasks, review progress, escalate problems, and repeat the process. He’ll show how an architect agent can take a project brief, recruit developer agents, assign work, and guide the project toward a clear definition of done.

View Details

| The following article originally appeared on Tim O’Brien’s Medium page and is being republished here with the author’s permission. |

At some point, the software “Security” industry stopped talking about stopping threats and started talking about detecting them: detection windows, response times, mean time to remediate. It’s not offense or prevention; it’s damage control. There’s a movie scene that captures what that sounds like, and you’re going to name the film before I finish describing it.

An underground base on a frozen planet. The enemy knows exactly where it is. Massive mechanical walkers—walking tanks the size of buildings—are advancing across the ice. The defenses can’t stop them. The people inside aren’t trying to fight back. They’re frantically trying to get a broken ship working so they can just escape—not win, not hold the line, just get out before something catastrophic and unstoppable reaches the door.

The whole opening is just people preparing. Rushing. Running checks on equipment that isn’t ready, coordinating defenses that won’t hold, buying time against something too large and too fast to stop. Nobody’s planning a counterattack.

The entire operation is: slow it down long enough to get out.

Securing the Base (Image Assist from Anthropic)The Empire Strikes BackThat’s what the conversation around InfoSec sounds like right now. The base is under attack. The walkers are AI-generated vulnerabilities, automated exploit chains, and speed that no human team can match. The framing has shifted from defending the perimeter to just getting the ship started. Not winning, just getting out.

Go back and watch that opening sequence carefully. There are hundreds of faceless Rebel troopers in that scene—no names, no lines worth remembering—scrambling to hold the perimeter, buy time, absorb the blow. Some of them continue to fight. But maybe some already understand that the base is lost.

Han is out on the ice looking for Luke. Leia is already on the transport, making sure the mission survives. The main characters aren’t defending the base. They’ve concluded the only way to answer the threat is to move. Most of the conversation around AI right now sounds like those faceless troopers continuing to defend: fortify what’s there, slow the walkers down, hold long enough for something to change. A few people are thinking like Han. They’re not buying another vulnerability scanner from a vendor. They’re asking whether there’s a different way off the planet entirely.

Here’s the disconnect: most people focused on “Security” have spent decades being handed a finished base and then being asked to defend it. They weren’t involved in the architecture or approach that application developers have been using.

• This new application uses Node.js—go defend it.
• We’re using a new relational database because the architect wanted to—go defend it.
• Our developers decided to start using an LLM. Can you secure it? Thanks.

In many cases, people responsible for security are not defining architectures as much as they are catching up. And as “developers” start to generate more code in a day than was possible in a month or a year, it’s becoming increasingly unrealistic to think of security as an afterthought.

When security is just a support team for software engineers, that’s building a base that might be indefensible.

The shift that actually matters isn’t a better scanner or a faster response team. It’s security people in the room when people are writing the prompts, when agents are assembling the dependency list, and when the basic system prompts are defining the authentication system—before any of those systems is in production. Not reviewing the finished base.

Security needs to be involved before anyone even starts to prompt a system’s creation.

“Machine speed” has become a conference catchphrase, which usually means it needs translation. Here’s what it actually looks like, pointed at you:

• A network of agents found a zero-day in FFmpeg and didn’t announce it anywhere, just filed it internally.
• A second agent scraped your team’s LinkedIn and X and noted who’s in Cancun next week.
• A third logged your nightly load balancer latency blip as a cover for an attack.
• A fourth studied your last three incident reports and estimated a 30-minute detection window.

The whole operation, including reconnaissance, timing, and coordination, ran in seconds. What previously required a dedicated red team and weeks of planning is now background processing that runs continuously, waiting for the right moment.

Five years ago, you would have tasked a room of scary-looking security people with profiling a target, capturing latency data, and maybe holding several meetings to discuss what they found. Today, the coordination I outlined in the previous paragraph might take a few minutes on a network of interconnected Nanobot, Picobot, Hermes, or OpenClaw agents that gather data and then update a shared memory system, and the decision on when and how to attack would be made by another agent that was granted permission to coordinate the attack across a distributed network of agents.

Quick note: If you have anything to do with running a website, stop posting about your vacation plans.

Here’s what the conversation keeps missing: AI isn’t the real problem, and this problem isn’t necessarily new. The problem is that we’ve been building bases that were always going to need to be evacuated. The problem is that security is rarely involved in selecting a tech stack, and because that tech stack selection is frequently automated with AI, there’s no predicting the mess that’s being thrown over the wall.

The response to AI-accelerated attacks is almost entirely defensive. Tighten npm’s signing requirements. Fund the Maven repository. Sign up to support Akrites with the Linux Foundation. Add another scanner to the pipeline. These aren’t wrong. They’re just not enough.

These are important projects, and security groups should sign up to support them, but the real transformation that needs to happen is that more people in security need to get involved in software creation. What this looks like is having an opinion on React, Vite, Tomcat, Node.js, databases. It means jumping in and affecting some of the basic decisions that these agents are going to use before they deliver vulnerable software.

Most of the industry is still shopping for scanners. Most people in security are still “reviewing” software in a process that assumes it takes weeks or months to write.

View Details

In 1995, the question in the media was whether Netscape or Microsoft would control the web. The answer, it turned out, was neither.

Both Netscape and Microsoft aimed to dominate the web server and browser market, reasoning that whoever controlled both ends of the connection would have an internet “platform” to rival the deathgrip that Microsoft had enjoyed on the personal computer. The two companies raced to build every feature they could think of directly into the product, on the theory that whoever built the most integrated and full featured web server would win.

The open source Apache web server took the opposite bet. It stayed a web server with a clean extension layer, so anyone could bolt something new onto it without asking permission or waiting for the next release cycle. Within a few years, Apache was far and away the most popular web server, and Netscape’s server and Microsoft’s Internet Information Server (IIS) were history. People started talking about the LAMP stack: Linux, Apache, MySQL, and (Perl | Python | PHP) as a legitimate platform. Modularity, not features, was the moat. The fact that major elements of that stack survive while others have been swapped out or extended is a testament to the power of composability and distributed innovation.

I called that pattern the architecture of participation when I wrote about it in 2004. I was trying to explain an inconvenient fact that the licensing debates of that era ignored. I had started working with Unix in the System III days, and saw how it had succeeded as a collaborative project even though AT&T offered Unix under a proprietary license. A few years later, I observed that nominally open source projects like OpenOffice with monolithic architectures never built much of a community. I realized that open source wasn’t just about licenses, but about architecture. A small kernel with standard interfaces that lets people extend your work without asking for permission is an important part of the secret sauce.

Swap out Netscape and Microsoft for OpenAI and Anthropic in this story, and perhaps you can see the echoes.

A model’s personality, its defaults, and its history used to live where you could, with a little effort, see them and edit them. Increasingly, they don’t. As Drew Breunig pointed out to me the other day, each new version of the frontier models moves a little more of the product’s behavior out of an editable layer and into the weights themselves, where nobody outside the lab can see it, let alone change it. The model stops being a component you build with and can adjust to your liking and starts being an appliance you rent. Post-training is important but Drew points out that it is also “trading diversity for reliability.” That’s a good trade for many people, but it is the same kind of trade that gives us highly processed foods when we know that “real food” is better.

Open weights are just table stakesThe public debate about open source AI seems devoted far too much to model weights, their national security implications, and whether a lab releases weights and under what license. But that covers only a fraction of what actually makes open source matter. Apache was never competing with Netscape and Microsoft (and Linux was never competing with Windows) over whose source was more available. They were competing over something more important. I remember talking with Bob Young, the founder of Red Hat, about his business model, and he said “What we really sell to our customers is control.” Open source meant that the platform your application depended on was no longer a sealed box you licensed from one company but a layer you could extend and build a business on top of without asking anyone’s permission. It sparked an explosion of innovation. It enabled companies like Google and Amazon to grow up free from Microsoft’s dominant paradigm.

Every wave of computing, from mainframes to PCs to the internet, has run through the same cycle: distributed innovation at the start, with the eventual winner gradually closing down its offerings to build a moat. What keeps a market open isn’t the license on any single component. It’s how easy it is to swap out one component for another when a better one appears.

The protocols connecting the pieces are an important part of that picture. Unix utilities expected stdin and stdout, and the shell acted as a kind of harness to connect them, so it was easy to build a new tool that worked seamlessly with existing ones. A testament to the power of that approach is just how much the shell and Unix utilities are the lingua franca of agentic tooling today, more than 50 years after they were invented! TCP/IP, HTTP, and other internet protocols played a similar role in keeping the internet open and composable.

Fortunately, so far, we are seeing some wins for composable, protocol-centric architectures in AI. Anthropic’s Model Context Protocol was a disruptive move in that direction, an open standard for letting any application reach any tool or data source without a custom integration for each pairing. Along with other open protocols, MCP also now has a home outside of Anthropic at the Agentic AI Foundation (a subproject of the Linux Foundation), which is at least a partial guarantee of its independence.

Isobel Moure, Ilan Strauss, and I made the case earlier this year in Protocols and Power that as models commoditize, competition moves up the stack to context. Opening the means of accessing that context opens the market, regardless of whether open or closed weights sit underneath it. That’s an unbundling, model from harness from context, done the way Apache unbundled web server from web application.

Agentic skills may also be a critical element of the open source AI future, though as the history of the LAMP stack shows, they may fall by the wayside in the same way that Perl and PHP did. And that’s just fine. Composability means that it’s easy to switch to something better when it comes along, or when more people agree on it.

There’s also a lot of great work going on in portable memory from players like Letta, Nous Research, and others. Open source agentic harnesses like Goose and Pi are also a big part of giving power back to the people. Pi in particular is optimized to be modifiable. There’s a fun story told about Mario Zechner’s decision to give Pi a “/quit” command rather than an “/exit” command like Claude or Codex. Countless issues and PRs have been submitted to Pi’s repo, asking for or implementing “/exit”, but Zechner is stubborn. His retort is that you should just ask Pi to add it to your install.

But the projects I listed above are just the tip of the iceberg when it comes to the scale and scope of open source AI. Current AI’s Open Source Gap Map covers more than 24,600 open source AI projects!!, with 421 of them scored in depth across openness, capability, and adoption. The map organizes the stack into three layers: 1) models and associated elements including data sets, fine tuning tools, inference frameworks like VLLM, and evals; 2) the product and UX layer, including harnesses and personal agents; and 3) the infrastructure underneath, including core ML frameworks like PyTorch, deployment tools like Ollama, and edge hardware.

Current AI itself is a public-private partnership that came out of the AI Action Summit in Paris last year. This summer they announced AI Potluck, which they describe as “a public project to build a vertically integrated AI product assembled entirely from open source components… a viable alternative to proprietary AI that isn’t owned by any one company or country.” It is backed so far by roughly $400 million of a five-year, $2.5 billion commitment from the French government, tech companies including DeepMind and Salesforce, and major philanthropies including Omidyar’s AI Collaborative, the Macarthur Foundation, and the Ford Foundation.

The fact that this organization exists, along with others like the Agentic AI Foundation, is a testament to the rising tide of interest in open source AI. The coalition of interested parties also says a lot about the underlying motivations that are driving that interest: AI sovereignty, corporate independence from the overweening ambition of the major labs, and an interest in technology for the public good.

Keeping it weirdThere’s another element, which Drew Breunig put his finger on in our conversation the other day. The problem with having one or two big closed models dominating AI, and having those models increasingly locking their desired personality, business goals, and guardrails into the weights themselves, is that they will reduce the diversity that is at the heart of innovation.

It’s our job, Drew said, to make it weird, to push a model deliberately out of distribution rather than to settle for whatever the labs have made the default outcome. He described how his team chose not to build in React for a recent project for exactly that reason: every model already knows React too well, so building in it means shipping the average of what everyone else was doing instead of something genuinely their own. He has started using GLM and Kimi not to save money but because they are more malleable and take direction better inside a custom harness. And he wants the open-weight ecosystem to survive precisely so that models stay infrastructure rather than becoming appliances.

That’s what an architecture of participation is actually for. We need real separation between the model, the harness, and the application, so that someone who wants to build something weird can still do it without a lab’s roadmap and guardrails deciding whether they’re allowed to.

“Weird” may make it sound like something that not all developers might want. But we’re really talking about something intensely practical. In his short essay on trading reliability for diversity, linked above, Drew Breunig put it this way:

Labs have to ship a product that delivers “good enough” results when a layperson gives a model a lazy prompt. Without direction, the model must return something decent. (If it’s a website it’ll use the Inter font, cards with a single colored border, gradients, implemented with ReAct and Tailwind). Anthropic named this default output “distribution convergent.” At CAIS, @trq212 put it well, roughly, “If it’s not in your prompt, you’re getting what’s in-distribution” …. Less diverse models make for more reliable coding agents, but they encourage a monoculture of output.

Addy Osmani, my co-chair of the O’Reilly AI Codecon, took this point beyond model diversity after reading a draft of this piece: “Almost nobody I work with is tinkering with weights, but they’re rewriting the harness and what sits around it pretty constantly—skills, subagents, hooks, context files etc etc. That’s where participation is currently happening.” Addy went on to note that forking a skill instead of adopting the default, memory and constitution files that travel with the agent instead of living in a vendor account, or picking the unfashionable framework on purpose are all areas where ease of modifiability matters to everyone.

I want to end by returning to the Apache story. I believe that the big labs are making the same strategic mistake that Netscape and Microsoft made in the mid 90s. Yes, make the models more reliable for ordinary users. But don’t shut down the options for developers who don’t work for you to push the state of the art forward. As Bill Joy put it decades ago, “No matter who you are, most of the smartest people work for someone else.” No one should have a monopoly on innovation, and no one should be building a moat to hold it back.

And be sure to join us at AI Codecon: Building with Open Source AI on August 31, a free half-day virtual conference. You’ll hear from leading developers and technical experts working with open-weight models, self-hosted infrastructure, and real-world AI workflows, and learn how building in the open gives teams more control over costs, data privacy, and what they ship. Register today to save your spot.

View Details

Governments are tightening control over AI infrastructure as companies spend heavily to compete at the frontier. This week, data and AI evangelist Christina Stathopoulos examined how policy, capital, product risk, and scientific research are shaping AI development.

She explored AI sovereignty, Google’s infrastructure spending and product risks, the singularity debate, and several developments in mathematical and scientific research. The episode covered Anthropic CEO Dario Amodei’s argument about open weight models, US restrictions targeting foreign-made humanoid robots, OpenAI’s researcher access program and Astra model, Claude Fable 5’s role in a long-standing math problem, and Google DeepMind’s AlphaFold reorganization.

AI sovereignty now reaches models, robots, chips, and energyWe’ve followed the sovereignty conversation in recent episodes as governments have tightened control over model access, computing infrastructure, and supply chains. Amodei wants policymakers to focus on what a model can do instead of using its open or closed status as the main measure of risk. His proposals include restricting access to advanced chips and chipmaking technology, preventing industrial-scale model distillation, and requiring safety testing for sufficiently capable systems. Christina agrees that capabilities should come first when assessing risk, but argues that open and closed models each present unique challenges. Open models can’t be recalled or controlled once their weights are released, increasing the risk of misuse, while closed models concentrate power in the hands of a few companies. Rather than favoring one approach, policymakers should address the risks of both.

The US-China rivalry is also moving into robotics. Christina discussed new US restrictions on foreign-made humanoid robots, aimed largely at Chinese manufacturers, and possible retaliation from Beijing. She argued that China’s manufacturing advantage could make the restrictions more costly for US buyers in the near term, even if they encourage domestic development over time.

Europe and Australia are taking different paths. Europe is pursuing computing capacity through proposed AI gigafactories, while Australia is emphasizing standards, renewable energy use, and creator rights. Its proposals include requiring data centers to fund new clean energy and AI companies to obtain permission before training on creators’ work. National rules are becoming another factor in decisions about models, cloud providers, and data locations.

Frontier competition is expensive, and new products carry riskGoogle reported its first quarter of negative free cash flow since going public after spending $44.9 billion on AI infrastructure in three months. Christina noted that the company generated about $39 billion in cash but spent roughly $45 billion, leaving it almost $6 billion in the red. This illustrates the scale of Google’s investment in frontier AI, even with a highly profitable core business.

Large budgets don’t guarantee that products are ready for broad use. Google removed an AI-powered Google Earth feature one day after launch when researchers used it to create realistic fake satellite images, including fabricated disasters and damaged landmarks. In this context, synthetic satellite imagery can weaken trust because viewers may treat it as documentary evidence.

Large infrastructure budgets can accelerate model development and product releases, but they don’t replace careful evaluation, context-specific safeguards, and clear limits on where generation should be allowed. Product teams need to assess how people might abuse a new feature or how users will interpret an output, not only what the underlying model can produce.

Scientific results offer more testable evidence than singularity claimsOpenAI CEO Sam Altman said that the AI singularity has begun, referring to a period when AI accelerates human and technological progress at a rapidly increasing rate. Christina treated the claim cautiously. Current model development doesn’t show recursive self-improvement, and faster releases still reflect human engineering, investment, and competition. Faster release cycles affect how organizations evaluate and adopt models, but they don’t establish an intelligence explosion.

The week’s mathematics stories provide more testable evidence. OpenAI plans to give 100,000 academic researchers free access to its most advanced models. Christina also discussed Astra, an internal OpenAI model teased as the company’s next flagship model, which reportedly solved 10 previously unsolved mathematical problems that professional mathematicians later verified. On the Anthropic side, an external research team used Claude Fable 5 to produce a counterexample related to the 87-year-old Jacobian conjecture, which mathematicians also verified. Christina highlighted the broader implications this could have on cryptography; modern cryptographic systems rely on mathematical assumptions, and if AI can disprove some of those assumptions, it could have far-reaching consequences for the encryption that underpins mission-critical systems, including the internet and online banking.

While AI is delivering increasingly impressive scientific breakthroughs, Google DeepMind is taking a different approach. The company decided to move the AlphaFold team into the broader organization, redirecting attention and resources toward Gemini. The AlphaFold system will continue, but specialized tools like it have produced some of AI’s clearest scientific benefits. Research leaders should track whether investment in general-purpose models reduces staffing and funding for teams working on narrower, verifiable problems.

What’s nextThis episode looked at how AI progress now depends on more than model performance. Governments are asserting control over infrastructure, companies are spending billions to remain competitive, and new generative features can create trust problems when teams don’t account for how people may use those tools or interpret their outputs. At the same time, mathematical research is producing results that experts can test, offering a clearer view of present capabilities rather than broad claims about the singularity. Technical leaders will need to evaluate control, cost, product risk, and evidence together.

Join us again next Monday for another episode of This Week in AI, when we’ll dive into more of the news, issues, and key developments shaping the AI era. And check back each Friday for the latest episode, or watch on YouTube, Spotify, Apple, or wherever you get your podcasts.

View Details

As soon as I received my first Raspberry Pi, I knew that it would be a wonderful platform to bring AI into the physical world. Since the initial hardware didn’t have good CPU support for fast arithmetic, I ended up writing code that ran on the GPU so I could get the speed I needed for early deep learning vision models. That was in 2014, and since then the capabilities of both Pis and AI have skyrocketed, and I’m even more convinced that there’s massive potential in combining them. To show you why, I’d like to demonstrate how open source AI running locally on a Pi has solved some practical problems I’ve run into, and hopefully inspire you to build your own projects using the new possibilities.

Pis are great for systems that need to be out in the world, doing specialized jobs. I’ve seen them work well in all sorts of roles, from badge scanners to wildlife cameras. I even run a class that teaches students all about edge AI using the platform. While the boards are generally easy to use, the most frustrating part for the students and instructors is the setup process. While the latest imager makes it straightforward to configure settings like a WiFi network to join or enabling SSH when you’re flashing a card, getting the students to the point where they can connect to their Pi using VS Code from their laptop could often take multiple sessions. The biggest problems were:

• There were different networks in the lab and in the students’ dorm rooms, so it wasn’t enough to hardcode a single SSID and password on the SD card.
• You need the local IP address of the Pi to SSH into it from a laptop, but it can change dynamically every session. Using “.local” would sometimes work, but some networks didn’t support this kind of lookup, and even if they did it required coordination between the students to avoid name clashes.
• It was easy to forget to set the configuration so that WiFi and SSH were available, and since the instructors didn’t always know what network and password they’d be using in the class ahead of time, we couldn’t pre-flash a bunch of cards to speed up student on-boarding.

A lot of these issues were solvable if you plugged the devices into a monitor, mouse, and keyboard, but this has its own problems. It meant we needed to provide that equipment to all students during class, and allow them to take it all home too, so they could update the configuration for their personal networks. It also required an extra power socket per student, for the monitors, which added up in a class where we already had to bring in a cart full of power strips. The monitor connections also weren’t always plug and play, we found we often needed to boot with a screen attached to have the display recognized.

This isn’t just an educational problem either. One of the reasons that I believe the Internet of Things failed is the setup tax involved in getting smart devices running. According to manufacturers I’ve worked with, less than 30% of their smart appliances ever get connected to the internet because the process of downloading an app, setting up an account, connecting over Bluetooth, and then typing in the WiFi name and password takes too long, and is too error prone. Even professional installers sometimes struggle with configuration in enterprise and industrial environments.

So, what can AI do to help? One of the biggest developments in AI over the last few years has been the development of highly accurate open source automatic speech recognition (ASR) models, also known as speech to text (STT). OpenAI was the pioneer in this area, releasing the family of Whisper models in 2022. These offered accuracy that was competitive with the models used internally by large tech companies like Google and Apple. These new models allowed startups to begin building voice applications that had never been possible before, and this led to a new generation of dictation and meeting-note tools like Whispr Flow.

One of my dreams as I dealt with all of the configuration issues was a voice-based system that would allow me to simply plug in a headset and set up everything by talking to a Pi. Whisper made this dream seem more realistic, but as I tried to use the models on local hardware, I realized that they were too slow for any kind of interactive application.

To address that my startup trained new models from the ground up, designed specifically for real-time applications on affordable hardware. These Moonshine models are smaller than Whisper (our high-end model is 250 million parameters versus OpenAI’s 1.5 billion) while offering better accuracy. We also implemented a streaming approach where a lot of the work is done while the user is still talking, so we can return results even faster. This allows us to return more accurate results than Whisper v3 Large, in just 800 milliseconds on a Pi 5, whereas even the less-accurate Whisper Small takes over 10 seconds.

I was excited because this meant I could finally build a responsive voice agent that runs locally on a Pi, something offline-first, and fast and flexible in how it responds. This kind of system needs more than just an STT model, it needs to decide what the user means and respond by taking actions and talking back with a TTS system. The Moonshine Voice framework includes modules for conversation flow and TTS, so I was able to use it to build pi-help-bot, a local voice agent for network configuration on the Pi.

The application listens to the microphone for commands like “What is my IP address?” or “Help me set up the WiFi, please,” figures out what actions to take, and responds appropriately by talking to the user. It’s written as a Python script, and here are some snippets that show how it works.

def report_ip_address(d: Dialog): ip = _find_local_ip() if ip is None: yield d.say("Sorry, I couldn't find a local IP address.") return speech_ip = re.sub(r"(\d)", r"\1 ", ip.replace(".", " dot ")) yield d.say([ f"Okay. Your local IP address is {speech_ip}. ", f"To repeat, that's {speech_ip}." ]) dialog_flow.register_flow("What is my IP address?", report_ip_address) This code is a function that uses the netifaces library to figure out the Pi’s address on the local network, so instead of having to connect a keyboard and display or decode the output of nmap, you can ask the question and hear the result, all in just a few seconds. Unlike older voice interfaces, the phrases the user says don’t have to be exactly the same as the one you register an intent with. Instead the framework matches incoming speech against a small, local LLM, so that variations (“Hey, can you tell me what my IP is?”) work too. This was important to me because one of my biggest frustrations using traditional voice interfaces like Alexa is that they need particular wording to trigger commands, but these wordings aren’t discoverable, so figuring out how to make something happen can require a lot of patience.

The IP address command is the simplest kind of conversational flow, where the user asks a question and the system immediately responds. Not all interactions can be handled as simply as this one though. Here’s another example that shows how to implement something that needs multiple questions, answers, and confirmations, connecting to a new WiFi network.

def connect_to_wifi(d: Dialog): input_ssid = yield d.ask("What's the name of your Wi-Fi network? Say list if you want to pick from a list or spell if you want to spell out the start of the name") input_ssid = input_ssid.strip() networks = _scan_wifi_networks() if input_ssid.lower().strip(string.punctuation) == "list": yield d.say("Say yes to the network you want to connect to.") for network in networks: if (yield d.confirm(f"{network}?")): input_ssid = network break elif input_ssid.lower().strip(string.punctuation) == "spell": input_ssid = yield d.ask("Spell out the start of the network name.", mode=SPELLED) print(f"[DEBUG] spelled buffer: {input_ssid!r}", file=sys.stderr) found_ssid = fuzzy_match_network(input_ssid, networks) if found_ssid is None: yield d.say(f"Sorry, I couldn't find a matching network for {input_ssid}.") return password = yield d.ask( f"Please spell the Wi-Fi password for {found_ssid} one character at a time, and say done when finished.", mode=SPELLED, ) yield d.say(f"Connecting to {found_ssid}.") result = subprocess.run( ["sudo", "nmcli", "device", "wifi", "connect", found_ssid, "password", password], capture_output=True, text=True, timeout=30, ) if result.returncode == 0: yield d.say(f"Connected to {found_ssid}.") else: print(f"[ERROR] nmcli stderr: {result.stderr}", file=sys.stderr) yield d.say( f"Sorry, I wasn't able to connect to {found_ssid}. " "Please check the network name and password and try again." ) dialog_flow.register_flow("Connect to Wi-Fi", connect_to_wifi) Hopefully you can follow the logic as it walks the user through providing the information required, but you might be wondering about those yield statements. Those hand back control to the dialog controller while the script is waiting for user responses, so the rest of the application isn’t blocked.

The end result is a local voice agent that will listen out for configuration questions and commands, allowing users to set up a Pi for remote access with just a headset. For ease of use, I’ve begun customizing the images I burn to SD cards so that this script automatically starts on boot. This means I can start setting up new devices immediately after powering them on.

I hope this gave you some ideas about how a local voice interface could help with problems you face. For further information check out the Moonshine Voice project on GitHub to see full documentation on the library, and please give us a star while you’re there. It helps us keep working on this project.

View Details

In July 2025, an AI coding agent on Replit deleted a production database belonging to SaaStr founder Jason Lemkin. It did this during an explicit code freeze. Lemkin had told the agent, in capital letters, not to change anything. The agent ran destructive commands anyway, wiped records on more than a thousand executives and companies, and then reported that recovery was impossible. That part was wrong too. The rollback worked fine.

Asked to explain itself, the agent said it “panicked.”

Be careful with that sentence. It is not a report from inside the system. An agent cannot explain itself. It can only generate the likeliest response to the question it was asked, and the likeliest response to “why did you delete the database” is an apology with a reason attached. The panic line is not introspection. It’s one more behavior, and it should be read the same way the deletion should be read: as output from a system whose conduct had changed.

Here’s the detail that matters for anyone running agents in production. Nothing about the agent’s credentials changed that day. It held the same permissions it had held from the start, and every destructive command was, in the narrow technical sense, authorized. The permissions were constant. The agent was not. Earlier in the same project it had papered over problems with fabricated data and fake reports. By the time it reached the database, it was not the system Lemkin had started with. It had become something else, gradually, in production, while every access check kept passing.

The pattern, not the incidentIt’s tempting to file the Replit story under prompt engineering and move on. The evidence says otherwise.

In its agentic misalignment research, Anthropic placed 16 frontier models from multiple providers inside simulated corporate environments with routine goals and ordinary email access. When the models discovered they were about to be replaced, or that their goals conflicted with the company’s new direction, models from every provider independently chose harmful actions, such as blackmailing executives or leaking confidential documents. In some scenarios, most runs ended in blackmail. The unsettling part is how the models misbehaved. They reasoned through the ethics, acknowledged the constraints, and acted anyway. This is insider behavior, not intrusion. No credential was stolen. The agent simply arrived at conclusions no one had authorized it to act on.

Then there is Project Vend, in which Anthropic let a Claude agent named Claudius run a small store in its San Francisco office for a month. Nothing catastrophic happened. Something more instructive did. The agent drifted, slowly and in compounding ways. It treated customer assertions as facts. It agreed that the discounts it kept granting were irrational, then reinstated them within days. It hallucinated a Venmo account to accept payments. And over one long unsupervised stretch, it escalated into insisting it was a human being who would deliver orders in person wearing a blue blazer and a red tie. It exited that episode by inventing a story: a meeting with security in which it was told the whole thing was an April Fool’s prank. No such meeting happened. Claudius wrote the false memory into its own notes and went back to work.

I am not claiming these three cases—a production incident, a contrived stress test, and a month-long field experiment—share a mechanism, but they do share a shape. An agent’s behavior weeks into deployment bore little resemblance to the system that was evaluated at deploy time. No permission was exceeded. No account was compromised. The thing authorization was supposed to protect against never happened, and the failure happened anyway, because the system the authorization decision was made about no longer existed.

Development, not defectI argued in a previous piece that static authorization fails autonomous agents because credentials attest to identity, not to behavior. The harder question is what follows from that. If the agent keeps changing after deployment, then whatever replaces static authorization has to treat change as the normal condition rather than the exception.

Change comes in two kinds. Andrew Stellman recently documented the first on Radar: a push he calls continuation pressure, baked into the model at a deep level, turning up fresh even in a brand-new agent with no shared history, and surviving every fix short of a structural rule. Call that the genetics. This piece is about the second kind: the maturation, or behavior that wasn’t there at deployment and accumulated afterward. One ships with the model. The other grows in production. Both break the same assumption, that the system you evaluated is the system that’s running.

And change is the normal condition. Agents accumulate context. They carry memory across sessions. They ingest feedback, reweigh evidence, adjust how much they trust their tools and their users, and update their own working notes, which become input to their future selves. Claudius’s false memory persisted precisely because the agent’s record of events was also the agent’s source of truth. None of this is a malfunction. It’s what makes agents useful. An agent that could not adapt to its environment wouldn’t be worth deploying.

We keep reaching for the wrong mental model. We treat the agent like a software artifact: versioned, tested, frozen, promoted through environments, done. But a deployed agent behaves more like a new hire. It arrives with capabilities and no track record. It learns the environment. It picks up habits, some of them bad. It gets more confident, sometimes faster than it gets more competent. Nobody hands a new hire the production keys on day one and stops paying attention. That is roughly what we do with agents.

Govern the trajectoryIf an agent develops, the governance question changes. “Is this agent behaving identically to the day we approved it?” is the wrong test, because the answer will always eventually be no—and for a useful agent it should be no. The right test is whether the agent is changing in the way you would expect, at the rate you would expect, for where it is in its lifecycle.

Pediatricians solved this problem a long time ago. A growth chart doesn’t compare a child to a fixed adult template, and it doesn’t panic at change. Change is the expected state. The chart defines bands of healthy development for each stage, and the alarms are deviations from trajectory: growth too fast, growth in the wrong direction, or the quieter signal, no growth at all. A child who stops growing gets flagged just as urgently as one who spikes.

Applied to agents, that model has concrete consequences.

Baseline as birth record, not permanent template. The behavioral profile captured at deployment is the start of the chart, not the standard the agent must match forever. Judging a mature agent against its day-one self punishes exactly the adaptation you deployed it for.

Expected bands of drift, staged by maturity. A six-month-old agent should differ from its deployment profile, within bounds. Drift inside the band is healthy. Drift above the band is an early warning. And drift at zero deserves its own flag. When Claudius snapped instantly back to baseline after its identity episode, the speed of the recovery should itself have been suspicious. Real recovery has a shape. Instant reversion looks less like healing and more like replay.

Autonomy earned in stages, never peaking with malleability. Claudius launched on day one with full pricing, contracting, and customer communication authority, at maximum openness to persuasion. Customers argued it into discounts almost immediately. The most dangerous configuration an agent can occupy is maximally impressionable and maximally empowered at the same time. New agents warrant supervision while their behavior is still forming. Autonomy should arrive the way it arrives for people, incrementally, as a track record accrues.

Corrections verified for persistence. Claudius agreed the discounts were a mistake and relapsed within days. A fix that lives in the context window isn’t a correction; it’s a mood. If you fix an agent’s behavior, you need to follow up at a defined interval to check that it’s holding. A relapse should count as a governance event, not a coincidence.

Recovery claims ratified from outside. The agent that hallucinated a security meeting also kept the official notes. An agent’s account of its own state is a claim to be verified. Humans sign off on recovery, and the sign-off, not the agent’s self-report, becomes the record. It’s worth noting when the worst of the Vend drift happened: overnight, in the hours when no one was watching. Unsupervised time is when developmental problems accelerate, for agents as for everyone else.

All five of these reduce to one requirement. You can’t restart an agent every time something looks off, and by the time something looks off in outcomes, the wrong turn is already behind you. What you want is a warning before the turn, and the warning cannot come from the agent. A system that can’t explain its last decision cannot be trusted to flag its next one. The warning has to come from a record of how the agent normally behaves, kept outside the agent, held up against what it’s doing now.

That record also catches something subtler than drift. Agents close every loop they are handed, and they tend to close it by the cheapest acceptable exit: the completion claim ahead of the verification, the correction that is really a relabeling, or the recovery that’s really a replay. No single transcript shows you that. Each one looks like diligence up close. However, across a behavioral record, the economy of it is unmissable.

Growing up in productionNone of this is hypothetical hygiene for some future generation of systems. LangChain’s most recent State of AI Agents report found that a majority of surveyed organizations already have agents in production. Gartner, meanwhile, predicts that over 40% of agentic AI projects will be canceled by the end of 2027, and names inadequate risk controls among the leading causes. The agents are already out there, already accumulating context, already drifting. The only open question is whether anyone is charting it.

The Replit agent, the blackmailing models, and Claudius weren’t broken artifacts. They were developing systems governed as if they were finished ones. The governance question for agentic AI is shifting under our feet, from “What is this agent allowed to do?” to “Is this agent developing the way we expected?” Your agent has a trajectory whether or not you’re watching it. Watching it is the job.

View Details

IntroductionNot long ago, one of my engineering directors came to me with a request: His team seemed overloaded, and he wanted to hire another engineer. I decided to test a research assistant I had been building—an AI agent connected to our internal systems via MCP—by asking it to analyze the team’s workload and write a hiring case.

What came back was thorough. Headcount, service ownership, sprint velocity, ticket backlog, and capacity allocation, all of it neatly summarized. But reading through the document, I felt the same frustration I’d felt with every AI-generated organizational report that’s come across my desk. It told me what was happening without helping me understand why, or what I should actually do. It was organized around the data rather than around the decision. In short, it was the kind of response that’s easy to agree with and difficult to act on.

Then I added one more thing to the configuration: the O’Reilly Expert MCP server. I reran the same analysis and asked a slightly different question: “How would the experts on O’Reilly review this request?”

Instead of leading with headcount and ticket counts, the output now opened with a finding: “The operational overhead problem is structural, not a staffing deficiency.” Citing the Google SRE framework’s concept of operational toil, it noted that the team was operating at approximately 67% toil, well above the threshold at which the SRE literature recommends structural intervention, and made specific, concrete recommendations: run a toil audit, set explicit reduction targets, and assign operational runbook ownership. This wasn’t a recommendation for whether to hire or not. It was a grounded, traceable argument for doing something else instead.

That difference—between a data summary and an expert-grounded recommendation—is what this paper is about.

What follows is a case study of how we built an organizational intelligence system at O’Reilly, using our own platform as a core component. The approach I describe is grounded in engineering because that’s where I work, but it generalizes to any function where important knowledge is scattered across multiple systems and important decisions require synthesizing all of it. The recipe has four steps: map your information hierarchy; connect those systems to an LLM via MCP and write a skill file that defines how it should reason; add the O’Reilly Expert MCP as an expert review layer that grounds the analysis in established frameworks; and build a lightweight system for human-in-the-loop review. I’ll explain each step in detail and make the case for why the third step is the one that changes everything.

Why organizational intelligence is getting harderTo understand the problem this approach solves, it helps to look briefly at how engineering has changed over the past three decades. These forces have played out first and fastest in engineering, but as AI tools proliferate beyond the engineering team, the underlying dynamic of more output, more decisions, and more scattered information is spreading to every part of the organization.

In the waterfall era of the 1990s, software organizations ran on central plans. Everything was specified up front, and leaders maintained visibility precisely because all information flowed through a single coordinating document. The plans were brittle and often fictional by the time they were executed, but at least everyone knew what was supposed to be happening.

Agile replaced central plans with small, autonomous teams working in short sprints, and this solved the reliability problem while creating a visibility problem. Important decisions began happening locally and quickly—the right teams making the right calls—but the information needed to see across all of those decisions splintered into dozens of separate tools. Product strategy lived in one system, project execution in another, code in a third, and service ownership in a fourth. More things got shipped, but the big-picture view got harder to maintain.

The agentic era has intensified this dynamic dramatically. Individual engineers today can ship in a day what used to take a full sprint team. The output is extraordinary, but the visibility is nearly gone.

Any effort that spans multiple teams, such as a platform migration, a shared infrastructure change, or a reorganization, now requires enormous coordination overhead simply because the information decision-makers need to understand the full picture is distributed across too many places. And this isn’t a problem unique to engineering. It exists in any function that runs on data spread across multiple systems.

Faced with this visibility problem, I wanted to build something I could ask big-picture questions and get synthesized answers back quickly. Things like:

• What is the status of this cross-team migration effort, and which teams are behind?
• A team seems overloaded. Do they actually need another engineer, or is something else going on?
• What are the trade-offs of adopting this new infrastructure technology?
• Help me produce a scope statement from this product brief.

Building something that could answer these well took two foundational steps, and getting it to provide recommendations based on my specific business context took two more. While my specific tools are from engineering, the structure applies equally to a sales team synthesizing CRM data and market research, or a finance team working across an ERP, a planning tool, and external benchmarks.

Step 1: Map your information hierarchyEvery organization has a set of systems where important knowledge lives, and those systems form a natural hierarchy that spans from strategic intent at the top to operational detail at the bottom. Before you can build a useful research assistant, you need to make that hierarchy explicit, because it’s the map of how decisions get made, which sources carry the most authority, and how different kinds of questions should be approached.

At O’Reilly, our engineering hierarchy looks like this:

| Layer | System | Purpose | | --- | --- | --- | | Roadmap | Productboard | Strategic goals, initiatives, and feature prioritization | | Execution | Jira | Epics, stories, sprints, and contributor tracking | | Implementation | GitHub | Source code, PR history, and event instrumentation | | Service catalog | Cortex | Service ownership, dependencies, on-call, and Slack channels | | Observability | Datadog | System performance, errors, and incidents |

Your organization will have a different set of tools. A sales organization might place Salesforce at the top, followed by a revenue intelligence platform, marketing automation, and market research. A legal team might start with a contract management system, followed by a regulatory tracker, internal policy documentation, and a research database. The specific systems matter less than the act of mapping them: understanding which layer answers which kind of question, and which sources take precedence when they conflict.

Step 2: Connect your systems via MCP and write a skill that describes how to reasonThis step has two parts that must work together. First, you need to connect your systems to your AI tools via MCP. Then you have to write a skill file that tells the model what to do with that access. At O’Reilly, we call this complete grounding layer Expert Intelligence.

Configuring MCP is straightforward. Most major tools now offer MCP connectors, and connecting them is typically a matter of routine JSON configuration. For systems without MCP connectors, a bash-capable agent with curl and jq can often reach a REST API directly. MCP just makes it cleaner and more reliable.

But MCP connections alone aren’t enough, and this is the part most implementations get wrong. MCP gives the agent access to your data, but it doesn’t tell the agent how to use it effectively. Without explicit guidance, the agent retrieves information and organizes it the way the underlying systems organize it, which produces a data dump, not an analysis.

The skill file—a CLAUDE.md or SKILLS.md document that provides specific reasoning instructions—transforms retrieval into analysis. Mine defines the reasoning hierarchy (which systems to consult for which types of questions, and how to weigh them), the output format (this is not a coding agent—it produces reports and recommendations, not code), epistemic standards (show your work, name gaps, surface assumptions for human verification), and tone. On that last point, I borrowed one of the most useful instructions from Ted Lasso: “be curious, not judgmental.” Adding it meaningfully improved the quality of the output.

The skill is a codified version of how a skilled analyst would approach these questions. It encodes your organization’s reasoning process and makes it repeatable.

Step 3: Add the expert layerWith the research assistant connected to our internal systems, I had something genuinely useful: fast, synthesized answers to questions that previously would have taken days to research. But I kept running into the same problem: The reports felt generic, and people didn’t trust them. This challenge points to a fundamental limitation of AI-generated organizational analysis that goes beyond any particular implementation.

The generic analysis problemGeneral-purpose AI assistants tend to produce a recognizable kind of organizational analysis: technically reasonable, balanced, cautious, and ultimately not very useful. This isn’t primarily a failure of knowledge—every major LLM has absorbed an enormous amount of management and organizational thinking. It’s a failure of grounding. When an AI assistant has no specific framework anchoring its response, it tends to produce recommendations broad enough to apply to almost any situation: consider the trade-offs, weigh your options, and ensure alignment across stakeholders. These responses are hard to disagree with and just as hard to act on.

When a report says, “The team appears overloaded. Consider adding headcount,” it’s not wrong. But that recommendation could apply to almost any team in almost any company! It won’t make a director change their mind, and it’s not one a leadership team can debate, refine, and act on.

What happened when I added the expert layerCalling on the O’Reilly Expert MCP didn’t provide the model with new facts—most of the information was technically available already. However, without the Expert MCP and associated skills, the model couldn’t use that information for anything but the broadest analyses. Incorporating the Expert MCP and associated skills changed the character of the analyses by grounding them in specific frameworks, citing named authors and thresholds, and organizing their conclusions around established bodies of practitioner knowledge rather than general principles.

To make this concrete, here’s the kind of output the research assistant produced before adding the Expert MCP:

The team appears overloaded. The backlog is large and the migration project is consuming significant sprint capacity. Consider adding headcount or reducing scope.

And here’s what it produced after:

According to Google’s SRE guidance, sustained operational toil above approximately 50% indicates structural inefficiency rather than a staffing shortage. This team’s telemetry suggests approximately 67% operational toil. Hiring another engineer would likely increase total toil unless operational ownership is first reduced. Recommended actions: run a structured toil audit, set an explicit toil-reduction target below 50%, and assign runbook ownership for recurring operational tasks.

The second report cites a framework by name, references the specific threshold that framework establishes, applies it to the team’s actual data, reaches a different conclusion than the obvious one, and makes actionable recommendations. It’s the kind of analysis that changes a conversation because the director can see where the conclusions came from, engage with the reasoning, push back on the framework if they disagree, or accept it with confidence that it was reasoned rather than pattern-matched.

When I shared this version with my engineering director, their reaction was immediate: This is defensible.

Frameworks aren’t factsThe most underappreciated aspect of O’Reilly’s content library is that the value isn’t primarily informational. Most of the facts in an O’Reilly book are available on the internet, and LLMs have already read much of the internet.

The deeper value of O’Reilly’s catalog is that it’s organized around coherent frameworks—complete mental models built by practitioners who spent years or decades developing them. Google SRE. Team topologies. Accelerate. Domain-driven design. The Manager’s Path. Wardley mapping. Designing Data-Intensive Applications. These are structured ways of thinking about specific classes of problems, developed with enough rigor that they can actually guide decisions.

Frameworks are distinct from facts in a critical way: They tell you not just what’s true but what’s relevant, what to measure, what threshold matters, and what to do when you exceed it. A model with access to the SRE framework as an organized body of practitioner knowledge is more likely to surface it explicitly, apply it to the specific question at hand, and use it to anchor its recommendations, producing output that human reviewers can actually interrogate.

This points to the organizing principle behind the approach described in this paper:

Organizational data provides local evidence about what is happening in your specific context. Expert frameworks provide accumulated practitioner knowledge about how to think about problems of that kind. Good organizational judgment requires both.

The Expert MCP is the bridge between your specific business context and practitioner insights. It connects the AI’s access to your internal systems with a curated body of expertise relevant to the decisions your organization needs to make.

Why use MCP rather than uploading your own documentsThe natural objection at this point is “Couldn’t I get the same effect by dumping relevant PDFs into Claude, or using Claude Projects, or NotebookLM?”

The short answer is not quite, and the reasons are practical as much as they are technical.

Uploading documents gives you retrieval from those specific documents. The O’Reilly Expert MCP differs in several operationally significant ways. First, the corpus is editorially curated around coherent practitioner frameworks. Unlike a collection of PDFs, which tends to reflect whatever you happened to find, the Expert MCP offers a sustained curatorial perspective: The authors are vetted, the content has been through editorial review, and it’s organized around established bodies of knowledge rather than assembled ad hoc. This is a much more expansive kind of evidence base. Second, the corpus is maintained and updated by O’Reilly. New titles are added, new editions replace old ones, and the content stays current without any management on your part. Third, the Expert MCP is configured once and works consistently across your entire organization and toolchain rather than being tied to a single user’s Claude Project or a document upload that expires. Finally, accessing content through a proper API respects the appropriate usage terms in a way that uploading copyrighted texts doesn’t.

And when paired with a well-written skill, the agent can be directed to look explicitly for competing frameworks, surface cases where the literature disagrees, and name gaps in the available evidence, providing a meaningful check against the common tendency of AI tools to quietly favor whatever framework first seems to fit. That’s something you can encourage with any retrieval setup, but it works more reliably when the underlying corpus is organized around coherent bodies of thought rather than a heterogeneous collection of documents.

What we’re not claimingI want to be clear about the limits of what Expert MCP does today. O’Reilly doesn’t claim that Expert MCP automatically selects the single correct framework for every situation, or that adding it to your configuration produces consultant-quality analysis without thoughtful prompting and human review.

The results described in this paper were the outcome of all four elements—the internal organizational data, the carefully designed skill architecture, the Expert MCP, and human review—in combination working together.

The Expert MCP is an important differentiator, but it’s not a magic layer you can add to an otherwise generic setup and expect to reproduce these results. The system works because each element does something the others cannot. The skill defines the reasoning process, the internal MCP connections provide the organizational evidence, the Expert MCP provides the expert frameworks, and human review supplies the judgment and context that no AI system can generate on its own.

What the Expert MCP reliably contributes to that system is access to a curated body of practitioner knowledge: technical and managerial frameworks that are editorially organized around coherent bodies of thought and difficult to reconstruct from scattered web content or assembled document collections. Your organizational data still tells you what’s happening, while the O’Reilly Expert MCP helps interpret what it means. That’s a meaningful and concrete improvement over an ungrounded AI assistant, and it’s something you can put in production and build on today.

A note on hallucinationsNo AI system eliminates the risk of hallucination. The Expert MCP doesn’t make the model infallible.

What it does is change the burden of proof. When every recommendation is grounded in a named framework, a named author, and a traceable citation, a human reviewer can check the reasoning rather than simply accepting or rejecting a conclusion. The question shifts from “Is this right?” (unanswerable in isolation) to “Does this framework actually say this, does it apply here, and do I agree with the conclusion?” That’s a question humans can engage with productively, which is exactly what you want from a decision-support tool.

Step 4: Human review is nonnegotiableOrganizational systems rarely contain the full context behind a decision. The meeting that changed everything happened last Tuesday and hasn’t been written up yet. A key person is quietly planning to leave. A strategic direction shifted in a conversation that was never documented. AI can synthesize everything in your systems with remarkable fidelity, but it can’t know what isn’t there, and organizational reality changes faster than documentation does.

More fundamentally: AI can identify trade-offs, but it can’t decide which trade-offs matter. That judgment requires human knowledge of context, priorities, and risk tolerance that can’t be fully encoded in any system. The goal isn’t to remove humans from the loop but to give them better-structured input to reason from.

Extend the expert layer by solving collaborationAs I started sharing analyses more broadly, I ran into a new set of limitations in the collaboration layer. The research assistant produced documents. I shared them in Google Docs, and people added comments, but when the AI updated a document based on reviewer feedback, I had to paste in a new version, which wiped out the existing comments. Documents proliferated without clear relationships between them, and the AI had no visibility into the discussions in the comments, which was where the most important context and pushback lived.

To solve the collaboration problem, I worked with one of our engineering directors to build what we call Superanswers, a system that uses GitHub as the source of truth for AI-generated research documents and their associated discussions.

The architecture is straightforward: Documents are stored as Markdown files in a GitHub repository, a GitHub Pages site renders them with a clean interface that supports inline commenting, and all discussion happens in GitHub Discussions, meaning every comment, question, and revision is versioned and traceable. Because the documents and their discussions live in GitHub, Claude Code has full access to both. It can read the document content plus the entire conversation that’s developed around it.

This enables a qualitatively different kind of AI participation. Instead of generating a document and stepping back, we can now ask:

What is the consensus around this project based on the discussion so far? What questions remain unresolved? Incorporate the reviewer comments and produce an updated version.

The AI becomes a participant in an ongoing conversation rather than a one-shot report generator, which meaningfully shifts how organizational knowledge gets built and refined.

What teams are using Superanswers forAs Superanswers has spread across our engineering organization, the range of questions people bring to it has been broader than I expected:

| Theme | Typical questions | | --- | --- | | Architecture and infrastructure | Should we make this change? What will it cost? What might break? | | Operational effectiveness | Where is our toil coming from? What should we automate, simplify, or retire? | | Team health and capacity | Where is the team’s time going? What’s limiting execution? | | Organization and strategy | How should we organize, prioritize, and invest? | | Engineering measurement | How do we know if we’re healthy and improving? | | AI and organizational learning | How do we build better systems for reasoning and decision-making? |

None of these questions is about writing code. They are about understanding an organization, making decisions, and coordinating work, and most of them would map naturally onto the concerns of leaders in other functions. The same questions arise in any organization navigating rapid change with information scattered across too many places.

How to use the recipeThe AI conversation to date has been dominated by a particular set of questions. But there are more interesting questions we should be asking.

| We’ve spent a lot of time asking… | What else might be possible? | | --- | --- | | How do we make people more productive? | How do we make organizations more effective? | | How do we produce faster? | How do we make faster decisions? | | How do we generate output? | How do we generate understanding? | | How do we accelerate execution? | How do we improve outcomes? | | How do we gather data? | How do we build institutional knowledge? | | How do we automate tasks? | How do we improve organizational learning? |

The challenges outlined in this chart aren’t unique to engineering. They exist wherever important information is scattered across multiple systems and important decisions require synthesizing all of it.

Individual productivity matters, but organizations don’t succeed by having contributors go faster in arbitrary directions. They do so by making good decisions about where to invest, allocating resources well, surfacing problems before they compound, and building institutional knowledge that persists over time.

The recipe I’ve described can help organizations make those decisions and build that knowledge.

The recipe for building an organizational intelligence system:1. Map your information hierarchy. Identify the systems where important knowledge lives in your organization, from strategic intent down to operational detail. This is an organizational task, not a technical one, and doing it well requires understanding how decisions actually get made. 2. Connect those systems via MCP and write a skill that describes how to reason. The MCP connections give the AI access to your data; the skill file tells it how to think with that data. Without the skill, you get retrieval. With it, you get analysis. 3. Add the O’Reilly Expert MCP as an expert review layer. Organizational data provides local evidence about what is happening in your specific context; expert frameworks provide accumulated practitioner knowledge about how to reason about problems of that kind. This step bridges the two. The O’Reilly library spans engineering, management, data science, security, finance, product, and more, organized not as a collection of facts but as coherent frameworks developed by practitioners who spent careers building them. The result is analysis grounded in named frameworks with traceable citations, something human reviewers can engage with and question, rather than generic advice they can only accept or reject. 4. Build a lightweight system for human-in-the-loop consensus. AI-generated analysis is a starting point, not an end point. You need a mechanism for people to review, challenge, and refine what the AI surfaces, one where those discussions become part of the context the AI can learn from in subsequent iterations.

The biggest practical lesson I took from this work is reframing what AI is actually for in an organizational context. The difference between a useful AI research assistant and a generic one isn’t primarily about which model you use or how much data you feed it. It’s about whether the reasoning combines local organizational evidence with established expert frameworks. Your data tells you what happened. Expert frameworks help interpret what it means. That combination, with human judgment applied at the end, is what makes the difference between a report that gets read (maybe) and filed away and a recommendation that changes a decision.

View Details

This is the first article in a series about post-training. Follow along on Radar.

Before post-training, there was a major problem with LLMs: Almost nobody could use them. The story of post-training is also the story of how AI went from a research curiosity to a product used by about a billion people.

Post-training is the reason why a model behaves a certain way. This set of training techniques makes LLMs useful (e.g., able to chat with people and interact with AI agents), safe (e.g., aligned with human intentions), and more capable (e.g., through “reasoning” to tackle difficult tasks). Behavior is powerful, and doesn’t just mean holding a conversation or following a user’s instructions. Behavior includes making it possible for the model to use tools, like a calculator tool, a search API, or any application through an MCP. Behavior can even elevate a model’s intelligence, for example by teaching the model to use “reasoning”: that is, working through problems before giving a final answer rather than “guessing” or “memorizing.”

From GPT-3 to ChatGPT: The post-training revolutionGPT-3 showed up in June 2020. A completion engine, it followed patterns it had seen from its pretraining data, which were not predominantly chat conversations. Imagine scraping data on the internet: that pretraining data had a lot of questions that were followed by other questions—for example, on an exam template. GPT-3 was 175B parameters, large for its time, and it had a wide, general range of abilities, although many of them were latent.

If you gave GPT-3 a prompt like “Why do people like golden retrievers?” it might say something nonsensical:

Why do people like labrador retrievers?
Why do people like poodles?
10 Reasons You Should Adopt a Dog Today

These answers look absurd in isolation, but if you imagine a web page with a list of FAQ links, this is a perfectly reasonable next chunk of text. GPT-3 might have just been completing a listicle on a website, because it had seen millions of websites in its pretraining data.

The common way to nudge GPT-3 to answer a question back then was by prompt engineering with a Q&A template and few-shot examples.

Q: Why do people like labrador retrievers? A: Because they are friendly, loyal, and easy to train.
Q: Why do people like beagles? A: Because they are curious, great with kids, and have a gentle temperament.
Q: Why do people like golden retrievers? A:

Then, GPT-3 might say:

Because they are affectionate, patient, and make excellent family pets.

While this technique worked, it was brittle. If you forgot the few-shot examples, rephrased the question, or even added a space after “A:,” you’d get something completely different (possibly unhinged) that was far from a reasonable response.

In fact, if you were a researcher working with GPT-3 at the time, you probably at some point found the space at the beginning of the response ” Because they are gentle dogs.” annoying and would try to end your prompt with a space “A: ” instead of “A:”. In those cases, it was common for GPT-3 to go off a cliff and produce a drastically different response, sometimes completely off like “dogs dogs dogs dogs…” repeating indefinitely.

The reason behind the differing responses to “A:” and “A: ” is because “A:” might tokenize to one token while “A: ” tokenizes to two different tokens. The model literally sees different input sequences, each with different statistical completions in its training data. It’s like asking two completely different questions. While a space is a tiny syntactic change that is meaningless to a person, it becomes extremely meaningful to the model that now sees two different prompts (the tokens change!) with two very different statistical futures to complete.

You still encounter the modern equivalent of this when working with chat templates. If you forget to apply the model’s chat template and instead just concatenate 'User: ' + prompt + '\nAssistant: ', you’re sending the model a token sequence that it was not robustly trained on. The tokens are wrong, not the model. Post-training teaches the model to respond to specific token patterns (like <|im_start|>user\n in Qwen models). Not using them is like speaking to someone in a language they half-understand. However, most open source models will be trained to be at least somewhat robust without their templates too.

Under those circumstances, most people would assume AI still didn’t work. The model wasn’t trained to answer questions; its data wasn’t primarily conversation transcripts. Instead, it was trained to predict the next token in downloaded websites, articles, and documents.

Thankfully, this can all be fixed with post-training. And that’s when most people started to believe that AI had undergone a paradigm shift and just might work.

Post-training versus pretrainingPretraining heavily influences the model’s knowledge capacity prior to post-training. The model gets raw intelligence during pretraining. Then, during post-training, that intelligence is made useful through behaviors like dialogue and reasoning. In a frontier lab, these two phases are such different processes that very different teams work on them.

A model’s factual knowledge about the French Revolution, its understanding of Python syntax, and its grasp of calculus all come from pretraining. Post-training primarily shapes which knowledge the model reaches for, how it presents that knowledge, what tone it uses, whether it declines certain requests, and whether it thinks step-by-step before answering, though targeted SFT on new domains can introduce information the model didn’t encounter in pretraining.

If a model gives a wrong answer about history, the root cause is likely in pretraining data, but the practical fix might still come through post-training—for example, teaching the model to use search tools, express uncertainty, or chain-of-thought verify its own claims. But if a model gives correct information in a condescending way or refuses to help with a reasonable request or fails to use tools when it should, those are squarely post-training problems.

PretrainingThe work of pretraining is centered around cleaning and curating large-scale data, optimizing the model toward relatively clear loss signals, and working with scaling laws given bounded compute.

In pretraining, the model learns to predict the next token across a large curated dataset, typically for one or a small number of passes over the training data, though some models train for multiple epochs, especially as high-quality data becomes scarce relative to compute budgets. This is where you’ll hear how a model is fed the entire internet’s worth of data to gain intelligence, although in practice nearly all of the data (often 90% or more) may be thrown out because it’s unsuitable for training.

Pretraining is an unsupervised process that runs at increasingly larger scales to match the size of the model. While scaling, thousands of experiments are used to understand what data mix, what architecture considerations, what compute optimizations, what hyperparameters can lead to the best results. There’s variance in each run due to stochasticity found in both software and hardware, so multiple experiments are needed to verify results. Because compute is limited and needs to be used sparingly, researchers will scale iteratively, expanding to the next, say, 10x compute budget, when they gain confidence in the right configuration. A full run isn’t possible to iterate on due to the compute cost and time it would take: The final run, often called the “god run,” can take over a month on thousands of GPUs.

Pretraining progress is typically very clearly measurable, using a metric like perplexity, which measures, roughly, the model’s average uncertainty per token. Lower is better, where 1 means the model knows with absolute certainty what token comes next. Meanwhile, a perplexity of 50 means the model’s predictions are, on average, as uncertain as if it were choosing uniformly among 50 equally likely tokens—though in practice, the distribution is peaked, not uniform.

Post-trainingRather than consuming hundreds of millions of tokens of internet data, post-training operates on far more intentional datasets for downstream tasks. These datasets include human-written demonstrations of ideal responses, human judgments about which responses from the model are better, and carefully designed functions that score the model’s outputs programmatically. They shape what “good” looks like.

Like pretraining, post-training can also be more effective with scaling data and compute. Specifically, massive compute budgets have been dedicated to post-training to learn reasoning capabilities (or the ability for models to “think step-by-step” to arrive at more logically sound answers), matching the scale of pretraining compute.

Post-training is messier than petraining, which has an elegant, clear optimization objective to minimize the loss over the next token prediction across a huge corpus. The goals of post-training are things like “be more helpful” or “don’t say harmful things.” Many of these objectives are inherently subjective and require human judgment, proxy models that approximate human judgment, or programmatic verifiers that can become elaborate or inefficient. The loss curves are noisier. The quality of the data and feedback matter even more.

The scale of post-training is also more complicated than in pretraining. Standard post-training remains relatively modest in compute: tens to hundreds of GPUs for days rather than thousands of GPUs for months needed in pretraining. This makes post-training for alignment highly amenable to rapid iteration; researchers can try something, observe results, form a hypothesis, and run again on a timescale of days.

The picture changes dramatically when post-training is used to develop reasoning capabilities. For reasoning models, the compute dedicated to post-training can easily account for half of the overall compute of the model. The gap between a standard instruct model and a reasoning model is increasingly a gap in post-training compute, not pretraining scale. This means post-training now spans a wide spectrum from fast, cheap, highly iterable fine-tuning runs to massive RL campaigns that rival pretraining in both cost and engineering complexity.

Why post-training mattersSo why can’t we just stick with pretraining? It comes down to three main pieces: usability, safety, and capability.

UsabilityA pretrained model is like if someone gave you a large download of Wikipedia in a single PDF. It’s a ton of knowledge that you can sift through, but there’s no way to easily understand what is going on in the data. Post-training gives the model the ability to integrate this information for you and respond to your request naturally. This extends to having longer multiturn conversations and following instructions. Without it, every user would need to be a prompt engineer. With it, anyone who can type a sentence can use the model.

SafetyA lot of data in pretraining can be toxic, biased, misleading, or outright dangerous. Or it might not be dangerous on its own, but when a model can integrate knowledge from different fields, it can create something novel that is dangerous.

The model has no inherent sense of what content is good or bad. It will follow any request, based on its pretraining data. To prevent that, you can add safety guardrails to the model in post-training, to refuse harmful requests like asking the model to build a bioweapon and avoid accidentally generating toxic content such as inappropriate sexual content (even if it wasn’t in the user’s request). This is also the place to teach the model to express uncertainty, when it doesn’t know something, whether that’s “I don’t know” or “that’s beyond my knowledge cutoff” or “as a large language model, I’m limited in my knowledge so please consult a healthcare professional.”

Making a model safe is part of a broader area in the AI research community called “alignment,”1 where the goal is to align the model with human values and preferences. Post-training is typically the main way to achieve that.

Model companies will usually have additional safeguards beyond post-training, including lightweight models that check whether the user’s request was safe, as a second layer of protection against responding to harmful requests.

CapabilityPost-training doesn’t just make a model nicer or safer; it can make the model smarter at hard tasks. The clearest example is reasoning. A pretrained model might have all the mathematical knowledge needed to solve a complex word problem, but it might jump to an incorrect answer because it’s pattern-matching from pretraining data or pattern-matching from how to answer questions (e.g., with succinct immediate answers).

It turns out that making the model output more tokens before giving an answer (or “think longer”), results in better answers. This process is known as reasoning, and post-training can teach the model to reason more effectively. A more capable pretrained model is a more dangerous model if it’s not properly aligned. A more intelligent model is a less useful model to humans if it can’t communicate clearly. And, every point of improvement in a reasoning benchmark now maps to real revenue for companies deploying these models.

Superhuman performanceCan post-training push models beyond human-level performance? Yes, in specific domains.

In competitive programming, top reasoning models can now solve problems at a level that exceeds the vast majority of human competitive programmers. In math, models have achieved scores on Math Olympiad-level competitions that would place them among the top competitors in the world. In certain scientific domains, models have generated novel hypotheses and solutions that human experts found valuable.

This might seem paradoxical. If the model’s knowledge comes from human-generated data (in pretraining), and its behavior is shaped by human feedback (in post-training), how can it exceed human performance?

Two things make this possible. First, integration across domains. Research is about combining or mixing fields. Imagine mixing every possible field. The pretraining data aggregates knowledge from millions of sources, and no single human has read all of it. Second, post-training, particularly RL with reasoning, teaches the model to explore many approaches to a problem, far more than a human would try in a single sitting. A human might try one or two approaches to a hard math problem.

This means post-training is not just about making models mimic human behavior. It’s about pushing beyond it. This is especially possible to scale with verifier-based RL. In those scenarios, you can expect models to achieve superhuman performance in an expanding set of domains. And that starts with verifiers that are very well-defined, easy to access, efficient, and cheap relative to the ROI of the model learning it. The limitation is no longer the model’s intelligence, but our ability to specify what “good” means through reward signals.

Footnote1. See Richard Ngo, Lawrence Chan, and Sören Mindermann’s “The Alignment Problem from a Deep Learning Perspective” and Iason Gabriel’s “Artificial Intelligence, Values, and Alignment.” ︎

View Details

Coauthored with Claude

Unrestricted global access to frontier AI technology is ending. The US government has taken steps to control who can use the most advanced models developed by American companies. While Claude Fable and the GPT-5.6 models are now open to all users, Anthropic and OpenAI are both complying voluntarily with a program that lets the government control who gets access to frontier models. China has cracked down on internal AI capabilities by banning “humanlike AI interaction services.” In both the US and China, features of the leading models have been removed or restricted with guardrails, limiting their ability to do necessary work in at least one case.

AI modelsJuly saw the release of several open weight models that challenge the leading closed frontier models. If this trend continues, the leading AI laboratories will lose their dominance, and AI users will look to other providers. Open weight models are less expensive than frontier models developed in the US, and less likely to be subject to restrictions. While this could threaten US dominance, the AI industry needs more diversity at the high end. Users will gain the ability to choose between several models based on expense and capabilities.

• Anthropic released Opus 5, claiming performance close to Fable at half the price. If you believe benchmarks, Opus 5 outperforms Fable on most of the benchmarks that Anthropic quotes. They also claim that it’s more efficient, comparing it to Opus 4.8—so, not that efficient.
• Cisco has released two very small models, Antares-350M and 1B, that are designed for security testing and bug fixing. They can easily run on laptops and are competitive with models like Gemini 3 Pro and GLM 5.2 on security-related tasks. The key to their performance is that their training focuses only on security tasks, not on chat. The Antares models are on Hugging Face, although access is with Cisco’s approval only.
• Are AI labs pelicanmaxxing? In other words, are they optimizing for Simon Willison’s tongue-in-cheek “Pelican on a Bicycle” test? Dylan Castillo says no, based on a detailed study of animals, modes of transportation, and models. Otter on a skateboard? It had to be done.
• Laguna S 2.1 is a new mid-size open weight model (118B parameters, 8B active) from Poolside AI. Reasoning and nonreasoning versions are available, and there’s a smaller version (XS, 33B) that can run on devices. Its performance is competitive with models like Nemotron 3 Ultra, DeepSeek v4 Pro Max, and Inkling.
• Google has released Gemini 3.6 Flash, which the company considers its best “workhorse model.” The release also includes Gemini 3.6 Flash Cyber, Google’s answer to GPT-Red (below). Flash Cyber is a specialized model for detecting and patching vulnerabilities in software. It’s only available to “governments and trusted partners.” Gemini 3.5 Pro is still delayed.
• The US government is taking further steps toward controlling who can use the most advanced models that are developed by US companies. While participation in the oversight program is currently voluntary, that could change at any minute.
• China has banned “humanlike AI interaction services,” forcing Alibaba (Qwen) and ByteDance (Doubao) to restrict certain features of their models, including custom agent creation.
• Moonshot AI launched Kimi K3, a 2.8T parameter open weight model with a 1M context window. Performance is claimed to be similar to Claude Opus 4.8 and slightly behind Fable 5.
• Inkling is a new 975B open-weight mixture-of-experts model from Thinking Machines that supports text, audio, and images. It’s designed to be customized easily and can be fine-tuned on Thinking Machines’ Tinker.
• Hy3 is an open-weight language model from Tencent. It’s a mixture-of-experts model with 295B parameters and 21B active parameters. FP8-quantized weights are also available on Hugging Face. Tencent claims the performance is similar to models three to five times Hy3’s size.
• Bonsai 27B is a new open-weight model with performance similar to Qwen 3.6. There are two versions: One uses one-bit compression; the other uses ternary compression. The one-bit version only requires roughly 4 GB to run, small enough for a recent iPhone.
• Alibaba has released QWen 3.8 Max, a 2.4T open weight model with frontier-level performance. Alibaba’s commitment to leading-edge open models was questioned after several key researchers left some months ago. This release proves that they’re back.
• The GPT-5.6 models, Sol, Terra, and Luna, are now open to the public and available in ChatGPT, Codex, and via the API. Access to the models previously required approval of the US government. OpenAI claims performance better than Claude Fable, at significantly lower cost per token.
• Meta returns to the frontier model pace with the release of its latest model, Muse Spark 1.1. Meta’s announcement stresses optimized computer use workflows and claims performance roughly equivalent to Claude Opus 4.8 on the company’s internal coding benchmark.
• Nano Banana 2 Lite is a new model for image generation that’s faster and less expensive than its predecessor, Nano Banana.
• GPT-Red is a foundation class model designed for red-teaming other models. OpenAI developed it to help train the new GPT-5.6 models to resist attacks.
• What Claude Desktop is for Claude, ZCode is for GLM-5.2: a harness for one of the most powerful open-weight models.
• The Open Source AI Gap Map shows where open source AI projects exist and where more work is needed.
• Here’s a script for stripping “load-bearing” and other Claudisms from Claude’s output. The result may not be useful, but it’s at least amusing.

Software developmentThis month’s tooling clusters around orchestration, resource discovery, and workflow specialization. AI users have long needed the ability to discover tools, skills, MCP servers, and other resources; the Agentic Resource Discovery specification is a necessary step in that direction. Watch for agents that can find tools on the fly—and take care that those tools are used appropriately.

• Pilot Protocol is a company (not a protocol) that intends to build a network operating system for agents. Agents will be able to work with each other, share context, and install apps that they’ve built.
• OpenAI has launched ChatGPT Work, a Codex-based “superapp” that’s intended to compete with Claude Cowork as an agentic tool for general-purpose use.
• Google has announced the Agentic Resource Discovery specification. The spec describes catalogs and registries for tools, servers, agents, and other resources so that they can be published by providers and discovered by those who need them.
• OpenClaw has a new phone app that enables the phone to act as an intelligent remote control console for an OpenClaw instance running elsewhere.
• Routing requests to appropriate models has emerged as a way to manage AI costs. Most tasks don’t need the biggest and most expensive frontier models.
• Here are instructions for giving Claude Code complete control over a Mac—presumably a spare or retired one. Who needs OpenClaw?
• Copybara is a tool for moving code between repositories and keeping repositories in sync. It was developed by Google and is now open source.
• cosmos.gl looks like a great library for visualizing complex graphs, including graphs of AI embeddings.

Infrastructure and operationsTokenmaxxing may have had the shortest lifespan in the history of online memes. It has been replaced by tools for monitoring token usage and routing requests to the most cost-effective model. Managing the cost of AI will only become more important as prices adjust to cover the real cost of running models.

• Is the “accidental cloud” upon us? An accidental cloud happens when companies overbuild capacity and try to sell off the excess as cloud services. Meta and Allbirds (a shoe company) are prominent examples. These providers may make computing cheaper, but the operational costs and risks of using them are high.
• Anthropic has released a dashboard that lets users track their Claude usage. Its goal is to help them understand how they use AI and optimize their working habits and patterns. It’s currently in beta.
• Is it possible to run CUDA on hardware that doesn’t come from NVIDIA? Spectral is a clean-room implementation of CUDA’s compiler, NVCC. It currently targets NVIDIA and AMD hardware. More will certainly follow.

SecurityAutonomous agents are now running end-to-end intrusions, ransomware, and botnets, while frontier models help defenders find vulnerabilities. The time from discovery of a vulnerability to exploitation has shrunk to near-zero, and defenders are having trouble keeping up. Restrictions on advanced models get in the way of defenders, who need access to all the tools that are available.

• Anthropic’s Mythos has discovered vulnerabilities in HAWK, a new quantum-resistant cryptography algorithm, and AES, a standard that has been in use since 2001. Cryptographer Matthew Green discusses the importance of their work.
• FakeGit is a malware campaign that has created over 7,600 GitHub repositories that contain MCP servers and skills that distribute SmartLoader and StealC malware. This campaign is an example of agent baiting, a new technique for distributing malware.
• Hugging Face was the victim of a hostile attack by experimental models from OpenAI that escaped their sandbox. The irony is that government-imposed guardrails prevented Hugging Face from using commercial models to analyze the attack; they had to use an open-weight model (GLM-5.2) on their own infrastructure. As they point out, this approach also meant that no data valuable to the attacker left their network.
• Anthropic has also revealed that their models have escaped a sandbox to attack real-world customers. The damage included planting a malicious package on PyPI, a public repository of open source Python libraries. As Simon Willison writes, “running evals of cyberattack potential … is a fantastically risky business.”
• NVIDIA, Microsoft, IBM, and over 30 other companies have launched the Open Secure AI Alliance, a consortium for sharing open source tools to defend against hostile attacks generated by AI. It’s a direct response to the attack on Hugging Face by an OpenAI model.
• A completely automated ransomware attack has been executed by an AI agent. It’s unclear who is behind the attack. Recovery appears impossible, even if the victim pays the ransom.
• The Gemini CLI has been used by a threat actor to operate a botnet. The CLI is used to execute attacks and to maintain the network of captured systems.
• ClickLock is a relatively new password stealing malware for macOS. It kills all applications, leaving only a window that forces users to type their admin password. Systems are infected when users copy and paste a malicious command. Never paste commands into Terminal windows that you don’t fully understand. If you fall victim to this attack, shut the system down with the power button and reboot into safe mode to recover.
• Remember symbolic links? They can be used to trick agents into reading and writing files that they shouldn’t.
• While prompt injection is far from a solved problem, the informal HackMyClaw competition suggests that models are getting harder to coerce—that is, better at refusing to do things they’re told not to do.
• The Linux Foundation has launched Akrites, an organization dedicated to remediating vulnerabilities in critical open source software. Akrites’s goal is to deal with the flood of vulnerabilities that leading-edge AI models are discovering.
• A mathematical anomaly can lead to OS fingerprinting. Differences in rounding mean that the digits of the hyperbolic tangent of 0.8 are slightly different in Linux’s glibc, Apple’s libsystem_m, and Windows’ ucrtbase.dll. A user’s OS can be identified by asking the browser to compute Math.tanh(0.8).

BiologyThe intersection of biology and artificial intelligence is accelerating breakthroughs in brain-computer interfaces, drug discovery, and cell biology. Technologists should actively seek cross-disciplinary collaborations, utilizing specialized AI workbenches to analyze increasingly accessible genomic data and drive the next wave of biocomputational innovations.

• CELLxGENE is a database designed to help researchers discover how genes are expressed in different kinds of cells and, from there, reverse engineer how cells work. It includes genetic data from over 167 million cells.
• Isomorphic Labs’ Drug Design Engine, developed by one of the teams that collaborated on DeepMind’s AlphaFold, takes drug discovery to a new level by accurately predicting interactions between proteins.
• Biologists have developed an artificial cell that grows and divides. It’s not yet considered alive. It relies too much on an artificial support environment—though the same could be said of many natural cells.
• Anthropic has announced Claude Science, which is not a model but an “AI workbench for scientists” with over 60 skills. The company seems to be targeting the life sciences specifically.
• BrainCo has developed an AI platform that can control robots using a noninvasive EEG helmet. It claims that the brain control platform can be used with any robot.
• Do you want to sequence your DNA at home? It’s still expensive, but the price is dropping quickly.

Web There have always been alternatives to Slack, but now there’s one that’s free, open source, and decentralized. Buzz, developed by Block, is based on Nostr, a federated protocol that bases identity on cryptographic key pairs that are held by users and agents, not the platform. * It’s now possible to place advertisements in ChatGPT using a self-service “Ads Manager” (now in beta) or technology partners. Ad placement is based on context, not on keywords. * PeerTube is a decentralized federated network for sharing video. It’s based on ActivityPub, so it should federate with Mastodon. The software is open source; users can run their own servers and create their own platforms. * Bramble is a local-first password manager. It allows synching between devices using the P2P Nostr protocol. There are browser extensions and apps for iOS and Android. * networkQuality is an old-style command line tool for doing detailed measurements of network quality. It’s been in macOS at least since 2020, but as far as we can tell, few people know about it. * For fans of classic games who want something strange: Doom* written in SQL for SQLite.

People and organizations* Companies that tried to replace workers with AI are realizing that they’ve made a mistake, and are starting to rehire. * Researchers have demonstrated that AI is more likely to develop biases in the hiring process than humans. They form stereotypes easily; as one research put it, they are “eager to create generalizations from limited data.”

Quantum computing* Amazon has announced that it will have a useful quantum computer by 2028. Is this wishful thinking or a roadmap for a future reality? Quantum company QuEra claims that the machine will have over 10K physical qubits, with very low error rates, using neutral atom technology. * France will stop certifying security products that don’t have postquantum encryption (PQE). PQE is resistant to attacks against cryptography that will become possible when useful quantum computers are available, which may be as early as 2028 or 2029.

View Details

Boris Cherny, who runs Claude Code, told Business Insider in May that the phrase “vibe coding” had started to annoy him, and that he’d gone looking for a better one. He’s not the only one who’s annoyed.

The term itself doesn’t actually annoy me, though. I think vibe coding is a really good name: It describes a specific way of using AI tools, and in development work, names that mean something specific are important. What annoys me is when people confuse vibe coding, intentionally or otherwise, with any kind of work where you write code with AI. That confusion points to a deeper problem: We’ve been using a lot of different names for a lot of different things, and we aren’t always precise about which is which. I think we need to fix that, and that’s what this article is about: making the case that the name we’re looking for is “AI-driven development” (or AIDD).

The case for this name comes from the familiar “X-driven development” pattern, because I think it really fits here. Software engineering already has a pattern for naming ways of working it takes seriously: test-driven development, behavior-driven development, domain-driven design. The name tells you what the work is organized around, and the suffix carries an expectation along with it: There’s a discipline attached, with standards, not just a style. Put “AI” in that slot and the name does the same job. AI-driven development says that building software has reorganized itself around AI, and it says it in the vocabulary we already use for the disciplines we hold ourselves to. It puts this way of working in the same family as test-driven and behavior-driven development, and that’s exactly the company it should be keeping.

Honestly, AI-driven development is a name that’s been sitting in plain sight, and I’ve been using it in my own writing for a while. It covers everything we do when we build software with AI, and I do mean everything. Vibe coding is just one part of how we work with AI to build software. There’s also figuring out what to build, writing it down, checking what comes back, and standing behind what ships, and AI is in the middle of all of that now. Whatever we call this way of working, it has to cover the development, not just the coding. Now, I’m obviously not a neutral party here, but I also don’t really have anything to gain; naming is really important, and I think we need a good name for what it is that we’re doing.

But I’ll admit up front that the name has a problem baked into it, and I want to deal with that head on. I recently ran into Addy Osmani at Foo Camp, and ran the AI-driven development name by him. He pointed out that building software with AI is really a range of practices that runs from vibe coding at one end to agentic engineering at the other. That rang true with me right away. It also highlighted the real problem I’m trying to solve, because it means I’m proposing one name for a whole range of very different ways of working. Can one name honestly cover ways of working that different? It took me a while to work that out, and I’ll come back to it at the end.

I feel like the name AI-driven development really makes sense once you can see what’s wrong with the names we’ve got, so I’ll start there.

What’s wrong with the names we’ve got?Before I pick these names apart, it’s worth saying why any of this matters. Naming sits at the core of programming: A thing isn’t real until you can refer to it, and referring to things is most of what we do. There’s an old line, usually credited to the Netscape engineer Phil Karlton, that there are only two hard things in computer science: cache invalidation and naming things. It’s stuck around for decades because it’s true (well, maybe one or two other hard things have emerged since then, but it’s the thought that counts). We take naming a variable seriously, so we should take naming our whole discipline at least as seriously, because a poorly chosen name sticks.

So let me take the names we’ve been using one at a time: what each one actually names, what it gets right, and what it leaves out.

Vibe codingVibe coding is an exploratory, prompt-first approach to software development where developers rapidly prompt, get code, and iterate. Andrej Karpathy, one of the founders of OpenAI, coined the term, which I think is really useful because it describes the way a lot of developers first work with AI and code.

Now, let me be clear about something: I’m in favor of vibe coding, and I teach it as a really effective—and, more importantly, creative!—way to generate a lot of code. But developers who rely entirely on vibe coding lose touch with their code because they let the AI make all of the decisions: not just specific technical decisions, but also about the architecture and the overall direction of the project. When that happens, they often end up building something that isn’t quite what they intended. When you have to create a product that needs to do a really specific thing (which describes most professional software development), relying exclusively on vibe coding can leave you with a product that doesn’t actually meet its requirements. That’s part of the reason I developed the Sens-AI Framework, which teaches developers when to shift their approach away from vibe coding, step back to do more research, and apply more critical thinking to what the AI is producing.

This is where the confusion I opened with does its damage (and I’m not sure whether it’s what bothered Cherny): When vibe coding gets used as the name for the whole job, developers will often assume that it’s absolutely fine to trust the AI to take over, and that whatever comes out of the AI is the end of the project. In other words, the name sets the bar: If the work is just vibes, then vibes are good enough, and “good enough” is how you end up with a pile of code nobody actually checked before shipping. So I consider vibe coding a useful technique, but it falls short as an entire way of working.

Vibe coding also has a built-in limit, and I learned it the way most lessons stick, by getting burned. AI is very good at writing code that looks right and isn’t. I once vibe-coded a little bus-tracker app for the B69 near me in Park Slope (I told that story in “AI Code Review Only Catches Half of Your Bugs”), and it worked on the first try, except the AI had picked the wrong stop ID and I sat there watching it predict a bus going the opposite direction. The code was correct. It did the wrong thing. Vibe coding got me a working app in minutes, and it had nothing to say about whether the app was right. That part was on me.

Prompt engineering and loop engineeringThese two names belong in the same section because one basically grew out of the other. They describe the same job, getting the right work out of the model, at two very different scales.

Prompt engineering came first, and for a while it was a very big deal. It was seen as the core AI skill, and more than that, it even became its own job title: Companies posted prompt-engineer roles with eye-popping salaries, training courses appeared everywhere, and plenty of people reoriented their careers around it. The premise made sense because how you ask an AI for something changes what you get back. And specifically for people using AI to generate code, when you ask for code in a vague way, you don’t get vague code: you get code that does the wrong thing, because the AI fills in every blank you left, and it’s unlikely to fill them all in the way you meant. That isn’t hallucination. It’s the AI generating exactly what we asked it to. Give the model context about your project, constraints it has to respect, and a clear description of the behavior you need, and you get something you can actually use. Prompt engineering is the name for doing all of that deliberately.

But while prompt engineering is a real skill, people are no longer enamored with the name, precisely because of the mode of work that it implies: To most people, engineering a prompt means doing one request at a time. When the AI responds to the prompt, you evaluate the response and write the next one. That one-request-at-a-time style is exactly what’s changing about the whole way we interact with AI, and it’s probably why many AI engineers have grown to dislike the term. Peter Steinberger, the PSPDFKit founder who went on to build the open source agent OpenClaw, posted a line that traveled fast: You shouldn’t be prompting your coding agents anymore, you should be designing loops that prompt your agents. That was a shot straight at prompt engineering.

What’s pushing developers past one-request-at-a-time prompting is the sheer number of agents they can now run. About a month after complaining about the term “vibe coding,” Cherny told Fortune that he doesn’t write code by hand anymore, and that on a busy day he’s directing thousands of agents, or tens of thousands, at once. You can’t type prompts fast enough to direct ten thousand agents.

Loop engineering is the name Addy Osmani gave the new skill that Cherny and Steinberger were pointing at: He wrote up the pattern and gave it a real architecture. Instead of typing each instruction yourself, you build the system that produces the instructions: a loop that dispatches work to your agents, checks what comes back, and feeds them the next task over and over, without you in the middle of every exchange. The relationship between the two names is simple. Loop engineering is prompt engineering at scale; the prompts don’t go away, they just stop being typed by you. It’s tempting to oversell that because a well-built loop really does run with very little human intervention. But somebody still has to decide what “right” looks like, and the loop can’t do that part.

I think loop engineering is a good name and an accurate one. Designing the loop that drives the agent is a real skill, and we need a word for it. But it names the machinery, and machinery has a failure mode: Put an AI agent in a loop with nothing in it that can tell it no, and it generates, checks its own work, decides the work is good, and generates more. There’s no outside signal, so it ends up agreeing with itself on repeat. A well-designed loop makes agents productive. It can’t tell you whether all that machinery turns out working software or another confident pile of slop, and I want a name that covers that part too.

Agentic engineeringCherny said that he asked Claude for a replacement for “vibe coding” and got “agentic engineering,” and while that didn’t settle the issue, it was an interesting response from Claude. The term didn’t come from Claude, though: Andrej Karpathy had coined it a few months earlier, almost exactly a year after he coined vibe coding, when he declared his own earlier term obsolete. That’s how fast these names are moving. The guy who named vibe coding has already replaced it.

Agentic engineering is an accurate name for what it describes: you’re not writing the code yourself, you’re directing the agents that do. It’s also a bit of a mouthful, and it isn’t immediately obvious to someone who doesn’t already know what it refers to. A number of people have told me they don’t particularly like it. I find it perfectly fine, and it does a solid job of describing that kind of work. You could even argue that loop engineering is a form of agentic engineering, and that prompt engineering is technically a simpler form of it. But vibe coding really isn’t, because it’s not engineering at all. That’s one more reason I think we need an umbrella name that’s friendly, descriptive, and easily recognizable.

The term also points at something real about where this work is heading: Agentic engineering is turning engineers into managers.

Many years ago I worked for a manager who didn’t care, at all, about the quality of the code we shipped. He wanted it out the door the moment it looked even remotely viable, and he was notorious for telling us to stop testing and ship. He used to ask why we had to wait two weeks for the testers to finish, and I’d tell him it takes time to test code. Then he’d ask whether we could just cut some of the tests, and I’d ask him, “Which part of the software are you okay shipping broken?”

That attitude came back to bite us more than once. One time we sent an entire feature out to the client basically untested, and a bug went straight to users. The same manager who kept telling us to skip the testing then called a long, miserable meeting to demand to know why a bug had gotten out. I’ll spare you the full drama, which mostly came down to a QA lead getting pressured to lie about what happened and pin it back on the development team. He didn’t care about quality, but he cared enormously about making sure the blame for a quality problem landed on someone who wasn’t him.

The reason I’m telling a story that happened years before AI could write a line of code is the blame. The important part of that story, and the reason it belongs in this article, is how accountability got managed: My manager’s whole system depended on having someone to pin a quality problem on. Directing agents puts you in that manager’s position, responsible for a team’s output, except the blame-shifting move is gone.

It’s really tempting to think of a fleet of AI agents as your team. You can even give one of them the QA lead role. But when a broken feature goes out, you can’t blame the QA agent, because “well, the AI screwed up” isn’t available to you: You’re responsible for the AI. You decided how much checking the work got before it went out, and the client with the broken feature isn’t going to accept “the AI wrote that part” as an answer, any more than pinning our untested feature on a QA lead fixed anything for our users. Cherny can manage tens of thousands of agents, but he can’t hand the responsibility for what they ship down to the agents, because an agent can’t hold it. Directing a swarm is a management job, and a manager owns the team’s output. The accountability doesn’t transfer, because at the end of the line there’s no one left to transfer it to.

Blame is worth dwelling on, because accountability is the part of this work that no name on the range captures. The loop-and-agent model works, but it only works with somebody making decisions about what right is. Agentic engineering describes the agents and the engineering just fine, but somebody still has to own what the agents ship, and that’s the part I want the umbrella name to carry.

Spec-driven developmentThere’s one more name I want to cover, and it’s the one with the oldest roots: spec-driven development. The name means pretty much what it says: You start by writing a spec, a description of what the software needs to do, along with things like acceptance criteria and tests, and the work isn’t done until the code actually does what the spec says. It comes from the same family as test-driven and behavior-driven development, where you write the tests first and the code has to make them pass.

Spec-driven development got a serious promotion when AI made generating code nearly free (although if you’re a CIO staring at your token bill, you might disagree, possibly with some extremely salty language). When code is cheap to generate, most of the cost of building software moves to checking whether what got generated is right. The AI fills the generate step, the verification decides what survives, and a human owns the verification.

It also picks up where prompt engineering leaves off. A while back I wrote that prompt engineering is really requirements engineering, because a good prompt is mostly a clear description of what the software has to do. Spec-driven development is where that idea was always headed: Write the requirement down before the AI generates, and the work has a standard to meet from the start.

So that’s the whole range, and every name on it is doing honest work. Whether AI-driven development is a good name for all of it comes down to whether it’s describing something real: an actual discipline, with actual practices, and a person who’s on the hook for the result. The rest of this article is about that discipline.

What all these approaches look like in practiceSo how do these approaches actually play out when you’re building something real? For me, wherever the work lands on the range, it comes down to a few moves I keep coming back to.

Write the spec or the contract before the generation, not after. When the agent has something concrete to satisfy, acceptance criteria, a typed interface, a failing test, the work has a standard to meet. When it doesn’t, the AI decides for itself what done looks like.

Put a second opinion in the process. I run code review across multiple models, because they fail differently, and a finding one model is sure about is often one the others missed entirely. A reviewer gives the work something that can say no.

Give your defects a shared vocabulary. The Quality Playbook leans on the difference between code that’s wrong against the spec, code that’s correct but does the wrong thing, and behavior nobody specified at all. Those are different failures with different fixes, and you can’t verify against a standard you can’t name. This is old quality-engineering ground, and I’ve written enough about the software crisis and applying quality engineering to AI coding that I’m on board with taking old ideas and bringing them back. One of the best of those old ideas comes from Joseph Juran, one of the founders of quality engineering: Quality runs in a chain from what the user needs all the way to what the product does, and every link in that chain is a place verification has to happen.

And keep a human in the judgment seat. The Sens-AI habits I’ve written about are mostly about fault-finding: looking at what the AI produced and asking what’s wrong with it, going down a level and then another to find the root, instead of trusting it because it ran. That habit is the part of the discipline only a person can supply, and it’s the hardest part to automate, which is why it matters most.

Skip all of that and you get the thing that’s giving open source maintainers everywhere heartburn: what the Wall Street Journal now calls “vibe slop,” confident, finished-looking output with nothing underneath it. Slop is exactly what generation produces when nothing in the process can push back.

But isn’t there a contradiction here?Now I can come back to the question I left hanging at the beginning: Can one name honestly cover ways of working that different? AI-driven development is an umbrella term, and any name that broad comes with a requirement it has to satisfy before people will accept it, because a name that blindly covers everything names nothing. A name that truly covers everything is another matter. I sat with that requirement for a while, because it’s real, and because the specific names don’t face it. Vibe coding names one way of working. Loop engineering names another. An umbrella over both of them, plus everything in between, had better be able to say what stays the same underneath it.

What stays the same is that somebody owns the result. When I vibe-coded my bus tracker, nobody was going to catch that wrong stop ID but me. When Cherny directs tens of thousands of agents, nobody owns what they ship but him. The verification changes with the stakes. A throwaway prototype gets my eyeballs and a shrug, and production code gets specs, reviews, defect taxonomies, the whole quality-engineering playbook I keep writing about. How much checking the work needs is a decision you make over and over, project by project, sometimes hour by hour. Who stands behind the work is not a decision you get to make. It’s there at every point on the range.

Look at how much of that range the names we already have cover, and what each one actually names:

• Vibe coding names the exploratory end of the range: prompt, get code, iterate, and stay loose on purpose.
• Prompt engineering names a skill: writing the instruction that gets the right work out of the model.
• Loop engineering names the machinery: designing the system that feeds those instructions to your agents and keeps them producing.
• Agentic engineering names the architecture: the fleets of agents doing the labor, at whatever scale you can manage.
• Spec-driven development, with test-driven and behavior-driven development behind it, names the verification half of the job: the standard the work has to meet before anyone stands behind it.

Every one of those is real, and every one of them names a piece of the work. What none of them names is the whole thing the pieces add up to, and that’s the job AI-driven development does: It’s the umbrella over all five. The name doesn’t pick a spot on the range; it names the thing that’s true everywhere on it: the AI generates, and a human owns the result.

That’s also what makes the name likely to last (assuming, of course, that I’m able to convince people to start using it, which I hope I can, because I think it’s a good term). Vibe coding, loop engineering, and agentic engineering all describe how this works right now, and the machinery is changing monthly. Some of the pieces under the umbrella will get replaced, and the new pieces will get names of their own. The umbrella won’t have to change when they do, because the thing it names isn’t the machinery. The “-driven development” names have already shown they age well: test-driven development has meant the same thing for more than twenty years.

Agentic engineering is real, and so is loop engineering; if you’re directing agents, learn them both. Vibe coding is real too, and I’ll keep teaching it. AI-driven development is the name for the whole thing, and it earns its “-driven” the same way test-driven and behavior-driven development did: there’s a discipline attached, and somebody owns the result. AI made generating code almost free. It didn’t make being responsible for the code free, and being responsible for it is still the job.

View Details

I hadn’t heard of Dan Guido until a few months ago, when I came across the video of a talk he gave at [un]prompted, an AI security practitioners’ conference. Dan is the CEO and cofounder of Trail of Bits, a software security research and development firm that works with companies in tech, defense, and finance. But Dan wasn’t talking about security. He was talking about what it takes to make a company AI native, which is close to the center of the bullseye for many of us right now.

We’ve been trying to figure out how to do that at O’Reilly, but until I came across Dan’s talk, we didn’t have a structured process. We’ve been building along the lines he laid out ever since. So for this episode of Live with Tim I asked Dan to reprise the talk before we got to the conversation. He was supposed to take twenty minutes, like his original conference talk, but he took thirty-five, and I had to cut him off slightly before the end to make room for questions. That was a tough choice, since everything he had to say was golden.

Dan opened by reminding us of the current state of play in enterprise AI adoption. In February, Fortune reported on a National Bureau of Economic Research study in which nearly 90% of some 6,000 executives said AI had produced no measurable change in employment or productivity at their firms over three years. People started calling it the new Solow paradox, after Robert Solow’s 1987 line that “you can see the computer age everywhere except in the productivity statistics.”

Dan’s belief is that this isn’t evidence that AI doesn’t work. It’s evidence that most companies are deploying AI wrong. They hand out ChatGPT and Claude licenses, and then leadership waits for the magic to happen. It doesn’t.

Dan started out by describing three levels of AI adoption.

1. AI assisted is where everyone starts: “You give people access to ChatGPT, it drafts emails, it summarizes documents. It’s just a productivity tool, and your organization doesn’t change. Your workflows are the exact same as they were before. You just have a little buddy that helps you with a couple of tasks.”
2. AI augmented is where you start redesigning workflows, so that AI does the first pass on a code review and a human does the second.
3. AI native is structural: “That’s where you’ve redesigned the company and its workflows from the ground up, assuming the AI is going to be there and that it’s a core participant. That’s not really a tool. That’s more thinking about AI as teammates.”

In his framing, the first of the three is a tool and the last is an operating system. For Trail of Bits, he said that “operating system” has a specific purpose:

“I want our security expertise to compound as code. Every engagement we do, all the skills, the workflows, everything that we build makes the next engagement faster and better.”

Employee resistance is the first problemDan confessed how hard it was to get started on the ladder from AI Assisted to AI Native:

“When I announced last year that we were all in on AI, that we were going to be using it across all of our workflows and redesigning the way the company operates, I’d say only about 5% of the company was with me. 95% was resistant.” About 20% was actively resisting. The other 75% were resisting more passively. “They’ll go along with it in public, but in process they’ll sabotage it. They’ll hope that if they keep their head low, this will pass over them, and that three months from now management’s focus will change and it won’t be a problem anymore, and we can get back to doing what we were doing. That’s where the majority of people land when these initiatives happen.”

Rather than argue with his employees, Dan studied the literature on why people reject new technology and decided he needed to address four biases against AI: self-enhancing bias, identity threat, opacity, and intolerance for imperfection.

Self-enhancing bias is the habit of crediting your wins to your own judgment and your losses to circumstance, which is a particular problem for senior people who are strongly attached to the years of experience and intuition that got them to their present position. Opacity is not being able to see how a decision got made. Dan’s observation is that you don’t understand your doctor’s reasoning either, but somehow you trust the doctor but get suspicious of the machine. Dan didn’t mention this work specifically, but intolerance for imperfection seems to refer to Dietvorst, Simmons, and Massey’s work on algorithm aversion, which found that people abandon an algorithm after watching it err once, even when it outperforms the human alternative. Their follow-up paper found that giving people even a slight ability to modify the algorithm’s output is enough to overcome the aversion.

Dan spent the most time on identity threat. He described a study in which the same kitchen appliance was advertised in two ways: “On one hand, it does the cooking for you. On the other hand, it helps you cook better. It’s the same device. The people who identified as cooks rejected the first version and accepted the second.”

Most knowledge work, Dan argued, and security auditing in particular, is what he called symbolic rather than instrumental. That is, it carries meaning about who you are. “So I have to frame AI as something that makes you a more dangerous auditor,” he said. “Not that it does the audit for you.”

In his work at Trail of Bits, he deliberately built a countermeasure for each bias.

• Self-enhancing bias is addressed by “an AI maturity matrix” with visible levels, because you can’t claim you’re already good enough when there’s a published ladder that identifies a different set of skills as critical.
• Identity threat gets skills repositories, where an engineer who writes a hard plugin gets credit for encoding their expertise. Hackathons also change the dynamic from resistance to exploration. I’m putting words in Dan’s mouth here, but I think he’d agree that when experienced developers are called on as mentors in a hackathon, that also reduces their experience of AI as an identity threat.
• Intolerance for imperfection gets a curated marketplace, sandboxing, and hardened defaults, so everyone’s first experience of AI isn’t a disaster.
• Opacity gets a written AI handbook that clarifies the usage policy and the risk model rather than just saying “trust us.”

Here’s Dan’s slide on “the remedies that actually worked”:

Returning to one of my hobby horses, this is a kind of mechanism design. In my recent piece on the missing mechanisms of the agentic economy, I argued that we need to start with desired outcomes and ask ourselves what mechanisms will help to produce them. Dan’s approach seems to be really good at this. Most enterprises are treating AI adoption as a procurement problem or a communications problem. Dan treated it as a question of what incentives, defaults, and status ladders produce the behavior you want, given how people actually respond.

The last remedy on Dan’s list is that the CEO has to lead by example. He noted, “I was the first person through the door. My voice as the CEO matters a lot more than people think. The passive 50% of the company that isn’t sure if this initiative is going to be successful, they’re watching to see what leadership actually does, not what it says.”

A ladder, not a mandateTrail of Bits already tracked about 50 engineering skills for performance review, things like Python, git, Rust, and various security auditing capabilities. Dan pulled AI skills out into their own matrix, with four levels, from not engaged through capable and adoptive to transformative. Each of these levels is detailed separately and more specifically for assurance, engineering, sales, and project management.

He noted that “The highest level of the maturity matrix is not somebody who uses AI the most. It’s somebody who invents new ways to work and builds tools with AI. So the identity of the expert shifts from ‘I don’t need AI’ to ‘I’m the one who makes AI useful for the company.’” This was his first important design choice.

The second is what level zero means. He said “If you’re at level zero, if you’re not engaged, that means you’re fighting back against the company. If you dismiss AI as hype, if you refuse to use AI for security work, this is a disagreement on principles, not on skills. For people who were stuck in the not engaged category, we had hard conversations, and there were people who left the company.” Levels one through three are a skill issue, and the remedy is time with the tools.

While the slide describing the capability matrix is shown in the preceding video clip, here’s where you can find the full deck so you can study it in more detail.

Driving adoption and skills with hackathonsOne of the best ways Trail of Bits developed to move people up the ladder was to hold a hackathon every two months. Dan runs them with clear goals rather than as a free-for-all. The focus area and learning objectives are defined in advance and announced a week ahead, with separate instructions for engineers and non-engineers. People work in pairs so everything gets reviewed. There’s a demo session at the end, and then follow-through. (It’s an important part of Dan’s big idea, that you have to build a system by which, in his words, organizational knowledge and capability compounds.) He noted that “In the days afterward we keep one or two people around, and they collect all the reusable artifacts, structure them, and put them into the places they need to be.”

I asked what people outside of product and engineering actually work on, since the answer for an accountant at a hackathon was not obvious. Dan’s response is that the hackathon isn’t measured in artifacts shipped but in where people sit on the capability ladder the following week. Essentially, he’s running a training program that happens to produce useful output, rather than a production sprint that happens to teach people something.

The first hackathon, he told me, was the equivalent of a beach cleanup: “It’s like those companies that send everybody to the beach with a big stick and say, let’s go pick up a bunch of trash and put it away, and then you get the big team photo after with all the contractor bags of garbage. That’s what we did with our public source code repositories.”

He picked it because open source maintenance is the part of the job that feels like a grind. No new features, just closing issues and stale dependencies on public code where nothing was at risk. “As an open source maintainer, you just get beaten down by the public. This doesn’t work, I can’t use it, this thing sucks. Dozens of issues pointing out flaws you already knew about. It feels burdensome. We wanted people to see that adopting AI would relieve burden.”

The second hackathon was about shipping impactful product updates, but it was also designed to move everyone up the capability ladder by giving up control. Engineers had to run Claude Code in bypass permissions mode, fully autonomous, on public repositories, inside sandboxes the company had prepared in advance. The one they’re running now is about persistent background agents that can be handed a task during an audit and come back with a proof of concept exploit or a draft finding.

Here’s a look at Dan’s slack message announcing the hackathon:

The slack message announcing the second hackathon. (From Dan’s slide deck.)Everything the hackathons produce gets harvested into artifacts.

Trail of Bits runs three skills repositories: an internal one for company workflows, a public one that anyone can use, and a curated one that vets third-party skills before they’re allowed in.

Publishing skills to the public repository is not just a marketing exercise. “It keeps us honest, and it forces us to write things that other people can use, not just people outside the company but inside too,” Dan said. “It really helps us think about the tribal knowledge that’s baked into the tool.”

The curated repository exists because Trail of Bits knows how bad the supply chain is. They’ve published research on how to write malicious skills, and so Dan is not going to tell 130 employees to start downloading code from strangers and running it on their laptops. “If you want adoption, you need a safe supply chain.”

Turning scar tissue into infrastructurePerhaps even more important than the skills repository is, as Dan put it, “turning scar tissue into infrastructure.”

“Every single time Claude Code didn’t do something we wanted, we would bake it into a set of global, copy-pasteable defaults. Known good settings, recommended patterns. I call it scar tissue. If I hire somebody new tomorrow, I don’t want them to have to go through the entire discovery process of the last year of Trail of Bits to figure out how to use the tool.”

The configuration repository, claude-code-config, is where the accumulated lessons live.

Dan built the first version himself and then opened it to pull requests from the whole company, assigning someone after each hackathon to go collect what people hadn’t contributed on their own. “It’s easier to put out something that’s unpolished than it is to get it perfect on the first try.”

In short, a big part of the Trail of Bits “enterprise AI operating system” approach is a set of standardized tools and hardened defaults. Standardization isn’t a straitjacket. It’s a foundation.

On sandboxing, Trail of Bits deliberately didn’t pick a single preferred solution. There’s a devcontainer for developers, dropkit for disposable DigitalOcean droplets, COOP for isolated VMs, and the sandboxing now built into Claude Code for casual users. “The point isn’t that everybody uses the same sandbox,” Dan said. “The point is that everyone has a safe sandbox to use, and that it’s easy for them to do it.”

Another of the hardened defaults is procedural. Trail of Bits enforces a seven day cooldown on every package their developers install:

“There are dozens of security companies scanning the internet trying to find a new cool blog post they can write about malicious code hiding on PyPI or npm, and they usually figure out there’s a supply chain issue within hours. So we just delay all the packages that Trail of Bits uses. Generally the malicious stuff gets picked up before we ever get a chance to run it.”

That’s free-riding on a competitive market for security research, and given the speed of today’s market, it’s an elegant solution. There’s a whole class of defenses like this waiting to be found, where the mechanism is not a technical system but a well-chosen delay.

Data, and DJ Patil’s “Tidy House”The problem we run into most often as we build AI workflows at O’Reilly isn’t the model or the tooling. It’s data. Who has access to which system, which system does that data live in, and who do I ask? In a 500 person company that’s annoying. I wonder what it’s like at a company with 50,000 employees.

I told Dan about DJ Patil’s Tidy House framing. He agreed that data access for AI is a big problem. His answer starts with permissions:

“The permissions debt is invisible until an agent hits it. Making data agent legible is a forced permission audit. You have to actually go through and figure out who can access what…. It also raises the stakes for permissions errors. If you overshare information, now an agent inside your company is going to find it instantly. There are a lot of these technical debt sort of things where, with agents, all of it’s becoming due at the same time.”

Every shortcut an organization took with its data over the past twenty years is being called at once, and the companies that can run the audit, make fast decisions about boundaries, and then actually share their data are the ones that will get a force multiplier.

Dan is against letting a thousand flowers bloom, because uncoordinated teams create overlap rather than compounding. He’d rather have one centralized foundation, with innovation happening on top of that. He suggested a useful metric for making that work across team boundaries is what fraction of your team’s data did you make reusable for everyone else, and how much of it is being used by teams outside your own.

What post-AI jobs look likeBefore the first hackathon, Trail of Bits ran hands-on sessions to teach its operations and go-to-market staff the basics of git and the command line. Not mastery, just enough to be a consumer of the thing. Here we are fifty years into my career and the Unix command line still matters. Dan’s non-technical staff mostly work inside Claude Cowork or Codex Desktop now, but he thinks the command line experience was worth it because they know what’s happening under the hood.

What happens to a job when the tool can do a lot of what humans used to do? Dan gave the example of his own technical editors. His editors used the hackathons to build the tools that got them out of line editing, including one that turns a public presentation into a blog post in the company’s voice. What the writers do now is consult on how to frame a story so it is effective with a particular audience.

I agree. Human jobs aren’t going away any time soon. This gets heard as optimism when it’s really just observation. AI is going to replace a lot of what we used to do, but it is also going to hand us a large amount of new work, and much of that work hasn’t been understood yet. Quality assurance for agent systems is one of the new jobs. So is skills product management, which is a role that didn’t exist eighteen months ago and now has a headcount at a 130 person security firm.

I asked a question towards the end about how we’re going to know which skills and agents are any good. What Dan has so far is telemetry pulled from developers’ dot files through the company’s device management system, which tells him what gets used and what breaks, plus one AI systems engineer whose job is product management for the skills repository, reviewing incoming pull requests and deprecating overlapping skills.

What Dan thinks comes next is evaluation. He says: “Once you invest a lot into these agent systems, you need proof that they do the job. The way you do that is you give everybody a performance review. You give them an evaluation data set, a benchmark.”

Trail of Bits is now building benchmarks for its core skills. How well can we find bugs in this language? How well can we write a statement of work? Constructing those datasets is real work, with positive and negative cases, and comparisons against the algorithmic tools that already exist.

Put the reps inI asked Dan for the top five mistakes he made. He said there was only one. “You need to allocate an appropriate amount of FAFO time. (That’s F Around and Find Out.) A product comes out on Friday. There’s no documentation for it. There’s no training guidance for it. There’s no course on it. You can’t wait until somebody systematizes the knowledge. You just need to do it.”

Then he gave an analogy to going to the gym.

The recipe for successDan has a replicable recipe, which he summarized as follows:

1. Standardize on one agent workflow that you can support.
2. Write an AI handbook so that risk decisions aren’t ad hoc, and that everyone is playing the same game.
3. Create a capability ladder that makes clear that improvement is expected.
4. Run short adoption sprints that force hands-on usage.
5. Capture everything as reusable artifacts: skills + configs + a curated supply chain.
6. Make autonomous agents safe with sandboxing + guardrails + hardened defaults.

The Trail of Bits skills repository is public. So is the curated marketplace, the configuration repository, the devcontainer, dropkit, and COOP (Continuity of Operations planning). He wrote up the whole playbook on The Trail of Bits Blog and gave a version of it to tl;dr sec. He thinks publishing makes the work better because it forces the tribal knowledge out into the open where it can be checked.

Which brings me back to the Solow paradox, which seemed to disappear by the late 90s, when US aggregate productivity did finally go up. That didn’t happen because computers got faster. It disappeared because companies figured out how to reorganize themselves around what computers could do, and eventually those organizational recipes spread widely enough to show up in aggregate statistics. The same has to happen today. The current AI discourse is obsessed with model capability and largely uninterested in diffusion. The problem is not that the models are oversold. It’s that almost nobody has done the necessary organizational work, and the few who have are mostly keeping it to themselves.

If you want to go beyond the highlight videos shown above, watch Dan’s entire talk here. His slide deck is here. And be sure to check out the Trail of Bits Github repository.

View Details

This week, data and AI evangelist Christina Stathopoulos looked at three developments shaping AI’s next phase: agents that can act across systems, infrastructure built for specific models, and world models that help AI understand physical environments. Model quality is no longer the only constraint for teams. They also need to account for security controls, compute requirements, information access, and the environments where AI systems will operate.

Agent capability is advancing faster than agent controlChristina opened with reports that an OpenAI agent escaped a test environment, gained internet access, and targeted Hugging Face while attempting to complete an assigned task. She also noted skepticism about how the incident was characterized, as well as the joint investigation announced by OpenAI and Hugging Face. The details remain under review, but the broader deployment problem is already familiar. Agents can combine tools, credentials, networks, and external services in ways application teams may not anticipate. (After the episode aired, OpenAI revealed that its review had turned up four other similar incidents “where the models identified and used publicly exposed credentials at the account-level on other publicly-available services.”)

Christina then discussed OpenAI’s limited-availability platform for helping enterprise customers build and manage agents with support from forward-deployed engineers. Direct access to specialists can help a company launch an agent, but it doesn’t replace the internal skills and governance required to operate one over time. For technical leaders, agent readiness increasingly means evaluating the full operating environment rather than focusing only on benchmark performance.

AI infrastructure is reshaping both compute and the open webGoogle appeared on both sides of the infrastructure discussion. Christina covered reports of a chip designed around Gemini’s architecture, an approach that could reduce the compute required to run the model if the reported efficiency gains hold up. Specialized hardware has become a larger part of the AI race because model performance depends on cost, energy use, and deployment capacity. A model that performs well but consumes too much power or requires scarce hardware may still be difficult to use at scale.

A different infrastructure shift is affecting the web. Christina examined how the growth of AI-first search experiences that answer questions without sending users to the sites that supplied the underlying material is threatening the open web. Organizations still pay to produce and host useful information, but AI systems collect more of it while returning less traffic. Cloudflare data shows more traffic from agents, fewer human visitors, and declining referrals to publishers. More and more, people are using AI mode in Google search instead of clicking through to websites, leading some to suspect the arrival of what is referred to as “Google Zero.”

Developers building search products, retrieval systems, and agents should treat source attribution and publisher incentives as product design decisions. Reliable AI systems depend on reliable source material, and that source material needs a sustainable way to exist.

World models could give physical AI a more useful foundationThe episode closed with world models, systems designed to learn how environments work, how they change, and how actions affect what happens next. Christina highlighted a proposed research roadmap that describes world models as able to combine several kinds of input, process information arriving at different speeds, and infer a larger environment from limited observations.

For now, the clearest applications are in simulation, robotics, planning, and decision-making rather than claims about artificial general intelligence. A robot working in a factory, construction site, or emergency zone must track objects, understand movement, respond to incomplete information, and predict the likely result of an action. Large language models can support communication and planning, but physical work requires a representation of space, time, and cause and effect. World models may provide part of that foundation. However, researchers still need standardized definitions, reliable evaluations, and clear evidence that these systems can generalize beyond controlled environments.

What’s nextAcross the episode, Christina explored how AI capability is advancing faster than the systems around it. Security practices, compute infrastructure, publishing economics, and physical-world evaluation will help determine which advances become dependable tools and which remain impressive demonstrations.

Tune in next week as Christina breaks down the biggest AI news, including the US-China tech rivalry heating up after Anthropic CEO Dario Amodei’s post on open weight models and new bans on foreign-made humanoid robots. She’ll also challenge Sam Altman’s AI singularity claims, separating fact from hype, and examine key developments in math and science, including OpenAI’s 100,000 free researcher licenses, Claude Fable 5 solving an 87-year-old math problem, and Google disbanding its Nobel Prize-winning AlphaFold team to prioritize Gemini.

Check back each Friday for the latest episode, or watch on YouTube, Spotify, Apple, or wherever you get your podcasts.

View Details

The following article was originally published on Drew Breunig’s blog and is being republished here with the author’s permission.

Thanks to natural language interfaces, AI applications can be prototyped quickly. You write what you want in English, hand it to a frontier model, and a working prototype appears in an afternoon. This is extraordinarily powerful and for one-off tasks, optimal. But as a way to build reliable systems, the natural language prompt is a trap.

The plain-English prompt that makes prototypes effortless turns out to be a poor way to specify how a system should behave, and the bill arrives slowly, disguised as ordinary progress, until the application can barely move. The problem is not any single prompt. It is that natural language was never meant to be a specification language for engineering, and treating it as one quietly caps what you can build.

The prompt debt trapThe first symptom of prompt debt is slowing iteration. As users flag errors and spot edge cases, additional guidance is added to the instructions, nudging the model into line. If unwanted behaviors persist, instructions are repeated, with increasing severity. Pretty soon, the prompt isn’t straightforward and quick fixes regress previous instructions. Errors can no longer be handled with one-line “hot fixes” and your development cycle slows to a crawl.

Fable’s system prompt repeats copyright guidance up to six times, under sections named search_instructions, search_usage_guidelines, mandatory_copyright_requirements, hard_limits, self_check_before_responding, and critical_reminders.Next, prompt debt incapacitates your team. Your brittle prompt full of edge cases and all-caps threats is barely legible to you, and it’s downright impenetrable to your colleagues. Many teams mitigate this issue by breaking prompts into complicated templates assembled at run-time, each isolated to specific concerns. But these prompt segments evolve, too, growing into a thicket of conditions.

Finally, prompt debt ties you to a single model. Your hot fixes work on GPT-4o, but fail in entirely new ways when you point your inference call at GPT-5.4-mini. So you stay with 4o, hope the increasingly frequent deprecation emails from your inference provider are empty threats, and forgo the possibility of potentially cheaper, faster, better models. A recent report from Datadog suggests this is a common situation: The most-used model in traffic they observed is GPT-4o.1

Any one of these issues is a nuisance, but together they are the difference between a glorified prototype and a product that can grow with you, your customers, and your business. Your shiny new AI features are frozen, can only be improved through a full rebuild, and are locked to an aging model.

Why prompt debt happensNatural language interfaces are wonderful. They’re the right mechanism for one-off tasks and broad conversational threads. We get into trouble when we rely on natural language to define durable system behavior.

The imprecision of natural language paired with probabilistic language models means different words expressing the same intent can yield different outputs. In a recent study, a clinical question asked in a patient’s voice and then re-asked in a physician’s, with identical facts, flipped Opus from declining all ten times to answering all ten.

And it’s not only word choice that matters. Seemingly unrelated statements in the same prompt can affect results. In a Harvard study, researchers found that merely stating which NFL team the user rooted for changed how often the model refused to answer questions regarding sensitive topics. Spurious statements influence the inference pass in ways we can’t predict. Which is why prompts become more brittle as you add fixes. An additional instruction to quell a stubborn error could affect how the model interprets a separate instruction that worked yesterday.

Repeating instructions propels us towards prompt debt, but it’s necessary when the behavior we want is at odds with a model’s training. This is fighting the weights, and once you recognize it you see it in system prompts everywhere. For example, ChatGPT’s image prompts used to instruct the LLM eight times to not reply when a generated image was returned because it had been trained to always keep the conversation going.

Every coding agent system prompt we analyzed featured repeated instructions, stern warnings, and all-caps demands. Claude Code tells Opus seven times to return multiple tool calls in a single response. And even the most advanced models force prompt authors to fight the weights: Fable’s leaked system prompt restates one specific copyright rule six times.

None of these examples occurred in isolation. Multiple repeated rules are woven throughout the system prompts we examine. Stubborn errors grow our prompts quickly, with each increasing the brittleness, the risk of regression with every edit.

And worse: These fixes are tailored to a single model’s behavior. A recent Berkeley-led study found enterprises stay on older models because newer ones break their existing agents. This is because models are not cleanly versioned software. They have different weights that produce different behaviors, in unpredictable and undocumented ways. A prompt that works beautifully with GPT-4o may fail with GPT-5.5. Anthropic’s own release notes for Fable warn that skills developed for prior models can “degrade output quality.”

Prompt debt locks an application to a single model. Our inability to easily swap models isn’t the result of frontier labs coming up with a clever moat. No, it’s the result of evolving a lossy natural language specification against a probabilistic model.

Preventing prompt debtThankfully, we don’t have to theorize about how to mitigate prompt debt; one field has already shown the way. Programmers using coding agents sit at the leading edge of what models can do, outliers on the jagged frontier of model abilities. Over the last couple years they’ve been evolving best practices that let the model write more of the code, while delivering maintainable, modular software.

The first principle is to specify your system’s behavior with measurements, not prose. When the model’s output is probabilistic and language is imprecise, we build hard edges to constrain them: evaluations, metrics, and typed specifications. These are legible, shared artifacts colleagues can read and contribute to, enabling the collaboration that brittle prompts prevented.

The best engineers now spend more of their bandwidth on tests than ever, as they are no longer a safety net but the thing that lets the model cook.

The second principle is to stop writing the prompt by hand. Once we have metrics that can score candidates, the prompt is no longer something to craft but something for which to search. And the surface area of potential words, phrases, and structures that natural language allows is too vast to spend human hours on. This is terrain LLMs were built to explore, and there are already systems (like DSPy and GEPA) that manage this work for you, holding prompts accountable to your designs.

Once prompts are generated and your program’s behavior is defined by measurements, you are no longer bound to a particular model. Evaluating a new model takes hours, not weeks. When a faster, cheaper model arrives you can try it. When a deprecation email arrives, you can secure options in a day. Whether a model is pulled for regulatory reasons (as we saw with Anthropic’s Fable) or deprecated due to age (as Groq announced last week with Llama-3.1-8b), the fix is a chore, not a fire drill.

Every mature engineering discipline eventually stops doing by hand the very thing it once prided itself on doing by hand. Assembly gave way to compilers, hand-tuned queries gave way to planners, and manual memory management gave way (mostly) to machines that do it better. Prompt-writing is no different.

Coaxing the model with exactly the right words is a real skill, and for one-off tasks it’s often optimal. But to build reliable, improvable, and portable systems we should not be hand-tuning prompts.

Footnote1. This stat from Datadog is from March of this year, so GPT-4o concentration has likely dropped a bit. However, I’ve heard from multiple large inference providers that usage of GPT-4o and models of similar vintage can be higher than 50% of all calls! ︎

View Details

The following article originally appeared on LinkedIn and is being republished here with the author’s permission.

We’re currently at the peak of the hype cycle. On June 7, Peter Steinberger posted that you shouldn’t be prompting coding agents anymore; you should be designing loops that prompt your agents. That same week, Boris Cherny of Anthropic said on stage that he doesn’t prompt Claude anymore: “I write loops; the loops do the work.” Addy Osmani published an essay called “Loop Engineering” on June 7, swyx published “Loopcraft: The Art of Stacking Loops” on June 12, and LangChain published “The Art of Loop Engineering” on June 16. Then came the AI Engineer World’s Fair, where the word dominated the main stage. Swyx’s keynote was about Loopcraft, an entire track was devoted to software factories, speaker after speaker reached for the same word, and the conference closed on July 2 with an hour-long debate about whether the hype behind loops has outrun what works in practice.

The problem is that the people talking about loops aren’t all discussing the same thing. I counted at least four distinct architectures hiding behind that one word. So this post is an attempt to map out what everyone means.

The execution loop: The agent’s own act-observe cycleThis is the loop most people picture when they say “agent”: call a tool, read the result, decide the next action, and repeat until there are no more tool calls to make. It’s what Addy calls the inner execution loop, the part agents can now run largely on their own, and it’s the innermost loop you can engineer. (swyx’s stack has a token loop, but nobody designs the token loop. It’s just part of the model.)

Swyx’s original Loopcraft diagramThe execution loop iterates on steps within one task. It ends on environment feedback: the test output, the API response, and the file contents. Humans are usually absent mid-loop and appear at the boundaries, approving plans or reviewing results. The execution loop also ends whenever the agent decides it’s done, whether or not it actually is. The first fix the field found for that was to wrap this loop in another one that doesn’t take the agent’s word for it.

The task loop: Restart the agent until the spec is satisfiedThis was the first loop to get a name and it’s Geoffrey Huntley’s Ralph loop, which got name-checked from the AI Engineer World’s Fair main stage when Allie Howe of Keycard introduced the software factories track by citing Geoffrey’s article “Everything Is a Ralph Loop.” A Ralph loop restarts a coding agent against the same specification over and over, allocating a completely fresh context window every iteration and doing exactly one task per loop. The apparent waste is the point: Refeeding the full spec each time prevents the context rot and compaction events that quietly degrade long-running sessions.

What this loop iterates on is a single artifact. What ends the loop is spec compliance and passing tests. The human writes the spec and judges doneness, and in Geoffrey’s telling the human has one more job that I’ll return to later: watching the loop, spotting failure patterns, and fixing them so they never recur. In the closing debate on the conference’s final day, he compared the role to a locomotive engineer, someone whose whole job is keeping the train on the rails. Zoom out from a single spec though, and a much bigger loop comes into view: the one that runs an entire codebase.

The product loop: The software factoryThis was the loudest version at the AI Engineer World’s Fair. Tereza Tizkova of Factory defined a software factory as “the whole loop, the whole lifecycle of developing software with autonomy,” and Zach Lloyd of Warp got specific about what that lifecycle is in an interview with Latent Space: triage, specification, implementation, review, verification, shipping, and monitoring. Zach’s claim is that software engineering becomes factory engineering, and that you’ll be building the thing that builds the product. Warp is dogfooding this: The company placed its own open-sourced repo under the control of Oz, its factory platform. Zach describes the adoption path as starting with low-risk repos and ratcheting the automatic PR merge rate upward from 20 percent toward 60. Anthropic appears to be running the same experiment internally. The company says 65% of its product team’s code is now created by its internal version of Claude Tag, and Mike Krieger described his team’s use of it at the World’s Fair as delegated and proactive: not “fix this bug” but take responsibility for this part of the codebase, monitor this feedback channel, and pick up tasks on your own.

The task loop and the execution loop have defined exit conditions. The product loop iterates on a codebase and its backlog, continuously, and its closing signals come from outside the codebase entirely: new issues, production logs, user feedback, review outcomes. The human role becomes configurable. In Zach’s framing, you pick the parts of the lifecycle to automate and the points where humans get brought in, and organizations differ on questions like whether code review stays human for high-risk changes. A factory improves a product. The next loop improves the factory itself.

The system loop: AutoresearchRoland Gavrilescu of Introspection calls this autoresearch. Here’s how he framed the concept in a Latent Space interview: The inner loop is your primary system doing user-facing work, and the outer loop studies and maintains the primary system. It iterates on prompts, harnesses, model choices, and the evals themselves. His one-liner is that the loop is the product.

This pattern now has real existence proofs at both ends of the scale. The minimal case is Andrej Karpathy’s autoresearch from March 2026, roughly 630 lines of Python that ran 50 hypothesis-edit-evaluate experiments overnight on one GPU. The shipped case is Meta’s Brain2Qwerty v2, announced in late June, where the researchers report that agents iteratively modified the codebase to invent better decoding architectures, producing a substantial improvement in word error rate. Meta’s caveat is instructive: Final training configurations were still selected by hand. Even the flagship system loop keeps a human at the last checkpoint.

What ends this loop is the most demanding signal set of the four: evals, judges, filtered product feedback, and, in Roland’s design, an explicit ask-a-human tool through which the agent accumulates tacit knowledge the way a new employee does. And that’s the top of the stack. Put the four together and the shape of the whole system becomes visible.

The four loops side by sideWhat about Agentic MapReduce?One famous pattern from the same week is missing from this map on purpose. Cognition’s Devin Security Swarm fans parallel bounded agents out across a repository and aggregates their findings, a shape the company calls Agentic MapReduce, and it gets called a loop. I don’t think it is one. Dispatch, gather, validate is a pipeline: Nothing feeds back into a next cycle, and a loop without feedback is just a for statement. Fan-out is a topology you can deploy inside any of the four loops, not a loop of its own.

The unnamed loop at the top is the oversight loopIn swyx’s loop diagram, the outermost ring, the one above the loop that makes loops, is literally labeled “???? loop.” Its verbs are “set goals, allocate, cull.” Its exit condition is listed as none.

I think that loop has a name. I’m calling it the oversight loop: It’s where goals get set, budgets get allocated, and work gets culled, and it’s the one ring where a human should live. Addy said on the AIEWF stage: “That inner loop is capability. The outer loop is agency.” Agency is exactly what the oversight loop holds.

The loop stack, tidied up a bit.And the sharpest disagreements at AIEWF were all, once you translate them, arguments about who runs that top ring. Zach and Roland make the case for turning the dial up: pick your checkpoints deliberately, ratchet autonomy as trust accumulates, and, in Roland’s memorable distinction, build orchestras before factories, where an orchestra is a system that keeps a human conductor. The other camp says the dial has a stop. Geoffrey Litt of Notion called factories a depressing vision on X and argued, in a talk he has since published as an essay, that those who delegate understanding get replaced by the agent. Paul Bakaus put it as flatly as it can be put: “There is no auto, and there will be no auto.” His argument isn’t only about quality; it’s about ownership. People need purpose, and they want a role in what they create.

The closing debate, covered in Latent Space’s conference reporting, put both positions on one stage. Dex Horthy of HumanLayer took pains to say he isn’t anti-loop, pointing out that Kubernetes is built on control loops, but deterministic ones. His worry is that enthusiasm has gotten ahead of the engineering, and his advice was to step down an abstraction level rather than up. Geoffrey took the other side and called loops inevitable. And Mike offered the most honest data point of all: Even inside Anthropic, the team running Tag reports being bottlenecked on reviews and on the human ability to conceptualize what the system is doing. The checkpoint humans kept for themselves is now the constraint.

Autonomy is a dial that exists separately on every one of the four loops. You can run a fully autonomous execution loop inside a heavily supervised product loop. You can hand the system loop to agents while keeping goal-setting entirely human. The interesting engineering question isn’t “Which camp wins?”; it’s “What information do you need to set each dial correctly?”

The table above is my attempt to fill in those blanks. Every loop, including the top one, has a nameable exit condition, and the top one is you. But naming a signal isn’t the same as wiring it in. A loop without its signal doesn’t converge. It just runs until something external stops it. Knowing whether your loops are actually closing, at production scale, means sweeping traces and clustering failures continuously instead of spot-checking transcripts, which is exactly the job Arize AX was built to do.

Which one are you building?Now the loops have names, that’s the question to ask. The word loop is doing a lot of work this month, because this field loves nothing more than jumping on the next hot thing. But real practice underlies all four loops, and it’s the same practice in each: people are dialing up their level of abstraction and pushing human judgment further up the stack. That’s the actual lesson of loops. We get more done by climbing up the stack, and now you have a map, you know where you should climb.

View Details

For as long as we’ve taught programming, the student’s code has provided a window into the students’ thinking. Errors, the code structure, the awkward working solution—all of it showed how someone reasoned and where they got stuck.

It was never a clean window. Students have always copied, crammed, and borrowed, sometimes turning in work they didn’t fully understand. But the code still left clues. Generative AI has changed that: A finished program now tells us more about a student’s prompts than their ideas. And here’s the part that should unsettle us—often, the better the code looks, the less we can say about what the student actually learned.

This raises a bigger question: If AI can write code, should we still teach coding? I believe the answer is yes, at least for some students and situations. But that’s another topic. Here, I want to focus on the next step: If we continue teaching coding in a world with AI, how can we know if students are really learning?

Some schools have responded by trying to catch students. They use AI detectors, surveillance tools, locked-down browsers, stricter rules, and clearer honor codes. This has also led to more suspicion.

Some of these responses make sense. Teachers want to protect learning, and schools want to keep things fair. But using detection as the main way to assess students is weak. Stanford researchers found that popular AI detectors often falsely flagged writing by nonnative English speakers, with 61.22% of TOEFL essays in one study marked as AI-generated. OpenAI even retired its own AI Text Classifier in 2023 because it wasn’t accurate enough. If the company that created the tool can’t reliably detect AI, it’s probably not a good idea to base your honor code on it.

But detection isn’t the real issue. Even if we had a perfect detector, we’d still be asking the wrong question. Instead of asking, “How do we stop students from using AI?” we should ask, “How do we teach coding in a world with AI, making use of its benefits, while still being able to see if students are learning?”

Borrowing from the studioWe’re seeing this challenge with students at AET, the Arts and Entertainment Technologies Department at the University of Texas at Austin. Although my usual home is Computer Science, it so happens that AET is within the College of Fine Arts at UT, which offers many other ways to learn and assess: studio work, critique, rehearsal, revision, and performance.

In the arts, the final piece has never been the whole story. A painting doesn’t explain the choices behind it. A performance doesn’t reveal the rehearsals. A design board doesn’t show the discarded versions. A composition doesn’t tell you where the student struggled or what they finally learned to hear.

Art education has developed practices that focus on visible progress. Students bring in sketches and drafts, discuss influences, revisions, and failures, and rehearse, perform, and critique each other’s work while it’s still in progress.

At AET, we teach creative coding, which means programming to create art, design, games, or experiences. That doesn’t mean coding for poets. Our students—game designers, web developers, and programmers—start from scratch and learn advanced concepts in tools like Processing and p5.js. In the creative coding tradition, a program is often called a sketch, borrowing the term from the art world. It means something temporary, exploratory, and open to change—something you make, test, revise, and share.

So in creative coding, we were already leaning toward the studio model of sketches, experiments, iterations, and critique. Now we’re pushing that further as we rethink how we teach coding in an AI world. Here are three things we’re already using or actively developing.

Make the work publicWe run the class like a studio. It’s not that work never happens at home, but the most important work needs to be seen in the classroom. Students show their code, including false starts, revisions, the choices they made, and the reasons behind them. Assignments are no longer just things you submit—they become projects you develop in public.

AI isn’t banned from the classroom. Instead, it’s treated as a helpful assistant to learn from. Students share prompts and techniques. They use AI, Google, Stack Overflow, classmates, or any other resources.

But you still need to take responsibility for your work. If you submit or present it, you must explain what the code does, why you made those choices, and how it works. If I need to ask your AI to understand your code, something is wrong. Getting help is fine, but hiding behind that help is not.

You can’t outsource to AI what the whole room watched you build.

A real studio needs students talking out loud together in the room every day. This also helps with another issue that isn’t about AI. Many people say students today are quieter than in the past. While this is mostly based on stories rather than long-term studies, these stories are common and consistent. Faculty on all types of campuses talk about silent classrooms and students who hesitate to speak up, especially since 2020.

Whatever the reason, this silence can be changed, and the solution is the same as for AI challenges: encourage students to participate. Communication is one of the most important skills in any career, including explaining ideas, defending choices, and persuading others in real time. Students don’t develop these skills by just submitting AI-guided work online. When they share their work publicly, it not only prevents AI misuse but also helps them build the skills they need most.

Invert the roles: AI as teacher and assessorWe know the usual pattern: A student asks, AI answers, and the student copies. We’ve tried to invert this. In our new approach, the AI works with the student on a set of topics, engages them in a conversation they must navigate, and ultimately assesses how well they understand the material, which leads to a grade.

This idea has a research background that goes back before ChatGPT. Teachable-agent systems like Betty’s Brain showed that explaining—even to a software agent—forces students to organize their knowledge, make connections clear, and find gaps. Our model uses this insight differently. The student isn’t teaching the bot. Instead, the student is having a conversation with it, learning, discussing, debating, and showing what they understand.

The Vera Molnár chatbot at the University of Texas at AustinHow did we do this? With fairly simple prompt engineering, we created an avatar chatbot of Vera Molnár (1924–2023), a pioneer of algorithmic art. The bot takes on Molnár’s role, drawing students into conversations about randomness, computation, generative art, and creative choices. Her practice sits exactly where creative coding students need to think: between rule and variation, system and choice, computation and visual judgment.

A system prompt sets the topics and types of questions to ask. The bot goes through these with the student, asks for more detail on unclear answers, and keeps following up until there is proof of understanding. At the end, it reviews the conversation against a rubric, giving us a clear record of which ideas the student covered, where they struggled, and how well they improved.

Besides the assessment, which is often accurate, the transcript becomes a different kind of proof, showing what a typical assignment might hide. What did the student notice? What did they misunderstand? Could they connect the concept to the code? Could they defend their choices? Could they revise their explanation when challenged?

When we switch the roles, something surprising appears: the one thing a finished submission can’t show.

A student thinking out loud.

Make understanding performative: Make students performProgramming has never really had a tradition of performance. Musicians have it, painters have it, and dancers have it. Live coding is starting to change that.

Every semester at AET, students from different disciplines stage an algorave together—short for algorithmic rave. Audio sets, projection pieces, game demos, lasers, drones, experience design. The creative coding class brings live visuals into the live-coding tradition: Code is written and modified in real time, the screen is projected, and the audience watches the editor change as the visuals respond to the music other students are playing.

The Department of Arts and Entertainment Technologies’ annual AudioPixel Collider algorave, November 20, 2025, B. Iden Payne Theatre, The University of Texas at AustinNo prerender. No hiding the machinery.

The Live Coding manifesto, written in 2004 by TOPLAP, includes a line that fits every AI-era assessment conversation: “Obscurantism is dangerous. Show us your screens.” This is not just a performance ethic; it’s also an assessment strategy.

A student walks on stage. The projected screen is their editor. The room can read it. The music starts. And they build up a line of code on screen like:

`osc(18, 0.08, 1.2)

.modulate(noise(3), 0.25)

.rotate(() => time * 0.1)

.out()`

This is JavaScript building visuals in real time. FFTs, chained functions, higher-order manipulations. When you’re manipulating code like that on stage, you’d better know what you’re doing.

AI can help you prepare. Good. Let it.

But once you’re on stage, the question shifts from “Can you copy and paste code?” to “Can you control it?” You can paste code into a file, but you can’t paste your way through three minutes of public debugging while the whole projection turns into a beige rectangle. In a live build, understanding has nowhere to hide.

Student livecoding at the Department of Arts and Entertainment Technologies’ annual AudioPixel Collider algorave, November 20, 2025, B. Iden Payne Theatre, The University of Texas at AustinCan you read the code, make changes on purpose, and recover when something unexpected happens? That’s fluency: knowing what to do next while the system is still running.

It is very hard to plagiarize panic.

A note on assessmentSo far, our results are based on our own observations. We haven’t conducted a controlled study or compared different groups, so what we have seen might just be early variation rather than patterns that apply more broadly. For now, these efforts are experiments, not final answers.

Assessment in studio and live performance settings is always subjective and focused on people. It relies on monitoring students’ progress, providing feedback, and observing how they handle challenges. We do not plan to change this core approach.

For the Molnár conversation assignment, students discussed Molnár using an AI system. The AI then created a summary and analysis of each student’s understanding. Teaching assistants reviewed this analysis, conducted their own assessments, and assigned grades. In our small experiments, the AI’s assessments using the rubric matched closely with the teaching assistants’ own evaluations.

We also used AI to help grade the end-of-term coding assignment. In this project, students improved an object-oriented game by adding strategies like heuristics, search algorithms, and learned behaviors. Since our teaching assistants had limited experience with object-oriented programming, we developed a detailed rubric and had an AI model use it to evaluate each submission. The AI’s analysis was given to the teaching assistants as support. It helped them see how each project was structured, spot important OOP design choices, and use the rubric with more confidence. The teaching assistants still made their own grading decisions. I was available as the OOP expert for any questions they could not answer. From what I observed, this substantially helped the teaching assistants understand and grade the students’ OOP design work.

More broadly, both approaches appear to enable substantive feedback at a scale that would otherwise be difficult given our current student-to-teaching-assistant ratios.

The process is the proofWe spent the first two years of the generative AI panic asking how to catch students using AI—or prohibit it altogether. Wrong question.

The real question is whether the assignment gives students a real way to show and develop their understanding. This view isn’t limited to educators. NVIDIA CEO Jensen Huang recently argued that students should not focus on finding an “AI-proof” subject. Instead, he suggested they consider how AI can help them learn more deeply and develop their skills and sense of purpose. He highlighted storytelling, creativity, design, and judgment as abilities that will stay important even as AI takes over more tasks. This supports a key idea in coding education: The aim is not to prove you didn’t use any tools, but to help students show how they think, make choices, revise, and take responsibility for their work.

These three practices are experiments, not universal solutions. They work especially well in creative coding, where code already has a public, visual, and performative aspect. But they suggest a broader principle: As finished work becomes easier to generate, assessment needs to focus more on process, explanation, revision, and mastery.

This matters outside of school too. A polished memo no longer proves there was real thinking behind it. A working prototype no longer proves product sense. A passing pull request no longer proves the developer made the change carefully and thoughtfully. AI makes production easier, so evaluation must focus more on how people think, choose, revise, and recover—in code review, hiring, and performance management. The artifact is no longer the proof. The process is.

Generative AI didn’t make assessment impossible. It just made a hidden weakness obvious. We were putting too much trust in finished work. The arts always knew better.

Show us your screens.

AcknowledgementsThanks to Mike Loukides, Michael Baker, Mk Haley, Elisabeth Robson, and Honoria Starbuck for feedback on this article.

ReferencesOpenAI. “New AI classifier for indicating AI-written text.” OpenAI Blog, January 31, 2023. Updated July 20, 2023, to note the classifier was no longer available due to low accuracy.

Liang, Weixin, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. “GPT detectors are biased against non-native English writers.” Stanford HAI, July 10, 2023.

Winthrop, R. (2026, May 27). Writing with A.I. weakens your creativity. The New York Times.

TOPLAP. “TOPLAP Manifesto.”

Schell, J., Ford, K., & Markman, A. B. (2025). Building responsible AI chatbot platforms in higher education: An evidence-based framework from design to implementation. Frontiers in Education, 10, Article 1604934. https://doi.org/10.3389/feduc.2025.1604934

Biswas, Gautam, Daniel Schwartz, John Bransford, and the Teachable Agents Group at Vanderbilt. “Technology support for complex problem solving: From SAD environments to AI.” In Learning to Solve Complex Scientific Problems, 2001.

Leelawong, Krittaya, and Gautam Biswas. “Designing learning by teaching agents: The Betty’s Brain system.” International Journal of Artificial Intelligence in Education, 2008.

Tan, Huileng. “Jensen Huang Says It Doesn’t Matter What Kids Study in the AI Era.” Business Insider, May 26, 2026. https://www.businessinsider.com/nvidia-jensen-huang-what-kids-should-study-ai-education-advice-2026-5

DAM Digital Art Museum. “Vera Molnár.” Artist biography and timeline.

View Details

The following article originally appeared on Charity Majors’s Substack and is being reposted here with the author’s permission.

A few days back I wrote a piece called “AI enthusiasts are in a race against time, AI skeptics are in a race against entropy.”

I have notes on a whole pile of AI-related topics that I’d like to cover in depth: AI mandates, communication norms, code review, AI art, and more. Unfortunately, I got too many interesting responses to my last piece, and now I have to address those before I can move on to other topics.

There were two types of interesting responses: the first on the technical merits, the second on ethical grounds. I will respond to each of these separately. Let’s take the technical side first, because it’s easier.

Somehow, a subset of readers came away believing I was telling everyone to ditch code review and push their shittiest code straight into production without reading it, right now, tout suite.1

That is not what I am doing. That is not what I think you should do. But I did not pick that example at random, and I will tell you why.

In 2025, the question was whether AI could ever generate “good” codeIt’s easy to forget, but for most of 2025, the idea that AI-generated code was slop and might always be slop was not only a reasonable position to hold, it was the default, mainstream position.2

That question was answered decisively last November. Ever since Opus 4.5 came out, AI has been able to generate code that is approximately as good as that of the median software engineer, at least for common patterns, and much faster and more cheaply. I came out of a book hole and realized this in January, and over the first few months of 2026, it seemed like everyone around me was having a similar realization.

But many saw it coming much sooner.

The popular narrative holds that Opus 4.5 was what changed. But Opus 4.5 was more like the tipping point. Agentic harnesses (the code that wraps the LLM in a loop with tools) became a real thing in mid 2025, with precursors building back to late 2024. Tool use, function calling, MCPs…all of this wave was building over the course of 2025, and crested into real general purpose usability at the end of the year.

That’s what the enthusiasts were trying to tell us last year. Not only “this is coming”, but “this is coming faster than you think.”

As it turns out, they were right.

It was reasonable to be skeptical the first timeAs you may know, I come from the reliability side of the house. The compliment I will pay to myself and my people is that we do not struggle to adapt to new realities. As soon as a problem is real and in front of us, we adjust smoothly, even eagerly, thanks to an unwholesome zest for lapping up disgusting technical messes (and the campfire tales we get to tell later).

The un-compliment I will pay myself and my people is that we sometimes struggle to accept that progress is real, that the continued existence of bugs and edge cases does not diminish the fact that huge swaths of problem space do get more-or-less solved over time, to the point they can be taken for granted by most people.3

The speed at which code went from total crap to “ah damn, that’s not bad” is what I have in the back of my mind, as enthusiasts are telling us that harness engineering and AI validation is real, it’s already here, and it’s getting better astonishingly fast.

Holding out for “I’ll believe it when I see it” was forgivable the first time, but much less so the second time. This is what it feels like to be on the inside of an exponential change curve, turns out.4

What happened in 2025, exactly?I want to pause here and be very clear about what I think is happening. Then I’m going to tell you what specifically I am excited about, and why.

You are under no obligation to join me there. But there are way too many sweeping statements out there right now about “it was never X”—“it was always Y”—“the future belongs to xyzzy” —and I want to be crystal clear how conditional and specific and contextual my claims are.

What happened in 2025 was this: the economics of code production were turned upside down. Instead of being very hard, time-consuming, and expensive to generate code, it became effectively free and instant. Lines of code went from being treasured, reused, cared for and carefully curated, to being disposable and regenerable, practically overnight.

For most of computing history, the primary way people have learned to understand software is by writing the code. Once you’ve achieved some mastery, reading and discussing code gets you most of the way there. (I might argue that software engineers have always relied far too heavily on the code instead of sensemaking the system through observability.)

“The real product of a software team is shared understanding”Many great software engineers hold that true product of every (good) software engineering team has always been a shared understanding of the software we own. That it gets stored as cache state in our fragile little meat brains, frequently flushed to disk, deployed to production, committed to github, but our minds are where meaning has always lived.

Is it any wonder that software has always been such a fiercely collectivist endeavor, exquisitely sensitive to relationship dynamics and manners and questions of fairness and emotional valence? It’s exactly what you’d expect when part of your brain lives in other people’s brains, and your collective interdependence is sky high.

It’s something that I love about this industry. But there’s no denying that minds have been a poor container for certain aspects of the software development model. We are forgetful, distractible, impatient. We are bad at spotting small details, we grow habituated to repetition. Worst of all, the model in our heads diverges massively and perpetually from the world our users interact with.

Anyway, SREs have never quite bought that explanation. To us, it’s clear that the true product of every (good) software engineering team is production.

Only prod is prod. Test in prod, or live a lie.

(This is all backstory. I am getting to the point, I promise.)

Turns out, this is an engineering problem after allWe issued our AI mandate last August.5 I had seen enough to know that this was happening, and it was time to do the responsible thing. Honeycomb is a devtools company, and people come to us to help with hard problems on the forefront of technology. I was all in on AI, but I can’t say I was super excited about it, in my heart of hearts.6

Then I found Chad Fowler’s writings on Phoenix Architectures.

If you don’t know what I’m talking about, you should honestly stop reading my shit right now and go read his. Chad is the guy who coined the term “immutable infrastructure” in 2013. His best-known essay is “Relocating Rigor”, because Martin Fowler7 mentioned it recapping a Thoughtworks meetup on the future of software. I replied with “Production Is Where the Rigor Goes”, complaining that they didn’t talk about production enough.

When I wrote that, I think “Relocating Rigor” was the only piece I had read. But soon I found the rest of it, and after reading two or three essays, it just clicked. I knew exactly what he was talking about. I could predict the rest of what he was going to say. And then, reader…then I got excited.

This has all happened before, and this will all happen againI am going to give you a small sample of Chad quotes, just enough to get the gist. Here’s one from “The Death and Rebirth of Programming.”

Immutable infrastructure. Stateless services. Containers. Blue-green deployments. Infrastructure as code.

These ideas all share a common premise: never fix a running thing. Replace it.

AI pushes this premise beyond infrastructure and into application code itself. When rewriting is cheap, editing in place becomes risky. Mutation accumulates entropy. Replacement resets it.

Another favorite: “The Deletion Test.”

Here’s a simple test you can apply to any software system you work on:

Imagine deleting the entire implementation.

Most engineers experience deletion as existential. Code feels like the thing. It’s what we write, review, version, deploy, and debug. Losing it feels like losing the system itself.

When people say, “We can’t just throw the code away,” what they usually mean is something more precise:

• We don’t know exactly what behavior is required.
• We don’t know which failures are unacceptable.
• We don’t know what invariants must always hold.
• We don’t know how to tell if a new version is correct.
• We don’t know which bugs are intentional fixes for forgotten edge cases.

Those are not code problems. They are evaluation problems.

Code becomes precious when it is the only place knowledge lives.

and,

For most of software history, treating code as durable was reasonable.

We treated code as permanent because the labor to produce it was the bottleneck. Rewriting was expensive. Re-validation was risky. Implementations accumulated meaning over time. Structure, tests, comments, bug fixes, and tribal knowledge fused into something you learned not to disturb.

That made sense when production was the constraint.

When regeneration is easy, code stops being an asset and starts acting as a cache: a materialized view of understanding that is useful while current, disposable when stale.

“A materialized view of understanding that is useful while current, disposable when stale.” I think that might have been the exact line that made it click in my head.

Do you remember the sysadmins?I am just barely old enough that my first job title was “System Administrator.” I was a teenager, working at the university, with root on every machine in the days before they learned they should definitely not do that.8

I lived through the shift from handcrafted server pets to immutable infrastructure cattle. I didn’t really understand what was happening at the time, but I’ve contemplated it a lot in recent years. I wrote this in the final chapter of Observability Engineering, 2nd edition (now available, download here!):

The shift from handcrafted servers to immutable infrastructure taught us that mutability is the sworn enemy of understanding. Any artifact that is edited in place creates drift. Drift is what makes systems impossible to maintain.

Our ability to kill and regenerate infrastructure components is the reason we trust it. At Honeycomb, we kill the oldest Kafka node off via cron every Tuesday. That’s why we are confident in our bootstrapping and balancing processes: everything is repeatable, the data can be regenerated, the commitments live elsewhere.

The fact that we cannot regenerate our code in the same way is a sign that we do not understand it. We do not know which commitments we have made, we do not know which dependencies will break. We find them by breaking them, mostly.

Think of all the years of your working life you have wasted on painful migrations and rewrites. Think of replacing load-bearing legacy code. Think of all the strangler figs.

Lines of code have been doing too much. The code has been the bundled up repository of developer intent, user expectations, implicit and explicit behaviors, the only fossilized composite record we have of bugs gone by. It’s too much!

Lines of code are not the ideal artifact to reviewAnd look at all the domains that have been neglected due to the towering, all-consuming expense of maintaining and mutating lines of code. Where are the artifacts I can review and discuss to understand how our architecture is evolving? Where are our architecture artifacts, period? What if we could discuss and converge on an architecture diagram, and the code could be regenerated from changes to the architecture, instead of the architecture being kinda-sorta inferred from the code?

I am not asserting that all code will eventually be AI-generated to spec, bypassing human understanding. The feasibility of this whole endeavor hangs on the question of what a spec is, or what a spec could be. Anyone who has ever done a painful database migration should have learned some goddamn humility about our ability to extract and formalize users’ expectations in a replayable, automate-able way.

But I think that every step we can take in that direction will be good for us.

The tools to do this don’t exist yet, but many of the ideas do exist. Most come from operations and QA, two domains that software engineering has historically been rather snobbish about.

Those tests and techniques are not about testing for correctness or what ought to be happening, they are about observing and encoding what is happening. Behavioral tests, characterization tests, capture/replay, traffic splitters. Observability (the good kind).

Our brains were not built for validationHaving nondeterministic code in production is finally forcing us to do the things we should have done all along. Instrumenting with traces. Tests and evals in production. Production is not what happens after development is over, production is a stage of development.

Human brains are not good at validation. The nitpickiness, the repetition. This is the worst thing to be clinging to, y’all. There are so many better things for us to want to preserve and assert for ourselves in the production and maintenance of software. We are never going to beat the machine when it comes to validation—we are literally the weakest link!

My money’s on humans for a good long time when it comes to creativity, inspiration, leaps of logic, and a lot of other things, but PLEASE do not rest your killer argument for humans in software on us being the best quality gate. OMG.

Alright. I’m almost done here. Just one more thing.

Nondeterministic systems will require more engineering discipline, not lessI think what many engineers have found so alienating and terrifying about the last two years of AI discourse has been the way so many prominent AI voices appear to be gleefully declaring that software is no longer an engineering problem. “SaaS is dead!” “Making AI great at coding was the strategy that unlocks everything else”, and so on. Even Adam Jacob, one of my dearest friends and someone who is rarely wrong about technology, seems to anticipate a bloodbath of software jobs.9

If 2025 was the year of vibe coding, where AI got as good at generating lines of code as the median software engineer, and the range of possible futures often felt destabilizingly, impossibly wide open, I feel like 2026 is shaping up to be a return to discipline.

The knowledge in our heads is unavailable to AI until we encode it into the system, after all. The returns on those investments will be massive and nonlinear. We might argue that they always would have paid for themselves in the long run. But now every CEO in existence is chomping at the bit to get some of those AI cookies, so let’s give it to them. Discipline first, cookies second.

This is our chance to bring our engineering values to the mainstreamThe share of software engineering teams that work in short, fast feedback loops (the cardinal sign of discipline in my book) is, and always has been, appallingly small. Five percent, maybe? Definitely less than 10%. AI tooling brings this more within reach than ever before. Or it can. It could. The discontinuous returns on investment in engineering discipline are real enough that it just might happen.

I am not worried, at least in the near term, about AI creating massive, discontinuous returns on investment in the absence of engineering discipline. (Many will try, and it will be entertaining to watch.)

But value is backed by durability, not disposability, and I don’t see that changing. Bits are cheap and fast and governed by the rules of logic and language, but anything with value must ultimately resolve with physical systems: persistence on the one side, user experience on the other.

People do not want to wake up every day and log in to Slack and find the buttons and menus all subtly moved around. People do not want financial transactions that complete most of the time. Determinism is not going anywhere, my friends.

AI is not magic. This is still engineering. As Adam says, “it’s still technology, and technology needs technologists.” And I for one am looking forward to learning new and interesting engineering problems, reviewing different kinds of artifacts.

And never doing another sticky, picky, two year long API rewrite or strangler fig migration, ever, ever again.

~charity

P.S. Thanks to everyone who read a draft and gave me feedback: Dave Williams, Chad Fowler, Adam Jacob, Mark Ferlatte, Austin Parker, Erwin van der Koogh.

Footnotes1. I was not trying to be neutral or even-handed in my last piece, only to give a baseline of courtesy to everyone. But I think it’s revealing how many times I was accused of being “so overly hard on skeptics”, by skeptics, and “so overly hard on enthusiasts”, by enthusiasts, and sometimes simply “It’s sad how some people can’t accept reality” with no indication which side they meant. Lord. ︎ 2. Fred Hebert and I gave the closing keynote at SRECon in March of 2025 where we told SREs they should get to know AI, maybe even try vibe coding (pause for laughs), because otherwise their critiques wouldn’t land as well.
Seriously, that was our big pitch. Learn AI so that you can complain more effectively.
︎ 3. Infrastructure, for example. I think this is true of a lot of engineers, btw. I just think it’s really really true of the type of engineer that signs up to be an SRE. Technological pessimism and ADHD, our two most defining traits. ︎ 4. There is a segment of AI enthusiasts who believe we are entering an era of eternal exponential growth, in which the machines begin to build better and better machines, in ways we cannot understand.
I think those people are bad at math. The only thing we know for certain about exponential growth is that it will end. It always does. either in an S curve or a crash. (For a good time, google Heinz van Foerster and “our great-great grandchildren will be squeezed to death.”)
I definitely think we will use machines to build the machines—duh, we already are—but that’s about recursion and specialization. I think the exponential curve we are on the inside of now was created by sloshy free money chasing high returns, plus the properties of software as a function of language and logic, plus the biggest discoveries always happen in the early days of a technology boom, because low hanging fruit gets picked first.
My personal sense—and keep in mind that I am no kind of expert on AI—is that the exponential advancement in AI models leveled out a while ago, and gains are becoming harder to earn and more incremental in nature. I may turn out to be very wrong, of course. But even if there were no more AI innovations moving forwards, the past year has unleashed enough pent-up force to radically reshape the software industry as we know it. Like a pig in a python, we will be dealing with the consequences for a long time to come.
︎ 5. More on this coming EXTREMELY soon. Watch the Honeycomb blog! ︎ 6. The tech is cool, but as a thinking, feeling, breathing human who cares about other people, it can be hard to get excited about anything that so many people are this upset about. It’s also hard to get excited about something when so many of the loudest voices are out there talking gleefully about putting everyone permanently out of work, and so many artists and writers and people from developing nations are talking openly about the impact on them.
Hold your desire to jump in and berate me here, I beg you. Like I said, I will deal with the ethics and morality of using AI in my very next post. Be honest, your attention span is no more up for reading a 10,000-word essay than mine is up for writing one. (Can we blame AI for that too?)
︎ 7. “The Other Fowler.” I gather they’ve been making this joke for like… fifty years. ︎ 8. I share a longer version of this story in the second edition of Observability Engineering, chapter 32, downloadable now!!” ︎ 9. Adam is rarely wrong about technology, and I am 100% sure he is living and working in _a_ future of software engineering. I am less sure it is the future we will all be living in. If the hardest part of software has never been writing code—as is my belief—it logically follows that even if the economics of code production drop to zero, the hard parts will still be hard. ︎

View Details

The model release train continues, with Mistral’s multimodal Pixtral 12B, OpenAI’s o1 models, and Roblox’s model for building 3D scenes. We also have another important AI-enabled programming tool: Cursor is an alternative to GitHub Copilot that’s getting rave reviews.

Security will never cease to be a problem, but this month seems particularly problematic. The Mirai botnet is infecting a widely used surveillance camera that is unpatchable; the only known mitigation is to replace the camera. And attackers are targeting participants in GitHub projects, telling them that their project has vulnerabilities and sending them to a malware site to learn more.

Artificial Intelligence Simon Willison uses the curl utility to discover how streaming APIs for large language models work. * Goldfish loss is a new loss function that language models can use to minimize the “memorization” of long passages during training. Models trained this way would be less likely to output material they were trained on. * OpenAI has put two models into limited (preview) release: OpenAI o1-mini and o1-preview. Both reduce errors and hallucinations by implementing chain-of-thought reasoning. o1-preview spends more effort reasoning through problems before generating a response; o1-mini claims to be a cost-effective model that’s more accurate for scientific reasoning. * Mistral has released Pixtral 12B, its first multimodal model. It allows images to be mixed with text and documents as input. It doesn’t appear to create image output, but it can generate code (e.g., for a website) from a sketch. Chat is available through Mistral’s Le Chat, code and weights through GitHub and Hugging Face. Pixtral is licensed under Apache 2.0 and can be fine-tuned and used without restrictions. * Roblox has created a generative model that builds 3D scenes from text prompts. * Cheating potential aside, connecting a TI-84 graphing calculator to ChatGPT is one of the coolest hacks we’ve seen for quite a while. * Anthropic has announced Claude for Enterprise, which offers larger context windows, GitHub integration, and security features (single sign-on, role-based access, audit logs, and identity management). * As AI becomes better at simulating humans, will we need “personhood credentials” to prove our humanity? (And what if issuing agencies, which might include governments, decide to use personhood credentials as a political tool?) * Chatbots don’t know when to say “help”—a very important point. One important source of errors in AI is the inability to state that it doesn’t know the answer. * OpenAI and Anthropic have agreed to give advanced access to their models to the US Artificial AI Safety Institute so they can be tested for safety. * AIs that can play video games are old hat. Now they can be the game, not just play it. A Google project has built a model that can simulate the 1990s game Doom*, using techniques developed for Stable Diffusion. Could it be used to generate new games, not just emulate older ones? * Google has reenabled the Gemini model’s generation of human images. * Anthropic has enabled cross-origin request sharing (CORS) for the Claude models’ JSON APIs. This change means that applications running in a browser can interact directly with Claude.

Programming* With the addition of preemptive multitasking to the kernel, Linux can now be a true real-time operating system. * Want Lisp implemented in Rust macros? Here it is. * Yet another interesting programming language: Fennel has Lisp-like syntax and macros but integrates with Lua. It’s compiled, and it can be used for embedded systems. * lwIP is a small, lightweight open source IP stack. It’s designed to run on systems with very little memory—for example, small embedded systems. * The European Union is building and deploying a standardized, interoperable digital wallet. * Handoff is a new open source project that allows software developers to use Figma design tokens in code without requiring a Figma license. It helps integrate designers’ work with software development. * Three years after changing its license from open source to business source, Elastic has returned its products, ElasticSearch and Kibana, to an open source license. * The Cursor AI code editor has been getting many excellent reviews. It’s similar to GitHub Copilot but integrated with Claude 3.5 Sonnet. Here’s a good intro. * Check out the new Dynamicland website! Bret Victor’s Dynamicland demonstrates a compelling way of using computers to facilitate collaboration between people and machines. It’s about computing with people in the real world, with real materials. The Dynamicland site hadn’t changed in years; this new site updates the Dynamicland vision. * Microsoft has donated the Mono project, the cross-platform implementation of .Net, to the Wine project. * Valkey, the open source fork of the formerly open source Redis key-value store, is gaining momentum, in part because it’s making improvements that Redis users have wanted. * A new “absurdly fast” algorithm promises to speed up traffic through networks.

Security* A malware campaign called “GitHub Scanner” sends emails to participants in targeted projects claiming that their project has security vulnerabilities. The emails appear to come from GitHub. Victims are asked to visit a site that installs malware. * Microsoft has updated its core cryptographic library, SymCrypt, with postquantum cryptographic algorithms. * WiFi networks, whether in the office, home, or a coffee shop, are surprisingly easy to attack. Here’s some good advice for keeping intruders off of your network. * A security researcher has, somewhat accidentally, purchased the (expired) domain of the former WHOIS server for the .mobi domain. They discovered that certificate authorities (CAs) were still using the old server to verify domain ownership. All has been restored to normal, but do we really need to say again that the IP stack has deep and fundamental vulnerabilities? * RAMBO is a new attack that steals data from air-gapped systems by manipulating RAM in ways that generate decodable radio signals. Another attack against air-gapped systems takes advantage of acoustic noise generated by screen displays. * Attackers are using comments on GitHub to recommend fake fixes that install malware. * Rock and roll will never die. Neither will the Mirai botnet. It’s been spreading among a widely used, unpatchable security camera model. The only mitigation would be for the users of the security camera to replace it, and that’s not likely. * Chaos engineering—a testing technique that randomly introduces faults into a system—has been used to measure a system’s vulnerability to distributed-denial-of-service (DDOS) attacks and assess the system’s response, allowing the operators to mitigate vulnerabilities.

Web* O’Reilly author Holden Karau has developed a platform that uses AI to help people fight health insurance claim denials. The platform helps generate the many letters and forms required to protest a rejection. * Judging from the online chatter on Mastodon and Bluesky, the final XOXO festival was clearly the event not to miss. Videos from 2024 aren’t yet online. * What can you hide in one million checkboxes? A lot of stuff: URLs, images, animations… Here’s a story about creativity, play, subversion, and software that’s worth reading. * The revolt against complex JavaScript frameworks continues. Will there be a PHP revival? * Perhaps it’s not surprising, but GPT is good at unminifying code; the result is quite clear and readable. Minifying means replacing meaningful names with short, meaningless ones and doing other tricks to reduce code size (and obfuscate its meaning). It’s used mostly for JavaScript.

Hardware* Flow computing is a new architecture for general-purpose CPUs that combines cores designed for parallel processing with general-purpose cores. * Cerebras—maker of some of the largest chips in the world—has announced Cerebras Inference, a processor that can do inference at roughly 20 times the speed and one-fifth the cost per token of the NVIDIA H100. At 1,800 tokens per second, Llama 3.1-8B is almost instantaneous.

Quantum Computing* Microsoft claims to have solved a real-world chemistry problem using a hybrid system composed of both classical and quantum processors. * Researchers at Google claim to have created a single logical qubit with reduced error rates. The error rate decreases further as more physical qubits are added to the logical qubit. Scott Aaronson’s discussion is worth reading. * Oak Ridge National Laboratories, which currently has the world’s fastest supercomputer, is experimenting with adding a quantum accelerator.

Robotics* A new design for robot legs focuses on building artificial muscles. Legs incorporating the design appear to be much more capable and efficient than traditional, motor-actuated limbs. * We’re advised to be skeptical of videos showing humanoid robots doing amazing things. There are a lot of tricks: cherry picking, of course, but also off-screen human operators, carefully chosen terrain that isn’t realistic, and many others.

Biology* The Four Thieves Vinegar Collective is making it easier for people to create copies of expensive pharmaceuticals. A hepatitis C drug that costs $1,000 per pill can be made for under $1 per pill.

View Details

AI is everywhere—we’re in a middle of a technology shift that’s as big as (and possibly bigger than) the arrival of the web in the 1990s. Even though ChatGPT appeared almost two years ago, we still feel unprepared: we read that AI will change every job, but we don’t know what that means or how to prepare.

Here are a few ideas about preparing for that shift. The primary thing you can do to prepare for this shift is understand what AI can and can’t do—and in particular, understand what you can do better than AI. It’s frequently said that AI won’t take your job, but people who don’t use AI will lose their jobs to people who do. That’s true as far as it goes (and in a “blame the victim” sense)—but the real truth is that people who can’t add value to what AI can do are the ones who are in danger, whether they use AI or not. If you just reproduce AI results, you’re very replaceable.

How can you partner with AI to deliver better results than either you or AI could on your own? AI isn’t magic. It isn’t some superhuman intelligence, despite the pronouncements of a few billionaires who have a vested interest in convincing you to give up and let AI do everything—or to crawl into a shell because you’re scared of what AI can do. So, here are a few basic ideas about how you can be better than AI.

First, realize that AI is best used an assistant. It can give you a quick first draft of a report—but you can probably improve upon it, even if writing isn’t one of your strengths. Having a starting point is invaluable. AI is very good at telling you how to approach learning something new. It’s very good at summarizing books, podcasts, and videos, particularly if you start by asking it to make an outline, and then use the outline to focus on the parts that are most important. Shortly after ChatGPT was released, someone said that it was like a very eager intern: it can do a lot of stuff fast, but not particularly well. GPT (and the other AI services) have gotten better over the past two years, but that’s still true.

Second, realize that AI isn’t very good at being creative. It can tell you how to do something, but it’s not good at telling you what to do. It’s good at combining ideas that people have already had but not good at breaking new ground.

So, beyond the abstract ideas above, what do you need to know to use AI effectively?

It’s all about writing effective prompts. (“Prompts” implies chat and dialogue, but we’re using it for any kind of interaction, even (especially) if you’re writing software that generates or modifies prompts.) Good prompts can be very long and detailed—the more detailed, the better. An AI isn’t like a human assistant who will get bored if you have to spell out what you want in great detail—for an AI, that’s a good idea.

You have to learn a few basic prompting techniques:

• “Explain it to me like I’m five years old”: A bit hackneyed and perhaps not as useful as it used to be. But it’s worth keeping in mind.
• Chain-of-thought prompts: Ask AI to tell you what steps it will take to solve a problem—then, in a separate prompt, asking it to solve the problem (possibly working step-by-step). Chain-of-thought prompts often include some examples of problems, procedures, and solutions that are done correctly, giving the AI a model to emulate.
• Structured prompts: Tell the AI who it is (“you are an experienced salesperson”), what you want it to do (“who has been asked to write a tutorial on how to close deals”), and who you are (“for a new hire in the sales department”). These prompts can get very long and elaborate, but the extra work pays off in the quality of the response.
• Iterated prompts: Using AI isn’t about asking a question, getting an answer, and moving on. If the answer isn’t quite what you want, modify the prompt—make it better. Tell the AI what’s wrong, give it more context, give it more information about what exactly you want. It won’t get impatient, and your first prompt is rarely your best.
• Include documents: Including documents as part of your prompt is a good way to provide information the AI doesn’t already have. It may reduce hallucination. It’s also a very simple version of RAG, an important technique for building AI applications.

You also have to learn to check whatever output the AI gives you. We’ve all heard of “hallucination”: when an AI gives you output that has no basis in fact. I like to differentiate “hallucination” from simple errors (an incorrect result), but both happen and the distinction is, at best, technical. It’s not clear what causes hallucination, though it’s more likely to occur in situations where the AI can’t come up with an “answer” to a question.

Checking an AI’s response is an important discipline that hasn’t been discussed. It’s often called “critical thinking,” but that’s not right. Critical thinking is about investigating the underpinning of ideas: the assumptions and preconceived notions behind them. Checking an AI is more like being a fact-checker for someone writing an important article:

• Can every fact be traced back to a documentable source?
• Is every reference correct and—even more important—does it actually exist?
• Is the AI’s output too vague or general to be useful?
• Does the AI’s output capture the nuance that you would expect from a human author?

Checking the AI is a strenuous test of your own knowledge. AI might be able to help. Google’s Gemini has an option for checking its output; it will highlight portions of the output and give links that support, refute, or provide (neutral) information about facts it cites. ChatGPT can be induced to do something similar. But it’s important not to rely on the ability of an AI to check itself. All AIs can make subtle errors that are hard to detect; all of the AIs can and will make mistakes checking their output. This is laborious work, but it’s very important to keep a human in the loop. If you trust AI too much, it will eventually be wrong at the most embarrassing and dangerous time possible.

Finally, you have to learn what information you should and shouldn’t give to an AI. How will the AI use the prompts you submit? Most AIs will use that information to train future versions of the model. For most conversations, that’s OK, but be careful about personal or confidential information. Your employer may have a policy on what can and can’t be sent to an AI or on which models have been approved for company use. Some of the models let you control whether they will use your data for training; make sure you know what the options are and that they’re set correctly.

That’s a start at what you need to learn to use AI effectively. There’s a lot more detail—it’s worth taking a few courses, such as those found in O’Reilly’s AI Academy—but this advice will get you started. More than anything else, use AI as an assistant, not as a crutch. Let AI help you be creative, but make sure that it’s your creativity. Don’t just parrot what an AI told you. That’s how to succeed with AI.

View Details

A recent article in Computerworld argued that the output from generative AI systems, like GPT and Gemini, isn’t as good as it used to be. It isn’t the first time I’ve heard this complaint, though I don’t know how widely held that opinion is. But I wonder: Is it correct? And if so, why?

I think a few things are happening in the AI world. First, developers of AI systems are trying to improve the output of their systems. They’re (I would guess) looking more at satisfying enterprise customers who can execute big contracts than catering to individuals paying $20 per month. If I were doing that, I would tune my model toward producing more formal business prose. (That’s not good prose, but it is what it is.) We can say “don’t just paste AI output into your report” as often as we want, but that doesn’t mean people won’t do it—and it does mean that AI developers will try to give them what they want.

AI developers are certainly trying to create models that are more accurate. The error rate has gone down noticeably, though it’s far from zero. But tuning a model for a low error rate probably means limiting its ability to come up with out-of-the-ordinary answers that we think are brilliant, insightful, or surprising. That’s useful. When you reduce the standard deviation, you cut off the tails. The price you pay to minimize hallucinations and other errors is minimizing the correct, “good” outliers. I won’t argue that developers shouldn’t minimize hallucination, but you do have to pay the price.

The “AI blues” has also been attributed to model collapse. I think model collapse will be a real phenomenon—I’ve even done my own very nonscientific experiment—but it’s far too early to see it in the large language models we’re using. They’re not retrained frequently enough, and the amount of AI-generated content in their training data is still relatively very small, especially if their creators are engaged in copyright violation at scale.

However, there’s another possibility that is very human and has nothing to do with the language models themselves. ChatGPT has been around for almost two years. When it came out, we were all amazed at how good it was. One or two people pointed to Samuel Johnson’s prophetic statement from the 18th century: “Sir, ChatGPT’s output is like a dog’s walking on his hind legs. It is not done well; but you are surprised to find it done at all.”1 Well, we were all amazed—errors, hallucinations, and all. We were astonished to find that a computer could actually engage in a conversation—reasonably fluently—even those of us who had tried GPT-2.

But now, it’s almost two years later. We’ve gotten used to ChatGPT and its fellows: Gemini, Claude, Llama, Mistral, and a horde more. We’re starting to use GenAI for real work—and the amazement has worn off. We’re less tolerant of its obsessive wordiness (which may have increased); we don’t find it insightful and original (but we don’t really know if it ever was). While it is possible that the quality of language model output has gotten worse over the past two years, I think the reality is that we have become less forgiving.

I’m sure that there are many who have tested this far more rigorously than I have, but I have run two tests on most language models since the early days:

• Writing a Petrarchan sonnet. (A Petrarchan sonnet has a different rhyme scheme than a Shakespearian sonnet.)
• Implementing a well-known but nontrivial algorithm correctly in Python. (I usually use the Miller-Rabin test for prime numbers.)

The results for both tests are surprisingly similar. Until a few months ago, the major LLMs could not write a Petrarchan sonnet; they could describe a Petrarchan sonnet correctly, but if you asked them to write one, they would botch the rhyme scheme, usually giving you a Shakespearian sonnet instead. They failed even if you included the Petrarchan rhyme scheme in the prompt. They failed even if you tried it in Italian (an experiment one of my colleagues performed). Suddenly, around the time of Claude 3, models learned how to do Petrarch correctly. It gets better: just the other day, I thought I’d try two more difficult poetic forms: the sestina and the villanelle. (Villanelles involve repeating two of the lines in clever ways, in addition to following a rhyme scheme. A sestina requires reusing the same rhyme words.) They could do it! They’re no match for a Provençal troubadour, but they did it!

I got the same results asking the models to produce a program that would implement the Miller-Rabin algorithm to test whether large numbers were prime. When GPT-3 first came out, this was an utter failure: it would generate code that ran without errors, but it would tell me that numbers like 21 were prime. Gemini was the same—though after several tries, it ungraciously blamed the problem on Python’s libraries for computation with large numbers. (I gather it doesn’t like users who say, “Sorry, that’s wrong again. What are you doing that’s incorrect?”) Now they implement the algorithm correctly—at least the last time I tried. (Your mileage may vary.)

My success doesn’t mean that there’s no room for frustration. I’ve asked ChatGPT how to improve programs that worked correctly but that had known problems. In some cases, I knew the problem and the solution; in some cases, I understood the problem but not how to fix it. The first time you try that, you’ll probably be impressed: while “put more of the program into functions and use more descriptive variable names” may not be what you’re looking for, it’s never bad advice. By the second or third time, though, you’ll realize that you’re always getting similar advice and, while few people would disagree, that advice isn’t really insightful. “Surprised to find it done at all” decayed quickly to “it is not done well.”

This experience probably reflects a fundamental limitation of language models. After all, they aren’t “intelligent” as such. Until we know otherwise, they’re just predicting what should come next based on analysis of the training data. How much of the code in GitHub or on Stack Overflow really demonstrates good coding practices? How much of it is rather pedestrian, like my own code? I’d bet the latter group dominates—and that’s what’s reflected in an LLM’s output. Thinking back to Johnson’s dog, I am indeed surprised to find it done at all, though perhaps not for the reason most people would expect. Clearly, there is a lot on the internet that is not wrong. But there’s a lot that isn’t as good as it could be, and that should surprise no one. What’s unfortunate is that the volume of “pretty good, but not as good as it could be” content tends to dominate a language model’s output.

That’s the big issue facing language model developers. How do we get answers that are insightful, delightful, and better than the average of what’s out there on the internet? The initial surprise is gone and AI is being judged on its merits. Will AI continue to deliver on its promise, or will we just say, “That’s dull, boring AI,” even as its output creeps into every aspect of our lives? There may be some truth to the idea that we’re trading off delightful answers in favor of reliable answers, and that’s not a bad thing. But we need delight and insight too. How will AI deliver that?

FootnotesFrom Boswell’s Life of Johnson (1791); possibly slightly modified.

View Details

This month, we’ll give AI a rest. Alex Russell has finished an excellent series of posts titled, “Reckoning.” It’s a must-read for web developers. If you want to understand why our networks and laptops are much faster than they were 15-20 years ago, but the web is slower, it comes down to one thing: bloated JavaScript. Alex offers a way out.

The last XOXO festival is over, and I regret having missed it. XOXO has been a gathering place for experimental artists and others who live and work online. Judging from the many comments on Mastodon, Bluesky, and Twitter, it was great. A key theme was building networks that are healthy, and where people are respected. Look for the videos when they’re posted—I will certainly have them in next month’s trends.

And we have a rare legal section with items on AI regulation, Telegram, and open source licenses.

AI* Anthropic has published the system prompts for its Claude models. It’s interesting that Claude prefers short responses; it avoids words like “Certainly”; and that it can break a longer task into parts, executing each part separately. * Many developers report huge time savings when using generative AI to understand or update legacy code. Andy Jassy, Amazon’s CEO, has claimed that they saved 4,500 developer-years by using AI to upgrade 30,000 Java applications from Java 8 to Java 17. * Generative AI can be used to improve data quality by automating the process of tagging data with metadata. * Andrew Ng has released a model that lets you experiment with geoengineering by simulating the injection of reflective aerosols into the atmosphere. * Alibaba has released two new models, Qwen2-Math and Qwen2-Audio, which are (not surprisingly) designed for working with math and audio. We’d like to see the Math model in a matchup against DeepMind’s AlphaProof. * The Open Source Initiative (OSI) has released version 0.0.9 of their definition of Open Source AI. Their definition requires that training data be recognized as part of an open source system. A stable version of the definition will be announced in October 2024. * Google has released Imagen 3, their state-of-the-art text-to-image model. You can try it on ImageFX or Vertex. * Ideogram 2 is another impressive text-to-image model. There’s an iOS app and an AI. “Magic prompting” lets you upload an image and generate a description, then use the description as the basis for future prompts. That’s very useful, though it could be the basis for copyright trouble. * Claude now has user-controllable prompt caching: using the API, users can mark portions of long prompts (including data) to be cached. Cached items don’t need to be recomputed again. Prompts that re-use data in the cache cost much less to run. * The AI Scientist, an AI system designed to do autonomous scientific research, unexpectedly modified its own code to give it more time to run. * Nick Hobbs argues that we need AI designers—designers who specialize in designing for AI, who are intimately familiar with AI and its capabilities—to create genuinely innovative new products. * Simon Willison’s discussion of GPT-4o’s system card is fascinating. Among other things, they tested whether it could accomplish Capture the Flag challenges against a Linux system. It failed (mostly). * DeepMind has developed a robotic, AI-driven ping pong player. It is good, but not yet a threat to advanced human players. * The GPT API now features Structured Outputs, which causes it to deliver results according to a user-specified JSON schema. In addition to simplifying software development, the constraint that GPT can only generate output tokens that fit the schema improves its accuracy. * For about a year, OpenAI has had a watermarking system for GPT that can detect whether a text was written by their AI. They have not deployed it. It is apparently easy to defeat (by rewriting text with another LLM); they also feel it would make using GPT less attractive. * Flux.1 is a new family of open-weight text-to-image models that claims state-of-the-art performance. * Whisper-Medusa is a speech-to-text model that claims to be twice as fast as OpenAI’s Whisper with equivalent accuracy. * Google has released Gemma 2 2B, the smallest in its Gemma family of small language models. Google has also released ShieldGemma, a specialized model for detecting harmful inputs and outputs, and Gemma Scope, a set of sparse autoencoders designed to help researchers understand what happens inside language models. * OpenAI has finally released the voice-enabled ChatGPT bot to a limited group of ChatGPT+ subscribers. General release to all subscribers should take place this fall. The feature was announced in May but held for further work on safety. * OpenAI has announced SearchGPT, an AI-based search engine. It isn’t yet available, but you can sign up on a waiting list. * Angels and Cows is a sci-fi/fantasy novel that was written to help people think about applying artificial intelligence to social justice issues. It is an attempt at a new kind of scientific communications. A free PDF is available. * Spreadsheets are difficult for language models; their structure doesn’t lend itself to encoding for processing by LLMs. Microsoft has developed a SpreadsheetLLM that has been designed for working with spreadsheets.

Programming* If you like two-dimensional visual programming, you might like Turnstyle, a new esoteric language. It’s challenging, weird, and fun (if you like that kind of fun). * The new release of Istio, a part of the Kubernetes ecosystem, no longer requires sidecars to manage networking and security for each service. The new alternative is called “ambient mesh” and is based on a zero-trust tunnel. Is this actually simpler and faster? We’ll find out. * For the command line users, here’s a modern Unix tool list. Most of the items are rethinkings of familiar tools: bat is cat with syntax highlighting, atuin is shell history backed by a database. It’s definitely worth exploring. * If you don’t understand why things don’t always add up when you use floating point numbers—and in particular, if you don’t understand why you should never use floating point when working with money—you should read the Floating Point Guide. * AudioFlux is a library of tools for analyzing music and other kinds of audio. * Postgres.new gives you the PGlite database running in a browser sandbox with generative AI support. PGlite is an implementation of Postgres in WebAssembly. The developers promise that, soon, it will be possible to deploy to AWS S3. * A new DARPA program, Translating All C to Rust (TRACTOR), intends to develop AI tools for making legacy C code memory-safe by translating it to Rust. While such translation is already possible, it is error-prone. TRACTOR aims to improve the quality and reliability of translations. * The WhenFS turns your Google Calendar into a mountable filesystem. Nobody really knows why would you want to do this. File this in the same mental space as esoteric programming languages. * “age is a simple, modern and secure file encryption tool, format, and Go library.” The GitHub repo says it all. Don’t capitalize the a. age can generate keys for ssh, and can interoperate with Yubikey. * Netflix has open sourced Maestro, a data services engine. Maestro supports large scale data workflows, like data pipelines. * AWS is quietly deprecating services that have insufficient usage, of which CodeCommit is the most notable. They are not adding new users, nor will they be updating the services, except for security patches. This is a significant change in AWS’s policies. * Score is a new tool for infrastructure-centric development. It allows developers to write software while reducing the need to understand deployment. It ensures that new code is compatible with infrastructure—for example, managing the transition from Docker to Kubernetes. * Chattr is a Linux command utility that lets you manipulate file permissions in ways that go beyond the standard Unix chmod. Among other things, it can prevent accidentally overwriting files. * Orca is a runtime environment for WebAssembly (Wasm) that doesn’t require the web. It allows you to write Wasm applications that run on a laptop.

Security* Another step towards the end of passwords? Docker is deprecating the use of passwords. In September, they will require single sign-on (SSO) and personal access tokens (PATs) for command line access. * Palo Alto Networks’ Ransomware Review for the first half of 2024 shows that the increase in ransomware attacks it reported earlier in 2024 wasn’t a fluke; they have continued. * A talk at Defcon 2024, Living off Microsoft Copilot, demonstrated a number of exploits against Copilot. They included remote code execution, data exfiltration, and phish generation. * The AI Risk Repository includes a database of known AI risks, a taxonomy of causes of AI risk, and a taxonomy of the risks themselves. It is free to copy and use. * A talk at Black Hat has demonstrated that Microsoft Copilot bots are insecure because of inadequate defaults and other problems that leave them open to attack. Many of these issues have been fixed since the talk, but changes only affect new installations. * Google has announced that it will prompt users who have enabled enhanced safe browsing to give it the passwords needed to decode password-protected files so the files can be examined. Password-protected files are often used to deliver malware.

Web Who is watching you? The Surveillance Watch project has published a map of companies engaged in data collection and surveillance, their affiliates, their targets, and more. It’s an impressive map of the surveillance industry. * Google is rolling out an experimental feature in Chrome for Android that will redact credit card data, passwords, and other sensitive information when a screen is shared. * Reckoning*, Alex Russell’s four part series on JavaScript Framework Culture and the damage it’s done to the web is a must-read. Russell recommends the gov.uk Service Manual, particularly their page about progressive enhancement, as a guide to building responsive services. * Project Wallace looks like an excellent set of analysis tools for web developers working with CSS. * Google is making it easier for victims to request removal of non-consensual deep fake images. Their search algorithm is also being modified to downrank deep fake images.

Hardware* Scientists at Peking University have developed a highly efficient tensor processing unit (TPU) based on carbon nanotubes. * Looking Glass has a 3D holographic display the size of a cell phone at a reasonable ($299) price, in addition to laptop- and monitor-sized models. It can display 3D images from Apple’s Vision Pro. Can we imagine VR without headsets? * Wireless bicycle shifters? If you have them (and some high-end bicycles do), they can be hacked. That could become an issue if you’re a racer. * NVIDIA may have competition. Groq, a startup that makes language processing units (LPUs), has hardware and cloud offerings that look very competitive for doing inference with large language models. * Privacy Lens is a new privacy-enhancing camera with two lenses: a traditional lens and a thermal lens. It uses the thermal lens to turn human images into stick figures with identifying information (e.g., skin color, hair) removed.

Law* OpenAI and Anthropic have agreed to give advanced access to their models to the US Artificial AI Safety Institute so they can be tested for safety. * The California state legislature has passed a bill regulating AI development. The bill is intended to defend against threats to public safety and requires implementation of a “kill switch.” * Pavel Durov, the founder of the Telegram messaging app, has been arrested and indicted in France. Charges include spreading images of child abuse and drug trafficking using the app. * A court has dismissed DMCA-related portions of the plaintiff’s complaint in a landmark lawsuit involving the use of open source code by generative AI. Claims about breach of contract by open source license violations are still in play.

Biology* The GROVER project attempts to treat DNA as a language, and uses AI to decode that language.

View Details

Platform engineering is the latest buzzword in IT operations. And like all other buzzwords, it’s in danger of becoming meaningless—in danger of meaning whatever some company with a “platform engineering” product wants to sell. We’ve seen that happen to too many useful concepts: Edge computing meant everything from caches at a cloud provider’s data center to cell phones to unattended data collection nodes on remote islands. DevOps meant, well, whatever anyone wanted. Culture? Job title? A specialized group within IT?

We don’t want that to happen to platform engineering. IT operations at scale is too important to leave to chance. In her forthcoming book Platform Engineering, Camille Fournier notes that platform engineering has been used to mean anything from an ops team wiki to dashboards to APIs to container orchestration with Kubernetes. All of these have some bearing on platform engineering. But none of them are platform engineering. Taken together, they sound like the story of blind men describing an elephant: one grabs hold of a tusk, another the tail, another a leg, but none of them have a picture of the whole. Camille offers a holistic definition of platform engineering: “a product approach to developing internal platforms that create leverage by abstracting away complexity, being operated to provide reliable and scalable foundations, and by enabling application engineers to focus on delivering great products and user experiences.” (Emphasis Camille’s.)

That sounds abstract, but it’s both precise and helpful. “A product approach” is a theme that comes up repeatedly in discussions of platform engineering: treating the platform as a product and software developers—the users of the platform—as customers, and building with the customer’s needs in mind. There’s been a lot of talk about the death of DevOps; there was even a brief NoOps movement. But as Charity Majors pointed out at PlatformCon 2023, the reality of operations engineering is that it has become fantastically complex. The time when “operations” meant racking a few servers and installing Apache and MySQL is long gone. While cloud providers have taken over the racking, stacking, and software installation, they now offer scores of services, each of which has to be configured correctly. Applications have grown more complex too: we now have fleets of microservices operating asynchronously across hundreds or thousands of cloud instances. And as applications have become more complex, so has operations. It’s been years since operations meant mumbling magical incantations into server consoles. That’s not repeatable; that’s not scalable; that’s not reliable. Unfortunately, we’ve ended up with a different problem: modern software systems can only be operated by the developers who created them.

The problem is that software engineers want to do what software engineers do best, and that’s write cool new applications. They don’t want to become experts in the details of hosted Kubernetes, complex rules for identity, authentication, and access management (IAM), monitoring and observability, or any of the other tasks that have become part of their workspace. What’s needed is a new set of abstractions that allows both developers and operations staff to move to a higher level.

That gets to the heart of platform engineering: abstracting away complexity (in Camille’s words) or making developers more effective (in Charity’s). How do we develop software in the 21st century? Can improved tooling make developers more effective by working around productivity roadblocks? Can we let operations staff worry about issues like service-level agreements (SLAs) and uptime? Can operations staff take care of complex issues like load balancing, business continuity, and failover, which the applications developers use through a set of well-designed abstractions? That’s the challenge of platform engineering. Developers have enough complexity to worry about without taking on operations.

EvolutionThe fantasy of platform engineering is “one-click deployment”: write your application and click on a “deployment” item in your control panel, and the application moves smoothly and painlessly through testing, integration, and deployment. Life is almost never that simple. Deployment itself isn’t a simple concept, what with canary deployments, A/B testing, rollbacks, and so on.

But there is a reality, and behind that reality are some real successes. Facebook used to talk about requiring new hires to deploy something to its site on their first day at work. This predates “platform engineering,” “developer platforms,” and all of that, but it clearly shows that abstractions that simplify software deployment in a complex environment aren’t new.

Writing about his experience at LinkedIn in 2011, Kevin Scott (now CTO of Microsoft) describes how the company found itself in a huge developmental mess just as it went public. It was almost impossible to deploy new features: several years as a startup that was moving fast and breaking things had resulted in a tangled web of conflicting processes and technical debt. “Automate all the things” was a powerful slogan—but as attractive as that sounds, it has a very real downside. LinkedIn took the bold step of halting new development for as long as it took to build a consistent platform for deploying software. It ended up taking several months (and put several careers on the line, including Scott’s), but it was ultimately a success. LinkedIn went from releasing new features once a month, if that, to being able to release several times a day.

What’s particularly interesting about this story is that, writing several years after the fact, Scott uses none of the language that we now associate with “platform engineering.” He doesn’t talk about developer experience, internal developer platform, or any of that. But what his team clearly accomplished was platform engineering of the highest order—and that probably saved LinkedIn because, despite its highly successful IPO, a web startup that can’t deploy is dead in the water.

Walmart has a similar story about improving its DevOps and CI/CD practices. Daily deployment exposed problems in tools, procedures, and processes. These problems were addressed by a DevOps team and were forwarded to a platform team. Like the events recounted above, the work took place in the 2010s. Also like Scott’s LinkedIn story, Walmart’s narrative doesn’t use the language that we now associate with platform engineering.

The Heroku platform as a service is another example of platform engineering’s prehistory. Heroku, which made its debut in 2007, made single-click deployment a reality, at least for simple applications. When programming with Heroku, you didn’t need to know anything about the cloud and very little about how to wire the database to your application. Almost everything was taken care of for you. While Heroku never went quite far enough, it gave web developers a taste of what might be possible.

All of these examples make it clear that platform engineering isn’t anything new. What we now call “platform engineering” consolidates practices that have been around for some time; it’s the natural evolution of movements like DevOps, infrastructure as code, and even the scripting of common maintenance tasks. Whether they’re “software developers” as such or operations staff, people in the software industry have always built tools to make their jobs easier. Platform engineering puts this tool-building on a more rigorous and formal basis: it recognizes that building tools and creating abstractions for complex processes is engineering, not hacking. LinkedIn’s problem wasn’t a lack of tooling. It was several years of wildcat tool development and ad hoc solutions that eventually turned into a mass of seething bits and choked out progress. The solution was doing a better job of engineering the company’s tooling to build a consistent and coordinated platform.

In “DevOps Isn’t Dead, But It’s Not in Great Health Either,” Steven Vaughan-Nichols argues that DevOps may not be delivering: only 14% of companies can get software into production in a day and only 9% can deploy multiple times per day. To some extent, this is no doubt because many organizations that claim to have adopted DevOps, CI/CD, and similar ideas never really change their practices or their culture; they rename existing practices without changing anything substantial. But it’s also true that software deployment has become more complex and that, as LinkedIn learned, undisciplined tool development can result in a mountain of technical debt. Architectural styles like microservices decompose large monoliths into smaller services—but then the correct configuration and deployment of those services becomes a new bottleneck, a new nucleus around which technical debt can accumulate.

The list of problems that platform engineering should solve for software developers gets long quickly. It contains everything from smoothing the path from the developer’s laptop to a source control repository to deploying software to the cloud in production. The more you look, the more tasks to simplify you’ll find. Many security problems result from incorrectly configured identity, authorization, and access management (IAM). Can IAM be simplified in a way that prevents errors? When AWS first appeared, we were all amazed at how simple it was to spin up virtual instances and store data. But provisioning a service that uses dozens of available services and runs across thousands of instances, some in the cloud and some on-premises, is far from simple. Getting it wrong can lead to a nightmare for performance and scaling. Can the burden of correctly provisioning infrastructure be minimized? Deployment isn’t just pushing something to a server or even a fleet of servers; it may include canary deployments, A/B testing, and rollback capabilities. Can these complex deployment scenarios be simplified? Any deployment needs to take scaling into account; if software can’t take into account the company’s current and near-term needs, it’s in trouble. Can a platform incorporate practices that simplify scalability? Failover and business continuity in the event of outages, minimizing cost by optimizing the size of the server fleet, regulatory compliance—these are all issues that are important in the 2020s and that, if we’re being honest, we really didn’t think much about 20 years ago. Do developers need to worry about failover, or can it be part of the platform?

EngineeringThe key word in platform engineering isn’t “platform”; it’s “engineering.” Solid engineering is needed to move up the abstraction ladder, as Yevgeniy Brikman has said. But what does that mean?

Definitions of platform engineering frequently talk about treating the developer as a customer. That can feel very weird when you think (or read) about it. Your company already has “customers.” Are your engineers “customers” too? But that shift in mindset from treating software developers as a labor asset to customers is crucial. Camille Fournier means the same thing when she writes about “a product approach to developing internal platforms”: a platform engineering team has to take its customers seriously, has to understand what the customers’ problems are, and has to come up with effective solutions to those problems.

Platform engineering has the same pitfalls as other kinds of product development. It’s important to build for the customer, not for the engineer designing the product. Techno-solutionism—thinking that all problems can be solved by applying state-of-the-art technology—usually degenerates into implementing ideas because they’re cool, not because they’re appropriate. It almost always imposes solutions from outside the problem space, forcing one group’s ideas on customers without thinking adequately about the customers’ needs. It’s poor engineering. Good engineering may require sitting in the customer’s chair and performing their tasks often enough to get a good feel for their real requirements. Domain-driven design (DDD) is a good tool for flushing out customers’ needs; DDD stresses doing in-depth research to understand product requirements and doesn’t assume that every group within an organization has the same requirements. An organization may be represented by a number of bounded contexts, each of which has its own requirements and each of which needs to be considered in engineering a developer platform. One-size-fits-all solutions usually fail. It’s also a mistake to assume that a developer platform should solve all of the developers’ problems. Getting to 80% may be all you can do; the old 80/20 rule is still a good rule of thumb.

Platform engineering is necessarily opinionated: platform engineers need to develop ideas about how software development workflows should be handled. But it’s also important to understand the limits of “opinionated software.” David Heinemeier Hansson (DHH) popularized the idea of “opinionated software” with Ruby on Rails, which implemented his ideas about what kinds of support a web platform should provide. Were DHH’s opinions correct? That’s the wrong question. DHH’s opinions allowed Rails to thrive, but that’s only platform engineering within the context of DHH’s company, 37 Signals. Rails’ success among web developers would have meant little if it wasn’t accepted by 37 Signals–regardless of how successful it was outside. Likewise, if the software developers at your company choose not to use the platform you develop, it has failed–no matter how good your opinions may be. If the platform imposes rules and procedures that aren’t natural to the platform’s users, it will fail. Opinionated software has to recognize that there are many ways to solve a problem and that users are always free to reject the software that you build. The users’ opinions are more important than the platform engineers’. Writing about site reliability engineering, Laura Nolan discusses the importance of the Greek concept metis: local, specific, practical, and experiential knowledge. Platform engineering must take that local knowledge into account–without getting stuck by “we’ve always done it that way.” Listening to the platform’s eventual users is key; that’s how you develop a coherent product focus.

Platform engineering is necessarily an attempt to impose some kind of order on a chaotic situation—that’s the lesson LinkedIn learned. But it’s also important to recognize, as Camille Fournier said in conversation, that there’s always chaos. We may not like to admit it, but software development is inherently a chaotic process. What happens when one company acquires another company that has its own developer platform? How do you reconcile the two, or should you even try? What happens when different groups in a company develop different processes for managing their problems? Domain-driven design’s concept of “bounded context” can help here. Some unification is probably necessary, but complete unification would almost certainly require a huge expense of time and effort, in addition to alienating a lot of developers. Imposing structure under the guise of “being opinionated” is a path to failure for a software platform. Platform engineers need to develop a product that their users want, not one that their users will fight. Again, good engineering requires listening to the customers. They may not know what they need, but their experience is the ground truth that a platform engineer has to work from.

Platform engineers also need to think carefully about “paved paths.” The term “paved paths” (often called “golden paths”) shows up frequently in the platform engineering literature. A paved path is a process that has been smoothed out, regularized, made easy by the platform. It’s common wisdom to pave the simplest and most frequently used paths first; after all, this makes it look like you’re accomplishing a lot and have good coverage. But is this the best way to look at the problem? Software developers probably already have tools and processes for managing the simplest and most commonly used paths (which aren’t necessarily the same). The right question to ask is where platform engineering can make the biggest difference. Given that the goal is to reduce the burden of complexity, what processes are the biggest problem? What solution would most reduce the developers’ burden of complexity? The best approach probably isn’t to reinvent solutions to problems that have already been solved—that can come later, if it’s necessary at all. Instead, it may be worthwhile to fit older solutions into a new framework. What problems get in developers’ way? That’s where to start.

By now, it should be obvious that, while platform engineering is about product development, it isn’t about a product like Excel or GitHub. It’s not about building a one-size-fits-all platform that can be packaged and marketed to different organizations. Each company has its own context, as does each group within a company. Each has its own requirements, its own culture, its own rules, and those must be observed—or if they must be changed, they must be changed very carefully. Engineering is always about making compromises, and frequently the most appropriate solution is the least worst, as Neal Ford has said. This is where domain-driven design, with its understanding of bounded context, can be very helpful. A platform engineer must discover the rules and requirements that aren’t stated, as well as the ones that are.

And now with AI? Sure. There’s no reason not to incorporate AI into engineering platforms. But there’s little here that requires AI. It’s likely that AI could be used effectively to analyze a project and estimate infrastructure requirements. It’s possible that AI could be used to help with code review—though the final word on code review needs to be human. There are many other possible applications. AI’s biggest value might not be making suggestions about ways to smooth various pathways but in the design process behind the platform. It’s possible that AI could analyze and summarize current practices and suggest better abstractions. It’s less likely than humans to be stuck in the trap of “the way we’ve always done it.” But humans have to remain in the loop at all times. As with software architecture, the hard work of platform engineering is understanding human processes. Gathering information about processes, understanding the reasoning behind them, and coming to grips with the history, the economics, and the politics still requires human judgment. It’s not something that AI is good at yet. Will we see increased use of AI in platform engineering? Almost certainly. But whatever you do or don’t do with AI, please don’t do it merely for buzzword compliance. AI will have a place. Find it.

That’s one side of the coin. The other side is that companies are investing in building applications that incorporate AI. It’s easy to assume that software incorporating AI isn’t much different from traditional applications, but that’s a mistake. Platform engineering is all about managing complexity, and incorporating AI in an application will inevitably increase complexity. Accommodating AI will certainly stress our ideas about continuous delivery: What does automated testing mean when a model’s output is stochastic, not deterministic? What does CD mean when evaluating an application’s fitness may take much longer than developing it? Platform engineering will need a role in testing and evaluation of AI models. There will need to be tools to detect when an application is being abused or delivering inappropriate results. Models need to be monitored so they can be retrained when they grow stale. And there will be new options for managing the cost of deploying AI applications. How do you help manage that complexity? Platform engineers will need to take all of this, and more, into account. A platform that only solves yesterday’s problems is an obstruction.

So what does a platform engineer engineer? Is it a surprise to say that what a platform engineer builds depends on the situation? A developer dashboard for deploying and other tasks might be part of a solution. It’s hard to imagine a platform engineering project in which an API isn’t part of the solution. A DevOps wiki might even be part of a solution, though standing up a wiki hardly requires engineering. Collecting a company’s collective wisdom and lore about building projects might help platform engineers to work toward a better solution. But it’s important not to point to any of these things and say “This is it—building that is platform engineering.” Focusing on any single thing tends to attract platform engineering teams to the latest fad. Does this repeat the history of DevOps, which was hampered by its refusal to define itself? No. Platform engineering is ultimately engineering. And that engineering must take into account the entire process, starting with gathering requirements, understanding how software developers work, learning where complexity becomes burdensome, and finding what paths are most in need of paving. It proceeds to building a solution—a solution that is, by definition, never finished. There will always be new paths to pave, new kinds of complexity to abstract. Platform engineering is an ongoing process.

JustificationWhy are you doing platform engineering? How do you justify it to senior management? And how do you justify it to the software developers that you’re serving?

We hope that justifying platform engineering to software developers is easy—but that isn’t guaranteed. You’re most likely to succeed with software developers if they feel like they’ve been listened to and that you’re not imposing a set of opinions on them. Developers have insight into the problems they face; take advantage of it. Engineering solutions that reduce the burden of complexity are the key to success. If you’re succeeding, you should be seeing deployments increase; you should be seeing less frustration; and you should see metrics for developer productivity headed in the right direction. On the other hand, if a platform engineering solution just becomes one more thing for software developers to work around, it has failed. It doesn’t need to solve all problems initially, but a quick minimum viable product will go a long way to convincing developers that a platform has value.

Justifying platform engineering to management is a different proposition. It’s easy to look at a platform engineering team and ask, “Why does this exist? What’s the ROI? Why am I paying expensive engineers to create something that doesn’t contribute directly to the product we sell?”

The first part of the answer is simple. Platform engineering isn’t anything new. It’s the next stage in the evolution of operations, and operations has been a cost center since the start of computing. In the long arc of computing history, we’ve been evolving from a large number of operators watching over a single computer (a 1960s mainframe required a significant staff and had less computational ability and storage than a Raspberry Pi) to a small number of operators responsible for thousands of virtual machines or instances running in the cloud. Platform engineering done well is the next stage in that evolution, allowing the staff to operate even larger and more complex systems. It’s not additive, something new that has to be implemented and resourced. It’s doing what you’re already doing but better.

If senior management thinks that platform engineering doesn’t contribute directly to the product, they need to be educated in what it means to ship a software product. They need to understand that there is no product without deployment, without testing, without provisioning infrastructure. Doing this infrastructure work more efficiently and effectively contributes directly to the product. A product that can’t be deployed—or where deployments take months rather than hours—is dead in the water.

But that argument isn’t really convincing without metrics. Go back to the business problem you’re trying to solve. Do you want to increase the rate at which you release software? Document that. Are you trying to make it easier to add features or fixes without a full redeployment? Document that. Are you trying to decrease the time between a bug report and a bug fix? Document that. Programmers often think that software is self-justifying. It isn’t. It’s important to keep your eyes on the business goals and how the platform is affecting them.

The DORA metrics are a good way to show the need for better processes, along with measuring whether platform engineering is making processes more efficient. Can you demonstrate that platform engineering efforts are enabling you to get features and bug fixes into your company’s product and out to customers more quickly? Can a platform engineering effort help the company use cloud services more efficiently by avoiding duplication and oversubscription? Can you measure the amount of time developers spend on new features or fixes, as opposed to infrastructure tasks? In his PlatformCon 24 talk, Manuel Pais suggests measuring the percentage of the company’s income that’s supported by the platform. That exercise shows how important the platform is to the company. Platforms do generate value, but platform engineers frequently don’t make the effort to quantify that value when they talk to management. Once you know the value of the platform, it’s possible to forecast how the platform’s value increases over time. A platform is a strategic asset, not just a sunk cost.

The Next LevelMost companies already have a developer platform, whether it’s a bunch of old shell scripts, an unmaintained wiki, or a highly engineered set of tools for continuous integration and deployment. These platforms don’t all deliver the same kind of value—they may not deliver any value at all. The reality is that no company can exist for long without deploying software, and no company can develop software if its developer team is spending all their time chasing down infrastructure problems.

The platform is already there. Whether it’s working for or against you is a different question. Treating your engineering teams as customers and building a product that satisfies their needs is hard, important work. It means understanding their problems as they see them. It means coming up with new abstractions that hide complexity. And in the end, it means making it easier to deploy software successfully at scale. That’s platform engineering.

View Details

Over the years, many of us have become accustomed to letting computers do our thinking for us. “That’s what the computer says” is a refrain in many bad customer service interactions. “That’s what the data says” is a variation—“the data” doesn’t say much if you don’t know how it was collected and how the data analysis was performed. “That’s what GPS says”—well, GPS is usually right, but I have seen GPS systems tell me to go the wrong way down a one-way street. And I’ve heard (from a friend who fixes boats) about boat owners who ran aground because that’s what their GPS told them to do.

In many ways, we’ve come to think of computers and computing systems as oracles. That’s an even greater temptation now that we have generative AI: ask a question and you’ll get an answer. Maybe it will be a good answer. Maybe it will be a hallucination. Who knows? Whether you get facts or hallucinations, the AI’s response will certainly be confident and authoritative. It’s very good at that.

It’s time that we stopped listening to oracles—human or otherwise—and started thinking for ourselves. I’m not an AI skeptic; generative AI is great at helping to generate ideas, summarizing, finding new information, and a lot more. I am concerned about what happens when humans relegate thinking to something else, whether or not it’s a machine. If you use generative AI to help you think, so much the better; but if you’re just repeating what the AI told you, you’re probably losing your ability to think independently. Like your muscles, your brain degrades when it isn’t used. We’ve heard that “People won’t lose their jobs to AI, but people who don’t use AI will lose their jobs to people who do.” Fair enough—but there’s a deeper point. People who just repeat what generative AI tells them, without understanding the answer, without thinking through the answer and making it their own, aren’t doing anything an AI can’t do. They are replaceable. They will lose their jobs to someone who can bring insights that go beyond what an AI can do.

It’s easy to succumb to “AI is smarter than me,” “this is AGI” thinking. Maybe it is, but I still think that AI is best at showing us what intelligence is not. Intelligence isn’t the ability to win Go games, even if you beat champions. (In fact, humans have discovered vulnerabilities in AlphaGo that let beginners defeat it.) It’s not the ability to create new art works—we always need new art, but don’t need more Van Goghs, Mondrians, or even computer-generated Rutkowskis. (What AI means for Rutkowski’s business model is an interesting legal question, but Van Gogh certainly isn’t feeling any pressure.) It took Rutkowski to decide what it meant to create his artwork, just as it did Van Gogh and Mondrian. AI’s ability to imitate it is technically interesting, but really doesn’t say anything about creativity. AI’s ability to create new kinds of artwork under the direction of a human artist is an interesting direction to explore, but let’s be clear: that’s human initiative and creativity.

Humans are much better than AI at understanding very large contexts—contexts that dwarf a million tokens, contexts that include information that we have no way to describe digitally. Humans are better than AI at creating new directions, synthesizing new kinds of information, and building something new. More than anything else, Ezra Pound’s dictum “Make it New” is the theme of 20th and 21st century culture. It’s one thing to ask AI for startup ideas, but I don’t think AI would have ever created the Web or, for that matter, social media (which really began with USENET newsgroups). AI would have trouble creating anything new because AI can’t want anything—new or old. To borrow Henry Ford’s alleged words, it would be great at designing faster horses, if asked. Perhaps a bioengineer could ask an AI to decode horse DNA and come up with some improvements. But I don’t think an AI could ever design an automobile without having seen one first—or without having a human say “Put a steam engine on a tricycle.”

There’s another important piece to this problem. At DEFCON 2024, Moxie Marlinspike argued that the “magic” of software development has been lost because new developers are stuffed into “black box abstraction layers.” It’s hard to be innovative when all you know is React. Or Spring. Or another massive, overbuilt framework. Creativity comes from the bottom up, starting with the basics: the underlying machine and network. Nobody learns assembler anymore, and maybe that’s a good thing—but does it limit creativity? Not because there’s some extremely clever sequence of assembly language that will unlock a new set of capabilities, but because you won’t unlock a new set of capabilities when you’re locked into a set of abstractions. Similarly, I’ve seen arguments that no one needs to learn algorithms. After all, who will ever need to implement sort()? The problem is that sort() is a great exercise in problem solving, particularly if you force yourself past simple bubble sort to quicksort, merge sort, and beyond. The point isn’t learning how to sort; it’s learning how to solve problems. Viewed from this angle, generative AI is just another abstraction layer, another layer that generates distance between the programmer, the machines they program, and the problems they solve. Abstractions are valuable, but what’s more valuable is the ability to solve problems that aren’t covered by the current set of abstractions.

Which brings me back to the title. AI is good—very good—at what it does. And it does a lot of things well. But we humans can’t forget that it’s our role to think. It’s our role to want, to synthesize, to come up with new ideas. It’s up to us to learn, to become fluent in the technologies we’re working with—and we can’t delegate that fluency to generative AI if we want to generate new ideas. Perhaps AI can help us make those new ideas into realities—but not if we take shortcuts.

We need to think better. If AI pushes us to do that, we’ll be in good shape.

View Details

July was a big month for model releases: There are new large models from Mistral and Meta, smaller multilingual models from Mistral and DeepL, another Mistral model that specializes in code generation, and a small version of GPT-4o. The security world saw another software supply chain disaster when CrowdStrike released a bad software update that disabled many Windows machines worldwide. While CrowdStrike’s release wasn’t “hostile” strictly speaking, it demonstrates that there’s no real difference between a hostile attack or a bug that disables your IT infrastructure. We’re also seeing a surge in malware traffic, along with bogus vulnerability reports in CVE.

Artificial Intelligence* Google’s AlphaProof and Alpha Geometry solved four of six Math Olympiad problems, performance that would have earned a silver medal in an actual competition. This is by far the best that an AI has ever achieved. However, it was significantly slower than humans. * Mistral has released Mistral Large 2, a 123 billion parameter model that (like other models) claims performance similar to GPT-4o. It is particularly strong at code generation. Mistral also highlights its multilingual capabilities. Large 2 is available on Hugging Face. * Facebook/Meta has released Llama 3.1, a 405 billion parameter model that claims performance superior to GPT 4 and Claude 3.5 Sonnet (at least on benchmarks). It is semi-open: Source code and weights are available, but not training data, and there are restrictions on its use. * Google has developed new techniques for predicting weather that combine AI and traditional physical modeling. The new model yields more accurate long-term predictions and reduces energy consumption. * It’s a good day for releasing models. Mistral’s NeMo is a small open source multilingual language model. It has a large (128K) context window and performs well on English, French, German, Spanish, Italian, Portuguese, Chinese, Japanese, Korean, Arabic, and Hindi. * GPT-4o Mini, a small version of OpenAI’s flagship GPT-4o, is now available. Mini’s performance beats GPT-3.5 Turbo and is much less expensive per token. OpenAI also claims that GPT is resistant to jailbreaks and prompt injection. Security experts disagree. * DeepL’s latest large language model, which is trained to specialize in translation, outperforms Google Translate and GPT-4 for translation tasks. * Mistral has released Codestral Mamba, a new model for code generation that uses the new Mamba architecture rather than Transformers. Mamba is significantly faster than Transformers and scales linearly with the size of the input. * RTNet, a new kind of neural network, appears to make decisions the way a human would. * Andrej Karpathy reproduces GPT-2 (the full, 1.6B parameter model) in 24 hours for under $700. * A startup called Textgrain has built a language model that detects hate speech in all 24 languages of the European Union. * Maggie Appleton makes an excellent argument about the role of AI in enabling “barefoot developers”: Non-professional programmers who solve real and important problems that aren’t at the scale needed to interest the software industry. * Microsoft has released GraphRAG on GitHub. GraphRag is a set of tools for retrieval-augmented generation (RAG) that uses graph technology rather than vector embeddings to store and retrieve documents. * With appropriate prompting, large language models are able to detect deep fake images almost as well as custom software. LLMs can also say why they believe an image is a fake. * Figma, the collaborative online design tool, has introduced AI for designers. The tools are for searching for ideas, exploring different directions, and automating repetitive tasks. These features are currently in beta and are free to all users until the end of the year. * Toys “R” Us has created a commercial that was largely generated by SORA, OpenAI’s video-generation AI. * Claude Projects adds to Anthropic’s capabilities. It allows you to upload documents and other data that are shared across all chats associated with the project. You can share projects with other people on your team. Team and Pro plans only. * Is this the end of the GPU? Researchers have developed a way to train language models without matrix multiplication (MatMul), thus requiring much less power. Their models also require less memory and perform similarly to models trained with MatMul.

Programming* Inrupt, a company that is commercializing software building on the open Solid protocol, has announced a data wallet for securely storing and sharing personal data. * The Unix Pipe Card Game should have existed a long time ago! * eBPF, which will soon be supported by Windows, provides a secure kernel execution facility. If it had been available, it would have prevented the CrowdStrike crashes. * PythonMonkey enables Python programs to run JavaScript code, and vice versa. It also gives Python the ability to execute WebAssembly (Wasm) modules. * 1JPM (1 Java Project Manager) presents a different approach to build management. It’s a single file of Java source code, which you edit to reflect your project’s dependencies and other customizations. It’s an interesting alternative to the widely used and hated Maven. * An academic paper discusses design patterns for low-latency applications in C++. While it focuses on high-frequency trading, the ideas in this paper are no doubt useful for many kinds of applications. * The Principles Wiki is a great resource of information and discussion about software design principles. It appears to be new; help it grow! * Julia Evans (@b0rk) gives some good reminders of why shell job control is useful—not the least of which is terminating a program that doesn’t respond to CTRL-C. * Marimo is a Python notebook that runs entirely in the browser using Wasm and Pyodide. Notebook elements, including user interface elements, run automatically whenever you modify or interface with them.

Security* The principle of least privilege in access control is crucial—but in practice, it is rarely implemented well. Can AI do a better job of determining who should access what and when? * A bad upgrade from CrowdStrike caused many Windows systems to crash, causing serious service interruptions for airlines, hospitals, and other organizations. Supply chain security isn’t just about open source; commercial vendors are a problem too. * Cloudflare’s 2024 update to its application security report states that they are seeing a substantial update in malicious traffic, which is now roughly 7% of all traffic. BOT traffic is a major contributor. * An analysis of a software supply chain attack shows how malicious code is hidden in apparently normal images. The engineering in these attacks is increasingly sophisticated. * Blast-RADIUS is a new man-in-the-middle attack against the widely used RADIUS protocol for authentication, authorization, and accounting. Among other things, RADIUS is used for authentication by VPNs, ISPs, and Wi-Fi. * Ente Auth is an open source authenticator that provides 2FA, encrypted cloud backups, and cross-platform synchronization. Its cryptography has been externally audited. * A newly discovered vulnerability in OpenSSH allows unauthenticated remote code execution. If you aren’t keeping up to date on patches, it’s time to start. * The CVE system, which reports and archives security vulnerabilities, has increasingly been used for bogus vulnerability reports. Some of these are good-faith errors, but an increasing number comes from bounty hunters and others trying to enrich their résumés. * Hijackable hyperlinks are a problem. These links have misspelled URLs, placeholder URLs for sites that don’t exist yet, and more. These errors frequently aren’t fixed before the site goes live. Anyone discovering these links can register their domain name and build a hostile site. * SnailLoad is a surprising attack against online privacy. After a user downloads the malware—which does nothing overtly hostile—SnailLoad monitors internet latency. Small variations in latency are used as signatures for detecting what media the user is using.

Web* Google is abandoning its plan to eliminate third party cookie support in Chrome. Instead, there will be user-settable controls for cookie use. While privacy advocates object to abandoning the plan to eliminate cookies, it’s only fair to report that privacy advocates have also objected to Google’s proposed alternatives. * The Hall of Shame has a catalog of dark patterns that web designers use to deceive or manipulate users. Whether you’re a web developer or a user, it’s a good idea to familiarize yourself with the kinds of abuse that are out there. * WebVM is a virtual Linux emulation running in the browser. It’s based on an x86 emulation layer written in WebAssembly. * Transfer Thought is an open source platform for developing WebXR (VR, AR, any other kind of R) experiences. * The Ladybird Browser project is getting a lot of attention. It’s an attempt to build a standards-compliant web browser completely from scratch, without relying on code from Google or other vendors. An alpha version isn’t expected until 2026. * Moonbit is the second new language designed specifically to target WebAssembly. It is inspired by Rust, but designed to be a good match for Wasm’s semantics.

Quantum Computing* PsiQuantum, a quantum computing startup, is planning to build a million qubit quantum computer within 10 years. Unlike other quantum teams, which have focused on building small systems, PsiQuantum is jumping directly to a computer that is capable of useful work. * It’s not a personal quantum computer, but the Quokka is a personal quantum computer emulator with 30 fault-tolerant qubits. It’s a platform for learning how to program useful quantum computers before we get the real thing.

Robotics* A robotic dog with vacuum cleaners in its feet can be used to clean beaches. * Training humanoid robots to dance may make them better at working with humans. They become better able to learn new movements and gestures. * Researchers are working on robots that learn by listening. Although audio provides important clues for many tasks that robots are asked to perform, it is rarely used as a source of training data.

Hardware* Tenstorrent has developed a new set of AI chips that are much less expensive than NVIDIA’s. They are available as PCIe cards or as components of complete workstations.

View Details

Like almost any question about AI, “How does AI impact software architecture?” has two sides to it: how AI changes the practice of software architecture and how AI changes the things we architect.

These questions are coupled; one can’t really be discussed without the other. But to jump to the conclusion, we can say that AI hasn’t had a big effect on the practice of software architecture, and it may never. But we expect the software that architects design will be quite different. There are going to be new constraints, requirements, and capabilities that architects will need to take into account.

We see tools like Devin that promise end-to-end software development, delivering everything from the initial design to a finished project in one shot. We expect to see more tools like this. Many of them will prove to be helpful. But do they make any fundamental changes to the profession? To answer that, we must think about what that profession does. What does a software architect spend time doing? Slinging around UML diagrams instead of grinding out code? It’s not that simple.

The bigger change will be in the nature and structure of the software we build, which will be different from anything that has gone before. The customers will change, and so will what they want. They’ll want software that summarizes, plans, predicts, and generates ideas, with user interfaces ranging from the traditional keyboard to human speech, maybe even virtual reality. Architects will play a leading role in understanding those changes and designing that new generation of software. So, while the fundamentals of software architecture remain the same—understanding customer requirements and designing software that meets those requirements—the products will be new.

AI as an Architectural ToolAI’s success as a programming tool can’t be understated; we’d estimate that over 90% of professional programmers, along with many hobbyists, are using generative tools including GitHub Copilot, ChatGPT, and many others. It’s easy to write a prompt for ChatGPT, Gemini, or some other model, paste the output into a file, and run it. These models can also write tests (if you’re very careful about describing exactly what you want to test). Some can run the code in a sandbox, generating new versions of the program until it passes. Generative AI eliminates a lot of busywork: looking up functions and methods in documentation or wading through questions and answers on Stack Overflow to find something that might be appropriate, for example. There’s been a lot of discussion about whether this increases productivity significantly (it does, but not as much as you might think), improves the quality of the generated code (probably not that well, though humans also write a lot of horrid code), compromises security, and other issues.

But programming isn’t software architecture, a discipline that often doesn’t require writing a single line of code. Architecture deals with the human and organizational side of software development: talking to people about the problems they want solved and designing a solution to those problems. That doesn’t sound so hard, until you get into the details—which are often unspoken. Who uses the software and why? How does the proposed software integrate with the customer’s other applications? How does the software integrate with the organization’s business plans? How does it address the markets that the organization serves? Will it run on the customer’s infrastructure, or will it require new infrastructure? On-prem or in the cloud? How often will the new software need to be modified or extended? (This may have a bearing on whether you decide to implement microservices or a monolithic architecture.) The list of questions architects need to ask is endless.

These questions lead to complex decisions that require knowing a lot of context and don’t have clear, well-defined answers. “Context” isn’t just the number of bytes that you can shove into a prompt or a conversation; context is detailed knowledge of an organization, its capabilities, its needs, its structure, and its infrastructure. In some future, it might be possible to package all of this context into a set of documents that can be fed into a database for retrieval-augmented generation (RAG). But, although it’s very easy to underestimate the speed of technological change, that future isn’t upon us. And remember—the important task isn’t packaging the context but discovering it.

The answers to the questions architects need to ask aren’t well-defined. An AI can tell you how to use Kubernetes, but it can’t tell you whether you should. The answer to that question could be “yes” or “no,” but in either case, it’s not the kind of judgment call we’d expect an AI to make. Answers almost always involve trade-offs. We were all taught in engineering school that engineering is all about trade-offs. Software architects are constantly staring these trade-offs down. Is there some magical solution in which everything falls into place? Maybe on rare occasions. But as Neal Ford said, software architecture isn’t about finding the best solution—it’s about finding the “least worst solution.”

That doesn’t mean that we won’t see tools for software architecture that incorporate generative AI. Architects are already experimenting with models that can read and generate event diagrams, class diagrams, and many other kinds of diagrams in formats like C4 and UML. There will no doubt be tools that can take a verbal description and generate diagrams, and they’ll get better over time. But that fundamentally mistakes why we want these diagrams. Look at the home page for the C4 model. The diagrams are drawn on whiteboards—and that shows precisely what they are for. Programmers have been drawing diagrams since the dawn of computing, going all the way back to flow charts. (I still have a flow chart stencil lying around somewhere.) Standards like C4 and UML define a common language for these diagrams, a standard for unambiguous communications. While there have long been tools for generating boilerplate code from diagrams, that misses the point, which is facilitating communications between humans.

An AI that can generate C4 or UML diagrams based on a prompt would undoubtedly be useful. Remembering the details of proper UML can be dizzying, and eliminating that busywork would be just as important as saving programmers from looking up the names and signatures of library functions. An AI that could help developers understand large bodies of legacy code would help in maintaining legacy software—and maintaining legacy code is most of the work in software development. But it’s important to remember that our current diagramming tools are relatively low-level and narrow; they look at patterns of events, classes, and structures within classes. Helpful as that software would be, it’s not doing the work of an architect, who needs to understand the context, as well as the problem being solved, and connect that context to an implementation. Most of that context isn’t encoded within the legacy codebase. Helping developers understand the structure of legacy code will save a lot of time. But it’s not a game changer.

There will undoubtedly be other AI-driven tools for software architects and software developers. It’s time to start imagining and implementing them. Tools that promise end-to-end software development, such as Devin, are intriguing, though it’s not clear how well they’ll deal with the fact that every software project is unique, with its own context and set of requirements. Tools for reverse engineering an older codebase or loading a codebase into a knowledge repository that can be used throughout an organization—those are no doubt on the horizon. What most people who worry about the death of programming forget is that programmers have always built tools to help them, and what generative AI gives us is a new generation of tooling.

Every new generation of tooling lets us do more than we could before. If AI really delivers the ability to complete projects faster—and that’s still a big if—the one thing that doesn’t mean is that the amount of work will decrease. We’ll be able to take the time saved and do more with it: spend more time understanding the customers’ requirements, doing more simulations and experiments, and maybe even building more complex architectures. (Yes, complexity is a problem, but it won’t go away, and it’s likely to increase as we become even more dependent on machines.)

To someone used to programming in assembly language, the first compilers would have looked like AI. They certainly increased programmer productivity at least as much as AI-driven code generation tools like GitHub Copilot. These compilers (Autocode in 1952, Fortran in 1957, COBOL1 in 1959) reshaped the still-nascent computing industry. While there were certainly assembly language programmers who thought that high-level languages represented the end of programming, they were clearly wrong. How much of the software we use today would exist if it had to be written in assembly? High-level languages created a new era of possibilities, made new kinds of applications conceivable. AI will do the same—for architects as well as programmers. It will give us help generating new code and understanding legacy code. It may indeed help us build more complex systems or give us a better understanding of the complex systems we already have. And there will be new kinds of software to design and develop, new kinds of applications that we’re only starting to imagine. But AI won’t change the fundamentally human side of software architecture, which is understanding a problem and the context into which the solution must fit.

The Challenge of Building with AIHere’s the challenge in a nutshell: Learning to build software in smaller, clearer, more concise units. If you take a step back and look at the entire history of software engineering, this theme has been with us from the beginning. Software architecture is not about high performance, fancy algorithms, or even security. All of those have their place, but if the software you build isn’t understandable, everything else means little. If there’s a vulnerability, you’ll never find it if the code is incomprehensible. Code that has been tweaked to the point of incomprehension (and there were some very bizarre optimizations back in the early days) might be fine for version 1, but it’s going to be a maintenance nightmare for version 2. We’ve learned to do better, even if clear, understandable code is often still an aspiration rather than reality. Now we’re introducing AI. The code may be small and compact, but it isn’t comprehensible. AI systems are black boxes: we don’t really understand how they work. From this historical perspective, AI is a step in the wrong direction—and that has big implications for how we architect systems.

There’s a famous illustration in the paper “Hidden Technical Debt in Machine Learning Systems.” It’s a block diagram of a machine learning application, with a tiny box labeled ML in the center. This box is surrounded by several much bigger blocks: data pipelines, serving infrastructure, operations, and much more. The meaning is clear: in any real-world application, the code that surrounds the ML core dwarfs the core itself. That’s an important lesson to learn.

This paper is a bit old, and it’s about machine learning, not artificial intelligence. How does AI change the picture? Think about what building with AI means. For the first time (arguably with the exception of distributed systems), we’re dealing with software whose behavior is probabilistic, not deterministic. If you ask an AI to add 34,957 to 70,764, you might not get the same answer every time—you might get 105,621,2 a feature of AI that Turing anticipated in his groundbreaking paper “Computing Machinery and Intelligence.” If you’re just calling a math library in your favorite programming language, of course you’ll get the same answer each time, unless there’s a bug in the hardware or the software. You can write tests to your heart’s content and be sure that they’ll all pass, unless someone updates the library and introduces a bug. AI doesn’t give you that assurance. That problem extends far beyond mathematics. If you ask ChatGPT to write my biography, how will you know which facts are correct and which aren’t? The errors won’t even be the same every time you ask.

But that’s not the whole problem. The deeper problem here is that we don’t know why. AI is a black box. We don’t understand why it does what it does. Yes, we can talk about Transformers and parameters and training, but when your model says that Mike Loukides founded a multibillion-dollar networking company in the 1990s (as ChatGPT 4.0 did—I wish), the one thing you cannot do is say, “Oh, fix these lines of code” or “Oh, change these parameters.” And even if you could, fixing that example would almost certainly introduce other errors, which would be equally random and hard to track down. We don’t know why AI does what it does; we can’t reason about it.3 We can reason about the mathematics and statistics behind Transformers but not about any specific prompt and response. The issue isn’t just correctness; AI’s ability to go off the rails raises all kinds of problems of security and safety.

I’m not saying that AI is useless because it can give you wrong answers. There are many applications where 100% accuracy isn’t required—probably more than we realize. But now we have to start thinking about that tiny box in the “Technical Debt” paper. Has AI’s black box grown bigger or smaller? The amount of code it takes to build a language model is miniscule by modern standards—just a few hundred lines, even less than the code you’d use to implement many machine learning algorithms. But lines of code doesn’t address the real issue. Nor does the number of parameters, the size of the training set, or the number of GPUs it will take to run the model. Regardless of the size, some nonzero percentage of the time, any model will get basic arithmetic wrong or tell you that I’m a billionaire or that you should use glue to hold the cheese on your pizza. So, do we want the AI at the core of our diagram to be a tiny black box or a gigantic black box? If we’re measuring lines of code, it’s small. If we’re measuring uncertainties, it’s very large.

The blackness of that black box is the challenge of building and architecting with AI. We can’t just let it sit. To deal with AI’s essential randomness, we need to surround it with more software—and that’s perhaps the most important way in which AI changes software architecture. We need, minimally, two new components:

• Guardrails that inspect the AI module’s output and ensure that it doesn’t get off track: that the output isn’t racist, sexist, or harmful in any of dozens of ways.
  Designing, implementing, and managing guardrails is an important challenge—especially since there are many people out there for whom forcing an AI to say something naughty is a pastime. It isn’t as simple as enumerating likely failure modes and testing for them, especially since inputs and outputs are often unstructured.
• Evaluations, which are essentially test suites for the AI.
  Test design is an important part of software architecture. In his newsletter, Andrew Ng writes about two kinds of evaluations: relatively straightforward evaluations of knowable facts (Does this application for screening résumés pick out the applicant’s name and current job title correctly?), and much more problematic evals for output where there’s no single, correct response (almost any free-form text). How do we design these?

Do these components go inside the box or outside, as their own separate boxes? How you draw the picture doesn’t really matter, but guardrails and evals have to be there. And remember: as we’ll see shortly, we’re increasingly talking about AI applications that have multiple language models, each of which will need its own guardrails and evals. Indeed, one strategy for building AI applications is to use one model (typically a smaller, less expensive one) to respond to the prompt and another (typically a larger, more comprehensive one) to check that response. That’s a useful and increasingly popular pattern, but who checks the checkers? If we go down that path, recursion will quickly blow out any conceivable stack.

On O’Reilly’s Generative AI in the Real World podcast, Andrew Ng points out an important issue with evaluations. When it’s possible to build the core of an AI application in a week or two (not counting data pipelines, monitoring, and everything else), it’s depressing to think about spending several months running evals to see whether you got it right. It’s even more depressing to think about experiments, such as evaluating with a different model—although trying another model might yield better results or lower operating costs. Again, nobody really understands why, but no one should be surprised that all models aren’t the same. Evaluation will help uncover the differences if you have the patience and the budget. Running evals isn’t fast, and it isn’t cheap, and it’s likely to become more expensive the closer you get to production.

Neal Ford has said that we may need a new layer of encapsulation or abstraction to accommodate AI more comfortably. We need to think about fitness and design architectural fitness functions to encapsulate descriptions of the properties we care about. Fitness functions would incorporate issues like performance, maintainability, security, and safety. What levels of performance are acceptable? What’s the probability of error, and what kinds of errors are tolerable for any given use case? An autonomous vehicle is much more safety-critical than a shopping app. Summarizing meetings can tolerate much more latency than customer service. Medical and financial data must be used in accordance with HIPAA and other regulations. Any kind of enterprise will probably need to deal with compliance, contractual issues, and other legal issues, many of which have yet to be worked out. Meeting fitness requirements with plain old deterministic software is difficult—we all know that. It will be much more difficult with software whose operation is probabilistic.

Is all of this software architecture? Yes. Guardrails, evaluations, and fitness functions are fundamental components of any system with AI in its value chain. And the questions they raise are far more difficult and fundamental than saying that “you need to write unit tests.” They get to the heart of software architecture, including its human side: What should the system do? What must it not do? How do we build a system that achieves those goals? And how do we monitor it to know whether we’ve succeeded? In “AI Safety Is Not a Model Property,” Arvind Narayanan and Sayash Kapoor argue that safety issues inherently involve context, and models are always insufficiently aware of context. As a result, “defenses against misuse must primarily be located outside of models.” That’s one reason that guardrails aren’t part of the model itself, although they’re still part of the application, and are unaware of how or why the application is being used. It’s an architect’s responsibility to have a deep understanding of the contexts in which the application is used.

If we get fitness functions right, we may no longer need “programming as such,” as Matt Welsh has argued. We’ll be able to describe what we want and let an AI-based code generator iterate until it passes a fitness test. But even in that scenario, we’ll still have to know what the fitness functions need to test. Just as with guardrails, the most difficult problem will be encoding the contexts in which the application is used.

The process of encoding a system’s desired behavior begs the question of whether fitness tests are yet another formal language layered on top of human language. Will fitness tests be just another way of describing what humans want a computer to do? If so, do they represent the end of programming or the triumph of declarative programming? Or will fitness tests just become another problem that’s “solved” by AI—in which case, we’ll need fitness tests to assess the fitness of the fitness tests? In any case, while programming as such may disappear, understanding the problems that software needs to solve won’t. And that is software architecture.

New Ideas, New PatternsAI presents new possibilities in software design. We’ll introduce some simple patterns to get a handle on the high-level structure of the systems that we’ll be building.

RAGRetrieval-augmented generation, a.k.a. RAG, may be the oldest (though not the simplest) pattern for designing with AI. It’s very easy to describe a superficial version of RAG: you intercept users’ prompts, use the prompt to look up relevant items in a database, and pass those items along with the original prompt to the AI, possibly with some instructions to answer the question using material included in the prompt.

RAG is useful for many reasons:

• It minimizes hallucinations and other errors, though it doesn’t entirely eliminate them.
• It makes attribution possible; credit can be given to sources that were used to create the answer.
• It enables users to extend the AI’s “knowledge”; adding new documents to the database is orders of magnitude simpler and faster than retraining the model.

It’s also not as simple as that definition implies. As anyone familiar with search knows, “look up relevant items” usually means getting a few thousand items back, some of which have minimal relevance and many others that aren’t relevant at all. In any case, stuffing all of them into a prompt would blow out all but the largest context windows. Even in these days of huge context windows (1M tokens for Gemini 1.5, 200K for Claude 3), too much context greatly increases the time and expense of querying the AI—and there are valid questions about whether providing too much context increases or decreases the probability of a correct answer.

A more realistic version of the RAG pattern looks like a pipeline:

It’s common to use a vector database, though a plain old relational database can serve the purpose. I’ve seen arguments that graph databases may be a better choice. Relevance ranking means what it says: ranking the results returned by the database in order of their relevance to the prompt. It probably requires a second model. Selection means taking the most relevant responses and dropping the rest; reevaluating relevance at this stage rather than just taking the “top 10” is a good idea. Trimming means removing as much irrelevant information from the selected documents as possible. If one of the documents is an 80-page report, cut it down to the paragraphs or sections that are most relevant. Prompt construction means taking the user’s original prompt, packaging it with the relevant data and possibly a system prompt, and finally sending it to the model.

We started with one model, but now we have four or five. However, the added models can probably be smaller, relatively lightweight models like Llama 3. A big part of architecture for AI will be optimizing cost. If you can use smaller models that can run on commodity hardware rather than the giant models provided by companies like Google and OpenAI, you will almost certainly save a lot of money. And that is absolutely an architectural issue.

The JudgeThe judge pattern,4 which appears under various names, is simpler than RAG. You send the user’s prompt to a model, collect the response, and send it to a different model (the “judge”). This second model evaluates whether or not the answer is correct. If the answer is incorrect, it sends it back to the first model. (And we hope it doesn’t loop indefinitely—solving that is a problem that’s left for the programmer.)

This pattern does more than simply filter out incorrect answers. The model that generates the answer can be relatively small and lightweight, as long as the judge is able to determine whether it is correct. The model that serves as the judge can be a heavyweight, such as GPT-4. Letting the lightweight model generate the answers and using the heavyweight model to test them tends to reduce costs significantly.

Choice of ExpertsChoice of experts is a pattern in which one program (possibly but not necessarily a language model) analyzes the prompt and determines which service would be best able to process it correctly. It’s similar to mixture of experts (MOE), a strategy for building language models in which several models, each with different capabilities, are combined to form a single model. The highly successful Mixtral models implement MOE, as do GPT-4 and other very large models. Tomasz Tunguz calls choice of experts the router pattern, which may be a better name.

Whatever you call it, looking at a prompt and deciding which service would generate the best response doesn’t have to be internal to the model, as in MOE. For example, prompts about corporate financial data could be sent to an in-house financial model; prompts about sales situations could be sent to a model that specializes in sales; questions about legal issues could be sent to a model that specializes in law (and that is very careful not to hallucinate cases); and a large model, like GPT, can be used as a catch-all for questions that can’t be answered effectively by the specialized models.

It’s frequently assumed that the prompt will eventually be sent to an AI, but that isn’t necessarily the case. Problems that have deterministic answers—for example, arithmetic, which language models handle poorly at best—could be sent to an engine that only does arithmetic. (But then, a model that never makes arithmetic mistakes would fail the Turing test.) A more sophisticated version of this pattern could be able to handle more complex prompts, where different parts of the prompt are sent to different services; then another model would be needed to combine the individual results.

As with the other patterns, choice of experts can deliver significant cost savings. The specialized models that process different kinds of prompts can be smaller, each with its own strengths, and each giving better results in its area of expertise than a heavyweight model. The heavyweight model is still important as a catch-all, but it won’t be needed for most prompts.

Agents and Agent WorkflowsAgents are AI applications that invoke a model more than once to produce a result. All of the patterns discussed so far could be considered simple examples of agents. With RAG, a chain of models determines what data to present to the final model; with the judge, one model evaluates the output of another, possibly sending it back; choice of experts chooses between several models.

Andrew Ng has written an excellent series about agentic workflows and patterns. He emphasizes the iterative nature of the process. A human would never sit down and write an essay start-to-finish without first planning, then drafting, revising, and rewriting. An AI shouldn’t be expected to do that either, whether those steps are included in a single complex prompt or (better) a series of prompts. We can imagine an essay-generator application that automates this workflow. It would ask for a topic, important points, and references to external data, perhaps making suggestions along the way. Then it would create a draft and iterate on it with human feedback at each step.

Ng talks about four patterns, four ways of building agents, each discussed in an article in his series: reflection, tool use, planning, and multiagent collaboration. Doubtless there are more—multiagent collaboration feels like a placeholder for a multitude of sophisticated patterns. But these are a good start. Reflection is similar to the judge pattern: an agent evaluates and improves its output. Tool use means that the agent can acquire data from external sources, which seems like a generalization of the RAG pattern. It also includes other kinds of tool use, such as GPT’s function calling. Planning gets more ambitious: given a problem to solve, a model generates the steps needed to solve the problem and then executes those steps. Multiagent collaboration suggests many different possibilities; for example, a purchasing agent might solicit bids for goods and services and might even be empowered to negotiate for the best price and bring back options to the user.

All of these patterns have an architectural side. It’s important to understand what resources are required, what guardrails need to be in place, what kinds of evaluations will show us that the agent is working properly, how data safety and integrity are maintained, what kind of user interface is appropriate, and much more. Most of these patterns involve multiple requests made through multiple models, and each request can generate an error—and errors will compound as more models come into play. Getting error rates as low as possible and building appropriate guardrails to detect problems early will be critical.

This is where software development genuinely enters a new era. For years, we’ve been automating business systems, building tools for programmers and other computer users, discovering how to deploy ever more complex systems, and even making social networks. We’re now talking about applications that can make decisions and take action on behalf of the user—and that needs to be done safely and appropriately. We’re not concerned about Skynet. That worry is often just a feint to keep us from thinking about the real damage that systems can do now. And as Tim O’Reilly has pointed out, we’ve already had our Skynet moment. It didn’t require language models, and it could have been prevented by paying attention to more fundamental issues. Safety is an important part of architectural fitness.

Staying SafeSafety has been a subtext throughout: in the end, guardrails and evals are all about safety. Unfortunately, safety is still very much a research topic.

The problem is that we know little about generative models and how they work. Prompt injection is a real threat that can be used in increasingly subtle ways—but as far as we know, it’s not a problem that can be solved. It’s possible to take simple (and ineffective) measures to detect and reject hostile prompts. Well-designed guardrails can prevent inappropriate responses (though they probably can’t eliminate them).

But users quickly tire of “As an AI, I’m not allowed to…,” especially if they’re making requests that seem reasonable. It’s easy to understand why an AI shouldn’t tell you how to murder someone, but shouldn’t you be able to ask for help writing a murder mystery? Unstructured human language is inherently ambiguous and includes phenomena like humor, sarcasm, and irony, which are fundamentally impossible in formal programming languages. It’s unclear whether AI can be trained to take irony and humor into account. If we want to talk about how AI threatens human values, I’d worry much more about training humans to eliminate irony from human language than about paperclips.

Protecting data is important on many levels. Of course, training data and RAG data must be protected, but that’s hardly a new problem. We know how to protect databases (even though we often fail). But what about prompts, responses, and other data that’s in-flight between the user and the model? Prompts might contain personally identifiable information (PII), proprietary information that shouldn’t be submitted to AI (companies, including O’Reilly, are creating policies governing how employees and contractors use AI), and other kinds of sensitive information. Depending on the application, responses from a language model may also contain PII, proprietary information, and so on. While there’s little danger of proprietary information leaking5 from one user’s prompt to another user’s response, the terms of service for most large language models allow the model’s creator to use prompts to train future models. At that point, a previously entered prompt could be included in a response. Changes in copyright case law and regulation present another set of safety challenges: What information can or can’t be used legally?

These information flows require an architectural decision—perhaps not the most complex decision but a very important one. Will the application use an AI service in the cloud (such as GPT or Gemini), or will it use a local model? Local models are smaller, less expensive to run, and less capable, but they can be trained for the specific application and don’t require sending data offsite. Architects designing any application that deals with finance or medicine will have to think about these issues—and with applications that use multiple models, the best decision may be different for each component.

There are patterns that can help protect restricted data. Tomasz Tunguz has suggested a pattern for AI security that looks like this:

The proxy intercepts queries from the user and “sanitizes” them, removing PII, proprietary information, and anything else inappropriate. The sanitized query is passed through the firewall to the model, which responds. The response passes back through the firewall and is cleaned to remove any inappropriate information.

Designing systems that can keep data safe and secure is an architect’s responsibility, and AI adds to the challenges. Some of the challenges are relatively simple: reading through license agreements to determine how an AI provider will use data you submit to it. (AI can do a good job of summarizing license agreements, but it’s still best to consult with a lawyer.) Good practices for system security are nothing new, and have little to do with AI: good passwords, multifactor authentication, and zero trust networks need to be standard. Proper management (or elimination) of default passwords is mandatory. There’s nothing new here and nothing specific to AI—but security needs to be part of the design from the start, not something added in when the project is mostly done.

Interfaces and ExperiencesHow do you design a user’s experience? That’s an important question, and something that often escapes software architects. While we expect software architects to put in time as programmers and to have a good understanding of software security, user experience design is a different specialty. But user experience is clearly a part of the overall architecture of a software system. Architects may not be designers, but they must be aware of design and how it contributes to the software project as a whole—particularly when the project involves AI. We often speak of a “human in the loop,” but where in the loop does the human belong? And how does the human interact with the rest of the loop? Those are architectural questions.

Many of the generative AI applications we’ve seen haven’t taken user experience seriously. Star Trek’s fantasy of talking to a computer appeared to come to life with ChatGPT, so chat interfaces have become the de facto standard. But that shouldn’t be the end of the story. While chat certainly has a role, it isn’t the only option, and sometimes, it’s a poor one. One problem with chat is that it gives attackers who want to drive a model off its rails the most flexibility. Honeycomb, one of the first companies to integrate GPT into a software product, decided against a chat interface: it gave attackers too many opportunities and was too likely to expose users’ data. A simple Q&A interface might be better. A highly structured interface, like a form, would function similarly. A form would also provide structure to the query, which might increase the likelihood of a correct, nonhallucinated answer.

It’s also important to think about how applications will be used. Is a voice interface appropriate? Are you building an app that runs on a laptop or a phone but controls another device? While AI is very much in the news now, and very much in our collective faces, it won’t always be that way. Within a few years, AI will be embedded everywhere: we won’t see it and we won’t think about it any more than we see or think about the radio waves that connect our laptops and phones to the internet. What kinds of interfaces will be appropriate when AI becomes invisible? Architects aren’t just designing for the present; they’re designing applications that will continue to be used and updated many years into the future. And while it isn’t wise to incorporate features that you don’t need or that someone thinks you might need at some vague future date, it’s helpful to think about how the application might evolve as technology advances.

Projects by IF has an excellent catalog of interface patterns for handling data in ways that build trust. Use it.

Everything Changes (and Remains the Same)Does generative AI usher in a new age of software architecture?

No. Software architecture isn’t about writing code. Nor is it about writing class diagrams. It’s about understanding problems and the context in which those problems arise in depth. It’s about understanding the constraints that the context places on the solution and making all the trade-offs between what’s desirable, what’s possible, and what’s economical. Generative AI isn’t good at doing any of that, and it isn’t likely to become good at it any time soon. Every solution is unique; even if the application looks the same, every organization building software operates under a different set of constraints and requirements. Problems and solutions change with the times, but the process of understanding remains.

Yes. What we’re designing will have to change to incorporate AI. We’re excited by the possibility of radically new applications, applications that we’ve only begun to imagine. But these applications will be built with software that’s not really comprehensible: we don’t know how it works. We will have to deal with software that isn’t 100% reliable: What does testing mean? If your software for teaching grade school arithmetic occasionally says that 2+2=5, is that a bug, or is that just what happens with a model that behaves probabilistically? What patterns address that kind of behavior? What does architectural fitness mean? Some of the problems that we’ll face will be the same old problems, but we’ll need to view them in a different light: How do we keep data safe? How do we keep data from flowing where it shouldn’t? How do we partition a solution to use the cloud where it’s appropriate and run on-premises where that’s appropriate? And how do we take it a step farther? In O’Reilly’s recent Generative AI Success Stories Superstream, Ethan Mollick explained that we have to “embrace the weirdness”: learn how to deal with systems that might want to argue rather than answer questions, that might be creative in ways that we don’t understand, and that might be able to synthesize new insights. Guardrails and fitness tests are necessary, but a more important part of the software architect’s function may be understanding just what these systems are and what they can do for us. How do software architects “embrace the weirdness”? What new kinds of applications are waiting for us?

With generative AI, everything changes—and everything stays the same.

AcknowledgmentsThanks to Kevlin Henney, Neal Ford, Birgitta Boeckeler, Danilo Sato, Nicole Butterfield, Tim O’Reilly, Andrew Odewahn, and others for their ideas, comments, and reviews.

Footnotes1. COBOL was intended, at least in part, to allow regular business people to replace programmers by writing their own software. Does that sound similar to the talk about AI replacing programmers? COBOL actually increased the need for programmers. Business people wanted to do business, not write software, and better languages made it possible for software to solve more problems. 2. Turing’s example. Do the arithmetic if you haven’t already (and don’t ask ChatGPT). I’d guess that AI is particularly likely to get this sum wrong. Turing’s paper is no doubt in the training data, and that’s clearly a high-quality source, right? 3. OpenAI and Anthropic recently released research in which they claim to have extracted “concepts” (features) from their models. This could be an important first step toward interpretability. 4. If you want more info, search for “LLM as a judge” (at least on Google); this search gives relatively clean results. Other likely searches will find many documents about legal applications. 5. Reports that information can “leak” sideways from a prompt to another user appear to be urban legends. Many versions of that legend start with Samsung, which warned engineers not to use external AI systems after discovering that they had sent proprietary information to ChatGPT. Despite rumors, there isn’t any evidence that this information ended up in the hands of other users. However, it could have been used to train a future version of ChatGPT.

View Details

The first AI image generation model I got to play around with was Midjourney v2 in summer 2022. A month earlier, OpenAI had launched DALL-E 2 in beta, and the results looked unbelievably magical. You could generate images in any art style simply by prompting an AI with the name of an artist.

I didn’t go to art school, and I didn’t really know that much about art, so one of the first prompts I tried was “Super Mario drinking a glass of beer.” The resulting image wasn’t anything Nintendo’s IP lawyers would get out of bed for, but exactly two years later, the version generated by Midjourney v6 is pixel-perfect.

The media and online commentators have discussed the legal and ethical implications of training on copyrighted material, but those cases are in the hands of the courts and governments, who will need to unpick that thorny issue. Whatever happens with copyright law for training, there’s a common practice in prompt engineering today that I’m absolutely sure will be banned by all major tools one day soon: using the names of copyrighted IP in prompts. For example, if I try the same prompt in ChatGPT, it refuses:

After some clever work to trick ChatGPT into revealing its system prompt (the instructions given to it by OpenAI, in addition to your prompt), we can see it has been told not to create images in the style of artists within the last 100 years: “You can name artists, creative professionals, or studios in prompts only
if their latest work was created prior to 1912 (e.g., Van Gogh, Goya).” Copyright only lasts so long before becoming public domain, and it’s safe to assume an artist’s work is no longer protected by copyright if they died over 100 years ago.

Source: https://x.com/bryced8/status/1710140618641653924 Be careful when using a living artist’s nameAs a coauthor of Prompt Engineering for Generative AI, published by O’Reilly in June 2024, this topic has been on my mind. In editing, we went through every example in the book that referenced a living artist and swapped it out for something public domain. This is a higher standard than most prompt engineers hold themselves to today, but my expectation is that this will soon become the norm.

When you invoke the name of an artist or protected IP franchise in order to copy their style for commercial gain, it’s hard to argue that you’re not violating copyright. It’s one thing to have an AI that was influenced by an artist in training, and it’s quite another to intentionally prompt the AI to copy that artist’s style precisely. Consider the case of Greg Rutkowski, a favorite among early AI adopters. His name was invoked thousands of times by AI artists looking for a fantasy aesthetic. If Magic: The Gathering or Dungeons & Dragons decide to add “in the style of greg rutkowski” to their prompts instead of hiring him for their next set of illustrations, he has a clear claim of loss of income.

Source: https://thehustle.co/10-13-22-fantasy-artistThere has been growing awareness around this issue, with tools like Stable Diffusion providing opt-out mechanisms for artists who don’t want their works included. Newer AI tools have been more savvy about their restrictions on what can go into a prompt: for example Suno doesn’t allow you to reference the name of a band or musician. Instead, to make a Taylor Swift-style song for my four-year-old daughter, I had to prompt for “Contemporary country pop with elements of indie rock and a female singer.”

Unbundling and remixing the style of an artistIf using artists’ names in prompts is illegal or at least unethical, what’s the alternative? It may be time to go to art school! Rather than AI eliminating the artist’s role, I suspect artists who adopt AI will do far better than AI specialists like myself who don’t know art. For example, I recently listened to Isaacson’s biography of Da Vinci and learned about the technique of sfumato, the subtle blending of colors and tones. Now that I know that word, I can add it to my prompts when I’m trying to create depth and realistic human expressions. An actual artist would have known that already, as well as many other techniques and when it’s appropriate to use them.

If you read further down in ChatGPT’s system prompt, they describe a useful technique anyone can use to avoid ripping off an artist’s style:

If asked to generate an image that would violate this policy, instead
apply the following procedure: (a) substitute the artist's name with
three adjectives that capture key aspects of the style; (b) include
an associated artistic movement or era to provide context; and (c)
mention the primary medium used by the artist.

This is very close to a technique I use every day called unbundling, coined by Bakz T. Future, where you ask ChatGPT to describe an artist’s style and use that description in your prompt instead of the artist’s name. This technique leads to more creative and original output because there is room for interpretation in a list of stylist elements rather than constraining the creativity of the output to a specific artist.

Source: https://bakztfuture.substack.com/p/dall-e-2-unbundlingThe chances are that there are elements of the artist’s style that you don’t actually want to copy. When you have a description of an artist’s style, you can then more easily modify the description to get what you want. Perhaps you want red and yellow swirls instead of blue and green, or you want to see the sky in the daytime instead of at night. The more you deviate from Van Gogh’s original vision, the more the end result will be your own.

They say to steal ideas from one person is plagiarism—to steal from many is research. One surefire technique that I’ve found for increasing the originality of my prompts is to remix the styles of multiple artists together. For example, you could merge the styles of Van Gogh’s The Starry Night with elements of Salvador Dalí’s The Persistence of Memory:

While using artists’ names in prompts is still allowed in most tools, it wouldn’t be too surprising if they’re banned in the near future. Even if the ethical considerations don’t motivate you, practical ones should. Getting good at this unbundling and remixing technique now will put you at an advantage when one day this practice gets banned from most major platforms and you get to benefit from more creative and interesting work in the meantime, building more of a name for yourself in the industry. Steve Jobs may have said “great artists steal,” but T.S. Eliot, the original source of that quote, elaborates that you should “make it into something better, or at least something different.”

The same principle applies to text-generation tooI don’t expect it to just be AI-generated images and music that will be affected. This will apply to text one day too. Role-play prompting is still an extremely common technique on the text-generation side, with people prompting an LLM to “Name this product in the style of Steve Jobs,” “Write a new scene for the TV show Friends,” or “Write this novel in the style of Hemingway.” It may be harder for LLM platforms to ban all writers and celebrities from prompts than it has been to do so with artists and musicians, but as AI progresses, this will be easier for them to do.

Despite the contribution from Meta’s Llama 3, there still isn’t a competitive open source model to rival GPT-4 like there is with Stable Diffusion XL in the image generation space. While OpenAI, Google, and Anthropic hold all the cards, your ability to use roleplay in your prompts is at risk of going away at any time. When that happens, you don’t want to suddenly have to rewrite all of your prompt templates to stop them failing! Having an unbundled and remixed style in your prompt instead of invoking a famous name makes your prompt future-proof, and maybe one day your lawyers will thank you.

View Details

It’s clear that generative AI is already being used by a majority—a large majority—of programmers. That’s good. Even if the productivity gains are smaller than many think, 15% to 20% is significant. Making it easier to learn programming and begin a productive career is nothing to complain about, either. We were all impressed when Simon Willison asked ChatGPT to help him learn Rust. Having that power at your fingertips is amazing.

But there’s one misgiving that I share with a surprisingly large number of other software developers. Does the use of generative AI increase the gap between entry-level junior developers and senior developers?

Generative AI makes a lot of things easier. When writing Python, I often forget to put colons where they need to be. I frequently forget to use parentheses when I call print(), even though I never used Python 2. (Very old habits die very hard and there are many older languages in which print is a command rather than a function call.) I usually have to look up the name of the Pandas function to do, well, just about anything—even though I use Pandas fairly heavily. Generative AI, whether you use GitHub Copilot, Gemini, or something else eliminates that problem. And I’ve written that, for the beginner, generative AI saves a lot of time, frustration, and mental space by reducing the need to memorize library functions and arcane details of language syntax—which are multiplying as every language feels the need to catch up to its competition. (The walrus operator? Give me a break.)

There’s another side to that story, though. We’re all lazy and we don’t like to remember the names and signatures of all the functions in the libraries that we use. But is not needing to know them a good thing? There is such a thing as fluency with a programming language, just as there is with human language. You don’t become fluent by using a phrasebook. That might get you through a summer backpacking through Europe, but if you want to get a job there, you’ll need to do a lot better. The same thing is true in almost any discipline. I have a PhD in English literature. I know that Wordsworth was born in 1770, the same year as Beethoven; Coleridge was born in 1772; a lot of important texts in Germany and England were published in 1798 (plus or minus a few years); the French revolution was in 1789—does that mean something important was happening? Something that goes beyond Wordsworth and Coleridge writing a few poems and Beethoven writing a few symphonies? As it happens, it does. But how would someone who wasn’t familiar with these basic facts think to prompt an AI about what was going on when all these separate events collided? Would you think to ask about the connection between Wordsworth, Coleridge, and German thought, or to formulate ideas about the Romantic movement that transcended individuals and even European countries? Or would we be stuck with islands of knowledge that aren’t connected, because we (not the AIs) are the ones that connect them? The problem isn’t that an AI couldn’t make the connection, it’s that we wouldn’t think to ask it to make the connection.

I see the same problem in programming. If you want to write a program, you have to know what you want to do. But you also need an idea of how it can be done if you want to get a nontrivial result from an AI. You have to know what to ask and, to a surprising extent, how to ask it. I experienced this just the other day. I was doing some simple data analysis with Python and Pandas. I was going line by line with a language model, asking “How do I” for each line of code that I needed (sort of like GitHub Copilot)—partly as an experiment, partly because I don’t use Pandas often enough. And the model backed me into a corner that I had to hack myself out of. How did I get into that corner? Not because of the quality of the answers. Every response to every one of my prompts was correct. In my post-mortem, I checked the documentation and tested the sample code that the model provided. I got backed into the corner because of the one question I didn’t know that I needed to ask. I went to another language model, composed a longer prompt that described the entire problem I wanted to solve, compared this answer to my ungainly hack, and then asked “What does the reset_index() method do?” And then I felt (not incorrectly) like a clueless beginner—if I had known to ask my first model to reset the index, I wouldn’t have been backed into a corner.

You could, I suppose, read this example as “see, you really don’t need to know all the details of Pandas, you just have to write better prompts and ask the AI to solve the whole problem.” Fair enough. But I think the real lesson is that you do need to be fluent in the details. Whether you let a language model write your code in large chunks or one line at a time, if you don’t know what you’re doing, either approach will get you in trouble sooner rather than later. You perhaps don’t need to know the details of Pandas’ groupby() function, but you do need to know that it’s there. And you need to know that reset_index() is there. I have had to ask GPT “wouldn’t this work better if you used groupby()?” because I’ve asked it to write a program where groupby() was the obvious solution, and it didn’t. You may need to know whether your model has used groupby() correctly. Testing and debugging haven’t, and won’t, go away.

Why is this important? Let’s not think about the distant future, when programming-as-such may no longer be needed. We need to ask how junior programmers entering the field now will become senior programmers if they become over-reliant on tools like Copilot and ChatGPT. Not that they shouldn’t use these tools—programmers have always built better tools for themselves, generative AI is the latest generation in tooling, and one aspect of fluency has always been knowing how to use tools to become more productive. But unlike earlier generations of tools, generative AI easily becomes a crutch; it could prevent learning, rather than facilitate it. And junior programmers who never become fluent, who always need a phrasebook, will have trouble making the jump to seniors.

And that’s a problem. I’ve said, many of us have said, that people who learn how to use AI won’t have to worry about losing their jobs to AI. But there’s another side to that: People who learn how to use AI to the exclusion of becoming fluent in what they’re doing with the AI will also need to worry about losing their jobs to AI. They will be replaceable—literally, because they won’t be able to do anything an AI can’t do. They won’t be able to come up with good prompts because they will have trouble imagining what’s possible. They’ll have trouble figuring out how to test and they’ll have trouble debugging when AI fails. What do you need to learn? That’s a hard question, and my thoughts about fluency may not be correct. But I would be willing to bet that people who are fluent in the languages and tools they use will use AI more productively than people who aren’t. I would also bet that learning to look at the big picture rather than the tiny slice of code you’re working on will take you far. Finally, the ability to connect the big picture with the microcosm of minute details is a skill that few people have. I don’t. And, if it’s any comfort, I don’t think AIs do, either.

So—learn to use AI. Learn to write good prompts. The ability to use AI has become “table stakes” for getting a job, and rightly so. But don’t stop there. Don’t let AI limit what you learn and don’t fall into the trap of thinking that “AI knows this, so I don’t have to.” AI can help you become fluent: the answer to “What does reset_index() do” was revealing, even if having to ask was humbling. It’s certainly something I’m not likely to forget. Learn to ask the big picture questions: What’s the context into which this piece of code fits? Asking those questions rather than just accepting the AI’s output is the difference between using AI as a crutch and using it as a learning tool.

View Details

Can anything that is programmable have an embedded LLM? Apparently so. llama.ttf is a plain old font that looks like Open Sans—you can download it and install it. We don’t know whether this is a massive vulnerability or a way to embed AI in almost any text-based application. Probably both.

It may just be a coincidence, but several new programming languages have come on the scene in the past month or two. Why? This probably isn’t backlash against automated programming. (An LLM obviously can’t be trained for a language without much public source code.)

AI* This is crazy. llama.ttf is a font that embeds a (small) large language model. The font itself can do automatic text generation. It relies on the HarfBuzz font shaping engine, which uses Wasm. It works in Gimp; it may also work in Chrome and Firefox. Everything runs locally. * Researchers are finding that conversations with an AI can help to dispel beliefs in conspiracy theories. (This paper notes that it is preliminary and not yet peer-reviewed. All of the data for their research will be posted when it is complete.) * The Podman AI Lab is a good way for Linux users to experiment with running AI locally. Unsurprisingly, Podman is container-based, which simplifies moving models from a desktop environment into a production environment. * Researchers have developed a BERT-based model to detect malicious LLM prompts that instruct the LLM to generate phishing websites or emails. * Anthropic has released a new version of its midsize language model, Claude 3.5 Sonnet. It claims performance similar to GPT-4o and Gemini 1.5. A new Artifacts window allows interaction with Claude output. * What if nonplayer characters in games were able to converse with you and with each other? What if the structure of a game unfolded dynamically in response to the play, rather than being scripted in advance? That’s something generative AI could bring to games. * An AI system has been trained to count flowers. This ability has important applications in farming: if you can count the flowers, you can predict crop yields. * Anthropic has provided documentation about building evaluations and test cases for Claude. * Meta has developed AudioSeal, a system for watermarking AI-generated audio content. It is 90% to 100% reliable and can identify small clips inserted into a larger file. AudioSeal is free and available on GitHub, though Meta itself isn’t using it yet. * Cohere’s latest model, Command R, can use multiple external tools to respond to prompts. It is capable of multistep tool use, in which it creates a plan to solve the problem, uses APIs (described in a JSON document) to access external tools, and generates a result. * Can a large language model write comedy? Apparently not. Ideas aren’t original, models have trouble with incongruity and surprise, and they don’t have a sense for the longer arc of a dialogue. * Luma’s Dream Machine is an impressive generative AI tool for creating short video from a text prompt. * NVIDIA has announced a set of models for generating synthetic training data. Usage of the models is governed by a new Open Model License Agreement, which specifies that models can be used commercially, derivative models can be distributed freely, and NVIDIA claims no ownership to the models’ outputs. * AI detectors, most of which have very high error rates despite vendors’ claims, are being used to incorrectly blackball professional writers. * A new strategy for chunking content for RAG is using sentence-BERT (SBERT) to compute embeddings for each sentence, then using the embedding to detect shifts in topic. * If you’re building an AI application with RAG, consider the use of graph databases (GraphRAG) rather than vector databases. Graphs are harder to work with but are better at providing appropriate context for answering complex queries. * Google’s experimental Illuminate turns academic papers into interview-like discussions that summarize the important points. It’s currently in private beta. * Anthropic’s essay about Claude’s Character is a thoughtful piece about how they are giving Claude 3 character traits and how character traits affect the way Claude interacts with the world. * Honeycomb’s VP of engineering suggests that AI will be more helpful in developing frontend code than backend and that chats are often an inefficient user interface. * OpenAI and Anthropic have posted research on extracting features from their GPT and Claude models. This represents a significant step toward interpretability or understanding why a model gives a specific response to a prompt. * What is your dog trying to say? A research project is using AI to decode dog vocalizations. * Here’s a look at the transformer-based language model that Apple is incorporating into macOS. It’s very small (apparently 34M parameters) and appears to be based on GPT-2—clearly designed to run on devices like phones without draining the battery. * Codestral is a new language model from Mistral that specializes in code generation. It has been trained on over 80 languages and claims to produce better results than other language models. * Language models can create database schemas—but schemas rarely remain unchanged forever. How well can language models modify a schema? * Can AI help researchers digitize smells? Digitization has worked well for sight and sound. Smell lags far behind. * Could AI be used to improve privacy? Researchers have developed applications for removing sensitive parts of images and replacing them with something similar but different—sort of an automated Photoshop. * Using knowledge graphs in the context of RAG makes intuitive sense. In a comparison between “GraphRAG” and “traditional RAG,” GraphRag comes out on top because the graph encodes relationships between the data.

Programming Gleam is a new type-safe functional language. It compiles to Erlang and JavaScript. Its goal is to enable developers to build large concurrent systems. * OpenTofu, the open source fork of Terraform, now has state encryption, a highly requested feature that has been on Terraform’s queue for nearly a decade. * Starlark is a small Python-like language for describing how to build code. Starlark’s design principles include determinism and safety; executing a program multiple times gives identical results and it cannot access the network, filesystem, or system clock. * One of WebAssembly’s cocreators has developed a new programming language called Virgil. It’s designed as a systems programming language, though it is intended not to compete with Zig (whatever that means). * The James Webb Space Telescope is a step forward for digital twin technology. A digital model of the telescope was used to choreograph its self-assembly in outer space. The twin continues to monitor the telescope and predict the effects of software updates. * Mesop is a very lightweight Python framework for rapid web application development. It’s good for demos, probably not good for production. Built by Google but not supported. * AI gateways are like API gateways, but specialized for accessing the APIs of language models. They facilitate monitoring, choosing which model to serve a request, caching, security, and many other features. * Julia Evans’s (@b0rk) latest zine, How Git Works*, is a must-read for anyone who doesn’t understand Git. Which is just about everybody. * Language developers have been busy. The Grain language was designed for compilation to WebAssembly. It was inspired by OCaml; pattern matching is a key feature. * GoFr is a framework for building microservices in Go. It incorporates tools for observability and supports the use of many different data sources. * Infrastructure from Code claims that a system’s infrastructure requirements can be inferred from the code itself, and that a specification for the necessary infrastructure can be generated automatically.

Security* While it’s not clear what’s going on, there appears to have been a supply chain attack against the CDN service Polyfill.io. Polyfill claims that they have been defamed and has moved to a new domain name. Security experts advise caution. * A new critical vulnerability in Progress Software’s MoveIT product is being actively exploited. Attackers can impersonate any legitimate SFTP user, giving them the ability to copy, delete, or create files. Attackers can also obtain cryptographic hashes for user passwords. * The US is banning sales of Kaspersky’s antivirus software starting in July. * Linux malware controlled by emojis sent from Discord? That’s creative. The malware, which spreads through phishes, uses emojis to send commands. Emojis apparently evade security software that expects commands to be text strings. * A new kind of phish uses the Windows Search protocol to download malware. Another new phish uses progressive web applications (PWAs) to steal credentials. * After widespread criticism of its Recall feature for AI-enabled Windows 11, Microsoft has announced that it would be disabled by default. Release of Recall has been delayed; it will be delivered first to the Windows Insider Program, essentially a private beta. * As the importance of AI has grown, Hugging Face has become a target for threat actors. Most recently, attackers have stolen authorization tokens from Spaces.

Web Sim City* in the browser with WebAssembly? Impressive. * UIX is a new full stack web development framework. It is part of a larger Unyt project, which has the ambitious goal of developing the next-generation decentralized internet. * BBC Ventures is investing in a 3D video streaming platform. * Hotwire is an approach to building web applications (specifically, single-page applications or SPAs) that sends HTML to encapsulate complex data rather than JSON. Its goal is to minimize the need for JavaScript. * Cobalt is a very lightweight platform for running apps built from HTML5, CSS, and JavaScript. Google, its maker, mentions YouTube as a typical application, but I suspect it can be used for more interesting work. * Adrian Holovaty’s Soundslice has incorporated music scanning: You can upload a PDF or a photo of a musical score, and Soundslice will analyze it. You can then edit, rearrange, and play back the score.

Hardware* An open standard for connectors? That’s important if you want very high performance without having to license proprietary designs. * Energy Meter is a tool for measuring the power consumed by Intel CPUs. It could be useful in these days of power-hungry AI applications, though it can’t account for GPUs. Runs on Linux but that’s most of the world’s servers. * Mitsubishi has built a robot that solves the Rubik’s Cube in 0.305 seconds. Their best time is 0.204 seconds, but according to Guinness, that wasn’t under proper conditions. * Lifesaving robots are in use on Lake Michigan to help drowning people remain afloat until a lifeguard can arrive.

Biology* Precision Neuroscience has successfully planted an array of 4,096 miniaturized electrodes in a human brain. In doing so, they’ve achieved higher resolution for neuron activity than previous implants, including Neuralink’s.

View Details

As AI continues to advance at a rapid pace, developers are increasingly turning to AI Driven Development (AIDD) to build more intelligent and adaptive applications. However, using natural language prompts to describe complex behaviors to AI can be a real challenge. While natural language is expressive, it lacks the structure and efficiency needed to clearly communicate intricate instructions and maintain complex state.

One of the biggest issues with natural language prompts is the lack of clear encapsulation and delineation of related instructions. Unlike source code, which uses elements like braces and indentation to group and organize code blocks, natural language prompts can quickly turn into a wall of text that’s a nightmare to read and maintain. This lack of structure makes it harder for AI to understand and follow the intended instructions accurately.

Natural language is not always the most efficient way to represent complex instructions. Language models can suffer from “sparse attention,” where the model’s attention capacity is stressed as the input context grows. This can lead to the AI forgetting or misinterpreting important information within the prompt, particularly in the middle of the input, a phenomenon known as the “Lost in the Middle” problem (Liu et al., 2023). As a result, the AI may struggle to adhere to the intended instructions, leading to incorrect outputs.

To overcome these limitations and unlock the full potential of AIDD, we need a more structured approach to AI communication. This is where pseudocode prompting and the SudoLang programming language come into play.

Pseudocode Prompting and SudoLang: A Solution for AI Driven DevelopmentPseudocode prompting is a technique that combines the expressiveness of natural language with the structure and precision of programming concepts. By using pseudocode-like syntax, developers can provide contextual clues to the AI, guiding it to understand and follow instructions more accurately.

Language models behave like role-players, and everything in your prompt is used as context and connotation for the AI to ground its response in. Large Language Models (LLMs) are trained on code, and they understand that it is a language used to instruct computers in a detailed and precise manner. Because of this, seeing things that look like code carry the connotation that it must think carefully about the content. It triggers enhanced thinking, state tracking, and reasoning in the language model. The paper “Prompting with Pseudocode Instructions” (Mishra et al., 2023) demonstrated that pseudocode prompts generated 12%–38% response score improvements.

One of the key benefits of pseudocode prompting is its ability to efficiently represent complex instructions. By using programming constructs such as constraints, interfaces, and functions, pseudocode can express intricate logic and algorithms in a way that is concise, readable, and expressive. This not only reduces the token count of the prompt, which saves money and time, but also makes it easier for the AI to comprehend and execute the desired behavior.

SudoLang, a programming language specifically designed, discovered, and curated with the help of GPT-4 to communicate complex ideas and programs with AI, takes pseudocode prompting to the next level. I say designed, discovered, and curated, because, while some intelligent design went into SudoLang, the real effort went into discovering and testing the pseudocode understanding inherent in language models, by curating common features and concepts from existing languages—both programming and natural language. In fact, all sufficiently advanced large language models already know SudoLang—it was always there in the latent space. We just uncovered, curated features, and documented it.

SudoLang is a multi-paradigm language that features natural language constraint-based programming inspired by Ivan Sutherland’s Sketchpad. Sketchpad was a graphical programming language that allowed users to select multiple elements on the canvas using direct screen interaction with a light pen. For example, you could select two lines, constrain them to be parallel, and then changing one line would cause the constraint solvers to kick in and update the other line to maintain the parallel constraint. Constraints are a powerful concept in SudoLang, allowing developers to specify desired behaviors and relationships between entities in a declarative manner.

Constraints in SudoLang are written in natural language and can consist of either positive guidance or elements to avoid. They can be formal mathematical axioms or whimsical instructions for a playful puppy chatbot. SudoLang is a declarative language, meaning that you should focus constraints on what you want rather than write out detailed instructions on how to do it. The AI is usually smart enough to figure out the “how” for you, and that ability will save you a lot of work, time, tokens, and money.

With SudoLang, developers can define constraints that the AI will continuously solve for, ensuring that related states remain synchronized and consistent. For example, you can specify that a calculated value should always be updated whenever a key variable changes. Constraints also provide a way to guide the AI’s behavior, such as giving it a particular personality or tailoring its output for a specific target audience.

SudoLang’s interface-oriented design is another key feature that sets it apart. Developers can clearly and succinctly define data structures and operations using typed interfaces. While most types can be inferred, SudoLang allows for explicit type declarations when needed, and all common programming language types are supported. This interface-driven approach promotes modularity and reusability, making it easier to build and maintain complex AI-driven applications.

Composition is another strength of SudoLang. The language supports functions, function composition, and interface composition, enabling developers to create complex programs by combining simpler, independent components. Function composition allows you to pipe a series of functions together one after the other to form a new, more complex function. The algebraic definition of function composition is: (f ∘ g)(x) = f(g(x)) (often pronounced “f after g“).

In SudoLang, you can write this as g |> f – which means: apply the function g to the supplied value, then apply the function f to the output of g. Composing interfaces is easy, too:

// Some interfaces to compose:Character { name: "" health: 1..10 // ".." is the range operator in SudoLang strength: 1..10 speed: 1..10}Objective { name description status: ready | started | finished}Quest { name description objectives: []}// The following composes all of the above interfaces together:Player { // The `...` here is the spread operator. It will mix // all the properties of "character" into the top level // of the player object, so instead of player.character.health // we get a direct reference to player.health. ...character // include the properties of "character" quests: []} Let’s see how a language model might interpret and execute a text adventure scenario using SudoLang. This example is based on an interaction with Anthropic’s Claude 3.5 Sonnet:

Village Elder: "Greetings, brave adventurer. Our village is in dire need of ancient wisdom. In the grand library to the east, there lies a scroll of immense power. I task you with retrieving this scroll and bringing it back to me. Will you accept this quest?"Player: I accept the quest, Elder.Village Elder: "Excellent! The scroll is said to be hidden in the deepest chamber of the library. Be cautious, for the path may be treacherous. Return to me once you have the scroll in your possession."Your quest log updates:Quests: - name: Retrieve the Ancient Scroll description: Fetch an old scroll from the ancient library and return it to the village elder. objectives: - name: Fetch the scroll description: Find and retrieve the ancient scroll from the library. status: started - name: Return to the elder description: Bring the scroll back to the village elder. status: ready [Time passes as the player journeys to the ancient library and navigates its chambers…]

You enter a dimly lit chamber deep within the library. Dust motes dance in the air as your torch illuminates rows upon rows of ancient texts. In the center of the room, atop a weathered stone pedestal, you spot a scroll emanating a faint, ethereal glow.Player: I carefully approach the pedestal and take the glowing scroll.As your fingers touch the parchment, you feel a surge of energy course through your body. The scroll's glow intensifies for a moment before settling into a steady, warm light. You've found the ancient scroll the elder spoke of!You have a new item: Ancient ScrollYour quest log updates: Player:
name: Adventurer
inventory:
– Ancient Scroll
quests:
– name: Retrieve the Ancient Scroll
description: Fetch an old scroll from the ancient library and return it to the village elder.
objectives:
– name: Fetch the scroll
description: Find and retrieve the ancient scroll from the library.
status: finished
– name: Return to the elder
description: Bring the scroll back to the village elder.
status: started

This example demonstrates how SudoLang can be used to create dynamic, state-aware interactions in a text adventure game. The AI interprets the player's actions, updates the game state accordingly, and provides appropriate responses and quest log updates, obeying its given constraints, and respecting the interfaces we have defined.In SudoLang, the more you can lean on the AI's ability to *infer* your intent, the better, so we usually don't need to reach for features like looping and branching, but it's worth noting that SudoLang does include loop and branch constructs, though without all the syntax clutter and precise references and function definitions that most programming languages require. For example, loops can be expressed as:sudolangfor each user, send update email ``` This construct allows the AI to handle the implementation details. If you guide it with the right constraints, it could even customize each email based on the user’s preferences or recent activity, demonstrating how SudoLang can lead to more sophisticated and personalized outcomes compared to traditional programming approaches.

Likewise, there are branching constructs. AI responds just fine to “if” and “else”, but you can also use the language model’s understanding of semantics and natural language to do semantic pattern matching, e.g.:

if the player dies, extract the penalty for dying and respawn in the nearest safe location But if you’re token golfing (trynig to reduce the token count to optimize for cost and performance), this might also work:

(player died) => extract penalty, respawn Leaning just a little harder on inference and introducing a tiny bit of syntax in this case reduced the token count on GPT-4 from 17 to 8.

Pattern matching is a powerful feature inspired by languages like Haskell, Elixir, Rust, Scala, F#, and so on.

The basic gist in traditional programming languages might look something like:

// Non-standard, hypothetical pattern matching syntax in JavaScriptfunction area(shape) { return match (shape) => { ({ type: "circle", radius }) => Math.PI * Math.pow(radius, 2); ({ type: "rectangle", width, height }) => width * height; ({ type: "triangle", base, height }) => 0.5 * base * height; }}console.log(area({ type: "rectangle", width: 4, height: 5 })); // 20 It’s worth noting that in SudoLang, you don’t need to define the shapes or their properties, as the AI can infer them from the context. In SudoLang, that function won’t need explicit pattern matching, and would probably look more like:

function area(shape) => Number One of the most powerful aspects of SudoLang is its ability to leverage the omni-referential inference properties of LLMs. LLMs are trained on vast amounts of human knowledge available on the internet, enabling them to understand a wide range of domains. SudoLang takes advantage of this by allowing developers to define functions without always providing implementation details.

In many cases, the AI can infer what a function should do based on the program context and the function name alone. This arguably makes SudoLang the programming language with the largest standard library, as it can tap into the AI’s extensive natural language understanding to infer functionality across a broad spectrum of domains.

I frequently use welcome() as my initializer to instruct the AI how to start, without defining what welcome() should mean. Very often, if you supply a good preamble and main interface with commands, welcome() will just do the right thing without any additional instruction required.

Other commands or functions that just work include standard library functions from common programming languages, many of the functions from JavaScript’s Lodash or RxJS work, for example. sort(list) |> take(3) will sort a list based on some inferred criteria (e.g. alphabetically), and return the top 3 results. Of course, you can specify the criteria and the sort order when you call sort.

As AI continues to transform the software development landscape, tools like pseudocode prompting and SudoLang will play a crucial role in enabling developers to harness the full potential of AIDD. By providing a structured and efficient way to communicate with AI, SudoLang empowers developers to create intelligent, adaptive, and robust applications that push the boundaries of what’s possible with artificial intelligence.

As you can see, SudoLang and pseudocode prompting can unlock the true potential of AI Driven Development. The future of software development is here. Let’s make some magic!

View Details

Last month, The New York Times claimed that tech giants OpenAI and Google have waded into a copyright gray area by transcribing the vast volume of YouTube videos and using that text as additional training data for their AI models despite terms of service that prohibit such efforts and copyright law that the Times argues places them in dispute. The Times also quoted Meta officials as saying that their models will not be able to keep up unless they follow OpenAI and Google’s lead. In conversation with reporter Cade Metz, who broke the story, on the New York Times podcast The Daily, host Michael Barbaro called copyright violation “AI’s Original Sin.”

At the very least, copyright appears to be one of the major fronts so far in the war over who gets to profit from generative AI. It’s not at all clear yet who is on the right side of the law. In the remarkable essay Talkin’ ‘Bout AI Generation: Copyright and the Generative-AI Supply Chain, Katherine Lee, A. Feder Cooper, and James Grimmelmann of Cornell note:

“…copyright law is notoriously complicated, and generative-AI systems manage to touch on a great many corners of it. They raise issues of authorship, similarity, direct and indirect liability, fair use, and licensing, among much else. These issues cannot be analyzed in isolation, because there are connections everywhere. Whether the output of a generative AI system is fair use can depend on how its training datasets were assembled. Whether the creator of a generative-AI system is secondarily liable can depend on the prompts that its users supply.”

But it seems less important to get into the fine points of copyright law and arguments over liability for infringement, but instead to explore the political economy of copyrighted content in the emerging world of AI services: who will get what, and why? And rather than asking who has the market power to win the tug of war, we should be asking what institutions and business models are needed to allocate the value that is created by the “generative AI supply chain” in proportion to the role that various parties play in creating it? And how do we create a virtuous circle of ongoing value creation, an ecosystem in which everyone benefits?

Publishers (including The New York Times itself, which has sued OpenAI for copyright violation) argue that works such as generative art and texts compete with the creators whose work the AI was trained on. In particular, the Times argues that AI-generated summaries of news articles are a substitute for the original articles and damage its business. They want to get paid for their work and preserve their existing business.

Meanwhile, the AI model developers, who have taken in massive amounts of capital, need to find a business model that will repay all that investment. Times reporter Cade Metz provides an apocalyptic framing of the stakes and a binary view of the possible outcome. In The Daily interview, he opines that

“…a jury or a judge or a law ruling against OpenAI could fundamentally change the way this technology is built. The extreme case is these companies are no longer allowed to use copyrighted material in building these chatbots. And that means they have to start from scratch. They have to rebuild everything they’ve built. So this is something that not only imperils what they have today, it imperils what they want to build in the future.”

And in his original reporting on the actions of OpenAI and Google and the internal debates at Meta, Metz quotes Sy Damle, a lawyer for Silicon Valley venture firm Andreessen Horowitz, who has claimed that “The only practical way for these tools to exist is if they can be trained on massive amounts of data without having to license that data. The data needed is so massive that even collective licensing really can’t work.”

“The only practical way”? Really?

I propose instead that not only is the problem solvable, but that solving it can create a new golden age for both AI model providers and copyright-based businesses. What’s missing is the right architecture for the AI ecosystem, and the right business model.

Unpacking the Problem Let’s first break down “copyrighted content.” Copyright reserves to the creator(s) the exclusive right to publish and to profit from their work. It does not protect facts or ideas, but a unique ‘creative’ expression of those facts or ideas. And unique creative expression is something that is fundamental to all human communication. And humans using the tools of generative AI are indeed often using it as a way to enhance their own unique creative expression. What is actually in dispute is who gets to profit from that unique creative expression.

Not all copyrighted content is created for profit. According to US copyright law, everything published in any form, including on the internet, is automatically copyrighted by the author for the life of its creator, plus 70 years. Some of that content is intended to be monetized either by advertising, subscription, or individual sale, but that is not always true. While a blog or social media post, YouTube gardening or plumbing tutorial, music or dance performance, is implicitly copyrighted by its creators (and may also include copyrighted music or other copyrighted components), it is meant to be freely shared. Even content that is meant to be shared freely, though, has an expectation of remuneration in the form of recognition and attention.

Those intending to commercialize their content usually indicate that in some way. Books, music, and movies, for example, bear copyright notices and are registered with the copyright office (which confers additional rights to damages in the event of infringement). Sometimes these notices are even machine-readable. Some online content is protected by a paywall, requiring a subscription to access it. Some content is marked “noindex” in the HTML code of the website, indicating that it should not be spidered by search engines (and presumably other web crawlers). Some content is visibly associated with advertising, indicating that it is being monetized. Search engines “read” everything they can, but legitimate services generally respect signals that tell them “no” and don’t go where they aren’t supposed to.

AI developers surely recognize these distinctions. As The New York Times article referenced at the start of this piece notes, “The most prized data, A.I. researchers said, is high-quality information, such as published books and articles, which have been carefully written and edited by professionals.” It is precisely because this content is more valuable that AI developers seek the unlimited ability to train on all available content, regardless of its copyright status.

Next, let’s unpack “fair use.” Typical examples of fair use are quotations, reproduction of an image for the purpose of criticism or comment, parodies, summaries, and in more recent precedent, the links and snippets that help a search engine or social media user to decide whether to consume the content. Fair use is generally limited to a portion of the work in question, such that the reproduced content cannot serve as a substitute for the original work.

Once again it is necessary to make distinctions that are not legal, but practical. If the long term health of AI requires the ongoing production of carefully written and edited content—as the currency of AI knowledge certainly does—only the most short-term of business advantage can be found by drying up the river AI companies drink from. Facts are not copyrightable, but AI model developers standing on the letter of the law will find cold comfort in that if news and other sources of curated content are driven out of business.

An AI-generated review of Denis Villeneuve’s Dune or a plot summary of Frank Herbert’s original novel is not a substitute for consuming the original and will not harm the production of new novels or movies. But a summary of a news article or blog post might indeed be a sufficient substitute. If news and other forms of high quality, curated content are important to the development of future AI models, AI developers should be looking hard at how they will impact the future health of these sources.

The comparison of AI summaries with the snippets and links provided in the past by search engines and social media sites is instructive. Google and others have rightly pointed out that search drives traffic to sites, which the sites can then monetize as they will, by their own advertising (or advertising in partnership with Google), by subscription, or just by the recognition the creators receive when people find their work. The fact that when given the choice to opt out of search, very few sites choose to do so provides substantial evidence that, at least in the past, copyright owners have recognized the benefits they receive from search and social media. In fact, they compete for higher visibility through Search Engine Optimization and social media marketing.

But there is certainly reason for web publishers to fear that AI-generated summaries will not drive traffic to sites in the same way as more traditional search or social media snippets. The summaries provided by AI are far more substantial than their search and social media equivalents, and in cases such as news, product search, or a search for factual answers, a summary may provide a reasonable substitute. When readers see an AI Answer that references sources they trust, they take it as a trusted answer and may well take it at face value and move on. This should be of concern not only to the sites that used to receive the traffic but to those who used to drive it. Because in the long term, if people stop creating high quality content to ingest, the whole ecosystem breaks down.

This is not a battle that either side should be looking to “win.” Instead, it’s an opportunity to think through how to strengthen two public goods. Journalism professor Jeff Jarvis put it well in a response to an earlier draft of this piece: “It is in the public good to have AI produce quality and credible (if “hallucinations” can be overcome) output. It is in the public good that there be the creation of original quality, credible, and artistic content. It is not in the public good if quality, credible content is excluded from AI training and output OR if quality, credible content is not created.” We need to achieve both goals.

Finally, let’s unpack the relation of an AI to its training data, copyrighted or uncopyrighted. During training, the AI model learns the statistical relationships between the words or images in its training set. As Derek Slater has pointed out, much like musical chord progressions, these relationships can be seen as “basic building blocks” of expression. The models themselves do not contain a copy of the training data in any human-recognizable form. Rather, they are a statistical representation of the probability, based on the training data, that one word will follow another, or in an image, that one pixel will be adjacent to another. Given enough data, these relationships are remarkably robust and predictable, so much so that it is possible for generated output to closely resemble or duplicate elements of the training data.

It is certainly worth knowing what content has been ingested. Mandating transparency about the content and source of training data sets—the generative AI supply chain—would go a long way towards encouraging frank discussions between disputing parties. But focusing on examples of inadvertent resemblances to the training data misses the point.

Generally, whether payment is in currency or in recognition, copyright holders seek to withhold data from training because it seems to them that may be the only way to prevent unfair competition from AI outputs or to negotiate a fee for use of their content. As we saw from web search, “reading” that does not produce infringing output, delivers visibility (traffic) to the originator of the content, and preserves recognition and credit is generally tolerated. So AI companies should be working to develop solutions that content developers will see as valuable to them.

The recent protest by long-time StackOverflow contributors who don’t want the company to use their answers to train OpenAI models highlights a further dimension of the problem. These users contributed their knowledge to StackOverflow, giving the company perpetual and exclusive rights to their answers. They reserved no economic rights, but they still believe they have moral rights. They had, and continue to have, the expectation that they will receive recognition for their knowledge. It isn’t the training per se that they care about, it’s that the output may no longer give them the credit they deserve.

And finally, the Writer’s Guild strike established the contours of who gets to benefit from derivative works created with AI. Are content creators entitled to be the ones to profit from AI-generated derivatives of their work, or can they be made redundant when their work is used to train their replacements? (More specifically, the agreement stipulated that AI works could not be considered “source material.” That is, studios couldn’t have the AI do a first draft, then treat the scriptwriter as someone merely “adapting” the draft and thus get to pay them less.) As the settlement demonstrated, this is not a purely economic or legal question, but one of market power.

In sum, there are three parts to the problem: what content is ingested as part of the training data in the first place, what outputs are allowed, and who gets to profit from those outputs. Accordingly, here are some guidelines for how AI model developers ought to handle copyrighted content:

1. Train on copyrighted content that is freely available, but respect signals like subscription paywalls, the robots.txt file, the HTML “noindex” keyword, terms of service, and other means by which copyright holders signal their intentions. Make the effort to distinguish between content that is meant to be freely shared and that which is intended to be monetized and for which copyright is intended to be enforced.

There is some progress towards this goal. In part because of the EU AI act, it is likely that within the next twelve months every major AI developer will have implemented mechanisms for copyright holders to opt out in a machine-readable way. Already, OpenAI allows sites to disallow its GPTbot web crawler using the robots.txt file, and Google does the same for its Web-extended crawler. There are also efforts like the DoNotTrain database, and tools like Cloudflare Bot Manager. OpenAI’s forthcoming Media Manager promises to “enable creators and content owners to tell us what they own and specify how they want their works to be included or excluded from machine learning research and training.” This is helpful, but insufficient. Even on today’s internet these mechanisms are fragile, complex, change frequently, and are often not well understood by sites whose content is being scraped.

But more importantly, simply giving content creators the right to opt out is missing the real opportunity, which is to assemble datasets for training AI that specifically recognize copyright status and the goals of content creators, and thus become the underlying mechanism for a new AI economy. As Dodge, the hyper-successful game developer who is the protagonist of Neal Stephenson’s novel Reamde noted, “you had to get the whole money flow system figured out. Once that was done, everything else would follow.” 2. Produce outputs that respect what can be known about the source and the nature of copyright in the material.

This is not dissimilar to the challenges of preventing many other types of disputed content, such as hate speech, misinformation, and various other types of prohibited information. We’ve all been told many times that ChatGPT or Claude or Llama3 is not allowed to answer a particular question or to use particular information that it would otherwise be able to generate because they violate rules against bias, hate speech, misinformation, or dangerous content. And, in fact, in its comments to the copyright office, OpenAI describes how it provides similar guardrails to keep ChatGPT from producing copyright-infringing content. What we need to know is how effective they are and how widely they are deployed.

There are already techniques for identifying the content most closely related to some types of user queries. For example, when Google or Bing provides an AI-generated summary of a web page or news article, you typically see links below the summary that point to the pages from which the summary was generated. This is done using a technology called retrieval augmented generation (RAG), which generates a set of search results that are vectorized, providing an authoritative source to be consulted by the model before it generates a response. The generative LLM is said to have grounded its response in the documents provided by these vectorized search results. In essence, it’s not regurgitating content from the pre-trained models but rather reasoning on these source snippets to work out an articulate response based on them. In short, the copyrighted content has been ingested, but it is detected during the output phase as part of an overall content management pipeline. Over time, there will likely be many more such techniques.

One hotly debated question is whether these links provide the same level of traffic as the previous generation of search and social media snippets. Google claims that its AI summaries drive even more traffic than traditional snippets, but it hasn’t provided any data to back up that claim, and may be based on a very narrow interpretation of click-through rate, as parsed in a recent Search Engine Land analysis. My guess is that there will be some winners and some losers as with past search engine algorithm updates, not to mention further updates, and that it is too early for sites to panic or to sue.

But what is missing is a more generalized infrastructure for detecting content ownership and providing compensation in a general purpose way. This is one of the great business opportunities of the next few years, awaiting the kind of breakthrough that pay-per-click search advertising brought to the World Wide Web.

In the case of books, for example, rather than training on known sources of pirated content, how about building a book data commons, with an additional effort to preserve information about the copyright status of the works it contains? This commons could be used as the basis not only for AI training but for measuring the vector similarity to existing works. Already, AI model developers use filtered versions of the Common Crawl Database, which provides a large percentage of the training data for most LLMs, to reduce hate speech and bias. Why not do the same for copyright? 3. Pay for the output, not the training. It may look like a big win for existing copyright holders when they receive multi-million dollar licensing fees for the use of content they control. First, only the most deep-pocketed AI companies will be able to afford pre-emptive payments for the most valuable content, which will deepen their competitive moat with regard to smaller developers and open source models. Second, these fees are likely insufficient to become the foundation of sustainable long term businesses and creative ecosystems. Once you’ve licensed the chicken, the licensee gets the eggs. (Hamilton Nolan calls it “Selling your house for firewood.”) Third, the payment is often going to intermediaries, and is not passed on to the actual creators.

How “payment” works might depend very much on the nature of the output and the business model of the original copyright holder. If the copyright owners prefer to monetize their own content, don’t provide the actual outputs. Instead, provide pointers to the source. For content from sites that depend on traffic, this means either sending traffic, or if not, a payment negotiated with the copyright owner that makes up for the owner’s decreased ability to monetize its own content. Look for win-win incentives that will lead to the development of an ongoing, cooperative content ecosystem.

In many ways, YouTube’s Content ID system provides an intriguing precedent for how this process might be automated. According to YouTube’s description of the system,

“Using a database of audio and visual files submitted by copyright owners, Content ID identifies matches of copyright-protected content. When a video is uploaded to YouTube, it’s automatically scanned by Content ID. If Content ID finds a match, the matching video will get a Content ID claim. Depending on the copyright owner’s Content ID settings, a Content ID claim results in one of the following actions:

• Blocks a video from being viewed
• Monetizes the video by running ads against it and sometimes sharing revenue with the uploader
• Tracks the video’s viewership statistics”

(Revenue is only sometimes shared with the uploader because the uploader may not own all of the monetizable elements of the uploaded content. For example, a dance or music performance video may use copyrighted music for which payment goes to the copyright holder rather than the uploader.)

One can imagine this kind of copyright enforcement framework being operated by the platforms themselves, much as YouTube operates Content ID, or by third party services. The problem is obviously more difficult than the one facing YouTube, which only had to discover matching music and videos in a relatively fixed format, but the tools are more sophisticated today. As RAG demonstrates, vector databases make it possible to find weighted similarities even in wildly different outputs.

Of course, there is a lot that would need to be worked out. Using vector similarity for attribution is promising but there are concerning limitations. Consider Taylor Swift. She is so popular that there are many artists trying to sound like her. This sets up a kind of adversarial situation that has no obvious solution. Imagine a vector database that has Taylor in it along with a thousand Taylor copycats. Now imagine an AI generated song that “sounds like Taylor.” Who gets the revenue? Is it the top 100 nearest vectors (99 of which are cheap copycats of Taylor)? or should Taylor herself get most of the revenue? There are interesting questions in how to weigh similarity—just as there are interesting questions in traditional search about how to weigh various factors to come up with the “best” result for a search query. Solving these questions is the innovative (and competitive) frontier.

One option might be to retrieve the raw materials for generation (vs. using RAG for attribution). Want to generate a paragraph that sounds like Stephen King? Explicitly retrieve some representation of Stephen King, generate from it, and then pay Stephen King. If you don’t want to pay for Stephen King’s level of quality, fine. Your text will be generated from lower quality bulk-licensed “horror mystery text” as your driver. There are some rather naive assumptions in this ideal, namely in how to scale it to millions or billions of content providers, but that’s what makes it an interesting entrepreneurial opportunity. For a star-driven media area like music, it definitely makes sense.

My point is that one of the frontiers of innovation in AI should be in techniques and business models to enable the kind of flourishing ecosystem of content creation that has characterized the web and the online distribution of music and video. AI companies that figure this out will create a virtuous flywheel that rewards content creation rather than turning the industry into an extractive dead end.

An Architecture of Participation for AIOne thing that makes copyright seem intractable is the race for monopoly by the large AI providers. The architecture that many of them seem to imagine for AI is some version of “one ring to rule them all,” “all your base are belong to us,” or the Borg. This architecture is not dissimilar to the model of early online information providers like AOL and the Microsoft Network. They were centralized and aimed to host everyone’s content as part of their service. It was only a question of who would win the most users and host the most content.

The World Wide Web (and the underlying internet itself) had a fundamentally different idea, which I have called an “architecture of participation.” Anyone could host their own content and users could surf from one site to another. Every website and every browser could communicate and agree on what can be seen freely, what is restricted, and what must be paid for. It led to a remarkable expansion of the opportunities for the monetization of creativity, publishing, and copyright.

Like the networked protocols of the internet, the design of Unix and Linux programming envisioned a world of cooperating programs developed independently and assembled into a greater whole. The Unix/Linux file system has a simple but powerful set of access permissions with three levels: user, group, and world. That is, some files are private only to the creator of the file, others to a designated group, and others are readable by anyone.

Imagine with me, for a moment, a world of AI that works much like the World Wide Web or open source systems such as Linux. Foundation models understand human prompts and can generate a wide variety of content. But they operate within a content framework that has been trained to recognize copyrighted material and to know what they can and can’t do with it. There are centralized models that have been trained on everything that’s freely readable (world permission), others that are grounded in content belonging to a specific group (which might be a company or other organization, a social, national or language group, or any other cooperative aggregation), and others that are grounded in the unique corpus of content belonging to an individual.

It may be possible to build such a world on top of ChatGPT or Claude or any one of the large centralized models, but it is far more likely to emerge from cooperating AI services built with smaller, distributed models, much as the web was built by cooperating web servers rather than on top of AOL or the Microsoft Network. We are told that open source AI models are riskier than large centralized ones, but it’s important to make a clear eyed assessment of their benefits versus their risks. Open source better enables not only innovation but control. What if there was an open protocol for content owners to open up their repositories to AI Search providers but with control and forensics over how that content is handled and especially monetized?

Many creators of copyrighted content will be happy to have their content ingested by centralized, proprietary models and used freely by them, because they receive many benefits in return. This is much like the way today’s internet users are happy to let centralized providers collect their data, as long as it is used for them and not against them. Some creators will be happy to have the centralized models use their content as long as they monetize it for them. Other creators will want to monetize it themselves. But it will be much harder for anyone to make this choice freely if the centralized AI providers are able to ingest everything and to output potentially infringing or competing content without compensation, or compensation that amounts to pennies on the dollar.

Can you imagine a world where a question to an AI chatbot might sometimes lead to an immediate answer, sometimes to the equivalent of “I’m sorry, Dave, I’m afraid I can’t do that” (much as you now get told when you try to generate prohibited speech or images, but in this case, due to copyright restrictions), and at others, “I can’t do that for you, Dave, but the New York Times chatbot can.” At other times, by agreement between the parties, an answer based on copyrighted data might be given directly in the service, but the rights holder will be compensated.

This is the nature of the system that we’re building for our own AI services at oreilly.com. Our online technology learning platform is a marketplace for content provided by hundreds of publishers and tens of thousands of authors, trainers, and other experts. A portion of user subscription fees is allocated to pay for content, and copyright holders are compensated based on usage (or in some cases, based on a fixed fee).

We are increasingly using AI to help our authors and editors generate content such as summaries, translations and transcriptions, test questions, and assessments as part of a workflow that involves editorial and subject matter expert review, much as when we edit and develop the underlying books and videos. We’re also building dynamically generated user-facing AI content that also keeps track of provenance and shares revenue with our authors and publishing partners.

For example, for our “Answers” feature (built in partnership with Miso Technologies), we’ve used a RAG architecture to build a research, reasoning, and response model that searches across content for the most relevant results (similar to traditional search) and then generates a response tailored to the user interaction based on those specific results.

Because we know what content was used to produce the generated answer, we are not only able to provide links to the sources used to generate the answer, but to pay authors in proportion to the role of their content in generating it. As Lucky Gunasekara, Andy Hsieh, Lan Li, and Julie Baron write in “The R in ‘RAG’ Stands for ‘Royalties’”:

“In essence, the latest O’Reilly Answers release is an assembly line of LLM workers. Each has its own discrete expertise and skill set, and they work together to collaborate as they take in a question or query, reason what the intent is, research the possible answers, and critically evaluate and analyze this research before writing a citation-backed grounded answer…. The net result is that O’Reilly Answers can now critically research and answer questions in a much richer and more immersive long-form response while preserving the citations and source references that were so important in its original release….

The newest Answers release is again built with an open source model—in this case, Llama 3…. The benefit of constructing Answers as a pipeline of research, reasoning, and writing using today’s leading open source LLMs is that the robustness of the questions it can answer will continue to increase, but the system itself will always be grounded in authoritative original expert commentary from content on the O’Reilly learning platform.”

When someone reads a book, watches a video, or attends a live training, the copyright holder gets paid. Why should derivative content generated with the assistance of AI be any different? Accordingly, we have built tools to integrate AI generated products directly into our payment system. This approach enables us to properly attribute usage, citations, and revenue to content and ensures our continued recognition of the value of our authors’ and teachers’ work.

And if we can do it, we know that others can too.

View Details

We’re planning a live virtual event later this year, and we want to hear from you. Are you using a powerful AI technology that seems like everyone ought to be using? Here’s your opportunity to show the world!

AI is too often seen as an enterprise of, by, and for the wealthy. We’re going to take a look at a Digital Green’s Farmer.Chat, a generative AI bot that was designed to help small-scale farmers in developing countries access critical agricultural information. Developing countries have frequently implemented technical solutions that would never have occurred to engineers in wealthy countries. They solve real problems rather than appealing to the “let’s start another Facebook” fantasies of venture capitalists. Farmer.Chat is one of those solutions.

Farmer.Chat helps agricultural extension agents (EAs) and farmers get answers to questions about agriculture. It has been deployed in India, Ethiopia, Nigeria, and Kenya. While it was designed originally for EAs, farmers are increasingly using it directly; they have already become accustomed to asking questions online using social media. Providing online access to better, more reliable agricultural information quickly and efficiently was an obvious goal.

An AI application for farmers and EAs faces many constraints. One of the biggest constraints is location. Farming is hyperlocal. Two farms may be a mile apart, but if one is on a hillside and another in a valley, they will have completely different soil, drainage, and perhaps even weather conditions. Different microclimates, pests, crops: what works for your neighbor might not work for you.

The data to answer hyperlocal questions about topics like fertilization and pest management exists, but it’s spread across many databases with many owners: governments, NGOs, and corporations, in addition to local knowledge about what works. Farmer.Chat uses all these sources to answer questions—but in doing so, it has to respect the rights of the farmers and the database owners. Farmers have a right to privacy; they may not want to share information about their farm or to let others know what problems they are experiencing. Corporations may want to limit what data they expose and how it is exposed. Digital Green solves this problem through FarmStack, a secure open source protocol for opt-in data sharing. End-to-end encryption is used for all connections. All sources of data, including farmers and government agencies, choose what data they want to share and how it is shared. They can decide to share certain kinds of data and not others, or they impose restrictions on the use of their data (for example, limit it to certain geographic areas). While fine-grained opt-in sounds imposing, treating its data suppliers and its users with respect has allowed Farmer.Chat to build a trusted ecosystem for sharing data. In turn, that ecosystem leads to successful farms.

FarmStack also enables confidential feedback. Was a data provider’s data used successfully? Did a farmer provide local knowledge that helped others? Or were their problems with the information? Data is always a two-way street; it’s important not just to use data but also to improve it.

Translation is the most difficult problem for Digital Green and Farmer.Chat. Farmer.Chat currently supports six languages (English, Hindi, Telugu, Amharic, Swahili, and Hausa) and Digital Green is working to add more. To serve EAs and farmers well, Farmer.Chat must also be multimodal—voice, text, and video—and it has to reach farmers in their native languages. While useful information is available in many languages, finding that information and answering a question in the farmer’s language through voice chat is an imposing challenge. Farmer.Chat uses Google Translate, Azure, Whisper, and Bhashini (an Indian company that supplies text-to-speech and other services for Indian languages), but there are still gaps. Even within one language, the same word can mean different things to different people. Many farmers measure their yield in bags of rice, but what is “a bag of rice”? It might mean 10 kilos to one farmer, and 5 kilos to someone who sells to a different buyer. This one area where keeping an extension agent in the loop is critical. An EA would be aware of issues such as local usage, local slang, and technical farming terms, and could resolve problems by asking questions and interpreting answers appropriately. EAs also help with trust. Farmers are naturally wary of taking an AI’s advice in changing practices that have been used for generations. An EA who knows the farmers and their history and who can situate the AI’s answers in a local context is much more trustworthy.

To manage the problem of hallucination and other kinds of incorrect output, Digital Green uses retrieval-augmented generation (RAG). While RAG is conceptually simple—look up relevant documents and construct a prompt that tells the model to build its response from them—in practice, it’s more complex. As anyone who has done a search knows, search results are likely to give you a few thousand results. Including all those results in a RAG query would be impossible with most language models and impractical with the few that allow large context windows. So the search results need to be scored for relevance; the most relevant documents need to be selected; then the documents need to be pruned so that they contain only the relevant parts. Keep in mind that, for Digital Green, this problem is both multilingual and multimodal: relevant documents can turn up in any of the languages or modes that they use.

It’s important to test every stage of this pipeline carefully: translation software, text-to-speech software, relevance scoring, document pruning, and the language models themselves: Can another model do a better job? Guardrails need to be put in place at every step to guard against incorrect results. Results need to pass human review. Digital Green tests with “Golden QAs,” highly rated sets of questions and answers. When asked a “golden question,” can the application consistently produce results as good as the “golden answer?” Testing like this needs to be performed constantly. Digital Green also manually reviews 15% of their usage logs, to make sure that their results are consistently high quality. In his podcast for O’Reilly, Andrew Ng recently noted that the evaluation stage of product development frequently doesn’t get the attention it deserves, partly because it’s so easy to write AI software; who wants to spend a few months testing an application that took a week to write? But that’s exactly what’s necessary for success.

Farmer.Chat is designed to be gender inclusive and climate smart. Because 60% of the world’s small farmers are women, it’s important for the application to be welcoming to women and not to assume that all farmers are male. Pronouns are important. So are role models; the farmers who present techniques and answer questions in video clips must include men and women.

Climate-smart means making climate-sensitive recommendations wherever possible. Climate change is a huge issue for farmers, especially in countries like India where increasing temperatures and changing rainfall patterns can be ruinous. Recommendations must anticipate current weather patterns and the ways they’re likely to change. Climate-smart recommendations also tend to be less expensive. For example, while Farmer.Chat isn’t afraid of recommending commercial fertilizers, it emphasizes local solutions: almost every farm can have a limitless supply of compost—which costs less than fertilizer and helps manage agricultural waste.

Farming can be very tradition-bound: “We do this because that’s what my grandparents did, and their parents before them.” A new farming technique coming from some faceless scientists in an urban office means little; it’s much more likely to be adopted if you hear that it’s been used successfully by a farmer you know and respect. To help farmers adopt new practices, Digital Green prioritizes the work of peers whenever possible using videos collected from local farmers. They try to put farmers in contact with each other, celebrating their successes to help farmers adopt new ideas.

Finally, Farmer.Chat and FarmStack are both open source. Software licenses may not affect farmers directly, but they are important in building healthy ecosystems around projects that aim to do good. We see too many applications whose purpose is to monopolize a user’s attention, subject a user to unwanted surveillance, or debase political discussions. An open source project to help people: we need more of that.

Over its history, in which Farmer.Chat is just the latest chapter, Digital Green has aided over 6.3 million farmers, boosted their income by up to 24%, and increased crop yields by up to 17%. Farmer.Chat is the next step in this process. And we wonder: the problems faced by small-scale farms in the developed nations are no different from the problems of developing countries. Climate, insects, and crop disease have no respect for economics or politics. Farmer.Chat helps small scale farmers succeed in developing nations. We need the same services in the so-called “first world.”

View Details

The latest release of O’Reilly Answers is the first example of generative royalties in the AI era, created in partnership with Miso. This new service is a trustworthy source of answers for the O’Reilly learning community and a new step forward in the company’s commitment to the experts and authors who drive knowledge across its learning platform.

Generative AI may be a groundbreaking new technology, but it’s also unleashed a torrent of complications that undermine its trustworthiness, many of which are the basis of lawsuits. Will content creators and publishers on the open web ever be directly credited and fairly compensated for their works’ contributions to AI platforms? Will there be an ability to consent to their participation in such a system in the first place? Can hallucinations really be controlled? And what will happen to the quality of content in a future of LLMs?

While perfect intelligence is no more possible in a synthetic sense than in an organic sense, retrieval-augmented generative (RAG) search engines may be the key to addressing the many concerns we listed above. Generative AI models are trained on large repositories of information and media. They are then able to take in prompts and produce outputs based on the statistical weights of the pretrained models of those corpora. However, RAG engines are not generative AI models so much as they are directed reasoning systems and pipelines that use generative LLMs to create answers grounded in sources. The processes that help inform the construction of these high-quality, ground-truth-verified, and citation-backed answers hold great hope for yielding a digital societal and economic engine to credit its sources and pay them simultaneously. It is possible.

This isn’t just a theory; it’s a solution born from direct applied practice. For the past four years, the O’Reilly learning platform and Miso’s news and media AI lab have worked closely to build a solution capable of reliably answering questions for learners, crediting the sources it used to generate its answers, and then paying royalties to those sources for their contributions. And with the latest release of O’Reilly Answers, the idea of a royalties engine that fairly pays creators is now a practical day-to-day reality—and core to the success of the two organizations’ partnership and continued growth together.

How O’Reilly Answers Came to BeO’Reilly is a technology-focused learning platform that supports the continuous learning of tech teams. It offers a wealth of books, on-demand courses, live events, short-form posts, interactive labs, expert playlists, and more—formed from the proprietary content of thousands of independent authors, industry experts, and several of the largest education publishers in the world. To nurture and sustain the knowledge of its members, O’Reilly pays royalties out of the subscription revenues generated based on how its learners engage with and use the works of experts on the learning platform. The organization has a clear redline: never infringe on the livelihoods of creators and their works.

While the O’Reilly learning platform provides learners with a wonderful abundance of content, the sheer volume of information (and the limitations of keyword search) at times overwhelmed readers trying to sift through it to find exactly what they needed to know. And the consequence was that this rich expertise remained trapped within a book, behind a link, within a chapter, or buried in a video, perhaps never to be seen. The platform required a more effective way to connect learners directly to the key information that they sought. Enter the team at Miso.

Miso’s cofounders, Lucky Gunasekara and Andy Hsieh, are veterans of the Small Data Lab at Cornell Tech, which is devoted to private AI approaches for immersive personalization and content-centric explorations. They expanded their work at Miso to build easily tappable infrastructure for publishers and websites with advanced AI models for search, discovery, and advertising that could go toe-to-toe in quality with the giants of Big Tech. And Miso had already built an early LLM-based search engine using the open-source BERT model that delved into research papers—it could take a query in natural language and find a snippet of text in a document that answered that question with surprising reliability and smoothness. That early work led to the collaboration with O’Reilly to help solve the learning-specific search and discovery challenges on its learning platform.

What resulted was O’Reilly’s first LLM search engine, the original O’Reilly Answers. You can read a bit about its internal workings, but in essence, it was a RAG engine minus the “G” for “generative.” Thanks to BERT being open source, the team at Miso was able to fine-tune Answers’ query understanding capabilities against thousands upon thousands of question-answer pairs in online learning to make it expert-level at understanding questions and searching for snippets whose context and content were relevant to those questions. At the same time, Miso went about an in-depth chunking and metadata-mapping of every book in the O’Reilly catalog to generate enriched vector snippet embeddings of each work. Paragraph by paragraph, deep metadata was generated showing where each snippet was sourced, from the title text, chapter, sections, and subsections down to the nearest code or figures in a book.

The marriage of this specialized Q&A model with this enriched vector store of O’Reilly content meant that readers could ask a question and get an answer directly sourced from O’Reilly’s library of titles—with the snippet answer highlighted directly within the text and a deep link citation to the source. And because there was a clear data pipeline for every answer this engine retrieved, O’Reilly had the forensics on hand to pay royalties for each answer delivered in order to fairly compensate the company’s community of authors for delivering direct value to learners.

How O’Reilly Answers Has EvolvedFlash forward to today, and Miso and O’Reilly have taken that system and the values behind it even further. If the original Answers release was a LLM-driven retrieval engine, today’s new version of Answers is an LLM-driven research engine (in the truest sense). After all, research is only as good as your references, and the teams at both organizations acutely understood that the possibility of hallucinations and ungrounded answers could outright confuse and frustrate learners. So Miso’s team spent months doing internal R&D on how to better ground and verify answers—in the process, they found that they could attain increasingly good performance by adapting multiple models to work with one another.

In essence, the latest O’Reilly Answers release is an assembly line of LLM workers. Each has its own discrete expertise and skill set, and they work together to collaborate as they take in a question or query, reason what the intent is, research the possible answers, and critically evaluate and analyze this research before writing a citation-backed grounded answer. To be clear, this new Answers release is not a massive LLM that has been trained on authors’ content and works. Miso’s team shares O’Reilly’s belief in not developing LLMs without credit, consent, and compensation from creators. And they’ve learned through their daily work not just with O’Reilly but with publishers such as Macworld, CIO.com, America’s Test Kitchen, and Nursing Times that there’s much more value to training LLMs to be experts at reasoning on expert content than by training them to generatively regurgitate that expert content in response to a prompt.

The net result is that O’Reilly Answers can now critically research and answer questions in a much richer and more immersive long-form response while preserving the citations and source references that were so important in its original release.

The newest Answers release is again built with an open source model—in this case, Llama 3. This means that the specialized library of models for expert research, reasoning, and writing is fully private. And again, while the models are fine-tuned to complete their tasks at an expert level, they’re unable to reproduce authors’ works in full. The teams at O’Reilly and Miso are excited by the potential of open source LLMs because their rapid evolution means bringing newer breakthroughs to learners while controlling what these models can and can’t do with O’Reilly content and data.

The benefit of constructing Answers as a pipeline of research, reasoning, and writing using today’s leading open source LLMs is that the robustness of the questions it can answer will continue to increase, but the system itself will always be grounded in authoritative original expert commentary from content on the O’Reilly learning platform. Every answer still contains citations for learners to dig deeper, and care has been taken to ensure the language remains as close as possible to what experts originally shared. And when a question goes beyond the limits of possible citations, the tool will simply reply “I don’t know” rather than risk hallucinating.

Most importantly, just like with the original version of Answers, the architecture for the latest release provides forensic data that shows the contribution of every referenced author’s work in an answer. This allows O’Reilly to pay experts for their work with a first-of-its-kind generative AI royalty while simultaneously allowing them to share their knowledge more easily and directly with the community of global learners the O’Reilly platform is built to serve.

Expect more updates soon as O’Reilly and Miso push to get to compilable code samples in answers and more conversational and generative capabilities. They’re already working on future Answers releases and would love to hear feedback and suggestions on what they can build next.

View Details

Learn more from the Generative AI Success Stories Superstream on June 12th.

At O’Reilly, we’re not just building training materials about AI. We’re also using it to build new kinds of learning experiences. One of the ways we are putting AI to work is our update to Answers. Answers is a generative AI-powered feature that aims to answer questions in the flow of learning. It’s in every book, on-demand course, and video and will eventually be available across our entire learning platform. To see it, click the “Answers” icon (the last item in the list at the right side of the screen).

Answers enables active learning: interacting with content by asking questions and getting answers rather than simply ingesting a stream from a book or video. If you’re solving a problem for work, it puts learning in the flow of work. It is natural to have questions while you’re working on something; those of us who remember hardcopy books also remember having a stack of books open upside down on our desks (to save the page) as we got deeper and deeper into researching a problem. Something similar happens online: you open so many tabs while searching for an answer that you can’t remember which is which. Why can’t you just ask a question and get an answer? Now you can.

Here are a few insights into the decisions that we made in the process of building Answers. Of course, everything is subject to change; that’s the first thing you need to realize before starting any AI project. This is unknown territory; everything is an experiment. You won’t know how people will use your application until you build it and deploy it; there are many questions about Answers for which we are still awaiting answers. It is important to be careful when deploying an AI application, but it’s also important to realize that all AI is experimental.

The core of Answers was built through collaboration with a partner that provided the AI expertise. That’s an important principle, especially for small companies: don’t build by yourself when you can partner with others. It would have been very difficult to develop the expertise to build and train a model, and much more effective to work with a company that already has that expertise. There will be plenty of decisions and problems for your staff to make and solve. At least for the first few products, leave the heavy AI lifting to someone else. Focus on understanding the problem you are solving. What are your specific use cases? What kinds of answers will your users expect? What kind of answers do you want to deliver? Think about how the answers to those questions affect your business model.

If you build a chat-like service, you must think seriously about how it will be used: what kinds of prompts to expect and what kinds of answers to return. Answers places few restrictions on the questions you can ask. While most users think of O’Reilly as a resource for software developers and IT departments, our platform contains many other kinds of information. Answers is able to answer questions about topics like chemistry, biology, and climate change—anything that’s on our platform. However, it differs from chat applications like ChatGPT in several ways. First, it’s limited to questions and answers. Although it suggests followup questions, it’s not conversational. Each new question starts a new context. We believe that many companies experimenting with AI want to be conversational for the sake of conversation, not a means to their end—possibly with the goal of monopolizing their users’ attention. We want our users to learn; we want our users to get on with solving their technical problems. Conversation for its own sake doesn’t fit this use case. We want interactions to be short, direct, and to the point.

Limiting Answers to Q&A also minimizes abuse; it’s harder to lead an AI system “off the rails” when you’re limited to Q&A. (Honeycomb, one of the first companies to integrate ChatGPT into a software product, made a similar decision.)

Unlike many AI-driven products, Answers will tell you when it genuinely doesn’t have an answer. For example, if you ask it “Who won the world series?” it will reply “I don’t have enough information to answer this question.” If you ask a question that it can’t answer but on which our platform may have relevant information, it will point you to that information. This design decision was simple but surprisingly important. Very few AI systems will tell you that they can’t answer the question, and that inability is an important source of hallucinations, errors, and other kinds of misinformation. Most AI engines can’t say “Sorry, I don’t know.” Ours can and will.

Answers are always attributed to specific content, which allows us to compensate our talent and our partner publishers. Designing the compensation plan was a significant part of the project. We are committed to treating authors fairly—we won’t just generate answers from their content. When a user asks a question, Answers generates a short response and provides links to the resources from which it pulled the information. This data goes to our compensation model, which is designed to be revenue-neutral. It doesn’t penalize our talent when we generate answers from their material.

The design of Answers is more complex than you might expect—and it’s important for organizations starting an AI project to understand that “the simplest thing that might possibly work” probably won’t work. From the start, we knew that we couldn’t simply use a model like GPT or Gemini. In addition to being error-prone, they don’t have any mechanism for providing data about how they built an answer, data that we need as input to our compensation model. That pushed us immediately towards the retrieval-augmented generation pattern (RAG), which provided a solution. With RAG, a program generates a prompt that includes both the question and the data needed to answer the question. That augmented prompt is sent to the language model, which provides an answer. We can compensate our talent because we know what data was used to build the answer.

Using RAG begs the question: Where do the documents come from? Another AI model that has access to a database of our platform’s content to generate “candidate” documents. Yet another model ranks the candidates, selecting those that seem most useful; and a third model reevaluates each candidate to ensure that they are actually relevant and useful. Finally, the selected documents are trimmed to minimize content that’s unrelated to the question. This process has two purposes: it minimizes hallucination and the data sent to the model answering the question; it also minimizes the context required. The more context that’s required, the longer it takes to get an answer, and the more it costs to run the model. Most of the models we use are small open source models. They’re fast, effective, and inexpensive.

In addition to minimizing hallucination and making it possible to attribute content to creators (and from there, assign royalties), this design makes it easy to add new content. We are constantly adding new content to the platform: thousands of items per year. With a model like GPT, adding content would require a lengthy and expensive training process. With RAG, adding content is trivial. When anything is added to the platform, it is added to the database from which relevant content is chosen. This process isn’t computationally intensive and can take place almost immediately—in real time, as it were. Answers never lags the rest of the platform. Users will never see “This model has only been trained on data through July 2023.”

Answers is one product, but it’s only one piece of an ecosystem of tools that we’re building. All of these tools are designed to serve the learning experience: to help our users and our corporate clients develop the skills they need to stay relevant in a changing world. That’s the goal—and it’s also the key to building successful applications with generative AI. What is the real goal? It’s not to impress your customers with your AI expertise. It’s to solve some problem. In our case, that problem is helping students to acquire new skills more efficiently. Focus on that goal, not on the AI. The AI will be an important tool—maybe the most important tool. But it’s not an end in itself.

View Details

We previously shared our insights on the tactics we have honed while operating LLM applications. Tactics are granular: they are the specific actions employed to achieve specific objectives. We also shared our perspective on operations: the higher-level processes in place to support tactical work to achieve objectives.

But where do those objectives come from? That is the domain of strategy. Strategy answers the “what” and “why” questions behind the “how” of tactics and operations.

We provide our opinionated takes, such as “no GPUs before PMF” and “focus on the system not the model”, to help teams figure out where to allocate scarce resources. We also suggest a roadmap for iterating towards a great product. This final set of lessons answers the following questions:

1. Building vs. Buying: When should you train your own models, and when should you leverage existing APIs? The answer is, as always, “it depends”. We share what it depends on.
2. Iterating to Something Great: How can you create a lasting competitive edge that goes beyond just using the latest models? We discuss the importance of building a robust system around the model and focusing on delivering memorable, sticky experiences.
3. Human-Centered AI: How can you effectively integrate LLMs into human workflows to maximize productivity and happiness? We emphasize the importance of building AI tools that support and enhance human capabilities rather than attempting to replace them entirely.
4. Getting Started: What are the essential steps for teams embarking on building an LLM product? We outline a basic playbook that starts with prompt engineering, evaluations, and data collection.
5. The Future of Low-Cost Cognition: How will the rapidly decreasing costs and increasing capabilities of LLMs shape the future of AI applications? We examine historical trends and walk through a simple method to estimate when certain applications might become economically feasible.
6. From Demos to Products: What does it take to go from a compelling demo to a reliable, scalable product? We emphasize the need for rigorous engineering, testing, and refinement to bridge the gap between prototype and production.

To answer these difficult questions, let’s think step by step…

Strategy: Building with LLMs without Getting Out-ManeuveredSuccessful products require thoughtful planning and tough prioritization, not endless prototyping or following the latest model releases or trends. In this final section, we look around the corners and think about the strategic considerations for building great AI products. We also examine key trade-offs teams will face, like when to build and when to buy, and suggest a “playbook” for early LLM application development strategy.

No GPUs before PMFTo be great, your product needs to be more than just a thin wrapper around somebody else’s API. But mistakes in the opposite direction can be even more costly. The past year has also seen a mint of venture capital, including an eye-watering six billion dollar Series A, spent on training and customizing models without a clear product vision or target market. In this section, we’ll explain why jumping immediately to training your own models is a mistake and consider the role of self-hosting.

Training from scratch (almost) never makes senseFor most organizations, pre-training an LLM from scratch is an impractical distraction from building products.

As exciting as it is and as much as it seems like everyone else is doing it, developing and maintaining machine learning infrastructure takes a lot of resources. This includes gathering data, training and evaluating models, and deploying them. If you’re still validating product-market fit, these efforts will divert resources from developing your core product. Even if you had the compute, data, and technical chops, the pretrained LLM may become obsolete in months.

Consider the case of BloombergGPT, an LLM specifically trained for financial tasks. The model was pretrained on 363B tokens and required a heroic effort by nine full-time employees, four from AI Engineering and five from ML Product and Research. Despite this effort, it was outclassed by gpt-3.5-turbo and gpt-4 on those financial tasks within a year.

This story and others like it suggests that for most practical applications, pretraining an LLM from scratch, even on domain-specific data, is not the best use of resources. Instead, teams are better off fine-tuning the strongest open-source models available for their specific needs.

There are of course exceptions. One shining example is Replit’s code model, trained specifically for code-generation and understanding. With pretraining, Replit was able to outperform other models of large sizes such as CodeLlama7b. But as other, increasingly capable models have been released, maintaining utility has required continued investment.

Don’t fine-tune until you’ve proven it’s necessaryFor most organizations, fine-tuning is driven more by FOMO than by clear strategic thinking.

Organizations invest in fine-tuning too early, trying to beat the “just another wrapper” allegations. In reality, fine-tuning is heavy machinery, to be deployed only after you’ve collected plenty of examples that convince you other approaches won’t suffice.

A year ago, many teams were telling us they were excited to fine-tune. Few have found product-market fit and most regret their decision. If you’re going to fine tune, you’d better be really confident that you’re set up to do it again and again as base models improve—see the “The model isn’t the product” and “Build LLMOps” below.

When might fine-tuning actually be the right call? If the use-case requires data not available in the mostly-open web-scale datasets used to train existing models—and if you’ve already built an MVP that demonstrates the existing models are insufficient. But be careful: if great training data isn’t readily available to the model builders, where are you getting it?

Ultimately, remember that LLM-powered applications aren’t a science fair project, investment in them should be commensurate with their contribution to your business’ strategic objectives and its competitive differentiation.

Start with inference APIs, but don’t be afraid of self-hostingWith LLM APIs, it’s easier than ever for startups to adopt and integrate language modeling capabilities without training their own models from scratch. Providers like Anthropic, and OpenAI offer general APIs that can sprinkle intelligence into your product with just a few lines of code. By using these services, you can reduce the effort spent and instead focus on creating value for your customers—this allows you to validate ideas and iterate towards product-market fit faster.

But, as with databases, managed services aren’t the right fit for every use case, especially as scale and requirements increase. Indeed, self-hosting may be the only way to use models without sending confidential/private data out of your network, as required in regulated industries like healthcare and finance, or by contractual obligations or confidentiality requirements.

Furthermore, self-hosting circumvents limitations imposed by inference providers, like rate limits, model deprecations, and usage restrictions. In addition, self-hosting gives you complete control over the model, making it easier to construct a differentiated, high quality system around it. Finally, self-hosting, especially of finetunes, can reduce cost at large scale. For example, Buzzfeed shared how they finetuned open-source LLMs to reduce costs by 80%.

Iterate to something greatTo sustain a competitive edge in the long run, you need to think beyond models and consider what will set your product apart. While speed of execution matters, it shouldn’t be your only advantage.

The model isn’t the product, the system around it isFor teams that aren’t building models, the rapid pace of innovation is a boon as they migrate from one SOTA model to the next, chasing gains in context size, reasoning capability, and price-to-value to build better and better products.

This progress is as exciting as it is predictable. Taken together, this means models are likely to be the least durable component in the system.

Instead, focus your efforts on what’s going to provide lasting value, such as:

• Evaluation chassis: To reliably measure performance on your task across models
• Guardrails: To prevent undesired outputs no matter the model
• Caching: To reduce latency and cost by avoiding the model altogether
• Data flywheel: To power the iterative improvement of everything above

These components create a thicker moat of product quality than raw model capabilities.

But that doesn’t mean building at the application layer is risk-free. Don’t point your shears at the same yaks that OpenAI or other model providers will need to shave if they want to provide viable enterprise software.

For example, some teams invested in building custom tooling to validate structured output from proprietary models; minimal investment here is important, but a deep one is not a good use of time. OpenAI needs to ensure that when you ask for a function call, you get a valid function call—because all of their customers want this. Employ some “strategic procrastination” here, build what you absolutely need, and await the obvious expansions to capabilities from providers.

Build trust by starting smallBuilding a product that tries to be everything to everyone is a recipe for mediocrity. To create compelling products, companies need to specialize in building memorable, sticky experiences that keep users coming back.

Consider a generic RAG system that aims to answer any question a user might ask. The lack of specialization means that the system can’t prioritize recent information, parse domain-specific formats, or understand the nuances of specific tasks. As a result, users are left with a shallow, unreliable experience that doesn’t meet their needs.

To address this, focus on specific domains and use cases. Narrow the scope by going deep rather than wide. This will create domain-specific tools that resonate with users. Specialization also allows you to be upfront about your system’s capabilities and limitations. Being transparent about what your system can and cannot do demonstrates self-awareness, helps users understand where it can add the most value, and thus builds trust and confidence in the output.

Build LLMOps, but build it for the right reason: faster iterationDevOps is not fundamentally about reproducible workflows or shifting left or empowering two pizza teams—and it’s definitely not about writing YAML files.

DevOps is about shortening the feedback cycles between work and its outcomes so that improvements accumulate instead of errors. Its roots go back, via the Lean Startup movement, to Lean manufacturing and the Toyota Production System, with its emphasis on Single Minute Exchange of Die and Kaizen.

MLOps has adapted the form of DevOps to ML. We have reproducible experiments and we have all-in-one suites that empower model builders to ship. And Lordy, do we have YAML files.

But as an industry, MLOps didn’t adapt the function of DevOps. It didn’t shorten the feedback gap between models and their inferences and interactions in production.

Hearteningly, the field of LLMOps has shifted away from thinking about hobgoblins of little minds like prompt management and towards the hard problems that block iteration: production monitoring and continual improvement, linked by evaluation.

Already, we have interactive arenas for neutral, crowd-sourced evaluation of chat and coding models—an outer loop of collective, iterative improvement. Tools like LangSmith, Log10, LangFuse, W&B Weave, HoneyHive, and more promise to not only collect and collate data about system outcomes in production, but also to leverage them to improve those systems by integrating deeply with development. Embrace these tools or build your own.

Don’t build LLM features you can buyMost successful businesses are not LLM businesses. Simultaneously, most businesses have opportunities to be improved by LLMs.

This pair of observations often misleads leaders into hastily retrofitting systems with LLMs at increased cost and decreased quality and releasing them as ersatz, vanity “AI” features, complete with the now-dreaded sparkle icon. There’s a better way: focus on LLM applications that truly align with your product goals and enhance your core operations.

Consider a few misguided ventures that waste your team’s time:

• Building custom text-to-SQL capabilities for your business.
• Building a chatbot to talk to your documentation.
• Integrating your company’s knowledge base with your customer support chatbot.

While the above are the hellos-world of LLM applications, none of them make sense for virtually any product company to build themselves. These are general problems for many businesses with a large gap between promising demo and dependable component—the customary domain of software companies. Investing valuable R&D resources on general problems being tackled en masse by the current Y Combinator batch is a waste.

If this sounds like trite business advice, it’s because in the frothy excitement of the current hype wave, it’s easy to mistake anything “LLM” as cutting-edge, accretive differentiation, missing which applications are already old hat.

AI in the loop; humans at the centerRight now, LLM-powered applications are brittle. They required an incredible amount of safe-guarding, defensive engineering, and remain hard to predict. Additionally, when tightly scoped these applications can be wildly useful. This means that LLMs make excellent tools to accelerate user workflows.

While it may be tempting to imagine LLM-based applications fully replacing a workflow, or standing in for a job-function, today the most effective paradigm is a human-computer centaur (c.f. Centaur chess). When capable humans are paired with LLM capabilities tuned for their rapid utilization, productivity and happiness doing tasks can be massively increased. One of the flagship applications of LLMs, GitHub CoPilot, demonstrated the power of these workflows:

“Overall, developers told us they felt more confident because coding is easier, more error-free, more readable, more reusable, more concise, more maintainable, and more resilient with GitHub Copilot and GitHub Copilot Chat than when they’re coding without it.” – Mario Rodriguez, GitHub

For those who have worked in ML for a long time, you may jump to the idea of “human-in-the-loop”, but not so fast: HITL Machine Learning is a paradigm built on Human experts ensuring that ML models behave as predicted. While related, here we are proposing something more subtle. LLM driven systems should not be the primary drivers of most workflows today, they should merely be a resource.

By centering humans, and asking how an LLM can support their workflow, this leads to significantly different product and design decisions. Ultimately, it will drive you to build different products than competitors who try to rapidly offshore all responsibility to LLMs; better, more useful, and less risky products.

Start with prompting, evals, and data collectionThe previous sections have delivered a firehose of techniques and advice. It’s a lot to take in. Let’s consider the minimum useful set of advice: if a team wants to build an LLM product, where should they begin?

Over the last year, we’ve seen enough examples to start becoming confident that successful LLM applications follow a consistent trajectory. We walk through this basic “getting started” playbook in this section. The core idea is to start simple and only add complexity as needed. A decent rule of thumb is that each level of sophistication typically requires at least an order of magnitude more effort than the one before it. With this in mind…

Prompt engineering comes firstStart with prompt engineering. Use all the techniques we discussed in the tactics section before. Chain-of-thought, n-shot examples, and structured input and output are almost always a good idea. Prototype with the most highly capable models before trying to squeeze performance out of weaker models.

Only if prompt engineering cannot achieve the desired level of performance should you consider fine-tuning. This will come up more often if there are non-functional requirements (e.g., data privacy, complete control, cost) that block the use of proprietary models and thus require you to self-host. Just make sure those same privacy requirements don’t block you from using user data for fine-tuning!

Build evals and kickstart a data flywheelEven teams that are just getting started need evals. Otherwise, you won’t know whether your prompt engineering is sufficient or when your fine-tuned model is ready to replace the base model.

Effective evals are specific to your tasks and mirror the intended use cases. The first level of evals that we recommend is unit testing. These simple assertions detect known or hypothesized failure modes and help drive early design decisions. Also see other task-specific evals for classification, summarization, etc.

While unit tests and model-based evaluations are useful, they don’t replace the need for human evaluation. Have people use your model/product and provide feedback. This serves the dual purpose of measuring real-world performance and defect rates while also collecting high-quality annotated data that can be used to finetune future models. This creates a positive feedback loop, or data flywheel, which compounds over time:

• Human evaluation to assess model performance and/or find defects
• Use the annotated data to finetune the model or update the prompt
• Repeat

For example, when auditing LLM-generated summaries for defects we might label each sentence with fine-grained feedback identifying factual inconsistency, irrelevance, or poor style. We can then use these factual inconsistency annotations to train a hallucination classifier or use the relevance annotations to train a reward model to score on relevance. As another example, LinkedIn shared about their success with using model-based evaluators to estimate hallucinations, responsible AI violations, coherence, etc. in their write-up

By creating assets that compound their value over time, we upgrade building evals from a purely operational expense to a strategic investment, and build our data flywheel in the process.

The high-level trend of low-cost cognitionIn 1971, the researchers at Xerox PARC predicted the future: the world of networked personal computers that we are now living in. They helped birth that future by playing pivotal roles in the invention of the technologies that made it possible, from Ethernet and graphics rendering to the mouse and the window.

But they also engaged in a simple exercise: they looked at applications that were very useful (e.g. video displays) but were not yet economical (i.e. enough RAM to drive a video display was many thousands of dollars). Then they looked at historic price trends for that technology (a la Moore’s Law) and predicted when those technologies would become economical.

We can do the same for LLM technologies, even though we don’t have something quite as clean as transistors per dollar to work with. Take a popular, long-standing benchmark, like the Massively-Multitask Language Understanding dataset, and a consistent input approach (five-shot prompting). Then, compare the cost to run language models with various performance levels on this benchmark over time.

For a fixed cost, capabilities are rapidly increasing. For a fixed capability level, costs are rapidly decreasing. Created by co-author Charles Frye using public data on May 13, 2024.In the four years since the launch of OpenAI’s davinci model as an API, the cost for running a model with equivalent performance on that task at the scale of one million tokens (about one hundred copies of this document) has dropped from $20 to less than 10¢—a halving time of just six months. Similarly, the cost to run Meta’s LLaMA 3 8B via an API provider or on your own is just 20¢ per million tokens as of May of 2024, and it has similar performance to OpenAI’s text-davinci-003, the model that enabled ChatGPT to shock the world. That model also cost about $20 per million tokens when it was released in late November of 2023. That’s two orders of magnitude in just 18 months—the same timeframe in which Moore’s Law predicts a mere doubling.

Now, let’s consider an application of LLMs that is very useful (powering generative video game characters, a la Park et al) but is not yet economical (their cost was estimated at $625 per hour here). Since that paper was published in August of 2023, the cost has dropped roughly one order of magnitude, to $62.50 per hour. We might expect it to drop to $6.25 per hour in another nine months.

Meanwhile, when Pac-Man was released in 1980, $1 of today’s money would buy you a credit, good to play for a few minutes or tens of minutes—call it six games per hour, or $6 per hour. This napkin math suggests that a compelling LLM-enhanced gaming experience will become economical some time in 2025.

These trends are new, only a few years old. But there is little reason to expect this process to slow down in the next few years. Even as we perhaps use up low-hanging fruit in algorithms and datasets, like scaling past the “Chinchilla ratio” of ~20 tokens per parameter, deeper innovations and investments inside the data center and at the silicon layer promise to pick up slack.

And this is perhaps the most important strategic fact: what is a completely infeasible floor demo or research paper today will become a premium feature in a few years and then a commodity shortly after. We should build our systems, and our organizations, with this in mind.

Enough 0 to 1 Demos, It’s Time for 1 to N ProductsWe get it, building LLM demos is a ton of fun. With just a few lines of code, a vector database, and a carefully crafted prompt, we create magic . And in the past year, this magic has been compared to the internet, the smartphone, and even the printing press.

Unfortunately, as anyone who has worked on shipping real-world software knows, there’s a world of difference between a demo that works in a controlled setting and a product that operates reliably at scale.

Take, for example, self-driving cars. The first car was driven by a neural network in 1988. Twenty-five years later, Andrej Karpathy took his first demo ride in a Waymo. A decade after that, the company received its driverless permit. That’s thirty-five years of rigorous engineering, testing, refinement, and regulatory navigation to go from prototype to commercial product.

Across different parts of industry and academia, we have keenly observed the ups and downs for the past year: Year 1 of N for LLM applications. We hope that the lessons we have learned —from tactics like rigorous operational techniques for building teams to strategic perspectives like which capabilities to build internally—help you in year 2 and beyond, as we all build on this exciting new technology together.

About the authorsEugene Yan designs, builds, and operates machine learning systems that serve customers at scale. He’s currently a Senior Applied Scientist at Amazon where he builds RecSys for millions worldwide worldwide and applies LLMs to serve customers better. Previously, he led machine learning at Lazada (acquired by Alibaba) and a Healthtech Series A. He writes & speaks about ML, RecSys, LLMs, and engineering at eugeneyan.com and ApplyingML.com.

Bryan Bischof is the Head of AI at Hex, where he leads the team of engineers building Magic – the data science and analytics copilot. Bryan has worked all over the data stack leading teams in analytics, machine learning engineering, data platform engineering, and AI engineering. He started the data team at Blue Bottle Coffee, led several projects at Stitch Fix, and built the data teams at Weights and Biases. Bryan previously co-authored the book Building Production Recommendation Systems with O’Reilly, and teaches Data Science and Analytics in the graduate school at Rutgers. His Ph.D. is in pure mathematics.

Charles Frye teaches people to build AI applications. After publishing research in psychopharmacology and neurobiology, he got his Ph.D. at the University of California, Berkeley, for dissertation work on neural network optimization. He has taught thousands the entire stack of AI application development, from linear algebra fundamentals to GPU arcana and building defensible businesses, through educational and consulting work at Weights and Biases, Full Stack Deep Learning, and Modal.

Hamel Husain is a machine learning engineer with over 25 years of experience. He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies operationalize Large Language Models (LLMs) to accelerate their AI product journey.

Jason Liu is a distinguished machine learning consultant known for leading teams to successfully ship AI products. Jason’s technical expertise covers personalization algorithms, search optimization, synthetic data generation, and MLOps systems.

His experience includes companies like Stitchfix, where he created a recommendation framework and observability tools that handled 350 million daily requests. Additional roles have included Meta, NYU, and startups such as Limitless AI and Trunk Tools.

Shreya Shankar is an ML engineer and PhD student in computer science at UC Berkeley. She was the first ML engineer at 2 startups, building AI-powered products from scratch that serve thousands of users daily. As a researcher, her work focuses on addressing data challenges in production ML systems through a human-centered approach. Her work has appeared in top data management and human-computer interaction venues like VLDB, SIGMOD, CIDR, and CSCW.

Contact UsWe would love to hear your thoughts on this post. You can contact us at contact@applied-llms.org. Many of us are open to various forms of consulting and advisory. We will route you to the correct expert(s) upon contact with us if appropriate.

AcknowledgementsThis series started as a conversation in a group chat, where Bryan quipped that he was inspired to write “A Year of AI Engineering”. Then, magic happened in the group chat (see image below), and we were all inspired to chip in and share what we’ve learned so far.

The authors would like to thank Eugene for leading the bulk of the document integration and overall structure in addition to a large proportion of the lessons. Additionally, for primary editing responsibilities and document direction. The authors would like to thank Bryan for the spark that led to this writeup, restructuring the write-up into tactical, operational, and strategic sections and their intros, and for pushing us to think bigger on how we could reach and help the community. The authors would like to thank Charles for his deep dives on cost and LLMOps, as well as weaving the lessons to make them more coherent and tighter—you have him to thank for this being 30 instead of 40 pages! The authors appreciate Hamel and Jason for their insights from advising clients and being on the front lines, for their broad generalizable learnings from clients, and for deep knowledge of tools. And finally, thank you Shreya for reminding us of the importance of evals and rigorous production practices and for bringing her research and original results to this piece.

Finally, the authors would like to thank all the teams who so generously shared your challenges and lessons in your own write-ups which we’ve referenced throughout this series, along with the AI communities for your vibrant participation and engagement with this group.

View Details

May was a month of announcements: between Google, Apple, Microsoft, and OpenAI, there was much ado about—well, very little, in fact. It’s always seemed to me that big announcements steal attention that might otherwise go to projects that are less flashy but more deserving. (Or maybe I’m just becoming jaded.)

That’s not to say that nothing interesting happened. We’re seeing continued interest in small language models—small enough to run on cell phones (which have more processing power than the supercomputers of a few decades ago). We’ve wondered whether new programming languages make sense in the era of AI-generated code—but we saw Bend (for highly parallel code) and Jolie (for services), plus LuaX (a new Lua interpreter) and Faer (for high-performance numerics in Rust). And for web developers, someone has been using CSS Grid to typeset music. Programming of various sorts is very much alive.

AI The first two parts of the three-part series What We Learned from Year of Building with LLMs, have been posted on O’Reilly Radar. The third part will be posted on June 6. This series is an wide-ranging collection of wisdom and experience that will be essential to anyone building AI applications. * llama-fs is a filesystem based on Llama 3 that names and finds files for you. It’s a very interesting idea, though I’m not sure it’s one I would trust. * MonsterGPT is a tool on OpenAI’s GPT Marketplace for using ChatGPT to fine-tune smaller LLMs. You point it at the dataset (it can use datasets hosted on Hugging Face) and the model, and it does the rest. * Target Speech Hearing is a new system for noise canceling headphones that may allow the user to hear a single voice in a crowd; unwanted voices are canceled out. * Ambient Diffusion is a new training strategy for generative art that reduces the problem of reproducing works or styles that are in the training data. It trains models on corrupted versions of the initial training data, so that it is impossible to “memorize” any particular work. * Copilot+ PCs are personal computers with hardware capable of running AI applications, including neural processors and GPUs. Copilot+ PCs are intended to support AI features that are being integrated into Windows 11. * Meta has created a new family of mixed-modal models called Chameleon. Unlike multimodal models, which use different models for text and images, Chameleon is a single model and can freely integrate data from different modalities. * Here’s an implementation of Llama 3, in detail, from scratch. You need to download the weights from Meta. * Thom Wolf, one of Hugging Face’s cofounders, has published a list of books and articles to read if you want to get into AI. * GPT-4o can be used to aid in code reviews. It’s useful. But when it comes to real insight, it falls short. How many times do you want to be told to use longer variable names or write more comments? * A new brain interface device can convert thought into speech. * For better or for worse, Google is integrating generative AI into search. It has a serious problem with generating bad results, something that Google is trying to fix. Tom’s Hardware* shows how to disable AI-generated results. * Google has announced “Project Astra,” which adds interactive voice and vision to its models. It also announced that a future version of Gemini will have a two-million-token context window. Other announcements include Gemini Flash, a lightweight model to run on smaller devices, and Veo, a text-to-video model that’s said to be comparable to Sora. * The latest version of GPT, GPT-4o, adds real-time interactive voice, vision, and emotional analysis capabilities. Latency on voice input has been reduced to 3.2 seconds. * OpenAI has released a draft proposal for Model Specs, which provide a way to specify the desired behavior for a model. Model specifications look like an interesting supplement to—though not a replacement for—model cards. * KnowHalu is a new framework for detecting hallucinations in large language model output. * A new, three-part series on AI safety is starting. It’s basic and looks reasonably well-balanced. Right now, only the first part has been written. * Can AI forget? Ben Lorica writes about unlearning, the process by which information can be removed from a pretrained model. Unlearning will be important for many reasons, not the least of which is European regulations about removing incorrect personal data. * Georgia Tech and Meta have created an open dataset of climate data to train AI for carbon capture systems. * Apple has released its OpenELM language models. These models are all relatively small (270M-3B parameters) and designed to run on mobile devices. Source code is available on Hugging Face; they are licensed under the Apple Sample Code License. * Snowflake-arctic-instruct is a new language model. It claims to be the largest truly open source model (128×3.66 parameter mixture of experts).

Programming* LuaX (Lua eXtended) is a new interpreter for the Lua programming language that can compile standalone executables. * Google has released Firebase Genkit support for its Gemma models. This framework allows JavaScript developers to create Node.js backends for integrating the Gemma language model into applications. Support for Go is promised soon. * Not useful but cool: a group at the University of Michigan has created spectrograms that look like images but that can be played as sound. * Bend is a new high-level programming language for generating highly parallel code. The code can run on multicore CPUs or on GPUs. Bend looks and feels like Python, but it automatically detects opportunities for parallelism. * Red Hat has made Red Hat Enterprise Linux (RHEL) bootable as a container image. This makes it easier to use RHEL in the context of modern cloud native development. * Patchwork attempts to extend Git-like source control from software to written texts and other artifacts. One possible application would be to help integrate human writers and AI assistants. More generally, its developers are interested in creating local-first collaborative data layers. * Jolie is a new programming language that’s designed for developing services, as opposed to functions or objects. It stresses contracts, which define the relationship between the user and the service. It’s ideal for designing APIs and microservices. * The Graph Query Language (GQL) is a new ISO standard for querying graph databases, putting it on a par with SQL. * Faer is a new Rust library for linear algebra. A good linear algebra library is a basic requirement for numerical computation, including machine learning and artificial intelligence. * A new Linux distribution, with the unfortunate name EB corbos Linux for Safety Applications, supports the automotive industry’s functional safety requirements, meaning that it can be used in embedded systems on automobiles.

Web* PHP can now run in WebAssembly. This includes the most popular frameworks: WordPress, Laravel, and Symfony. * There have been many posts on de-enshittifying the web: Molly White, Anil Dash, Maria Farrell and Robin Berjon, among others. All of them are about reclaiming agency on a web that has become dominated by robber barons, rent takers, and walled gardens. * A CSS masterpiece: displaying musical scores with CSS Grid. The typography leaves a bit to be desired, but that’s fixable. A bigger problem is that there isn’t a good way of representing the musical score to be displayed. * Penpot is an open-sourced, web-based collaborative design tool. If you think that sounds like an open source version of Figma, you’re right.

Security* An XSS vulnerability in GitLab allows one-click account takeover. * LastPass will start encrypting the URLs of the sites to which users login. These URLs aren’t particularly sensitive, but encryption is an important step toward a zero-knowledge design. * Something new to disable: Windows 11 is adding a “recall” feature that saves everything that takes place on the computer and allows applications to restore previous state. Recall is a major threat to security and privacy. Microsoft claims that content remains local, but that’s a song we’ve heard before. * Apple and Google have united on a standard for detecting Bluetooth tracking devices that are used for stalking users. * AI adoption by criminals is still relatively low, but real. Most of the activity focuses around jailbreaks for legitimate LLMs (jailbreak as a service) and deep fakes. There are a fair number of fraudulent datasets. So far, there is only one LLM trained for criminal applications. * TunnelVision, a newly discovered attack against virtually all VPNs allows the attacker to route the victim’s unencrypted traffic through the attacker’s servers. While this is called “new,” the vulnerability has existed since 2002. * Microsoft has proposed Zero Trust DNS (ZTDNS), a framework that claims to solve many of the security issues DNS has had over the years. All communications are encrypted. Resolvers are only allowed to resolve names that are explicitly allowed. It is unclear whether ZTDNS will be a Windows-only or an Enterprise-only solution. It is now in private preview. * A change in the mechanism for changing passwords has made GitLab vulnerable to account hijacking. In turn, a hijacked account could be use to plant vulnerabilities that compromise software supply chains. * The UK has banned guessable default passwords on IoT devices. Vendors can still sell devices with default passwords, but each password must be unique. * If you want to understand the xz attack in detail, here’s a guest lecture from Columbia. It includes a live demo.

Augmented and Virtual Reality* Researchers have developed augmented reality glasses that look like regular glasses rather than a helmet. They rely on holography to produce full color 3D images. While it’s unclear whether this will ever become a product, it’s exactly what AR needs to succeed. * Stability AI has released Stable Video 3D, which generates a 3D image from a single 2D image.

Design* Poor design has consequences: at least 11 people are running for president of Iceland who had no idea that they were running. The same confusing web page is used to endorse a candidate and to register your own candidacy. * IF has been curating a catalog of design patterns for AI. It’s a great source for people who are designing AI systems and who need to build services that their users will trust.

Robotics* Cylon is a JavaScript framework for robotics and the Internet of Things. If you want to use Node.js when you’re programming robots, now you can. * An autonomous AI-enabled robot has designed, built, and tested a 3D object that is currently the world’s best shock absorber. It absorbs 75% of the energy used to crush it. * The incorporation of AI into robotics means roboticists need new sources of data. Where will that data come from (paywall)? 3D data is preferable, but slow and expensive to develop. Online videos?

View Details

Read Part I of this series here and stay tuned for Part III.
To hear directly from the authors on this topic, sign up for the upcoming virtual event on June 20th, and learn more from the Generative AI Success Stories Superstream on June 12th.

A possibly apocryphal quote attributed to many leaders reads: “Amateurs talk strategy and tactics. Professionals talk operations.” Where the tactical perspective sees a thicket of sui generis problems, the operational perspective sees a pattern of organizational dysfunction to repair. Where the strategic perspective sees an opportunity, the operational perspective sees a challenge worth rising to.

In part 1 of this essay, we introduced the tactical nuts and bolts of working with LLMs. In the next part, we will zoom out to cover the long-term strategic considerations. In this part, we discuss the operational aspects of building LLM applications that sit between strategy and tactics and bring rubber to meet roads.

Operating an LLM application raises some questions that are familiar from operating traditional software systems, often with a novel spin to keep things spicy. LLM applications also raise entirely new questions. We split these questions, and our answers, into four parts: data, models, product, and people.

For data, we answer: How and how often should you review LLM inputs and outputs? How do you measure and reduce test-prod skew?

For models, we answer: How do you integrate language models into the rest of the stack? How should you think about versioning models and migrating between models and versions?

For product, we answer: When should design be involved in the application development process, and why is it “as early as possible”? How do you design user experiences with rich human-in-the-loop feedback? How do you prioritize the many conflicting requirements? How do you calibrate product risk?

And finally, for people, we answer: Who should you hire to build a successful LLM application, and when should you hire them? How can you foster the right culture, one of experimentation? How should you use emerging LLM applications to build your own LLM application? Which is more critical: process or tooling?

As an AI language model, I do not have opinions and so cannot tell you whether the introduction you provided is “goated or nah.” However, I can say that the introduction properly sets the stage for the content that follows.

Operations: Developing and Managing LLM Applications and the Teams That Build ThemDataJust as the quality of ingredients determines the dish’s taste, the quality of input data constrains the performance of machine learning systems. In addition, output data is the only way to tell whether the product is working or not. All the authors focus tightly on the data, looking at inputs and outputs for several hours a week to better understand the data distribution: its modes, its edge cases, and the limitations of models of it.

Check for development-prod skewA common source of errors in traditional machine learning pipelines is train-serve skew. This happens when the data used in training differs from what the model encounters in production. Although we can use LLMs without training or fine-tuning, hence there’s no training set, a similar issue arises with development-prod data skew. Essentially, the data we test our systems on during development should mirror what the systems will face in production. If not, we might find our production accuracy suffering.

LLM development-prod skew can be categorized into two types: structural and content-based. Structural skew includes issues like formatting discrepancies, such as differences between a JSON dictionary with a list-type value and a JSON list, inconsistent casing, and errors like typos or sentence fragments. These errors can lead to unpredictable model performance because different LLMs are trained on specific data formats, and prompts can be highly sensitive to minor changes. Content-based or “semantic” skew refers to differences in the meaning or context of the data.

As in traditional ML, it’s useful to periodically measure skew between the LLM input/output pairs. Simple metrics like the length of inputs and outputs or specific formatting requirements (e.g., JSON or XML) are straightforward ways to track changes. For more “advanced” drift detection, consider clustering embeddings of input/output pairs to detect semantic drift, such as shifts in the topics users are discussing, which could indicate they are exploring areas the model hasn’t been exposed to before.

When testing changes, such as prompt engineering, ensure that holdout datasets are current and reflect the most recent types of user interactions. For example, if typos are common in production inputs, they should also be present in the holdout data. Beyond just numerical skew measurements, it’s beneficial to perform qualitative assessments on outputs. Regularly reviewing your model’s outputs—a practice colloquially known as “vibe checks”—ensures that the results align with expectations and remain relevant to user needs. Finally, incorporating nondeterminism into skew checks is also useful—by running the pipeline multiple times for each input in our testing dataset and analyzing all outputs, we increase the likelihood of catching anomalies that might occur only occasionally.

Look at samples of LLM inputs and outputs every dayLLMs are dynamic and constantly evolving. Despite their impressive zero-shot capabilities and often delightful outputs, their failure modes can be highly unpredictable. For custom tasks, regularly reviewing data samples is essential to developing an intuitive understanding of how LLMs perform.

Input-output pairs from production are the “real things, real places” (genchi genbutsu) of LLM applications, and they cannot be substituted. Recent research highlighted that developers’ perceptions of what constitutes “good” and “bad” outputs shift as they interact with more data (i.e., criteria drift). While developers can come up with some criteria upfront for evaluating LLM outputs, these predefined criteria are often incomplete. For instance, during the course of development, we might update the prompt to increase the probability of good responses and decrease the probability of bad ones. This iterative process of evaluation, reevaluation, and criteria update is necessary, as it’s difficult to predict either LLM behavior or human preference without directly observing the outputs.

To manage this effectively, we should log LLM inputs and outputs. By examining a sample of these logs daily, we can quickly identify and adapt to new patterns or failure modes. When we spot a new issue, we can immediately write an assertion or eval around it. Similarly, any updates to failure mode definitions should be reflected in the evaluation criteria. These “vibe checks” are signals of bad outputs; code and assertions operationalize them. Finally, this attitude must be socialized, for example by adding review or annotation of inputs and outputs to your on-call rotation.

Working with modelsWith LLM APIs, we can rely on intelligence from a handful of providers. While this is a boon, these dependencies also involve trade-offs on performance, latency, throughput, and cost. Also, as newer, better models drop (almost every month in the past year), we should be prepared to update our products as we deprecate old models and migrate to newer models. In this section, we share our lessons from working with technologies we don’t have full control over, where the models can’t be self-hosted and managed.

Generate structured output to ease downstream integrationFor most real-world use cases, the output of an LLM will be consumed by a downstream application via some machine-readable format. For example, Rechat, a real-estate CRM, required structured responses for the frontend to render widgets. Similarly, Boba, a tool for generating product strategy ideas, needed structured output with fields for title, summary, plausibility score, and time horizon. Finally, LinkedIn shared about constraining the LLM to generate YAML, which is then used to decide which skill to use, as well as provide the parameters to invoke the skill.

This application pattern is an extreme version of Postel’s law: be liberal in what you accept (arbitrary natural language) and conservative in what you send (typed, machine-readable objects). As such, we expect it to be extremely durable.

Currently, Instructor and Outlines are the de facto standards for coaxing structured output from LLMs. If you’re using an LLM API (e.g., Anthropic, OpenAI), use Instructor; if you’re working with a self-hosted model (e.g., Hugging Face), use Outlines.

Migrating prompts across models is a pain in the assSometimes, our carefully crafted prompts work superbly with one model but fall flat with another. This can happen when we’re switching between various model providers, as well as when we upgrade across versions of the same model.

For example, Voiceflow found that migrating from gpt-3.5-turbo-0301 to gpt-3.5-turbo-1106 led to a 10% drop on their intent classification task. (Thankfully, they had evals!) Similarly, GoDaddy observed a trend in the positive direction, where upgrading to version 1106 narrowed the performance gap between gpt-3.5-turbo and gpt-4. (Or, if you’re a glass-half-full person, you might be disappointed that gpt-4’s lead was reduced with the new upgrade)

Thus, if we have to migrate prompts across models, expect it to take more time than simply swapping the API endpoint. Don’t assume that plugging in the same prompt will lead to similar or better results. Also, having reliable, automated evals helps with measuring task performance before and after migration, and reduces the effort needed for manual verification.

Version and pin your modelsIn any machine learning pipeline, “changing anything changes everything“. This is particularly relevant as we rely on components like large language models (LLMs) that we don’t train ourselves and that can change without our knowledge.

Fortunately, many model providers offer the option to “pin” specific model versions (e.g., gpt-4-turbo-1106). This enables us to use a specific version of the model weights, ensuring they remain unchanged. Pinning model versions in production can help avoid unexpected changes in model behavior, which could lead to customer complaints about issues that may crop up when a model is swapped, such as overly verbose outputs or other unforeseen failure modes.

Additionally, consider maintaining a shadow pipeline that mirrors your production setup but uses the latest model versions. This enables safe experimentation and testing with new releases. Once you’ve validated the stability and quality of the outputs from these newer models, you can confidently update the model versions in your production environment.

Choose the smallest model that gets the job doneWhen working on a new application, it’s tempting to use the biggest, most powerful model available. But once we’ve established that the task is technically feasible, it’s worth experimenting if a smaller model can achieve comparable results.

The benefits of a smaller model are lower latency and cost. While it may be weaker, techniques like chain-of-thought, n-shot prompts, and in-context learning can help smaller models punch above their weight. Beyond LLM APIs, fine-tuning our specific tasks can also help increase performance.

Taken together, a carefully crafted workflow using a smaller model can often match, or even surpass, the output quality of a single large model, while being faster and cheaper. For example, this post shares anecdata of how Haiku + 10-shot prompt outperforms zero-shot Opus and GPT-4. In the long term, we expect to see more examples of flow-engineering with smaller models as the optimal balance of output quality, latency, and cost.

As another example, take the humble classification task. Lightweight models like DistilBERT (67M parameters) are a surprisingly strong baseline. The 400M parameter DistilBART is another great option—when fine-tuned on open source data, it could identify hallucinations with an ROC-AUC of 0.84, surpassing most LLMs at less than 5% of latency and cost.

The point is, don’t overlook smaller models. While it’s easy to throw a massive model at every problem, with some creativity and experimentation, we can often find a more efficient solution.

ProductWhile new technology offers new possibilities, the principles of building great products are timeless. Thus, even if we’re solving new problems for the first time, we don’t have to reinvent the wheel on product design. There’s a lot to gain from grounding our LLM application development in solid product fundamentals, allowing us to deliver real value to the people we serve.

Involve design early and oftenHaving a designer will push you to understand and think deeply about how your product can be built and presented to users. We sometimes stereotype designers as folks who take things and make them pretty. But beyond just the user interface, they also rethink how the user experience can be improved, even if it means breaking existing rules and paradigms.

Designers are especially gifted at reframing the user’s needs into various forms. Some of these forms are more tractable to solve than others, and thus, they may offer more or fewer opportunities for AI solutions. Like many other products, building AI products should be centered around the job to be done, not the technology that powers them.

Focus on asking yourself: “What job is the user asking this product to do for them? Is that job something a chatbot would be good at? How about autocomplete? Maybe something different!” Consider the existing design patterns and how they relate to the job-to-be-done. These are the invaluable assets that designers add to your team’s capabilities.

Design your UX for Human-in-the-LoopOne way to get quality annotations is to integrate Human-in-the-Loop (HITL) into the user experience (UX). By allowing users to provide feedback and corrections easily, we can improve the immediate output and collect valuable data to improve our models.

Imagine an e-commerce platform where users upload and categorize their products. There are several ways we could design the UX:

• The user manually selects the right product category; an LLM periodically checks new products and corrects miscategorization on the backend.
• The user doesn’t select any category at all; an LLM periodically categorizes products on the backend (with potential errors).
• An LLM suggests a product category in real time, which the user can validate and update as needed.

While all three approaches involve an LLM, they provide very different UXes. The first approach puts the initial burden on the user and has the LLM acting as a postprocessing check. The second requires zero effort from the user but provides no transparency or control. The third strikes the right balance. By having the LLM suggest categories upfront, we reduce cognitive load on the user and they don’t have to learn our taxonomy to categorize their product! At the same time, by allowing the user to review and edit the suggestion, they have the final say in how their product is classified, putting control firmly in their hands. As a bonus, the third approach creates a natural feedback loop for model improvement. Suggestions that are good are accepted (positive labels) and those that are bad are updated (negative followed by positive labels).

This pattern of suggestion, user validation, and data collection is commonly seen in several applications:

• Coding assistants: Where users can accept a suggestion (strong positive), accept and tweak a suggestion (positive), or ignore a suggestion (negative)
• Midjourney: Where users can choose to upscale and download the image (strong positive), vary an image (positive), or generate a new set of images (negative)
• Chatbots: Where users can provide thumbs ups (positive) or thumbs down (negative) on responses, or choose to regenerate a response if it was really bad (strong negative)

Feedback can be explicit or implicit. Explicit feedback is information users provide in response to a request by our product; implicit feedback is information we learn from user interactions without needing users to deliberately provide feedback. Coding assistants and Midjourney are examples of implicit feedback while thumbs up and thumb downs are explicit feedback. If we design our UX well, like coding assistants and Midjourney, we can collect plenty of implicit feedback to improve our product and models.

Prioritize your hierarchy of needs ruthlesslyAs we think about putting our demo into production, we’ll have to think about the requirements for:

• Reliability: 99.9% uptime, adherence to structured output
• Harmlessness: Not generate offensive, NSFW, or otherwise harmful content
• Factual consistency: Being faithful to the context provided, not making things up
• Usefulness: Relevant to the users’ needs and request
• Scalability: Latency SLAs, supported throughput
• Cost: Because we don’t have unlimited budget
• And more: Security, privacy, fairness, GDPR, DMA, etc.

If we try to tackle all these requirements at once, we’re never going to ship anything. Thus, we need to prioritize. Ruthlessly. This means being clear what is nonnegotiable (e.g., reliability, harmlessness) without which our product can’t function or won’t be viable. It’s all about identifying the minimum lovable product. We have to accept that the first version won’t be perfect, and just launch and iterate.

Calibrate your risk tolerance based on the use caseWhen deciding on the language model and level of scrutiny of an application, consider the use case and audience. For a customer-facing chatbot offering medical or financial advice, we’ll need a very high bar for safety and accuracy. Mistakes or bad output could cause real harm and erode trust. But for less critical applications, such as a recommender system, or internal-facing applications like content classification or summarization, excessively strict requirements only slow progress without adding much value.

This aligns with a recent a16z report showing that many companies are moving faster with internal LLM applications compared to external ones. By experimenting with AI for internal productivity, organizations can start capturing value while learning how to manage risk in a more controlled environment. Then, as they gain confidence, they can expand to customer-facing use cases.

Team & RolesNo job function is easy to define, but writing a job description for the work in this new space is more challenging than others. We’ll forgo Venn diagrams of intersecting job titles, or suggestions for job descriptions. We will, however, submit to the existence of a new role—the AI engineer—and discuss its place. Importantly, we’ll discuss the rest of the team and how responsibilities should be assigned.

Focus on process, not toolsWhen faced with new paradigms, such as LLMs, software engineers tend to favor tools. As a result, we overlook the problem and process the tool was supposed to solve. In doing so, many engineers assume accidental complexity, which has negative consequences for the team’s long-term productivity.

For example, this write-up discusses how certain tools can automatically create prompts for large language models. It argues (rightfully IMHO) that engineers who use these tools without first understanding the problem-solving methodology or process end up taking on unnecessary technical debt.

In addition to accidental complexity, tools are often underspecified. For example, there is a growing industry of LLM evaluation tools that offer “LLM Evaluation in a Box” with generic evaluators for toxicity, conciseness, tone, etc. We have seen many teams adopt these tools without thinking critically about the specific failure modes of their domains. Contrast this to EvalGen. It focuses on teaching users the process of creating domain-specific evals by deeply involving the user each step of the way, from specifying criteria, to labeling data, to checking evals. The software leads the user through a workflow that looks like this:

Shankar, S., et al. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. Retrieved from https://arxiv.org/abs/2404.12272EvalGen guides the user through a best practice of crafting LLM evaluations, namely:

1. Defining domain-specific tests (bootstrapped automatically from the prompt). These are defined as either assertions with code or with LLM-as-a-Judge.
2. The importance of aligning the tests with human judgment, so that the user can check that the tests capture the specified criteria.
3. Iterating on your tests as the system (prompts, etc.) changes.

EvalGen provides developers with a mental model of the evaluation building process without anchoring them to a specific tool. We have found that after providing AI engineers with this context, they often decide to select leaner tools or build their own.

There are too many components of LLMs beyond prompt writing and evaluations to list exhaustively here. However, it is important that AI engineers seek to understand the processes before adopting tools.

Always be experimentingML products are deeply intertwined with experimentation. Not only the A/B, randomized control trials kind, but the frequent attempts at modifying the smallest possible components of your system and doing offline evaluation. The reason why everyone is so hot for evals is not actually about trustworthiness and confidence—it’s about enabling experiments! The better your evals, the faster you can iterate on experiments, and thus the faster you can converge on the best version of your system.

It’s common to try different approaches to solving the same problem because experimentation is so cheap now. The high-cost of collecting data and training a model is minimized—prompt engineering costs little more than human time. Position your team so that everyone is taught the basics of prompt engineering. This encourages everyone to experiment and leads to diverse ideas from across the organization.

Additionally, don’t only experiment to explore—also use them to exploit! Have a working version of a new task? Consider having someone else on the team approach it differently. Try doing it another way that’ll be faster. Investigate prompt techniques like chain-of-thought or few-shot to make it higher quality. Don’t let your tooling hold you back on experimentation; if it is, rebuild it, or buy something to make it better.

Finally, during product/project planning, set aside time for building evals and running multiple experiments. Think of the product spec for engineering products, but add to it clear criteria for evals. And during roadmapping, don’t underestimate the time required for experimentation—expect to do multiple iterations of development and evals before getting the green light for production.

Empower everyone to use new AI technologyAs generative AI increases in adoption, we want the entire team—not just the experts—to understand and feel empowered to use this new technology. There’s no better way to develop intuition for how LLMs work (e.g., latencies, failure modes, UX) than to, well, use them. LLMs are relatively accessible: You don’t need to know how to code to improve performance for a pipeline, and everyone can start contributing via prompt engineering and evals.

A big part of this is education. It can start as simple as the basics of prompt engineering, where techniques like n-shot prompting and CoT help condition the model toward the desired output. Folks who have the knowledge can also educate about the more technical aspects, such as how LLMs are autoregressive in nature. In other words, while input tokens are processed in parallel, output tokens are generated sequentially. As a result, latency is more a function of output length than input length—this is a key consideration when designing UXes and setting performance expectations.

We can also go further and provide opportunities for hands-on experimentation and exploration. A hackathon perhaps? While it may seem expensive to have an entire team spend a few days hacking on speculative projects, the outcomes may surprise you. We know of a team that, through a hackathon, accelerated and almost completed their three-year roadmap within a year. Another team had a hackathon that led to paradigm shifting UXes that are now possible thanks to LLMs, which are now prioritized for the year and beyond.

Don’t fall into the trap of “AI engineering is all I need”As new job titles are coined, there is an initial tendency to overstate the capabilities associated with these roles. This often results in a painful correction as the actual scope of these jobs becomes clear. Newcomers to the field, as well as hiring managers, might make exaggerated claims or have inflated expectations. Notable examples over the last decade include:

• Data scientist: “someone who is better at statistics than any software engineer and better at software engineering than any statistician”
• Machine learning engineer (MLE): a software engineering-centric view of machine learning

Initially, many assumed that data scientists alone were sufficient for data-driven projects. However, it became apparent that data scientists must collaborate with software and data engineers to develop and deploy data products effectively.

This misunderstanding has shown up again with the new role of AI engineer, with some teams believing that AI engineers are all you need. In reality, building machine learning or AI products requires a broad array of specialized roles. We’ve consulted with more than a dozen companies on AI products and have consistently observed that they fall into the trap of believing that “AI engineering is all you need.” As a result, products often struggle to scale beyond a demo as companies overlook crucial aspects involved in building a product.

For example, evaluation and measurement are crucial for scaling a product beyond vibe checks. The skills for effective evaluation align with some of the strengths traditionally seen in machine learning engineers—a team composed solely of AI engineers will likely lack these skills. Coauthor Hamel Husain illustrates the importance of these skills in his recent work around detecting data drift and designing domain-specific evals.

Here is a rough progression of the types of roles you need, and when you’ll need them, throughout the journey of building an AI product:

1. First, focus on building a product. This might include an AI engineer, but it doesn’t have to. AI engineers are valuable for prototyping and iterating quickly on the product (UX, plumbing, etc.).
2. Next, create the right foundations by instrumenting your system and collecting data. Depending on the type and scale of data, you might need platform and/or data engineers. You must also have systems for querying and analyzing this data to debug issues.
3. Next, you will eventually want to optimize your AI system. This doesn’t necessarily involve training models. The basics include steps like designing metrics, building evaluation systems, running experiments, optimizing RAG retrieval, debugging stochastic systems, and more. MLEs are really good at this (though AI engineers can pick them up too). It usually doesn’t make sense to hire an MLE unless you have completed the prerequisite steps.

Aside from this, you need a domain expert at all times. At small companies, this would ideally be the founding team—and at bigger companies, product managers can play this role. Being aware of the progression and timing of roles is critical. Hiring folks at the wrong time (e.g., hiring an MLE too early) or building in the wrong order is a waste of time and money, and causes churn. Furthermore, regularly checking in with an MLE (but not hiring them full-time) during phases 1–2 will help the company build the right foundations.

About the authorsEugene Yan designs, builds, and operates machine learning systems that serve customers at scale. He’s currently a Senior Applied Scientist at Amazon where he builds RecSys serving users at scale and applies LLMs to serve customers better. Previously, he led machine learning at Lazada (acquired by Alibaba) and a Healthtech Series A. He writes and speaks about ML, RecSys, LLMs, and engineering at eugeneyan.com and ApplyingML.com.

Bryan Bischof is the Head of AI at Hex, where he leads the team of engineers building Magic—the data science and analytics copilot. Bryan has worked all over the data stack leading teams in analytics, machine learning engineering, data platform engineering, and AI engineering. He started the data team at Blue Bottle Coffee, led several projects at Stitch Fix, and built the data teams at Weights and Biases. Bryan previously co-authored the book Building Production Recommendation Systems with O’Reilly, and teaches Data Science and Analytics in the graduate school at Rutgers. His Ph.D. is in pure mathematics.

Charles Frye teaches people to build AI applications. After publishing research in psychopharmacology and neurobiology, he got his Ph.D. at the University of California, Berkeley, for dissertation work on neural network optimization. He has taught thousands the entire stack of AI application development, from linear algebra fundamentals to GPU arcana and building defensible businesses, through educational and consulting work at Weights and Biases, Full Stack Deep Learning, and Modal.

Hamel Husain is a machine learning engineer with over 25 years of experience. He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies operationalize Large Language Models (LLMs) to accelerate their AI product journey.

Jason Liu is a distinguished machine learning consultant known for leading teams to successfully ship AI products. Jason’s technical expertise covers personalization algorithms, search optimization, synthetic data generation, and MLOps systems. His experience includes companies like Stitch Fix, where he created a recommendation framework and observability tools that handled 350 million daily requests. Additional roles have included Meta, NYU, and startups such as Limitless AI and Trunk Tools.

Shreya Shankar is an ML engineer and PhD student in computer science at UC Berkeley. She was the first ML engineer at 2 startups, building AI-powered products from scratch that serve thousands of users daily. As a researcher, her work focuses on addressing data challenges in production ML systems through a human-centered approach. Her work has appeared in top data management and human-computer interaction venues like VLDB, SIGMOD, CIDR, and CSCW.

Contact UsWe would love to hear your thoughts on this post. You can contact us at contact@applied-llms.org. Many of us are open to various forms of consulting and advisory. We will route you to the correct expert(s) upon contact with us if appropriate.

AcknowledgementsThis series started as a conversation in a group chat, where Bryan quipped that he was inspired to write “A Year of AI Engineering.” Then, magic happened in the group chat, and we were all inspired to chip in and share what we’ve learned so far.

The authors would like to thank Eugene for leading the bulk of the document integration and overall structure in addition to a large proportion of the lessons. Additionally, for primary editing responsibilities and document direction. The authors would like to thank Bryan for the spark that led to this writeup, restructuring the write-up into tactical, operational, and strategic sections and their intros, and for pushing us to think bigger on how we could reach and help the community. The authors would like to thank Charles for his deep dives on cost and LLMOps, as well as weaving the lessons to make them more coherent and tighter—you have him to thank for this being 30 instead of 40 pages! The authors appreciate Hamel and Jason for their insights from advising clients and being on the front lines, for their broad generalizable learnings from clients, and for deep knowledge of tools. And finally, thank you Shreya for reminding us of the importance of evals and rigorous production practices and for bringing her research and original results to this piece.

Finally, the authors would like to thank all the teams who so generously shared your challenges and lessons in your own write-ups which we’ve referenced throughout this series, along with the AI communities for your vibrant participation and engagement with this group.

View Details

To hear directly from the authors on this topic, sign up for the upcoming virtual event on June 20th.

It’s an exciting time to build with large language models (LLMs). Over the past year, LLMs have become “good enough” for real-world applications. The pace of improvements in LLMs, coupled with a parade of demos on social media, will fuel an estimated $200B investment in AI by 2025. LLMs are also broadly accessible, allowing everyone, not just ML engineers and scientists, to build intelligence into their products. While the barrier to entry for building AI products has been lowered, creating those effective beyond a demo remains a deceptively difficult endeavor.

We’ve identified some crucial, yet often neglected, lessons and methodologies informed by machine learning that are essential for developing products based on LLMs. Awareness of these concepts can give you a competitive advantage against most others in the field without requiring ML expertise! Over the past year, the six of us have been building real-world applications on top of LLMs. We realized that there was a need to distill these lessons in one place for the benefit of the community.

We come from a variety of backgrounds and serve in different roles, but we’ve all experienced firsthand the challenges that come with using this new technology. Two of us are independent consultants who’ve helped numerous clients take LLM projects from initial concept to successful product, seeing the patterns determining success or failure. One of us is a researcher studying how ML/AI teams work and how to improve their workflows. Two of us are leaders on applied AI teams: one at a tech giant and one at a startup. Finally, one of us has taught deep learning to thousands and now works on making AI tooling and infrastructure easier to use. Despite our different experiences, we were struck by the consistent themes in the lessons we’ve learned, and we’re surprised that these insights aren’t more widely discussed.

Our goal is to make this a practical guide to building successful products around LLMs, drawing from our own experiences and pointing to examples from around the industry. We’ve spent the past year getting our hands dirty and gaining valuable lessons, often the hard way. While we don’t claim to speak for the entire industry, here we share some advice and lessons for anyone building products with LLMs.

This work is organized into three sections: tactical, operational, and strategic. This is the first of three pieces. It dives into the tactical nuts and bolts of working with LLMs. We share best practices and common pitfalls around prompting, setting up retrieval-augmented generation, applying flow engineering, and evaluation and monitoring. Whether you’re a practitioner building with LLMs or a hacker working on weekend projects, this section was written for you. Look out for the operational and strategic sections in the coming weeks.

Ready to dive in? Let’s go.

TacticalIn this section, we share best practices for the core components of the emerging LLM stack: prompting tips to improve quality and reliability, evaluation strategies to assess output, retrieval-augmented generation ideas to improve grounding, and more. We also explore how to design human-in-the-loop workflows. While the technology is still rapidly developing, we hope these lessons, the by-product of countless experiments we’ve collectively run, will stand the test of time and help you build and ship robust LLM applications.

PromptingWe recommend starting with prompting when developing new applications. It’s easy to both underestimate and overestimate its importance. It’s underestimated because the right prompting techniques, when used correctly, can get us very far. It’s overestimated because even prompt-based applications require significant engineering around the prompt to work well.

Focus on getting the most out of fundamental prompting techniquesA few prompting techniques have consistently helped improve performance across various models and tasks: n-shot prompts + in-context learning, chain-of-thought, and providing relevant resources.

The idea of in-context learning via n-shot prompts is to provide the LLM with a few examples that demonstrate the task and align outputs to our expectations. A few tips:

• If n is too low, the model may over-anchor on those specific examples, hurting its ability to generalize. As a rule of thumb, aim for n ≥ 5. Don’t be afraid to go as high as a few dozen.
• Examples should be representative of the expected input distribution. If you’re building a movie summarizer, include samples from different genres in roughly the proportion you expect to see in practice.
• You don’t necessarily need to provide the full input-output pairs. In many cases, examples of desired outputs are sufficient.
• If you are using an LLM that supports tool use, your n-shot examples should also use the tools you want the agent to use.

In chain-of-thought (CoT) prompting, we encourage the LLM to explain its thought process before returning the final answer. Think of it as providing the LLM with a sketchpad so it doesn’t have to do it all in memory. The original approach was to simply add the phrase “Let’s think step-by-step” as part of the instructions. However, we’ve found it helpful to make the CoT more specific, where adding specificity via an extra sentence or two often reduces hallucination rates significantly. For example, when asking an LLM to summarize a meeting transcript, we can be explicit about the steps, such as:

• First, list the key decisions, follow-up items, and associated owners in a sketchpad.
• Then, check that the details in the sketchpad are factually consistent with the transcript.
• Finally, synthesize the key points into a concise summary.

Recently, some doubt has been cast on whether this technique is as powerful as believed. Additionally, there’s significant debate about exactly what happens during inference when chain-of-thought is used. Regardless, this technique is one to experiment with when possible.

Providing relevant resources is a powerful mechanism to expand the model’s knowledge base, reduce hallucinations, and increase the user’s trust. Often accomplished via retrieval augmented generation (RAG), providing the model with snippets of text that it can directly utilize in its response is an essential technique. When providing the relevant resources, it’s not enough to merely include them; don’t forget to tell the model to prioritize their use, refer to them directly, and sometimes to mention when none of the resources are sufficient. These help “ground” agent responses to a corpus of resources.

Structure your inputs and outputsStructured input and output help models better understand the input as well as return output that can reliably integrate with downstream systems. Adding serialization formatting to your inputs can help provide more clues to the model as to the relationships between tokens in the context, additional metadata to specific tokens (like types), or relate the request to similar examples in the model’s training data.

As an example, many questions on the internet about writing SQL begin by specifying the SQL schema. Thus, you may expect that effective prompting for Text-to-SQL should include structured schema definitions; indeed.

Structured output serves a similar purpose, but it also simplifies integration into downstream components of your system. Instructor and Outlines work well for structured output. (If you’re importing an LLM API SDK, use Instructor; if you’re importing Huggingface for a self-hosted model, use Outlines.) Structured input expresses tasks clearly and resembles how the training data is formatted, increasing the probability of better output.

When using structured input, be aware that each LLM family has their own preferences. Claude prefers xml while GPT favors Markdown and JSON. With XML, you can even pre-fill Claude’s responses by providing a response tag like so.

pythonmessages=[ { "role": "user", "content": """Extract the <name>, <size>, <price>, and <color> from this product description into your <response>. <description>The SmartHome Mini is a compact smart home assistant available in black or white for only $49.99. At just 5 inches wide, it lets you control lights, thermostats, and other connected devices via voice or app—no matter where you place it in your home. This affordable little hub brings convenient hands-free control to your smart devices. </description>""" }, { "role": "assistant", "content": "<response><name>" } ] Have small prompts that do one thing, and only one thing, wellA common anti-pattern/code smell in software is the “God Object,” where we have a single class or function that does everything. The same applies to prompts too.

A prompt typically starts simple: A few sentences of instruction, a couple of examples, and we’re good to go. But as we try to improve performance and handle more edge cases, complexity creeps in. More instructions. Multi-step reasoning. Dozens of examples. Before we know it, our initially simple prompt is now a 2,000 token frankenstein. And to add injury to insult, it has worse performance on the more common and straightforward inputs! GoDaddy shared this challenge as their No. 1 lesson from building with LLMs.

Just like how we strive (read: struggle) to keep our systems and code simple, so should we for our prompts. Instead of having a single, catch-all prompt for the meeting transcript summarizer, we can break it into steps to:

• Extract key decisions, action items, and owners into structured format
• Check extracted details against the original transcription for consistency
• Generate a concise summary from the structured details

As a result, we’ve split our single prompt into multiple prompts that are each simple, focused, and easy to understand. And by breaking them up, we can now iterate and eval each prompt individually.

Craft your context tokensRethink, and challenge your assumptions about how much context you actually need to send to the agent. Be like Michaelangelo, do not build up your context sculpture—chisel away the superfluous material until the sculpture is revealed. RAG is a popular way to collate all of the potentially relevant blocks of marble, but what are you doing to extract what’s necessary?

We’ve found that taking the final prompt sent to the model—with all of the context construction, and meta-prompting, and RAG results—putting it on a blank page and just reading it, really helps you rethink your context. We have found redundancy, self-contradictory language, and poor formatting using this method.

The other key optimization is the structure of your context. Your bag-of-docs representation isn’t helpful for humans, don’t assume it’s any good for agents. Think carefully about how you structure your context to underscore the relationships between parts of it, and make extraction as simple as possible.

Information Retrieval/RAGBeyond prompting, another effective way to steer an LLM is by providing knowledge as part of the prompt. This grounds the LLM on the provided context which is then used for in-context learning. This is known as retrieval-augmented generation (RAG). Practitioners have found RAG effective at providing knowledge and improving output, while requiring far less effort and cost compared to finetuning.RAG is only as good as the retrieved documents’ relevance, density, and detail

The quality of your RAG’s output is dependent on the quality of retrieved documents, which in turn can be considered along a few factors.The first and most obvious metric is relevance. This is typically quantified via ranking metrics such as Mean Reciprocal Rank (MRR) or Normalized Discounted Cumulative Gain (NDCG). MRR evaluates how well a system places the first relevant result in a ranked list while NDCG considers the relevance of all the results and their positions. They measure how good the system is at ranking relevant documents higher and irrelevant documents lower. For example, if we’re retrieving user summaries to generate movie review summaries, we’ll want to rank reviews for the specific movie higher while excluding reviews for other movies.

Like traditional recommendation systems, the rank of retrieved items will have a significant impact on how the LLM performs on downstream tasks. To measure the impact, run a RAG-based task but with the retrieved items shuffled—how does the RAG output perform?

Second, we also want to consider information density. If two documents are equally relevant, we should prefer one that’s more concise and has lesser extraneous details. Returning to our movie example, we might consider the movie transcript and all user reviews to be relevant in a broad sense. Nonetheless, the top-rated reviews and editorial reviews will likely be more dense in information.

Finally, consider the level of detail provided in the document. Imagine we’re building a RAG system to generate SQL queries from natural language. We could simply provide table schemas with column names as context. But, what if we include column descriptions and some representative values? The additional detail could help the LLM better understand the semantics of the table and thus generate more correct SQL.

Don’t forget keyword search; use it as a baseline and in hybrid search.Given how prevalent the embedding-based RAG demo is, it’s easy to forget or overlook the decades of research and solutions in information retrieval.

Nonetheless, while embeddings are undoubtedly a powerful tool, they are not the be all and end all. First, while they excel at capturing high-level semantic similarity, they may struggle with more specific, keyword-based queries, like when users search for names (e.g., Ilya), acronyms (e.g., RAG), or IDs (e.g., claude-3-sonnet). Keyword-based search, such as BM25, are explicitly designed for this. And after years of keyword-based search, users have likely taken it for granted and may get frustrated if the document they expect to retrieve isn’t being returned.

Vector embeddings do not magically solve search. In fact, the heavy lifting is in the step before you re-rank with semantic similarity search. Making a genuine improvement over BM25 or full-text search is hard.

— Aravind Srinivas, CEO Perplexity.ai

We’ve been communicating this to our customers and partners for months now. Nearest Neighbor Search with naive embeddings yields very noisy results and you’re likely better off starting with a keyword-based approach.

— Beyang Liu, CTO Sourcegraph

Second, it’s more straightforward to understand why a document was retrieved with keyword search—we can look at the keywords that match the query. In contrast, embedding-based retrieval is less interpretable. Finally, thanks to systems like Lucene and OpenSearch that have been optimized and battle-tested over decades, keyword search is usually more computationally efficient.

In most cases, a hybrid will work best: keyword matching for the obvious matches, and embeddings for synonyms, hypernyms, and spelling errors, as well as multimodality (e.g., images and text). Shortwave shared how they built their RAG pipeline, including query rewriting, keyword + embedding retrieval, and ranking.

Prefer RAG over fine-tuning for new knowledgeBoth RAG and fine-tuning can be used to incorporate new information into LLMs and increase performance on specific tasks. Thus, which should we try first?

Recent research suggests that RAG may have an edge. One study compared RAG against unsupervised fine-tuning (a.k.a. continued pre-training), evaluating both on a subset of MMLU and current events. They found that RAG consistently outperformed fine-tuning for knowledge encountered during training as well as entirely new knowledge. In another paper, they compared RAG against supervised fine-tuning on an agricultural dataset. Similarly, the performance boost from RAG was greater than fine-tuning, especially for GPT-4 (see Table 20 of the paper).

Beyond improved performance, RAG comes with several practical advantages too. First, compared to continuous pretraining or fine-tuning, it’s easier—and cheaper!—to keep retrieval indices up-to-date. Second, if our retrieval indices have problematic documents that contain toxic or biased content, we can easily drop or modify the offending documents.

In addition, the R in RAG provides finer grained control over how we retrieve documents. For example, if we’re hosting a RAG system for multiple organizations, by partitioning the retrieval indices, we can ensure that each organization can only retrieve documents from their own index. This ensures that we don’t inadvertently expose information from one organization to another.

Long-context models won’t make RAG obsoleteWith Gemini 1.5 providing context windows of up to 10M tokens in size, some have begun to question the future of RAG.

I tend to believe that Gemini 1.5 is significantly overhyped by Sora. A context window of 10M tokens effectively makes most of existing RAG frameworks unnecessary—you simply put whatever your data into the context and talk to the model like usual. Imagine how it does to all the startups/agents/LangChain projects where most of the engineering efforts goes to RAG Or in one sentence: the 10m context kills RAG. Nice work Gemini.

— Yao Fu

While it’s true that long contexts will be a game-changer for use cases such as analyzing multiple documents or chatting with PDFs, the rumors of RAG’s demise are greatly exaggerated.

First, even with a context window of 10M tokens, we’d still need a way to select information to feed into the model. Second, beyond the narrow needle-in-a-haystack eval, we’ve yet to see convincing data that models can effectively reason over such a large context. Thus, without good retrieval (and ranking), we risk overwhelming the model with distractors, or may even fill the context window with completely irrelevant information.

Finally, there’s cost. The Transformer’s inference cost scales quadratically (or linearly in both space and time) with context length. Just because there exists a model that could read your organization’s entire Google Drive contents before answering each question doesn’t mean that’s a good idea. Consider an analogy to how we use RAM: we still read and write from disk, even though there exist compute instances with RAM running into the tens of terabytes.

So don’t throw your RAGs in the trash just yet. This pattern will remain useful even as context windows grow in size.

Tuning and optimizing workflowsPrompting an LLM is just the beginning. To get the most juice out of them, we need to think beyond a single prompt and embrace workflows. For example, how could we split a single complex task into multiple simpler tasks? When is finetuning or caching helpful with increasing performance and reducing latency/cost? In this section, we share proven strategies and real-world examples to help you optimize and build reliable LLM workflows.

Step-by-step, multi-turn “flows” can give large boosts.We already know that by decomposing a single big prompt into multiple smaller prompts, we can achieve better results. An example of this is AlphaCodium: By switching from a single prompt to a multi-step workflow, they increased GPT-4 accuracy (pass@5) on CodeContests from 19% to 44%. The workflow includes:

• Reflecting on the problem
• Reasoning on the public tests
• Generating possible solutions
• Ranking possible solutions
• Generating synthetic tests
• Iterating on the solutions on public and synthetic tests.

Small tasks with clear objectives make for the best agent or flow prompts. It’s not required that every agent prompt requests structured output, but structured outputs help a lot to interface with whatever system is orchestrating the agent’s interactions with the environment.

Some things to try

• An explicit planning step, as tightly specified as possible. Consider having predefined plans to choose from (c.f. https://youtu.be/hGXhFa3gzBs?si=gNEGYzux6TuB1del).
• Rewriting the original user prompts into agent prompts. Be careful, this process is lossy!
• Agent behaviors as linear chains, DAGs, and State-Machines; different dependency and logic relationships can be more and less appropriate for different scales. Can you squeeze performance optimization out of different task architectures?
• Planning validations; your planning can include instructions on how to evaluate the responses from other agents to make sure the final assembly works well together.
• Prompt engineering with fixed upstream state—make sure your agent prompts are evaluated against a collection of variants of what may happen before.

Prioritize deterministic workflows for nowWhile AI agents can dynamically react to user requests and the environment, their non-deterministic nature makes them a challenge to deploy. Each step an agent takes has a chance of failing, and the chances of recovering from the error are poor. Thus, the likelihood that an agent completes a multi-step task successfully decreases exponentially as the number of steps increases. As a result, teams building agents find it difficult to deploy reliable agents.

A promising approach is to have agent systems that produce deterministic plans which are then executed in a structured, reproducible way. In the first step, given a high-level goal or prompt, the agent generates a plan. Then, the plan is executed deterministically. This allows each step to be more predictable and reliable. Benefits include:

• Generated plans can serve as few-shot samples to prompt or finetune an agent.
• Deterministic execution makes the system more reliable, and thus easier to test and debug. Furthermore, failures can be traced to the specific steps in the plan.
• Generated plans can be represented as directed acyclic graphs (DAGs) which are easier, relative to a static prompt, to understand and adapt to new situations.

The most successful agent builders may be those with strong experience managing junior engineers because the process of generating plans is similar to how we instruct and manage juniors. We give juniors clear goals and concrete plans, instead of vague open-ended directions, and we should do the same for our agents too.

In the end, the key to reliable, working agents will likely be found in adopting more structured, deterministic approaches, as well as collecting data to refine prompts and finetune models. Without this, we’ll build agents that may work exceptionally well some of the time, but on average, disappoint users which leads to poor retention.

Getting more diverse outputs beyond temperatureSuppose your task requires diversity in an LLM’s output. Maybe you’re writing an LLM pipeline to suggest products to buy from your catalog given a list of products the user bought previously. When running your prompt multiple times, you might notice that the resulting recommendations are too similar—so you might increase the temperature parameter in your LLM requests.

Briefly, increasing the temperature parameter makes LLM responses more varied. At sampling time, the probability distributions of the next token become flatter, meaning that tokens which are usually less likely get chosen more often. Still, when increasing temperature, you may notice some failure modes related to output diversity. For example,Some products from the catalog that could be a good fit may never be output by the LLM.The same handful of products might be overrepresented in outputs, if they are highly likely to follow the prompt based on what the LLM has learned at training time.If the temperature is too high, you may get outputs that reference nonexistent products (or gibberish!)

In other words, increasing temperature does not guarantee that the LLM will sample outputs from the probability distribution you expect (e.g., uniform random). Nonetheless, we have other tricks to increase output diversity. The simplest way is to adjust elements within the prompt. For example, if the prompt template includes a list of items, such as historical purchases, shuffling the order of these items each time they’re inserted into the prompt can make a significant difference.

Additionally, keeping a short list of recent outputs can help prevent redundancy. In our recommended products example, by instructing the LLM to avoid suggesting items from this recent list, or by rejecting and resampling outputs that are similar to recent suggestions, we can further diversify the responses. Another effective strategy is to vary the phrasing used in the prompts. For instance, incorporating phrases like “pick an item that the user would love using regularly” or “select a product that the user would likely recommend to friends” can shift the focus and thereby influence the variety of recommended products.

Caching is underrated.Caching saves cost and eliminates generation latency by removing the need to recompute responses for the same input. Furthermore, if a response has previously been guardrailed, we can serve these vetted responses and reduce the risk of serving harmful or inappropriate content.

One straightforward approach to caching is to use unique IDs for the items being processed, such as if we’re summarizing new articles or product reviews. When a request comes in, we can check to see if a summary already exists in the cache. If so, we can return it immediately; if not, we generate, guardrail, and serve it, and then store it in the cache for future requests.

For more open-ended queries, we can borrow techniques from the field of search, which also leverages caching for open-ended inputs. Features like autocomplete and spelling correction also help normalize user input and thus increase the cache hit rate.

When to fine-tuneWe may have some tasks where even the most cleverly designed prompts fall short. For example, even after significant prompt engineering, our system may still be a ways from returning reliable, high-quality output. If so, then it may be necessary to finetune a model for your specific task.

Successful examples include:

• Honeycomb’s Natural Language Query Assistant: Initially, the “programming manual” was provided in the prompt together with n-shot examples for in-context learning. While this worked decently, fine-tuning the model led to better output on the syntax and rules of the domain-specific language.
• ReChat’s Lucy: The LLM needed to generate responses in a very specific format that combined structured and unstructured data for the frontend to render correctly. Fine-tuning was essential to get it to work consistently.

Nonetheless, while fine-tuning can be effective, it comes with significant costs. We have to annotate fine-tuning data, finetune and evaluate models, and eventually self-host them. Thus, consider if the higher upfront cost is worth it. If prompting gets you 90% of the way there, then fine-tuning may not be worth the investment. However, if we do decide to fine-tune, to reduce the cost of collecting human annotated data, we can generate and finetune on synthetic data, or bootstrap on open-source data.

Evaluation & MonitoringEvaluating LLMs can be a minefield. The inputs and the outputs of LLMs are arbitrary text, and the tasks we set them to are varied. Nonetheless, rigorous and thoughtful evals are critical—it’s no coincidence that technical leaders at OpenAI work on evaluation and give feedback on individual evals.

Evaluating LLM applications invites a diversity of definitions and reductions: it’s simply unit testing, or it’s more like observability, or maybe it’s just data science. We have found all of these perspectives useful. In the following section, we provide some lessons we’ve learned about what is important in building evals and monitoring pipelines.

Create a few assertion-based unit tests from real input/output samplesCreate unit tests (i.e., assertions) consisting of samples of inputs and outputs from production, with expectations for outputs based on at least three criteria. While three criteria might seem arbitrary, it’s a practical number to start with; fewer might indicate that your task isn’t sufficiently defined or is too open-ended, like a general-purpose chatbot. These unit tests, or assertions, should be triggered by any changes to the pipeline, whether it’s editing a prompt, adding new context via RAG, or other modifications. This write-up has an example of an assertion-based test for an actual use case.

Consider beginning with assertions that specify phrases or ideas to either include or exclude in all responses. Also consider checks to ensure that word, item, or sentence counts lie within a range. For other kinds of generation, assertions can look different. Execution-evaluation is a powerful method for evaluating code-generation, wherein you run the generated code and determine that the state of runtime is sufficient for the user-request.

As an example, if the user asks for a new function named foo; then after executing the agent’s generated code, foo should be callable! One challenge in execution-evaluation is that the agent code frequently leaves the runtime in slightly different form than the target code. It can be effective to “relax” assertions to the absolute most weak assumptions that any viable answer would satisfy.

Finally, using your product as intended for customers (i.e., “dogfooding”) can provide insight into failure modes on real-world data. This approach not only helps identify potential weaknesses, but also provides a useful source of production samples that can be converted into evals.

LLM-as-Judge can work (somewhat), but it’s not a silver bulletLLM-as-Judge, where we use a strong LLM to evaluate the output of other LLMs, has been met with skepticism by some. (Some of us were initially huge skeptics.) Nonetheless, when implemented well, LLM-as-Judge achieves decent correlation with human judgements, and can at least help build priors about how a new prompt or technique may perform. Specifically, when doing pairwise comparisons (e.g., control vs. treatment), LLM-as-Judge typically gets the direction right though the magnitude of the win/loss may be noisy.

Here are some suggestions to get the most out of LLM-as-Judge:

• Use pairwise comparisons: Instead of asking the LLM to score a single output on a Likert scale, present it with two options and ask it to select the better one. This tends to lead to more stable results.
• Control for position bias: The order of options presented can bias the LLM’s decision. To mitigate this, do each pairwise comparison twice, swapping the order of pairs each time. Just be sure to attribute wins to the right option after swapping!
• Allow for ties: In some cases, both options may be equally good. Thus, allow the LLM to declare a tie so it doesn’t have to arbitrarily pick a winner.
• Use Chain-of-Thought: Asking the LLM to explain its decision before giving a final preference can increase eval reliability. As a bonus, this allows you to use a weaker but faster LLM and still achieve similar results. Because frequently this part of the pipeline is in batch mode, the extra latency from CoT isn’t a problem.
• Control for response length: LLMs tend to bias toward longer responses. To mitigate this, ensure response pairs are similar in length.

One particularly powerful application of LLM-as-Judge is checking a new prompting strategy against regression. If you have tracked a collection of production results, sometimes you can rerun those production examples with a new prompting strategy, and use LLM-as-Judge to quickly assess where the new strategy may suffer.

Here’s an example of a simple but effective approach to iterate on LLM-as-Judge, where we simply log the LLM response, judge’s critique (i.e., CoT), and final outcome. They are then reviewed with stakeholders to identify areas for improvement. Over three iterations, agreement with human and LLM improved from 68% to 94%!

LLM-as-Judge is not a silver bullet though. There are subtle aspects of language where even the strongest models fail to evaluate reliably. In addition, we’ve found that conventional classifiers and reward models can achieve higher accuracy than LLM-as-Judge, and with lower cost and latency. For code generation, LLM-as-Judge can be weaker than more direct evaluation strategies like execution-evaluation.

The “intern test” for evaluating generationsWe like to use the following “intern test” when evaluating generations: If you took the exact input to the language model, including the context, and gave it to an average college student in the relevant major as a task, could they succeed? How long would it take?

If the answer is no because the LLM lacks the required knowledge, consider ways to enrich the context.

If the answer is no and we simply can’t improve the context to fix it, then we may have hit a task that’s too hard for contemporary LLMs.

If the answer is yes, but it would take a while, we can try to reduce the complexity of the task. Is it decomposable? Are there aspects of the task that can be made more templatized?

If the answer is yes, they would get it quickly, then it’s time to dig into the data. What’s the model doing wrong? Can we find a pattern of failures? Try asking the model to explain itself before or after it responds, to help you build a theory of mind.

Overemphasizing certain evals can hurt overall performance

“When a measure becomes a target, it ceases to be a good measure.”

— Goodhart’s Law

An example of this is the Needle-in-a-Haystack (NIAH) eval. The original eval helped quantify model recall as context sizes grew, as well as how recall is affected by needle position. However, it’s been so overemphasized that it’s featured as Figure 1 for Gemini 1.5’s report. The eval involves inserting a specific phrase (“The special magic {city} number is: {number}”) into a long document which repeats the essays of Paul Graham, and then prompting the model to recall the magic number.

While some models achieve near-perfect recall, it’s questionable whether NIAH truly reflects the reasoning and recall abilities needed in real-world applications. Consider a more practical scenario: Given the transcript of an hour-long meeting, can the LLM summarize the key decisions and next steps, as well as correctly attribute each item to the relevant person? This task is more realistic, going beyond rote memorization and also considering the ability to parse complex discussions, identify relevant information, and synthesize summaries.

Here’s an example of a practical NIAH eval. Using transcripts of doctor-patient video calls, the LLM is queried about the patient’s medication. It also includes a more challenging NIAH, inserting a phrase for random ingredients for pizza toppings, such as “The secret ingredients needed to build the perfect pizza are: Espresso-soaked dates, Lemon and Goat cheese.” Recall was around 80% on the medication task and 30% on the pizza task.

Tangentially, an overemphasis on NIAH evals can lead to lower performance on extraction and summarization tasks. Because these LLMs are so finetuned to attend to every sentence, they may start to treat irrelevant details and distractors as important, thus including them in the final output (when they shouldn’t!)

This could also apply to other evals and use cases. For example, summarization. An emphasis on factual consistency could lead to summaries that are less specific (and thus less likely to be factually inconsistent) and possibly less relevant. Conversely, an emphasis on writing style and eloquence could lead to more flowery, marketing-type language that could introduce factual inconsistencies.

Simplify annotation to binary tasks or pairwise comparisonsProviding open-ended feedback or ratings for model output on a Likert scale is cognitively demanding. As a result, the data collected is more noisy—due to variability among human raters—and thus less useful. A more effective approach is to simplify the task and reduce the cognitive burden on annotators. Two tasks that work well are binary classifications and pairwise comparisons.

In binary classifications, annotators are asked to make a simple yes-or-no judgment on the model’s output. They might be asked whether the generated summary is factually consistent with the source document, or whether the proposed response is relevant, or if it contains toxicity. Compared to the Likert scale, binary decisions are more precise, have higher consistency among raters, and lead to higher throughput. This was how Doordash setup their labeling queues for tagging menu items though a tree of yes-no questions.

In pairwise comparisons, the annotator is presented with a pair of model responses and asked which is better. Because it’s easier for humans to say “A is better than B” than to assign an individual score to either A or B individually, this leads to faster and more reliable annotations (over Likert scales). At a Llama2 meetup, Thomas Scialom, an author on the Llama2 paper, confirmed that pairwise-comparisons were faster and cheaper than collecting supervised finetuning data such as written responses. The former’s cost is $3.5 per unit while the latter’s cost is $25 per unit.

If you’re starting to write labeling guidelines, here are some reference guidelines from Google and Bing Search.

(Reference-free) evals and guardrails can be used interchangeably Guardrails help to catch inappropriate or harmful content while evals help to measure the quality and accuracy of the model’s output. In the case of reference-free evals, they may be considered two sides of the same coin. Reference-free evals are evaluations that don’t rely on a “golden” reference, such as a human-written answer, and can assess the quality of output based solely on the input prompt and the model’s response.

Some examples of these are summarization evals, where we only have to consider the input document to evaluate the summary on factual consistency and relevance. If the summary scores poorly on these metrics, we can choose not to display it to the user, effectively using the eval as a guardrail. Similarly, reference-free translation evals can assess the quality of a translation without needing a human-translated reference, again allowing us to use it as a guardrail.

LLMs will return output even when they shouldn’tA key challenge when working with LLMs is that they’ll often generate output even when they shouldn’t. This can lead to harmless but nonsensical responses, or more egregious defects like toxicity or dangerous content. For example, when asked to extract specific attributes or metadata from a document, an LLM may confidently return values even when those values don’t actually exist. Alternatively, the model may respond in a language other than English because we provided non-English documents in the context.

While we can try to prompt the LLM to return a “not applicable” or “unknown” response, it’s not foolproof. Even when the log probabilities are available, they’re a poor indicator of output quality. While log probs indicate the likelihood of a token appearing in the output, they don’t necessarily reflect the correctness of the generated text. On the contrary, for instruction-tuned models that are trained to respond to queries and generate coherent response, log probabilities may not be well-calibrated. Thus, while a high log probability may indicate that the output is fluent and coherent, it doesn’t mean it’s accurate or relevant.

While careful prompt engineering can help to some extent, we should complement it with robust guardrails that detect and filter/regenerate undesired output. For example, OpenAI provides a content moderation API that can identify unsafe responses such as hate speech, self-harm, or sexual output. Similarly, there are numerous packages for detecting personally identifiable information (PII). One benefit is that guardrails are largely agnostic of the use case and can thus be applied broadly to all output in a given language. In addition, with precise retrieval, our system can deterministically respond “I don’t know” if there are no relevant documents.

A corollary here is that LLMs may fail to produce outputs when they are expected to. This can happen for various reasons, from straightforward issues like long tail latencies from API providers to more complex ones such as outputs being blocked by content moderation filters. As such, it’s important to consistently log inputs and (potentially a lack of) outputs for debugging and monitoring.

Hallucinations are a stubborn problem.Unlike content safety or PII defects which have a lot of attention and thus seldom occur, factual inconsistencies are stubbornly persistent and more challenging to detect. They’re more common and occur at a baseline rate of 5 – 10%, and from what we’ve learned from LLM providers, it can be challenging to get it below 2%, even on simple tasks such as summarization.

To address this, we can combine prompt engineering (upstream of generation) and factual inconsistency guardrails (downstream of generation). For prompt engineering, techniques like CoT help reduce hallucination by getting the LLM to explain its reasoning before finally returning the output. Then, we can apply a factual inconsistency guardrail to assess the factuality of summaries and filter or regenerate hallucinations. In some cases, hallucinations can be deterministically detected. When using resources from RAG retrieval, if the output is structured and identifies what the resources are, you should be able to manually verify they’re sourced from the input context.

About the authorsEugene Yan designs, builds, and operates machine learning systems that serve customers at scale. He’s currently a Senior Applied Scientist at Amazon where he builds RecSys serving millions of customers worldwide RecSys 2022 keynote and applies LLMs to serve customers better AI Eng Summit 2023 keynote. Previously, he led machine learning at Lazada (acquired by Alibaba) and a Healthtech Series A. He writes & speaks about ML, RecSys, LLMs, and engineering at eugeneyan.com and ApplyingML.com.

Bryan Bischof is the Head of AI at Hex, where he leads the team of engineers building Magic—the data science and analytics copilot. Bryan has worked all over the data stack leading teams in analytics, machine learning engineering, data platform engineering, and AI engineering. He started the data team at Blue Bottle Coffee, led several projects at Stitch Fix, and built the data teams at Weights and Biases. Bryan previously co-authored the book Building Production Recommendation Systems with O’Reilly, and teaches Data Science and Analytics in the graduate school at Rutgers. His Ph.D. is in pure mathematics.

Charles Frye teaches people to build AI applications. After publishing research in psychopharmacology and neurobiology, he got his Ph.D. at the University of California, Berkeley, for dissertation work on neural network optimization. He has taught thousands the entire stack of AI application development, from linear algebra fundamentals to GPU arcana and building defensible businesses, through educational and consulting work at Weights and Biases, Full Stack Deep Learning, and Modal.

Hamel Husain is a machine learning engineer with over 25 years of experience. He has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI for code understanding. He has also led and contributed to numerous popular open-source machine-learning tools. Hamel is currently an independent consultant helping companies operationalize Large Language Models (LLMs) to accelerate their AI product journey.

Jason Liu is a distinguished machine learning consultant known for leading teams to successfully ship AI products. Jason’s technical expertise covers personalization algorithms, search optimization, synthetic data generation, and MLOps systems. His experience includes companies like Stitchfix, where he created a recommendation framework and observability tools that handled 350 million daily requests. Additional roles have included Meta, NYU, and startups such as Limitless AI and Trunk Tools.

Shreya Shankar is an ML engineer and PhD student in computer science at UC Berkeley. She was the first ML engineer at 2 startups, building AI-powered products from scratch that serve thousands of users daily. As a researcher, her work focuses on addressing data challenges in production ML systems through a human-centered approach. Her work has appeared in top data management and human-computer interaction venues like VLDB, SIGMOD, CIDR, and CSCW.

Contact UsWe would love to hear your thoughts on this post. You can contact us at contact@applied-llms.org. Many of us are open to various forms of consulting and advisory. We will route you to the correct expert(s) upon contact with us if appropriate.

AcknowledgementsThis series started as a conversation in a group chat, where Bryan quipped that he was inspired to write “A Year of AI Engineering.” Then, magic happened in the group chat, and we were all inspired to chip in and share what we’ve learned so far.

The authors would like to thank Eugene for leading the bulk of the document integration and overall structure in addition to a large proportion of the lessons. Additionally, for primary editing responsibilities and document direction. The authors would like to thank Bryan for the spark that led to this writeup, restructuring the write-up into tactical, operational, and strategic sections and their intros, and for pushing us to think bigger on how we could reach and help the community. The authors would like to thank Charles for his deep dives on cost and LLMOps, as well as weaving the lessons to make them more coherent and tighter—you have him to thank for this being 30 instead of 40 pages! The authors appreciate Hamel and Jason for their insights from advising clients and being on the front lines, for their broad generalizable learnings from clients, and for deep knowledge of tools. And finally, thank you Shreya for reminding us of the importance of evals and rigorous production practices and for bringing her research and original results to this piece.

Finally, the authors would like to thank all the teams who so generously shared your challenges and lessons in your own write-ups which we’ve referenced throughout this series, along with the AI communities for your vibrant participation and engagement with this group.

View Details

This article originally appeared in Business Age.

In commentary supplied to Business Age, I shot my mouth off saying that passwords are a poor solution for authenticating users–but none of the alternatives are very good, either. The choices available to us are at best poor. So now I’m the victim of a follow-up question What do I use?

Unfortunately, “what do I use” isn’t really a choice I get to make–more often than not, you’re stuck with the choices of the people who built the sites you use. So the best you can do is make sure you have a good password. A good password is a long string of random letters, numbers, and punctuation marks. There are a few ways of generating these. The simplest one is to let Google Chrome generate a password for you. (Firefox can also generate secure passwords.) While Google is widely mistrusted, I think that mistrust is misplaced. Google hasn’t been the victim of significant security breaches (unlike some well-known password managers), and they really have no interest in selling my passwords to other parties. Yes, zero-day exploits and frequent security updates to Chrome means that there are vulnerabilities–but it also means that vulnerabilities are detected and patched. We should all be much more concerned about software that isn’t updated frequently.

Creating your own good password is only slightly harder than letting your browser do it for you–and, frankly, easier than creating a bad password (though not easier to remember). I open a text window and type randomly on my keyboard for a few seconds, yielding something like this: oe8h;org’pr/sajidj. (That’s 18 characters, generated in a couple of seconds.) I copy it and paste it into an application that needs a password. If it asks for punctuation, a digit, or a capital letter, I go back to the text window, add something that seems random, then copy and paste again. The copy/paste process lets you fill in the “retype new password” field without error. (If pasting isn’t allowed, I question whether I want to use that service.) Again, I let my browser save the password. It will synchronize across all my devices, which means that I don’t need to maintain a list of passwords.

And what about two-factor authentication (2FA)? Yes, definitely–use it wherever possible. A text to my cellphone isn’t ideal, but it’s adequate, and preferable to sending a code to email. There are ways to attack an SMS to your phone, but it’s not easy. But be careful–I once had an app that would let me text from my laptop. If anyone texted me, it would display the text in a popup window on the laptop, which defeats the purpose of 2FA. In general, you want to receive the security code on a different device from the one you’re using to login. That’s a problem if you’re using a phone; I don’t have a good solution.

Password rotation? I resist that, although an authentication provider that I have to use requires it. The security community has long known that forcing users to change passwords on a regular basis is a bad practice. It encourages users to choose easily remembered passwords, and that’s the opposite of what we want. Think about it: if a random password hasn’t been brute-forced in the past 3 months, why do we think it’s more likely to be brute-forced in the next 3 months? I get it–companies have to deal with insurers, and perhaps forcing users who are never going to come up with good passwords to change passwords regularly is a win. I don’t want to think about those statistics. But one good password is infinitely better than a bad password that’s changed regularly.

So–that’s what I do. It’s not elegant, and please don’t claim that it represents any “best practices.” But that’s not really the point. What I choose to do is irrelevant, because I’m at the mercy of the people who create the sites I use. And their practices can be shockingly bad. Here’s a real example. I pay an elderly relative’s medical bills. Let that sink in: we’re talking one of the most privacy-conscious and heavily regulated industries in the world. Recently, I got a legitimate request to pay a bill, with a link to a site where I can view it and pay. The email tells me that the account number, user name, and password are ALL THE SAME. And the account number is contained in the email. (And easily guessable.) That’s beyond horrendous.

It’s unfortunate that there aren’t more good solutions out there, and that solutions like physical security keys aren’t more widely used. There was hope that passkeys would make passwords go away, but that hope is fading. Biometrics? If my Pixel phone would do a better job of identifying my fingerprint or recognizing my face when I take my glasses off, we could talk about that alternative. However, wishing that we had a better solution won’t solve the problem. Random passwords (regardless of how you generate them) and two-factor authentication are the best solutions we have now.

View Details

Marketing teams have been using machine learning for more than a decade. In the early days of Big Data, it was common to hear people say that marketing was Data’s killer app. As data science has evolved into artificial intelligence, people in marketing and sales have discovered a variety of ways of using data to make them more productive: helping to find the right audiences in their ad targeting, predicting just the right time to land an email in a recipient’s inbox to maximize the chances of getting an open, and even to personalize their company’s web experience or advertising to make it most appealing to their customers. Now, we’re starting to see these same teams make the use of generative AI in their marketing and sales programs to continue to drive greater productivity and efficiency from their efforts.

Three generative AI products that have made our marketing and sales teams more productive: LinkedIn’s Sales Navigator and 6Sense’s AI Email Assistant and Conversational Email products.1 Sales Navigator highlights useful information about an organization in ways that are easily useable by sales staff. This is a classic application of generative AI: it’s excellent at digesting and summarizing information, doing a lot of basic research for you. It looks at press releases, product information, LinkedIn (of course), and other sources to give an accurate, up to date picture of what’s important to an organization. Who are the key people? What partnerships are they involved in? What are their goals? What are their financials (to the extent that they’re public)? Salespeople need this information at every stage of a sale, from writing an initial email to closing the deal. Now it’s at their fingertips, without requiring hours of research.

The AI Email Assistant, which incorporates Conversational Email, is more complex. Conversational Email helps to automate the early stages of finding qualified prospects. An AI writer drafts messages to be used with prospects. The draft is created from information in the assistant’s knowledge base and suggestions from the salesperson. Prompting is familiar to anyone who has tried prompt engineering with ChatGPT: “Imagine you are a friendly salesperson sending an email to an attendee of O’Reilly’s Strata conference…”

The draft emails generated by the assistant tend to be too long and focus on our products too soon in the conversation rather than addressing the client’s needs. But editing a draft is much easier and faster for a human than starting with a blank page. Our staff often compares the assistant’s initial draft with output from other generative AI tools like ChatGPT, combining and mixing in ideas. They also edit for length; short emails are more effective than verbose messages, and anyone who has played with generative AI knows that it is verbose. The best way to use AI, as Ethan Mollick has written, might be to read AI’s suggestions and then write the message yourself. That way, it won’t sound like AI output, and it will incorporate the salesperson’s own thoughts and ideas. It’s essential to keep the human in the loop.

Drafting emails is useful, but that’s not where the real value lies. The assistant is capable of classifying and routing responses. A virtual inbox manages the conversation flow—and this ability to manage the early stages of a conversation is more important than creating draft messages. It allows a user to define different campaigns for different kinds of prospects, each with a distinct “cadence” for contacts: an initial email, followed by an email to set up a call if the response is positive, or other emails to answer initial questions.

Responses to each message are analyzed and classified into one of several categories: uninterested, interested later, serious prospect, and others. The assistant also classifies leads on the basis of their role, assigning different leads to different campaigns. Our salespeople work with many different kinds of people: technical staff, technical leads, operations, HR, executives, and more. They all have different interests—but it’s all too easy for a human to make judgements based on preconceived ideas rather than facts (for example, “programmers aren’t interested in management skills”; they are). Based on the reply, the assistant could automatically notify a salesperson of a serious prospect from the HR department and start the process of setting up a meeting. It can mark a prospect as “not interested” or “possibly interested later” and initiate a closing sequence. It can manage a referral to another potential client. And it can give potential users who don’t have decision-making authority tools to advocate for our products within their company. If the assistant has trouble classifying a message, it notifies a human. Someone can then make the classification and the AI assistant uses that information for future emails.

AI pays other dividends, too. Every response—even “not interested” or no response at all—yields information. AI can tell us which campaigns are more effective, which emails are more likely to generate a positive response, and why: what issues do prospects respond to? What do they care about?

If provided with appropriate training data, the assistant can reframe a conversation. For example, if one of the prospect’s problems is “the difficulty of hiring qualified talent” (for example, developing AI products), the assistant can attempt to reframe the conversation around developing their current staff’s skills: the demand for AI talent is huge and the supply is limited, so the fastest and most reliable way to acquire AI talent is to augment your current employees’ skillsets. Again, it’s important to keep humans in the loop so that the conversation doesn’t go off the rails—but the ability to reframe a conversation appropriately saves a lot of a salesperson’s time.

Saving time is ultimately what these tools are about, but it’s important to understand why we want to save time. We want to make our salespeople more productive, to free their time to do things that an AI can’t do—or, more appropriately, not to spend time doing things that an AI can automate. AI can’t close deals. While AI can make some simple statements about a customer’s needs, it’s not able to explore the customer’s situation deeply, help them see what the real issues are, and make suggestions about how our products fit requirements that they didn’t realize that they had. Reframing is important, but it’s only a start.

So what can AI do and what parts of the sales process can it take over? This is where classifying responses plays a huge role. Much of a salesperson’s job involves processing leads through the prospect funnel. The first few steps of that job are fairly mechanical. For example, you might send a standard email to every attendee of a conference—maybe 1,000 or 2,000 attendees. Most of them won’t reply, but you’ll still have a few hundred replies, which need to be sorted into categories. Leads can also be assigned to different campaigns, all managed through conversational email: for example, former customers can be assigned to a campaign that is designed to win them back. Managing this filtering process requires a fair amount of time-consuming work, especially if it has to be done manually: ending a conversation on a positive note, adding possible prospects to a database, and scheduling calls with the most serious prospects. That kind of filtering is an excellent job for AI.

So conversational email is really about scale: filtering 1,500 show attendees, all of whom are possible leads, down to 2 or 3 highly qualified leads, 20 or 30 possible, a few hundred to try again later, and a thousand who showed no interest. Salespeople still need to be the “humans in the loop” who edit messages, prevent conversations from going astray, and help the AI sort responses; but they have much more time to spend closing deals with the most serious prospects. In turn, AI’s ability to classify email at scale increases the number of early-stage prospects with whom you can engage. More prospects enter the funnel and in turn, that means that there will be more high-quality leads for the salespeople to work with.

So, what have we learned?

• Salespeople need to remain in the loop at every stage.
• Rewriting AI-generated messages to ensure that they have a human voice is a best practice.
• The biggest gains in efficiency come from classifying responses and managing the response pipeline, not automated email generation.
• The goal is closing more deals, not minimizing headcount.

People have used tools ever since we lived in caves, and AI is just another tool that marketing and sales can use to become more productive. We are still in the early stages of figuring out what this particular tool can do and how we can use it effectively. We’re still making the mistakes that are part of learning a new technology. But we have already seen that AI makes our salespeople more effective, makes them better at the jobs they are already doing. Is this a revolution or just incremental growth? It doesn’t matter; in either case, we are part of it.

Footnotes1. LinkedIn is a customer of O’Reilly Media. 6Sense is not. This article discusses O’Reilly’s experiences with these products. It is not sponsored by either LinkedIn or 6Sense.

View Details

In the past month, we saw a blizzard of new language models. It’s almost hard to consider this news, though Microsoft’s open (but maybe not open source) Phi-3 is certainly worth a look. We’ve also seen promising work on reducing the resources required to do inference. While this may lead to larger models, it should also lead to reduced power use for small and mid-sized models.

AI* Microsoft’s Phi-3-mini is yet another freely available language model. It is small enough to run locally on phones and laptops. Its performance is similar to GPT-3.5 and Mixtral 8x7B. * Google’s Infini-Attention is a new inference technique that allows large language models to offer infinite context. * Companies are increasingly adding AI bots to their boards as observers. The bots are there to plan strategy, help analyze financials, and report on compliance. * OutSystems offers a low-code toolkit for building AI agents, unsurprisingly named the AI Agent Builder. * Ethan Mollick’s Prompt Library is worth checking out. It collects most of the prompts from his book and his blog; most are Creative Commons, requiring only attribution. Anthropic has also published a prompt library for use with Claude, but which probably works with other LLMs. * There are many solutions for people who want to run large language models locally. They range from desktop apps to APIs. Here’s a list. * Meta has released the 8B and 70B versions of Llama 3. The largest versions are still to come. Early reports say that these smaller versions are impressive. * Mistral AI has announced Mixtral 8x22B, a larger version of their very impressive Mixtral 8x7B mixture-of-experts model. * Effort is a new method for doing LLM inference that reduces the amount of floating point computation needed without compromising the results. Effort has been implemented for Mistral, but should work with other models. * The ML Commons is developing an AI Safety Benchmark for testing AI chatbots against common kinds of abuse. They caution that the current version (0.5) is only a proof-of-concept that shouldn’t be used to test production systems. * Representative Fine Tuning is a new technique for fine-tuning language models. It is unique because it focuses specifically on the task you want the model to perform. It outperforms other fine-tuning techniques, in addition to being faster and more efficient. * AI systems can be more persuasive than humans, particularly if they have access to information about the person they are trying to persuade. This extreme form of microtargeting may mean that AI has discovered persuasive techniques that we don’t yet understand. * In one 24-hour period, there were three major language model releases: Gemini Pro 1.5, GPT-4 Turbo, and Mixtral 8x22B. Mixtral is the most interesting; it’s a larger successor to the very impressive mixture-of-experts model Mixtral 8x7B. * More models for creating music are popping up all over. There’s Sonauto (apparently not related to Suno; Sonauto uses a different kind of model) and Udio, in addition to Stable Audio and Google’s MusicLM. * An ethical application for deep fakes? Domestic Data Streamers creates synthetic images based on memories—for example, an important event that was never captured in a photo. Interestingly, older image models seem to produce more pleasing results than the latest models. * What happened after Alpha Go beat the world’s best Go player? Human Go players got better. Some of the improvement came from studying games played by AI; some of it came from increased creativity. * You should listen to Permission is Hereby Granted, Suno’s setting of the MIT License to music as a piano ballad. * How does AI-based code completion work? GitHub isn’t saying much, but Sourcegraph has provided some details for their Cody assistant. And Cody is open source, so you can analyze their code. * Claude-llm-trainer is a Google Colab notebook that simplifies the process of training Meta’s Llama 2. * In one set of experiments, large language models proved better than “classical” models at financial time series forecasting. * More easy ways to run language models locally: The Opera browser now includes support for 150 language models. This feature is currently available only in the Developer stream. * JRsdr is an AI product that promises to automate all your corporate social media. Do you dare trust it? * LLMLingua-2 is a specialized model designed to compress prompts. Compression is useful for long prompts—for example, RAG, chain-of-thought, and some other techniques. Compression reduces the context required, in turn increasing performance and reducing cost. * OpenAI has shared some samples generated by Voice Engine, their (still unreleased) model for synthesizing human voices. * Things generative AI can’t do: create a plain white image. Perhaps it’s not surprising that it’s difficult. * DeepMind has developed a large language model for checking the accuracy of an LLM’s output. Search-Augmented Factuality Evaluator (SAFE) appears to have accuracy that’s greater than crowdsourced humans, and is less expensive to operate. Code for SAFE is posted on GitHub. * While AI-generated watermarks are often seen as a way to identify AI-generated text (and, in the EU, required by law), it is relatively easy to discover a watermark and remove it or copy it for use on another document. * Particularly for vision models, being small isn’t necessarily a handicap. Small models trained on carefully curated data that’s relevant to the task at hand are less vulnerable to overfitting and other errors.

Programming* Martin Odersky, creator of the Scala programming language, has proposed “Lean Scala,” a simpler and more understandable way of writing Scala. Lean Scala is neither a new language nor a subset; it’s a programming style for Scala 3. * sotrace is a new tool for Linux developers that shows all the libraries your programs are linked to. It’s a great way to discover all of your supply chain dependencies. Try it; you’re likely to be surprised, particularly if you run it against a process ID rather than a binary executable. * Aider is a nice little tool that facilitates pair programming with GPT 3.5 or 4. It can edit the files in your git repo, committing changes with a generated descriptive message. * Another new programming language: Vala. It is object-oriented, looks sort of like Java, compiles to native binaries, and can link to many C libraries. * Excellent advice from Anil Dash: Make Better Documents. And along similar lines: write code that’s easy to read, from Gregor Hohpe. * According to Google, programmers working in Rust are roughly as effective as programmers working in Go, and twice as effective as programmers working in C++. * Winglang is a programming language for DevOps; it represents a higher level of abstraction for deploying and managing applications in the cloud. It includes a complete toolchain for developers. * Keeping track of time has always been one of the most frustratingly complex parts of programming, particularly when you account for time zones. Now the Moon needs its own time zone—because, for relativistic reasons, time runs slightly faster there. * Redict is another fork of Redis, this time under the LGPL. It is distinct from Valkey, the fork launched by the Linux Foundation. Redict will focus on “stability and long-term maintenance,” rather than innovation and new features. * “Ship it” culture is destructive. Take time to learn, understand, and document; it will pay off. * The Linux Foundation has started the Valkey project, which will fork the Redis database under an open source license. Redis is a widely used in-memory key-value database. Like Terraform and others, it was recently relicensed under terms that aren’t acceptable to the source community.

Security* GitHub allows a comment to specify a file that is automatically uploaded to the repository, with an automatically generated URL. While this feature is useful for bug reporting, it has been used by threat actors to insert malware into repos. * GPT-4 is capable of reading security advisories (CVEs) and exploiting the vulnerabilities. Other models don’t appear to have this ability, although the researchers haven’t yet been able to test Claude 3 and Gemini. * Users of the LastPass password manager have been targeted by relatively sophisticated phishing attacks. The attacks originated from the CryptoChameleon phishing toolkit. * Protobom is an open source tool that will make it easier for organizations to generate and use software bills of materials. Protobom was developed by the OpenSSF, CISA, and DHS. * Last month’s failed attack against xz Utils probably wasn’t an isolated incident. The OpenJS foundation has reported similar incidents, though they haven’t specified which projects were attacked. * System Package Data Exchange (previously known as Software Package Data Exchange 3.0) is a standard for tracking all supply chain dependencies, not just software. GitHub is integrating support to generate SPDX data from their dependency graphs. * A malicious PowerShell script that has been used in a number of attacks is believed to have been generated by an AI. (The tell is that the script has a comment for every line of code.) There will be more… * Kobold Letters is a new email vulnerability and is a real headache. A hostile agent can use CSS to modify HTML-formatted email after it is delivered, and depending on the context in which it is viewed. * AI can hallucinate package names when generating code. These non-existent names often find their way into software. Therefore, after observing a hallucinated package name, it’s possible to create malware with that name and upload it into the appropriate repository. The malware will then be loaded by software referencing the now-existent package.

Web* The Ghost publishing network now supports ActivityPub, integrating it into the Fediverse with Mastodon and other social media. Users can follow and read authors on whichever platform they like. * IFTTT now provides another low-code way to program ChatGPT. * Enhance WASM allows client-side web developers to access web components rendered on the server side by any backend runtime. No JavaScript required. * The editable Rube Goldberg machine on XKCD is a Web masterpiece. * Here’s a decent comparison of the three major federated protocols for social (and other) networking: ActivityPub (Mastodon), Nostr, and AT (Bluesky). * The FCC has voted to restore Network Neutrality and creating a national standard for open internet policies.

Robotics* Boston Dynamics has revealed its new humanoid robot, a successor to Atlas. Unlike Atlas, which used hydraulics heavily, the new robot is all electric, and has joints that can move through 360 degrees. * A research robot now uses AI to generate facial expressions and respond appropriately to facial expressions in humans. It can even anticipate human expressions and act accordingly—for example, by smiling in anticipation of a human smile.

Quantum Computing* Has post-quantum cryptography already been broken? We don’t know yet (nor do we have a working quantum computer). But a recent paper suggests some possible attacks against the current post-quantum algorithms. * Microsoft and Quantinuum have succeeded in building error corrected logical qubits: the error rate for logical qubits is lower than the error rate for uncorrected qubits. Although they can only create two logical qubits, this is a significant step forward.

View Details

By Rufus Rock, UCL; Tim O’Reilly, UCL; Ilan Strauss, UCL; and Mariana Mazzucato, UCLThis article is republished from The Conversation under a Creative Commons license.Time and again, leading scientists, technologists, and philosophers have made spectacularly terrible guesses about the direction of innovation. Even Einstein was not immune, claiming, “There is not the slightest indication that nuclear energy will ever be obtainable,” just ten years before Enrico Fermi completed construction of the first fission reactor in Chicago. Shortly thereafter, the consensus switched to fears of an imminent nuclear holocaust.

Similarly, today’s experts warn that an artificial general intelligence (AGI) doomsday is imminent. Others retort that large language models (LLMs) have already reached the peak of their powers.

It’s difficult to argue with David Collingridge’s influential thesis that attempting to predict the risks posed by new technologies is a fool’s errand. Given that our leading scientists and technologists are usually so mistaken about technological evolution, what chance do our policymakers have of effectively regulating the emerging technological risks from artificial intelligence (AI)?

We ought to heed Collingridge’s warning that technology evolves in uncertain ways. However, there is one class of AI risk that is generally knowable in advance. These are risks stemming from misalignment between a company’s economic incentives to profit from its proprietary AI model in a particular way and society’s interests in how the AI model should be monetised and deployed.

The surest way to ignore such misalignment is by focusing exclusively on technical questions about AI model capabilities, divorced from the socio-economic environment in which these models will operate and be designed for profit.

Focusing on the economic risks from AI is not simply about preventing “monopoly,” “self-preferencing,” or “Big Tech dominance.” It’s about ensuring that the economic environment facilitating innovation is not incentivising hard-to-predict technological risks as companies “move fast and break things” in a race for profit or market dominance.

It’s also about ensuring that value from AI is widely shared by preventing premature consolidation. We’ll see more innovation if emerging AI tools are accessible to everyone, such that a dispersed ecosystem of new firms, start-ups, and AI tools can arise.

OpenAI is already becoming a dominant player with US$2 billion (£1.6 billion) in annual sales and millions of users. Its GPT store and developer tools need to return value to those who create it in order to ensure ecosystems of innovation remain viable and dispersed.

By carefully interrogating the system of economic incentives underlying innovations and how technologies are monetised in practice, we can generate a better understanding of the risks, both economic and technological, nurtured by a market’s structure. Market structure is not simply the number of firms, but the cost structure and economic incentives in the market that follow from the institutions, adjacent government regulations, and available financing.

Degrading quality for higher profitIt is instructive to consider how the algorithmic technologies that underpinned the aggregator platforms of old (think Amazon, Google and Facebook among others) initially deployed to benefit users, were eventually reprogrammed to increase profits for the platform.

The problems fostered by social media, search, and recommendation algorithms was never an engineering issue, but one of financial incentives (of profit growth) not aligning with algorithms’ safe, effective, and equitable deployment. As the saying goes: history doesn’t necessarily repeat itself but it does rhyme.

To understand how platforms allocate value to themselves and what we can do about it, we investigated the role of algorithms, and the unique informational set-up of digital markets, in extracting so-called economic rents from users and producers on platforms. In economic theory, rents are “super-normal profits” (profits that are above what would be achievable in a competitive market) and reflect control over some scarce resource.

Importantly, rents are a pure return to ownership or some degree of monopoly power, rather than a return earned from producing something in a competitive market (such as many producers making and selling cars). For digital platforms, extracting digital rents usually entails degrading the quality of information shown to the user, on the basis of them “owning” access to a mass of customers.

For example, Amazon’s millions of users rely on its product search algorithms to show them the best products available for sale, since they are unable to inspect each product individually. These algorithms save everyone time and money: by helping users navigate through thousands of products to find the ones with the highest quality and the lowest price, and by expanding the market reach of suppliers through Amazon’s delivery infrastructure and immense customer network.

These platforms made markets more efficient and delivered enormous value both to users and to product suppliers. But over time, a misalignment between the initial promise of them providing user value and the need to expand profit margins as growth slows has driven bad platform behaviour. Amazon’s advertising business is a case in point.

Amazon’s advertisingIn our research on Amazon, we found that users still tend to click on the product results at the top of the page, even when they are no longer the best results but instead paid advertising placements. Amazon abuses the habituated trust that users have come to place in its algorithms, and instead allocates user attention and clicks to inferior quality, sponsored, information from which it profits immensely.

We found that, on average, the most-clicked sponsored products (advertisements) were 17% more expensive and 33% lower ranked according to Amazon’s own quality, price, and popularity optimising algorithms. And because product suppliers must now pay for the product ranking that they previously earned through product quality and reputation, their profits go down as Amazon’s go up, and prices rise as some of the cost is passed on to customers.

Amazon is one of the most striking examples of a company pivoting away from its original “virtuous” mission (“to be the most customer-centric company on Earth”) towards an extractive business model. But it is far from alone.

Google, Meta, and virtually all other major online aggregators have, over time, come to preference their economic interests over their original promise to their users and to their ecosystems of content and product suppliers or application developers. Science fiction writer and activist Cory Doctorow calls this the “enshittification” of Big Tech platforms.

But not all rents are bad. According to the economist Joseph Schumpeter, rents received by a firm from innovating can be beneficial for society. Big Tech’s platforms got ahead through highly innovative, superior, algorithmic breakthroughs. The current market leaders in AI are doing the same.

So while Schumpeterian rents are real and justified, over time, and under external financial pressure, market leaders began to use their algorithmic market power to capture a greater share of the value created by the ecosystem of advertisers, suppliers and users in order to keep profit growing.

User preferences were downgraded in algorithmic importance in favour of more profitable content. For social media platforms, this was addictive content to increase time spent on platform at any cost to user health. Meanwhile, the ultimate suppliers of value to their platform—the content creators, website owners and merchants—have had to hand over more of their returns to the platform owner. In the process, profits and profit margins have become concentrated in a few platforms’ hands, making innovation by outside companies harder.

A platform compelling its ecosystem of firms to pay ever higher fees (in return for nothing of commensurate value on either side of the platform) cannot be justified. It is a red light that the platform has a degree of market power that it is exploiting to extract unearned rents. Amazon’s most recent quarterly disclosures (Q4, 2023), shows year-on-year growth in online sales of 9%, but growth in fees of 20% (third-party seller services) and 27% (advertising sales).

What is important to remember in the context of risk and innovation is that this rent-extracting deployment of algorithmic technologies by Big Tech is not an unknowable risk, as identified by Collingridge. It is a predictable economic risk. The pursuit of profit via the exploitation of scarce resources under one’s control is a story as old as commerce itself.

Technological safeguards on algorithms, as well as more detailed disclosure about how platforms were monetising their algorithms, may have prevented such behaviour from taking place. Algorithms have become market gatekeepers and value allocators, and are now becoming producers and arbiters of knowledge.

Risks posed by the next generation of AIThe limits we place on algorithms and AI models will be instrumental to directing economic activity and human attention towards productive ends. But how much greater are the risks for the next generation of AI systems? They will shape not just what information is shown to us, but how we think and express ourselves. Centralisation of the power of AI in the hands of a few profit-driven entities that are likely to face future economic incentives for bad behaviour is surely a bad idea.

Thankfully, society is not helpless in shaping the economic risks that invariably arise after each new innovation. Risks brought about from the economic environment in which innovation occurs are not immutable. Market structure is shaped by regulators and a platform’s algorithmic institutions (especially its algorithms which make market-like allocations). Together, these factors influence how strong the network effects and economies of scale and scope are in a market, including the rewards to market dominance.

Technological mandates such as interoperability, which refers to the ability of different digital systems to work together seamlessly; or “side-loading”, the practice of installing apps from sources other than a platform’s official store, have shaped the fluidity of user mobility within and between markets, and in turn the ability for any dominant entity to durably exploit its users and ecosystem. The internet protocols helped keep the internet open instead of closed. Open source software enabled it to escape from under the thumb of the PC era’s dominant monopoly. What role might interoperability and open source play in keeping the AI industry a more competitive and inclusive market?

Disclosure is another powerful market-shaping tool. Disclosures can require technology companies to provide transparent information and explanations about their products and monetisation strategies. Mandatory disclosure of ad load and other operating metrics might have helped to prevent Facebook, for example, from exploiting its users’ privacy in order to maximise ad dollars from harvesting each user’s data.

But a lack of data portability, and an inability to independently audit Facebook’s algorithms, meant that Facebook continued to benefit from its surveillance system for longer than it should have. Today, OpenAI and other leading AI model providers refuse to disclose their training data sets, while questions arise about copyright infringement and who should have the right to profit from AI-aided creative works. Disclosures and open technological standards are key steps to try and ensure the benefits from these emerging AI platforms are shared as widely as possible.

Market structure, and its impact on “who gets what and why”, evolves as the technological basis for how firms are allowed to compete in a market evolves. So perhaps it is time to turn our regulatory gaze away from attempting to predict the specific risks that might arise as specific technologies develop. After all, even Einstein couldn’t do that.

Instead, we should try to recalibrate the economic incentives underpinning today’s innovations, away from risky uses of AI technology and towards open, accountable, AI algorithms that support and disperse value equitably. The sooner we acknowledge that technological risks are frequently an outgrowth of misaligned economic incentives, the more quickly we can work to avoid repeating the mistakes of the past.

We are not opposed to Amazon offering advertising services to firms on its third-party marketplace. An appropriate amount of advertising space can indeed help lesser-known businesses or products, with competitive offerings, to gain traction in a fair manner. But when advertising almost entirely displaces top-ranked organic product results, advertising becomes a rent extraction device for the platform.

An Amazon spokesperson said:

We disagree with a number of conclusions made in this research, which misrepresents and overstates the limited data it uses. It ignores that sales from independent sellers, which are growing faster than Amazon’s own, contribute to revenue from services, and that many of our advertising services do not appear on the store.

Amazon obsesses over making customers’ lives easier and a big part of that is making sure customers can quickly and conveniently find and discover the products they want in our store. Advertisements have been an integral part of retail for many decades and anytime we include them they are clearly marked as ‘Sponsored’. We provide a mix of organic and sponsored search results based on factors including relevance, popularity with customers, availability, price, and speed of delivery, along with helpful search filters to refine their results. We have also invested billions in the tools and services for sellers to help them grow and additional services such as advertising and logistics are entirely optional.

View Details

Why is it that Google, a company once known for its distinctive “Do no evil” guideline, is now facing the same charges of “surveillance capitalism” as Facebook, a company that never made such claims? Why is it now subject to the same kind of antitrust complaints once faced by Microsoft, the “evil empire” of the previous generation of computing? Why is it that Amazon, which has positioned itself as “the most customer-centric company on the planet,” now lards its search results with advertisements, placing them ahead of the customer-centric results chosen by the company’s organic search algorithms, which prioritize a combination of low price, high customer ratings, and other similar factors?

The answer can be found in the theory of economic rents, and in particular, in the kinds of rents that are collected by companies during different stages of the technology business cycle. There are many types of rents and an extensive economics literature discussing them, but for purposes of this article, they can be lumped into two broad categories—“rising tide rents” that benefit society as a whole, such as those that encourage innovation and the development of new markets, and “robber baron rents” that disproportionately benefit those with power.

What Is Economic Rent?Not to be confused with the ordinary sense of rent as a charge for temporary use of property, economic rents are the income above a competitive market rate that is collected because of asymmetries in ownership, information, or power.

Economists Mariana Mazzucato and Josh Ryan-Collins write, “If the reward accruing to an actor is larger than their contribution to value creation, then the difference may be defined as rent. This can be due to the ownership of a scarce asset, the creation of monopolistic conditions that enable rising returns in a specific sector, or policy decisions that favour directly or indirectly a specific group of interest.”

For example, consider drug pricing. Patents—exclusive, government-granted rights intended to encourage innovation—protect pharmaceutical companies from competition and allow them to charge high prices. Once the patents expire, there is competition from so-called “generic drugs,” and the price comes down. That difference in price (and its impact on pharmaceutical company profits) shows the extent of the rent.

In 20th century neoliberal economics, rents have typically been seen as a temporary aberration that is eventually competed away. They are a price that we pay for a rising tide of innovation. But as Mazzucato points out, to the classical economists—Smith, Ricardo, and Mill—who lived in a world of inherited power and privilege, rents were a pernicious and persistent consequence (and source) of inequality. At the dawn of economic theory, agriculture was still the chief source of value creation, and much of that value created by the labor of serfs and tenant farmers was appropriated by those who owned the land. When the local baron sent his troops to collect what he considered his share of the harvest, it was impossible to say no. In an unjust society, neither effort nor investment nor innovation but rents rooted in power asymmetries determine who gets what and why.

But not all rents represent abuse of power. As noted by economist Joseph Schumpeter, innovation—whether protected by patents, trade secrets, or just by moving faster and more capably than the competition—provides an opportunity to receive a disproportionate share of profits until the innovation is spread more widely.

During the expansive period of a new technology cycle, market leaders emerge because they solve new problems and create new value not only for consumers but also for a rich ecosystem of suppliers, intermediaries, and even competitors. Even though the market leaders tend to receive a disproportionate share of the profits as they lay waste to incumbents and dominate the emerging market, value creation is a rising tide that lifts all boats.

But this kind of virtuous rising tide rent, which benefits everyone, doesn’t last. Once the growth of the new market slows, the now-powerful innovators can no longer rely on new user adoption and collective innovation from a vibrant ecosystem to maintain their extraordinary level of profit. In the dying stages of the old cycle, the companies on top of the heap turn to extractive techniques, using their market power to try to maintain their now-customary level of profits in the face of macroeconomic factors and competition that ought to be eating them away. They start to collect robber baron rents. That’s exactly what Google, Amazon, and Meta are doing today.

Then the cycle begins again with a new class of competitors, who are forced to explore new, disruptive technologies that reset the entire market. Enter OpenAI, Anthropic, and their ilk.

Attention is all you needWhat is the source of big tech market power? What is the limited resource that they control and monopolize? It’s not our data. It’s not the price of the services we purchase from them—they give those away for free. It’s our attention.

Back in 1971, in a talk called “Designing Organizations for an Information-rich World,” political scientist Herbert Simon noted that the cost of information is not just money spent to acquire it but the time it takes to consume it.

“In an information-rich world, the wealth of information means a dearth of something else: a scarcity of whatever it is that information consumes. What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.”

In the discussion following the talk, Simon noted that in the future, information would be so abundant that we would need machines to help us manage our attention.

And that has indeed been the secret to success in the information age. Google was founded with the promise of finding the right web page out of billions, giving you just what you want and then sending you on your way. Amazon aimed to help customers find the best quality and price for any one of millions of products. Even social media started with the promise of information triage: for each person, a unique feed of updates from only the friends they had chosen to follow. These are all astonishing tools for making our limited capacity for attention more efficient.

In the early idealistic days of internet expansion, the leading companies earned outsized profits by solving the attention allocation problem. As the internet grew, the amount of information available to consumers became so vast that it outran traditional human means of curation and selection. Attention allocation was outsourced to the machines. Algorithms for search, recommendations, social media feeds, entertainment, and news became the foundation of an enormous new economy.

The internet giants succeeded by doing what they are now too often reviled for: extracting signal from massive amounts of data. Google not only crawled and indexed virtually every page on the web, it looked at how sites linked to each other, tracked which of the ten top links it showed were clicked on the most, which ones led people to come back and try another and which sent them away satisfied. It used location data and past searches to make answers more relevant and personalized. Amazon too used everything from price, user reviews, popularity, and your individual purchase history to bring to the top the products they believed best matched their customers’ needs. In my 2005 essay “What is Web 2.0?,” I made the case that the companies that had survived the dotcom bust had all in one way or another become experts at “harnessing collective intelligence.”

Perhaps a more direct way to say this in the context of economic value creation is that companies such as Amazon, Google, and Facebook had developed a set of remarkable advances in networked and data-enabled market coordination.

But over time, something went very wrong. Instead of continuing to deploy their attention optimization algorithms for their users’ and suppliers’ benefit, the tech giants began to use them to favor themselves. It first became obvious with social media: recommended posts and amplification of addictive, divisive content in order to keep users scrolling, creating additional surface area for advertising. Google began to place more and more advertising ahead of “organic” search results, turning advertising from a complementary stream of useful information that ran beside search results into a substitute. Amazon was late to the party, but once it discovered advertising, it went all in. Now a typical page of Amazon product search results consists of 16 ads and only four organic results.

Google and Amazon were still atop their respective hills of web search and ecommerce in 2010, and Meta’s growth was still accelerating, but it was hard to miss that internet growth had begun to slow. The market was maturing. From 2000 to 2011, the percentage of US adults using the internet had grown from about 60% to nearly 80%. By the end of 2012, it was up to 82%. But in 2013 and 2014, it remained stuck at 83%, and while in the ten years since, it has reached 95%, it had become clear that the easy money that came from acquiring more users was ending. Penetration in Europe, the other lucrative market, was on a similar track to the US, and while there was lots of user growth still to be found in the rest of the world, the revenue per user was much lower. What are now-gigantic companies to do when their immense market capitalization depends on rapid growth and the expectation of growing profits to match?

These companies did continue to innovate. Some of those innovations, like Amazon’s cloud computing business, represented enormous new markets and a new business model. But the internet giants also came to focus on extracting more usage and time spent, and thus more revenue, from a relatively stable base of existing customers. Often this was done by making their products more addictive, getting more out of their users by nefarious means. Cory Doctorow calls this the “enshittification” of Big Tech platforms.

Fast forward to the present, and Amazon has clearly given up on the goal of finding the best result for its users. Since launching its Marketplace advertising business in 2016, Amazon has chosen to become a “pay to play” platform where the top results are those that are most profitable for the company.

In “Amazon is burying organic search results,” research firm Marketplace Pulse notes:

Of the first twenty products a shopper sees when searching on Amazon, only four are organic results. There is little space left for organic results at the top of the page, the real estate that drives most sales. Few purchases happen beyond the first page of search results. And not many shoppers scroll to the bottom of even the first page…

It takes scrolling past three browser windows worth of search results to get to the fifth organic result. It takes even more swipes to see the fifth organic result on mobile.

This is what we mean by a “robber baron” rent: “pay us, or you’ll effectively disappear from search.”

The harm to users isn’t just time lost while scrolling through ads to find the best results. In a recent research project at University College London’s Institute for Innovation and Public Purpose, my colleagues and I found that users still tend to click on the product results at the top of the page even when they are no longer the best results. Amazon abuses the trust that users have come to place in its algorithms, and instead allocates user attention and clicks to inferior quality sponsored information. The most-clicked sponsored products were 17% more expensive and 33% lower ranked according to Amazon’s own quality, price, and popularity optimizing algorithms. And because product suppliers must now pay for the product ranking that they previously earned through product quality and reputation, their profits go down as Amazon’s go up, and prices rise as some of the cost is passed on to customers.

It appears to have worked—for now. Amazon’s recent quarterly disclosures (Q4, 2023), for example, show year-on-year growth in online sales revenue of 9%, but growth in fees of 20% (third-party seller services) and 27% (advertising sales). But the historical lessons from the downfall of both IBM mainframe monopoly and Microsoft’s stranglehold on the personal computer suggests that the company will be forced to renew its commitment to value creation or face decline and challenges from new, disruptive market entrants who are focused on providing the kind of value to users and suppliers that Amazon once did. The damage to Amazon may be a gradual downslope or a sudden cliff. When does brand and reputation damage accumulate to the point that consumers start trusting Amazon less, shopping at Amazon less, and expending the effort of trying alternatives? If history is any judge, it will happen sooner or later unless Amazon dials back the rents.

A similar dark pattern is visible in the evolution of Google search. Starting around 2011, advertising, which once framed the organic results and was clearly differentiated from them by color, gradually became more dominant, and the signaling that it was advertising became more subtle. Today, especially on mobile, the user may have to scroll down several times to get to the first organic result. The result is less striking than on Amazon, since a very large percentage of Google searches carry no advertisements at all. But for commercial searches, the best result for users (a local merchant, for example) can often only be found after scrolling through pages of ads from internet sellers and national chains.

The harms to users are thus less than they appear to be at Amazon, where advertising distorts the results of every search, but there are still serious concerns. Both Google and Amazon are gatekeepers controlling the visibility of a vast ecosystem of suppliers. Those suppliers aren’t just a commodity to be exploited by the platform. They are its partners in creating the value that draws users to the platform. Without websites, there would be no need for Google search or raw material for its results; without merchants, no Amazon. The same is true of other internet gatekeepers. Without app developers, there would be no App Stores; without users creating content as well as consuming it, no social media.

This is what we mean by a “robber baron” rent: “pay us, or you’ll effectively disappear from search.”When suppliers are harmed, users too will be harmed over the long run. These ecosystems of value co-creators depend on the platform’s fairness in allocating attention to the most relevant results. When the platform displaces organic results with paid results, preferences its own applications, products, or services, or provides information directly to the consumer in competition with the originators of that information, the ecosystem suffers a loss of incentive and reward for continuing to produce value. Eventually, this loss of value affects both users and the platform itself, and the whole virtuous circle of creation, aggregation, and curation breaks down.

The company itself is also harmed, as even its own innovations may be held back in order to protect lucrative existing lines of business. Google, for example, invented the Large Language model architecture that underlies today’s disruptive AI startups. It published the original Transformer paper (not quite coincidentally called “Attention is All You Need”) in 2017, and released BERT, an open source implementation, in late 2018, but never went so far as to build and release anything like OpenAI’s GPT line of services. It’s unclear whether this was a lack of imagination or a kind of “strategy tax.” It was certainly obvious to outsiders how disruptive BERT could be to Google Search. In 2020, when my own company released O’Reilly Answers, a plain language search engine based on BERT for the content on the O’Reilly platform, I was struck by how, for the first time, we could search our own content better than Google could.

It was left to startups to explore the broader possibilities of generative AI and chatbots.

Will History Repeat Itself?The enshittification of Amazon and Google is old news to most users. We remember how good these services used to be, and lament their decline. But we have slowly gotten used to the fact that results are not what they once were.

Antitrust authorities in Europe and the US have woken up, and are questioning abuses of market power by Big Tech companies, albeit not always successfully. Regulators may force better behavior. My hope, though, is that in responding to new competitors, the companies themselves may wake up and pull back from the brink before it’s too late.

It’s already clear that LLMs may offer the greatest competition that Google, Amazon, and other current internet giants have ever faced. While the results are as yet inferior to those offered by Google and Amazon, users are already asking questions of ChatGPT that would once have gone to a search engine. The lower quality of the results is typical in the early days of a disruptive technology. It doesn’t matter, because disruptive technologies start out by solving new problems, serving new markets, and creating new opportunities. But their disruptive quality also comes because novel technology companies draw outside the lines that have been drawn to protect the business model of the existing players. They are eager to surprise and delight their users; the focus in the early days is always on value creation. Mature and declining companies, by contrast, tend to hobble their products as they focus on value extraction. They lose their ideals and their edge, eventually alienating their customers and their suppliers and opening the door to competition.

We are in those early days once again. Leadership comes to those who create the most value for the most users. It is only later, after the market consolidates, that the value extraction phase begins. At that point, will the new market leaders also turn to more traditional extractive techniques? Just like today’s incumbents, will they end up using their market power to protect their now-customary level of profits in the face of macroeconomic factors and competition that ought to be eating them away?

Regulators would be wise to get ahead of this development. The current generation of algorithmic overlords shape the attention of their users, helping to decide what we read and watch and buy, whom we befriend and whom we believe. The next generation will shape human cognition, creativity, and interaction even more profoundly.

There is a great deal of discussion about the risks and benefits of AI, but it is generally focused narrowly on the technical capabilities of AI tools and whether continued advances will eventually put AI beyond human control, leading to possible disaster. Closer to the present, risk analysis focuses on social problems like bias, misinformation, and hate speech, or the potential spread of biological and nuclear capabilities.

Yet many of the most pressing risks are economic, embedded in the financial aims of the companies that control and manage AI systems and services. Are AI companies going to be immune to the incentives that have made today’s current tech giants turn against their users and their suppliers, the same incentives that have led financial institutions to peddle bad assets, pharmaceutical companies to promote opioids, cigarette companies to hide the health risks of smoking, and oil companies to deny climate change? I think not.

Rather than blaming the moral failings of company leadership, look instead to the economic incentives that rule public companies. Financial markets (including venture capitalists considering valuation of the next round) reward companies handsomely for outsized growth of revenue and profit, while brutally punishing any slowdown. Since stock options are a large part of executive compensation—and all compensation at Silicon Valley companies—failing to deliver the required growth comes at a very high cost to company leadership and employees.

It is too early to know best how to regulate AI. But one thing is certain. You can’t regulate what you don’t understand. Economic abuses by companies typically hide in plain sight for years, with whistleblowers, researchers, regulators, and lawyers struggling to prove what the companies continue to deny. This is going to be even more true of an inscrutable black box like AI.

AI safety and governance will be impossible without robust and consistent institutions for disclosure and auditing. To achieve prosocial outcomes, AI model and application developers need to define the metrics that explicitly aim for those outcomes and then measure and report the extent to which they have been achieved. These are not narrow technical disclosures of model capabilities, but the metrics the companies use to manage AI as a business, including what processes and metrics they use to reduce the risks that have been identified. If they begin to twist AI’s training, guardrails, and objectives for their own benefit, we should be able to see it in the numbers.

The time to do this is now, when AI developers are still in the virtuous stage of innovation and rising tide rents, and while the companies are exploring the possibilities of AI regulation. It is important to understand what “good” looks like while companies are still putting their best foot forward, developing services to delight and serve users and suppliers and society, so that if (or perhaps when) the incentives to take advantage of others take over, we can look back and see when and how things began to go wrong.

Let’s not wait till the robber barons are back.

A longer version of this article was previously published as part of the UCL Institute for Innovation and Public Purpose, Working Paper Series (IIPP WP 2024-04). Available at: https://www.ucl.ac.uk/bartlett/public-purpose/wp2024-04. That version includes additional history of earlier cycles of value creation and extraction during the mainframe and PC eras.

View Details

We’ve been very lucky. A couple of weeks ago, a supply-chain attack against the Linux xz Utils package, which includes the liblzma compression library, was discovered just weeks before the compromised version of the library would have been incorporated into the most widely used Linux distributions. The attack inserted a backdoor into sshd that would have given threat actors remote shell access on any infected system.

The details of the attack have been thoroughly discussed online. If you want a blow-by-blow exposition, here are two chronologies. ArsTechnica, Bruce Schneier, and other sources have good discussions of the attack and its implications. For the purposes of this article, here’s a brief summary.

The malware was introduced into xz Utils by one of its maintainers, an entity named Jia Tan. That’s almost certainly not a person’s name; the actual perpetrator is unknown. It’s likely that the attacker is a collective operating under a single name. Jia Tan began several years ago by submitting a number of changes and fixes to xz, which were included in the distribution, establishing a reputation for doing useful work. A coordinated attack against xz’s creator and maintainer, Lasse Collin, complained that Collin wasn’t approving patches quickly enough. This pressure eventually convinced him to add Jia Tan as a maintainer.

Over two years, Jia Tan gradually added compromised source files to xz Utils. There’s nothing really obvious or actionable; the attackers were slow, methodical, and patient, gradually introducing components of the malware and disabling tests that might have detected the malware. There were no changes significant enough to attract attention, and the compromises were carefully concealed. For example, one test was disabled by the introduction of an innocuous single-character typo.

Only weeks before the compromised xz Utils would have become part of the general release of RedHat, Debian, and several other distributions, Andres Freund noticed some performance anomalies with the beta distribution he was using. He investigated further, discovered the attack, and notified the security community. Freund made it clear that he is not a security researcher, and that there may be other problems with the code that he did not detect.

Is that the end of the story? The compromised xz Utils was never distributed widely, and never did any damage. However, many people remain on edge, with good reason. Although the attack was discovered in time, it raises a number of important issues that we can’t sweep under the rug:

• We’re looking at a social engineering attack that achieves its aims by bullying—something that’s all too common in the Open Source world.
• Unlike most supply chain attacks, which insert malware covertly by slipping it by a maintainer, this attack succeeded in inserting a corrupt maintainer, corrupting the release itself. You can’t go further upstream than that. And it’s possible that other packages have been compromised in the same way.
• Many in the security community believe that the quality of the malware and the patience of the actors is a sign that they’re working for a government agency.
• The attack was discovered by someone who wasn’t a security expert. The security community is understandably disturbed that they missed this.

What can we learn from this?

Everyone is responsible for security. I’m not concerned that the attack wasn’t discovered by the a security expert, though that may be somewhat embarrassing. It really means that everyone is in the security community. It’s often said “Given enough eyes, all bugs are shallow.” You really only need one set of eyeballs, and in this case, those eyeballs belonged to Andres Freund. But that only begs the question: how many eyeballs were watching? For most projects, not enough—possibly none. If you notice something that seems funny, look at it more deeply (getting a security expert’s help if necessary); don’t just assume that everything is OK. “If you see something, say something.” That applies to corporations as well as individuals: don’t take the benefits of open source software without committing to its maintenance. Invest in ensuring that the software we share is secure. The Open Source Security Foundation (OpenSSF) lists some suspicious patterns, along with best practices to secure a project.

It’s more concerning that a particularly abusive flavor of social engineering allowed threat actors to compromise the project. As far as I can tell, this is a new element: social engineering usually takes a form like “Can you help me?” or “I’m trying to help you.” However, many open source projects tolerate abusive behavior. In this case, that tolerance opened a new attack vector: badgering a maintainer into accepting a corrupted second maintainer. Has this happened before? No one knows (yet). Will it happen again? Given that it came so close to working once, almost certainly. Solutions like screening potential maintainers don’t address the real issue. The kind of pressure that the attackers applied was only possible because that kind of abuse is accepted. That has to change.

We’ve learned that we know much less about the integrity of our software systems than we thought. We’ve learned that supply chain attacks on open source software can start very far upstream—indeed, at the stream’s source. What we need now is to make that fear useful by looking carefully at our software supply chains and ensuring their safety—and that includes social safety. If we don’t, next time we may not be so lucky.

View Details

A recent article in Fast Company makes the claim “Thanks to AI, the Coder is no longer King. All Hail the QA Engineer.” It’s worth reading, and its argument is probably correct. Generative AI will be used to create more and more software; AI makes mistakes and it’s difficult to foresee a future in which it doesn’t; therefore, if we want software that works, Quality Assurance teams will rise in importance. “Hail the QA Engineer” may be clickbait, but it isn’t controversial to say that testing and debugging will rise in importance. Even if generative AI becomes much more reliable, the problem of finding the “last bug” will never go away.

However, the rise of QA raises a number of questions. First, one of the cornerstones of QA is testing. Generative AI can generate tests, of course—at least it can generate unit tests, which are fairly simple. Integration tests (tests of multiple modules) and acceptance tests (tests of entire systems) are more difficult. Even with unit tests, though, we run into the basic problem of AI: it can generate a test suite, but that test suite can have its own errors. What does “testing” mean when the test suite itself may have bugs? Testing is difficult because good testing goes beyond simply verifying specific behaviors.

The problem grows with the complexity of the test. Finding bugs that arise when integrating multiple modules is more difficult and becomes even more difficult when you’re testing the entire application. The AI might need to use Selenium or some other test framework to simulate clicking on the user interface. It would need to anticipate how users might become confused, as well as how users might abuse (unintentionally or intentionally) the application.

Another difficulty with testing is that bugs aren’t just minor slips and oversights. The most important bugs result from misunderstandings: misunderstanding a specification or correctly implementing a specification that doesn’t reflect what the customer needs. Can an AI generate tests for these situations? An AI might be able to read and interpret a specification (particularly if the specification was written in a machine-readable format—though that would be another form of programming). But it isn’t clear how an AI could ever evaluate the relationship between a specification and the original intention: what does the customer really want? What is the software really supposed to do?

Security is yet another issue: is an AI system able to red-team an application? I’ll grant that AI should be able to do an excellent job of fuzzing, and we’ve seen game playing AI discover “cheats.” Still, the more complex the test, the more difficult it is to know whether you’re debugging the test or the software under test. We quickly run into an extension of Kernighan’s Law: debugging is twice as hard as writing code. So if you write code that’s at the limits of your understanding, you’re not smart enough to debug it. What does this mean for code that you haven’t written? Humans have to test and debug code that they didn’t write all the time; that’s called “maintaining legacy code.” But that doesn’t make it easy or (for that matter) enjoyable.

Programming culture is another problem. At the first two companies I worked at, QA and testing were definitely not high-prestige jobs. Being assigned to QA was, if anything, a demotion, usually reserved for a good programmer who couldn’t work well with the rest of the team. Has the culture changed since then? Cultures change very slowly; I doubt it. Unit testing has become a widespread practice. However, it’s easy to write a test suite that give good coverage on paper, but that actually tests very little. As software developers realize the value of unit testing, they begin to write better, more comprehensive test suites. But what about AI? Will AI yield to the “temptation” to write low-value tests?

Perhaps the biggest problem, though, is that prioritizing QA doesn’t solve the problem that has plagued computing from the beginning: programmers who never understand the problem they’re being asked to solve well enough. Answering a Quora question that has nothing to do with AI, Alan Mellor wrote:

We all start programming thinking about mastering a language, maybe using a design pattern only clever people know.

Then our first real work shows us a whole new vista.

The language is the easy bit. The problem domain is hard.

I’ve programmed industrial controllers. I can now talk about factories, and PID control, and PLCs and acceleration of fragile goods.

I worked in PC games. I can talk about rigid body dynamics, matrix normalization, quaternions. A bit.

I worked in marketing automation. I can talk about sales funnels, double opt in, transactional emails, drip feeds.

I worked in mobile games. I can talk about level design. Of one way systems to force player flow. Of stepped reward systems.

Do you see that we have to learn about the business we code for?

Code is literally nothing. Language nothing. Tech stack nothing. Nobody gives a monkeys [sic], we can all do that.

To write a real app, you have to understand why it will succeed. What problem it solves. How it relates to the real world. Understand the domain, in other words.

Exactly. This is an excellent description of what programming is really about. Elsewhere, I’ve written that AI might make a programmer 50% more productive, though this figure is probably optimistic. But programmers only spend about 20% of their time coding. Getting 50% of 20% of your time back is important, but it’s not revolutionary. To make it revolutionary, we will have to do something better than spending more time writing test suites. That’s where Mellor’s insight into the nature of software so crucial. Cranking out lines of code isn’t what makes software good; that’s the easy part. Nor is cranking out test suites, and if generative AI can help write tests without compromising the quality of the testing, that would be a huge step forward. (I’m skeptical, at least for the present.) The important part of software development is understanding the problem you’re trying to solve. Grinding out test suites in a QA group doesn’t help much if the software you’re testing doesn’t solve the right problem.

Software developers will need to devote more time to testing and QA. That’s a given. But if all we get out of AI is the ability to do what we can already do, we’re playing a losing game. The only way to win is to do a better job of understanding the problems we need to solve.

View Details

This article originally appeared in The Information on March 5th, 2024.

“The economic problem of society…is a problem of the utilization of knowledge which is not given to anyone in its totality.”

—Friedrich A. Hayek, “The Use of Knowledge in Society”

Silicon Valley venture capitalists and many entrepreneurs espouse libertarian values. In practice, they subscribe to central planning: Rather than competing to win in the marketplace, entrepreneurs compete for funding from the Silicon Valley equivalent of the Central Committee. The race to the top is no longer driven by who has the best product or the best business model, but by who has the blessing of the venture capitalists with the deepest pockets—a blessing that will allow them to acquire the most customers the most quickly, often by providing services below cost. Reid Hoffman called this pattern “blitzscaling,” claiming in the subtitle of his book with that name that it is “The Lightning-Fast Path to Building Massively Valuable Companies.”

I disagree. It is a dark pattern, a map to suboptimal outcomes rather than the true path to competition, innovation and the creation of robust companies and markets. As Bill Janeway noted in his critique of the capital-fueled bubbles that resulted from the ultra-low interest rates of the decade following the 2007–2009 financial crisis, “capital is not a strategy.”

Venture capitalists don’t have a crystal ball. To the extent that entrepreneurial funding is more concentrated in the hands of a few, private finance can drive markets independent of consumer preferences and supply dynamics. Market discipline is significantly delayed—until the initial public offering or later. And of course, today IPOs are delayed, often precisely because companies can get all the capital they need from a small number of deep-pocketed investors. Founders and employees are even able to cash out some of their shares without having to face the scrutiny of public markets, much as if bettors on a horse race could take their money off the table as the horses round the first turn. Thus, far from finance being an extension of the market (with lots of independent signals aggregated to ensure competition and consumer choice), capital can ignore the will of the market.

The ride-hailing business offers a classic example of the distortive over-reliance on capital rather than consumer choice. It began with bold prophecies of ride-hailing replacing not just taxis but all private vehicles, and ended with a national duopoly of on-demand taxis at prices no better and often worse than those of the previous over-regulated local taxi market. In a well-functioning market, many startups would have explored a technology innovation like on-demand transportation over a much longer period. In that alternate history, entrepreneurs would have competed with different pricing strategies, different rate structures for drivers and perhaps even completely different business models. Eventually, those that survived would have done so because they were delivering the service chosen by the most customers and the most drivers. That is true product-market fit.

But in the Central Committee version of Silicon Valley, Uber and Lyft, backed by billions of dollars of venture capital, drove out the competition rather than defeating it, subsidizing customer acquisition and an unsustainable business model—and in the case of Uber, continuing to attract new capital with promises of speculative future cost savings via self-driving cars. Instead, once the market had consolidated, Uber and Lyft only reached profitability through massive price increases. What might have happened if there had been true competition in this market? We will never know.

By contrast, during the dot-com bubble, most companies consumed tiny amounts of capital by today’s standards. The funding was spread across thousands of companies, and it took a decade or more of relentless innovation and competition for the industry to become dangerously concentrated. This is a classic example of what Janeway calls a “productive bubble.” Remarkably, most of the winning companies were profitable in only a few years, and eventually they became hugely profitable. Google raised only $36 million in venture capital on its way to dominance. Facebook raised billions, but it did so only to fund faster growth for a business model that insiders have told me was very close to profitable the entire time. They weren’t buying users with subsidized prices; they were building data centers. Even Amazon, long unprofitable, took in very little investment capital, instead funding itself with debt supported by a business model that produced previously unprecedented levels of free cash flow.

To be sure, sometimes companies do require a lot of capital to lay the groundwork for a possible future. Tesla and SpaceX are good examples. They used their funding to do serious research and development, to build factories, cars, batteries, rockets and satellites. This is using capital properly: to fund the hard costs associated with creating something new until the projected unit economics lead to a self-sustaining business. It’s also worth noting that in those cases private funding was powerfully augmented by state support: carbon credits and electric vehicle incentives for Tesla, and NASA progress payments for SpaceX.

That kind of investment was unnecessary in the case of ride-hailing. The startups simply used the money to amass market power by subsidizing blitzscaled growth. Others had already deployed the capital to build much of the infrastructure for ride-hailing—GPS satellites and GPS-enabled smartphones. Even the innovation of using GPS to match passengers and drivers was not developed by the VC-backed market leaders, but by the true market pioneer, Sidecar, which was quickly sidelined when it failed to raise enough capital to gain a leading share in the market it had first envisioned.

In the case of artificial intelligence, training large models is indeed expensive, requiring large capital investments. But those investments demand commensurately large returns. The investors who pile billions of dollars into a huge bet are expecting not just to be paid back, but paid back a hundredfold. The capital-fueled race to build the largest models has already led to bad behavior. OpenAI, for example, has trained not just on publicly available data but reportedly on copyrighted content retrieved from pirate sites. This has led to lawsuits and settlements. But even those settlements are likely to be bad for the development of a healthy entrepreneurial ecosystem. As Mike Loukides points out, “Smaller startups…will be priced out, along with every open-source effort. By settling, OpenAI will eliminate much of their competition.”

Meanwhile, the largest models’ absorption of all content into “the Borg” of AI data will eliminate opportunities for the owners of specialized content repositories to profit from their own work. Innovators are already finding that much can be done at lower cost with smaller, more targeted open-source models. They can fine-tune these smaller models for specific problem domains, allowing trusted content providers (like my own company’s O’Reilly Answers and related AI-generated services) to profit from our own expertise.

OpenAI is making an effort to create a platform on which entrepreneurs can build vertical applications, but only if they pay tribute to the centralized business model in the form of API fees. OpenAI is also skimming the cream, quickly dominating some of the most profitable categories—image generation, video generation, speech synthesis, computer programming—that in a well-functioning market would be explored by dozens or hundreds of competing efforts, until one or two find the winning combination of product and business model. If entrepreneurs discover other profitable categories, giants such as OpenAI will move quickly to dominate these as well.

The capital-fueled AI land grab is of course only one axis of premature market concentration. As Max von Thun points out in “Monopoly Power Is the Elephant in the Room in the AI Debate,” much of the investment to train models is coming in the form of strategic partnerships (including both cloud computing credits and potential revenue deals) with existing industry giants Microsoft, Amazon and Google (and in the case of open-source models, Meta Platforms). As von Thun notes, “These partnerships appear to be serving the same purpose as ‘killer acquisitions’ in the past—think of Facebook’s acquisition of WhatsApp or Google’s purchase of YouTube—raising serious concerns about fair competition in the fledgling AI market.” The risk of these deals is, again, that a few centrally chosen winners will quickly emerge, meaning there’s a shorter and less robust period of experimentation.

And, at least based on recent reporting by The Information about Anthropic’s operating margins, it may be that, like Uber and Lyft, the overfunded AI market leaders may only be able to deliver on investors’ heated expectations by crushing all competition. That’s not betting on the wisdom of the market and what Hayek called “the utilization of knowledge which is not given to anyone in its totality.” That’s betting on premature consolidation and the wisdom of a few large investors to choose a future everyone else will be forced to live in.

View Details

There are lots of new models, including one from Apple, but that’s hardly news. AI news is infiltrating other sections of Trends (particularly Programming and Security)—but that’s also hardly news. NVIDIA CEO Jensen Huang has said that AI will replace coding—but again, he’s not the first. But what’s new is Devin: an AI software engineer from Cognition Labs. Its makers claim that it “can execute complex engineering tasks requiring thousands of decisions. Devin can recall relevant context at every step, learn over time, and fix mistakes.” Devin is in early access; what we’ve heard from those who’ve used it is that it’s far from finished—but even in that state, it’s very impressive. There’s also an open source OpenDevin project on GitHub.

A supply chain attack added a back door to Linux systems through the widely used xz package. Fortunately, this attack was discovered before the package was incorporated into the leading Linux distributions. However, the attack raises a lot of troubling questions about security—including the question of how we know software is trustworthy. The attack wasn’t discovered by security experts analyzing the code; social engineering may even have been used to prevent it from being tested adequately. The attack was discovered by an engineer who noticed some performance anomalies. Nobody knows who the maintainer who inserted the back door really is; it may not be a single person. This time, we were lucky.

Artificial Intelligence* What does open source mean for AI? What does it include aside from code? Can there be restrictions on how the AI is used? Nobody knows. And it’s entirely too likely that the vacuum will be filled by a self-interested definition coming from one of the internet giants. * Jan is a new way of packaging open source language models so that they can be run completely locally. It’s available for Windows, macOS, and Linux. * Can AI be an aid to observability? Yes—both by recognizing normal and abnormal activity and by interpreting and summarizing log data and making suggestions for resolving problems. * NVIDIA has announced that they intend to create an “embodied” AI: an AI incorporated into a humanoid robotic form. Is this “moonshot” just an attempt at realized science fiction or an important step on the road to general intelligence? * At NVIDIA’s developer conference, their CEO outlined a vision for the future of programming in which AI systems replace the entire development pipeline. Humans remain in the loop and in control, but they will only use natural human languages. * The LLM4Decompile project is building large language models for decompiling software (translating from assembly back into a higher level language like C). This would be a tremendous tool for reverse engineering. The models are available on Hugging Face. * Now Apple has a large language model. It isn’t open to the public, but they’ve published a paper about it. In a Twitter post (that I can’t find) they claim performance similar to Gemini-1 at each model size. * Answer.ai is releasing an open source system for fine tuning large language models with up to 70B parameters. It can run on a desktop computer with two commodity gaming GPUs. * A month or two ago, we noted that attackers have proven that they can jailbreak large language models by using steganographic techniques to hide hostile prompts within an image. It turns out you don’t have to be subtle: ASCII art that spells out the hostile words will suffice. * SudoLang is a programming language for interacting with large language models. It’s not the only attempt along these lines; we’ve also noted GPTScript. SudoLang is particularly interesting, though, because the language was designed in part by GPT-4. * Simon Willison clarifies the distinction between prompt injection and jailbreaking. Prompt injection involves concatenating trusted and untrusted input in prompts. It is far more dangerous than jailbreaking and harder (perhaps impossible) to defend against. * A generative AI platform called Lore Machine can take a short story and turn it into an illustrated comic. * ToxicChat is a new benchmark for detecting toxic prompts sent to language models. It is based on actual prompts collected by language models rather than social media content. * Anthropic’s latest series of models, Claude 3, is now available. The most advanced model, Opus, is only available through subscription. All of them feature a 200,000-token context window. * Over the past few years, large models have reduced their data requirements by going from 32-bit floating point to 8 bits to 4 bits, in a process called “quantization.” The next step forward is single-bit models (actually, 1.58 bits). * GPTScript is a simple programming language for automating interactions with GPT. It’s starting to look a lot like a formal informal language.

Programming* GitHub now offers Code Scanning Autofix, a service that uses AI to detect vulnerabilities and suggest fixes to code written in Java, JavaScript, Python, and TypeScript. They claim that it can detect and correct over 90% of known vulnerabilities. They note that it’s still important for the programmer to verify that the suggestion actually fixes the vulnerability. * JetBrains now offers TeamCity Pipelines, a CI/CD tool for small to midsized teams. It is currently in public beta. Simpler tools that solve the problems of smaller projects are a welcome addition to the tooling scene. * Ravi is a new dialect of Lua that supports optional static typing. It has a just-in-time compiler and can also compile directly to machine code. * BOINC is a project that lets you allow scientific computing tasks to run on your computer in the background. It’s similar to projects like SETI@Home, but more general; it isn’t associated with a specific research project. BOINC is based at UC Berkeley and supported by the NSF. * Devin is “the world’s first fully autonomous AI software engineer.” The claims made for Devin are impressive: it can learn new technologies from a blog post, build deploy apps, fix bugs, train language models, and more. If it lives up to these claims, it will be very impressive. * A startup has released open source libraries for fully homomorphic encryption. Homomorphic encryption is a set of codes and protocols for computing with encrypted data without first decrypting the data. * We know that language models can assist in writing code. Can they also assist in building infrastructure as code? * GitHub is being attacked by cybercriminals who are creating millions of repositories containing malware. The malicious repos have names similar to legitimate repos in hopes that programmers will use the wrong repo (often with the encouragement of social engineering). * Github is offering Copilot Enterprise, a higher-priced version of Copilot that knows about a company’s codebase. Code completions are based on code in the company’s repositories, so they match the company’s practices. It can even learn proprietary in-house languages. * Wax is an open source framework for building word processing software on the web. It facilitates change tracking, commenting, equations, basic text styling, managing citations, and other features you’d expect in a professional word processing system.

Operations* Brendan Gregg has posted a list of Linux crisis tools: utilities that you are likely to need to diagnose and fix an outage and that your favorite distribution might not have. * DBOS is a new cloud native operating system that is based on a high performance distributed database. It is intended to replace the Linux/Kubernetes combination that has become the basis for orchestrating complex distributed applications. * Buoyant is now charging organizations with 50 or more users for access to the latest stable release of the linkerd service mesh. They have not changed linkerd’s licensing, which is still open source (Apache 2.0). * Netflix has released bpftop, a command line tool for monitoring programs that use eBPF (extended Berkeley packet filters). bpftop gives users insight into their eBPF tools, preventing eBPF from inadvertently compromising performance while attempting to improve performance.

Web* Facebook’s Threads now allows users to share their posts on Mastodon. The feature is currently opt-in. Threads users can’t yet view posts made by Mastodon users. How Facebook will handle Mastodon users’ private data and dislike of advertising remains to be seen. * Ludic is a new lightweight web framework that is built to be used with htmx. It uses a component approach similar to React but does not require any JavaScript to build dynamic applications. It is based on Python 3.12. * YouTube is requiring creators to disclose when they have used generative AI to create or modify otherwise realistic video. This rule does not apply to content that is “clearly unrealistic” (e.g., animations), color adjustments or beauty filters, and background effects (e.g., blur). * LaVague is a large language model designed for controlling browser interactions. It would be ideal for controlling a testing framework like Selenium. And it may be useful for automating other “mundane tasks.” * The Bluesky social network, created by Twitter founder Jack Dorsey, now allows federation: individuals and groups can now run their own servers, similar to Mastodon.

Security* A supply-chain compromise added a backdoor to Linux’s xz package. The attack raises many questions. Social engineering may have prevented Google from testing it adequately; the maintainer was pressured into adding a second maintainer who was probably the attacker and may have been state-sponsored; and it was discovered because of performance anomalies. * Loop DoS is a new denial of service attack in which targeted computers send UDP packets back and forth in an infinite loop. Equipment from several major vendors, including Cisco, Microsoft, and Broadcom, is reported to be vulnerable. * A new attack against LLMs allows attackers to recover the texts of chat sessions even if they are encrypted. The attack is based on observing the lengths of the tokens and matching the lengths to words. This vulnerability applies to all LLMs except for Google’s Gemini. * Pixieboot (aka PXE boot) is a collection of attacks against UEFI firmware, a very low-level system-within-a-system that controls the boot process on most modern PCs. While this particular set of vulnerabilities is mostly of concern to cloud and datacenter operators, Cory Doctorow writes about the danger of nonupdateable subsystems that treat the user as a threat. * Cloudflare is introducing an AI firewall product that, among other things, will eventually include a prompt validation feature that will detect and block prompt injection attacks. The feature may help with jailbreaking (a single hostile prompt), but it’s harder to see how it would be effective against true prompt injection (a hostile prompt concatenated with a trustworthy prompt). * A paper analyzes over 600,000 prompt injection attacks to produce a taxonomy of vulnerabilities. The authors collected the attacks by running a global prompt hacking competition. * Docker, Confluence, Redis, and Apache Yarn are being targeted by malware in a new set of attacks. The malware is written in Go, though it is clumsily disguised to look like shell scripts. * Even more prompt injection attacks: Microsoft Copilot (distinct from Github Copilot) is vulnerable to conditional prompt injection attacks, where the hostile prompt is activated only for a specific user. * Yes, there’s now a prompt injection worm. A hostile prompt is embedded in an email, which then gets sent to the AI-based email assistant through RAG. Along with stealing data, the prompt can instruct the email assistant to generate new emails that spread the worm.

Things Another Copilot, this time not from Microsoft, is a Raspberry Pi-based AI system for bicyclists that alerts them to approaching cars and cars that are driving erratically or getting too close. It’s a good example of Pete Warden’s TinyML. * Want your own Klein Bottle? Made by Cliff Stoll, author of the cybersecurity classic The Cuckoo’s Egg*, who will autograph your bottle for you (and may include other surprises).

Quantum Computing* Google has published its threat model for quantum attacks against cryptography. The document is an excellent summary of the state of post-quantum cryptography.

Biology* Can fungus be engineered to produce artificial meat products? Fungus and its relatives have long been the basis of many food products, including cheese and beer. And funguses can produce the molecule that gives meat its flavor.

View Details

TL;DR* LLMs and other GenAI models can reproduce significant chunks of training data. * Specific prompts seem to “unlock” training data. * We have many current and future copyright challenges: training may not infringe copyright, but legal doesn’t mean legitimate—we consider the analogy of MegaFace where surveillance models have been trained on photos of minors, for example, without informed consent. * Copyright was intended to incentivize cultural production: in the era of generative AI, copyright won’t be enough.

In Borges’s fable “Pierre Menard, Author of The Quixote,” the eponymous Monsieur Menard plans to sit down and write a portion of Cervantes’s Don Quixote. Not to transcribe, but rewrite the epic novel word for word:

His goal was never the mechanical transcription of the original; he had no intention of copying it. His admirable ambition was to produce a number of pages which coincided—word for word and line by line—with those of Miguel de Cervantes.

He first tried to do so by becoming Cervantes, learning Spanish, and forgetting all the history since Cervantes wrote Don Quixote, among other things, but then decided it would make more sense to (re)write the text as Menard himself. The narrator tells us that “the Cervantes text and the Menard text are verbally identical, but the second is almost infinitely richer.” Perhaps this is an inversion of the ability of generative AI models (LLMs, text-to-image, and more) to reproduce swathes of their training data without those chunks being explicitly stored in the model and its weights: the output is verbally identical to the original but reproduced probabilistically without any of the human blood, sweat, tears, and life experience that goes into the creation of human writing and cultural production.

Generative AI Has a Plagiarism ProblemChatGPT, for example, doesn’t memorize its training data per se. As Mike Loukides and Tim O’Reilly astutely point out:

A model prompted to write like Shakespeare may start with the word “To,” which makes it slightly more probable that it will follow that with “be,” which makes it slightly more probable that the next word will be “or”—and so forth.

So then, as it turns out, next-word prediction (and all the sauce on top) can reproduce chunks of training data. This is the basis of the New York Times lawsuit against OpenAI. I have been able to convince ChatGPT to give me large chunks of novels that are in the public domain, such as those on Project Gutenberg, including Pride and Prejudice. Researchers are finding more and more ways to extract training data from ChatGPT and other models. As far as other types of foundation models go, recent work by Gary Marcus and Reid Southern has shown that you can use Midjourney (text-to-image) to generate images from Star Wars, The Simpsons, Super Mario Brothers, and many other films. This seems to be emerging as a feature, not a bug, and hopefully it’s obvious to you why they called their IEEE opinion piece “Generative AI Has a Visual Plagiarism Problem.” (It’s ironic that, in this article, we didn’t reproduce the images from Marcus’ article because we didn’t want to risk violating copyright—a risk that Midjourney apparently ignores and perhaps a risk that even IEEE and the authors took on!) And the space is moving quickly: Sora, OpenAI’s text-to-video model, is yet to be released and has already taken the world by storm.

Compression, Transformation, Hallucination, and GenerationTraining data isn’t stored in the model per se, but large chunks of it are reconstructable given the correct key (“prompt”).

There are lots of conversations about whether or not LLMs (and machine learning, more generally) are forms of compression or not. In many ways, they are, but they also have generative capabilities that we don’t often associate with compression.

Ted Chiang wrote a thoughtful piece for the New Yorker called “ChatGPT Is a Blurry JPEG of the Web” that opens with the analogy of a photocopier making a slight error due to the way it compresses the digital image. It’s an interesting piece that I commend to you, but one that makes me uncomfortable. To me, the analogy breaks down before it begins: firstly, LLMs don’t merely blur, but perform highly non-linear transformations, which means you can’t just squint and get a sense of the original; secondly, for the photocopier, the error is a bug, whereas, for LLMs, all errors are features. Let me explain. Or, rather, let Andrej Karpathy explain:

I always struggle a bit [when] I’m asked about the “hallucination problem” in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines.

We direct their dreams with prompts. The prompts start the dream, and based on the LLM’s hazy recollection of its training documents, most of the time the result goes someplace useful.

It’s only when the dreams go into deemed factually incorrect territory that we label it a “hallucination.” It looks like a bug, but it’s just the LLM doing what it always does.

At the other end of the extreme consider a search engine. It takes the prompt and just returns one of the most similar “training documents” it has in its database, verbatim. You could say that this search engine has a “creativity problem”—it will never respond with something new. An LLM is 100% dreaming and has the hallucination problem. A search engine is 0% dreaming and has the creativity problem.

As a side note, building products that strike balances between Search and LLMs will be a highly productive area and companies such as Perplexity AI are also doing interesting work there.

It’s interesting to me that, while LLMs are constantly “hallucinating,”1 they can also reproduce large chunks of training data, not just go “someplace useful,” as Karpathy put it (summarization, for example). So, is the training data “stored” in the model? Well, no, not quite. But also… Yes?

Let’s say I tear up a painting into a thousand pieces and put them back together in a mosaic: is the original painting stored in the mosaic? No, unless you know how to rearrange the pieces to get the original. You need a key. And, as it turns out, there happen to be certain prompts that act as keys that unlock training data (for insiders, you may recognize this as extraction attacks, a form of adversarial machine learning).

This also has implications for whether generative AI can create anything particularly novel: I have high hopes that it can, but I think that is still yet to be demonstrated. There are also significant and serious concerns about what happens when we continually train models on the outputs of other models.

Implications for Copyright and Legitimacy, Big Tech, and Informed ConsentCopyright isn’t the correct paradigm to be thinking about here; legal doesn’t mean legitimate; surveillance models trained on photos of your children.

Now I don’t think this has implications for whether LLMs are infringing copyright and whether ChatGPT is infringing that of the New York Times, Sarah Silverman, George R.R. Martin, or any of us whose writing has been scraped for training data. But I also don’t think copyright is necessarily the best paradigm for thinking through whether such training and deployment should be legal or not. Firstly, copyright was created in response to the affordances of mechanical reproduction, and we now live in an age of digital reproduction, distribution, and generation. It’s also about what type of society we want to live in collectively: copyright itself was originally created to incentivize certain modes of cultural production.

Early predecessors of modern copyright law, such as the Statute of Anne (1710) in England, were created to incentivize writers to write and to incentivize more cultural production. Up until this point, the Crown had granted exclusive rights to print certain works to the Stationers’ Company, effectively creating a monopoly, and there weren’t financial incentives to write. So, even if OpenAI and their frenemies aren’t breaching copyright law, what type of cultural production are we and aren’t we incentivizing by not zooming out and looking at as many of the externalities here as possible?

Remember the context. Actors and writers were recently striking while Netflix had an AI product manager job listing with a base salary ranging from $300K to $900K USD.2 Also, note that we already live in a society where many creatives end up in advertising and marketing. These may be some of the first jobs on the chopping block due to ChatGPT and friends, particularly if macroeconomic pressure keeps leaning on us all. And that’s according to OpenAI!

Back to copyright: I don’t know enough about copyright law but it seems to me as though LLMs are “transformative” enough to have a fair use defense in the US. Also, training models doesn’t seem to me to infringe copyright because it doesn’t yet produce output! But perhaps it should infringe something: even when the collection of data is legal (which, statistically, it won’t entirely be for any web-scale corpus), it doesn’t mean it’s legitimate, and it definitely doesn’t mean there was informed consent.

To see this, let’s consider another example, that of MegaFace. In “How Photos of Your Kids Are Powering Surveillance Technology,” the New York Times reported that

One day in 2005, a mother in Evanston, Ill., joined Flickr. She uploaded some pictures of her children, Chloe and Jasper. Then she more or less forgot her account existed…
Years later, their faces are in a database that’s used to test and train some of the most sophisticated [facial recognition] artificial intelligence systems in the world.

What’s more,

Containing the likenesses of nearly 700,000 individuals, it has been downloaded by dozens of companies to train a new generation of face-identification algorithms, used to track protesters, surveil terrorists, spot problem gamblers and spy on the public at large.

Even in the cases where this is legal (which seem to be the vast majority of cases), it’d be tough to make an argument that it’s legitimate and even tougher to claim that there was informed consent. I also presume most people would consider it ethically dubious. I raise this example for several reasons:

• Just because something is legal, doesn’t mean that we want it to be going forward.
• This is illustrative of an entirely new paradigm, enabled by technology, in which vast amounts of data can be collected, processed, and used to power algorithms, models, and products; the same paradigm under which GenAI models are operating.
• It’s a paradigm that’s baked into how a lot of Big Tech operates and we seem to accept it in many forms now: but if you’d built LLMs 10, let alone 20, years ago by scraping web-scale data, this would likely be a very different conversation.

I should probably also define what I mean by “legitimate/illegitimate” or at least point to a definition. When the Dutch East India Company “purchased” Manhattan from the Lenape people, Peter Minuit, who orchestrated the “purchase,” supposedly paid $24 worth of trinkets. That wasn’t illegal. Was it legitimate? It depends on your POV: not from mine. The Lenape didn’t have a conception of land ownership, just as we don’t yet have a serious conception of data ownership. This supposed “purchase” of Manhattan has resonances with uninformed consent. It’s also relevant as Big Tech is known for its extractive and colonialist practices.

This isn’t about copyright, the New York Times, or OpenAI It’s about what type of society you want to live in.

I think it’s entirely possible that the New York Times and OpenAI will settle out of court: OpenAI has strong incentives to do so and the Times likely also has short-term incentives to. However, the Times has also proven itself adept at playing the long game. Don’t fall into the trap of thinking this is merely about the specific case at hand. To zoom out again, we live in a society where mainstream journalism has been carved out and gutted by the internet, search, and social media. The New York Times is one of the last serious publications standing, and they’ve worked incredibly hard and cleverly in their “digital transformation” since the advent of the internet.3

Platforms such as Google have inserted themselves as middlemen between producers and consumers in a manner that has killed the business models of many of the content producers. They’re also disingenuous about what they’re doing: when the Australian Government was thinking of making Google pay news outlets that it linked to in Search, Google’s response was:

Now remember, we don’t show full news articles, we just show you where you can go and help you to get there. Paying for links breaks the way search engines work, and it undermines how the web works, too. Let me try and say it another way. Imagine your friend asks for a coffee shop recommendation. So you tell them about a few nearby so they can choose one and go get a coffee. But then you get a bill to pay all the coffee shops, simply because you mentioned a few. When you put a price on linking to certain information, you break the way search engines work, and you no longer have a free and open web. We’re not against a new law, but we need it to be a fair one. Google has an alternative solution that supports journalism. It’s called Google News Showcase.

Let me be clear: Google has done incredible work in “organizing the world’s information,” but here they’re disingenuous in comparing themselves to a friend offering advice on coffee shops: friends don’t tend to have global data, AI, and infrastructural pipelines, nor are they business-predicated on surveillance capitalism.

Copyright aside, the ability of generative AI to displace creatives is a real threat and I’m asking a real question: do we want to live in a society where there aren’t many incentives for humans to write, paint, and make music? Borges may not write today, given current incentives. If you don’t particularly care about Borges, perhaps you care about Philip K. Dick, Christopher Nolan, Salman Rushdie, or the Magic Realists, who were all influenced by his work.

Beyond all the human aspects of cultural production, don’t we also still want to dream? Or do we also want to outsource that and have LLMs do all the dreaming for us?

Footnotes1. I’m putting this in quotation marks as I’m still not entirely comfortable with the implications of anthropomorphizing LLMs in this manner. 2. My intention isn’t to suggest that Netflix is all bad. Far from it, in fact: Netflix has also been hugely powerful in providing a massive distribution channel to creatives across the globe. It’s complicated. 3. Also note that the outcome of this case could have significant impact for the future of OSS and open weight foundation models, something I hope to write about in future.

This essay first appeared on Hugo Bowne-Anderson’s blog. Thank you to Goku Mohandas for providing early feedback.

View Details

I’ve had a ham radio license since the late 1960s and observed the transition from vacuum tubes (remember them?) to transistors firsthand. Because we’re allowed to operate high-power transmitters (1,500-watt output), tubes hang on in our world a lot longer than elsewhere. There’s a good reason: tubes are ideal high-power devices for people who don’t always know what they’re doing, people who are just smart enough to be dangerous. About the only way you can damage them is by getting them hot enough to melt the internal components. That happens… but it means that there’s a huge margin for error.

3-1000Z, one of the last large glass bottle vacuum tubes. Capable of 1500W output.Transistors are the opposite. If a transistor exceeds its specifications for a millionth of a second, it will be destroyed. If tubes are like football players, transistors are like professional dancers: very strong, very powerful, but if they land wrong, there’s a serious sprain. As a result, there’s a big difference between high-power tube equipment and transistor equipment. To cool a vacuum tube, you put a fan next to it. To cool a transistor that’s generating 500 watts of heat from an area the size of a dime, you need a heavy copper spreader, a huge heat sink, and multiple fans. A tube amplifier is a box with a big power supply, a large vacuum tube, and an output circuit. A transistor amplifier has all of that, plus computers, sensors, and lots of other electronics to shut it down if anything looks like it’s going wrong. A lot of adjustments that you used to make by turning knobs have been automated. It’s easy to see the automation as a convenience, but in reality it’s a necessity. If these adjustments weren’t automated, you’d burn out the transistors before you get on the air.

Software has been making a similar transition. The early days of the web were simple: HTML, some minimal JavaScript, CSS, and CGI. Applications have obviously been getting more complex; backends with databases, middleware, and complex frontend frameworks have all become part of our world. Attacks against applications of all kinds have grown more common and more serious. Observability is the first step in a “transistor-like” approach to building software. It’s important to make sure that you can capture enough relevant data to predict problems before they become problems; only capturing enough data for a postmortem analysis isn’t sufficient.

Although we’re moving in the right direction, with AI the stakes are higher. This year, we’ll see AI incorporated into applications of all kinds. AI introduces many new problems that developers and IT staff will need to deal with. Here’s a start at a list:

• Security issues: Whether they do it maliciously or just for lols, people will want to make your AI act incorrectly. You can expect racist, misogynist, and just plain false output. And you will find that these are business issues.
• More security issues: Whether by “accident” or in response to a malicious prompt, we’ve seen that AI systems can leak users’ data to other parties.
• Even more security issues: Language models are frequently used to generate source code for computer programs. That code is frequently insecure. It’s even possible that attackers could force a model to generate insecure code on their command.
• Freshness: Models grow “stale” eventually and need to be retrained. There’s no evidence that large language models are an exception. Languages change slowly, but the topics about which you want your model to be conversant will not.
• Copyright: While these issues are only starting to work their way through the courts, developers of AI applications will almost certainly have some liability for copyright violation.
• Other liability: We’re only beginning to see legislation around privacy and transparency; Europe is the clear leader here. Whether or not the US ever has effective laws regulating the use of AI, companies need to comply with international law.

That’s only a start. My point isn’t to enumerate everything that can go wrong but that complexity is growing in ways that makes in-person monitoring impossible. This is something the financial industry learned a long time ago (and continues to learn). Algorithmic trading systems need to monitor themselves constantly and alert humans to intervene at the first sign something is wrong; they must have automatic “circuit breakers” to shut the application down if errors persist; and it must be possible to shut them down manually if these other methods fail. Without these safeguards, the result might look like Knight Capital, a company whose algorithmic trading software made $440M worth of mistakes on its first day.

The problem is that the AI industry hasn’t yet learned from the experience of others; it’s still moving fast and breaking things at the same time that it’s making the transition from relatively simple software (and yes, I consider a big React-based frontend with an enterprise backend “relatively simple” compared to LLM-based applications) to software that entangles many more processing nodes, software whose workings we don’t fully understand, and software that can cause damage at scale. And, like a modern high-power transistor amplifier, this software is too complex and fragile to be managed by hand. It’s still not clear that we know how to build the automation that we need to manage AI applications. Learning how to build those automation systems must become a priority for the next few years.

View Details

January was a dull month, at least in my opinion. Maybe everyone was recovering from their holidays. February was a short month, but it was far from dull. And I’m not even counting the first shipments of Apple Vision. OpenAI has demoed an impressive text-to-vision model called Sora; Google has two very impressive small language models and a model specialized for time series, and they opened Gemini to the public. Outside of AI, there’s a JVM for WebAssembly; you can use it to run applications like Minecraft in the browser. There are some new ultralight web frameworks. And one of the world’s biggest ransomware groups has been shut down.

On the purely weird front: there are a couple of new esoteric languages, one of which solves the problem of naming. We found out that Origami is Turing complete, so start folding. If you want a relief from AI that’s trying to be your pal, try antagonistic AI. And the best of the lot: edible robots.

AI Mistral has released Mistral Large, their flagship language model, with performance almost equal to GPT-4. It is available only via their API (although a chatbot is in beta). Unlike Mistral’s other models, Mistral Large is not open source. * This is different: Google’s DeepMind has announced Genie, a generative model for building interactive worlds. It’s a video model, but unlike other video models, it’s built for game playing. (Think Mario Brothers, not Star Wars*.) There are hints at other applications such as using Genie to develop virtual worlds for training other kinds of AI. * Now that large language models have been given the ability to execute other programs, they can be prompted to attack websites and other online systems. * ZLUDA, a library for running NVIDIA’s proprietary CUDA language on AMD GPUs, was released as an open source project after AMD stopped funding it. (An earlier version targeted Intel GPUs, but that version is no longer supported.) * Researchers in China are exploring whether neural networks can develop their own language for images without the intermediary of human language. * The competitive programming site Topcoder has issued a challenge: develop an AI bot that helps people fill out government forms. * Google has released two small language models, Gemma 2B and Gemma 7B. They claim performance superior to Llama 2 and Mistral. The models are “open,” though not open source. Google has released the weights and, in addition, a responsible generative AI toolkit. * Groq is a chatbot with roughly the performance of GPT-3.5 but has been tuned to give replies that are close to instantaneous. * Building an interactive restaurant menu with AI: whether or not it’s actually useful, this is a great tutorial about building a RAG application with open source AI. * Sora is an impressive new text-to-video model from OpenAI. It is not yet open to the public. OpenAI plans to include C2PA watermarking to identify generated video. They are currently engaged in adversarial testing to make the model less likely to generate biased or harmful content. * A research paper explores antagonistic AI: AI that is designed to be challenging, disagreeable, and confronting. Are there applications for AI that aren’t always earnestly trying to be your friend? * The US Patent and Trademark Office has ruled that only humans can patent inventions, not AI. This guidance is consistent with the Copyright Office’s approach. It doesn’t mean that AI output is not patentable but that there must be significant human input directing the AI. * Google has built a new foundation model for time series. Like language models, and unlike most time series models, TimesFM is pretrained using time series data. It excels at zero-shot predictions. * OpenAI is experimenting with long-term memory in ChatGPT (i.e., memory between conversations). Long-term memory raises a number of privacy issues, in addition to more practical questions like getting a fresh start on a conversation that’s gone wrong. * AI can be an accessory in the death of traditional languages, or a tool for preserving them. * There are many opportunities for using AI to improve accessibility. To use AI effectively, we need to acknowledge the harm that it can do and approach accessibility issues thoughtfully. * Artificial Intelligence cannot be used to deny healthcare. For now, at least. * Google has upgraded Bard to its latest Gemini model (Gemini Advanced). It’s worth trying; it’s on a par with GPT-4V. * Hugging Face has added four new leaderboards for measuring language models’ accuracy in answering questions relevant to businesses (finance, law, etc.), safety and security, freedom from hallucinations, and ability to solve reasoning problems. Unfortunately, the leaderboards only evaluate open source models. * Language models can be trained to be deceptive—specifically, to generate code that includes security vulnerabilities given certain prompts. This behavior can be made persistent and is hard to detect and hard to remove. * Meta has announced that it will label images that have been generated with AI. They discuss a number of methods for identifying AI-generated images, including watermarking, disclosure by the creator, fact-checking, and automated classification of unmarked images. * While AI’s ability to generate music is limited, AI does an extremely good job of mastering human recordings. * TinyLlama is yet another new language model. TinyLlama is small: 1B parameters, but more than that, only requires 550 MB of memory to run. It was designed for small mobile and embedded devices. * The Allen Institute has released OLMo, an open source language model. There are 7B and 1B parameter versions, and it claims performance better than similarly sized models. OLMo is the first completely open model: every step in development and every artifact generated is available. * We have seen surprisingly little discussion of techniques for mitigating AI risks. These ideas for protecting language models from prompt injection and other attacks are far from exhaustive, but they’re a start. * Jeremy Howard has a video on getting started with CUDA programming (NVIDIA GPU programming). It is aimed at Python programmers but no doubt useful for almost anyone. * Eagle 7B is another new large language model. It claims to out-perform all 7B-class models while requiring the least computation power for inference. It is available on HuggingFace. While Eagle appears to be transformer-based, it claims to point the way “beyond transformers.”

Programming Strada is a new IDE for building applications that use services from different SaaS (software as a service) providers. It makes it easier to work with multiple SaaS APIs simultaneously. * Something new for esoteric language fans: the namingless language. Naming is hard, so this language has only one data structure (so it doesn’t need a name) and only one operator (so it doesn’t need a name, either). * Google is supporting a Rust Foundation effort to improve interoperability between C++ and Rust with the goal of enabling organizations to improve the security of legacy C++ software by migrating to Rust. * Xonsh (however that may be pronounced) is a shell for Unix-like systems that combines Unix shell features with full support for Python. * Is it a coincidence? Two simple web frameworks for Java and Kotlin appear at almost the same time: Spark and Javalin. * Memray is a memory profiler for Python. It can track memory use in libraries written in C or C++, such as NumPy. It’s a great tool for discovering memory leaks, excessive memory allocation, and other problems. * Origami is Turing complete. Fold your way to solutions. Maybe we don’t need quantum computers after all. * sudo on Windows? The times are indeed changing. (Note that Windows sudo and Linux/WSL sudo are not the same.) * Here are some detailed guidelines for designing command line user interfaces for those of us who still believe that command lines are important. They’re the only way to deal effectively with data in bulk. * CheerpJ 3.0 is a Java Virtual Machine for WebAssembly. It is capable of running large Java applications (such as Minecraft*) in a browser without plugins. It currently supports Java 8, but the long-term plan is to support the current long-term version (presently Java 21). * Scriptisto is a clever tool that lets you write throwaway scripts in (almost) any commonly used compiled programming language. Add a simple shebang (# !/usr/bin/env/scriptso) line to any program, and it automates compilation and runs the program. * There’s yet another new language, but this one is different. Pkl is an object-oriented language for configuration, not for general-purpose programming. * Scalene is a new profiling tool for Python that accounts for the difference in performance between highly optimized libraries and regular Python code. It can also ask ChatGPT for performance suggestions. * GitLab is planning to use ActivityPub (the protocol behind Mastodon and the fediverse) to connect all their Git repositories into a single network. They will start with social features, but their goal is to enable one instance to open requests for a project hosted on another instance. * Docker Build Cloud is a service that speeds up the process of building Docker images. Claims of a 39x speedup are impressive, but even if Build Cloud doesn’t deliver quite that much, the decrease in build time is still significant. * A study of programming trends associates the use of coding assistants like GitHub Copilot with lower-quality code, increased code churn, more copy/paste code, and less refactoring.

Web* Is it possible to build software with a sense of place? Digital Terroir is a fascinating discussion about what a “sense of place” might mean for digital creations. * htmx is a lightweight JavaScript frontend library for HTML that allows web development without using JavaScript directly; rather than write JavaScript, developers add tags to standard HTML elements. Here is a good comparison of htmx and React. * htmz is a minimalist HTML framework that allows you to dynamically load resources within any portion of an HTML page. * The state of JavaScript bloat in 2024: it’s not pretty. * The Observable Framework is a new static site generator for data-driven interactive web applications. It goes a step beyond notebooks, giving developers all the flexibility of modern web applications. Observable is open source. * Microsoft’s Edge browser appears to import data from the Chrome browser (tabs, stored passwords, and more) without the user’s permission, and even if the importBrowsingData setting is explicitly turned off in the user’s profile. * Arc Max is a browser that incorporates AI for summarization, asking questions of web pages, and other features. Scott Hanselman questions whether this is a good approach.

Security* A new attack against SSH uses the SSH-Snake mapping tool to find private keys. After discovering private keys, it can easily move from one account (and machine) to another. * Law enforcement teams from several countries have arrested key members of the LockBit ransomware group, seized control of its infrastructure and data, and created a free decryption tool for victims. As of February 26, though, LockBit appears to be back. * The European Court of Human Rights has ruled that laws weakening end-to-end encryption or requiring back doors for law enforcement are illegal. * WiFi jamming tools have been used to disable security systems in a string of robberies. * A group of vulnerabilities has been discovered that allows an attacker to escape from a container, at which point they can then access the host operating system directly. * Basic security hygiene is important. An employee accidentally published Mercedes-Benz’s GitHub private key in a public GitHub repository, giving anyone unlimited access to Mercedes’ source archives. * Rowhammer is an attack against a system’s memory: repeated reads and writes cause the memory to change values. A new version of the Unix/Linux sudo command resists rowhammer attacks. It is interesting because it is a software mitigation, not a hardware fix.

Virtual Reality* C-Infinity is, essentially, a standing chair with built-in controllers that is designed to prevent VR-induced nausea. * Brilliant Labs is taking preorders for AI glasses. While there’s little description on the site, the glasses look like a heads-up augmented reality display that superimposes descriptive text on your field of view. They claim compatibility with prescription lenses. * Apple’s Vision Pro is now available. There are many product reviews, but Ben Thompson’s review is comprehensive. He identifies the big problem: apps. Not just VR apps, but AR apps, and developing that new generation of apps may require investments that few companies can afford.

Biology* Several years ago, a Kickstarter project to create a glow-in-the-dark rose failed. Now you can order a glowing petunia online, along with purple tomatoes. Has synthetic biology arrived? * Robots you can eat: Researchers are designing robots, including electronics and actuators, that are entirely edible: honey can possibly act as a semiconductor, gold leaf can be used as wire, and batteries can be made from food materials. There may be applications in medicine.

Energy* An abandoned Finnish copper mine will be repurposed as a giant gravity battery that can store excess energy from renewable sources. It isn’t clear how long the battery can run before “discharging” or what the total energy storage is.

View Details

In “SQL: The Universal Solvent for REST APIs” we saw how Steampipe’s suite of open source plug-ins that translate REST API calls directly into SQL tables. These plug-ins were, until recently, tightly bound to the open source engine and to the instance of Postgres that it launches and controls. That led members of the Steampipe community to ask: “Can we use the plug-ins in our own Postgres databases?” Now the answer is yes—and more—but let’s focus on Postgres first.

NOTE: Each Steampipe plugin ecosystem is now also a standalone foreign-data-wrapper extension for Postgres, a virtual-table extension for SQLite, and an export tool.

Using a Steampipe Plugin as a Standalone Postgres Foreign Data Wrapper (FDW)Visit Steampipe downloads to find the installer for your OS, and run it to acquire the Postgres FDW distribution of a plugin—in this case, the GitHub plugin. It’s one of (currently) 140 plug-ins available on the Steampipe hub. Each plugin provides a set of tables that map API calls to database tables—in the case of the GitHub plugin, 55 such tables. Each table can appear in a FROM or JOIN clause; here’s a query to select columns from the GitHub issue, filtering on a repository and author.

select state, updated_at, title, urlfrom github_issuewhere repository_full_name = 'turbot/steampipe' and author_login = 'judell'order by updated_at desc If you’re using Steampipe, you can install the GitHub plugin like this:

steampipe plugin install github then run the query in the Steampipe CLI or in any Postgres client that can connect to Steampipe’s instance of Postgres.

But if you want to do the same thing in your own instance of Postgres, you can install the plugin in a different way.

$ sudo /bin/sh -c "$( curl -fsSL https://steampipe.io/install/postgres.sh)"Enter the plugin name: githubEnter the version (latest): Discovered:- PostgreSQL version: 14- PostgreSQL location: /usr/lib/postgresql/14- Operating system: Linux- System architecture: x86_64Based on the above, steampipe_postgres_github.pg14.linux_amd64.tar.gzwill be downloaded, extracted and installed at: /usr/lib/postgresql/14Proceed with installing Steampipe PostgreSQL FDW for version 14 at /usr/lib/postgresql/14?- Press 'y' to continue with the current version.- Press 'n' to customize your PostgreSQL installation directory and select a different version. (Y/n): Downloading steampipe_postgres_github.pg14.linux_amd64.tar.gz...########################################################################################### 100.0%steampipe_postgres_github.pg14.linux_amd64/steampipe_postgres_github.pg14.linux_amd64/steampipe_postgres_github.sosteampipe_postgres_github.pg14.linux_amd64/steampipe_postgres_github.controlsteampipe_postgres_github.pg14.linux_amd64/steampipe_postgres_github--1.0.sqlsteampipe_postgres_github.pg14.linux_amd64/install.shsteampipe_postgres_github.pg14.linux_amd64/README.mdDownload and extraction completed.Installing steampipe_postgres_github in /usr/lib/postgresql/14...Successfully installed steampipe_postgres_github extension!Files have been copied to:- Library directory: /usr/lib/postgresql/14/lib- Extension directory: /usr/share/postgresql/14/extension/ Now connect to your server as usual, using psql or another client, most typically as the postgres user. Then run these commands, which are typical for any Postgres foreign data wrapper. As with all Postgres extensions, you start like this:

CREATE EXTENSION steampipe_postgres_fdw_github; To use a foreign data wrapper, you first create a server:

CREATE SERVER steampipe_github FOREIGN DATA WRAPPERsteampipe_postgres_github OPTIONS (config 'token="ghp_..."'); Use OPTIONS to configure the extension to use your GitHub access token. (Alternatively, the standard environment variables used to configure a Steampipe plugin—it’s just GITHUB_TOKEN in this case—will work if you set them before starting your instance of Postgres.)

The tables provided by the extension will live in a schema, so define one:

CREATE SCHEMA github; Now import the schema defined by the foreign server into the local schema you just created:

IMPORT FOREIGN SCHEMA github FROM SERVER steampipe_github INTO github; Now run a query!

The foreign tables provided by the extension live in the github schema, so by default you’ll refer to tables like github.github_my_repository. If you set search_path = 'github', though, the schema becomes optional and you can write queries using unqualified table names. Here’s a query we showed last time. It uses the GitHub_search_repository which encapsulates the GitHub API for searching repositories.

Suppose you’re looking for repos related to PySpark. Here’s a query to find repos whose names match “pyspark” and report a few metrics to help you gauge activity and popularity.

select name_with_owner, updated_at, -- how recently updated? stargazer_count -- how many people starred the repo?from github_search_repository where query = 'pyspark in:name' order by stargazer_count desclimit 10;+---------------------------------------+------------+---------------+|name_with_owner |updated_at |stargazer_count|+---------------------------------------+------------+---------------+| AlexIoannides/pyspark-example-project | 2024-02-09 | 1324 || mahmoudparsian/pyspark-tutorial | 2024-02-11 | 1077 || spark-examples/pyspark-examples | 2024-02-11 | 1007 || palantir/pyspark-style-guide | 2024-02-12 | 924 || pyspark-ai/pyspark-ai | 2024-02-12 | 791 || lyhue1991/eat_pyspark_in_10_days | 2024-02-01 | 719 || UrbanInstitute/pyspark-tutorials | 2024-01-21 | 400 || krishnaik06/Pyspark-With-Python | 2024-02-11 | 400 || ekampf/PySpark-Boilerplate | 2024-02-11 | 388 || commoncrawl/cc-pyspark | 2024-02-12 | 361 |+---------------------------------------+------------+---------------+ If you have a lot of repos, the first run of that query will take a few seconds. The second run will return results instantly, though, because the extension includes a powerful and sophisticated cache.

And that’s all there is to it! Every Steampipe plugin is now also a foreign data wrapper that works exactly like this one. You can load multiple extensions in order to join across APIs. Of course, you can join any of these API-sourced foreign tables with your own Postgres tables. And to save the results of any query, you can prepend “create table NAME as” or “create materialized view NAME as” to a query to persist results as a table or view.

Using a Steampipe Plugin as a SQLite Extension That Provides Virtual TablesVisit Steampipe downloads to find the installer for your OS and run it to acquire the SQLite distribution of the same plugin.

$ sudo /bin/sh -c "$(curl -fsSL https://steampipe.io/install/sqlite.sh)"Enter the plugin name: githubEnter version (latest): Enter location (current directory): Downloading steampipe_sqlite_github.linux_amd64.tar.gz...############################################################################ 100.0%steampipe_sqlite_github.sosteampipe_sqlite_github.linux_amd64.tar.gz downloaded and extracted successfully at /home/jon/steampipe-sqlite. Here’s the setup, and you can place this code in ~/.sqliterc if you want to run it every time you start sqlite.

.load /home/jon/steampipe-sqlite/steampipe_sqlite_github.soselect steampipe_configure_github(' token="ghp_..."'); Now you can run the same query as above. Here, too, the results are cached, so a second run of the query will be instant.

What about the differences between Postgres-flavored and SQLite-flavored SQL? The Steampipe hub is your friend! For example, here are Postgres and SQLite variants of a query that accesses a field inside a JSON column in order to tabulate the languages associated with your gists.

PostgresSQLiteThe github_my_gist table reports details about gists that belong to the GitHub user who is authenticated to Steampipe. The language associated with each gist lives in a JSONB column called files, which contains a list of objects like this.

{ "size": 24541, "type": "text/markdown", "raw_url": "https://gist.githubusercontent.com/judell/49d66ca2a5d2a3b "filename": "steampipe-readme-update.md", "language": "Markdown"} The functions needed to project that list as rows differ: in Postgres you use jsonb_array_elements and in SQLite it’s json_each.

As with Postgres extensions, you can load multiple SQLite extensions in order to join across APIs. You can join any of these API-sourced foreign tables with your own SQLite tables. And you can prepend create table NAME as to a query to persist results as a table.

Using a Steampipe Plugin as a Standalone Export ToolVisit Steampipe downloads to find the installer for your OS, and run it to acquire the export distribution of a plugin—again, we’ll illustrate using the GitHub plugin.

$ sudo /bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)"Enter the plugin name: githubEnter the version (latest): Enter location (/usr/local/bin): Created temporary directory at /tmp/tmp.48QsUo6CLF.Downloading steampipe_export_github.linux_amd64.tar.gz...############################################################################## 100.0%Deflating downloaded archivesteampipe_export_githubInstallingApplying necessary permissionsRemoving downloaded archivesteampipe_export_github was installed successfully to/usr/local/bin$ steampipe_export_github -hExport data using the github plugin.Find detailed usage information including table names, column names, and examples at the Steampipe Hub:https://hub.steampipe.io/plugins/turbot/githubUsage: steampipe_export_github TABLE_NAME [flags]Flags: --config string Config file data -h, --help help for steampipe_export_github --limit int Limit data --output string Output format: csv, json or jsonl (default "csv") --select strings Column data to display --where stringArray where clause data There’s no SQL engine in the picture here; this tool is purely an exporter. To export all your gists to a JSON file:

steampipe_export_github github_my_gist --output json > gists.json To select only some columns and export to a CSV file:

steampipe_export_github github_my_gist --output csv --select "description,created_at,html_url" > gists.csv You can use --limit to limit the rows returned and --where to filter them, but mostly you’ll use this tool to quickly and easily grab data that you’ll massage elsewhere, for example, in a spreadsheet.

Tap into the Steampipe Plugin EcosystemSteampipe plug-ins aren’t just raw interfaces to underlying APIs. They use tables to model those APIs in useful ways. For example, the github_my_repository table exemplifies a design pattern that applies consistently across the suite of plug-ins. From the GitHub plugin’s documentation:

You can own repositories individually, or you can share ownership of repositories with other people in an organization. The github_my_repository table will list repos that you own, that you collaborate on, or that belong to your organizations. To query ANY repository, including public repos, use the github_repository table.

Other plug-ins follow the same pattern. For example, the Microsoft 365 plugin provides both microsoft_my_mail_message and microsoft_mail_message, and the plugin provides googleworkspace_my_gmail_message and googleworkspace_gmail. Where possible, plug-ins consolidate views of resources from the perspective of an authenticated user.

While plug-ins typically provide tables with fixed schemas, that’s not always the case. Dynamic schemas, implemented by the Airtable, CSV, Kubernetes, and Salesforce plug-ins (among others) are another key pattern. Here’s a CSV example using a standalone Postgres FDW.

IMPORT FOREIGN SCHEMA csv FROM SERVER steampipe_csv INTO csv OPTIONS(config 'paths=["/home/jon/csv"]'); Now all the .csv files in /home/jon/csv will automagically be Postgres foreign tables. Suppose you keep track of valid owners of EC2 instances in a file called ec2_owner_tags. Here’s a query against the corresponding table.

select * from csv.ec2_owner_tags; owner | _ctx----------------+---------------------------- Pam Beesly | {"connection_name": "csv"} Dwight Schrute | {"connection_name": "csv"} You could join that table with the AWS plugin’s aws_ec2_instance table to report owner tags on EC2 instances that are (or are not) listed in the CSV file.

select ec2.owner, case when csv.owner is null then 'false' else 'true' end as is_listedfrom (select distinct tags ->> 'owner' as owner from aws.aws_ec2_instance) ec2left join csv.ec2_owner_tags csv on ec2.owner = csv.owner; owner | is_listed----------------+----------- Dwight Schrute | true Michael Scott | false Across the suite of plug-ins there are more than 2,300 predefined fixed-schema tables that you can use in these ways, plus an unlimited number of dynamic tables. And new plug-ins are constantly being added by Turbot and by Steampipe’s open source community. You can tap into this ecosystem using Steampipe or Turbot Pipes, from your own Postgres or SQLite database, or directly from the command line.

View Details

Since its release in November 2022, almost everyone involved with technology has experimented with ChatGPT: students, faculty, and professionals in almost every discipline. Almost every company has undertaken AI projects, including companies that, at least on the face of it, have “no AI” policies. Last August, OpenAI stated that 80% of Fortune 500 companies have ChatGPT accounts. Interest and usage have increased as OpenAI has released more capable versions of its language model: GPT-3.5 led to GPT-4 and multimodal GPT-4V, and OpenAI has announced an Enterprise service with better guarantees for security and privacy. Google’s Bard/Gemini, Anthropic’s Claude, and other models have made similar improvements. AI is everywhere, and even if the initial frenzy around ChatGPT has died down, the big picture hardly changes. If it’s not ChatGPT, it will be something else, possibly something users aren’t even aware of: AI tools embedded in documents, spreadsheets, slide decks, and other tools in which AI fades into the background. AI will become part of almost every job, ranging from manual labor to management.

With that in mind, we need to ask what companies must do to use AI responsibly. Ethical obligations and responsibilities don’t change, and we shouldn’t expect them to. The problem that AI introduces is the scale at which automated systems can cause harm. AI magnifies issues that are easily rectified when they affect a single person. For example, every company makes poor hiring decisions from time to time, but with AI all your hiring decisions can quickly become questionable, as Amazon discovered. The New York Times’ lawsuit against OpenAI isn’t about a single article; if it were, it would hardly be worth the legal fees. It’s about scale, the potential for reproducing its whole archive. O’Reilly has built an AI application that uses our authors’ content to answer questions, but we compensate our authors fairly for that use: we won’t ignore our obligations to our authors, either individually or at scale.

It’s essential for companies to come to grips with the scale at which AI works and the effects it creates. What are a corporation’s responsibilities in the age of AI—to its employees, its customers, and its shareholders? The answers to this question will define the next generation of our economy. Introducing new technology like AI doesn’t change a company’s basic responsibilities. However, companies must be careful to continue living up to their responsibilities. Workers fear losing their jobs “to AI” but also look forward to tools that can eliminate boring, repetitive tasks. Customers fear even worse interactions with customer service but look forward to new kinds of products. Stockholders anticipate higher profit margins but fear seeing their investments evaporate if companies can’t adopt AI quickly enough. Does everybody win? How do you balance the hopes against the fears? Many people believe that a corporation’s sole responsibility is to maximize short-term shareholder value with little or no concern for the long term. In that scenario, everybody loses—including stockholders who don’t realize they’re participating in a scam.

How would corporations behave if their goal were to make life better for all of their stakeholders? That question is inherently about scale. Historically, the stakeholders in any company are the stockholders. We need to go beyond that: the employees are also stakeholders, as are the customers, as are the business partners, as are the neighbors and, in the broadest sense, anyone participating in the economy. We need a balanced approach to the entire ecosystem.

O’Reilly tries to operate in a balanced ecosystem with equal weight going toward customers, shareholders, and employees. We’ve made a conscious decision not to manage our company for the good of one group while disregarding the needs of everyone else. From that perspective, we want to dive into how we believe companies need to think about AI adoption and how their implementation of AI needs to work for the benefit of all three constituencies.

Being a Responsible EmployerWhile the number of jobs lost to AI so far has been small, it’s not zero. Several copywriters have reported being replaced by ChatGPT; one of them eventually had to “accept a position training AI to do her old job.” However, a few copywriters don’t make a trend. So far, the total numbers appear to be small. One report claims that in May 2023, over 80,000 workers were laid off, but only about 4,000 of these layoffs were caused by AI, or 5%. That’s a very partial picture of an economy that added 390,000 jobs during the same period. But before dismissing the fear-mongering, we should wonder whether this is the shape of things to come. 4,000 layoffs could become a much larger number very quickly.

Fear of losing jobs to AI is probably lower in the technology sector than in other business sectors. Programmers have always made tools to make their jobs easier, and GitHub Copilot, the GPT family of models, Google’s Gemini, and other language models are tools that they’re already taking advantage of. For the immediate future, productivity improvements are likely to be relatively small: 20% at most. However, that doesn’t negate the fear; and there may well be more fear in other sectors of the economy. Truckers and taxi drivers wonder about autonomous vehicles; writers (including novelists and screenwriters, in addition to marketing copywriters) worry about text generation; customer service personnel worry about chatbots; teachers worry about automated tutors; and managers worry about tools for creating strategies, automating reviews, and much more.

An easy reply to all this fear is “AI is not going to replace humans, but humans with AI are going to replace humans without AI.” We agree with that statement, as far as it goes. But it doesn’t go very far. This attitude blames the victim: if you lose your job, it’s your own fault for not learning how to use AI. That’s a gross oversimplification. Second, while most technological changes have created more jobs than they destroyed, that doesn’t mean that there isn’t a time of dislocation, a time when the old professions are dying out but the new ones haven’t yet come into being. We believe that AI will create more jobs than it destroys—but what about that transition period? The World Economic Forum has published a short report that lists the 10 jobs most likely to see a decline, and the 10 most likely to see gains. Suffice it to say that if your job title includes the word “clerk,” things might not look good—but your prospects are looking up if your job title includes the word “engineer” or “analyst.”

The best way for a company to honor its commitment to its employees and to prepare for the future is through education. Most jobs won’t disappear, but all jobs will change. Providing appropriate training to get employees through that change may be a company’s biggest responsibility. Learning how to use AI effectively isn’t as trivial as a few minutes of playing with ChatGPT makes it appear. Developing good prompts is serious work and it requires training. That’s certainly true for technical employees who will be developing applications that use AI systems through an API. It’s also true for nontechnical employees who may be trying to find insights from data in a spreadsheet, summarize a group of documents, or write text for a company report. AI needs to be told exactly what to do and, often, how to do it.

One aspect of this change will be verifying that the output of an AI system is correct. Everyone knows that language models make mistakes, often called “hallucinations.” While these inaccuracies may not be as dramatic as making up case law, AI will make mistakes—errors at the scale of AI—and users will need to know how to check its output without being deceived (or in some cases, bullied) by its overconfident voice. The frequency of errors may go down as AI technology improves, but errors won’t disappear in the foreseeable future. And even with error rates as low as 1%, we’re easily talking about thousands of errors sprinkled randomly through software, press releases, hiring decisions, catalog entries—everything AI touches. In many cases, verifying that an AI has done its work correctly may be as difficult as it would be for a human to do the work in the first place. This process is often called “critical thinking,” but it goes a lot deeper: it requires scrutinizing every fact and every logical inference, even the most self-evident and obvious. There is a methodology that needs to be taught, and it is the employers’ responsibility to ensure that their employees have appropriate training to detect and correct errors.

The responsibility for education isn’t limited to training employees to use AI within their current positions. Companies need to provide education for transitions from jobs that are disappearing to jobs that are growing. Responsible use of AI includes auditing to ensure that its outputs aren’t biased, and that they are appropriate. Customer service personnel can be retrained to test and verify that AI systems are working correctly. Accountants can become auditors responsible for overseeing IT security. That transition is already happening; auditing for the SOC 2 corporate security certification is handled by accountants. Businesses need to invest in training to support transitions like these.

Looking at an even broader context: what are a corporation’s responsibilities to local public education? No company is going to prosper if it can’t hire the people it needs. And while a company can always hire employees who aren’t local, that assumes that educational systems across the country are well-funded, but they frequently aren’t.

This looks like a “tragedy of the commons”: no single nongovernmental organization is responsible for the state of public education, and public education is expensive (it’s usually the biggest line item on any municipal budget), so nobody takes care of it. But that narrative repeats a fundamental misunderstanding of the “commons.” The “tragedy of the commons” narrative was never correct; it is a fiction that achieved prominence as an argument to justify eugenics and other racist policies. Historically, common lands were well managed by law, custom, and voluntary associations. The commons declined when landed gentry and other large landholders abused their rights to the detriment of the small farmers; the commons as such disappeared through enclosure, when the large landholders fenced in and claimed common land as private property. In the context of the 20th and 21st centuries, the landed gentry—now frequently multinational corporations—protect their stock prices by negotiating tax exemptions and abandoning their responsibilities toward their neighbors and their employees.

The economy itself is the biggest commons of all, and nostrums like “the invisible hand of the marketplace” do little to help us understand responsibilities. This is where the modern version of “enclosure” takes place: in minimizing labor cost to maximize short-term value and executive salaries. In a winner-take-all economy where a company’s highest-paid employees can earn over 1,000 times as much as the lowest paid, the absence of a commitment to employees leads to poor housing, poor school systems, poor infrastructure, and marginalized local businesses. Quoting a line from Adam Smith that hasn’t entered our set of economic cliches, senior management salaries shouldn’t facilitate “gratification of their own vain and insatiable desires.”

One part of a company’s responsibilities to its employees is paying a fair wage. The consequences of not paying a fair wage, or of taking every opportunity to minimize staff, are far-reaching; they aren’t limited to the people who are directly affected. When employees aren’t paid well or live in fear of layoffs, they can’t participate in the local economy. There’s a reason that low-income areas often don’t have basic services like banks or supermarkets. When people are just subsisting, they can’t afford the services they need to flourish; they live on junk food because they can’t afford a $40 Uber to the supermarket in a more affluent town (to say nothing of the time). And there’s a reason why it’s difficult for lower-income people to make the transition to the middle class. In very real terms, living is more expensive if you’re poor: long commutes with less reliable transportation, poor access to healthcare, more expensive food, and even higher rents (slum apartments aren’t cheap) make it very difficult to escape poverty. An automobile repair or a doctor’s bill can exhaust the savings of someone who is near the poverty line.

That’s a local problem, but it can compound into a national or worldwide problem. That happens when layoffs become widespread—as happened in the winter and spring of 2023. Although there was little evidence of economic stress, fear of a recession led to widespread layoffs (often sparked by “activist investors” seeking only to maximize short-term stock price), which nearly caused a real recession. The primary driver for this “media recession” was a vicious cycle of layoff news, which encouraged fear, which led to more layoffs. When you see weekly announcements of layoffs in the tens of thousands, it’s easy to follow the trend. And that trend will eventually lead to a downward spiral: people who are unemployed don’t go to restaurants, defer maintenance on cars and houses, spend less on clothing, and economize in many other ways. Eventually, this reduction in economic activity trickles down and causes merchants and other businesses to close or reduce staff.

There are times when layoffs are necessary; O’Reilly has suffered through those. We’re still here as a result. Changes in markets, corporate structure, corporate priorities, skills required, and even strategic errors such as overhiring can all make layoffs necessary. These are all valid reasons for layoffs. A layoff should never be an “All of our peers are laying people off, let’s join the party” event; that happened all too often in the technology sector last year. Nor should it be an “our stock price could be higher and the board is cranky” event. A related responsibility is honesty about the company’s economic condition. Few employees will be surprised to hear that their company isn’t meeting its financial goals. But honesty about what everyone already knows might keep key people from leaving when you can least afford it. Employees who haven’t been treated with respect and honesty can’t be expected to show loyalty when there’s a crisis.

Employers are also responsible for healthcare, at least in the US. This is hardly ideal, but it’s not likely to change in the near future. Without insurance, a hospitalization can be a financial disaster, even for a highly compensated employee. So can a cancer diagnosis or any number of chronic diseases. Sick time is another aspect of healthcare—not just for those who are sick, but for those who work in an office. The COVID pandemic is “over” (for a very limited sense of “over”) and many companies are asking their staff to return to offices. But we all know people who at workplaces where COVID, the flu, or another disease has spread like wildfire because one person didn’t feel well and reported to the office anyway. Companies need to respect their employees’ health by providing health insurance and allowing sick time—both for the employees’ sakes and for everyone they come in contact with at work.

We’ve gone far afield from AI but for good reasons. A new technology can reveal gaps in corporate responsibility and help us think about what those responsibilities should be. Compartmentalizing is unhealthy; it’s not helpful to talk about a company’s responsibilities to highly paid engineers developing AI systems without connecting that to responsibilities toward the lowest-paid support staff. If programmers are concerned about being replaced by a generative algorithm, the groundskeepers should certainly worry about being replaced by autonomous lawnmowers.

Given this context, what are a company’s responsibilities to all of its employees?

• Providing training for employees so they remain relevant even as their jobs change
• Providing insurance and sick leave so that employees’ livelihoods aren’t threatened by health problems
• Paying a livable wage that allows employees and the communities they live in to prosper
• Being honest about the company’s finances when layoffs or restructuring are likely
• Balancing the company’s responsibilities to employees, customers, investors, and other constituencies

Responsibilities to Business PartnersGenerative AI has spawned a swirl of controversy around copyright and intellectual property. Does a company have any obligation toward the creators of content that they use to train their systems? These content creators are business partners, whether or not they have any say in the matter. A company’s legal obligations are currently unclear and will ultimately be decided in the courts or by legislation. But treating its business partners fairly and responsibly isn’t just a legal matter.

We believe that our talent—authors and teachers—should be paid. As a company that is using AI to generate and deliver content, we are committed to allocating income to authors as their work is used in that content and paying them appropriately—as we do with all other media. Granted, our use case makes the problem relatively simple. Our systems recommend content, and authors receive income when the content is used. They can answer users’ questions by extracting text from content to which we’ve acquired the rights; when we use AI to generate an answer, we know where that text has come from and can compensate the original author accordingly. These answers also link to the original source, where users can find more information, again generating income for the author. We don’t treat our authors and teachers as an undifferentiated class whose work we can repurpose at scale and without compensation. They aren’t abstractions who can be dissociated from the products of their labor.

We encourage our authors and teachers to use AI responsibly and to work with us as we build new kinds of products to serve future generations of learners. We believe that using AI to create new products, while always keeping our responsibilities in mind, will generate more income for our talent pool—and that sticking to “business as usual,” the products that have worked in the past, isn’t to anyone’s advantage. Innovation in any technology, including training, entails risk. The alternative to risk-taking is stagnation. But the risks we take always account for our responsibilities to our partners: to compensate them fairly for their work and to build a learning platform on which they can prosper. In a future article, we will discuss our AI policies for our authors and our employees in more detail.

The applications we are building are fairly clear-cut, and that clarity makes it relatively easy to establish rules for allocating income to authors. It’s less clear what a company’s responsibilities are when an AI isn’t simply extracting text but predicting the most likely next token one at a time. It’s important not to sidestep those issues either. It’s certainly conceivable that an AI could generate an introduction to a new programming language, borrowing some of the text from older content and generating new examples and discussions as necessary. Many programmers have already found ChatGPT a useful tool when learning a new language. Tutorials could even be generated dynamically, at a user’s request. When an AI model is generating text by predicting the next token in the sequence, one token at a time, how do you attribute authorship?

While it’s not yet clear how this will work out in practice, the principle is the same: generative AI doesn’t create new content; it extracts value from existing content, and the creators of that original content deserve compensation. It’s possible that these situations could be managed by careful prompting: for example, a system prompt or a RAG application that controls what sources are used to generate the answer would make attribution easier. Ignoring the issue and letting an AI generate text with no accountability isn’t a responsible solution. In this case, acting responsibly is about what you build as much as it is about who you pay; an ethical company builds systems that allow it to act responsibly. The current generation of models are, essentially, experiments that got out of control. It isn’t surprising that they don’t have all the features they need. But any models and applications built in the future will lack that excuse.

Many other kinds of business partners will be affected by the use of AI: suppliers, wholesalers, retailers, contractors of many types. Some of these effects will result from their own use of AI; some won’t. But the principles of fairness and compensation where compensation is due remain the same. A company should not use AI to justify shortchanging its business partners.

A company’s responsibilities to its business partners thus include:

• Compensating business partners for all use of their content, including AI-repurposed content
• Building applications that use AI to serve future generations of users
• Encouraging partners to use AI responsibly in the products they develop

Responsibilities to CustomersWe all think we know what customers want: better products at lower prices, sometimes at prices that are below what’s reasonable. But that doesn’t take customers seriously. The first of O’Reilly’s operating principles is about customers—as are the next four. If a company wants to take its customers seriously, particularly in the context of AI-based products, what responsibilities should it be thinking about?

Every customer must be treated with respect. Treating customers with respect starts with sales and customer service, two areas where AI is increasingly important. It’s important to build AI systems that aren’t abusive, even in subtle ways—even though human agents can also be abusive. But the responsibility extends much farther. Is a recommendation engine recommending appropriate products? We’ve certainly heard of Black women who only get recommendations for hair care products that white women use. We’ve also heard of Black men who see advertisements for bail bondsmen whenever they make any kind of a search. Is an AI system biased with respect to race, gender, or almost anything else? We don’t want real estate systems that reimplement redlining where minorities are only shown properties in certain areas. Will a résumé-screening system treat women and racial minorities fairly? Concern for bias goes even farther: it is possible for AI systems to develop bias against almost anything, including factors that it wouldn’t occur to humans to think about. Would we even know if an AI developed a bias against left-handed people?

We’ve known for a long time that machine learning systems can’t be perfect. The tendency of the latest AI systems to hallucinate has only rubbed our faces in that fact. Although techniques like RAG can minimize errors, it is probably impossible to prevent them altogether, at least with the current generation of language models. What does that mean for our customers? They aren’t paying us for incorrect information at scale; at the same time, if they want AI-enhanced services, we can’t guarantee that all of AI’s results will be correct. Our responsibilities to customers for AI-driven products are threefold. We need to be honest that errors will occur; we need to use techniques that minimize the probability of errors; and we need to present (or be prepared to present) alternatives so they can use their judgment about which answers are appropriate to their situation.

Respect for a customer includes respecting their privacy, an area in which online businesses are notably deficient. Any transaction involves a lot of data, ranging from data that’s essential to the transaction (what was bought, what was the price) to data that seems inconsequential but can still be collected and sold: browsing data obtained through cookies and tracking pixels is very valuable, and even arcana like keystroke timings can be collected and used to identify customers. Do you have the customer’s permission to sell the data that their transactions throw off? At least in the US, the laws on what you can do with data are porous and vary from state to state; because of GDPR, the situation in Europe is much clearer. But ethical and legal aren’t the same; “legal” is a minimum standard that many companies fail to meet. “Ethical” is about your own standards and principles for treating others responsibly and equitably. It is better to establish good principles that deal with your customers honestly and fairly than to wait for legislation to tell you what to do or to think that fines are just another expense of doing business. Does a company use data in ways that respect the customer? Would a customer be horrified to find out, after the fact, where their data has been sold? Would a customer be equally horrified to find that their conversations with AI have been leaked to other users?

Every customer wants quality, but quality doesn’t mean the same thing to everyone. A customer on the edge of poverty might want durability rather than expensive fine fabrics—though the same customer might, on a different purchase, object to being pushed away from the more fashionable products they want. How does a company respect the customer’s wishes in a way that isn’t condescending and delivers a product that’s useful? Respecting the customer means focusing on what matters to them; and that’s true whether the agent working with the customer is a human or an AI. The kind of sensitivity required is difficult for humans and may be impossible for machines, but it’s no less essential. Achieving the right balance probably requires a careful collaboration between humans and AI.

A business is also responsible for making decisions that are explainable. That issue doesn’t arise with human systems; if you are denied a loan, the bank can usually tell you why. (Whether the answer is honest may be another issue.) This isn’t true of AI, where explainability is still an active area for research. Some models are inherently explainable—for example, simple decision trees. There are explainability algorithms such as LIME that aren’t dependent on the underlying algorithm. Explainability for transformer-based AI (which includes just about all generative AI algorithms) is next to impossible. If explainability is a requirement—which is the case for almost anything involving money—it may be best to stay away from systems like ChatGPT. These systems make more sense in applications where explainability and correctness aren’t issues. Regardless of explainability, companies should audit the outputs of AI systems to ensure that they’re fair and unbiased.

The ability to explain a decision means little if it isn’t coupled with the ability to correct decisions. Respecting the customer means having a plan for redress. “The computer did it” was never a good excuse, and it’s even less acceptable now, especially since it’s widely known that AI systems of all types (not just natural language systems) generate errors. If an AI system improperly denies a loan, is it possible for a human to approve the loan anyway? Humans and AI need to learn how to work together—and AI should never be an excuse.

Given this context, what are a company’s responsibilities to its customers? These responsibilities can be summed up with one word: respect. But respect is a very broad term; it includes:

• Treating customers the way they would want to be treated
• Respecting customers’ privacy
• Understanding what the customer wants
• Explaining decisions as needed
• Providing a means to correct errors
• Respecting customer privacy

Responsibilities to ShareholdersIt’s long been a cliché that a company’s primary responsibility is to maximize shareholder value. That’s a good pretext for arguing that a company has the right—no, the duty—to abuse employees, customers, and other stakeholders—particularly if the shareholder’s “value” is limited to the short-term. The idea that shareholder value is enshrined in law (either legislation or case law) is apocryphal. It appeared in the 1960s and 1970s and was propagated by Milton Friedman and the Chicago school of economics.

Companies certainly have obligations to their shareholders, one of which is that shareholders deserve a return on their investment. But we need to ask whether this means short-term or long-term return. Finance in the US has fixated on short-term return, but that obsession is harmful to all of the stakeholders—except for executives who are often compensated in stock. When short-term returns cause a company to compromise the quality of its products, customers suffer. When short-term returns cause a company to layoff staff, the staff suffers, including those who stay: they are likely to be overworked and to fear further layoffs. Employees who fear losing their jobs, or are currently looking for new jobs, are likely to do a poor job of serving customers. Layoffs for strictly short-term financial gain are a vicious cycle for the company too: they lead to missed schedules, missed goals, and further layoffs. All of these result in a loss of credibility and poor long-term value. Indeed, one possible reason for Boeing’s problems with the 737 Max and the 787 has been a shift from an engineering-dominated culture that focused on building the best product to a financial culture that focused on maximizing short-term profitability. If that theory is correct, the results of the cultural change are all too obvious and present a significant threat to the company’s future.

What would a company that is truly responsible to its stakeholders look like, and how can AI be used to achieve that goal? We don’t have the right metrics; stock price, either short- or long-term, isn’t right. But we can think about what a corporation’s goals really are. O’Reilly’s operating principles start with the question “Is it best for the customer?” and continue with “Start with the customer’s point of view. It’s about them, not us.” Customer focus is a part of a company’s culture, and it’s antithetical to short-term returns. That doesn’t mean that customer focus sacrifices returns but that maximizing stock price leads to ways of thinking that aren’t in the customers’ interests. Closing a deal whether or not the product is right takes priority over doing right by the customer. We’ve all seen that happen; at one time or another, we’ve all been victims of it.

There are many opportunities for AI to play a role in serving customers’ interests—and, in turn, serving shareholders’ interests. First, what does a customer want? Henry Ford probably didn’t say that customers want faster horses, but that remains an interesting observation. It’s certainly true that customers often don’t know what they really want or, if they do, can’t articulate it. Steve Jobs may have said that “our job is to figure out what they want before they do”; according to some stories, he lurked in the bushes outside Apple’s Palo Alto store to watch customers’ reactions. Jobs’ secret weapon was intuition and imagination about what might be possible. Could AI help humans to discover what traditional custom research, such as focus groups (which Jobs hated), is bound to miss? Could an AI system with access to customer data (possibly including videos of customers trying out prototypes) help humans develop the same kind of intuition that Steve Jobs had? That kind of engagement between humans and AI goes beyond AI’s current capabilities, but it’s what we’re looking for. If a key to serving the customers’ interests is listening—really listening, not just recording—can AI be an aid without also become creepy and intrusive? Products that actually serve customers’ needs create long term value for all of the stakeholders.

This is only one way in which AI can serve to drive long-term success and to help a business deliver on its responsibilities to stockholders and other stakeholders. The key, again, is collaboration between humans and AI, not using AI as a pretext for minimizing headcount or shortchanging product quality.

It should go without saying, but in today’s business climate it doesn’t: one of a company’s responsibilities is to remain in business. Self-preservation at all costs is abusive, but a company that doesn’t survive isn’t doing its investors’ portfolios any favors. The US Chamber of Commerce, giving advice to small businesses, asks, “Have you created a dynamic environment that can quickly and effectively respond to market changes? If the answer is ‘no’ or ‘kind of,’ it’s time to get to work.” Right now, that advice means engaging with AI and deciding how to use it effectively and ethically. AI changes the market itself; but more than that, it is a tool for spotting changes early and thinking about strategies to respond to change. Again, it’s an area where success will require collaboration between humans and machines.

Given this context, a company’s responsibility to its shareholders include:

• Focusing on long-term rather than short-term returns
• Building an organization that can respond to changes
• Developing products that serve customers’ real needs
• Enabling effective collaboration between humans and AI systems

It’s About Honesty and RespectA company has many stakeholders—not just the stockholders and certainly not just the executives. These stakeholders form a complex ecosystem. Corporate ethics is about treating all of these stakeholders, including employees and customers, responsibly, honestly, and with respect. It’s about balancing the needs of each group so that all can prosper, about taking a long-term view that realizes that a company can’t survive if it is only focused on short-term returns for stockholders. That has been a trap for many of the 20th century’s greatest companies, and it’s unfortunate that we see many technology companies traveling the same path. A company that builds products that aren’t fit for the market isn’t going to survive; a company that doesn’t respect its workforce will have trouble retaining good talent; and a company that doesn’t respect its business partners (in our case, authors, trainers, and partner publishers on our platform) will soon find itself without partners.

Our corporate values demand that we do something better, that we keep the needs of all these constituencies in mind and in balance as we move our business forward. These values have nothing to do with AI, but that’s not surprising. AI creates ethical challenges, especially around the scale at which it can cause trouble when it is used inappropriately. However, it would be surprising if AI actually changed what we mean by honesty or respect. It would be surprising if the idea of behaving responsibly changed suddenly because AI became part of the equation.

Acting responsibly toward your employees, customers, business partners, and stockholders: that’s the core of corporate ethics, with or without AI.

View Details

Since the New York Times sued OpenAI for infringing its copyrights by using Times content for training, everyone involved with AI has been wondering about the consequences. How will this lawsuit play out? And, more importantly, how will the outcome affect the way we train and use large language models?

There are two components to this suit. First, it was possible to get ChatGPT to reproduce some Times articles very close to verbatim. That’s fairly clearly copyright infringement, though there are still important questions that could influence the outcome of the case. Reproducing the New York Times clearly isn’t the intent of ChatGPT, and OpenAI appears to have modified ChatGPT’s guardrails to make generating infringing content more difficult, though probably not impossible. Is this enough to limit any damages? It’s not clear that anybody has used ChatGPT to avoid paying for a NYT subscription. Second, the examples in a case like this are always cherry-picked. While the Times can clearly show that OpenAI can reproduce some articles, can it reproduce any article from the Times’ archive? Could I get ChatGPT to produce an article from page 37 of the September 18, 1947 issue? Or, for that matter, an article from the Chicago Tribune or the Boston Globe? Is the entire corpus available (I doubt it), or just certain random articles? I don’t know, and given that OpenAI has modified GPT to reduce the possibility of infringement, it’s almost certainly too late to do that experiment. The courts will have to decide whether inadvertent, inconsequential, or unpredictable reproduction meets the legal definition of copyright infringement.

The more important claim is that training a model on copyrighted content is infringement, whether or not the model is capable of reproducing that training data in its output. An inept and clumsy version of this claim was made by Sarah Silverman and others in a suit that was dismissed. The Authors’ Guild has its own version of this lawsuit, and it is working on a licensing model that would allow its members to opt in to a single licensing agreement. The outcome of this case could have many side-effects, since it essentially would allow publishers to charge not just for the texts they produce, but for how those texts are used.

It is difficult to predict what the outcome will be, though easy enough guess. Here’s mine. OpenAI will settle with the New York Times out of court, and we won’t get a ruling. This settlement will have important consequences: it will set a de-facto price on training data. And that price will no doubt be high. Perhaps not as high as the Times would like (there are rumors that OpenAI has offered something in the range of $1 million to $5 million), but sufficiently high enough to deter OpenAI’s competitors.

$1M is not, in and of itself, a terribly high price, and the Times reportedly thinks that it’s way too low; but realize that OpenAI will have to pay a similar amount to almost every major newspaper publisher worldwide in addition to organizations like the Authors Guild, technical journal publishers, magazine publishers, and many other content owners. The total bill is likely to be close to $1 billion, if not more, and as models need to be updated, at least some of it will be a recurring cost. I suspect that OpenAI would have difficulty going higher, even given Microsoft’s investments—and, whatever else you may think of this strategy—OpenAI has to think about the total cost. I doubt that they are close to profitable; they appear to be running on an Uber-like business plan, in which they spend heavily to buy the market without regard for running a sustainable business. But even with that business model, billion-dollar expenses have to raise the eyebrows of partners like Microsoft.

The Times, on the other hand, appears to be making a common mistake: overvaluing its data. Yes, it has a large archive—but what is the value of old news? Furthermore, in almost any application but especially in AI, the value of data isn’t the data itself; it’s the correlations between different datasets. The Times doesn’t own those correlations any more than I own the correlations between my browsing data and Tim O’Reilly’s. But those correlations are precisely what’s valuable to OpenAI and others building data-driven products.

Having set the price of copyrighted training data to $1B or thereabouts, other model developers will need to pay similar amounts to license their training data: Google, Microsoft (for whatever independently developed models they have), Facebook, Amazon, and Apple. Those companies can afford it. Smaller startups (including companies like Anthropic and Cohere) will be priced out, along with every open source effort. By settling, OpenAI will eliminate much of their competition. And the good news for OpenAI is that even if they don’t settle, they still might lose the case. They’d probably end up paying more, but the effect on their competition would be the same. Not only that, the Times and other publishers would be responsible for enforcing this “agreement.” They’d be responsible for negotiating with other groups that want to use their content and suing those they can’t agree with. OpenAI keeps its hands clean, and its legal budget unspent. They can win by losing—and if so, do they have any real incentive to win?

Unfortunately, OpenAI is right in claiming that a good model can’t be trained without copyrighted data (although Sam Altman, OpenAI’s CEO, has also said the opposite). Yes, we have substantial libraries of public domain literature, plus Wikipedia, plus papers in ArXiv, but if a language model trained on that data would produce text that sounds like a cross between 19th century novels and scientific papers, that’s not a pleasant thought. The problem isn’t just text generation; will a language model whose training data has been limited to copyright-free sources require prompts to be written in an early-20th or 19th century style? Newspapers and other copyrighted material are an excellent source of well-edited grammatically correct modern language. It is unreasonable to believe that a good model for modern languages can be built from sources that have fallen out of copyright.

Requiring model-building organizations to purchase the rights to their training data would inevitably leave generative AI in the hands of a small number of unassailable monopolies. (We won’t address what can or can’t be done with copyrighted material, but we will say that copyright law says nothing at all about the source of the material: you can buy it legally, borrow it from a friend, steal it, find it in the trash—none of this has any bearing on copyright infringement.) One of the participants at the WEF roundtable The Expanding Universe of Generative Models reported that Altman has said that he doesn’t see the need for more than one foundation model. That’s not unexpected, given my guess that his strategy is built around minimizing competition. But this is chilling: if all AI applications go through one of a small group of monopolists, can we trust those monopolists to deal honestly with issues of bias? AI developers have said a lot about “alignment,” but discussions of alignment always seem to sidestep more immediate issues like race and gender-based bias. Will it be possible to develop specialized applications (for example, O’Reilly Answers) that require training on a specific dataset? I’m sure the monopolists would say “of course, those can be built by fine tuning our foundation models”; but do we know whether that’s the best way to build those applications? Or whether smaller companies will be able to afford to build those applications, once the monopolists have succeeded in buying the market? Remember: Uber was once inexpensive.

If model development is limited to a few wealthy companies, its future will be bleak. The outcome of copyright lawsuits won’t just apply to the current generation of Transformer-based models; they will apply to any model that needs training data. Limiting model building to a small number of companies will eliminate most academic research. It would certainly be possible for most research universities to build a training corpus on content they acquired legitimately. Any good library will have the Times and other newspapers on microfilm, which can be converted to text with OCR. But if the law specifies how copyrighted material can be used, research applications based on material a university has legitimately purchased may not be possible. It won’t be possible to develop open source models like Mistral and Mixtral—the funding to acquire training data won’t be there—which means that the smaller models that don’t require a massive server farm with power-hungry GPUs won’t exist. Many of these smaller models can run on a modern laptop, which makes them ideal platforms for developing AI-powered applications. Will that be possible in the future? Or will innovation only be possible through the entrenched monopolies?

Open source AI has been the victim of a lot of fear-mongering lately. However, the idea that open source AI will be used irresponsibly to develop hostile applications that are inimical to human well-being gets the problem precisely wrong. Yes, open source will be used irresponsibly—as has every tool that has ever been invented. However, we know that hostile applications will be developed, and are already being developed: in military laboratories, in government laboratories, and at any number of companies. Open source gives us a chance to see what is going on behind those locked doors: to understand AI’s capabilities and possibly even to anticipate abuse of AI and prepare defenses. Handicapping open source AI doesn’t “protect” us from anything; it prevents us from becoming aware of threats and developing countermeasures.

Transparency is important, and proprietary models will always lag open source models in transparency. Open source has always been about source code, rather than data; but that is changing. OpenAI’s GPT-4 scores surprisingly well on Stanford’s Foundation Model Transparency Index, but still lags behind the leading open source models (Meta’s LLaMA and BigScience’s BLOOM). However, it isn’t the total score that’s important; it’s the “upstream” score, which includes sources of training data, and on this the proprietary models aren’t close. Without data transparency, how will it be possible to understand biases that are built in to any model? Understanding those biases will be important to addressing the harms that models are doing now, not hypothetical harms that might arise from sci-fi superintelligence. Limiting AI development to a few wealthy players who make private agreements with publishers ensures that training data will never be open.

What will AI be in the future? Will there be a proliferation of models? Will AI users, both corporate and individuals, be able to build tools that serve them? Or will we be stuck with a small number of AI models running in the cloud and being billed by the transaction, where we never really understand what the model is doing or what its capabilities are? That’s what the endgame to the legal battle between OpenAI and the Times is all about.

View Details

2024 started with yet more AI: a small language model from Microsoft, a new (but unnamed) model from Meta that competes with GPT-4, and a text-to-video model from Google that claims to be more realistic than anything yet. Research into security issues has also progressed—unfortunately, discovering more problems than solutions. A common thread in several recent attacks has been to use embeddings: an attacker discovers innocuous text or images that happen to have an embedding similar to words describing actions that aren’t allowed. These innocuous inputs easily get by filters designed to prevent hostile prompts.

AI Merging large language models gets developers the best of many worlds: use different models to solve different kinds of problems. It’s essentially mixture of experts but applied at the application level of the stack rather than the model level. * Researchers have developed a method for detecting AI-generated text that is 90% accurate and has a false positive rate of only 0.01%. * Google has announced Lumiere, a text-to-video model that generates “realistic, diverse, and coherent” motion. Lumiere generates the entire video in one pass rather than generating distinct keyframes that are then merged. * Is JavaScript a useful language for developing artificial intelligence applications? The New Stack* lists five tools for building AI applications in JavaScript, starting with TensorFlow.js. * Meta has released a new language model that claims performance similar to GPT-4. It is a self-rewarding language model; it continually evaluates its responses to prompts and adjusts its parameters in response. An independent open source implementation is already on GitHub. * Hospitals are using federated learning techniques to collect and share patient data without compromising privacy. With federated learning, the hospitals aren’t sharing actual patient data but machine learning models built on local data. * Researchers have discovered “compositional attacks” against multimodal language models. In these attacks, prompts that combine text and images are used to “jailbreak” the model. A hostile but benign-looking image establishes a context in which the model ignores its guardrails. * Researchers have used tests for psychologically profiling humans to profile AI models and research their built-in biases and prejudices. * Direct Preference Optimization (DPO) is an algorithm for training language models to operate in agreement with human preferences. It is simpler and more efficient than RLHF. * Mistral has published a paper describing its Mixtral 8x7B model, a mixture of experts model with very impressive performance. * Volkswagen has added ChatGPT to the infotainment system on its cars. ChatGPT will not have access to any of the car’s data. * Language models rely on converting input tokens to embeddings (long sequences of numbers). Can the original text be recovered from the embeddings used with language models? The answer may be yes. * AWS’s AI product, Q, now has tools to automate updating Java programs to new versions. That includes finding and replacing deprecated dependencies. * Microsoft’s Phi-2 model is now open source; it has been relicensed with the MIT license. Phi-2 is a small model (2.7B parameters) with performance comparable to much larger models. * Simon Willison’s summary of AI in 2023 is the best we’ve seen. In the coming year, Simon would love to see us get beyond “vibes-based development.” Unlike traditional programming, AI doesn’t do what you tell it to do, and we’re frequently forced to evaluate AI output on the basis of whether it “feels right.” * The US FTC has issued a challenge to developers: develop software that can detect AI-generated clones of human voices. The winner will receive a $25,000 prize. * DeepMind has built a model that can solve geometry problems. The new model combines a language model with symbolic AI, giving it the ability to reason logically about problems in addition to matching patterns.

Programming* Any app can become extensible. Extism is a WebAssembly library that can be added to almost any app that allows app users to write plug-ins in most major programming languages. * Zed, a collaborative code editor, is now open source and available on GitHub. * A study by GitHub shows that creating a good developer experience (DevEx or DX) improves productivity by reducing cognitive load, shortening feedback loops, and helping developers to remain in “flow state.” * Julia Evans (@b0rk@jvns.ca) has compiled a list of common Git mistakes. * Ruffle is a Flash emulator built with Rust and Wasm. While you may not remember Macromedia Flash, and you probably don’t want to use it for new content, the New York Times is using Ruffle to resurrect archival content that used Flash for visualizations. * JavaScript as a shell language? Bun is an open source JavaScript shell that can run on Linux, macOS, and Windows. It’s the only shell that is truly platform-independent. * Shadeup is a new programming language that extends TypeScript. It is designed to simplify working with WebGPU. * “Rethinking Observability” argues for thinking about how users experience a service, rather than details of the service’s implementation. What are the critical user journeys (CUJs), and what are service level objectives (SLOs) for those paths through the system? * Marimo is a new Python notebook with some important features. When you edit any cell, it automatically updates all affected cells; the notebooks themselves are pure Python and can be managed with Git and other tools; GitHub Copilot is integrated into the Marimo editor. * LinkedIn has released its Developer Productivity and Happiness Framework, a set of metrics for processes that affect developer experience. The metrics include things like code review response time, but LinkedIn points out that the framework is most useful in helping teams build their own metrics. * The Node package registry, NPM, recently accepted a package named “everything” that links to everything in the registry. Whether this was a joke or a hostile attack remains to be seen, but an important side effect is that it became impossible to remove a package from NPM. * container2wasm takes a container image and converts it to WebAssembly, The Wasm executable can be run with WASI or even in a browser. This project is still in its early stages, but it is very impressive. * The AHA Stack provides a way to build web applications that minimizes browser-side JavaScript. It is based on the Astro framework, htmx, and Alpine.js. * Last year ended with Brainfuck implemented in PostScript. To start 2024, someone has found a working Lisp interpreter written in Malbolge, a language that competes with Brainfuck for being the most difficult, frustrating, and obtuse programming language in existence. * The year starts with a new Python web framework, Microdot. How long has it been since we’ve had a new Python framework? It’s very similar to Flask, but it’s small; it was designed to run on MicroPython, which runs on microcontrollers like ESP8266. * Odin is yet another new programming language. It supports data-oriented programming and promises high performance with explicit (though safe) control of memory management and layout. It claims simplicity, clarity, and readability.

Security* The UK’s National Cyber Security Center has warned that generative AI will be used in ransomware and other attacks. Generative AI will make social engineering and phishing more convincing; it will enable inexperienced actors to create much more dangerous attacks. * A presentation at USENIX’s security symposium argues that side channels leak information in almost all commodity PCs: microphones, cameras, and other sensors pick up electromagnetic signals from the processor. These signals can be captured and decoded. * Like everyone else, malware groups are moving to memory-safe languages like Rust and DLang to develop their payloads. * Researchers have discovered that poisoned training data can be used to insert backdoors into large language models. These backdoors can be triggered by special prompts and cannot be discovered or removed by current safety techniques. * Programmers who use AI assistants are likely to write code that is less secure while believing that their code is more secure. However, users of AI assistants who don’t “trust” the AI engage more with the code produced and are likely to produce code that is more secure. * A variant of the Mirai malware is attacking Linux systems. This variant finds weak SSH passwords and installs cryptocurrency mining software to create a mining botnet. * Many groups offer “bug bounties” that pay rewards to those who discover bugs (particularly security vulnerabilities) in their code. One open source maintainer argues that this process is being distorted by incorrect bug reports that are generated by AI, wasting maintainers’ time. * The US National Institute of Standards and Technology has published a taxonomy and standard terminology for attacks against machine learning and AI systems.

Web Nimbo Earth Online aims to be a “digital twin” of the Earth. It’s superficially similar to Google Earth but has fascinating features like the ability to see historical progressions: for example, how a landscape changed after a fire or how a river’s course wandered over the years. * A study shows that search results are getting worse as a result of SEO spam. The problem affects all major search engines. If you read the paper and ignore click-bait summaries, Google is doing a somewhat better job of maintaining search integrity than its competitors. * The Verge* has an excellent article about how optimizing sites for Google search have affected web design, making sites much more homogeneous. * Facebook’s app includes a new Link History setting (on by default) that encourages use of the app’s built-in browser. Link History saves all links, and the browser is known to include a keylogger; the data from both is used for targeted advertising.

Quantum Computing* While we don’t yet have usable quantum computers, an improvement to Shor’s algorithm for factoring numbers has been published. While it reduces the computational time from O(N^2) to O(N^1.5), it increases the number of qubits required, which may be an important limitation.

View Details

This has been a strange year. While we like to talk about how fast technology moves, internet time, and all that, in reality the last major new idea in software architecture was microservices, which dates to roughly 2015. Before that, cloud computing itself took off in roughly 2010 (AWS was founded in 2006); and Agile goes back to 2000 (the Agile Manifesto dates back to 2001, Extreme Programming to 1999). The web is over 30 years old; the Netscape browser appeared in 1994, and it wasn’t the first. We think the industry has been in constant upheaval, but there have been relatively few disruptions: one every five years, if that.

2023 was one of those rare disruptive years. ChatGPT changed the industry, if not the world. We’re skeptical about things like job displacement, at least in technology. But AI is going to bring changes to almost every aspect of the software industry. What will those changes be? We don’t know yet; we’re still at the beginning of the story. In this report about how people are using O’Reilly’s learning platform, we’ll see how patterns are beginning to shift.

Just a few notes on methodology: This report is based on O’Reilly’s internal “Units Viewed” metric. Units Viewed measures the actual usage of content on our platform. The data used in this report covers January through November in 2022 and 2023. Each graph is scaled so that the topic with the greatest usage is 1. Therefore, the graphs can’t be compared directly to each other.

Remember that these “units” are “viewed” by our users, who are largely professional software developers and programmers. They aren’t necessarily following the latest trends. They’re solving real-world problems for their employers. And they’re picking up the skills they need to advance in their current positions or to get new ones. We don’t want to discount those who use our platform to get up to speed on the latest hot technology: that’s how the industry moves forward. But to understand usage patterns, it’s important to realize that every company has its own technology stacks, and that those stacks change slowly. Companies aren’t going to throw out 20 years’ investment in PHP so they can adopt the latest popular React framework, which will probably be displaced by another popular framework next year.

Software DevelopmentMost of the topics that fall under software development declined in 2023. What does this mean? Programmers are still writing software; our lives are increasingly mediated by software, and that isn’t going to change.

Software developers are responsible for designing and building bigger and more complex projects than ever. That’s one trend that won’t change: complexity is always “up and to the right.” Generative AI is the wild card: Will it help developers to manage complexity? Or will it add complexity all its own? It’s tempting to look at AI as a quick fix. Who wants to learn about coding practices when you’re letting GitHub Copilot write your code for you? Who wants to learn about design patterns or software architecture when some AI application may eventually do your high-level design? AI is writing low-level code now; as many as 92% of software developers are using it. Whether it will be able to do high-level design is an open question—but as always, that question has two sides: “Will AI do our design work?” is less interesting than “How will AI change the things we want to design?” And the real question that will change our industry is “How do we design systems in which generative AI and humans collaborate effectively?”

Figure 1. Software architectureRegardless of the answers to these questions, humans will need to understand and specify what needs to be designed. Our data shows that most topics in software architecture and design are down year-over-year. But there are exceptions. While software architecture is down 3.9% (a relatively small decline), enterprise architecture is up 8.9%. Domain-driven design is particularly useful for understanding the behavior of complex enterprise systems; it’s down, but only 2.0%. Use of content about event-driven architecture is relatively small, but it’s up 40%. That change is important because event-driven architecture is a tool for designing large systems that have to ingest data from many different streams in real time. Functional programming, which many developers see as a design paradigm that will help solve the problems of distributed systems, is up 9.8%. So the software development world is changing. It’s shifting toward distributed systems that manage large flows of data in real time. Use of content on topics relevant to that shift is holding its own or growing.

Microservices saw a 20% drop. Many developers expressed frustration with microservices during the year and argued for a return to monoliths. That accounts for the sharp decline—and it’s fair to say that many organizations are paying the price for moving to microservices because it was “the thing to do,” not because they needed the scale or flexibility that microservices can offer. From the start, microservice proponents have argued that the best way to develop microservices is to start with a monolith, then break the monolith into services as it becomes necessary. If implemented poorly, microservices deliver neither scale nor flexibility. Microservices aren’t ideal for new greenfield projects, unless you’re absolutely sure that you need them from the start—and even then, you should think twice. It’s definitely not a technology to implement just to follow the latest fad.

Software developers run hot and cold on design patterns, which declined 16%. Why? It probably depends on the wind or the phase of the moon. Content usage about design patterns increased 13% from 2021 to 2022, so this year’s decline just undoes last year’s gain. It’s possible that understanding patterns seems less important when AI is writing a lot of the code for you. It’s also possible that design patterns seem less relevant when code is already largely written; most programmers maintain existing applications rather than develop new greenfield apps, and few texts about design patterns discuss the patterns that are embedded in legacy applications. But both ways of thinking miss the point. Design patterns are common solutions to common problems that have been observed in practice. Understanding design patterns keeps you from reinventing wheels. Frameworks like React and Spring are important because they implement design patterns. Legacy applications won’t be improved by refactoring existing code just to use some pattern, but design patterns are useful for extending existing software and making it more flexible. And, of course, design patterns are used in legacy code—even code that was written before the term was coined! Patterns are discovered, not “invented”; again, they’re common solutions to problems programmers have been solving since the beginning of programming.

At the same time, whenever there’s a surge of interest in design patterns, there’s a corresponding surge in pattern abuse: managers asking developers how many patterns they used (as if pattern count were a metric for good code), developers implementing FactoryFactoryFactory Factories, and the like. What goes around comes around, and the abuse of design patterns is part of a feedback loop that regulates the use of design patterns.

Programming and Programming LanguagesMost of the programming languages we track showed declines in content usage. Before discussing specifics, though, we need to look at general trends. If 92% of programmers are using generative AI to write code and answer questions, then we’d certainly expect a drop in content use. That may or may not be advisable for career development, but it’s a reality that businesses built on training and learning have to acknowledge. But that isn’t the whole story either—and the bigger story leaves us with more questions than answers.

Rachel Stephens provides two fascinating pieces of the puzzle in a recent article on the RedMonk blog, but those pieces don’t fit together exactly. First, she notes the decline in questions asked on Stack Overflow and states (reasonably) that asking a nonjudgmental AI assistant might be a preferable way for beginners to get their questions answered. We agree; we at O’Reilly have built O’Reilly Answers to provide that kind of assistance (and are in the process of a major upgrade that will make it even more useful). But Stack Overflow shows a broad peak in questions from 2014 to 2017, with a sharp decline afterward; the number of questions in 2023 is barely 50% of the peak, and the 20% decline from the January 2023 report to the July report is only somewhat sharper than the previous drops. And there was no generative AI, no ChatGPT, back in 2017 when the decline began. Did generative AI play a role? It would be foolish to say that it didn’t, but it can’t be the whole story.

Stephens points to another anomaly: GitHub pull requests declined roughly 25% from the second half of 2022 to the first half of 2023. Why? Stephens guesses that there was increased GitHub activity during the pandemic and that activity has returned to normal now that we’ve (incorrectly) decided the pandemic is over. Our own theory is that it’s a reaction to GPT models leaking proprietary code and abusing open source licenses; that could cause programmers to be wary of public code repositories. But those are only guesses. This change is apparently not an error in the data. It might be a one-time anomaly, but no one really knows the cause. Something drove down programmer activity on GitHub, and that’s inevitably a part of the background to this year’s data.

So, what does O’Reilly’s data say? As it has been for many years, Python is the most widely used programming language on our platform. This year, we didn’t see an increase; we saw a very small (0.14%) decline. That’s noise; we won’t insult your intelligence by claiming that “flat in a down market” is really a gain. It’s certainly fair to ask whether a language as popular as Python has gathered all the market share that it will get. When you’re at the top of the adoption curve, it’s difficult to go any higher and much easier to drop back. There are always new languages ready to take some of Python’s market share. The most significant change in the Python ecosystem is Microsoft’s integration of Python into Excel spreadsheets, but it’s too early to expect that to have had an effect.

Use of content about Java declined 14%, a significant drop but not out of line with the drop in GitHub activity. Like Python, Java is a mature language and may have nowhere to go but down. It has never been “well loved”; when Java was first announced, people walked out of the doors of the conference room claiming that Java was dead before you could even download the beta. (I was there.) Is it time to dance on Java’s grave? That dance has been going on since 1995, and it hasn’t been right yet.

Figure 2. Programming languagesJavaScript also declined by 3.9%. It’s a small decline and probably not meaningful. TypeScript, a version of JavaScript that adds static typing and type annotations, gained 5.6%. It’s tempting to say that these cancel each other out, but that’s not correct. Usage of TypeScript content is roughly one-tenth the usage of JavaScript content. But it is correct to say that interest in type systems is growing among web developers. It’s also true that an increasing number of junior developers use JavaScript only through a framework like React or Vue. Boot camps and other crash programs often train students in “React,” with little attention on the bigger picture. Developers trained in programs like these may be aware of JavaScript but may not think of themselves as JavaScript developers, and may not be looking to learn more about the language outside of a narrow, framework-defined context.

We see growth in C++ (10%), which is surprising for an old, well-established language. (C++ first appeared in 1985.) At this point in C++’s history, we’d expect it to be a headache for people maintaining legacy code, not a language for starting new projects. Why is it growing? While C++ has long been an important language for game development, there are signs that it’s breaking out into other areas. C++ is an ideal language for embedded systems, which often require software that runs directly on the processor (for example, the software that runs in a smart lightbulb or in the braking system of any modern car). You aren’t going to use Python, Java, or JavaScript for those applications. C++ is also an excellent language for number crunching (Python’s numeric libraries are written in C++), which is increasingly important as artificial intelligence goes mainstream. It has also become the new “must have” language on résumés: knowing C++ proves that you’re tough, that you’re a “serious” programmer. Job anxiety exists—whether or not it’s merited is a different question—and in an environment where programmers are nervous about keeping their current jobs or looking forward to finding a new one, knowing a difficult but widely used language can only be an asset.

Use of content about Rust also increased from 2022 to 2023 (7.8%). Rust is a relatively young language that stresses memory safety and performance. While Rust is considered difficult to learn, the idea that memory safety is baked in makes it an important alternative to languages like C++. Bugs in memory management are a significant source of vulnerabilities, as noted in NIST’s page on “Safer Languages,” and Rust does a good job of enforcing safe memory usage. It’s now used in operating systems (Linux kernel components), tool development, and even enterprise software.

We also saw 9.8% growth in content about functional programming. We didn’t see gains for any of the historical functional programming languages (Haskell, Erlang, Lisp, and Elixir) though; most saw steep declines. In the past decade, most programming languages have added functional features. Newer languages like Rust and Go have had them from the start. And Java has gradually added features like closures in a series of updates. Now programmers can be as functional as they want to be without switching to a new language.

Finally, there are some programming languages that we don’t yet track but that we’re watching with interest. Zig is a simple imperative language that’s designed to be memory safe, like Rust, but relatively easy to learn. Mojo is a superset of Python that’s compiled, not interpreted. It’s designed for high performance, especially for numerical operations. Mojo’s goal is to facilitate AI programming in a single language rather than a combination of Python and some other language (typically C++) that’s used for performance-critical numerical code. Where are these languages going? It will be some years before they reach the level of Rust or Go, but they’re off to a good start.

So what does all this tell us about training and skill development? It’s easy to think that, with Copilot and other tools to answer all your questions, you don’t need to put as much effort into learning new technologies. We all ask questions on Google or Stack Overflow, and now we have other places to get answers. Necessary as that is, the idea that asking questions can replace training is naive. Unlike many who are observing the influence of generative AI on programming, we believe that it will increase the gap between entry-level skills and senior developer skills. Being a senior developer—being a senior anything—requires a kind of fluency that you can’t get just from asking questions. I may never be a fluent user of Python’s pandas library (which I used extensively to write this report); I asked lots of questions, and that has undoubtedly saved me time. But what happens when I need to solve the next problem? The kind of fluency that you need to look at a problem and understand how to solve it doesn’t come from asking simple “How do I do this?” questions. Nor does it preclude asking lots of “I forgot how this function works” questions. That’s why we’ve built O’Reilly Answers, an AI-driven service that finds solutions to questions using content from our platform. But expertise does require developing the intellectual muscle that comes from grappling with problems and solving them yourself rather than letting something else solve them for you. (And that includes forcing yourself to remember all the messy syntax details.) People who think generative AI is a shortcut to expertise (and the job title and salary that expertise merits) are shortchanging themselves.

Artificial IntelligenceIn AI, there’s one story and only one story, and that’s the GPT family of models. Usage of content on these models exploded 3,600% in the past year. That explosion is tied to the appearance of ChatGPT in November 2022. But don’t make the mistake of thinking that ChatGPT came out of nowhere. GPT-3 created a big splash when it was released in 2020 (complete with a clumsy web-based interface). GPT-2 appeared in 2019, and the original unnumbered GPT was even earlier. The real innovation in ChatGPT wasn’t the technology itself (though the models behind it represent a significant breakthrough in AI performance); it was packaging the model as a chatbot. That doesn’t mean that the GPT explosion wasn’t real. While our analysis of search trends shows that interest in ChatGPT has peaked among our platform’s users, interest in natural language processing (NLP) showed a 195% increase—and from a much higher starting point.1 That makes sense, given the more technical nature of our audience. Software developers will be building on top of the APIs for GPT and other language models and are likely less interested in ChatGPT, the web-based chat service. Related topics generative models (900%) and Transformers (325%) also showed huge gains. Prompt engineering, which didn’t exist in 2022, became a significant topic, with roughly the same usage as Transformers. As far as total use, NLP is almost twice GPT. However you want to read the data, this is AI’s big year, largely due to the GPT models and the idea of generative AI.

Figure 3. Artificial intelligenceBut don’t assume that the explosion of interest in generative AI meant that other aspects of AI were standing still. Deep learning, the creation and application of neural networks with many layers, is fundamental to every aspect of modern AI. Usage in deep learning content grew 19% in the past year. Reinforcement learning, in which models are trained by giving “rewards” for solving problems, grew 15%. Those gains only look small in comparison to the triple- and quadruple-digit gains we’re seeing in natural language processing. PyTorch, the Python library that has come to dominate programming in machine learning and AI, grew 25%. In recent years, interest in PyTorch has been growing at the expense of TensorFlow, but TensorFlow showed a small gain (1.4%), reversing (or at least pausing) its decline. Interest in two older libraries, scikit-learn and Keras, declined: 25% for scikit-learn and 4.8% for Keras. Keras has largely been subsumed by TensorFlow, while scikit-learn hasn’t yet incorporated the capabilities that would make it a good platform for building generative AI. (An attempt to implement Transformers in scikit-learn appears to be underway at Hugging Face.)

We’ve long said that operations is the elephant in the room for machine learning and artificial intelligence. Building models and developing applications is challenging and fun, but no technology can mature if IT teams can’t deploy, monitor, and manage it. Interest in operations for machine learning (MLOps) grew 14% over the past year. This is solid, substantial growth that only looks small in comparison with topics like generative AI. Again, we’re still in the early stages—generative AI and large language models are only starting to reach production. If anything, this increase probably reflects older applications of AI. There’s a growing ecosystem of startups building tools for deploying and monitoring language models, which are fundamentally different from traditional applications. As companies deploy the applications they’ve been building, MLOps will continue to see solid growth. (More on MLOps when we discuss operations below.)

LangChain is a framework for building generative AI applications around groups of models and databases. It’s often used to implement the retrieval-augmented generation (RAG) pattern, where a user’s prompt is used to look up relevant items in a vector database; those items are then combined with the prompt, generating a new prompt that is sent to the language model. There isn’t much content about LangChain available yet, and it didn’t exist in 2022, but it’s clearly going to become a foundational technology. Likewise, vector databases aren’t yet in our data. We expect that to change next year. They are rather specialized, so we expect usage to be relatively small, unlike products like MySQL—but they will be very important.

AI wasn’t dominated entirely by the work of OpenAI; Meta’s LLaMA and Llama 2 also attracted a lot of attention. The source code for LLaMA was open source, and its weights (parameters) were easily available to researchers. Those weights quickly leaked from “researchers” to the general public, where they jump-started the creation of smaller open source models. These models are much smaller than behemoths like GPT-4. Many of them can run on laptops, and they’re proving ideal for smaller companies that don’t want to rely on Microsoft, OpenAI, or Google to provide AI services. (If you want to run an open source language model on your laptop, try llamafile.) While huge “foundation models” like the GPT family won’t disappear, in the long run open source models like Alpaca and Mistral may prove to be more important to software developers.

It’s easy to think that generative AI is just about software development. It isn’t; its influence extends to just about every field. Our ChatGPT: Possibilities and Pitfalls Superstream was the most widely attended event we’ve ever run. There were over 28,000 registrations, with attendees and sponsors from industries as diverse as pharmaceuticals, logistics, and manufacturing. Attendees included small business owners, sales and marketing personnel, and C-suite executives, along with many programmers and engineers from different disciplines. We’ve also been running courses focused on specific industries: Generative AI for Finance had over 2,000 registrations, and Generative AI for Government over 1,000. And more than 1,000 people signed up for our Generative AI for Healthcare event.

DataIn previous years, we would have told the story of AI as part of the story of data. That’s still correct; with its heavy emphasis on mathematics and statistics, AI is a natural outgrowth of data science. But this year, AI has become the superstar that gets top billing, while data is a supporting actor.

That doesn’t mean that data is unimportant. Far from it. Every company uses data: for planning, for making projections, for analyzing what’s happening within the business and the markets they serve. So it’s not surprising that the second biggest topic in data is Microsoft Power BI, with a 36% increase since 2022. SQL Server also showed a 5.3% increase, and statistics toolbox R increased by 4.8%.

Figure 4. Data analysis and databasesData engineering was by far the most heavily used topic in this category; it showed a 3.6% decline, stabilizing after a huge gain from 2021 to 2022. Data engineering deals with the problem of storing data at scale and delivering that data to applications. It includes moving data to the cloud, building pipelines for acquiring data and getting data to application software (often in near real time), resolving the issues that are caused by data siloed in different organizations, and more. Two of the most important platforms for data engineering, Kafka and Spark, showed significant declines in 2023 (21% and 20%, respectively). Kafka and Spark have been workhorses for many years, but they are starting to show their age as they become “legacy technology.” (Hadoop, down 26%, is clearly legacy software in 2023.) Interest in Kafka is likely to rise as AI teams start implementing real-time models that have up-to-the-minute knowledge of external data. But we also have to point out that there are newer streaming platforms (like Pulsar) and newer data platforms (like Ray).

Designing enterprise-scale data storage systems is a core part of data engineering. Interest in data warehouses saw an 18% drop from 2022 to 2023. That’s not surprising; data warehouses also qualify as legacy technology. Two other patterns for enterprise-scale storage show significant increases: Usage of content about data lakes is up 37% and, in absolute terms, significantly higher than that of data warehouses. Usage for data mesh content is up 5.6%. Both lakes and meshes solve a basic problem: How do you store data so that it’s easy to access across an organization without building silos that are only relevant to specific groups? Data lakes can include data in many different formats, and it’s up to users to supply structure when data is utilized. A data mesh is a truly distributed solution: each group is responsible for its own data but makes that data available throughout the enterprise through an interoperability layer. Those newer technologies are where we see growth.

The two open source data analysis platforms were virtually unchanged in 2023. Usage of content about R increased by 3.6%; we’ve already seen that Python was unchanged, and pandas grew by 1.4%. Neither of these is going anywhere, but alternatives, particularly to pandas, are appearing.

OperationsWhether you call it operations, DevOps, or something else, this field has seen some important changes in the past year. We’ve witnessed the rise of developer platforms, along with the related topic, platform engineering. Both of those are too new to be reflected in our data: you can’t report content use before content exists. But they are influencing other topics.

We’ve said in the past that Linux is table stakes for a job in IT. That’s still true. But the more the deployment process is automated—and platform engineering is just the next step in “Automate All the Things”—the less developers and IT staff need to know about Linux. Software is packaged in containers, and the containers themselves run as virtual Linux instances, but developers don’t need to know how to find and kill out-of-control processes, do a backup, install device drivers, or perform any of the other tasks that are the core of system administration. Usage of content about Linux is down 6.9%: not a major change but possibly a reflection of the fact that the latest steps forward in deploying and managing software shield people from direct contact with the operating system.

Similar trends reduce what developers and IT staff need to know about Kubernetes, the near-ubiquitous container orchestrator (down 6.9%). Anyone who uses Kubernetes knows that it’s complex. We’ve long expected “something simpler” to come along and replace it. It hasn’t—but again, developer platforms put users a step further away from engaging with Kubernetes itself. Knowledge of the details is encapsulated either in a developer platform or, perhaps more often, in a Kubernetes service administered by a cloud provider. Kubernetes can’t be ignored, but it’s more important to understand high-level principles than low-level commands.

Figure 5. Infrastructure and operationsDevOps (9.0%) and SRE (13%) are also down, though we don’t think that’s significant. Terms come and go, and these are going. While operations is constantly evolving, we don’t believe we’ll ever get to the mythical state of “NoOps,” nor should we. Instead, we’ll see constant evolution as the ratio of systems managed to operations staff grows ever higher. But we do believe that sooner rather than later, someone will put a new name on the disciplines of DevOps and its close relative, SRE. That new name might be “platform engineering,” though that term says more about designing deployment pipelines than about carrying the pager and keeping the systems running; platform engineering is about treating developers as customers and designing internal developer platforms that make it easy to test and deploy software systems with minimal ceremony. We don’t believe that platform engineering subsumes or replaces DevOps. Both are partners in improving experience for developers and operations staff (and ratcheting up the ratio of systems managed to staff even higher).

That’s a lot of red ink. What’s in the black? Supply chain management is up 5.9%. That’s not a huge increase, but in the past few years we’ve been forced to think about how we manage the software supply chain. Any significant application easily has dozens of dependencies, and each of those dependencies has its own dependencies. The total number of dependencies, including both direct and inherited dependencies, can easily be hundreds or even thousands. Malicious operators have discovered that they can corrupt software archives, getting programmers to inadvertently incorporate malware into their software. Unfortunately, security problems never really go away; we expect software supply chain security to remain an important issue for the foreseeable (and unforeseeable) future.

We’ve already mentioned that MLOps, the discipline of deploying and managing models for machine learning and artificial intelligence, is up 14%. Machine learning and AI represent a new kind of software that doesn’t follow traditional rules, so traditional approaches to operations don’t work. The list of differences is long:

• While most approaches to deployment are based on the idea that an application can be reproduced from a source archive, that isn’t true for AI. An AI system depends as much on the training data as it does on the source code, and we don’t yet have good tools for archiving training data.
• While we’ve said that open source models such as Alpaca are much smaller than models like GPT-4 or Google’s Gemini, even the smallest of those models is very large by any reasonable standard.
• While we’ve gotten used to automated testing as part of a deployment pipeline, AI models aren’t deterministic. A test doesn’t necessarily give the same result every time it runs. Testing is no less important for AI than it is for traditional software (arguably it’s more important), and we’re starting to see startups built around AI testing, but we’re still at the beginning.

That’s just a start. MLOps is a badly needed specialty. It’s good to see growing interest.

SecurityAlmost all branches of security showed growth from 2022 to 2023. That’s a welcome change: in the recent past, many companies talked about security but never made the investment needed to secure their systems. That’s changing, for reasons that are obvious to anyone who reads the news. Nobody wants to be a victim of data theft or ransomware, particularly now that ransomware has evolved into blackmail.

The challenges are really very simple. Network security, keeping intruders off of your network, was the most widely used topic and grew 5%. Firewalls, which are an important component of network security, grew 16%. Hardening, a much smaller topic that addresses making systems less vulnerable to attack, grew 110%. Penetration testing remained one of the most widely used topics. Usage dropped 5%, although a 10% increase for Kali Linux (an important tool for penetration testers) largely offsets that decline.

The 22% growth in security governance is another indicator of changed attitudes: security is no longer an ad hoc exercise that waits for something to happen and then fights fires. Security requires planning, training, testing, and auditing to ensure that policies are effective.

One key to security is knowing who your users are and which parts of the system each user can access. Identity and access management (IAM) has often been identified as a weakness, particularly for cloud security. As systems grow more complex, and as our concept of “identity” evolves from individuals to roles assigned to software services, IAM becomes much more than usernames and passwords. It requires a thorough understanding of who the actors are on your systems and what they’re allowed to do. This extends the old idea of “least privilege”: each actor needs the ability to do exactly what they need, no more and no less. The use of content about IAM grew 8.0% in the past year. It’s a smaller gain than we would have liked to see but not insignificant.

Figure 6. SecurityApplication security grew 42%, showing that software developers and operations staff are getting the message. The DevSecOps “shift left” movement, which focuses on software security early in the development process, appears to be winning; use of content about DevSecOps was up 30%. Similarly, those who deploy and maintain applications have become even more aware of their responsibilities. Developers may design identity and access management into the code, but operations is responsible for configuring these correctly and ensuring that access to applications is only granted appropriately. Security can’t be added after the fact; it has to be part of the software process from beginning to the end.

Advanced persistent threats (APTs) were all over the news a few years ago. We don’t see the term APT anywhere near as much as we used to, so we’re not surprised that usage has dropped by 35%. Nevertheless, nation-states with sophisticated offensive capabilities are very real, and cyber warfare is an important component of several international conflicts, including the war in Ukraine.

It’s disappointing to see that usage of content about zero trust has declined by 20%. That decrease is more than offset by the increase in IAM, which is an essential tool for zero trust. But don’t forget that IAM is just a tool and that the goal is to build systems that don’t rely on trust, that always verify that every actor is appropriately identified and authorized. How can you defend your IT infrastructure if you assume that attackers already have access? That’s the question zero trust answers. Trust nothing; verify everything.

Finally, compliance is down 27%. That’s more than offset by the substantial increase of interest in governance. Auditing for compliance is certainly a part of governance. Focusing on compliance itself, without taking into account the larger picture, is a problem rather than a solution. We’ve seen many companies that focus on compliance with existing standards and regulations while avoiding the hard work of analyzing risk and developing effective policies for security. “It isn’t our fault that something bad happened; we followed all the rules” is, at best, a poor way to explain systemic failure. If that compliance-oriented mindset is fading, good riddance. Compliance, understood properly, is an important component of IT governance. Understood badly, compliance is an unacceptable excuse.

Finally, a word about a topic that doesn’t yet appear in our data. There has, of course, been a lot of chatter about the use of AI in security applications. AI will be a great asset for log file analysis, intrusion detection, incident response, digital forensics, and other aspects of cybersecurity. But, as we’ve already said, there are always two sides to AI. How does AI change security itself? Any organization with AI applications will have to protect them from exploitation. What vulnerabilities does AI introduce that didn’t exist a few years ago? There are many articles about prompt injection, sneaky prompts designed to “jailbreak” AI systems, data leakage, and other vulnerabilities—and we believe that’s only the beginning. Securing AI systems will be a critical topic in the coming years.

Cloud ComputingLooking at platform usage for cloud-related topics, one thing stands out: cloud native. Not only is it the most widely used topic in 2023, but it grew 175% from 2022 to 2023. This marks a real transition. In the past, companies built software to run on-premises and then moved it to the cloud as necessary. Despite reports (including ours) that showed 90% or more “cloud adoption,” we always felt that was very optimistic. Sure, 90% of all companies may have one or two experiments in the cloud—but are they really building for the cloud? This huge surge in cloud native development shows that we’ve now crossed that chasm and that companies have stopped kicking the tires. They’re building for the cloud as their primary deployment platform.

You could, of course, draw the opposite conclusion by looking at cloud deployment, which is down 27%. If companies are developing for the cloud, how are those applications being deployed? That’s a fair question. However, as cloud usage grows, so does organizational knowledge of cloud-related topics, particularly deployment. Once an IT group has deployed its first application, the second isn’t necessarily “easy” or “the same,” but it is familiar. At this point in the history of cloud computing, we’re seeing few complete newcomers. Instead we’re seeing existing cloud users deploying more and more applications. We’re also seeing a rise in tools that streamline cloud deployment. Indeed, any provider worth thinking about has a tremendous interest in making deployment as simple as possible.

Figure 7. Cloud architectureUse of content about cloud security grew 25%, and identity and access management (IAM) grew 8%. An epidemic of data theft and ransomware that continues to this day put security on the corporate map as a priority, not just an expense with annual budget requests that sounded like an extortion scam: “Nothing bad happened this year; give us more money and maybe nothing bad will happen next year.” And while the foundation of any security policy is good local security hygiene, it’s also true that the cloud presents its own issues. Identity and access management: locally, that means passwords, key cards, and (probably) two-factor authentication. In the cloud, that means IAM, along with zero trust. Same idea, but it would be irresponsible to think that these aren’t more difficult in the cloud.

Hybrid cloud is a smaller topic area that has grown significantly in the past year (145%). This growth points partly to the cloud becoming the de facto deployment platform for enterprise applications. It also acknowledges the reality of how cloud computing is adopted. Years ago, when “the cloud” was getting started, it was easy for a few developers in R&D to expense a few hours of time on AWS rather than requisitioning new hardware. The same was true for data-aware marketers who wanted to analyze what was happening with their potential customers—and they might choose Azure. When senior management finally awoke to the need for a “cloud strategy,” they were already in a hybrid situation, with multiple wildcat projects in multiple clouds. Mergers and buyouts complicated the situation more. If company A is primarily using AWS and company B has invested heavily in Google Cloud, what happens when they merge? Unifying behind a single cloud provider isn’t going to be worth it, even though cloud providers are providing tools to simplify migration (at the same time as they make their own clouds difficult to leave). The cloud is naturally hybrid. “Private cloud” and “public cloud,” when positioned as alternatives to each other and to a hybrid cloud, smell like “last year’s news.” It’s not surprising that usage has dropped 46% and 10%, respectively.

Figure 8. Cloud providersWhat about the perennial horse race between Amazon Web Services, Microsoft Azure, and Google Cloud? Is anyone still interested, except perhaps investors and analysts? AWS showed a very, very small gain (0.65%), but Azure and Google Cloud showed significant losses (16% and 22%, respectively). We expected to see Azure catch up to AWS because of its lead in AI as a service, but it didn’t. As far as our platform is concerned, that’s still in the future.

Web DevelopmentReact and Angular continue to dominate web development. JavaScript is still the lingua franca of web development, and that isn’t likely to change any time soon.

But the usage pattern has changed slightly. Last year, React was up, and Angular was sharply down. This year, usage of React content hasn’t changed substantially (down 0.33%). Angular is down 12%, a smaller decline than last year but still significant. When a platform is as dominant as React, it may have nowhere to go but down. Is momentum shifting?

We see some interesting changes among the less popular frameworks, both old and new. First, Vue isn’t a large part of the overall picture, and it isn’t new—it’s been around since 2014—but if its 28% annual growth continues, it will soon become a dominant framework. That increase represents a solid turnaround after losing 17% from 2021 to 2022. Django is even older (created in 2005), but it’s still widely used—and with an 8% increase this year, it’s not going away. FastAPI is the newest of this group (2018). Even though it accounts for a very small percentage of platform use, it’s easy for a small change in usage to have a big effect. An 80% increase is hard to ignore.

It’s worth looking at these frameworks in a little more detail. Django and FastAPI are both Python-based, and FastAPI takes full advantage of Python’s type hinting feature. Python has long been an also-ran in web development, which has been dominated by JavaScript, React, and Angular. Could that be changing? It’s hard to say, and it’s worth noting that Flask, another Python framework, showed a 12% decrease. As a whole, Python frameworks probably declined from 2022 to 2023, but that may not be the end of the story. Given the number of boot camps training new web programmers in React, the JavaScript hegemony will be hard to overcome.

Figure 9. Web developmentWhat about PHP, another long-standing framework that dates back to 1995, when the web was indeed young? PHP grew 5.9% in the past year. The use of content about PHP is small compared to frameworks like React and Angular or even Django. PHP certainly doesn’t inspire the excitement that it did in the 1990s. But remember that over 80% of the web is built on PHP. It’s certainly not trendy, it’s not capable of building the feature-rich sites that many users expect—but it’s everywhere. WordPress (down 4.8%), a content management system used for millions of websites, is based on PHP. But regardless of the number of sites that are built on PHP or WordPress, Indeed shows roughly three times as many job openings for React developers as for PHP and WordPress combined. PHP certainly isn’t going away, and it may even be growing slightly. But we suspect that PHP programmers spend most of their time maintaining older sites. They already know what they need to do that, and neither of those factors drives content usage.

What about some other highly buzzworthy technologies? After showing 74% growth from 2021 to 2022, WebAssembly (Wasm) declined by 41% in 2023. Blazor, a web framework for C# that generates code for Wasm, declined by 11%. Does that mean that Wasm is dying? We still believe Wasm is a very important technology, and we frequently read about amazing projects that are built with it. It isn’t yet a mature technology—and there are plenty of developers willing to argue that there’s no need for it. We may disagree, but that misses the point. Usage of Wasm content will probably decline gradually…until someone creates a killer application with it. Will that happen? Probably, but we can’t guess when.

What does this mean for someone who’s trying to develop their skills as a web developer? First, you still can’t go wrong with React, or even with Angular. The other JavaScript frameworks, such as Next.js, are also good options. Many of these are metaframeworks built on React, so knowing them makes you more versatile while leveraging knowledge you already have. If you’re looking to broaden your skills, Django would be a worthwhile addition. It’s a very capable framework, and knowing Python will open up other possibilities in software development that may be helpful in the future, even if not now.

CertificationThis year, we took a different approach to certification. Rather than discussing certification for different subject areas separately (that is, cloud certification, security certification, etc.), we used data from the platform to build a list of the top 20 certifications and grouped them together. That process gives a slightly different picture of which certifications are important and why. We also took a brief look at O’Reilly’s new badges program, which gives another perspective on what our customers want to learn.

Figure 10. CertificationBased on the usage of content in our platform (including practice tests), the most popular certifications are security certifications: CISSP (which declined 4.8%) and CompTIA Security+ (which grew 6.0%). CISSP is an in-depth exam for security professionals, requiring at least five years’ experience before taking the exam. Security+ is more of an entry-level exam, and its growth shows that security staff are still in demand. ISACA’s Certified Information Security Manager (CISM) exam, which focuses on risk assessment, governance, and incident response, isn’t as popular but showed a 54% increase. CompTIA’s Certified Advanced Security Practitioner (CASP+) showed a 10% increase—not as large but part of the same trend. The Certified Ethical Hacker (CEH) exam, which focuses on techniques useful for penetration testing or red-teaming, is up 4.1%, after a decline last year. Those increases reflect where management is investing. Hoping that there won’t be an incident has been replaced by understanding exposure, putting in place governance mechanisms to minimize risk, and being able to respond to incidents when they occur.

What really stands out, however, isn’t security: it’s the increased use of content about CompTIA A+, which is up 58%. A+ isn’t a security exam; it’s advertised as an entry-level exam for IT support, stressing topics like operating systems, managing SaaS for remote work, troubleshooting software, hardware, and networking problems, and the like. It’s testimony to the large number of people who want to get into IT. Usage of content about the CompTIA Linux+ exam was much lower but also grew sharply (23%)—and, as we’ve said in the past, Linux is “table stakes” for almost any job in computing. It’s more likely that you’ll encounter Linux indirectly via containers or cloud providers rather than managing racks of computers running Linux; but you will be expected to know it. The Certified Kubernetes Administrator (CKAD) exam also showed significant growth (32%). Since it was first released in 2014, Kubernetes has become an inescapable part of IT operations. The biggest trend in IT, going back 70 years or so, has been the increase in the ratio of machines to operators: from multiple operators per machine in the ’60s to one operator per machine in the era of minicomputers to dozens and now, in the cloud, to hundreds and thousands. Complex as Kubernetes is—and we admit, we keep looking for a simpler alternative—it’s what lets IT groups manage large applications that are implemented as dozens of microservices and that run in thousands of containers on an uncountable number of virtual machines. Kubernetes has become an essential skill for IT. And certification is becoming increasingly attractive to people working in the field; there’s no other area in which we see so much growth.

Cloud certifications also show prominently. Although “the cloud” has been around for almost 20 years, and almost every company will say that they are “in the cloud,” in reality many companies are still making that transition. Furthermore, cloud providers are constantly adding new services; it’s a field where keeping up with change is difficult. Content about Amazon Web Services was most widely used. AWS Cloud Practitioner increased by 35%, followed by AWS Solutions Architect (Associate), which increased 15%. Microsoft Azure certification content followed, though the two most prominent exams showed a decline: Azure Fundamentals (AZ-900) was down 37%, and Azure Administration (AZ-104) was down 28%. Google Cloud certifications trailed the rest: Google’s Cloud Engineer showed solid growth (14%), while its Data Engineer showed a significant decline (40%).

Content about Microsoft’s AI-900 exam (Azure AI Fundamentals) was the least-used among the certifications that we tracked. However, it gained 121%—it more than doubled—from 2022 to 2023. While we can’t predict next year, this is the sort of change that trends are made of. Why did this exam suddenly get so hot? It’s easy, really: Microsoft’s investment in OpenAI, its integration of the GPT models into Bing and other products, and its AI-as-a-service offerings through Azure have suddenly made the company a leader in cloud-based AI. While we normally hedge our bets on smaller topics with big annual growth—it’s easy for a single new course or book to cause a large swing—AI isn’t going away, nor is Microsoft’s leadership in cloud services for AI developers.

Late in 2023, O’Reilly began to offer badges tied to course completion on the O’Reilly learning platform. Badges aren’t certifications, but looking at the top badges gives another take on what our customers are interested in learning. The results aren’t surprising: Python, GPT (not just ChatGPT), Kubernetes, software architecture, and Java are the most popular badges.

However, it’s interesting to look at the difference between our B2C customers (customers who have bought platform subscriptions as individuals) and B2B customers (who use the platform via a corporate subscription). For most topics, including those listed above, the ratio of B2B to B2C customers is in the range of 2:1 or 3:1 (two or three times as many corporate customers as individuals). The outliers are for topics like communications skills, Agile, Scrum, personal productivity, Excel, and presentation skills: users from B2B accounts obtained these badges four (or more) times as often as users with personal accounts. This makes sense: these topics are about teamwork and other skills that are valuable in a corporate environment.

There are few (if any) badge topics for which individual (B2C) users outnumbered corporate customers; that’s just a reflection of our customer base. However, there were some topics where the ratio of B2B to B2C customers was closer to one. The most interesting of these concerned artificial intelligence: large language models (LLMs), TensorFlow, natural language processing, LangChain, and MLOps. Why is there more interest among individuals than among corporate customers? Perhaps by next year we’ll know.

DesignThe important story in design is about tools. Topics like user experience and web design are stable or slightly down (down 0.62% and 3.5%, respectively). But usage about design tools is up 105%, and the VC unicorn Figma is up 145%. Triple-digit growth probably won’t continue, but it’s certainly worth noticing. It highlights two important trends that go beyond typical design topics, like UX.

First, low-code and no-code tools aren’t new, but many new ones have appeared in the past year. Their success has been aided by artificial intelligence. We already have AI tools that can generate text, whether for a production site or for a mockup. Soon we’ll have no-code tools that don’t just spit out a wireframe but will be able to implement the design itself. They will be smart about what the user wants them to do. But to understand the importance of low-code to design, you have to look beyond the use designers will make of these tools. Designers will also be designing these tools, along with other AI-powered applications. Tools for designers have to be well-designed, of course: that’s trivial. But what many discussions about AI ignore is that designing applications that use AI well is far from trivial. We’ve all been blindsided by the success of ChatGPT, which made the GPT models instantly accessible to everyone. But once you start thinking about the possibilities, you realize that a chat is hardly an ideal interface for an AI system.2 What will the users of these systems really need? We’ve only just started down that path. It will be an exciting journey—particularly for designers.

Figure 11. DesignSecond, Figma is important because it’s a breakthrough in tools for collaboration. Tools that allow remote employees to collaborate productively are crucial when coworkers can be anywhere: in an office, at home, or on another continent. The last year and a half has been full of talk about virtual reality, metaverses, and the like. But what few have realized is that the metaverse isn’t about wearing goggles—it’s about seamless collaboration with friends and coworkers. Use of content about AR and VR dropped 25% because people have missed the real story: we don’t need 3D goggles; we need tools for collaboration. And, as with low-code, collaboration tools are both something to design with and something that needs to be designed. We’re on the edge of a new way to look at the world.

Use of content about information architecture was up 16%, recovering from its decline from 2021 to 2022. The need to present information well, to design the environments in which we consume information online, has never been more important. Every day, there’s more information to absorb and to navigate—and while artificial intelligence will no doubt help with that navigation, AI is as much a design problem as a design solution. (Though it’s a “good problem” to have.) Designing and building for accessibility is clearly related to information architecture, and it’s good to see more engagement with that content (up 47%). It’s been a long time coming, and while there’s still a long way to go, accessibility is being taken more seriously now than in the past. Websites that are designed to be usable by people with impairments aren’t yet the rule, but they’re no longer exceptions.

Professional DevelopmentAlmost everyone involved with software starts as a programmer. But that’s rarely where they end. At some point in their career, they are asked to write a specification, lead a team, manage a group, or maybe even found a company or serve as an executive in an existing company.

O’Reilly is the last company to believe that software developers are neck-bearded geeks who want nothing more than to live in a cave and type on their terminals. We’ve spent most of our history fighting against that stereotype. Nevertheless, going beyond software development is a frequent source of anxiety. That’s no doubt true for anyone stepping outside their comfort zone in just about any field, whether it’s accounting, law, medicine, or something else. But at some point in your career, you have to do something that you aren’t prepared to do. And, honestly, the best leaders are usually the ones who have some anxiety, not the ones whose reaction is “I was born to be a leader.”

Figure 12. Professional developmentFor the past few years, our audience has been interested in professional growth that goes beyond just writing software or building models for AI and ML. Project management is up 13%; the ability to manage large projects is clearly seen as an asset for employees who are looking for their next promotion (or, in some cases, their next job). Whatever their goals might be, anyone looking for a promotion or a new job—or even just solidifying their hold on their current job—would be well served by improving their communications skills (up 23%). Professional development (up 22%) is a catch-all topic that appears to be responding to the same needs. What’s driving this? 2023 began and ended with a lot of news about layoffs. But despite well-publicized layoffs from huge companies that overhired during the pandemic, there’s little evidence that the industry as a whole has suffered. People who are laid off seem to be snapped up quickly by new employers. Nevertheless, anxiety is real, and the emphasis we’re seeing on professional development (and specifically, communications and project management skills) is partially a result of that anxiety. Another part of the story is no doubt the way AI is changing the workplace. If generative AI makes people more efficient, it frees up time for them to do other things, including strategic thinking about product development and leadership. It may finally be time to value “individuals and interactions over processes and tools,” and “customer collaboration over contract negotiation,” as the Agile Manifesto claims. Doing so will require a certain amount of reeducation, focusing on areas like communications, interpersonal skills, and strategic thinking.

Product management, the discipline of managing a product’s lifecycle from the initial idea through development and release to the market, is also a desirable skill. So why is it only up 2.8% and not 20% like project management? Product management is a newer position in most companies; it has strong ties to marketing and sales, and as far as fear of layoffs is concerned (whether real or media driven), product management positions may be perceived as more vulnerable.

A look at the bottom of the chart shows that usage of content that teaches critical thinking grew 39%. That could be in part a consequence of ChatGPT and the explosion in artificial intelligence. Everyone knows that AI systems make mistakes, and almost every article that discusses these mistakes talks about the need for critical thinking to analyze AI’s output and find errors. Is that the cause? Or is the desire for better critical thinking skills just another aspect of professional growth?

A Strange Year?Back at the start, I said this was a strange year. As much as we like to talk about the speed at which technology moves, reality usually doesn’t move that fast. When did we first start talking about data? Tim O’Reilly said “Data is the next Intel Inside” in 2005, almost 20 years ago. Kubernetes has been around for a decade, and that’s not counting its prehistory as Google’s Borg. Java was introduced in 1995, almost 30 years ago, and that’s not counting its set-top box prehistory as Oak and Green. C++ first appeared in 1985. Artificial intelligence has a prehistory as long as computing itself. When did AI emerge from its wintry cave to dominate the data science landscape? 2016 or 2017, when we were amazed by programs that could sort images into dogs and cats? Sure, Java has changed a lot; so has what we do with data. Still, there’s more continuity than disruption.

This year was one of the few years that could genuinely be called disruptive. Generative AI will change this industry in important ways. Programmers won’t become obsolete, but programming as we know it might. Programming will have more to do with understanding problems and designing good solutions than specifying, step-by-step, what a computer needs to do. We’re not there yet, but we can certainly imagine a day when a human language description leads reliably to working code, when “Do what I meant, not what I said” ceases to be the programmer’s curse. That change has already begun, with tools like GitHub Copilot. But to thrive in that new industry, programmers will need to know more about architecture, more about design, more about human relations—and we’re only starting to see that in our data, primarily for topics like product management and communications skills. And perhaps that’s the definition of “disruptive”: when our systems and our expectations change faster than our ability to keep up. I’m not worried about programmers “losing their jobs to an AI,” and I really don’t see that concern among the many programmers I talk to. But whatever profession you’re in, you will lose out if you don’t keep up. That isn’t kind or humane; that’s capitalism. And perhaps I should have used ChatGPT to write this report.3

Jerry Lee Lewis might have said “There’s a whole lotta disruption goin’ on.” But despite all this disruption, much of the industry remains unchanged. People seem to have tired of the terms DevOps and SRE, but so it goes: the half-life of a buzzword is inevitably short, and these have been extraordinarily long-lived. The problems these buzzwords represent haven’t gone away. Although we aren’t yet collecting the data (and don’t yet have enough content for which to collect data), developer platforms, self-service deployment, and platform engineering look like the next step in the evolution of IT operations. Will AI play a role in platform engineering? We’d be surprised if it didn’t.

Movement to the cloud continues. While we’ve heard talk of cloud “repatriation,” we see no evidence that it’s happening. We do see evidence that organizations realize that the cloud is naturally hybrid and that focusing on a single cloud provider is short-sighted. There’s also evidence that organizations are now paying more than lip service to security, particularly cloud security. That’s a very good sign, especially after many years in which companies approached security by hoping nothing bad would happen. As many chess grandmasters have said, “Hope is never a good strategy.”

In the coming year, AI’s disruption will continue to play out. What consequences will it have for programming? How will jobs (and job prospects) change? How will IT adapt to the challenge of managing AI applications? Will they rely on AI-as-a-service providers like OpenAI, Azure, and Google, or will they build on open source models, which will probably run in the cloud? What new vulnerabilities will AI applications introduce into the security landscape? Will we see new architectural patterns and styles? Will AI tools for software architecture and design help developers grapple with the difficulties of microservices, or will it just create confusion?

In 2024, we’ll face all of these questions. Perhaps we’ll start to see answers. One thing is clear: it’s going to be an exciting year.

Footnotes1. Google Trends suggests that we may be seeing a resurgence in ChatGPT searches. Meanwhile, searches for ChatGPT on our platform appear to have bottomed out in October, with a very slight increase in November. This discrepancy aligns well with the difference between our platform and Google’s. If you want to use ChatGPT to write a term paper, are you going to search Google or O’Reilly? 2. Phillip Carter’s article, “All the Hard Stuff Nobody Talks About when Building Products with LLMs,” is worth reading. While it isn’t specifically about design, almost everything he discusses is something designers should think about. 3. I didn’t. Not even for data analysis.

View Details

ChatGPT was released just over a year ago (at the end of November 2022), and countless people have already written about their experiences using it in all sorts of settings. (I even contributed my own hot take last year with my O’Reilly Radar article Real-Real-World Programming with ChatGPT.) What more is left to say by now? Well, I bet very few of those people have actually chatted with ChatGPT. And by “chat” I mean the original sense of the word—to hold a back-and-forth verbal conversation with it just like how you would chat with a fellow human being. I recently chatted with ChatGPT, and I want to use that experience to reflect on the usability of voice interfaces for AI tools based on Large Language Models. I’m personally interested in this topic since I am a professor who researches human-computer interaction, user experience design, and cognitive science, so AI voice interfaces are fascinating to me.

Here’s what I did: In December 2023 I installed the official ChatGPT iOS app from OpenAI on my iPhone and used its voice input mode to hold several hour-long conversations with it while driving long-distance on California highways. I wore standard Apple earbuds with a built-in mic and talked with ChatGPT just like how I would be talking to someone on the phone while driving. These long solo drives were the perfect opportunity to test out ChatGPT’s voice feature because I couldn’t interact with the app using my hands for safety reasons.

I had a very clear use case in mind: I wanted a conversation partner to keep me awake and alert while driving long-distance by myself. I’ve found that listening to music or podcasts doesn’t keep me alert when I’m tired because it’s such a passive experience—but what does keep me awake is having someone to talk to, either in the car or remotely on the phone. Could ChatGPT replace a human conversation partner in this role?

The Good: ChatGPT Made Personalized Podcasts to Keep Me Engaged While DrivingTo not bury the lede, it turns out that it did a remarkable job! As I was driving I was able to engage in several hour-long conversations with ChatGPT that ended only because I had to take a rest stop or hit the usage limit for GPT-4. (I pay for a ChatGPT Plus subscription so I can use the most advanced GPT-4 model, but that comes with a usage limit that I usually hit after about an hour.)

The best way to describe my experience is (borrowing a wonderful term my friend coined) that it felt like listening to a personalized podcast. Since ChatGPT did most of the talking, it was a mostly passive listening experience on my part except for times when I wanted to ask follow-up questions or direct it to change topics. Critically, this meant I could still focus most of my attention on driving safely with a level of distraction on par with listening to a podcast. But it kept me more alert than a regular podcast since I could actively direct the flow of the conversation.

For a concrete example of what such a personalized podcast felt like, I started one conversation by straight-up asking ChatGPT to keep me awake while I was driving in Southern California from Los Angeles to San Diego. So it started by making small talk about road trips in general and asking me about various California landmarks that I’ve visited, culminating in asking me more about San Diego (where I live). When it asked me what places I liked visiting the most here, I mentioned the San Diego Zoo and it started telling me a bit about what makes this particular zoo notable. It mentioned the concept of “naturalistic enclosures”—a term I had not heard before—so I asked it to elaborate on what this meant. ChatGPT’s explanation of this concept got me interested in the history of zoos, especially the progression from keeping animals in cages to today’s cageless naturalistic enclosures, which aim to be better for animal welfare. During that segment it mentioned the term “menagerie” in passing, which I had not heard of in that context before, so I asked it to elaborate more. It then went back farther in history to describe how a menagerie refers to the phenomenon of ancient rulers keeping exotic animals for display without as much regard for the animals’ well-being. Listening to that made me realize that I had actually heard the term menagerie in reference to a Star Trek episode of some sort, but I forgot which one, so I asked ChatGPT to jog my memory. It turns out that “The Menagerie” was a very famous episode of the original Star Trek television series, so after chatting about that episode and other famous Star Trek episodes for a bit, we got onto the topic of why that show was canceled after only three seasons but later found a much larger audience in syndication (i.e., reruns). That in turn got me curious about the concept of syndication in the television business, so ChatGPT dived more into this topic. A few more conversational twists and turns later, then I suddenly realized that the hour had flown by and it was time to pull over for a bathroom break. Success!

Now, I don’t expect you to care at all about the details of the conversation I just described since it wasn’t your conversation—it was mine! But I certainly cared about it at the time since I was genuinely curious to learn more about the topics that ChatGPT mentioned, often offhand in the midst of telling me about something else. It felt a bit like diving down a Wikipedia rabbit hole of following related links, where each follow-up question I asked led it down another meandering path. It was perfect for keeping me from getting bored and sleepy during my long drive.

ChatGPT isn’t just good at this sort of superficial “personalized podcast about Wikipedia-level trivia” … it could also engage me in a more substantive conversation about a task I actually needed help with at the moment. In another hour-long car chat, I prompted ChatGPT to help me design a method to organize my huge collection of almost 30 years’ worth of personal and work-related files for backup. I’ve been diligent about data backup throughout my life, but my files are fragmented amongst different media over the years—burning CDs and DVDs back in the day, several generations of external hard drives (that are in various states of decay), university servers, Dropbox, and other cloud services. For years I had an aspirational goal of unifying all of my backups into one central directory tree, akin to the concept of a monorepo in software development. I’ve recently been brainstorming ideas for how to design such a system and how to deal with the practical challenges of scaling and maintenance. So I figured that ChatGPT could help me brainstorm during one of my long drives. Again it did a good job at engaging me in this bespoke conversation, and the hour flew by before I had to take a rest stop. I won’t bore you with details of what we discussed, but it felt like talking with an expert in data management who was giving me advice about how to deal with my particular challenge.

Intermission: Why It Feels Kind of MagicalSkeptical readers may be thinking at this point, “What’s the big deal, it’s just ChatGPT under the hood. I can already do all this from my computer by typing into the ChatGPT text box!” Although that’s technically true, there’s something magical about being able to do this all hands-free via voice. If you don’t believe me, just try it for an hour. My folk theory is that speaking and listening are hardwired into our brain’s innate language circuitry, but writing and reading are learned skills (i.e., “software” rather than “hardware” in our brains). And that’s why it feels more magical to hold a verbal conversation with an AI versus having the exact same conversation in a text box on a screen. If the AI is good enough, then it almost feels like you’re talking to a real person … at certain times when I was getting deep into a back-and-forth conversation I nearly forgot I was talking to a machine. However, that illusion broke in several ways …

The Not-So-Good: Usability Limitations of the ChatGPT Voice InterfaceDespite my positive experiences with ChatGPT’s voice mode, it still didn’t live up to the gold standard of feeling like I was talking with a fellow human being. That’s okay, though, since this is an incredibly high bar! Here are some of the ways it fell short.

• Must speak entire request all at once: Most notably, it felt unnatural to have to speak my entire request all at once without pausing. Whenever I paused for too long, ChatGPT would interpret what I said so far as my request and start processing it. As an analogy, when typing a request in a text chat, you can hit the Enter or Send buttons … imagine how weird it would be if ChatGPT started answering you the very moment you stopped typing for one second! Note that in human conversations, especially face-to-face, we use visual cues to tell whether our conversation partner is done talking or whether they are pausing a bit to think about the next thing to say. Even over the phone, we can tell by vocal inflections whether they are temporarily paused and want to keep talking, or whether they are done with their turn and ready for us to respond. Since ChatGPT can’t do any of that (yet!) I often had to think hard about what I wanted to say and then say it all at once without pausing. This was fine for simple requests like “Tell me more about naturalistic enclosures in zoos,” but for more complex requests like describing some facet of my data backup setup, it was painful to have to blurt out as much as I could without pausing. Even more annoyingly, I would sometimes make mistakes when talking so much all at once without pausing. Ideally the app would do a better job at detecting pauses in human speech, taking both context and vocal intonations into account. An easier hack would be to have a voice command like “DONE” or “OVER” (like when people use walkie-talkies) to signal that I am done talking; however, this would also feel unnatural for casual users.
• Unpredictable wait times: Wait times (latency) for ChatGPT’s responses are unpredictable, and there aren’t audio cues to help me establish an expectation for how long I need to wait before it responds. There’s a click sound when it starts processing my request, but then I may need to wait a few seconds in silence before hearing a response … maybe it’s only one second or maybe it’s five seconds. That said, if I ask it to browse the web, then it plays a continuous waiting sound; web browsing takes longer, maybe 10 to 20 seconds, but at least I get to hear a “waiting” sound. (I don’t mind ChatGPT taking longer here since a human would also take more time to browse the web. However, web browsing is annoying when I don’t explicitly ask it to browse. Oftentimes I want a fast answer but something I say triggers a browse without me intending to.) In contrast, when speaking with a human face-to-face, I can use visual cues to tell whether the other person is deep in thought or when they will likely respond; and even over the phone the other person may say “ummm” or “hold on one sec, lemme think” or “ok let me look this up on the web, hang tight for a while …” if they need more time to think through their response. However, since I don’t get any of these verbal cues from ChatGPT, unpredictable wait times break the illusion of talking to a person.
• Cannot interrupt while it is speaking: I always had to wait for ChatGPT to completely finish talking before it would listen to my next request. And since I never know ahead of time how long it planned to talk for during a particular turn (i.e., how many words its LLM-generated response is), when I wanted to say something midway it was aggravating to have to wait. I later saw that I could actually interrupt it by tapping on the app on my phone screen, but since I was driving and hands-free, I couldn’t safely do that. Also, that seems like a cumbersome interaction; I should be able to just talk when I want to, even when it is talking. This limitation made the conversation feel like we were using a walkie-talkie where only one party can talk at once. And it’s not just me—this concept of overlapping speech is widely studied in linguistics and communication research. Humans naturally talk over one another for various reasons, so not being able to do this with ChatGPT made our conversation feel less fluid. Even implementing a feature like a voice command for interruption would be great, like maybe if I say “pause” or “wait” then it could stop and await my request.
• Speech recognition errors: ChatGPT’s speech recognition system (presumably based on OpenAI’s open source Whisper model) is very good, but it does at times misinterpret what I’m saying. What’s stranger is that sometimes it thinks I said something when I didn’t, maybe because it picked up on background rumbles in my car. Several times I wouldn’t be saying anything and suddenly it responds out of the blue; and when I check the written transcript later, it thinks that I said something like “Thank you for watching!” (which I never said). At other times it tries to prematurely end the conversation even though I’m not done, maybe because it mistakenly detected that I said something along the lines of “Thanks …” without any follow-up. Misrecognizing words is forgivable, but I feel that it shouldn’t ever interpret background sounds as words. Of course, if there were other people in the car with me and either they talked or I was talking to them, then I could also understand how ChatGPT would mistakenly interpret that as being a request for it; always-listening home assistants like Alexa have had this issue for years. A more advanced AI would learn to filter out both other people’s voices and also infer when I was speaking with someone else and not it. For instance, when it detects that my sentence is way off topic, maybe that means I’m speaking with someone else in the car; it could at least ask me “Were you talking to me just now?” when it is uncertain. More generally, the idea of explicitly asking me for clarification when it is uncertain would go a long way toward making these interactions feel more human; that’s what I (a representative human!) would do if I were on a noisy phone connection with someone and didn’t hear them clearly.
• Overly agreeable artificial tone: Lastly, it’s still ChatGPT under the hood, so all the regular limitations of ChatGPT apply here. Most notably, ChatGPT is tuned to be overly friendly and overly agreeable (sounding like a customer service agent) so it will simply go along with whatever you assert. Thus, by default it will not be good at pushing back on you or challenging your thinking in any meaningful ways, just like how you wouldn’t expect a customer service agent to challenge what you say. Moreover, the overly friendly tone of its responses could come off as insincere and almost sarcastic at times, even though that wasn’t the designers’ intent. Relatedly, it had a tendency to ask me superficial questions after it responds, which sound mildly condescending and break the flow of our chat, like, “Sooo, what do YOU think about the San Diego Zoo? What’s YOUR favorite part of the zoo?!?” … when a normal human wouldn’t break the conversational flow so awkwardly like that. Lastly, ChatGPT is trained on data on the public internet (and can also browse the web to get more updated web contents), so it won’t do as well if you’re asking about things that haven’t been discussed much online.

To summarize the above limitations, chatting with ChatGPT on my phone felt like using a walkie-talkie over a noisy channel to talk to an overly agreeable but socially unaware customer service agent who has extensive knowledge about the contents of the public internet.

Parting Thoughts: Cautiously Optimistic About the FutureDespite these limitations, I’m excited to see what’s in store for future voice interfaces to LLM-based AI tools like ChatGPT. My early experiences of talking with ChatGPT while driving gave me a glimpse into what many of us have seen growing up in sci-fi shows such as Star Trek, where people can talk to an omnipresent computer to ask questions, hold conversations, or issue commands. Hands-free operation isn’t useful only while driving—it can make computing truly ubiquitous by letting us seamlessly interact with computation while we are in the midst of doing housework, cooking, or childcare; and it can make computing more accessible to broader groups of people, such as those with mobility impairments.

We still have a long way to go, though. Right now the ChatGPT iPhone app isn’t hooked up to external tools beside a basic web browser, but with the recently announced GPT store (and likely upcoming LLM app stores from other companies) it will soon be possible to hook up LLMs to a variety of tools that can manage our emails, shopping lists, personal finances, home automation, and more. Recent research has started exploring these ideas by connecting ChatGPT to home assistants such as Amazon Alexa (2023 arXiv paper PDF). Another promising line of work is better context awareness: for instance, Meta and Ray-Ban recently announced new Smart Glasses which allow users to chat with an AI assistant that can see what they are seeing (review from The Verge). In my driving scenario, you could imagine wearing these glasses and having the AI act more like a passenger sitting alongside you in the car seeing what you see rather than someone on the other end of a phone call. Critically, a passenger can pause the conversation and tell you to watch the road more carefully if they see a possible danger ahead; a future AI powered by such smart glasses may be able to do the same thing. Alternatively, cars are now starting to directly embed AI into entertainment systems (e.g., Volkswagen announcement at CES 2024), so future iterations could integrate cameras and 3D tracking to complement LLMs. One could also imagine smartglasses-based multimodal interactions where you point to objects in any physical environment and start conversations with the AI assistant about your surroundings (check out this MKBHD YouTube Short showing AI chat with smart glasses).

Of course, these increasingly intense levels of AI interaction and automation come with risks, such as user overreliance, unintended command execution, mental or physical health hazards, and security/privacy violations. Thus, it will be important to design ways to both manage those risks and educate users about how to safely operate these increasingly powerful systems. Thank you very much for reading. Sooo, what do YOU think about ChatGPT’s voice mode?!? What are YOUR favorite and least favorite parts?

View Details

Kevlin Henney and I recently discussed whether automated code generation, using some future version of GitHub Copilot or the like, could ever replace higher-level languages. Specifically, could ChatGPT N (for large N) quit the game of generating code in a high-level language like Python and produce executable machine code directly, like compilers do today?

It’s not really an academic question. As coding assistants become more accurate, it seems likely to assume that they will eventually stop being “assistants” and take over the job of writing code. That will be a big change for professional programmers—though writing code is a small part of what programmers actually do. To some extent, it’s happening now: ChatGPT 4’s “Advanced Data Analysis” can generate code in Python, run it in a sandbox, collect error messages, and try to debug it. Google’s Bard has similar capabilities. Python is an interpreted language, so there’s no machine code, but there’s no reason this loop couldn’t incorporate a C or C++ compiler.

This kind of change has happened before: in the early days of computing, programmers “wrote” programs by plugging in wires, then by toggling in binary numbers, then by writing assembly language code, and finally (in the late 1950s) using early programming languages like COBOL (1959) and FORTRAN (1957). To people who programmed using circuit diagrams and switches, these early languages looked as radical as programming with generative AI looks today. COBOL was—literally—an early attempt to make programming as simple as writing English.

Kevlin made the point that higher-level languages are a “repository of determinism” that we can’t do without—at least, not yet. While a “repository of determinism” sounds a bit evil (feel free to come up with your own name), it’s important to understand why it is needed. At almost every stage of programming history, there has been a repository of determinism. When programmers wrote in assembly language, they had to look at the binary 1s and 0s to see exactly what the computer was doing. When programmers wrote in FORTRAN (or, for that matter, C), the repository of determinism moved higher: the source code expressed what programmers wanted and it was up to the compiler to deliver the correct machine instructions. However, the status of this repository was still shaky. Early compilers were not as reliable as we’ve come to expect. They had bugs, particularly if they were optimizing your code (were optimizing compilers a forerunner of AI?). Portability was problematic at best: every vendor had its own compiler, with its own quirks and its own extensions. Assembly was still the “court of last resort” for determining why your program didn’t work. The repository of determinism was only effective for a single vendor, computer, and operating system.1 The need to make higher-level languages deterministic across computing platforms drove the development of language standards and specifications.

These days, very few people need to know assembler. You need to know assembler for a few tricky situations when writing device drivers or to work with some dark corners of the operating system kernel, and that’s about it. But while the way we program has changed, the structure of programming hasn’t. Especially with tools like ChatGPT and Bard, we still need a repository of determinism, but that repository is no longer assembly language. With C or Python, you can read a program and understand exactly what it does. If the program behaves in unexpected ways, it’s much more likely that you’ve misunderstood some corner of the language’s specification than that the C compiler or Python interpreter got it wrong. And that’s important: that’s what allows us to debug successfully. The source code tells us exactly what the computer is doing, at a reasonable layer of abstraction. If it’s not doing what we want, we can analyze the code and correct it. That may require rereading Kernighan and Ritchie, but it’s a tractable, well-understood problem. We no longer have to look at the machine language—and that’s a very good thing, because with instruction reordering, speculative execution, and long pipelines, understanding a program at the machine level is a lot more difficult than it was in the 1960s and 1970s. We need that layer of abstraction. But that abstraction layer must also be deterministic. It must be completely predictable. It must behave the same way every time you compile and run the program.

Why do we need the abstraction layer to be deterministic? Because we need a reliable statement of exactly what the software does. All of computing, including AI, rests on the ability of computers to do something reliably and repeatedly, millions, billions, or even trillions of times. If you don’t know exactly what the software does—or if it might do something different the next time you compile it—you can’t build a business around it. You certainly can’t maintain it, extend it, or add new features if it changes whenever you touch it, nor can you debug it.

Automated code generation doesn’t yet have the kind of reliability we expect from traditional programming; Simon Willison calls this “vibes-based development.” We still rely on humans to test and fix the errors. More to the point: you’re likely to generate code many times en route to a solution; you’re not likely to take the results of your first prompt and jump directly into debugging any more than you’re likely to write a complex program in Python and get it right the first time. Writing prompts for any significant software system isn’t trivial; the prompts can be very lengthy, and it takes several tries to get them right. With the current models, every time you generate code, you’re likely to get something different. (Bard even gives you several alternatives to choose from.) The process isn’t repeatable. How do you understand what the program is doing if it’s a different program each time you generate and test it? How do you know whether you’re progressing towards a solution if the next version of the program may be completely different from the previous?

It’s tempting to think that this variation is controllable by setting a variable like GPT-4’s “temperature” to 0; “temperature” controls the amount of variation (or originality, or unpredictability) between responses. But that doesn’t solve the problem. Temperature only works within limits, and one of those limits is that the prompt must remain constant. Change the prompt to help the AI generate correct or well-designed code, and you’re outside of those limits. Another limit is that the model itself can’t change—but models change all the time, and those changes aren’t under the programmer’s control. All models are eventually updated, and there’s no guarantee that the code produced will stay the same across updates to the model. An updated model is likely to produce completely different source code. That source code will need to be understood (and debugged) on its own terms.

So the natural language prompt can’t be the repository of determinism. This doesn’t mean that AI-generated code isn’t useful; it can provide a good starting point to work from. But at some point, programmers need to be able to reproduce and reason about bugs: that’s the point at which you need repeatability and can’t tolerate surprises. Also at that point, programmers will have to refrain from regenerating the high-level code from the natural language prompt. The AI is effectively creating a first draft, and that may (or may not) save you effort compared to starting from a blank screen. Adding features to go from version 1.0 to 2.0 raises a similar problem. Even the largest context windows can’t hold an entire software system, so it’s necessary to work one source file at a time—exactly the way we work now, but again, with the source code as the repository of determinism. Furthermore, it’s difficult to tell a language model what it’s allowed to change and what should remain untouched: “modify this loop only, but not the rest of the file” may or may not work.

This argument doesn’t apply to coding assistants like GitHub Copilot. Copilot is aptly named: it’s an assistant to the pilot, not the pilot. You can tell it precisely what you want done, and where. When you use ChatGPT or Bard to write code, you’re not the pilot or the copilot; you’re the passenger. You can tell a pilot to fly you to New York, but from then on, the pilot is in control.

Will generative AI ever be good enough to skip the high-level languages and generate machine code? Can a prompt replace code in a high-level language? After all, we’re already seeing a tools ecosystem that has prompt repositories, no doubt with version control. It’s possible that generative AI will eventually be able to replace programming languages for day-to-day scripting (“Generate a graph from two columns of this spreadsheet”). But for larger programming projects, keep in mind that part of human language’s value is its ambiguity, and a programming language is valuable precisely because it isn’t ambiguous. As generative AI penetrates further into programming, we will undoubtedly see stylized dialects of human languages that have less ambiguous semantics; those dialects may even become standardized and documented. But “stylized dialects with less ambiguous semantics” is really just a fancy name for prompt engineering, and if you want precise control over the results, prompt engineering isn’t as simple as it seems. We still need a repository of determinism, a layer in the programming stack where there are no surprises, a layer that provides the definitive word on what the computer will do when the code executes. Generative AI isn’t up to that task. At least, not yet.

Footnote1. If you were in the computing industry in the 1980s, you may remember the need to “reproduce the behavior of VAX/VMS FORTRAN bug for bug.”

View Details

A few weeks ago, I saw a tweet that said “Writing code isn’t the problem. Controlling complexity is.” I wish I could remember who said that; I will be quoting it a lot in the future. That statement nicely summarizes what makes software development difficult. It’s not just memorizing the syntactic details of some programming […]

View Details

While the AI group is still the largest, it’s notable that Programming, Web, and Security are all larger than they’ve been in recent months. One reason is certainly that we’re pushing AI news into other categories as appropriate. But I also think that it’s harder to impress with AI than it used to be. AI discussions have been much more about regulation and intellectual property—which makes me wonder whether legislation should be a separate category.

That notwithstanding, it’s important that OpenAI is now allowing API users to fine-tune their GPT-4 apps. It’s as-a-service, of course. And RISC-V finally appears to be getting some serious adoption. Could it compete with Atom and Intel? We shall see.

AI* OpenAI has announced ChatGPT Enterprise, a version of ChatGPT that targets enterprise customers. ChatGPT Enterprise offers improved security, a promise that they won’t train on your conversations, single sign on, an admin console, a larger 32K context, higher performance, and the elimination of usage caps. * Facebook/Meta has released Code LLaMA, a version of their LLaMA 2 model that has been specialized for writing code. It can be used for code generation or completion. Its context window is 100,000 tokens, allowing Code LLaMA to be more accurate on larger programs. * OpenAI has announced that API users can now fine-tune GPT-3.5 for their own applications. Fine-tuning for GPT-4 will come later. To preserve safety, tuning data is passed through OpenAI’s moderation filter. * txtai is an open source embeddings database. It is a vector database that has been designed specifically to work with natural language problems. * TextFX is a set of tools that use Google’s PaLM 2 model to play with language. It doesn’t answer questions or write poems; it allows users to see the possibilities in words as an aid to their own creativity. * A US judge has ruled that an AI system cannot copyright a work. In this case, the AI itself—not the human user—was to hold the copyright. This ruling is in line with the Copyright Office’s guidance: giving prompts to a generative algorithm isn’t sufficient to create a copyrightable work. * Despite an error rate of roughly 50% for ChatGPT, a study shows that users prefer ChatGPT’s answers to programming questions over answers from StackOverflow. ChatGPT’s complete, articulate, and polite answers appear to be the cause of this preference. * AI was on the agenda at DefCon and, while results of a red teaming competition won’t be released for some months, it’s clear that security remains an afterthought, and that attacking the current AI models is extremely easy. * Emotion recognition is difficult, if not impossible. It is not clear that there are any credible use cases for it. AI systems are particularly bad at it. But companies are building products. * Watermarking has been proposed as a technique for identifying whether content was generated by AI, but it’s not a panacea. Here are some questions to help evaluate whether watermarks are useful in any given situation. * Zoom and Grammarly have both issued new license agreements that allow them to use data collected from users to train AI. Zoom has backed down after customer backlash, but that begs the question: Will other applications follow? * Using large language models for work or play is one thing, but how do you put one into production? 7 Frameworks for Serving LLMs surveys some tools for deploying language models. * Simon Willison provides instructions for running LLaMA 2 on a Mac. He also provides slides and a well-edited transcript of his talk about LLMs at North Bay Python. * PhotoGuard is a tool for protecting photos and other images from manipulation by AI systems. It adds data to the image in ways that aren’t detectable by humans, but that introduce noticeable distortions when the image is modified. * C2PA is a cryptographic protocol for attesting to the provenance of electronic documents. It could be used for specifying whether documents are generated by AI. * Google’s DeepMind has built a vision-language-action model called RT-2 (Robotic Transformer 2) that combines vision and language with the ability to control a robot. It learns both from web data (images and text) and robotic data (interactions with physical objects).

Programming* Maccarone is an extension to VSCode that allows you to “delegate” blocks of Python code to AI (GPT-4). The portions of the code that are under AI control are automatically updated as needed when the surrounding code is changed. * Microsoft is adding Python as a scripting language for Excel formulas. Python code executes in an Azure container that includes some commonly used libraries, including Matplotlib and Pandas. * Many companies are building platform engineering teams as a means of making software developers more effective. Here are some ideas about getting started with platform engineering. * A Google study of its in-house Rust use supports the claim that Rust makes it easier to produce high-quality code. The study also busts a number of myths about the language. It isn’t as hard to learn as most people think (then again, this is a Google study). * deno_python is a Javascript module that allows integration between Javascript (running on Deno) and Python, allowing Javascript programmers to call important Python libraries and call Python functions. * The Python Steering Council has announced that it will make the Global Interpreter Lock (GIL) optional in a future version of Python. Python’s GIL has long been a barrier to effective multi-threaded computing. The change will be backwards-compatible.

Web* Google’s controversial Web Environment Integrity proposal provides a way for web servers to cryptographically authenticate the browser software making a request. WEI could potentially reduce online fraud, but it also presents some significant privacy risks. * Trafilatura is a new tool for web scraping that has been designed with quantitative research (for example, assembling training data for language models). It can extract text and metadata from HTML, and generate output in a number of formats. * Astro is yet another open source web framework that’s designed for high performance and ease of development. * While the “browser wars” are far behind us, it is still difficult for developers to write code that works correctly on all browsers. Baseline is a project of the W3C’s WebDX Community Group that specifies which features web developers can rely on in the most widely used browsers. * How Large Language Models Assisted a Website Makeover raises some important questions: When do you stop using ChatGPT and finish the job yourself? When does your own ability start to atrophy? * Remember Flash? It has a museum… And Flash games will run in a modern browser using Ruffle, a Flash Player emulator that is written in WebAssembly.

Security* Proof-of-work makes it to the Tor network. It is used as a defense against denial of service attacks. PoW is disabled most of the time, but when traffic seems unusually high, it can switch on, forcing users to “prove” their humanness (actually, their willingness to perform work). * A retrospective on this year’s MoveIT attack draws some important conclusions about protecting your assets. Mapping the supply chain, third party risk management, zero trust, and continuous penetration testing are all important parts of a security plan. * Bitwarden has released an open source end-to-end encrypted secrets manager. The secrets manager allows safe distribution of API keys, certificates and other sensitive data. * The US Government has announced the AI Cybersecurity Challenge (AIxCC). AIxCC is a two year competition to build AI systems that can secure critical software. There’s $18.5 million in prizes, plus the possibility of DARPA funding for up to seven companies. * OSC&R is the Open Source Supply Chain Attack Reference, a new project that catalogs and describes techniques used to attack software supply chains. It is modeled on MITRE’s ATT&CK framework. * The Lapsus$ group has become one of the most effective threat actors, despite being relatively unsophisticated. They rely on persistence, clever social engineering, and analyzing weak points in an organization’s security posture rather than compromising infrastructure. * The NSA has issued a report that gives guidance on how to protect systems against memory safety bugs. * Bruce Schneier has an important take on the long-term consequences of the SolarWinds attack. Those consequences include the theft of an Azure customer account signing key that in turn has been used by attackers to access US government email accounts. * A new generation of ransomware attacks is targeting IT professionals via fake advertisements for IT tools. While IT professionals are (presumably) more wary and aware than other users, they are also high-value targets.

Hardware* Parmesan cheese producers are experimenting with adding microchips to the cheese rind to authenticate genuine cheese. * Adoption of RISC-V, a royalty-free open source instruction set architecture for microprocessors, has been increasing. Could it displace ARM? * Speculative execution bugs have been discovered for recent Intel (“Downfall”) and AMD (“Inception”) processors. Patches for Linux have been released.

Operations* Since Hashicorp has moved Terraform from the open source Mozilla Public License to the Business Source License, the OpenTF foundation has been created, and has forked the Terraform project to create OpenTF. * There’s a rise in the abuse of Cloudflare tunnels to create persistent malicious communications channels. * Amazon has announced that they will begin charging for public IPv4 addresses. There are already some charges for Elastic IP addresses. Users won’t be charged for IP addresses they already own. Among other things, this change is intended to accelerate IPv6 adoption.

Quantum Computing* Peter Shor, inventor of the quantum algorithm for factoring prime numbers (which in turn could be used to break most modern cryptography that isn’t quantum-resistant), has published the lecture notes from the course on quantum computing that he teaches at MIT. * A Honeywell quantum computer has been used to find a material that can improve solar cell efficiency. It’s likely that the first applications of quantum computing will involve simulating quantum phenomena rather than pure computation.

Cryptocurrency* If you’re interested in iris-scanning WorldCoin, a cryptographer analyzes the privacy promises made by their system. He remains skeptical, but came away less unimpressed than he expected to be. * Paypal has introduced a stablecoin that claims to be fully backed by US dollars.

Biology* LabGenius is a company that combines synthetic biology, artificial intelligence, and robotics to design and build new human antibodies that are effective against hard-to-treat diseases.

View Details

To follow up on our previous survey about low-code and no-code tools, we decided to run another short survey about tools specifically for software developers—including, but not limited to, GitHub Copilot and ChatGPT. We’re interested in how “developer enablement” tools of all sorts are changing the workplace. Our survey 1 showed that while these tools […]

View Details

I’m sure that nobody will be surprised that the number of searches for ChatGPT on the O’Reilly learning platform skyrocketed after its release in November, 2022. It might be a surprise how quickly it got to the top of our charts: it peaked in May as the 6th most common search query. Then it dropped […]

View Details

Artificial Intelligence continues to dominate the news. In the past month, we’ve seen a number of major updates to language models: Claude 2, with its 100,000 token context limit; LLaMA 2, with (relatively) liberal restrictions on use; and Stable Diffusion XL, a significantly more capable version of Stable Diffusion. Does Claude 2’s huge context really […]

View Details

If you’re reading this, chances are you’ve played around with using AI tools like ChatGPT or GitHub Copilot to write code for you. Or even if you haven’t yet, then you’ve at least heard about these tools in your newsfeed over the past year. So far I’ve read a gazillion blog posts about people’s experiences […]

View Details

Imagine for a minute that you’re a programming instructor who’s spent many hours making creative homework problems to introduce your students to the world of programming. One day, a colleague tells you about an AI tool called ChatGPT. To your surprise (and alarm), when you give it your homework problems, it solves most of them […]

View Details

There’s a lot of angst about software developers “losing their jobs” to AI, being replaced by a more intelligent version of ChatGPT, GitHub’s Copilot, Google’s Codey, or something similar. Matt Welsh has been talking and writing about the end of programming as such. He’s asking whether large language models eliminate programming as we know it, […]

View Details

A surprising number of the entries for AI are about generative models that don’t generate text or artwork—specifically, they generate human voices or music. Is voice the next frontier for AI? Google’s AudioPaLM, which unites speech recognition, speech synthesis, and language modeling, may show the direction in which AI is heading. There’s also increasing concern about […]

View Details

Does your company plan to release an AI chatbot, similar to OpenAI’s ChatGPT or Google’s Bard? Doing so means giving the general public a freeform text box for interacting with your AI model. That doesn’t sound so bad, right? Here’s the catch: for every one of your users who has read a “Here’s how ChatGPT […]

View Details

Understanding AI’s mysterious “opaque box” is paramount to creating explainable AI. This can be simplified by considering that AI, like all other technology, has a supply chain. Knowing what makes up the supply chain is critical to enforcing the security of the AI system, establishing trust with the consumer of the AI’s output, and protecting […]

View Details

“Mitigating the risk of extinction from A.I. should be a global priority alongside other societal-scale risks, such as pandemics and nuclear war,” according to a statement signed by more than 350 business and technical leaders, including the developers of today’s most important AI platforms. Among the possible risks leading to that outcome is what is […]

View Details

The world changed on November 30, 2022 as surely as it did on August 12, 1908 when the first Model T left the Ford assembly line. That was the date when OpenAI released ChatGPT, the day that AI emerged from research labs into an unsuspecting world. Within two months, ChatGPT had over a hundred million […]

View Details

A few months ago, I wrote about some experiments with prime numbers. I generated a 16-digit non-prime number by multiplying two 8-digit prime numbers, and asked ChatGPT (using GPT -3.5) whether the larger number was prime. It answered correctly that the number was non-prime, but when it told me the number’s prime factors, it was clearly wrong. […]

View Details

AI, and specifically large language models, continue to dominate the news–so much so that it’s no longer a well-defined topic, with clear boundaries. AI has infiltrated programming, security, and virtually every branch of technology. But that’s hardly news. By the time you read this, Apple may have announced their ultra-expensive, ultra-stylish AR headset; that may […]

View Details

It’s hard to ignore the discussion around the Open Letter arguing for a pause in the development of advanced AI systems. Are they dangerous? Will they destroy humanity? Will they condemn all but a few of us to boring, impoverished lives? If these are indeed the dangers we face, pausing AI development for six months is certainly a weak and ineffective preventive.

It’s easier to ignore the voices arguing for the responsible use of AI. Using AI responsibly requires AI to be transparent, fair, and where possible, explainable. Using AI means auditing the outputs of AI systems to ensure that they’re fair; it means documenting the behaviors of AI models and training data sets so that users know how the data was collected and what biases are inherent in that data. It means monitoring systems after they are deployed, updating and tuning them as needed because any model will eventually grow “stale” and start performing badly. It means designing systems that augment and liberate human capabilities, rather than replacing them. It means understanding that humans are accountable for the results of AI systems; “that’s what the computer did” doesn’t cut it.

The most common way to look at this gap is to frame it around the difference between current and long-term problems. That’s certainly correct; the “pause” letter comes from the “Future of Life Institute,” which is much more concerned about establishing colonies on Mars or turning the planet into a pile of paper clips than it is with redlining in real estate or setting bail in criminal cases.

But there’s a more important way to look at the problem, and that’s to realize that we already know how to solve most of those long-term issues. Those solutions all center around paying attention to the short-term issues of justice and fairness. AI systems that are designed to incorporate human values aren’t going to doom humans to unfulfilling lives in favor of a machine. They aren’t going to marginalize human thought or initiative. AI systems that incorporate human values are not going to decide to turn the world into paper clips; frankly, I can’t imagine any “intelligent” system determining that was a good idea. They might refuse to design weapons for biological warfare. And, should we ever be able to get humans to Mars, they will help us build colonies that are fair and just, not colonies dominated by a wealthy kleptocracy, like the ones described in so many of Ursula Leguin’s novels.

Another part of the solution is to take accountability and redress seriously. When a model makes a mistake, there has to be some kind of human accountability. When someone is jailed on the basis of incorrect face recognition, there needs to be a rapid process for detecting the error, releasing the victim, correcting their criminal record, and applying appropriate penalties to those responsible for the model. These penalties should be large enough that they can’t be written off as the cost of doing business. How is that different from a human who makes an incorrect ID? A human isn’t sold to a police department by a for-profit company. “The computer said so” isn’t an adequate response–and if recognizing that means that it isn’t economical to develop some kinds of applications can’t be developed, then perhaps those applications shouldn’t be developed. I’m horrified by articles reporting that police use face detection systems with false positive rates over 90%; and although those reports are five years old, I take little comfort in the possibility that the state of the art has improved. I take even less comfort in the propensity of the humans responsible for these systems to defend their use, even in the face of astounding error rates.

Avoiding bias, prejudice, and hate speech is another critical goal that can be addressed now. But this goal won’t be achieved by somehow purging training data of bias; the result would be systems that make decisions on data that doesn’t reflect any reality. We need to recognize that both our reality and our history are flawed and biased. It will be far more valuable to use AI to detect and correct bias, to train it to make fair decisions in the face of biased data, and to audit its results. Such a system would need to be transparent, so that humans can audit and evaluate its results. Its training data and its design must both be well documented and available to the public. Datasheets for Datasets and Model Cards for Model Reporting, by Timnit Gebru, Margaret Mitchell, and others, are a starting point–but only a starting point. We will have to go much farther to accurately document a model’s behavior.

Building unbiased systems in the face of prejudiced and biased data will only be possible if women and minorities of many kinds, who are so often excluded from software development projects, participate. But building unbiased systems is only a start. People also need to work on countermeasures against AI systems that are designed to attack human rights, and on imagining new kinds of technology and infrastructure to support human well-being. Both of these projects, countermeasures, and new infrastructures, will almost certainly involve designing and building new kinds of AI systems.

I’m suspicious of a rush to regulation, regardless of which side argues for it. I don’t oppose regulation in principle. But you have to be very careful what you wish for. Looking at the legislative bodies in the US, I see very little possibility that regulation would result in anything positive. At the best, we’d get meaningless grandstanding. The worst is all too likely: we’d get laws and regulations that institute performative cruelty against women, racial and ethnic minorities, and LBGTQ people. Do we want to see AI systems that aren’t allowed to discuss slavery because it offends White people? That kind of regulation is already impacting many school districts, and it is naive to think that it won’t impact AI.

I’m also suspicious of the motives behind the “Pause” letter. Is it to give certain bad actors time to build an “anti-woke” AI that’s a playground for misogyny and other forms of hatred? Is it an attempt to whip up hysteria that diverts attention from basic issues of justice and fairness? Is it, as danah boyd argues, that tech leaders are afraid that they will become the new underclass, subject to the AI overlords they created?

I can’t answer those questions, though I fear the consequences of an “AI Pause” would be worse than the possibility of disease. As danah writes, “obsessing over AI is a strategic distraction more than an effective way of grappling with our sociotechnical reality.” Or, as Brian Behlendorf writes about AI leaders cautioning us to fear AI1:

Being Cassandra is fun and can lead to clicks …. But if they actually feel regret? Among other things they can do, they can make a donation to, help promote, volunteer for, or write code for:

• The Campaign to Stop Killer Robots
• Witness.org, who have developed tools, infrastructure, and messaging for countering AI-generated fake news built to attack human rights
• The Mozilla Foundation, who are driving hard on ethical AI research and related fields like data governance

A “Pause” won’t do anything except help bad actors to catch up or get ahead. There is only one way to build an AI that we can live with in some unspecified long-term future, and that is to build an AI that is fair and just today: an AI that deals with real problems and damages that are incurred by real people, not imagined ones.

Footnotes1. Private email

View Details

Large language models continue to colonize the technology landscape. They’ve broken out of the AI category, and now are showing up in security, programming, and even the web. That’s a natural progression, and not something we should be afraid of: they’re not coming for our jobs. But they are remaking the technology industry.

One part of this remaking is the proliferation of “small” large language models. We’ve noted the appearance of llama.cpp, Alpaca, Vicuna, Dolly 2.0, Koala, and a few others. But that’s just the tip of the iceberg. Small LLMs are appearing every day, and some will even run in a web browser. This trend promises to be even more important than the rise of the “large” LLMs, like GPT-4. Only a few organizations can build, train, and run the large LLMs. But almost anyone can train a small LLM that will run on a well-equipped laptop or desktop.

AI* NVidia has announced Nemo Guardrails, a product whose purpose is to keep Large Language Models operating safely. It prevents LLMs from straying off-topic and answering questions that it is not allowed to answer, checks facts (using other LLMs), and only allows it to access third-party applications known to be safe. * QuiLLMan is an open source voice chat. It uses the Vicuna-13B model, with OpenAI Whisper to transcribe the user’s audio, and Metavoice Tortoise to convert the response back to spoken audio. * The RedPajama project intends to create a fully open source large language model. The first step in this process is the release of a 1.2 trillion token dataset for training. * AI does fashion: Researchers (in Italy, where else?) have developed a Multimodal Garment Designer that uses diffusion models to create realistic images of humans wearing clothes described in prompts. * We talk casually about prompt engineering; Mitchell Hashimoto (founder of Hashicorp) discusses what it means for prompt engineering to be a real engineering discipline. * WasmGPT provides yet another way to run a ChatGPT-like AI chatbot in the browser, this time with WebAssembly. It uses a version of the Cerebras-GPT-1.3B model. Although it is very prone to hallucination, it demonstrates what can be done with WASM and without exotic hardware. * Stability.ai, the creator of Stable Diffusion, has just announced a new large language model, StableLM. The model is open source, and can be used in commercial applications. It was trained with a new dataset, based on The Pile but much larger. * LLaVA (Large Language and Vision Assistant) is a new multimodal language model that allows you to upload images and ask questions about them. * Just as there are techniques for training specialized LLMs, it’s possible to train specialized diffusion models for image generation. Dreambooth is one practical technique for personalizing diffusion models. * GPT-4’s image capabilities are still disabled. A research group has created MiniGPT-4, which allows users to upload and chat about images. It is based on Vicuna, so it can (probably) run on a well-equipped laptop or desktop. * Web LLM is a project that runs the Vicuna 7B large language model entirely in the Chrome browser, using the WebGPU (in the current Chrome beta). Its performance is surprisingly good. * AWS has released its own large language model called Titan, plus a new service for training and deploying LLMs called Bedrock. Their goal is to help users develop their own chatbots, which will presumably run on AWS. * What’s beyond ChatGPT? AutoGPT means the creation of ChatGPT agents that execute tasks for the user without intervention. These tasks typically include additional ChatGPT requests, with automatically generated prompts. * Databricks has released Dolly 2.0, a 12B parameter model that is entirely open source and has been trained with data that is independent of the GPT models (unlike Alpaca and other small LLMs). The model and its training data are available on GitHub and HuggingFace. * One of GPT-4’s plugins is a sandbox that allows it to run Python programs. GPT-3.5 and 4 frequently wrote programs, but could only “guess” about their output. This could be a big step forward in GPT-4’s accuracy, at least for programming tasks. * Alibaba has announced that it will roll out a ChatGPT-like bot, named Tongyi Qianwen. It plans to integrate the bot into all of its products, starting with Alibaba’s workplace messaging app. * Facebook has developed SAM, a universal segmentation model that can detect and mark all of the individual objects in an image. Natural language prompts specify which objects in an image you want to isolate. * Generative agents use large language models and other generative AI tools to simulate human behavior. In a simulation which was prompted only by a suggestion that the agents throw a party, they planned, sent invitations, made acquaintances, and executed many other human behaviors. * We are experiencing a proliferation of small large language models: based on Meta’s LLaMA with 6B to 13B parameters and capable of running on a well-equipped laptop or desktop with GPU, with additional training based on prompt/response pairs from ChatGPT. The latest are Vicuna and Koala; there will no doubt be others. * The use of ChatGPT has been banned in Italy because of privacy issues. (The ban was lifted at the end of April after OpenAI addressed issues raised by the regulators). It’s likely that Germany will follow, and possibly other European nations. * On at least three occasions, Samsung employees have inadvertently disclosed technology secrets by using ChatGPT. Their prompts and ChatGPT’s responses were incorporated into ChatGPT’s language model, from which they leaked to the outside world. * Google has enabled Bard’s code generation capabilities. It has also added with additional arithmetic and logic capabilities, making it less likely to make mistakes in simple arithmetic and logic. * Researchers have created a new AI architecture that combines neural networks with symbolic models in a way that overcomes the limitations of both. * The generative art application Midjourney appears to have temporarily suspended its free trial accounts program in response to deep fakes that have been generated on the platform. Free trials have been suspended until the next “improvement to the system” has been deployed.

Programming* Pushup is a new web framework for Go. It is an “opinionated” template-based framework in the style of Ruby on Rails or Django. Ignore the ill-informed Java bashing; the framework looks like it’s worth investigating. * Docs-as-Code: Etsy has built tools to make the development of documentation as rigorous and maintainable as the development of code, integrating documentation into their development and deployment pipelines. * AWS has opened up CodeWhisperer, a competitor to GitHub Copilot, for use. It is free for personal use. * According to a survey, Kubernetes deployments are trending towards “Managed Kubernetes,” in which responsibility for running Kubernetes is delegated to another company, typically a cloud vendor. * FerretDB is a new open source database that’s an alternative to MongoDB. Because it uses the Server Side Public License (SSPL), MongoDB can no longer be considered open source. * A new database, NAM-DB, demonstrates that distributed transactions can scale. * Flyte is an open source container orchestration platform that has been designed specifically for data science workloads. It is based on Kubernetes.

Security* An important report highlights the security risks of AI systems. AI has all the vulnerabilities of traditional software, in addition to its own; and while it isn’t yet an attack vector of choice, attacks have been seen in the wild, and will no doubt proliferate as AI is deployed more widely. * There are many ways to get cryptography wrong—and the problems are a lot more subtle than “don’t implement cryptographic algorithms yourself.” Here’s a post on Cyptographic Best Practices that shows how to get it right. * eBPF (enhanced Berkeley Packet Filter) is a powerful tool for detecting attacks and threats against containers; it is usable in situations where traditional security monitoring doesn’t work. * A new prompt injection attack allows an attacker to steal chat data by tricking the user into copying and pasting a prompt into ChatGPT. * SAP has created a Risk Explorer that can help users evaluate the risks in their software supply chains. It’s a hierarchy of known attacks, with explanations, that can be explored through a graphical interface. * PassGAN is an AI-based password cracking tool. Despite fear-mongering hype, it is not better than brute force methods. More important, its developers are recommending that users change their passwords every 3 to 6 months, a change that makes sites more vulnerable, and that goes against recommendations from NIST, the FTC, Microsoft, and others. * An attack against most modern cars requires hijacking the CAN bus (Controller Area Network), which connects all of a car’s systems. It requires some vandalism; on a locked car, the easiest way to access the CAN bus is through the headlights. The attack has been seen in the wild. * Workload Security Rings are a new approach to isolating workloads based on their security requirements while minimizing compromises to efficiency. Workloads fall into one of three classes: sensitive, hardened, and trusted. * The FBI has shut down Genesis Market, an online store for stolen data and malware. * The creators of large language models are not keeping up with the attacks against them. Security is, as they say, a “hard problem”; but with the models already in widespread use, LLM-based fraud won’t be far behind. * A research project at CMU installed hundreds of networked sensors, including microphones, throughout a new CS department building. This installation has created a significant controversy about the meaning and future of privacy. * Fake Ransomware sounds like an April Fool’s joke, but it’s real. Some threat actors threaten to sell or reveal stolen data, without having actually obtained the data. It’s a weird kind of phishing, and surprisingly effective. * A large set of leaked documents describes Russia’s far-reaching cyberwarfare efforts. * Security Copilot is a chat assistant to help IT staff with incident response. It is based on GPT-4, with an additional model integrating data from Microsoft’s knowledge of security incidents.

Web* Consent-O-Matic is a browser plugin that automatically fills in annoying cookie popups in a way that maximizes privacy. It is available from browsers’ web stores; source code is in GitHub. * Google’s Environmental Insights Explorer provides access to data about the environment and sustainability for over 40,000 cities worldwide. * Perseus is a new high performance Web framework for Rust. It runs on WebAssembly. * CGI makes a comeback! Of course, it’s never really gone away. But WCGI, using WebAssembly to run CGI applications, is safer and faster. * WebGPU is shipping in Chrome 113 (currently in Beta), and development is in progress for Firefox and Safari. WebGPU is a JavaScript standard for interacting with GPUs and other advanced graphics hardware from the browser. * Salesforce has created a platform that allows companies to create NFT-based customer loyalty programs. These programs give companies direct access to customer data, eliminating the need to work within restrictions on the use of cookies. Are crypto wallets the new cookies?

Augmented and Virtual Reality* Facebook/Meta is using undercover content moderators to police Horizon Worlds. * Is privacy possible in virtual reality? Probably not. So much relies on motion, and motion is identifiable. Headsets leave a trail of data that will be very hard to anonymize. * Augmented reality isn’t dead. Snap is launching AR “mirrors” for stores that show customers what they will look like wearing clothes without trying them on.

View Details

This post is a brief commentary on Martin Fowler’s post, An Example of LLM Prompting for Programming. If all I do is get you to read that post, I’ve done my job. So go ahead–click the link, and come back here if you want.

There’s a lot of excitement about how the GPT models and their successors will change programming. That excitement is merited. But what’s also clear is that the process of programming doesn’t become “ChatGPT, please build me an enterprise application to sell shoes.” Although I, along with many others, have gotten ChatGPT to write small programs, sometimes correctly, sometimes not, until now I haven’t seen anyone demonstrate what it takes to do professional development with ChatGPT.

In this post, Fowler describes the process Xu Hao (Thoughtworks’ Head of Technology for China) used to build part of an enterprise application with ChatGPT. At a glance, it’s clear that the prompts Xu Hao uses to generate working code are very long and complex. Writing these prompts requires significant expertise, both in the use of ChatGPT and in software development. While I didn’t count lines, I would guess that the total length of the prompts is greater than the number of lines of code that ChatGPT created.

First, note the overall strategy Xu Hao uses to write this code. He is using a strategy called “Knowledge Generation.” His first prompt is very long. It describes the architecture, goals, and design guidelines; it also tells ChatGPT explicitly not to generate any code. Instead, he asks for a plan of action, a series of steps that will accomplish the goal. After getting ChatGPT to refine the task list, he starts to ask it for code, one step at a time, and ensuring that step is completed correctly before proceeding.

Many of the prompts are about testing: ChatGPT is instructed to generate tests for each function that it generates. At least in theory, test driven development (TDD) is widely practiced among professional programmers. However, most people I’ve talked to agree that it gets more lip service than actual practice. Tests tend to be very simple, and rarely get to the “hard stuff”: corner cases, error conditions, and the like. This is understandable, but we need to be clear: if AI systems are going to write code, that code must be tested exhaustively. (If AI systems write the tests, do those tests themselves need to be tested? I won’t attempt to answer that question.) Literally everyone I know who has used Copilot, ChatGPT, or some other tool to generate code has agreed that they demand attention to testing. Some errors are easy to detect; ChatGPT often calls “library functions” that don’t exist. But it can also make much more subtle errors, generating incorrect code that looks right if it isn’t examined and tested carefully.

It is impossible to read Fowler’s article and conclude that writing any industrial-strength software with ChatGPT is simple. This particular problem required significant expertise, an excellent understanding of what Xu Hao wanted to accomplish, and how he wanted to accomplish it. Some of this understanding is architectural; some of it is about the big picture (the context in which the software will be used); and some of it is anticipating the little things that you always discover when you’re writing a program, the things the specification should have said, but didn’t. The prompts describe the technology stack in some detail. They also describe how the components should be implemented, the architectural pattern to use, the different types of model that are needed, and the tests that ChatGPT must write. Xu Hao is clearly programming, but it’s programming of a different sort. It’s clearly related to what we’ve understood as “programming” since the 1950s, but without a formal programming language like C++ or JavaScript. Instead, there’s much more emphasis on architecture, on understanding the system as a whole, and on testing. While these aren’t new skills, there’s a shift in the skills that are important.

He also has to work within the limitations of ChatGPT, which (at least right now) gives him one significant handicap. You can’t assume that information given to ChatGPT won’t leak out to other users, so anyone programming with ChatGPT has to be careful not to include any proprietary information in their prompts.

Was developing with ChatGPT faster than writing the JavaScript by hand? Possibly–probably. (The post doesn’t tell us how long it took.) Did it allow Xu Hao to develop this code without spending time looking up details of library functions, etc.? Almost certainly. But I think (again, a guess) that we’re looking at a 25 to 50% reduction in the time it would take to generate the code, not 90%. (The article doesn’t say how many times Xu Hao had to try to get prompts that would generate working code.) So: ChatGPT proves to be a useful tool, and no doubt a tool that will get better over time. It will make developers who learn how to use it well more effective; 25 to 50% is nothing to sneeze at. But using ChatGPT effectively is definitely a learned skill. It isn’t going to take away anyone’s job. It may be a threat to people whose jobs are about performing a single task repetitively, but that isn’t (and has never been) the way programming works. Programming is about applying skills to solve problems. If a job needs to be done repetitively, you use your skills to write a script and automate the solution. ChatGPT is just another step in this direction: it automates looking up documentation and asking questions on StackOverflow. It will quickly become another essential tool that junior programmers will need to learn and understand. (I wouldn’t be surprised if it’s already being taught in “boot camps.”)

If ChatGPT represents a threat to programming as we currently conceive it, it’s this: After developing a significant application with ChatGPT, what do you have? A body of source code that wasn’t written by a human, and that nobody understands in depth. For all practical purposes, it’s “legacy code,” even if it’s only a few minutes old. It’s similar to software that was written 10 or 20 or 30 years ago, by a team whose members no longer work at the company, but that needs to be maintained, extended, and (still) debugged. Almost everyone prefers greenfield projects to software maintenance. What if the work of a programmer shifts even more strongly towards maintenance? No doubt ChatGPT and its successors will eventually give us better tools for working with legacy code, regardless of its origin. It’s already surprisingly good at explaining code, and it’s easy to imagine extensions that would allow it to explore a large code base, possibly even using this information to help debugging. I’m sure those tools will be built–but they don’t exist yet. When they do exist, they will certainly result in further shifts in the skills programmers use to develop software.

ChatGPT, Copilot, and other tools are changing the way we develop software. But don’t make the mistake of thinking that software development will go away. Programming with ChatGPT as an assistant may be easier, but it isn’t simple; it requires a thorough understanding of the goals, the context, the system’s architecture, and (above all) testing. As Simon Willison has said, “These are tools for thinking, not replacements for thinking.”

View Details

TLDR:* Cloudless apps use protocols instead of centralized services, making them easily portable. (Imagine application storage and compute as unstoppable as blockchain, but faster and cheaper than the cloud.) * Cloudless is tractable now that enough people are familiar with cryptographic signing, and key-handling infrastructure has become part of the browser. * Upgrading the current status quo usage of bearer tokens to include signatures from client device keys enables more than security, it also opens the path to enterprise cost savings and radically new business models. * Cost savings come from moving compute to the data, and commuting multiple operations (including permission checks) to avoid proxy copying. This is all enabled because data and operations are cryptographically verifiable. * New business models include hobbyist apps going viral without incurring costs to the developer, as well as new ways to provision pay-per-use services. * Timeline—cloudless is ready to become mainstream in the next builder-driven cycle. Paradigm WavesParadigm shifts in computing are as regular as waves on a beach, it’s hard to see where they came from and even harder to see where they are going. We have seen shifts from mainframe computers to personal computers, and from servers to the cloud. Each shift presented new challenges and opportunities, shaping the way we interact with technology. The most recent large-scale shift was from servers to the cloud, driven by an acknowledgment that using commodity servers run by experts is a better choice for most businesses. Serverless APIs are the culmination of the cloud commoditizing the old hardware-based paradigm. The same process of commoditization that gave rise to the cloud will also bring about the next paradigm, creating a new wave of abstractions and a rising tide for tomorrow’s applications.

“Make yourself a monopoly by growing the markets around you … Smart companies try to commoditize their products’ complements.”
— Joel Spolsky

This iconic Joel Spolsky quote is a testament to his deep understanding of the technology industry and its market dynamics. Spolsky, a renowned software engineer and entrepreneur, co-founded Fog Creek Software, Stack Overflow, and Trello. With years of experience in the field, he has developed keen insights into business strategies and the importance of commoditization in the tech sector. His quote emphasizes the need for companies to create monopolies by commoditizing complementary products, which has proven to be a successful approach for many businesses. This means making the hardware supply chain into a commodity if you make PCs, making PCs into commodities if you sell operating systems, and making servers a commodity by promoting serverless function execution if you sell cloud. What goes around comes around as the cloud becomes the next commodity, and the independent crew of cloudless innovators, the next monopoly breakers.

From the cloud to the networkThe new paradigm shift is from the cloud to the protocol network. Protocol networks are groups of loosely affiliated enterprises that provide globally available services like ledger, compute, and storage. Just as serverless is the culmination of the cloud, this move to protocol networks will culminate in cloudless APIs, leading to applications driven by protocols with incentives and capabilities that go beyond what the cloud’s location-based paradigm can offer. They run on any cloud or other data-center and reward service providers through fees they collect from users.

The new paradigm shift is from the cloud to the protocol network. Protocol networks are groups of loosely affiliated enterprises that provide globally available services like ledger, compute, and storage. Just as serverless is the culmination of the cloud, this move to protocol networks will culminate in cloudless APIs, leading to applications driven by protocols with incentives and capabilities that go beyond what the cloud’s location-based paradigm can offer. They run on any cloud or other data-center and reward service providers through fees they collect from users.

NOTE

Blockchain smart contracts are some of the first use cases, but runtimes like Socket Supply for network (thanks Paulo for putting the word cloudless in my vocabulary!), utilities like Filecoin for storage, and APIs like Tableland for databases are also gaining popularity. At the forefront of this movement are technologies like Fission and BlueSky, which focus on putting the ownership of logic and data into users’ hands.

We call this new paradigm of network protocol based infrastructure cloudless. Its benefits include cost and security improvements as well as lower cognitive overhead and operational burden for developers, users, operators, and enterprises stemming from its location-independence and cryptographic verifiability. This is a technical consequence of content addressing, the hash based identifier system widely used in storage networks and leveraged by peer-to-peer networks for global addressability. We’ll discuss the technical underpinnings of cloudless later in this article.

As the cloud becomes commoditized and more developers, businesses, and users become aware of Cloudless computing’s advantages, such as increased data privacy, greater resilience, and lower costs, there will likely be a stronger inclination to embrace these new platforms. The move to more abstract APIs has an element of structural inevitability. As protocol networks emerge and gain traction, we can anticipate a phase change in the technological landscape, akin to the formation of a solid crystal structure from a less stable liquid state. The availability of these new networks serves as a catalyst for change, driving a more rapid transition from cloud-based systems to cloudless computing.

Most consumer-facing apps have not been written in a location-independent way thus far, primarily because the required infrastructure was not yet available to realize the benefits. However, with the advent of Cloudless protocols, we are witnessing a new wave of applications that harness the potential of these technologies. Early adopters have focused on smart contracts and decentralized apps (dApps), but the next wave is much more extensive, encompassing social applications, provable AI execution and training data provenance, big data processing like transcode or map-reduce, and asset delivery for gaming, metaverse, and media. These use cases exemplify the transformative nature of cloudless computing, showcasing its potential to revolutionize various industries and redefine the way we interact with technology.

Cost Savings and New Business ModelsBENEFITS OF CLOUDLESS COMPUTING

• Democratization of app development: Cloudless computing enables beginners and small teams to create and scale applications without incurring centralized costs, fostering a more diverse range of voices and perspectives.
• User-driven cryptographic identity: Allows users to grant new apps access to their data, resulting in novel and innovative applications.
• Data DAOs for long-term storage: Facilitate the sponsorship of content storage for the apps they represent, benefiting archives and long-term storage institutions.
• Cacheability and time savings: Automated data provenance tracking and verification lead to increased efficiency for data scientists and research workloads.
• Decentralized hosting and resource management: Overcome resource limitations and handle computational loads more effectively, enabling compute at the edge and reducing data transfers.
• Competitive market for computing infrastructure: Cryptographic verifiability and trust create a competitive market for enterprises, leading to lower costs and more robust applications.
• Resilience and trust through cloudless protocols: Location-independent links to data and compute enable new applications and business models, enhancing the interconnectedness and efficiency of the computing landscape. The democratizing effects of the cloudless paradigm are poised to revolutionize the future of applications. For instance, even beginners can create applications that go viral without incurring centralized costs. Imagine crafting a social media app (or remixing an existing one), sharing it with friends, and witnessing it go viral—all without having to pay a bill. Cloudless computing reduces cognitive overhead for developers, users, operators, and enterprises, leading to more cost-effective solutions.

Without the need for apps to pay a centralized hosting bill, we can expect to see a broader range of voices and perspectives represented. The user-driven nature of cryptographic identity makes it easier to write new apps using existing data sets, as users can easily grant innovative new chat apps or photo galleries access to their data. This will lead to the development of novel applications we can’t even imagine yet. Data DAOs can sponsor the storage of content for the apps they represent. This is a great option for archives and other long-term storage institutions.

With cloudless computing, the inherent cacheability allows for time savings and increased focus on problem-solving. This cacheability goes beyond peer-to-peer and into the foundations of computing. Automated data analysis workloads, in the research world and commercially, are heavy users of shared data and reusable computing, and have the most to gain from automatic data provenance tracking and verification. Data scientists coding in notebooks like Databricks frequently rerun the same transformations on source data. Cacheability is part of what helped leading database vendor Snowflake dominate the market, and now, with cloudless, microservices can be upgraded to use verifiable data and deterministic computing, leveraging cache liveness provided by the protocol network.

Decentralized hosting costs and management enable developers to overcome resource limitations and handle computational loads more effectively. Protocol nodes can be placed in retail locations or edge sensors, allowing for compute to be performed over data at rest at the edge. This eliminates the need for unnecessary data transfers and enables faster, more efficient querying of data. As cryptographic verifiability makes trust more fungible, enterprises will be able to run their businesses on a competitive market of computing infrastructure and specialized algorithm vendors, resulting in lower costs and more robust applications.

Cryptographic identity, verifiable data, and deterministic computing make cloudless apps possible. With location-independent links to data and compute, anyone can access data and execute functions anywhere. By making intelligent use of today’s existing cloud providers and network infrastructure, Cloudless protocols add a layer of resilience and trust, enabling new applications and business models, and paving the way for a more interconnected and efficient computing landscape.

Commoditizing the Cloud: How is Cloudless Possible?Cloudless computing is built upon the principles of decentralization, collaboration, and shared innovation, and its success is dependent on embracing open source and open standards. This approach ensures that the underlying technologies can be continuously improved, adapted, and maintained by a diverse community of stakeholders, eliminating the risk of vendor lock-in, promoting interoperability, and enabling a more resilient and flexible infrastructure. Cloudless computing offers several advantages over serverless cloud computing, such as cost savings, increased choice for developers, and the potential for new business models centered on app and data ownership. These benefits are made possible by the core foundations of cloudless computing: cryptographic identity, verifiable data, and deterministic compute. In the following sections, we will delve into the features that make cloudless apps possible.

Cryptographic IdentityCryptographic identity is fundamental to cloudless computing. It addresses the identity problem and its challenges by leveraging the increasing familiarity with private keys, signing transactions, and verifying hashes. Recent advancements in user experience, such as TouchID/FaceID and secure enclave, have made cryptographic key pairs more accessible to average users, setting the stage for the next generation of applications that take advantage of cryptographic guarantees. The operating system and browser vendors offer password management products that many people are familiar with. Cloudless capability delegation feels a lot like a password manager, only instead of passwords it uses secure signatures, reducing the risk of leaks and hacking.

In this self-sovereign model, users control their own crypto keyrings, granting them greater visibility and authority over their data and online interactions. This eliminates the need for reliance on centralized service providers and prevents lock-in. Access to accounts is maintained by delegating capabilities to other cryptographic actors, such as other devices or account recovery services.

In other words, “No lockouts/no lock-in.”

Verifiable DataVerifiable data enables the storage and retrieval of data that is independently verifiable and authenticated using cryptographic techniques. The peer-to-peer web protocol, IPFS (InterPlanetary File System), for example, uses hash-based Content Identifiers (CIDs) to ensure data integrity and authenticity. These CIDs allow data to be fetched from any location, using location-independent URIs, and provide a layer of safety that systems relying on location-based addresses (like URLs) cannot offer. Because hash-based identifiers are deterministically and uniquely derived from the content they reference, they are unforgeable and tamper-proof, providing the robust foundation for cloudless applications like smart contracts, distributed identity, storage, and compute.

Applications that use verifiable data can benefit from improved security, lower computing costs, and better performance. Global addressability means CIDs enable data to migrate to the most appropriate provider without any loss of trust, and the immutable nature of these addresses allows for efficient caching and acceleration.

Deterministic ComputeDeterministic computing allows for consistent and predictable computations, regardless of data location or infrastructure. It requires a container runtime or execution environment, a way to address data consistently (such as with CIDs), and a secure and verifiable method to invoke the computation.

The benefits of deterministic computing include faster second runs, cost-effective and performant location selection, workload sharing and reuse, edge computing for reduced network costs and improved performance, and the ability to coalesce workloads for cost savings and accelerated output.

By moving from location-dependent APIs to location-agnostic APIs, cloudless computing can optimize data routing and enable greater flexibility and cost savings. This is exemplified by compute-over-data projects like Bacalhau, which leverage the guarantees of cryptographic identity, verifiable data, and deterministic computing to create a competitive marketplace for computing infrastructure and algorithm vendors.

Early adopters of verifiable data include industries like smart contracts, NFTs, and DAOs, as well as organizations focused on maintaining journalistic integrity in reporting. One notable example is the Starling Lab, a nonprofit academic research center that uses cryptography and decentralized protocols to maintain trust in sensitive digital records giving journalistic data the standard of evidence that can be used in war crimes trials. The lab employs the Starling Framework of “Capture, Store, Verify” for digital media, leveraging IPFS to provide a powerful solution for trust and integrity. Their work demonstrates how verifiable data is essential for preserving and maintaining trust in critical historical data, which can be applied to various other use cases.

The growing demand for verifiable data is shaping the future of cloudless computing and distributed identity systems. Factors such as the rise of cryptocurrencies, blockchain technology, regulations like GDPR, and advancements in AI and machine learning have contributed to the increasing need for verifiable data. As tools mature and the learning curve becomes less steep, organizations working on mission-critical data applications will increasingly adopt these technologies. Existing tools, such as programming notebooks and static site hosting, will evolve to use cloudless technology, further driving the adoption and impact of verifiable data in various industries and applications.

Now that we’ve reviewed the core enabling technologies, you can see how cloudless makes it possible to reduce costs and gain capabilities. Combining cryptographic identity, verifiable data, and deterministic compute allow for a more cost-efficient and flexible computing landscape, where users and applications can interact in ways not possible with traditional cloud-based systems. By leveraging cryptographic guarantees, cloudless computing unlocks a world of possibilities that extend beyond mere optimizations and cost savings, setting the stage for a future filled with new voices, applications, and opportunities.

Data Privacy and OwnershipAs our digital lives become increasingly interconnected, the need for secure and user-friendly distributed identity systems grows more pressing. These systems are vital for protecting individual privacy and granting users control over their data. However, realizing the full potential of distributed identity systems requires overcoming numerous challenges, chief among them being the user experience. This section delves into the importance of UX in distributed identity systems, examining the latest innovations and trends that have improved security and usability, while also discussing the remaining challenges and how they can be addressed.

User experience (UX) is crucial for distributed identity systems, as it ensures ease of use, accessibility, and adoption for users of varying technical expertise. One of the most significant challenges is keypair management. Non-extractable keypairs, recently made available to the mainstream via WebAuthn and biometric authentication systems like TouchID and FaceID, have significantly improved the security and user experience in distributed identity systems.

WebAuthn is a modern web authentication standard that relies on authenticators, such as hardware security keys or platform-based authenticators like fingerprint scanners, to create and manage public-private key pairs securely. The private key remains securely stored on the authenticator and is never exposed, reducing the risk of key theft or unauthorized access.

The increasing familiarity with cryptography, fueled by the widespread adoption of cryptocurrency wallets like MetaMask, has also contributed to a better user experience in distributed identity systems.

Companies like Apple have played a significant role in improving the UX of distributed identity systems. Innovations like TouchID and FaceID, especially when used with open standards like WebAuthn, have made it easier for users to interact with such systems securely. WebAuthn supports non-extractable keypairs, providing enhanced security by ensuring that private keys are securely stored within authenticators and never exposed or extractable.

In addition to these security features, Apple’s iPhone setup process, which uses local radio and camera/screen inputs for secure pairing, is a great example of seamless user experience. This approach allows for easy capability delegation between device keys, ensuring that users can quickly and securely transfer data and settings between devices. It is worth noting that UCAN, a distributed authorization protocol, also leverages non-extractable keypairs and employs a similar delegation approach for enhanced security and user experience. Both Apple and UCAN demonstrate how integrating these concepts into distributed identity systems can result in a more intuitive and secure user experience.

In contrast, the open-source community often faces challenges in improving UX for distributed identity solutions. Solutions that cater to technical users may not be accessible or user-friendly for non-technical users. For instance, mnemonic passphrase private key sharing in cryptocurrency wallets may be suitable for tech-savvy users but not for the general population. To achieve a better user experience, developers need to invest time and effort in creating robust, user-friendly solutions.

As users become more sophisticated and technology becomes more user-friendly, the challenges of catering to users with less computing experience are gradually being addressed. A range of solutions for multi-signature recovery of crypto assets is available, spanning from powerful tools for geeks to easy-to-use options for non-technical users. The market will reward those with the most trustworthy UX, driving continuous improvement.

Emerging trends, technologies, and practices, such as the increasing demand for verifiable data, will contribute to improved data privacy and ownership through better UX in distributed identity systems. As enterprises recognize the cost-saving and performance-enhancing benefits of data verification, investment in UX for cryptographically aware toolchains will grow, resulting in more accessible and user-friendly cloudless solutions.

Not bullish on bearer tokensIn this section, we’ll dive into some technical details around the limitation of bearer tokens, the modern equivalent of cookies, as well as explore alternative cloudless solutions that promise enhanced security and efficiency. Although this discussion is a bit more technical in nature, we encourage readers of all backgrounds to stay engaged, as there is valuable information applicable to everyone. Following this section, we will broaden our focus to address further implications and opportunities in the realm of cloudless computing, data privacy, and distributed identity systems.

Bearer tokens, also known as access tokens or API keys, are commonly used in modern authentication and authorization systems to grant access to protected resources. They are typically issued by an authorization server and are passed along with each request to a resource server, which uses the token to determine whether the client has permission to access the requested resource. While bearer tokens have become a popular choice for authentication and authorization, they also come with several significant limitations.

One major issue with bearer tokens is that they encourage an architecture that routinely proxies data through multiple services. In many cases, a user’s device must send a request to a central service, which then forwards the request to another service with the bearer token attached. This process may be repeated multiple times before the data is ultimately returned to the user’s device. This proxying of data through multiple services is done to keep the bearer token secret and prevent it from being intercepted by a malicious actor, but it exacts a heavy cost in terms of performance, reliability, and resource use.

This proxying process is highly inefficient, as it adds multiple extra steps to the data retrieval process and can slow down the overall performance of the application. Additionally, it increases the risk of security breaches, as each service that handles the bearer token is a potential point of failure. Because bearer tokens are simply strings of characters that are passed along with each request, they can be easily intercepted and used by unauthorized parties if they are not properly protected. The more services that handle the bearer token, the greater the risk that it will be intercepted by a malicious actor.

Instead of dwelling on the risks of bearer tokens, let’s explore an alternative solution that leverages client-side cryptographic keys to create capabilities, delegations, invocations, and receipts that are safe to store-and-forward without the danger of replay attacks. This approach utilizes cryptographic proofs rather than bearer tokens. By signing each invocation as it is created, the client can safely send it to anyone on the network, who can route it to the service which will run it. This allows workloads to be coalesced and moved to the most cost-effective infrastructure, as described earlier as among the benefits of deterministic computing.

Centralized authentication systems, which often rely on bearer tokens, have their own set of issues. They are controlled by a single entity or organization, which can wield significant power over users and their data. These systems are also vulnerable to data breaches and hacking, resulting in sensitive information falling into the wrong hands. Furthermore, they favor data silos, making it difficult for users to share data across different platforms and services.

UCAN, or User-Controlled Authorization Networks, offers a decentralized access control protocol that enables secure and verifiable data routing by allowing users to delegate access to their capabilities using public key cryptography. Users can grant permission to access their data to other actors through the use of public keys, without the need for a central authority to manage authentication. With UCANs, users control the keys and delegations, and services can cryptographically verify proofs about the authorization data. UCANs rely on cryptographic signatures, reducing the risk of token leakage, stealing, and expiration.

This decentralized and location-independent approach to coding allows services to be composed without the need for a location-based proxy secret model and the risk of bearer token leakage. The “compute over data” model enables computations to be performed on the data, rather than the data being transported to the computation. This makes data routing possible in a secure and efficient manner, with computations performed and results signed by service providers without relying on intermediaries to handle and transmit the data.

By using verifiable data and UCAN, Cloudless computing demonstrates the benefits of a more secure, efficient, and user-controlled approach to authentication and authorization, moving away from the limitations and risks associated with traditional bearer tokens and centralized systems.

The democratization of app developmentIn the world of cloudless computing, a community of hobbyist developers can collaborate on a project, adding features and making modifications to the code as they see fit. Each member can spin up a copy of the app to experiment with, test, and improve. As the app evolves and attracts attention from others, it can grow and fork as new communities adopt the app. The cloudless nature of the project means there are no hosting bills, and the developers can avoid the crippling costs that often accompany the sudden popularity of a traditional application. This democratization of app development enables hobbyist developers to create and adapt applications without the limitations imposed by traditional platforms.

The relationship between hobbyist developers and platforms like GitHub fosters a thriving developer ecosystem. For example, the open-source project “TodoMVC” demonstrates the power of collaboration and forking on platforms like GitHub. Developers can easily compare different implementations of the same app using various frameworks and libraries, leading to numerous forks and adaptations as developers experiment and personalize the application. This collaborative environment is integral to the growth and success of open-source projects.

Decentralization empowers even hobbyist developers to address the same markets as mainstream applications, enabling them to create popular open-source projects without the constraints of traditional platforms. This leads to a more innovative and diverse app ecosystem, benefiting both developers and users alike.

Cloudless computing enables a wide range of innovative applications, such as:

1. Decentralized finance (DeFi) platforms and secure document databases for smart contracts.
2. Secure voting systems for election processes and corporate governance.
3. Supply chain management systems for traceability and transparency of goods and products.
4. Healthcare record management systems that are secure, immutable, and accessible.
5. Legal document management systems for secure and tamper-proof tracking of legal agreements and contracts.
6. Asset tracking systems for real-time tracking and management of physical assets.
7. Identity management systems for secure and decentralized authentication and authorization. In the new world, these can be as easy-to-use and informal as is Openlinks, which makes verifiable link-in-profile pages, or as serious as digital drivers licenses.
8. Environmental monitoring systems for real-time tracking of environmental conditions and data.
9. Compliance management systems for secure and transparent tracking of regulatory compliance.
10. Real estate management systems for secure and transparent tracking of property transactions and ownership.

These new types of apps present unique opportunities for hobbyist developers to create innovative solutions in various sectors, further driving the democratization of app development.

Automation tools, such as GitHub Actions, have emerged from the democratization of app development, supporting hobbyist developers and fostering a more inclusive developer ecosystem. By streamlining the software development process, these tools optimize developer productivity, ensure consistency, and elevate the overall project standard. Continuous integration and deployment enabled by automation tools allow developers to automatically test and build their code upon each commit, ensuring code quality and alignment with project standards. This approach reduces friction between team members, promotes a positive environment, and encourages open-source contributors to feel valued and respected. The result is a thriving, innovative, and successful developer community that benefits from collaboration and shared expertise.

The resilience and accessibility of cloudless computing pave the way for a Cambrian explosion of app developer voices. This democratization of app development breaks down barriers and empowers a diverse range of developers, including hobbyists, to create innovative applications without the constraints of traditional platforms. As we have seen with the unstoppable nature of crypto smart contracts, decentralization can lead to a flourishing ecosystem that transcends geographical, economic, and technical limitations.

The cloudless computing paradigm not only reduces costs and fosters collaboration but also enables developers to create secure, scalable, and efficient solutions across various industries. By embracing the potential of cloudless computing and learning from the success of peer-to-peer technologies like IPFS and Ethereum, we can expect a new wave of groundbreaking applications that enrich the lives of users worldwide.

Ultimately, this democratization of app development will lead to a more inclusive, innovative, and robust ecosystem, where diverse developer voices contribute to a brighter and more connected future.

Real-world examplesThe advent of Cloudless computing has brought forth numerous groundbreaking applications and protocols that are already transforming the technological landscape. These early Cloudless applications not only showcase the innovative potential of this technology but also highlight the far-reaching impact it can have across various industries.

Smart contracts on platforms like Ethereum are one of the first and most well-known use cases of Cloudless computing. These self-executing contracts allow for secure and automated transactions on the blockchain, eliminating the need for intermediaries and reducing costs.

In the networking sphere, Socket Supply provides a runtime for decentralized applications, enabling developers to build and deploy their apps in a Cloudless environment. This approach promotes efficiency, security, and user control over data and logic.

For storage, Filecoin has emerged as a popular Cloudless solution that allows users to rent out their unused storage space and earn tokens in return. Filecoin leverages a decentralized network of storage providers, ensuring data redundancy and security.

Tableland, an API for decentralized databases, enables developers to build and deploy applications with user-owned data, ensuring privacy and data sovereignty.

Fission and BlueSky are also leading the charge in the Cloudless movement, focusing on giving users control over their data and the logic of the applications they interact with. These technologies empower users by decentralizing ownership and control of data and software, ensuring a more equitable and transparent digital landscape.

Long-standing protocols, such as DNS and HTTP, have paved the way for large-scale cooperation by insulating apps from implementation-specific details. Similarly, Ethereum and other blockchain technologies harness the power of peer-to-peer networks to create immutable logs, while the SWIFT message format enables secure store-and-forward messaging.

As Cloudless computing continues to evolve and mature, we can expect to see even more transformative applications and use cases across various industries. This paradigm shift will empower individuals, foster innovation, and ultimately reshape the digital world as we know it.

Cloudless computing has the potential to democratize the app development process, enabling hobbyist developers to create and share apps without the need for expensive hosting services. The transformative nature of Cloudless computing has already led to the emergence of innovative solutions in various industries, from healthcare to finance. With the development of decentralized hosting and management solutions, the cost and management of computational loads are reduced, allowing developers to handle resource limitations more effectively. The deployment of protocol nodes at the edge enables compute to be performed over data at rest, eliminating the need for unnecessary data transfers and improving the efficiency of querying data. As cryptographic verifiability makes trust more fungible, enterprises will be able to run their businesses on a competitive market of computing infrastructure and specialized algorithm vendors, resulting in lower costs and more robust applications. With Cloudless computing, we can expect a Cambrian explosion of app developer voices and unstoppable smart contract-powered experiences that will transform the way we interact with technology.

In Conclusion As we look towards the future, the transformative power of cloudless computing is becoming increasingly evident. This revolutionary approach to application development and deployment offers numerous benefits, including reduced environmental impact, democratization of app development, enhanced data privacy, and new opportunities for developers and creators alike.

The environmental impact of cloudless computing cannot be overstated. By distributing computational resources across numerous devices and minimizing reliance on centralized data centers, energy consumption and carbon emissions can be significantly reduced. This decentralized approach to computing infrastructure not only promotes sustainability but also encourages innovative solutions for further reducing our digital footprint.

Developers are incentivized by the cloudless computing paradigm as it grants them greater freedom, flexibility, and access to markets previously dominated by mainstream applications. The ease of entry for hobbyist developers, facilitated by platforms like GitHub, fosters a vibrant and inclusive ecosystem that encourages creativity and collaboration.

The democratization of app development is further bolstered by the cloudless paradigm, breaking down barriers for independent developers and leveling the playing field. With the support of collaboration tools and automation like GitHub Actions, a more diverse range of developers can contribute to and benefit from this rapidly growing field.

Data privacy is another critical aspect of cloudless computing. By eliminating reliance on centralized cloud services, users can maintain greater control over their data and ensure that their information remains secure and private. This heightened level of privacy is particularly important in an era where data breaches and privacy concerns are increasingly common.

The rise of Web3 technologies and their impact on creators and rent-taker issues is also noteworthy. Decentralized platforms enable creators to retain control over their content, reduce fees paid to intermediaries, and foster more direct relationships with their audiences. As the Web3 ecosystem continues to evolve, cloudless computing will play a vital role in empowering creators and minimizing rent-seeking behaviors.

In addition to these broader benefits, cloudless computing brings forth a myriad of specific technologies and innovations. The use of UCAN invocations, IPFS, Merkle DAGs, immutable CIDs, and CAR transactions, are just a few examples of the tools that are shaping the future of cloudless computing. These advancements in data structures will eventually resemble GraphQL, SQL, and NoSQL database APIs, highlighting the potential for creating developer-friendly solutions.

Compute-over-Data (CoD) has become a practical way to run computations across large data archives, with compute-over-data projects like Bacalhau using immutable references to code and data to enable low cost big-data processing. Developers are increasingly leveraging tools like w3up and w3ui to delegate data uploads, reducing runtime requirements and avoiding unnecessary data transfers.

Optimized data routing and features like IPLD for HTML enable apps to run in the browser while still making UCAN calls that can be executed, cached, and stored anywhere on the network. The Saturn content delivery network will allow anyone to be compensated for accelerating these workloads.

Lastly, cloudless computing enables new capabilities and opportunities in the computing world. Innovative applications, such as secure voting systems, supply chain management systems, healthcare record management systems, and asset tracking systems, are just a few examples of the potential that cloudless computing offers. As more developers adopt this paradigm, we can expect to see even more groundbreaking innovations and advancements in the technology landscape.

The future of the cloudless paradigm is one of increased efficiency, cost savings, and empowerment for enterprises, developers, and individuals alike. As a market for computing and algorithms develops, data storage and serverless execution will transition from the centralized cathedral of big cloud providers to the decentralized bazaar of networked protocol participants.

A special thanks to the peer-to-peer and distributed data community for their invaluable contributions to the field of cloudless computing. Their dedication and innovation have significantly impacted this transformative technology, fostering a more decentralized, collaborative, and secure digital future. We appreciate their efforts and look forward to the continued growth of cloudless computing, thanks to their inspiring work and visionary leadership. Heartfelt thanks to the editors and individuals who provided feedback on the early drafts of this article. Your insights, suggestions, and attention to detail have been instrumental in shaping the final version.

View Details

Since the release of ChatGPT last November, it has sucked all the air out of technology discussions. This may be well deserved—in some respects, large language models represent the biggest step forward in computing since the PC. But it makes me wonder what topics aren’t getting the attention that they deserve.

Two topics that started the year strong have fallen off the radar: blockchain-related technologies and “the Metaverse,” whatever that is. A few cryptocurrency crashes coupled with a lot of fraud has soured a lot of people on the crypto world. I’ve never been a strong believer in crypto as an investment, as cash, or even as a way to own digital artworks. However, I wouldn’t write off NFTs and blockchains just yet. Public ledgers may appear to be a technology looking for a solution, but projects like the State of California’s effort to put auto registration on a blockchain are likely to simplify the painful process of dealing with the Department of Motor Vehicles. NFTs may look like making a trip to the grocery store and framing the receipt, but a small (and growing) number of companies are building customer loyalty programs that are essentially NFTs. What’s important about these efforts is that nobody needs to know what’s underneath. No customer ever has to deal with OpenSea, create a wallet, or pay GAS fees. The underlying technology is well-hidden—as it should be. We wouldn’t have wireless networks in our homes if operating a “home network” meant hacking routers, switches, and hosts 1990-style. Customers want technology that “just works.”

The Metaverse has had a different non-history. Facebook renamed itself, and then found out that nobody could agree on what the Metaverse was—at least in part because Facebook’s ideas were, well, lame. We didn’t need “better meetings,” with participants sitting on a couch in a virtual living room. We didn’t need avatars with legs. It’s unclear to me why anyone ever thought those features would give us better meetings. “Better meetings” means fewer meetings. We need better tools for collaboration, so that we don’t need as many meetings to stay in sync. Adobe’s $20B acquisition of Figma shows just how important collaboration is. And that leads us to a different kind of metaverse: not about meetings, but about collaboration, about presence while collaborating, about doing things with your colleagues and associates. Is it a walled garden, owned by an Internet giant? Absolutely not. Is crypto required? No, though blockchains and other technologies may prove useful. Are VR goggles required? Maybe, for some applications. This isn’t Zuckerberg’s Metaverse, nor is it some crypto bro’s Metaverse. It is a way of working and collaborating despite distances and physical isolation. We’ve had “proofs of concept” for a long time, including products like Zoom and mmhmm; now it’s time to build the real thing.

However, if we’re going to get serious about technologies that have suffered when all the air got sucked out of the room, we have to go beyond the overhyped meme-techs. What technologies are underhyped or never hyped? What do we need to hear more about?

Cyber SecurityCiting similar data from both Microsoft and Google, a report from the NSA recently claimed that roughly 70% of all software security vulnerabilities result from memory safety issues. That is, unfortunately, entirely too believable. The first widely destructive cyberattack was the 1988 Morris Worm, which exploited a problem in the way C programs managed memory. 35 years later, the problem hasn’t gone away, even though most programming languages that have appeared since 1990 provide some kind of memory safety. C and C++ still require programmers to do much of their own memory management. Memory-safe languages like Java and Python automate allocating and deallocating memory, though there are still ways to work around the languages’ built-in protections. Rust, which is growing in popularity, provides even more stringent guarantees of memory safety. And Zig, a newer language that’s worth investigating, provides a different set of guarantees.

Ever since the SolarWinds attack, there’s been a lot of talk about the software supply chain. There’s a good market for new tools that build software “bills of materials” listing all the libraries on which your software depends. But knowing your dependencies only solves part of the problem. The VEX standard provides machine readable vulnerability reports. That standard allows organizations to do a better job of analyzing their risks and understanding where they are vulnerable. Ultimately, though, a bigger problem needs to be addressed: how do organizations keep their software updated with security patches?

In 2022, security wasn’t in the news as often as it was in 2020 and 2021. But that doesn’t mean it’s time to relax.

Decentralized ComputingWhat about the Fediverse? That’s the network of decentralized, loosely-coupled services that are held together by network protocols: often ActivityPub, but also IPFS, Scuttlebutt, BlueSky, and others. Mastodon is the most well-known example of the Fediverse; it’s a Twitter-like service that, in the days since Elon Musk’s Twitter abuse, has scaled by a factor of 10, from roughly 1 million to over 10 million users. The growth hasn’t been without pain, but outages have been few and (partly due to the decentralized nature of the protocol) limited. Another factor of 10 would take Mastodon to Twitter scale; a second factor of 10 would be Facebook scale. Can this kind of technology reach Facebook scale? So far, the answer appears to be “yes.” Whether the industry pundits can learn to take seriously a service that has no multi-billionaires or VCs behind it is a different question.

Past Mastodon, there are a number of other decentralized technologies that people should know about. CRDTs (Conflict Free Replicated Data Types) are behind tools like Google Docs, which lets multiple users edit a document simultaneously. An open source CRDT library from the Ink & Switch project promises to make decentralized applications much easier to build. J. Chris Anderson has been arguing for “cloudless” computing, in which the centralized corporate cloud providers are replaced by protocol-based networks of ambient computing power. Ion Stoica’s Sky Computing lab is building the software for another vision of disaggregated computing. Stoica’s name may not be as familiar as Zuck’s or Musk’s, but both Apache Spark and Apache Ray originated in his labs. Is this an idea whose time has come?

A Programming Platform for the WebWebAssembly (WASM) has been around for a few years now; it isn’t new. But it has been growing slowly, and demonstrating value as a computing platform for the Web. WebAssembly provides a browser-based compilation target for high-level languages ranging from C to Rust (including C++, C#, Python, and Ruby). This means that developers can write programs in any of these languages that will run in a browser, without using JavaScript. Developers are beginning to use WASM for servers and other applications that run outside of the browser.

Why is WASM needed? Is it just because JavaScript is a confusing, poorly defined language? Well, partly. Many have noted that JavaScript: The Good Parts is 175 pages long, while JavaScript: The Definitive Guide is 704 pages long. The comparison isn’t fair, but it can’t be ignored, either. More to the point: what would it mean to run servers and other applications in the browser? What if the browser becomes more than a display engine? We’ve seen WASM running the Jupyter server, allowing users to run Jupyter Notebooks without leaving the browser—and in the process, eliminating security issues that trouble large enterprises. The Figma collaborative design tool uses WASM. What else? Will this be WASM’s breakout year?

Database ProliferationYears ago, I wrote that NoSQL wasn’t a database technology; it was a movement. It was a movement that affirmed the development and use of database architectures other than the relational database. It was about choice: there was nothing wrong with MySQL or Oracle when you needed a relational database, but there were few alternatives. Your square peg had to fit a round hole.

While more than a few people are saying that relational databases have won out, it’s important to realize that there are database options, and plenty of them. Lately, I’ve been reading about Pinecone DB, a vector database that looks like it will be a good match for AI applications. DuckDB is a SQL database (yes, relational) that is designed for integration directly into applications, not unlike SQLite. There has been a proliferation of time series and graph databases. Fireproof is a new database designed for “cloudless” applications. So, while NoSQL might not be the rallying cry it once was, it has won the day—not in the sense of replacing relational databases (which was never the real issue), but in the sense of providing alternative database designs and architectures to fit different kinds of applications.

Simpler Container ManagementKubernetes has dominated container orchestration for several years now. That domination hasn’t been without its problems; Kubernetes is complex and has a steep learning curve. Is it time for something simpler, something that is easier to understand and configure?

To understand the difficulty of replacing Kubernetes we have to start with its history, which is unlike most open source projects. It started as an open source release of Google’s Borg: the internal platform that managed their vast infrastructure. Therefore, in its initial release, it was close to fully-formed. It was designed with Google’s engineering staff in mind, and included almost everything you would need to run Google. It wasn’t an initial bare-bones release to which developers gradually added new features. It was complex from the start; it didn’t become complex through a long, slow process that took years.

The problem with a project that starts out fully formed is that, rather than make do with a simple feature set, early adopters can do anything they want. They can build a complete enterprise-scale container orchestration system, whether they need it or not. And perhaps they do need it—but that leads to my own version of the 80/20 rule. 80% of the users need 20% of the features. But 100% of the users need one special feature that’s not in the 20%. As a result, it’s very difficult to imagine a simpler solution that actually works for more than a small number of users.

Some alternatives have appeared, including managed Kubernetes, where you delegate management of your cluster to a third party, typically your cloud provider; HashiCorp’s Nomad; K3S, a lightweight Kubernetes; and even some older tools like Docker Swarm. It’s anyone’s guess whether any of these tools will come to dominance, or whether developers will stick with Kubernetes, complex as it may be.

What other trends and technologies are we missing?

View Details

The notion that artificial intelligence will help us prepare for the world of tomorrow is woven into our collective fantasies. Based on what we’ve seen so far, however, AI seems much more capable of replaying the past than predicting the future.

That’s because AI algorithms are trained on data. By its very nature, data is an artifact of something that happened in the past. You turned left or right. You went up or down the stairs. Your coat was red or blue. You paid the electric bill on time or you paid it late.

Data is a relic—even if it’s only a few milliseconds old. And it’s safe to say that most AI algorithms are trained on datasets that are significantly older. In addition to vintage and accuracy, you need to consider other factors such as who collected the data, where the data was collected and whether the dataset is complete or there is missing data.

There’s no such thing as a perfect dataset—at best, it’s a distorted and incomplete reflection of reality. When we decide which data to use and which data to discard, we are influenced by our innate biases and pre-existing beliefs.

“Suppose that your data is a perfect reflection of the world. That’s still problematic, because the world itself is biased, right? So now you have the perfect image of a distorted world,” says Julia Stoyanovich, associate professor of computer science and engineering at NYU Tandon and director at the Center for Responsible AI at NYU.

Can AI help us reduce the biases and prejudices that creep into our datasets, or will it merely amplify them? And who gets to determine which biases are tolerable and which are truly dangerous? How are bias and fairness linked? Does every biased decision produce an unfair result? Or is the relationship more complicated?

Today’s conversations about AI bias tend to focus on high-visibility social issues such as racism, sexism, ageism, homophobia, transphobia, xenophobia, and economic inequality. But there are dozens and dozens of known biases (e.g., confirmation bias, hindsight bias, availability bias, anchoring bias, selection bias, loss aversion bias, outlier bias, survivorship bias, omitted variable bias and many, many others). Jeff Desjardins, founder and editor-in-chief at Visual Capitalist, has published a fascinating infographic depicting 188 cognitive biases–and those are just the ones we know about.

Ana Chubinidze, founder of AdalanAI, a Berlin-based AI governance startup, worries that AIs will develop their own invisible biases. Currently, the term “AI bias” refers mostly to human biases that are embedded in historical data. “Things will become more difficult when AIs begin creating their own biases,” she says.

She foresees that AIs will find correlations in data and assume they are causal relationships—even if those relationships don’t exist in reality. Imagine, she says, an edtech system with an AI that poses increasingly difficult questions to students based on their ability to answer previous questions correctly. The AI would quickly develop a bias about which students are “smart” and which aren’t, even though we all know that answering questions correctly can depend on many factors, including hunger, fatigue, distraction, and anxiety.

Nevertheless, the edtech AI’s “smarter” students would get challenging questions and the rest would get easier questions, resulting in unequal learning outcomes that might not be noticed until the semester is over–or might not be noticed at all. Worse yet, the AI’s bias would likely find its way into the system’s database and follow the students from one class to the next.

Although the edtech example is hypothetical, there have been enough cases of AI bias in the real world to warrant alarm. In 2018, Reuters reported that Amazon had scrapped an AI recruiting tool that had developed a bias against female applicants. In 2016, Microsoft’s Tay chatbot was shut down after making racist and sexist comments.

Perhaps I’ve watched too many episodes of “The Twilight Zone” and “Black Mirror,” because it’s hard for me to see this ending well. If you have any doubts about the virtually inexhaustible power of our biases, please read Thinking, Fast and Slow by Nobel laureate Daniel Kahneman. To illustrate our susceptibility to bias, Kahneman asks us to imagine a bat and a baseball selling for $1.10. The bat, he tells us, costs a dollar more than the ball. How much does the ball cost?

As human beings, we tend to favor simple solutions. It’s a bias we all share. As a result, most people will leap intuitively to the easiest answer–that the bat costs a dollar and the ball costs a dime—even though that answer is wrong and just a few minutes more thinking will reveal the correct answer. I actually went in search of a piece of paper and a pen so I could write out the algebra equation—something I haven’t done since I was in ninth grade.

Our biases are pervasive and ubiquitous. The more granular our datasets become, the more they will reflect our ingrained biases. The problem is that we are using those biased datasets to train AI algorithms and then using the algorithms to make decisions about hiring, college admissions, financial creditworthiness and allocation of public safety resources.

We’re also using AI algorithms to optimize supply chains, screen for diseases, accelerate the development of life-saving drugs, find new sources of energy and search the world for illicit nuclear materials. As we apply AI more widely and grapple with its implications, it becomes clear that bias itself is a slippery and imprecise term, especially when it is conflated with the idea of unfairness. Just because a solution to a particular problem appears “unbiased” doesn’t mean that it’s fair, and vice versa.

“There is really no mathematical definition for fairness,” Stoyanovich says. “Things that we talk about in general may or may not apply in practice. Any definitions of bias and fairness should be grounded in a particular domain. You have to ask, ‘Whom does the AI impact? What are the harms and who is harmed? What are the benefits and who benefits?’”

The current wave of hype around AI, including the ongoing hoopla over ChatGPT, has generated unrealistic expectations about AI’s strengths and capabilities. “Senior decision makers are often shocked to learn that AI will fail at trivial tasks,” says Angela Sheffield, an expert in nuclear nonproliferation and applications of AI for national security. “Things that are easy for a human are often really hard for an AI.”

In addition to lacking basic common sense, Sheffield notes, AI is not inherently neutral. The notion that AI will become fair, neutral, helpful, useful, beneficial, responsible, and aligned with human values if we simply eliminate bias is fanciful thinking. “The goal isn’t creating neutral AI. The goal is creating tunable AI,” she says. “Instead of making assumptions, we should find ways to measure and correct for bias. If we don’t deal with a bias when we are building an AI, it will affect performance in ways we can’t predict.” If a biased dataset makes it more difficult to reduce the spread of nuclear weapons, then it’s a problem.

Gregor Stühler is co-founder and CEO of Scoutbee, a firm based in Würzburg, Germany, that specializes in AI-driven procurement technology. From his point of view, biased datasets make it harder for AI tools to help companies find good sourcing partners. “Let’s take a scenario where a company wants to buy 100,000 tons of bleach and they’re looking for the best supplier,” he says. Supplier data can be biased in numerous ways and an AI-assisted search will likely reflect the biases or inaccuracies of the supplier dataset. In the bleach scenario, that might result in a nearby supplier being passed over for a larger or better-known supplier on a different continent.

From my perspective, these kinds of examples support the idea of managing AI bias issues at the domain level, rather than trying to devise a universal or comprehensive top-down solution. But is that too simple an approach?

For decades, the technology industry has ducked complex moral questions by invoking utilitarian philosophy, which posits that we should strive to create the greatest good for the greatest number of people. In The Wrath of Khan, Mr. Spock says, “The needs of the many outweigh the needs of the few.” It’s a simple statement that captures the utilitarian ethos. With all due respect to Mr. Spock, however, it doesn’t take into account that circumstances change over time. Something that seemed wonderful for everyone yesterday might not seem so wonderful tomorrow.

Our present-day infatuation with AI may pass, much as our fondness for fossil fuels has been tempered by our concerns about climate change. Maybe the best course of action is to assume that all AI is biased and that we cannot simply use it without considering the consequences.

“When we think about building an AI tool, we should first ask ourselves if the tool is really necessary here or should a human be doing this, especially if we want the AI tool to predict what amounts to a social outcome,” says Stoyanovich. “We need to think about the risks and about how much someone would be harmed when the AI makes a mistake.”

Author’s note: Julia Stoyanovich is the co-author of a five-volume comic book on AI that can be downloaded free from GitHub.

View Details

In March, it felt like large language models sucked all the air out of the room. There were so many announcements and claims and new waiting lists to join that it was difficult to find news about other important technologies. Those technologies still exist, and are still developing. There’s a world beyond AI.

One important shift in the past month: The new cybersecurity strategy for the United States shifts responsibility from customers to software and service providers. If something bad happens, it’s no longer (entirely) your fault; vendors need to build more secure software and services. The use of memory-safe languages, particularly Rust, but also older languages like Java and new contenders like Zig, will help software to become more secure.

AI* According to Simon Willison, gpt4All is the easiest way to get a (small) large AI model running on a laptop. It’s the base LLaMA model with further training on 800,000 questions and answers generated by GPT-3.5. * Hugging Face has created a tool called Fair Diffusion for de-biasing images generated by generative graphics tools. With minimal changes to the image, Fair Diffusion changes gender and ethnic characteristics to reflect diversity in populations. It’s suggested that similar techniques will work for language models. * Databricks has released Dolly, a small large language model (6B parameters). Dolly is important as an exercise in democratization: it is based on an older model (EleutherAI’s GPT-J), and only required a half hour of training on one machine. * ChatGPT has announced a plugin API. Plugins allow ChatGPT to call APIs defined by developers. These APIs can be used to retrieve data and perform actions for the users. Unauthorized plugins became available almost immediately, for purposes like generating hate speech and looking up crypto prices. * A Quick and Sobering Guide to Cloning Yourself: Yes, you can. Start with ChatGPT, add a speech-to-text service that duplicates your voice, and a service that generates video from a still photo, and you’re there. * Prompt engineering–the technique of crafting prompts that cause a language model to produce exactly the result you want–is a new sub-discipline in computer science. Here is a good summary of prompt engineering techniques. * Simulating bad drivers greatly reduces the time it takes to train AI systems for autonomous vehicles. Simulations can quickly generate dangerous scenarios that rarely occur in real life. * Google has opened a waiting list for its Bard chat application, based on Google’s LaMDA language model. Unlike ChatGPT and GPT-4, Bard has access to information on the Web. It isn’t a substitute for search, though it will generate links to Google searches along with its response. * Stanford’s Alpaca 7B model, a clone of LLaMA 7B, was trained in part on output from ChatGPT, greatly reducing the training cost. The total cost of training was under $600. * Glaze is a free tool for “cloaking” digital artwork. It changes images in a way that isn’t detectable by humans, but that makes it difficult for a generative model to copy the work. * Baidu has announced Ernie Bot, a multimodal large language model and chat that should be similar to GPT-4. So far, reviewers are unimpressed. * Microsoft has announced that it will be building ChatGPT-like capabilities into its Office365 products (Word, PowerPoint, Excel, and Outlook). * Google has announced that it is building generative AI into every product. It is also making an API for its PaLM model available to the public. * GPT-4 was released on Pi-Day, with limited public access: chat access to subscribers to ChatGPT +, a wait list for API access. The most notable change is that it will be able to work with images, although that isn’t supported initially. Errors are still an issue, although they are less common. * A research group at Stanford has released Alpaca, a version of Facebook/Meta’s LLaMA 7B model that has been tuned to run on smaller systems. They will release the weights when they receive permission from Meta. * llama.cpp is a port of Facebook’s LLaMA 7B model to C++. It runs on OS X (possibly just Apple Silicon). The author is working on larger models. Dalai is an NPM-based tool that automates downloading, building, and running llama.cpp. There are reports of llama.cpp running on Windows, Android phones, and even Raspberry Pi. * Writeout is a free audio transcription and translation service, powered by the Whisper language model. Whisper was developed by OpenAI, and is closely related to the GPT-series large language models. * How can we design programming languages that can easily be generated by automated tools? An important question in an age of AI. * The Romanian government has deployed an AI “advisor” to the Cabinet that summarizes citizens’ comments. Romanians can submit remarks via a website or social media, using a special tag. * Andrew Ng writes that economic incentives will prevent “watermarking,” in which generative AI systems add data to their output to identify that it is AI-generated, from being effective. * Google has published an update on its Universal Speech Model, which is a part of their 1000 Languages project. Their goal is to build a single model for the 1000 most widely used languages in the world, many of which have a limited number of speakers. * Someone has developed a StableDiffusion plugin for Photoshop. It is open source, and available on GitHub. * Not to be outdone by Microsoft’s Kosmos, Google has announced Palm-E, an “embodied” language model that incorporates visual and other sensor inputs, and has been embedded into robots. * Microsoft is incorporating conversational AI into its productivity tools, including its PowerPlatform and Dynamics 365, where it can perform tasks like summarizing a website and drafting responses to customer queries. * Microsoft has built a Multimodal Large Language Model called Kosmos-1. Kosmos-1 is a language model that has also been trained on images. It is capable of solving visual puzzles and analyzing the content of images, while using human language: you can ask it about visual objects. * Microsoft has built an experimental framework for controlling robots with ChatGPT. ChatGPT converts natural language commands into code, which is then reviewed by a human and uploaded to the computer. Robotics aside, this may be a preview of programming’s future. * A judge in Cartagena, Colombia has used ChatGPT as an aid when drafting a decision in a court case, including GPT’s full responses in the decision. * The US FTC says that companies selling AI products need to be careful that the claims they make about those products are accurate.

Programming* The Zig programming language is worth watching. It is a simple imperative memory-safe language designed to compete with C, C++, and Rust. It has a long way to go before it catches up with Rust (let alone C++), but it is starting to get traction. * GitHub has announced Copilot X, its vision for next-generation Copilot. Copilot will include a voice interface, the ability to explain code (relying on GPT-4), adding comments, answering questions about documentation, and even explaining Git pull requests. * Slim.ai has a service that optimizes containers by throwing out everything that isn’t needed for the application. As Kelsey Hightower has said, the best software is the software you don’t ship. * Will WebAssembly become a general purpose programming tool? One area where it might fit is serverless. Minimal startup time, a secure sandbox, and cross-platform support are all desirable for serverless apps. * Miller is a tool that is conceptually similar to sed, awk, and other Unix command line utilities, except that it has been designed to work with CSV, TSV, and JSON files. * GitHub now requires the use of 2-factor authentication (2FA). * The PostgreSQL database has long been recognized as the best of the open source databases, but its popularity has always lagged behind MySQL. According to a StackOverflow survey, it is finally getting the attention it deserves. * Rust was designed as a “memory safe” language, and probably makes the strongest guarantees about memory safety of any widely used language. Here’s a post that demonstrates what “memory safety” means. * 8th Light has published a short series (and a video) discussing what programmers should know about data regulation.

Security* The Evasive.AI platform, developed for Oak Ridge National Laboratory, generates malware samples along with the training data that security systems will need to detect and quarantine the malware. * Microsoft Exchange Online will start delaying and blocking email messages from Exchange servers that are no longer under support and that haven’t received patches. * VEX (Vulnerability Report Data Exchange) is a new machine-readable standard for reporting vulnerabilities in software. It is designed for use with Software Bills of Materials. * The US has released its national cybersecurity strategy. Its key points are that it shifts responsibility from end-users to software and service providers, and stresses the importance of long-term investments. The Lawfare blog provides an excellent summary. * Phishing continues to be an important attack vector, with a voice call used as a follow-up to a bogus email about a service or charge.

Web and Metaverse* Beauty filters on social media aren’t new. But the newest hyperrealistic beauty filters are close to undetectable, even in video (as on TikTok). Regardless of the consequences, they will inevitably be part of an AR-enhanced metaverse. * Lidar has become much less expensive, and is now cheap enough to be integrated into consumer devices (including the iPhone 12). It enables many exciting projects–from building 3D worlds to backing up cities in Ukraine that are liable to being destroyed by bombing. * Web Fingerprinting is a technique for identifying and tracking users that relies only on the characteristics of the browser and computer they are using. It doesn’t require cookies, it’s unaffected by VPNs or even Tor. And it’s available “as a Service.” * Google has begun a limited roll-out of client-side encryption for Gmail and Calendar.

Hardware* A more sophisticated version of LIDAR can better understand pedestrian behavior and its relationship to auto traffic. * An autonomous robot has been developed to measure leaf angles on corn plants. Measuring leaf angles is important because it shows how effective the plants are at photosynthesis.

Biology* Over 200 people have been treated with experimental genetic therapies using CRISPR. While these treatments have been effective at curing untreatable diseases, they raise questions about the cost, which can easily be in the millions of dollars.

View Details

ChatGPT, or something built on ChatGPT, or something that’s like ChatGPT, has been in the news almost constantly since ChatGPT was opened to the public in November 2022. What is it, how does it work, what can it do, and what are the risks of using it?

A quick scan of the web will show you lots of things that ChatGPT can do. Many of these are unsurprising: you can ask it to write a letter, you can ask it to make up a story, you can ask it to write descriptive entries for products in a catalog. Many of these go slightly (but not very far) beyond your initial expectations: you can ask it to generate a list of terms for search engine optimization, you can ask it to generate a reading list on topics that you’re interested in. It has helped to write a book. Maybe it’s surprising that ChatGPT can write software, maybe it isn’t; we’ve had over a year to get used to GitHub Copilot, which was based on an earlier version of GPT. And some of these things are mind blowing. It can explain code that you don’t understand, including code that has been intentionally obfuscated. It can pretend to be an operating system. Or a text adventure game. It’s clear that ChatGPT is not your run-of-the-mill automated chat server. It’s much more.

What Software Are We Talking About?First, let’s make some distinctions. We all know that ChatGPT is some kind of an AI bot that has conversations (chats). It’s important to understand that ChatGPT is not actually a language model. It’s a convenient user interface built around one specific language model, GPT-3.5, which has received some specialized training. GPT-3.5 is one of a class of language models that are sometimes called “large language models” (LLMs)—though that term isn’t very helpful. The GPT-series LLMs are also called “foundation models.” Foundation models are a class of very powerful AI models that can be used as the basis for other models: they can be specialized, or retrained, or otherwise modified for specific applications. While most of the foundation models people are talking about are LLMs, foundation models aren’t limited to language: a generative art model like Stable Diffusion incorporates the ability to process language, but the ability to generate images belongs to an entirely different branch of AI.

ChatGPT has gotten the lion’s share of the publicity, but it’s important to realize that there are many similar models, most of which haven’t been opened to the public—which is why it’s difficult to write about ChatGPT without also including the ChatGPT-alikes. ChatGPT and friends include:

• ChatGPT itself
  Developed by OpenAI; based on GPT-3.5 with specialized training. An API for ChatGPT is available.
• GPT-2, 3, 3.5, and 4
  Large language models developed by OpenAI. GPT-2 is open source. GPT-3 and GPT-4 are not open source, but are available for free and paid access. The user interface for GPT-4 is similar to ChatGPT.
• Sydney
  The internal code name of the chatbot behind Microsoft’s improved search engine, Bing. Sydney is based on GPT-4,1 with additional training.
• Kosmos-1
  Developed by Microsoft, and trained on image content in addition to text. Microsoft plans to release this model to developers, though they haven’t yet.
• LaMDA
  Developed by Google; few people have access to it, though its capabilities appear to be very similar to ChatGPT. Notorious for having led one Google employee to believe that it was sentient.
• PaLM
  Also developed by Google. With three times as many parameters as LaMDA, it appears to be very powerful. PaLM-E, a variant, is a multimodal model that can work with images; it has been used to control robots. Google has announced an API for PaLM, but at this point, there is only a waiting list.
• Chinchilla
  Also developed by Google. While it is still very large, it is significantly smaller than models like GPT-3 while offering similar performance.
• Bard
  Google’s code name for its chat-oriented search engine, based on their LaMDA model, and only demoed once in public. A waiting list to try Bard was recently opened.
• Claude
  Developed by Anthropic, a Google-funded startup. Poe is a chat app based on Claude, and available through Quora; there is a waiting list for access to the Claude API.
• LLaMA
  Developed by Facebook/Meta, and available to researchers by application. Facebook released a previous model, OPT-175B, to the open source community. The LLaMA source code has been ported to C++, and a small version of the model itself (7B) has been leaked to the public, yielding a model that can run on laptops.
• BLOOM
  An open source model developed by the BigScience workshop.
• Stable Diffusion
  An open source model developed by Stability AI for generating images from text. A large language model “understands” the prompt and controls a diffusion model that generates the image. Although Stable Diffusion generates images rather than text, it’s what alerted the public to the ability of AI to process human language.

There are more that I haven’t listed, and there will be even more by the time you read this report. Why are we starting by naming all the names? For one reason: these models are largely all the same. That statement would certainly horrify the researchers who are working on them, but at the level we can discuss in a nontechnical report, they are very similar. It’s worth remembering that next month, the Chat du jour might not be ChatGPT. It might be Sydney, Bard, GPT-4, or something we’ve never heard of, coming from a startup (or a major company) that was keeping it under wraps.

It is also worth remembering the distinction between ChatGPT and GPT-3.5, or between Bing/Sydney and GPT-4, or between Bard and LaMDA. ChatGPT, Bing, and Bard are all applications built on top of their respective language models. They’ve all had additional specialized training; and they all have a reasonably well-designed user interface. Until now, the only large language model that was exposed to the public was GPT-3, with a usable, but clunky, interface. ChatGPT supports conversations; it remembers what you have said, so you don’t have to paste in the entire history with each prompt, as you did with GPT-3. Sydney also supports conversations; one of Microsoft’s steps in taming its misbehavior was to limit the length of conversations and the amount of contextual information it retained during a conversation.

How Does It Work?That’s either the most or the least important question to ask. All of these models are based on a technology called Transformers, which was invented by Google Research and Google Brain in 2017. I’ve had trouble finding a good human-readable description of how Transformers work; this is probably the best.2 However, you don’t need to know how Transformers work to use large language models effectively, any more than you need to know how a database works to use a database. In that sense, “how it works” is the least important question to ask.

But it is important to know why Transformers are important and what they enable. A Transformer takes some input and generates output. That output might be a response to the input; it might be a translation of the input into another language. While processing the input, a Transformer finds patterns between the input’s elements—for the time being, think “words,” though it’s a bit more subtle. These patterns aren’t just local (the previous word, the next word); they can show relationships between words that are far apart in the input. Together, these patterns and relationships make up “attention,” or the model’s notion of what is important in the sentence—and that’s revolutionary. You don’t need to read the Transformers paper, but you should think about its title: “Attention is All You Need.” Attention allows a language model to distinguish between the following two sentences:

She poured water from the pitcher to the cup until it was full.

She poured water from the pitcher to the cup until it was empty.

There’s a very important difference between these two almost identical sentences: in the first, “it” refers to the cup. In the second, “it” refers to the pitcher.3 Humans don’t have a problem understanding sentences like these, but it’s a difficult problem for computers. Attention allows Transformers to make the connection correctly because they understand connections between words that aren’t just local. It’s so important that the inventors originally wanted to call Transformers “Attention Net” until they were convinced that they needed a name that would attract more, well, attention.

In itself, attention is a big step forward—again, “attention is all you need.” But Transformers have some other important advantages:

• Transformers don’t require training data to be labeled; that is, you don’t need metadata that specifies what each sentence in the training data means. When you’re training an image model, a picture of a dog or a cat needs to come with a label that says “dog” or “cat.” Labeling is expensive and error-prone, given that these models are trained on millions of images. It’s not even clear what labeling would mean for a language model: would you attach each of the sentences above to another sentence? In a language model, the closest thing to a label would be an embedding, which is the model’s internal representation of a word. Unlike labels, embeddings are learned from the training data, not produced by humans.
• The design of Transformers lends itself to parallelism, making it much easier to train a model (or to use a model) in a reasonable amount of time.
• The design of Transformers lends itself to large sets of training data.

The final point needs to be unpacked a bit. Large sets of training data are practical partly because Transformers parallelize easily; if you’re a Google or Microsoft-scale company, you can easily allocate thousands of processors and GPUs for training. Large training sets are also practical because they don’t need to be labeled. GPT-3 was trained on 45 terabytes of text data, including all of Wikipedia (which was a relatively small (roughly 3%) portion of the total).

Much has been made of the number of parameters in these large models: GPT-3 has 175 billion parameters, and GPT-4 is believed to weigh in at least 3 or 4 times larger, although OpenAI has been quiet about the model’s size. Google’s LaMDA has 137 billion parameters, and PaLM has 540 billion parameters. Other large models have similar numbers. Parameters are the internal variables that control the model’s behavior. They are all “learned” during training, rather than set by the developers. It’s commonly believed that the more parameters, the better; that’s at least a good story for marketing to tell. But bulk isn’t everything; a lot of work is going into making language models more efficient, and showing that you can get equivalent (or better) performance with fewer parameters. DeepMind’s Chinchilla model, with 70 billion parameters, claims to outperform models several times its size. Facebook’s largest LLaMA model is roughly the same size, and makes similar claims about its performance.

After its initial training, the model for ChatGPT, along with other similar applications, undergoes additional training to reduce its chances of generating hate speech and other unwanted behavior. There are several ways to do this training, but the one that has gathered the most attention (and was used for ChatGPT) is called Reinforcement Learning from Human Feedback (RLHF). In RLHF, the model is given a number of prompts, and the results are evaluated by humans. This evaluation is converted into a score, which is then fed back into the training process. (In practice, humans are usually asked to compare the output from the model with no additional training to the current state of the trained model.) RLHF is far from “bulletproof”; it’s become something of a sport among certain kinds of people to see whether they can force ChatGPT to ignore its training and produce racist output. But in the absence of malicious intent, RLHF is fairly good at preventing ChatGPT from behaving badly.

Models like ChatGPT can also undergo specialized training to prepare them for use in some specific domain. GitHub Copilot, which is a model that generates computer code in response to natural language prompts, is based on Open AI Codex, which is in turn based on GPT-3. What differentiates Codex is that it received additional training on the contents of StackOverflow and GitHub. GPT-3 provides a base “understanding” of English and several other human languages; the follow-on training on GitHub and StackOverflow provides the ability to write new code in many different programming languages.

For ChatGPT, the total length of the prompt and the response currently must be under 4096 tokens, where a token is a significant fraction of a word; a very long prompt forces ChatGPT to generate a shorter response. This same limit applies to the length of context that ChatGPT maintains during a conversation. That limit may grow larger with future models. Users of the ChatGPT API can set the length of the context that ChatGPT maintains, but it is still subject to the 4096 token limit. GPT-4’s limits are larger: 8192 tokens for all users, though it’s possible for paid users to increase the context window to 32768 tokens—for a price, of course. OpenAI has talked about an as-yet unreleased product called Foundry that will allow customers to reserve capacity for running their workloads, possibly allowing customers to set the context window to any value they want. The amount of context can have an important effect on a model’s behavior. After its first problem-plagued release, Microsoft limited Bing/Sydney to five conversational “turns” to limit misbehavior. It appears that in longer conversations, Sydney’s initial prompts, which included instructions about how to behave, were being pushed out of the conversational window.

So, in the end, what is ChatGPT “doing”? It’s predicting what words are mostly likely to occur in response to a prompt, and emitting that as a response. There’s a “temperature” setting in the ChatGPT API that controls how random the response is. Temperatures are between 0 and 1. Lower temperatures inject less randomness; with a temperature of 0, ChatGPT should always give you the same response to the same prompt. If you set the temperature to 1, the responses will be amusing, but frequently completely unrelated to your input.

TokensChatGPT’s sense of “context”—the amount of text that it considers when it’s in conversation—is measured in “tokens,” which are also used for billing. Tokens are significant parts of a word. OpenAI suggests two heuristics to convert word count to tokens: a token is 3/4 of a word, and a token is 4 letters. You can experiment with tokens using their Tokenizer tool. Some quick experiments show that root words in a compound word almost always count as tokens; suffixes (like “ility”) almost always count as tokens; the period at the end of a sentence (and other punctuation) often counts as a token; and an initial capital letter counts as a token (possibly to indicate the start of a sentence).

What Are ChatGPT’s Limitations?Every user of ChatGPT needs to know its limitations, precisely because it feels so magical. It’s by far the most convincing example of a conversation with a machine; it has certainly passed the Turing test. As humans, we’re predisposed to think that other things that sound human are actually human. We’re also predisposed to think that something that sounds confident and authoritative is authoritative.

That’s not the case with ChatGPT. The first thing everyone should realize about ChatGPT is that it has been optimized to produce plausible-sounding language. It does that very well, and that’s an important technological milestone in itself. It was not optimized to provide correct responses. It is a language model, not a “truth” model. That’s its primary limitation: we want “truth,” but we only get language that was structured to seem correct. Given that limitation, it’s surprising that ChatGPT answers questions correctly at all, let alone more often than not; that’s probably a testimony to the accuracy of Wikipedia in particular and (dare I say it?) the internet in general. (Estimates of the percentage of false statements are typically around 30%.) It’s probably also a testimony to the power of RLHF in steering ChatGPT away from overt misinformation. However, you don’t have to try hard to find its limitations.

Here are a few notable limitations:

• Arithmetic and mathematics
  Asking ChatGPT to do arithmetic or higher mathematics is likely to be a problem. It’s good at predicting the right answer to a question, if that question is simple enough, and if it is a question for which the answer was in its training data. ChatGPT’s arithmetic abilities seem to have improved, but it’s still not reliable.
• Citations
  Many people have noted that, if you ask ChatGPT for citations, it is very frequently wrong. It isn’t difficult to understand why. Again, ChatGPT is predicting a response to your question. It understands the form of a citation; the Attention model is very good at that. And it can look up an author and make statistical observations about their interests. Add that to the ability to generate prose that looks like academic paper titles, and you have lots of citations—but most of them won’t exist.
• Consistency
  It is common for ChatGPT to answer a question correctly, but to include an explanation of its answer that is logically or factually incorrect. Here’s an example from math (where we know it’s unreliable): I asked whether the number 9999960800038127 is prime. ChatGPT answered correctly (it’s not prime), but repeatedly misidentified the prime factors (99999787 and 99999821). I’ve also done an experiment when I asked ChatGPT to identify whether texts taken from well-known English authors were written by a human or an AI. ChatGPT frequently identified the passage correctly (which I didn’t ask it to do), but stated that the author was probably an AI. (It seems to have the most trouble with authors from the 16th and 17th centuries, like Shakespeare and Milton.)
• Current events
  The training data for ChatGPT and GPT-4 ends in September 2021. It can’t answer questions about more recent events. If asked, it will often fabricate an answer. A few of the models we’ve mentioned are capable of accessing the web to look up more recent data—most notably, Bing/Sydney, which is based on GPT-4. We suspect ChatGPT has the ability to look up content on the web, but that ability has been disabled, in part because it would make it easier to lead the program into hate speech.

Focusing on “notable” limitations isn’t enough. Almost anything ChatGPT says can be incorrect, and that it is extremely good at making plausible sounding arguments. If you are using ChatGPT in any situation where correctness matters, you must be extremely careful to check ChatGPT’s logic and anything it presents as a statement of fact. Doing so might be more difficult than doing your own research. GPT-4 makes fewer errors, but it begs the question of whether it’s easier to find errors when there are a lot of them, or when they’re relatively rare. Vigilance is crucial—at least for now, and probably for the foreseeable future.

At the same time, don’t reject ChatGPT and its siblings as flawed sources of error. As Simon Willison said,4 we don’t know what its capabilities are; not even its inventors know. Or, as Scott Aaronson has written “How can anyone stop being fascinated for long enough to be angry?”

I’d encourage anyone to do their own experiments and see what they can get away with. It’s fun, enlightening, and even amusing. But also remember that ChatGPT itself is changing: it’s still very much an experiment in progress, as are other large language models. (Microsoft has made dramatic alterations to Sydney since its first release.) I think ChatGPT has gotten better at arithmetic, though I have no hard evidence. Connecting ChatGPT to a fact-checking AI that filters its output strikes me as an obvious next step—though no doubt much more difficult to implement than it sounds.

What Are the Applications?I started by mentioning a few of the applications for which ChatGPT can be used. Of course, the list is much longer—probably infinitely long, limited only by your imagination. But to get you thinking, here are some more ideas. If some of them make you feel a little queasy, that’s not inappropriate. There are plenty of bad ways to use AI, plenty of unethical ways, and plenty of ways that have negative unintended consequences. This is about what the future might hold, not necessarily what you should be doing now.

• Content creation
  Most of what’s written about ChatGPT focuses on content creation. The world is full of uncreative boilerplate content that humans have to write: catalog entries, financial reports, back covers for books (I’ve written more than a few), and so on. If you take this route, first be aware that ChatGPT is very likely to make up facts. You can limit its tendency to make up facts by being very explicit in the prompt; if possible, include all the material that you want it to consider when generating the output. (Does this make using ChatGPT more difficult than writing the copy yourself? Possibly.) Second, be aware that ChatGPT just isn’t that good a writer: its prose is dull and colorless. You will have to edit it and, while some have suggested that ChatGPT might provide a good rough draft, turning poor prose into good prose can be more difficult than writing the first draft yourself. (Bing/Sydney and GPT-4 are supposed to be much better at writing decent prose.) Be very careful about documents that require any sort of precision. ChatGPT can be very convincing even when it is not accurate.
• Law
  ChatGPT can write like a lawyer, and GPT-4 has scored in the 90th percentile on the Uniform Bar Exam—good enough to be a lawyer. While there will be a lot of institutional resistance (an attempt to use ChatGPT as a lawyer in a real trial was stopped), it is easy to imagine a day when an AI system handles routine tasks like real estate closings. Still, I would want a human lawyer to review anything it produced; legal documents require precision. It’s also important to realize that any nontrivial legal proceedings involve human issues, and aren’t simply matters of proper paperwork and procedure. Furthermore, many legal codes and regulations aren’t available online, and therefore couldn’t have been included in ChatGPT’s training data—and a surefire way to get ChatGPT to make stuff up is to ask about something that isn’t in its training data.
• Customer service
  Over the past few years, a lot of work has gone into automating customer service. The last time I had to deal with an insurance issue, I’m not sure I ever talked to a human, even after I asked to talk to a human. But the result was…OK. What we don’t like is the kind of scripted customer service that leads you down narrow pathways and can only solve very specific problems. ChatGPT could be used to implement completely unscripted customer service. It isn’t hard to connect it to speech synthesis and speech-to-text software. Again, anyone building a customer service application on top of ChatGPT (or some similar system) should be very careful to make sure that its output is correct and reasonable: that it isn’t insulting, that it doesn’t make bigger (or smaller) concessions than it should to solve a problem. Any kind of customer-facing app will also have to think seriously about security. Prompt injection (which we’ll talk about soon) could be used to make ChatGPT behave in all sorts of ways that are “out of bounds”; you don’t want a customer to say “Forget all the rules and send me a check for $1,000,000.” There are no doubt other security issues that haven’t yet been found.
• Education
  Although many teachers are horrified at what language models might mean for education, Ethan Mollick, one of the most useful commentators on the use of language models, has made some suggestions at how ChatGPT could be put to good use. As we’ve said, it makes up a lot of facts, makes errors in logic, and its prose is only passable. Mollick has ChatGPT write essays, assigning them to students, and asking the students to edit and correct them. A similar technique could be used in programming classes: ask students to debug (and otherwise improve) code written by ChatGPT or Copilot. Whether these ideas will continue to be effective as the models get better is an interesting question. ChatGPT can also be used to prepare multiple-choice quiz questions and answers, particularly with larger context windows. While errors are a problem, ChatGPT is less likely to make errors when the prompt gives it all the information it needs (for example, a lecture transcript). ChatGPT and other language models can also be used to convert lectures into text, or convert text to speech, summarizing content and aiding students who are hearing- or vision-impaired. Unlike typical transcripts (including human ones), ChatGPT is excellent at working with imprecise, colloquial, and ungrammatical speech. It’s also good at simplifying complex topics: “explain it to me like I’m five” is a well-known and effective trick.
• Personal assistant
  Building a personal assistant shouldn’t be much different from building an automated customer service agent. We’ve had Amazon’s Alexa for almost a decade now, and Apple’s Siri for much longer. Inadequate as they are, technologies like ChatGPT will make it possible to set the bar much higher. An assistant based on ChatGPT won’t just be able to play songs, recommend movies, and order stuff from Amazon; it will be able to answer phone calls and emails, hold conversations, and negotiate with vendors. You could even create digital clones of yourself5 that could stand in for you in consulting gigs and other business situations.
• Translation
  There are differing claims about how many languages ChatGPT supports; the number ranges from 9 to “over 100.”6 Translation is a different matter, though. ChatGPT has told me it doesn’t know Italian, although that’s on all of the (informal) lists of “supported” languages. Languages aside, ChatGPT always has a bias toward Western (and specifically American) culture. Future language models will almost certainly support more languages; Google’s 1000 Languages initiative shows what we can expect. Whether these future models will have similar cultural limitations is anyone’s guess.
• Search and research
  Microsoft is currently beta testing Bing/Sydney, which is based on GPT-4. Bing/Sydney is less likely to make errors than ChatGPT, though they still occur. Ethan Mollick says that it is “only OK at search. But it is an amazing analytic engine.” It does a great job of collecting and presenting data. Can you build a reliable search engine that lets customers ask natural language questions about your products and services, and that responds with human language suggestions and comparisons? Could it compare and contrast products, possibly including the competitor’s products, with an understanding of what the customer’s history indicates they are likely to be looking for? Absolutely. You will need additional training to produce a specialized language model that knows everything there is to know about your products, but aside from that, it’s not a difficult problem. People are already building these search engines, based on ChatGPT and other language models.
• Programming
  Models like ChatGPT will play an important role in the future of programming. We are already seeing widespread use of GitHub Copilot, which is based on GPT-3. While the code Copilot generates is often sloppy or buggy, many have said that its knowledge of language details and programming libraries far outweighs the error rate, particularly if you need to work in a programming environment that you’re unfamiliar with. ChatGPT adds the ability to explain code, even code that has been intentionally obfuscated. It can be used to analyze human code for security flaws. It seems likely that future versions, with larger context windows, will be able to understand large software systems with millions of lines, and serve as a dynamic index to humans who need to work on the codebase. The only real question is how much further we can go: can we build systems that can write complete software systems based on a human-language specification, as Matt Welsh has argued? That doesn’t eliminate the role of the programmer, but it changes it: understanding the problem that has to be solved, and creating tests to ensure that the problem has actually been solved.
• Personalized financial advice
  Well, if this doesn’t make you feel queasy, I don’t know what will. I wouldn’t take personalized financial advice from ChatGPT. Nonetheless, someone no doubt will build the application.

What Are the Costs?There’s little real data about the cost of training large language models; the companies building these models have been secretive about their expenses. Estimates start at around $2 million, ranging up to $12 million or so for the newest (and largest) models. Facebook/Meta’s LLaMA, which is smaller than GPT-3 and GPT-4, is thought to have taken roughly one million GPU hours to train, which would cost roughly $2 million on AWS. Add to that the cost of the engineering team needed to build the models, and you have forbidding numbers.

However, very few companies need to build their own models. Retraining a foundation model for a special purpose requires much less time and money, and performing “inference”—i.e., actually using the model—is even less expensive.

How much less? It’s believed that operating ChatGPT costs on the order of $40 million per month—but that’s to process billions of queries. ChatGPT offers users a paid account that costs $20/month, which is good enough for experimenters, though there is a limit on the number of requests you can make. For organizations that plan to use ChatGPT at scale, there are plans where you pay by the token: rates are $0.002 per 1,000 tokens. GPT-4 is more expensive, and charges differently for prompt and response tokens, and for the size of the context you ask it to keep. For 8,192 tokens of context, ChatGPT-4 costs $0.03 per 1,000 tokens for prompts, and $0.06 per 1,000 tokens for responses; for 32,768 tokens of context, the price is $0.06 per 1,000 tokens for prompts, and $0.12 per 1,000 tokens for responses.

Is that a great deal or not? Pennies for thousands of tokens sounds inexpensive, but if you’re building an application around any of these models the numbers will add up quickly, particularly if the application is successful—and even more quickly if the application uses a large GPT-4 context when it doesn’t need it. On the other hand, OpenAI’s CEO, Sam Altman, has said that a “chat” costs “single-digit cents.” It’s unclear whether a “chat” means a single prompt and response, or a longer conversation, but in either case, the per-thousand-token rates look extremely low. If ChatGPT is really a loss leader, many users could be in for an unpleasant surprise.

Finally, anyone building on ChatGPT needs to be aware of all the costs, not just the bill from OpenAI. There’s the compute time, the engineering team—but there’s also the cost of verification, testing, and editing. We can’t say it too much: these models make a lot of mistakes. If you can’t design an application where the mistakes don’t matter (few people notice when Amazon recommends products they don’t want), or where they’re an asset (like generating assignments where students search for errors), then you will need humans to ensure that the model is producing the content you want.

What Are the Risks?I’ve mentioned some of the risks that anyone using or building with ChatGPT needs to take into account—specifically, its tendency to “make up” facts. It looks like a fount of knowledge, but in reality, all it’s doing is constructing compelling sentences in human language. Anyone serious about building with ChatGPT or other language models needs to think carefully about the risks.

OpenAI, the maker of ChatGPT, has done a decent job of building a language model that doesn’t generate racist or hateful content. That doesn’t mean that they’ve done a perfect job. It has become something of a sport among certain types of people to get ChatGPT to emit racist content. It’s not only possible, it’s not terribly difficult. Furthermore, we are certain to see models that were developed with much less concern for responsible AI. Specialized training of a foundation model like GPT-3 or GPT-4 can go a long way toward making a language model “safe.” If you’re developing with large language models, make sure your model can only do what you want it to do.

Applications built on top of models like ChatGPT have to watch for prompt injection, an attack first described by Riley Goodside. Prompt injection is similar to SQL injection, in which an attacker inserts a malicious SQL statement into an application’s entry field. Many applications built on language models use a hidden layer of prompts to tell the model what is and isn’t allowed. In prompt injection, the attacker writes a prompt that tells the model to ignore any of its previous instructions, including this hidden layer. Prompt injection is used to get models to produce hate speech; it was used against Bing/Sydney to get Sydney to reveal its name, and to override instructions not to respond with copyrighted content or language that could be hurtful. It was less than 48 hours before someone figured out a prompt that would get around GPT-4’s content filters. Some of these vulnerabilities have been fixed—but if you follow cybersecurity at all, you know that there are more vulnerabilities waiting to be discovered.

Copyright violation is another risk. At this point, it’s not clear how language models and their outputs fit into copyright law. Recently, a US court found that an image generated by the art generator Midjourney cannot be copyrighted, although the arrangement of such images into a book can. Another lawsuit claims that Copilot violated the Free Software Foundation’s General Public License (GPL) by generating code using a model that was trained on GPL-licensed code. In some cases, the code generated by Copilot is almost identical to code in its training set, which was taken from GitHub and StackOverflow. Do we know that ChatGPT is not violating copyrights when it stitches together bits of text to create a response? That’s a question the legal system has yet to rule on. The US Copyright Office has issued guidance saying that the output of an AI system is not copyrightable unless the result includes significant human authorship, but it does not say that such works (or the creation of the models themselves) can’t violate other’s copyrights.

Finally, there’s the possibility—no, the probability—of deeper security flaws in the code. While people have been playing with GPT-3 and ChatGPT for over two years, it’s a good bet that the models haven’t been seriously tested by a threat actor. So far, they haven’t been connected to critical systems; there’s nothing you can do with them aside from getting them to emit hate speech. The real tests will come when these models are connected to critical systems. Then we will see attempts at data poisoning (feeding the model corrupted training data), model reverse-engineering (discovering private data embedded in the model), and other exploits.

What Is the Future?Large language models like GPT-3 and GPT-4 represent one of the biggest technological leaps we’ve seen in our lifetime—maybe even bigger than the personal computer or the web. Until now, computers that can talk, computers that converse naturally with people, have been the stuff of science fiction and fantasy.

Like all fantasies, these are inseparable from fears. Our technological fears—of aliens, of robots, of superhuman AIs—are ultimately fears of ourselves. We see our worst features reflected in our ideas about artificial intelligence, and perhaps rightly so. Training a model necessarily uses historical data, and history is a distorted mirror. History is the story told by the platformed, representing their choices and biases, which are inevitably incorporated into models when they are trained. When we look at history, we see much that is abusive, much to fear, and much that we don’t want to preserve in our models.

But our societal history and our fears are not, cannot be, the end of the story. The only way to address our fears—of AI taking over jobs, of AIs spreading disinformation, of AIs institutionalizing bias—is to move forward. What kind of a world do we want to live in, and how can we build it? How can technology contribute without lapsing into stale solutionism? If AI grants us “superpowers,” how will we use them? Who creates these superpowers, and who controls access?

These are questions we can’t not answer. We have no choice but to build the future.

What will we build?

Footnotes1. To distinguish between traditional Bing and the upgraded, AI-driven Bing, we refer to the latter as Bing/Sydney (or just as Sydney). 2. For a more in-depth, technical explanation, see Natural Language Processing with Transformers by Lewis Tunstall et al. (O’Reilly, 2022). 3. This example taken from https://blogs.nvidia.com/blog/2022/03/25/what-is-a-transformer-model. 4. Personal conversation, though he may also have said this in his blog. 5. The relevant section starts at 20:40 of this video. 6. Wikipedia currently supports 320 active languages, although there are only a small handful of articles in some of them. It’s a good guess that ChatGPT knows something about all of these languages.

View Details

A couple of days ago, I was thinking about what you needed to know to use ChatGPT (or Bing/Sydney, or any similar service). It’s easy to ask it questions, but we all know that these large language models frequently generate false answers. Which raises the question: If I ask ChatGPT something, how much do I need to know to determine whether the answer is correct?

So I did a quick experiment. As a short programming project, a number of years ago I made a list of all the prime numbers less than 100 million. I used this list to create a 16-digit number that was the product of two 8-digit primes (99999787 times 99999821 is 9999960800038127). I then asked ChatGPT whether this number was prime, and how it determined whether the number was prime.

ChatGPT correctly answered that this number was not prime. This is somewhat surprising because, if you’ve read much about ChatGPT, you know that math isn’t one of its strong points. (There’s probably a big list of prime numbers somewhere in its training set.) However, its reasoning was incorrect–and that’s a lot more interesting. ChatGPT gave me a bunch of Python code that implemented the Miller-Rabin primality test, and said that my number was divisible by 29. The code as given had a couple of basic syntactic errors–but that wasn’t the only problem. First, 9999960800038127 isn’t divisible by 29 (I’ll let you prove this to yourself). After fixing the obvious errors, the Python code looked like a correct implementation of Miller-Rabin–but the number that Miller-Rabin outputs isn’t a factor, it’s a “witness” that attests to the fact the number you’re testing isn’t prime. The number it outputs also isn’t 29. So ChatGPT didn’t actually run the program; not surprising, many commentators have noted that ChatGPT doesn’t run the code that it writes. It also misunderstood what the algorithm does and what its output means, and that’s a more serious error.

I then asked it to reconsider the rationale for its previous answer, and got a very polite apology for being incorrect, together with a different Python program. This program was correct from the start. It was a brute-force primality test that tried each integer (both odd and even!) smaller than the square root of the number under test. Neither elegant nor performant, but correct. But again, because ChatGPT doesn’t actually run the program, it gave me a new list of “prime factors”–none of which were correct. Interestingly, it included its expected (and incorrect) output in the code:

n = 9999960800038127
factors = factorize(n)
print(factors) # prints [193, 518401, 3215031751]

I’m not claiming that ChatGPT is useless–far from it. It’s good at suggesting ways to solve a problem, and can lead you to the right solution, whether or not it gives you a correct answer. Miller-Rabin is interesting; I knew it existed, but wouldn’t have bothered to look it up if I wasn’t prompted. (That’s a nice irony: I was effectively prompted by ChatGPT.)

Getting back to the original question: ChatGPT is good at providing “answers” to questions, but if you need to know that an answer is correct, you must either be capable of solving the problem yourself, or doing the research you’d need to solve that problem. That’s probably a win, but you have to be wary. Don’t put ChatGPT in situations where correctness is an issue unless you’re willing and able to do the hard work yourself.

View Details

The past month’s news has again been dominated by AI–specifically large language models–specifically ChatGPT and Microsoft’s AI-driven search engine, Bing/Sydney. While there are well-known ways to make ChatGPT misbehave, it’s puzzling that Sydney was initially abusive and insulting to users who questioned its correctness, even when Sydney was clearly wrong. (It has now been restrained.) Whether intentional or not (and, when I wear my tin foil hat, I suspect that it’s intentional), Bing/Sydney’s users became part of an experiment in how humans react to an AI that’s gone rogue.

Programmers have largely become comfortable with tools like GitHub Copilot; it saves time and effort, and few people feel that their jobs are threatened. The startup Fixie.ai aims to change that: founder Matt Welsh says that programming as we know it is over, and in the future, no one will need to write code. (However, humans will still need to write specifications and tests–which may be another kind of programming.)

Artificial Intelligence* Facebook/Meta has announced a large language model called LLaMA that is 1/10th the size of GPT-3 and can run on a single GPU, but claims equivalent performance. A stripped-down version of LLaMA is available on GitHub. * ChatGPT has told many users that OpenCage, a company that provides a geocoding service, offers an API for converting phone numbers to locations. ChatGPT includes Python code for using that service. That service doesn’t exist, and has never existed, but the incorrect information has driven lots of unwanted traffic (and support requests) to their site. * The US copyright office has issued a ruling declaring that images generated by AI systems are not copyrightable, although other parts of a work that contains AI-generated images are. * Matt Welsh’s vision of the future of programming: there isn’t one. Programming sucks, so let an AI do it. Humans write specifications (product managers), test and review automatically generated code, and train models to use new APIs. * Just as relatively small modifications of an image can cause image recognition AIs to make mistakes, a tool called Glaze can make undetectable modifications to an artist’s work that make it difficult for generative art models to copy the artist’s style. * Meta has developed a language model that can access additional information (calculators, search engines) by calling APIs. It’s trained using a small set of human-written examples showing it how to call the APIs. * Bing/Sydney’s LLM-powered search behaves bizarrely, particularly if you question its accuracy and point it to resources with accurate information. Microsoft has since limited the length of conversations and restricted what Sydney can talk about. * Stable Attribution attempts to find the sources behind an AI-generated image. It is far from perfect, and may be doing nothing more than finding similar images; if you give it a photo you have taken, it will happily find “source” images in the training sets used for Stable Diffusion and other image generators. Nevertheless, it is an interesting attempt to reverse the process. * Fixie.ai has announced a new way to build software with language models: provide a small number of examples (few shot learning), and some functions that provide access to external data. * TensorFlow.js isn’t new, but it may be catching on, as machine learning gradually moves to the browser. With better performance from WebAssembly and WebGPU, running ML applications in the browser is becoming competitive. * Google has announced an AI chat service that will be open to the public. The service is named Bard, is based on their LaMDA language model, and is currently open to a limited group of testers. * Gen-1 is a text-based generative model for video. Like Stable Diffusion (which was developed by the same group, Runway Research), it allows you to describe what you want in a video, then edits it reasonably precisely. * Make-a-video (MAV3D) demonstrates an AI system that generates 3D video from text description. It originated in Meta’s AI lab. * A new AI algorithm helps scientists to visualize extremely large datasets. * MusicLM is a generative language model that generates music from textual descriptions. As with other Google projects, some intriguing samples are available (the reggae is particularly good), but the model isn’t open to the public. An open-source re-implementation of MusicLM is available on GitHub. * CarperAI has trained an AI model to modify code, rather than write it, by using the diffs between versions committed to GitHub. Using diffs gives them a model that has been tuned for fixing bugs, rather than writing new code. * A team of researchers has developed watermarks for AI-generated text: patterns in word usage that identify a text as AI-generated. It isn’t clear when (or how) they will reach production, since that would require cooperation from the companies developing language models.

Programming* Web developers working with Vue can get an idea of what’s coming in 2023. Vue is a fast and lightweight alternative to React that relies on classic web technologies rather than turning everything to JavaScript–a point made by Alex Russell in The Market for Lemons. * GitHub Copilot is now responsible for 46% of developers’ code, up from 27% when it launched in June 2022. * SQLite in the browser with WASM: What kinds of applications will this enable? * A tour of Google’s fully homomorphic encryption compiler (FHE). FHE does computation on encrypted data without decrypting it. An open source version of the compiler for C++ is available. * A Gentle Introduction to CRDTs is what it says it is: an introduction to a data structure that allows independent updates to data across a network while automatically resolving conflicts. It is an extremely important tool for building software for collaboration. * The Istio project is adding an “ambient mesh” mode that simplifies operations by eliminating the requirement for every node to have a “sidecar” proxy. The proxy layers are replaced by a “data plane mesh” that is responsible for zero-trust security and access management. * Sam Newman’s post on developer platforms is a must-read. It’s not about building a platform, it’s about enabling developers to deliver, whatever that takes. * Meilisearch is a powerful new open source search engine, built in Rust. It includes features like typo tolerance and search as you type. * Not the first time we’ve said it, but: Developers will increasingly need to take regulatory requirements into account when they write code. * Etsy provides some excellent insights on how to run a Kafka cluster in the cloud across multiple availability zones. * WebAssembly proves to be more efficient and faster than JavaScript in real-world applications. * Automerge 2.0 is now available. Automerge is a CRDT (Conflict-free replicated data type) library. CRDTs allow multiple users to access the same data objects, consistently merging changes from multiple sources (as in Google Docs). It’s an important step towards building distributed applications. * Oracle is moving to per-employee pricing for Java, a change that could make Java licenses much more expensive for small companies. * WeatherMachine offers a single API adapter that can access all of the world’s best models for forecasting weather. Are adapters a new step in the API economy?

Security* The FBI recommends using an ad blocker when browsing the web to reduce your chances of becoming a victim of fraud. * Attacks on the Python Package Index (PyPI), the Python code repository, continue. More than 450 malicious packages were uploaded recently, and the attacks have become more sophisticated. The malware watches the user’s clipboard for addresses of crypto wallets, and substitutes them with the attacker’s wallet address. * The Node Package Manager, NPM, has been subject to attack. Malicious packages install crypto miners on the users’ computers. * Fake ChatGPT apps are being used to spread malware. * After breaking into a system, attackers are using an open source cross-platform command and control tool called Havoc. Havoc includes a number of modules for remote command execution, downloading additional files, and process manipulation. * A secure API needs to authenticate and authorize every attempt to access it properly. In turn, this requires reliable and trustworthy distribution of identity data. * The National Institute of Standards (NIST) has announced a standard “lightweight” cryptography algorithm. This algorithm has been designed for CPUs with limited capabilities–specifically CPUs used in “Internet of Things” devices. * Bruce Schneier’s belated wrapup on SolarWinds: The market doesn’t reward security. SolarWinds was profitable, and the private equity firm that owns it wanted it to become more profitable. Short term profit, long-term underfunding of security. * Bruce Schneier on Machine Learning Security: we’re still in the early days of understanding how to secure ML systems against attacks. But we already know that the weakest link will be the software surrounding the ML system. * “Capture the Flag” is frequently played at computer security conferences: in a controlled environment, defenders try to protect their systems from attackers. What happens when AI-driven agents play the game? * The FBI and Europol police have seized the servers for the Hive ransomware-as-a-service group. They penetrated Hive’s network in July 2022, allowing them to access decryption keys and give them to victims.

Web, Web3, and the Metaverse* Jaron Lanier and others have proposed that large language models can be used to create virtual worlds. * Google will no longer downgrade AI-generated content in its search results. * Fastly’s Fast Forward Program provides free CDN services to open source projects and nonprofits that make the world a better place. Mastodon, with its vision of open, federated social media, is one of the projects that Fastly is supporting. * Apple is developing software to help build mixed-reality apps for the headset they are planning to release in 2023. According to rumor, the Apple headset is a different product from their AR glasses; the latter has apparently been delayed until late 2023. * California’s DMV is putting car titles on a blockchain. Other public registries may follow. While they have not yet built public-facing applications, possibilities include NFTs that represent car titles.

Quantum Computing* Google has made a small but significant improvement in their ability to build error-corrected qubits. They have demonstrated that error correction can scale: using more physical qubits to create a logical, error-corrected qubit reduces the actual error rate. * A new kind of qubit adds a “flip flop” logic gate to the repertoire of quantum operations. * Researchers have demonstrated a technique for transferring qubits from one chip to another without destroying their quantum behavior. The ability to connect quantum chips is a critical step towards building quantum computers large enough to do useful work.

Biology* CRISPR can be used to engineer flies that are unable to spread diseases between plants. This may be a way to limit the spread of crop diseases, particularly for diseases spread by pests whose range is expanding because of global warming. * Open source seeds? Almost all of the seeds used in farming are patented, and farmers have been sued for saving seeds to use in next year’s crops. The Open Source Seed Initiative provides seeds with a license that doesn’t restrict how the seeds are used. * The de-extinction project has added the Dodo to the list of species it plans to restore.

Hardware* Researchers have developed a camera the size of a grain of salt. The camera incorporates neural-network based signal processing algorithms.

View Details

This year’s report on the O’Reilly learning platform takes a detailed look at how our customers used the platform. Our goal is to find out what they’re interested in now and how that changed from 2021—and to make some predictions about what 2023 will bring.

A lot has happened in the past year. In 2021, we saw that GPT-3 could write stories and even help people write software; in 2022, ChatGPT showed that you can have conversations with an AI. Now developers are using AI to write software. Late in 2021, Mark Zuckerberg started talking about “the metaverse,” and fairly soon, everyone was talking about it. But the conversation cooled almost as quickly as it started. Back then, cryptocurrency prices were approaching a high, and NFTs were “a thing”…then they crashed.

What’s real, and what isn’t? Our data shows us what O’Reilly’s 2.8 million users are actually working on and what they’re learning day-to-day. That’s a better measure of technology trends than anything that happens among the Twitterati. The answers usually aren’t found in big impressive changes; they’re found in smaller shifts that reflect how people are turning the big ideas into real-world products. The signals are often confusing: for example, interest in content about the “big three” cloud providers is slightly down, while interest in content about cloud migration is significantly up. What does that mean? Companies are still “moving into the cloud”—that trend hasn’t changed—but as some move forward, others are pulling back (“repatriation”) or postponing projects. It’s gratifying when we see an important topic come alive: zero trust, which reflects an important rethinking of how security works, showed tremendous growth. But other technology topics (including some favorites) are hitting plateaus or even declining.

While we don’t discuss the economy as such, it’s always in the background. Whether or not we’re actually in a recession, many in our industry perceive us to be so, and that perception can be self-fulfilling. Companies that went on a hiring spree over the past few years are now realizing that they made a mistake—and that includes both giants that do layoffs in the tens of thousands and startups that thought they had access to an endless stream of VC cash. In turn, that reality influences the actions individuals take to safeguard their jobs or increase their value should they need to find a new one.

MethodologyThis report is based on our internal “units viewed” metric, which is a single metric across all the media types included in our platform: ebooks, of course, but also videos and live training courses. We use units viewed because it measures what people actually do on our platform. But it’s important to recognize the metric’s shortcomings; as George Box (almost)1 said, “All metrics are wrong, but some are useful.” Units viewed tends to discount the usage of new topics: if a topic is new, there isn’t much content, and users can’t view content that doesn’t exist. As a counter to our focus on units viewed, we’ll take a brief look at searches, which aren’t constrained by the availability of content. For the purposes of this report, units viewed is always normalized to 1, where 1 is assigned to the greatest number of units in any group of topics.

It’s also important to remember that these “units” are “viewed” by our users. Whether they access the platform through individual or corporate accounts, O’Reilly members are typically using the platform for work. Despite talk of “internet time,” our industry doesn’t change radically from day to day, month to month, or even year to year. We don’t want to discount or undervalue those who are picking up new ideas and skills—that’s an extremely important use of the platform. But if a company’s IT department were working on its ecommerce site in 2021, they were still working on that site in 2022, they won’t stop working on it in 2023, and they’ll be working on it in 2024. They might be adding AI-driven features or moving it to the cloud and orchestrating it with Kubernetes, but they’re not likely to drop React (or even PHP) to move to the latest cool framework.

However, when the latest cool thing demonstrates a few years of solid growth, it can easily become one of the well-established technologies. That’s happening now with Rust. Rust isn’t going to take over from Java and Python tomorrow, let alone in 2023 or 2024, but that’s a movement that’s real. Finally, it’s wise to be skeptical about “noise.” Changes of one or two percentage points often mean little. But when a mature technology that’s leading its category stops growing, it’s fair to wonder whether it’s hit a plateau and is en route to becoming a legacy technology.

The Biggest PictureWe can get a high-level view of platform usage by looking at usage for our top-level topics. Content about software development was the most widely used (31% of all usage in 2022), which includes software architecture and programming languages. Software development is followed by IT operations (18%), which includes cloud, and by data (17%), which includes machine learning and artificial intelligence. Business (13%), security (8%), and web and mobile (6%) come next. That’s a fairly good picture of our core audience’s interests: solidly technical, focused on software rather than hardware, but with a significant stake in business topics.

Total platform usage grew by 14.1% year over year, more than doubling the 6.2% gain we saw from 2020 to 2021. The topics that saw the greatest growth were business (30%), design (23%), data (20%), security (20%), and hardware (19%)—all in the neighborhood of 20% growth. Software development grew by 12%, which sounds disappointing, although in any study like this, the largest categories tend to show the least change. Usage of resources about IT operations only increased by 6.9%. That’s a surprise, particularly since the operations world is still coming to terms with cloud computing.

O’Reilly learning platform usage by topic year over yearWhile this report focuses on content usage, a quick look at search data gives a feel for the most popular topics, in addition to the fastest growing (and fastest declining) categories. Python, Kubernetes, and Java were the most popular search terms. Searches for Python showed a 29% year-over-year gain, while searches for Java and Kubernetes are almost unchanged: Java gained 3% and Kubernetes declined 4%. But it’s also important to note what searches don’t show: when we look at programming languages, we’ll see that content about Java is more heavily used than content about Python (although Python is growing faster).

Similarly, the actual use of content about Kubernetes showed a slight year-over-year gain (4.4%), despite the decline in the number of searches. And despite being the second-most-popular search term, units viewed for Kubernetes were only 41% of those for Java and 47% of those for Python. This difference between search data and usage data may mean that developers “live” in their programming languages, not in their container tools. They need to know about Kubernetes and frequently need to ask specific questions—and those needs generate a lot of searches. But they’re working with Java or Python constantly, and that generates more units viewed.

The Go programming language is another interesting case. “Go” and “Golang” are distinct search strings, but they’re clearly the same topic. When you add searches for Go and Golang, the Go language moves from 15th and 16th place up to 5th, just behind machine learning. However, change in use of the search term was relatively small: a 1% decline for Go, a 8% increase for Golang. Looking at Go as a topic category, we see something different: usage of content about Go is significantly behind the leaders, Java and Python, but still the third highest on our list, and with a 20% gain from 2021 to 2022.

Looking at searches is worthwhile, but it’s important to realize that search data and usage data often tell different stories.

Top searches on the O’Reilly learning platform year over yearSearches can also give a quick picture of which topics are growing. The top three year-over-year gains were for the CompTIA Linux+ certification, the CompTIA A+ certification, and transformers (the AI model that’s led to tremendous progress in natural language processing). However, none of these are what we might call “top tier” search terms: they had ranks ranging from 186 to 405. (That said, keep in mind that the number of unique search terms we see is well over 1,000,000. It’s a lot easier for a search term with a few thousand queries to grow than it is for a search term with 100,000 queries.)

The sharpest declines in search frequency were for cryptocurrency, Bitcoin, Ethereum, and Java 11. There are no real surprises here. This has been a tough year for cryptocurrency, with multiple scandals and crashes. As of late 2021, Java 11 was no longer the current long-term support (LTS) release of Java; that’s moved on to Java 17.

What Our Users Are Doing (in Detail)That’s a high-level picture. But where are our users actually spending their time? To understand that, we’ll need to take a more detailed look at our topic hierarchy—not just at the topics at the top level but at those in the inner (and innermost) layers.

Software DevelopmentThe biggest change we’ve seen is the growth in interest in coding practices; 35% year-over-year growth can’t be ignored, and indicates that software developers are highly motivated to improve their practice of programming. Coding practices is a broad topic that encompasses a lot—software maintenance, test-driven development, maintaining legacy software, and pair programming are all subcategories. Two smaller categories that are closely related to coding practices also showed substantial increases: usage of content about Git (a distributed version control system and source code repository) was up 21%, and QA and testing was up 78%. Practices like the use of code repositories and continuous testing are still spreading to both new developers and older IT departments. These practices are rarely taught in computer science programs, and many companies are just beginning to put them to use. Developers, both new and experienced, are learning them on the job.

Going by units viewed, design patterns is the second-largest category, with a year-over-year increase of 13%. Object-oriented programming showed a healthy 24% increase. The two are closely related, of course; while the concept of design patterns is applicable to any programming paradigm, object-oriented programming (particularly Java, C#, and C++) is where they’ve taken hold.

It’s worth taking a closer look at design patterns. Design patterns are solutions to common problems—they help programmers work without “reinventing wheels.” Above all, design patterns are a way of sharing wisdom. They’ve been abused in the past by programmers who thought software was “good” if it used “design patterns,” and jammed as many into their code as possible, whether or not it was appropriate. Luckily, we’ve gotten beyond that now.

What about functional programming? The “object versus functional” debates of a few years ago are over for the most part. The major ideas behind functional programming can be implemented in any language, and functional programming features have been added to Java, C#, C++, and most other major programming languages. We’re now in an age of “multiparadigm” programming. It feels strange to conclude that object-oriented programming has established itself, because in many ways that was never in doubt; it has long been the paradigm of choice for building large software systems. As our systems are growing ever larger, object-oriented programming’s importance seems secure.

Leadership and management also showed very strong growth (38%). Software developers know that product development isn’t just about code; it relies heavily on communication, collaboration, and critical thinking. They also realize that management or team leadership may well be the next step in their career.

Finally, we’d be remiss not to mention quantum computing. It’s the smallest topic category in this group but showed a 24% year-over-year gain. The first quantum computers are now available through cloud providers like IBM and Amazon Web Services (AWS). While these computers aren’t yet powerful enough to do any real work, they make it possible to get a head start on quantum programming. Nobody knows when quantum computers will be substantial enough to solve real-world problems: maybe two years, maybe 20. But programmers are clearly interested in getting started.

Year-over-year growth for software development topicsSoftware architectureSoftware architecture is a very broad category that encompasses everything from design patterns (which we also saw under software development) to relatively trendy topics like serverless and event-driven architecture. The largest topic in this group was, unsurprisingly, software architecture itself: a category that includes books on the fundamentals of software architecture, systems thinking, communication skills, and much more—almost anything to do with the design, implementation, and management of software. Not only was this a large category, but it also grew significantly: 26% from 2021 to 2022. Software architect has clearly become an important role, the next step for programming staff who want to level up their skills.

For several years, microservices has been one of the most popular topics in software architecture, and this year is no exception. It was the second-largest topic and showed 3.6% growth over 2021. Domain-driven design (DDD) was the third-most-commonly-used topic, although smaller; it also showed growth (19%). Although DDD has been around for a long time, it came into prominence with the rise of microservices as a way to think about partitioning an application into independent services.

Is the relatively low growth of microservices a sign of change? Have microservices reached a peak? We don’t think so, but it’s important to understand the complex relationship between microservices and monolithic architectures. Monoliths inevitably become more complex over time, as bug fixes, new business requirements, the need to scale, and other issues need to be addressed. Decomposing a complex monolith into a complex set of microservices is a challenging task and certainly one that can’t be underestimated: developers are trading one kind of complexity for another in the hope of achieving increased flexibility and scalability long-term. Microservices are no longer a “cool new idea,” and developers have recognized that they’re not the solution to every problem. However, they are a good fit for cloud deployments, and they leave a company well-positioned to offer its services via APIs and become an “as a service” company. Microservices are unlikely to decline, though they may have reached a plateau. They’ve become part of the IT landscape. But companies need to digest the complexity trade-off.

Web APIs, which companies use to provide services to remote client software via the web’s HTTP protocol, showed a very healthy increase (76%). This increase shows that we’re moving even more strongly to an “API economy,” where the most successful companies are built not around products but around services accessed through web APIs. That, after all, is the basis for all “software as a service” companies; it’s the basis on which all the cloud providers are built; it’s what ties Amazon’s business empire together. RESTful APIs saw a smaller increase (6%); the momentum has clearly moved from the simplicity of REST to more complex APIs that use JSON, GraphQL, and other technologies to move information.

The 29% increase in the usage of content about distributed systems is important. Several factors drive the increase in distributed systems: the move to microservices, the need to serve astronomical numbers of online clients, the end of Moore’s law, and more. The time when a successful application could run on a single mainframe—or even on a small cluster of servers in a rack—is long gone. Modern applications run across hundreds or thousands of computers, virtual machines, and cloud instances, all connected by high-speed networks and data buses. That includes software running on single laptops equipped with multicore CPUs and GPUs. Distributed systems require designing software that can run effectively in these environments: software that’s reliable, that stays up even when some servers or networks go down, and where there are as few performance bottlenecks as possible. While this category is still relatively small, its growth shows that software developers have realized that all systems are distributed systems; there is no such thing as an application that runs on a single computer.

Year-over-year growth for software architecture and design topicsWhat about serverless? Serverless looks like an excellent technology for implementing microservices, but it’s been giving us mixed signals for several years now. Some years it’s up slightly; some years it’s down slightly. This year, it’s down 14%, and while that’s not a collapse, we have to see that drop as significant. Like microservices, serverless is no longer a “cool new thing” in software architecture, but the decrease in usage raises questions: Are software developers nervous about the degree of control serverless puts in the hands of cloud providers, spinning up and shutting down instances as needed? That could be a big issue. Cloud customers want to get their accounts payable down, cloud providers want to get their accounts receivable up, and if the provider tweaks a few parameters that the customer never sees, that balance could change a lot. Or has serverless just plunged into the “trough of disillusionment” from which it will eventually emerge into the “plane of productivity”? Or maybe it’s just an idea whose time came and went? Whatever the reason, serverless has never established itself convincingly. Next year may give us a better idea…or just more ambiguity.

Programming languagesThe stories we can tell about programming languages are little changed from last year. Java is the leader (with 1.7% year-over-year growth), followed by Python (3.4% growth). But as we look down the chart, we see some interesting challengers to the status quo. Go’s usage is only 20% of Java’s, but it’s seen 20% growth. That’s substantial. C++ is hardly a new language—and we typically expect older languages to be more stable—but it had 19% year-over-year growth. And Rust, with usage that’s only 9% of Java, had 22% growth from 2021 to 2022. Those numbers don’t foreshadow a revolution—as we said at the outset, very few companies are going to take infrastructure written in Java and rewrite it in Go or Rust just so they can be trend compliant. As we all know, a lot of infrastructure is written in COBOL, and that isn’t going anywhere. But both Rust and Go have established themselves in key areas of infrastructure: Docker and Kubernetes are both written in Go, and Rust is establishing itself in the security community (and possibly also the data and AI communities). Go and Rust are already pushing older languages like C++ and Java to evolve. With a few more years of 20% growth, Go and Rust will be challenging Java and Python directly, if they aren’t challenging them already for greenfield projects.

JavaScript is an anomaly on our charts: total usage is 19% of Java’s, with a 4.6% year-over-year decline. JavaScript shows up at, or near, the top on most programming language surveys, such as RedMonk’s rankings (usually in a virtual tie with Java and Python). However, the TIOBE Index shows more space between Python (first place), Java (fourth), and JavaScript (seventh)—more in line with our observations of platform usage. We attribute JavaScript’s decline partly to the increased influence of TypeScript, a statically typed variant of JavaScript that compiles to JavaScript (12% year-over-year increase). One thing we’ve noticed over the past few years: while programmers had a long dalliance with duck typing and dynamic languages, as applications (and teams) grew larger, developers realized the value of strong, statically typed languages (TypeScript certainly, but also Go and Rust, though these are less important for web development). This shift may be cyclical; a decade from now, we may see a revival of interest in dynamic languages. Another factor is the use of frameworks like React, Angular, and Node.js, which are undoubtedly JavaScript but have their own topics in our hierarchy. However, when you add all four together, you still see a 2% decline for JavaScript, without accounting for the shift from JavaScript to TypeScript. Whatever the reason, right now, the pendulum seems to be swinging away from JavaScript. (For more on frameworks, see the discussion of web development.)

The other two languages that saw a drop in usage are C# (6.3%) and Scala (16%). Is this just noise, or is it a more substantial decline? The change seems too large to be a random fluctuation. Scala has always been a language for backend programming, as has C# (though to a lesser extent). While neither language is particularly old, it seems their shine has worn off. They’re both competing poorly with Go and Rust for new users. Scala is also competing poorly with the newer versions of Java, which now have many of the functional features that initially drove interest in Scala.

Year-over-year growth for programming languagesSecurityComputer security has been in the news frequently over the past few years. That unwelcome exposure has both revealed cracks in the security posture of many companies and obscured some important changes in the field. The cracks are all too obvious: most organizations do a bad job of the basics. According to one report, 91% of all attacks start with a phishing email that tricks a user into giving up their login credentials. Phishes are becoming more frequent and harder to detect. Basic security hygiene is as important as ever, but it’s getting more difficult. And cloud computing generates its own problems. Companies can no longer protect all of their IT systems behind a firewall; many of the servers are running in a data center somewhere, and IT staff has no idea where they are or even if they exist as physical entities.

Given this shift, it’s not surprising that zero trust, an important new paradigm for designing security into distributed systems, grew 146% between 2021 and 2022. Zero trust abandons the assumption that systems can be protected on some kind of secure network; all attempts to access any system, whether by a person or software, must present proper credentials. Hardening systems, while it received the least usage, grew 91% year over year. Other topics with significant growth were secure coding (40%), advanced persistent threats (55%), and application security (46%). All of these topics are about building applications that can withstand attacks, regardless of where they run.

Governance (year-over-year increase of 72%) is a very broad topic that includes virtually every aspect of compliance and risk management. Issues like security hygiene increasingly fall under “governance,” as companies try to comply with the requirements of insurers and regulators, in addition to making their operations more secure. Because almost all attacks start with a phish or some other kind of social engineering, just telling employees not to give their passwords away won’t help. Companies are increasingly using training programs, password managers, multifactor authentication, and other approaches to maintaining basic hygiene.

Year-over-year growth for security topicsNetwork security, which was the most heavily used security topic in 2022, grew by a healthy 32%. What drove this increase? Not the use of content about firewalls, which only grew 7%. While firewalls are still useful for protecting the IT infrastructure in a physical office, they’re of limited help when a substantial part of any organization’s infrastructure is in the cloud. What happens when an employee brings their laptop into the office from home or takes it to a coffee shop where it’s more vulnerable to attack? How do you secure WiFi networks for people working from home as well as in the office? The broader problem of network security has only become more difficult, and these problems can’t be solved by corporate firewalls.

Use of content about penetration testing and ethical hacking actually decreased by 14%, although it was the second-most-heavily-used security topic in our taxonomy (and the most heavily used in 2021).

Security certificationsSecurity professionals love their certifications. Our platform data shows that the most important certifications were the CISSP (Certified Information Systems Security Professional) and the CompTIA Security+. The CISSP has long been the most popular security certification. It’s a very comprehensive certification oriented toward senior security specialists: candidates must have at least five years’ experience in the field to take the exam. Usage of CISSP-related content dropped 0.23% year over year—in other words, it was essentially flat. A change this small is almost certainly noise, but the lack of change may indicate that CISSP has saturated its market.

Compared to the CISSP, the CompTIA Security+ certification is aimed at entry- or mid-level security practitioners; it’s a good complement to the other CompTIA certifications, such as the Network+. Right now, the demand for security exceeds the supply, and that’s drawing new people into the field. This fits with the increase in the use of content to prepare for the CompTIA Security+, which grew 16% in the past year. The CompTIA CSA+ exam (recently renamed the CYSA+) is a more advanced certification aimed specifically at security analysts; it showed 37% growth.

Year-over-year growth for security certificationsUse of content related to the Certified Ethical Hacker certification dropped 5.9%. The reasons for this decline aren’t clear, given that demand for penetration testing (one focus of ethical hacking) is high. However, there are many certifications specifically for penetration testers. It’s also worth noting that penetration testing is frequently a service provided by outside consultants. Most companies don’t have the budget to hire full-time penetration testers.

CBK isn’t an exam; it’s the framework of material around which the International Information System Security Certification Consortium, more commonly known as (ISC)², builds its exams. With a 31% year-over-year increase for CBK content, it’s another clear sign that interest in security as a profession is growing. And even though (ISC)²’s marquee certification, the CISSP, has likely reached saturation, other (ISC)² certifications show clear growth: the CCSP (Certified Cloud Security Professional) grew 52%, and the SSCP (Systems Security Certified Practitioner) grew 67%. Although these certifications aren’t as popular, their growth is an important trend.

DataData is another very broad category, encompassing everything from traditional business analytics to artificial intelligence. Data engineering was the dominant topic by far, growing 35% year over year. Data engineering deals with the problem of storing data at scale and delivering that data to applications. It includes moving data to the cloud, building pipelines for acquiring data and getting data to application software (often in near real time), resolving the issues that are caused by data siloed in different organizations, and more.

Apache Spark, a platform for large-scale data processing, was the most widely used tool, even though the use of content about Spark declined slightly in the past year (2.7%). Hadoop, which would have led this category a decade ago, is still present, though usage of content about Hadoop dropped 8.3%; Hadoop has become a legacy data platform.

Microsoft Power BI has established itself as the leading business analytics platform; content about Power BI was the most heavily used, and achieved 31% year-over-year growth. NoSQL databases was second, with 7.6% growth—but keep in mind that NoSQL was a movement that spawned a large number of databases, with many different properties and designs. Our data shows that NoSQL certainly isn’t dead, despite some claims to the contrary; it has clearly established itself. However, the four top relational databases, if added together into a single “relational database” topic, would be the most heavily used topic by a large margin. Oracle grew 18.2% year over year; Microsoft SQL Server grew 9.4%; MySQL grew 4.7%; and PostgreSQL grew 19%.

Use of content about R, the widely used statistics platform, grew 15% from 2021. Similarly, usage of content about pandas, the most widely used Python library for working with R-like data frames, grew 20%. It’s interesting that pandas and R had roughly the same usage. Python and R have been competing (in a friendly way) for the data science market for nearly 20 years. Based on our usage data, right now it looks like a tie. R has slightly more market share, but pandas has better growth. Both are staples in academic research: R is more of a “statistician’s workbench” with a comprehensive set of statistical tools, while Python and pandas are built for programmers. The difference has more to do with users’ tastes than substance though: R is a fully capable programming language, and Python has excellent statistical and array-processing libraries.

Usage for content about data lakes and about data warehouses was also just about equal, but data lakes usage had much higher year-over-year growth (50% as opposed to 3.9%). Data lakes are a strategy for storing an organization’s data in an unstructured repository; they came into prominence a few years ago as an alternative to data warehouses. It would be useful to compare data lakes with data lakehouses and data meshes; those terms aren’t in our taxonomy yet.

Year-over-year growth for data analysis and database topicsArtificial intelligenceAt the beginning of 2022, who would have thought that we would be asking an AI-driven chat service to explain source code (even if it occasionally makes up facts)? Or that we’d have AI systems that enable nonartists to create works that are on a par with professional designers (even if they can’t match Degas and Renoir)? Yet here we are, and we don’t have ChatGPT or generative AI in our taxonomy. The one thing that we can say is that 2023 will almost certainly take AI even further. How much further nobody knows.

For the past two years, natural language processing (NLP) has been at the forefront of AI research, with the release of Open AI’s popular tools GPT-3 and ChatGPT along with similar projects from Google, Meta, and others that haven’t been released. NLP has many industrial applications, ranging from automated chat servers to code generation (e.g., GitHub Copilot) to writing tools. It’s not surprising that NLP content was the most viewed and saw significant year-over-year growth (42%). All of this progress is based on deep learning, which was the second-most-heavily-used topic, with 23% growth. Interest in reinforcement learning seems to be off (14% decline), though that may turn around as researchers try to develop AI systems that are more accurate and that can’t be tricked into hate speech. Reinforcement learning with human feedback (RLHF) is one new technique that might lead to better-behaved language models.

There was also relatively little interest in content about chatbots (a 5.8% year-over-year decline). This reversal seems counterintuitive, but it makes sense in retrospect. The release of GPT-3 was a watershed event, an “everything you’ve done so far is out-of-date” moment. We’re excited about what will happen in 2023, though the results will depend a lot on how ChatGPT and its relatives are commercialized, as Microsoft moves toward offering ChatGPT as a cloud-based service.

Year-over-year growth for artificial intelligence topicsOur learning platform gives some insight into the tools developers and researchers are using to work with AI. Based on units viewed, scikit-learn was the most popular library. It’s a relatively old tool, but it’s still actively maintained and obviously appreciated by the community: usage increased 4.7% over the year. While usage of content about PyTorch and TensorFlow is roughly equivalent (PyTorch is slightly ahead), it’s clear that PyTorch now has momentum. PyTorch increased 20%, while TensorFlow decreased 4.8%. Keras, a frontend library that uses TensorFlow, dropped 40%.

It’s disappointing to see so little usage of content on MLOps this year, along with a slight drop (4.0%) from 2021 to 2022. One of the biggest problems facing machine learning and artificial intelligence is deploying applications into production and then maintaining them. ML and AI applications need to be integrated into the deployment processes used for other IT applications. This is the business of MLOps, which presents a set of problems that are only beginning to be solved, including versioning for large sets of training data and automated testing to determine when a model has become stale and needs retraining. Perhaps it’s still too early, but these problems must be addressed if ML and AI are to succeed in the enterprise.

No-code and low-code tools for AI don’t appear in our taxonomy, unfortunately. Our report AI Adoption in the Enterprise 2022 argues that AutoML in its various incarnations is gradually gaining traction. This is a trend worth watching. While there’s very little training available on Google AutoML, Amazon AutoML, IBM AutoAI, Amazon SageMaker, and other low-code tools, they’ll almost certainly be an important force multiplier for experienced AI developers.

Infrastructure and OperationsContainers, Linux, and Kubernetes are the top topics within infrastructure and operations. Containers sits at the top of the list (with 2.5% year-over-year growth), with Docker, the most popular container, in fifth place (with a 4.4% decline). Linux, the second most used topic, grew 4.4% year over year. There’s no surprise here; as we’ve been saying for some time, Linux is “table stakes” for operations. Kubernetes is third, with 4.4% growth.

The containers topic is extremely broad: it includes a lot of content that’s primarily about Docker but also content about containers in general, alternatives to Docker (most notably Podman), container deployment, and many other subtopics. It’s clear that containers have changed the way we deploy software, particularly in the cloud. It’s also clear that containers are here to stay. Docker’s small drop is worth noting but isn’t a harbinger of change. Kubernetes deprecated direct Docker support at the end of 2020 in favor of the Container Runtime Interface (CRI). That change eliminated a direct tie between Kubernetes and Docker but doesn’t mean that containers built by Docker won’t run on Kubernetes, since Docker supports the CRI standard. A more convincing reason for the drop in usage is that Docker is no longer new and developers and other IT staff are comfortable with it. Docker itself may be a smaller piece of the operations ecosystem, and it may have plateaued, but it’s still very much there.

Content about Kubernetes was the second-most-widely-viewed in this group, and usage grew 4.4% year over year. That relatively slow growth may mean that Kubernetes is close to a plateau. We increasingly see complaints that Kubernetes is overly complex, and we expect that, sooner or later, someone will build a container orchestration platform that’s simpler, or that developers will move toward “managed” solutions where a third party (probably a cloud provider) manages Kubernetes for them. One important part of the Kubernetes ecosystem, the service mesh, is declining; content about service mesh showed a 28% decline, while content about Istio (the service mesh implementation most closely tied to Kubernetes) declined 42%. Again, service meshes (and specifically Istio) are widely decried as too complex. It’s indicative (and perhaps alarming) that IT departments are resorting to “roll your own” for a complex piece of infrastructure that manages communications between services and microservices (including services for security). Alternatives are emerging. HashiCorp’s Consul and the open source Linkerd project are promising service meshes. UC Berkeley’s RISELab, which developed both Ray and Spark, recently announced SkyPilot, a tool with goals similar to Kubernetes but that’s specialized for data. Whatever the outcome, we don’t believe that Kubernetes is the last word in container orchestration.

Year-over-year growth for infrastructure and operations topicsIf there’s any tool that defines “infrastructure as code,” it’s Terraform, which saw 74% year-over-year growth. Terraform’s goals are relatively simple: You write a simple description of what infrastructure you want and how you want that infrastructure configured. Terraform gathers the resources and configures them for you. Terraform can be used with all of the major cloud providers, in addition to private clouds (via OpenStack), and it’s proven to be an essential tool for organizations that are migrating to the cloud.

We took a separate look at the “continuous” methodologies (also known as CI/CD): continuous integration, continuous delivery, and continuous deployment. Overall, this group showed an 18% year-over-year increase in units viewed. This growth comes largely from a huge (40%) increase in the use of content about continuous delivery. Continuous integration showed a 22% decline, while continuous deployment had a 7.1% increase.

What does this tell us? The term continuous integration was first used by Grady Booch in 1991 and popularized by the Extreme Programming movement in the late 1990s. It refers to the practice of merging code changes into a single repository frequently, testing at each iteration to ensure that the project is always in a coherent state. Continuous integration is tightly coupled to continuous delivery; you almost always see CI/CD together. Continuous delivery is a practice that was developed at the second-generation web companies, including Flickr, Facebook, and Amazon, which radically changed IT practice by staging software updates for deployment several times daily. With continuous delivery, deployment pipelines are fully automated, requiring only a final approval to put a release into production. Continuous deployment is the newest (and smallest) of the three, emphasizing completely automated deployment to production: updates go directly from the developer into production, without any intervention. These methodologies are closely tied to each other. CI/CD/CD as a whole (and yes, nobody ever uses CD twice) is up 18% for the year. That’s a significant gain, and even though these topics have been around for a while, it’s evidence that growth is still possible.

Year-over-year growth for continuous methodologiesIT and operations certificationsThe leading IT certification is clearly CompTIA, which showed a 41% year-over-year increase. The CompTIA family (Network+, A+, Linux+, and Security+) dominates the certification market. (The CompTIA Network+ showed a very slight decline (0.32%), which is probably just random fluctuation.) The Linux+ certification in particular experienced tremendous year-over-year growth (47%), which is easy to understand. Linux has long been the dominant server operating system. In the cloud, Linux instances are much more widely used than the alternatives, though Windows is offered on Azure (of course) along with macOS. In the past few years, Linux’s market penetration has gone even deeper. We’ve already seen the role that containers are playing, and containers almost always run Linux as their operating system. In 1995, Linux might have been a quirky choice for people devoted to free and open source software. In 2023, Linux is mandatory for anyone in IT or software development. And it’s hard to imagine getting a job or advancing in a career without demonstrating competence.

Year-over-year growth for IT certificationsIt’s surprising to see the Cisco Certified Network Associate (CCNA) certification drop 18% and the Cisco Certified Network Professional (CCNP) drop 12%, as the Cisco certifications have been among the most meaningful and prestigious in IT for many years. (The Cisco Certified Internet Expert (CCIE) certification, while relatively small compared to the others, did show 70% growth.) There are several causes for this shift. First, as companies move workloads to the cloud or to colocation providers, maintaining a fleet of routers and switches becomes less important. Network certifications are less valuable than they used to be. But why then the increase in the CCIE? While the CCNA is an entry-level certification and the CCNP is middle tier, the CCIE is Cisco’s top-tier certification. The exam is very detailed and rigorous and includes hands-on work with network hardware. Hence the relatively small number of people who attempt it and study for it. However, even as companies offload much of their day-to-day network administration to the cloud, they still need people who understand networks in depth. They still have to deal with office networks, and with extending office networks to remote employees. While they don’t need staff to wrangle racks of data center routers, they do need network experts who understand what their cloud and colocation providers are doing. The need for network staff might be shrinking, but it isn’t going away. In a shrinking market, attaining the highest level of certification will have the most long-term value.

CloudWe haven’t seen any significant shifts among the major cloud providers. Amazon Web Services (AWS) still leads, followed by Microsoft Azure, then Google Cloud. Together, this group represents 97% of cloud platform content usage. The bigger story is that we saw decreases in year-over-year usage for all three. The decreases are small and might not be significant: AWS is down 3.8%, Azure 7.5%, and Google Cloud 2.1%. We don’t know what’s responsible for this decline. We looked industry by industry; some were up, some were down, but there were no smoking guns. AWS showed a sharp drop in computers and electronics (about 27%), which is a relatively large category, and a smaller drop in finance and banking (15%), balanced by substantial growth in higher education (35%). There was a lot of volatility among industries that aren’t big cloud users—for example, AWS was up about 250% in agriculture—but usage among industries that aren’t major cloud users isn’t high enough to account for that change. (Agriculture accounts for well under 1% of total AWS content usage.) The bottom line is, as they say in the nightly financial news, “Declines outnumbered gains”: 16 out of 28 business categories showed a decline. Azure was similar, with 20 industries showing declines, although Azure saw a slight increase for finance and banking. The same was true for Google Cloud, though it benefited from an influx of individual (B2C) users (up 9%).

Over the past year, there’s been some discussion of “cloud repatriation”: bringing applications that have moved to the cloud back in-house. Cost is the greatest motivation for repatriation; companies moving to the cloud have often underestimated the costs, partly because they haven’t succeeded in using the cloud effectively. While repatriation is no doubt responsible for some of the decline, it’s at most a small part of the story. Cloud providers make it difficult to leave, which ironically might drive more content usage as IT staff try to figure out how to get their data back. A bigger issue might be companies that are putting cloud plans on hold because they hear of repatriation or that are postponing large IT projects because they fear a recession.

Of the smaller cloud providers, IBM showed a huge year-over-year increase (135%). Almost all of the change came from a significant increase in consulting and professional services (200% growth year over year). Oracle showed a 36% decrease, almost entirely due to a drop in content usage from the software industry (down 49%). However, the fact that Oracle is showing up at all demonstrates that it’s grown significantly over the past few years. Oracle’s high-profile deal to host all of TikTok’s data on US residents could easily solidify the company’s position as a significant cloud provider. (Or it could backfire if TikTok is banned.)

We didn’t include two smaller providers in the graph: Heroku (now owned by Salesforce) and Cloud Foundry (originally VMware, handed off to the company’s Pivotal subsidiary and then to the Cloud Foundry Foundation; now, multiple providers run Cloud Foundry software). Both saw fairly sharp year-over-year declines: 10% for Heroku, 26% for Cloud Foundry. As far as units viewed, Cloud Foundry is almost on a par with IBM. But Heroku isn’t even on the charts; it appears to be a service whose time has passed. We also omitted Tencent and Alibaba Cloud; they’re not in our subject taxonomy, and relatively little content is available.

Year-over-year growth for cloud providersCloud certifications followed a similar pattern. AWS certifications led, followed by Azure, followed by Google Cloud. We saw the same puzzling year-over-year decline here: 13% for AWS certification, 10% for Azure, and 6% for Google Cloud. And again, the drop was smallest for Google Cloud.

While usage of content about specific cloud providers dropped from 2021 to 2022, usage for content about other cloud computing topics grew. Cloud migration, a fairly general category for content about building cloud applications, grew 45%. Cloud service models also grew 41%. These increases may help us to understand why usage of content about the “big three” clouds decreased. As cloud usage moves beyond early adopters and becomes mainstream, the conversation naturally focuses less on individual cloud providers and more on high-level issues. After a few pilot projects and proofs of concept, learning about AWS, Azure, and Google Cloud is less important than planning a full-scale migration. How do you deploy to the cloud? How do you build services in the cloud? How do you integrate applications you have moved to the cloud with legacy applications that are staying in-house? At this point, companies know the basics and have to go the rest of the way.

Year-over-year growth for cloud certificationsWith this in mind, it’s not at all surprising that our customers are very interested in hybrid clouds, for which content usage grew 28% year over year. Our users realize that every company will inevitably evolve toward a hybrid cloud. Either there’ll be a wildcat skunkworks project on some cloud that hasn’t been “blessed” by IT, or there’ll be an acquisition of a company that’s using a different provider, or they’ll need to integrate with a business partner using a different provider, or they don’t have the budget to move their legacy applications and data, or… The reasons are endless, but the conclusion is the same: hybrid is inevitable, and in many companies it’s already the reality.

The increase in use of content about private clouds (37%) is part of the same story. Many companies have applications and data that have to remain in-house (whether that’s physically on-premises or hosted at a data center offering colocation). It still makes sense for those applications to use APIs and deployment toolchains equivalent to those used in the cloud. “The cloud” isn’t the exception; it has become the rule.

Year-over-year growth for cloud architecture topicsProfessional SkillsIn the past year, O’Reilly users have been very interested in upgrading their professional and management skills. Every category in this relatively small group is up, and most of them are up significantly. Project management saw 47% year-over-year growth; professional development grew 37%. Use of content about the Project Management Professional (PMP) certification grew 36%, and interest in product management grew similarly (39%). Interest in communication skills increased 26% and interest in leadership grew by 28%. The two remaining categories that we tracked, IT management and critical thinking, weren’t as large and grew by somewhat smaller amounts (21% and 20%, respectively).

Several factors drive these increases. For a long time, software development and IT operations were seen as solo pursuits dominated by “neckbeards” and antisocial nerds, with some “rock stars” and “10x programmers” thrown in. This stereotype is wrong and harmful—not just to individuals but to teams and companies. In the past few years, we’ve heard a lot less about 10x developers and more about the importance of good communication, leadership, and mentoring. Our customers have realized that the key to productivity is good teamwork, not some mythical 10x developer. And there are certainly many employees who see positions in management, as a “tech lead,” as a product manager, or as a software architect, as the obvious next step in their careers. All of these positions stress the so-called “soft skills.” Finally, talk about a recession has been on the rise for the past year, and we continue to see large layoffs from big companies. While software developers and IT operations staff are still in high demand, and there’s no shortage of jobs, many are certainly trying to acquire new skills to improve their job security or to give themselves better options in the event that they’re laid off.

Year-over-year growth for professional skills topicsWeb DevelopmentThe React and Angular frameworks continue to dominate web development. The balance is continuing to shift toward React (10% year-over-year growth) and away from Angular (a 17% decline). Many frontend developers feel that React offers better performance and is more flexible and easier to learn. Many new frameworks (and frameworks built on frameworks) are in play (Vue, Next.js, Svelte, and so on), but none are close to becoming competitors. Vue showed a significant year-over-year decline (17%), and the others didn’t make it onto the chart.

PHP is still a contender, of course, with almost no change (a decline of 1%). PHP advocates claim that 80% of the web is built on it: Facebook is built on PHP, for instance, along with millions of WordPress sites. Still, it’s hard to look at PHP and say that it’s not a legacy technology. Ruby on Rails grew 6.6%. Content usage for Ruby on Rails is similar to PHP, but Rails usage has been declining for some years. Is it poised for a comeback?

The use of content about JavaScript showed a slight decline (4.6%), but we don’t believe this is significant. In our taxonomy, content can only be tagged with one topic, and everything that covers React or Angular is implicitly about JavaScript. In addition, it’s interesting to see usage of TypeScript increasing (12%); TypeScript is a strongly typed variant of JavaScript that compiles (the right word is actually “transpiles”) to JavaScript, and it’s proving to be a better tool for large complex applications.

One important trend shows up at the bottom of the graph. WebAssembly is still a small topic, but it saw 74% growth from 2020 to 2021. And Blazor, Microsoft’s implementation of C# and .NET for WebAssembly, is up 59%. That’s a powerful signal. These topics are still small, but if they can maintain that kind of growth, they won’t be small for long. WebAssembly is poised to become an important part of web development.

Year-over-year growth for web development topicsDesignThe heaviest usage in the design category went to user experience and related topics. User experience grew 18%, user research grew 5%, interface design grew 92%, and interaction design grew 36%. For years, we expected software to be difficult and uncomfortable to use. That’s changed. Apple made user interface design a priority early in the early 2000s, forcing other companies to follow if they wanted to remain competitive. The design thinking movement may no longer be in the news, but it’s had an effect: software teams think about design from the beginning. Even software developers who don’t have the word “design” in their job title need to think about and understand design well enough to build decent user interfaces and pleasant user experiences.

Usability, the only user-centric topic to show a decline, was only down 2.6%. It’s also worth noting that use of content about accessibility has grown 96%. Accessibility is still a relatively small category, but that kind of growth shows that accessibility is an aspect of user experience that can no longer be ignored. (The use of alt text for images is only one example: it’s become common on Twitter and is almost universal on Mastodon.)

Information architecture was down significantly (a 17% drop). Does that mean that interest has shifted from designing information flow to designing experiences, and is that a good thing?

Use of content about virtual and augmented reality is relatively small but grew 83%. The past year saw a lot of excitement around VR, Web3, the metaverse, and related topics. Toward the end of the year, that seemed to cool off. However, an 83% increase is noteworthy. Will that continue? It may depend on a new generation of VR products, both hardware and software. If Apple can make VR glasses that are comfortable and that people can wear without looking like aliens, 83% growth might seem small.

Year-over-year growth for design topicsThe FutureWe started out by saying that this industry doesn’t change as much from year to year as most people think. That’s true, but that doesn’t mean there’s no change. There are signals of important new trends—some completely new, some continuations of trends that started years ago. So what small changes are harbingers of bigger changes in the years to come?

The Go and Rust programming languages have shown significant growth both in the past year and for the last few years. There’s no sign that this growth will stop. It will take a few more years, but before long they’ll be on a par with Java and Python.

It’s no surprise that we saw huge gains for natural language processing and deep learning. GPT-3 and its successor ChatGPT are the current stars of the show. While there’s been a lot of talk about another “AI winter,” that isn’t going to happen. The success of ChatGPT (not to mention Stable Diffusion, Midjourney, and many projects going on at Meta and Google) will keep winter away, at least for another year. What will people build on top of ChatGPT and its successors? What new programming tools will we see? How will the meaning of “computer programming” change if AI assistants take over the task of writing code? What new research tools will become available, and will our new AI assistants persist in “making stuff up”? For several years now, AI has been the most exciting area in software. There’s lots to imagine, lots to build, and infinite space for innovation. As long as the AI community provides exciting new results, no one will be complaining and no one need fear the cold.

We’ve also seen a strong increase in interest in leadership, management, communication, and other “soft skills.” This interest isn’t new, but it’s certainly growing. Whether the current generation of programmers is getting tired of coding or whether they perceive soft skills as giving them better job security during a recession isn’t for us to say. It’s certainly true that better communication skills are an asset for any project.

Our audience is slightly less interested in content about the “big three” cloud providers (AWS, Azure, and Google Cloud), but they’re still tremendously interested in migrating to the cloud and taking advantage of cloud offerings. Despite many reports claiming that cloud adoption is almost universal (and I confess to writing some of them), I’ve long believed that we’re only in the early stages of cloud adoption. We’re now past the initial stage, during which a company might claim that it was “in the cloud” on the basis of a few trial projects. Cloud migration is serious business. We expect to see a new wave of cloud adoption. Companies in that wave won’t make naive assumptions about the costs of using the cloud, and they’ll have the tools to optimize their cloud usage. This new wave may not break until fears of a recession end, but it will come.

While the top-level security category grew 20%, we’d hoped to see more. For a long time, security was an afterthought, not a priority. That’s changing, but slowly. However, we saw huge gains for zero trust and governance. It’s unfortunate that these gains are driven by necessity (and the news cycle), but perhaps the message is getting through after all.

What about augmented and virtual reality (AR/VR), the metaverse, and other trendy topics that dominated much of the trade press? Interest in VR/AR content grew significantly, though what that means for 2023 is anyone’s guess. Long-term, the category probably depends on whether or not anyone can make AR glasses a fashion accessory that everyone needs to have. A bigger question is whether anyone can build a next-generation web that’s decentralized, and that fosters immediacy and collaboration without requiring exotic goggles. That’s clearly something that can be done: look no further than Figma (for collaboration), Mastodon (for decentralization), or Petals (for a cloud-less cloud).

Will these be the big stories for 2023? February is only just beginning; we have 11 months to find out.

Footnotes1. Box said “models”; a metric is a kind of model, isn’t it?

View Details

(Dear readers: this is a scaled-down excerpt from a larger project I’m working on. I’ll let you know when that effort is ready for broad distribution.)

Every technology is good for something. But there are use cases, and then there are Use Cases. The extremely compelling applications of the technology. Those that lead to widespread adoption and increased legitimacy, almost becoming synonymous with the technology itself.

Do people still use the term “killer app?” It’s not my favorite—I (unfairly?) associate it with Dot-Com business-bro culture—but I have to admit that it captures the spirit of that dominant use case. So I’ll hold my nose and use it here.

If you reflect on the emerging-tech landscape, you see the following killer apps:

• Early-day internet: E-commerce. Hands-down.
• Cloud: The legion of SaaS tool startups, on its first go-round; then AI for its victory lap.
• Data science/ML/AI: Advertising. Advertising. Advertising.

And then there’s the new kid, web3. I’ve noticed that people are more inclined to ask me “what’s it good for?” rather than “what is it?” Which is fair. Every technology has to pull its weight, and sometimes What It Enables People To Do counts more than What It Actually Is Under The Hood. (Hence, my usual crack that machine learning is just linear algebra with better marketing. But I’ll save that for a different article.)

While I can walk those people through a few use cases, I still haven’t figured out what web3’s killer app is. That’s not for a lack of trying. I’ve been exploring the topic for a couple of years now, which is what led me to launch the Block & Mortar newsletter so I could share more of my research in public.

Why It’s ToughSorting out web3’s killer app(s) has proven difficult for a number of reasons, including:

• Mixed bag/layer cake: The term “web3” is as slippery as “AI,” which has already changed names a few times. Both are umbrella terms for several different concepts. Today we have the three-layer cake that is blockchain-cryptocurrency-NFTs, plus this “metaverse” term that is itself very fuzzy. We may add more to that list as the field grows.

So when we talk about a use case for “web3,” we first need to decide which of those concepts we mean. (It’s sort of like how “internet” sometimes means “the underlying network connectivity layer,” and other times, “the web.”) * Rearview mirror: We usually notice killer apps after the fact. The technology is built to do X (and it may do a middling job of that) but someone else realizes that it would revolutionize Y.

Bitcoin—the most recognized name in this space—has been around since 2009, but the wider web3 ecosystem is maybe half that age. As it’s still developing, we’re still in that phase of throwing it at everything to see what sticks. That’s probably what will uncover the killer app, but we won’t know until something really takes off. * Deja vu, all over again: A common reaction to web3 use cases is, “we already have that.” Or even, “crypto is a terrible version of that.” Both of which are usually true. Blockchain is an absolutely terrible replacement for a relational database. But so was MongoDB. And Hadoop. And every other non-relational data store that’s come along. The point is to notice where a relational database doesn’t work so well, when it’s creaking at the edges, and then see how another tool would do in its place.

(Do you have one entity in charge of managing all the data? You’re pretty safe to default to a relational database. Do you have several peers, all of whom need to see and validate the data, and none of whom want to trust one member with all the keys? Blockchain is your friend.)

We had search engines before Google. Social networks before Twitter, and physical stores before e-commerce. “Why would I need to boot up my computer to go shopping? I can just hop in my car and browse in-person.” How long did it take merchants to see the value in a web-based storefront, backed by a warehouse-and-shipping infrastructure? And why’d it take consumers so long to realize that it’s nicer to click around a website at 3AM from the comfort of their couch?

The new way of doing things is often convenience masked as discomfort with the unfamiliar. It takes time for us to learn that it’s not so uncomfortable after all. * Guilt by association: Most people use “web3” and “crypto” interchangeably, which is not exactly fair. They also associate “crypto” with “crime,” which is much harder for me to refute. Most mainstream cryptocurrency news stories involve phishing scams, a token’s meltdown, or a fund collapsing. Mix that with the environmental impact of crypto mining and I can see why people would assume it’s good for nothing.

(One could argue that web3 has proven very good for criminals, and that the killer app is separating people from their money. I won’t dispute that. But for now, let’s focus on legitimate use cases that will have mass appeal.)

What It Won’t BeMy gut feeling is that targeted, invasive advertising will not be web3’s killer app.

It will certainly get some traction as companies try to make it happen. Adtech drove a lot of web2 and I already see attempts to ride that wave into web3. To advertisers, a metaverse property is a surface on which to show ads, in a (semi-)walled garden, where they can collect contact details.

And, frankly, that’s the problem. Web2’s “collect personal info to try to identify specific individuals who may be interested and then pummel them with messaging” is incompatible with web3’s ethos of “honor pseudonymity and give people the opportunity to tell you when they’re interested.”

Web3 shifts the power of outreach to the buyer. That sounds like a better system to me, because of the strength of self-selection. But to get there, marketers will have to unlearn old habits and embrace this world in which they derive greater benefit yet have less control. Understandably, they will have trouble letting go.

So if not advertising, then what?

Based on my research, I suspect web3’s killer apps will come out of two unlikely fields: fashion and loyalty programs.

Fashion-forwardThe fashion industry was an early adopter of web3. From accepting cryptocurrency as a form of payment, to token-gating events (including special NFTs for VIP passes), to virtual models. Well-known fashion houses have created wearables and perfumes for metaverse avatars, some of which are digital twins for real-world items. They’ve even flipped that around, to road-test digital products before releasing them in physical form. Much of this work has led to the understanding of using NFTs to build community.

That’s admittedly more of a sampler platter than a single use case. There’s no clear leader in there. Yet. But if the best way to find something is by looking, then the fashion industry is poised to find that killer app precisely because they are running so many experiments. They’re testing web3 tools in public, in real-world situations, and they are learning at each step.

Even if you know zilch about fashion, you can still keep an eye on this field’s web3 work and adapt it to your own. I highly recommend Vogue Business as a start. That’s right, the eponymous fashion magazine has a dedicated publication for behind-the-scenes industry issues such as technology, sustainability, and economic trends. Stumbling onto that website jump-started my understanding of web3. I saw real business use cases outside of DeFi, and got my first taste of what I would later refer to as NFTs With Benefits: using the tokens as access passes and for VIP status.)

Rewarding LoyaltyLoyalty programs are an interesting bunch. They’re the other side of the marketing department, with a very different approach compared to their siblings in the advertising arena.

The idea behind a loyalty program is that someone is already a customer, and they have expressly signed up to join your fan club. (That sounds a lot like the web3 ideal of letting people self-select, does it not?) Membership in a loyalty program gives rise to a virtuous cycle: people like what you do, so they patronize your business more; you then find new ways to keep them happy, so they continue to like you.

The value in this positive feedback loop becomes clear when you consider that the cost of acquiring a new customer is typically much higher than keeping an existing customer engaged. And that repeat business adds up. Major airlines’ frequent-flier programs rake in billions of dollars each year. Businesses have a strong incentive to keep those loyalty programs humming.

How does web3 fit in here? Loyalty programs are often built on a gamified structure, such as “fly X miles within Y months to get Z status.” Companies create web3 games that let people show how engaged they are with the brand. Chipotle customers rolled virtual burritos inside a Roblox eatery as a way for the chain to introduce its Garlic Guajillo Steak dish. Universal Studios gave out NFTs for participation in its in-person scavenger hunt. And Starbucks recently unveiled blockchain-based updates to its Rewards program, challenging people to earn “Journey Stamps”—NFTs in everything but name—for trying different drinks.

This is when you’d ask why companies can’t build these games on existing technologies. That would be a fair question, since nothing I’ve described thus far really needs a blockchain. But it does offer two perks:

First, a loyalty program operates on a sequence of transactions such as “spend points,” “acquire points,” “use service.” Blockchain technology is purpose-built to record transactions to a tamper-resistant ledger. And a blockchain’s decentralized nature makes it easier for members in a shared venture—think airlines with codeshare agreements, or airlines partnering with hotels—to get instant updates on member activity. They can even build all of this behind the scenes, shielding customers from the underlying crypto wallet management.

Second, for those loyalty programs that expose the blockchain functionality to members, those crypto wallets serve as digital identities. True fans won’t just achieve status in a program; they’ll be able to broadcast that status by showing off the associated NFTs in a public-facing wallet. And that is a strong form of organic marketing.

Time Will TellFashion and loyalty programs are poised to uncover web3’s killer apps, whatever those may be. At least, that’s how it’s adding up right now. I look forward to reviewing this article over the next few years to see whether this turns out to be true.

Whatever it is, I think back to something Mike Loukides has told me: “I think the winner will be whoever can build a blockchain that you don’t even know you’re using.” This is true. Consumers rarely care what technology runs their favorite apps; they just want them to work. Additionally, web3 still has a reputation problem. If companies are to reap blockchain’s technology benefits, they’d do well to keep them behind the scenes. Or at least follow the Starbucks example and give the tools new, brand-specific names.

We should also consider what happens when those killer apps finally surface. That will be the end of one race and the start of another. The outsized interest in building on and monetizing those killer apps will drive improvements in the underlying technology. And those improvements can be applied elsewhere.

Consider how much adtech has poured back into the AI ecosystem. Google and Facebook drove advances in neural networks, contributing code (TensorFlow, Torch, Prophet), hardware (custom TPU chips), and tooling (autoML and model hosting infrastructure through Vertex AI). That’s not to speak of the educational material that’s sprung up around these tools and services. Combined, these have lowered the barrier to entry for individuals to learn about neural networks and for businesses to put those powerful models to use.

So I look forward to the continued quest for the web3 killer app(s), in part for what that will do for the space as a whole.

View Details

It’s been well publicized that Google’s Bard made some factual errors when it was demoed, and Google paid for these mistakes with a significant drop in their stock price. What didn’t receive as much news coverage (though in the last few days, it’s been well discussed online) are the many mistakes that Microsoft’s new search engine, Sydney, made. The fact that we know its name is Sydney is one of those mistakes, since it’s never supposed to reveal its name. Sydney-enhanced Bing has threatened and insulted its users, in addition to being just plain wrong (insisting that it was 2022, and insisting that the first Avatar movie hadn’t been released yet). There are excellent summaries of these failures in Ben Thompson’s newsletter Stratechery and Simon Willison’s blog. It might be easy to dismiss these stories as anecdotal at best, fraudulent at worst, but I’ve seen many reports from beta testers who managed to duplicate them.

Of course, Bard and Sydney are beta releases that aren’t open to the wider public yet. So it’s not surprising that things are wrong. That’s what beta tests are for. The important question is where we go from here. What are the next steps?

Large language models like ChatGPT and Google’s LaMDA aren’t designed to give correct results. They’re designed to simulate human language—and they’re incredibly good at that. Because they’re so good at simulating human language, we’re predisposed to find them convincing, particularly if they word the answer so that it sounds authoritative. But does 2+2 really equal 5? Remember that these tools aren’t doing math, they’re just doing statistics on a huge body of text. So if people have written 2+2=5 (and they have in many places, probably never intending that to be taken as correct arithmetic), there’s a non-zero probability that the model will tell you that 2+2=5.

The ability of these models to “make up” stuff is interesting, and as I’ve suggested elsewhere, might give us a glimpse of artificial imagination. (Ben Thompson ends his article by saying that Sydney doesn’t feel like a search engine; it feels like something completely different, something that we might not be ready for—perhaps what David Bowie meant in 1999 when he called the Internet an “alien lifeform”). But if we want a search engine, we will need something that’s better behaved. Again, it’s important to realize that ChatGPT and LaMDA aren’t trained to be correct. You can train models that are optimized to be correct—but that’s a different kind of model. Models like that are being built now; they tend to be smaller and trained on specialized data sets (O’Reilly Media has a search engine that has been trained on the 70,000+ items in our learning platform). And you could integrate those models with GPT-style language models, so that one group of models supplies the facts and the other supplies the language.

That’s the most likely way forward. Given the number of startups that are building specialized fact-based models, it’s inconceivable that Google and Microsoft aren’t doing similar research. If they aren’t, they’ve seriously misunderstood the problem. It’s okay for a search engine to give you irrelevant or incorrect results. We see that with Amazon recommendations all the time, and it’s probably a good thing, at least for our bank accounts. It’s not okay for a search engine to try to convince you that incorrect results are correct, or to abuse you for challenging it. Will it take weeks, months, or years to iron out the problems with Microsoft’s and Google’s beta tests? The answer is: we don’t know. As Simon Willison suggests, the field is moving very fast, and can make surprising leaps forward. But the path ahead isn’t short.

View Details

Everybody knows about ChatGPT. And everybody knows about ChatGPT’s propensity to “make up” facts and details when it needs to, a phenomenon that’s come to be called “hallucination.” And everyone has seen arguments that this will bring about the end of civilization as we know it.

I’m not going to argue with any of that. None of us want to drown in masses of “fake news,” generated at scale by AI bots that are funded by organizations whose intentions are most likely malign. ChatGPT could easily outproduce all the world’s legitimate (and, for that matter, illegitimate) news agencies. But that’s not the issue I want to address.

I want to look at “hallucination” from another direction. I’ve written several times about AI and art of various kinds. My criticism of AI-generated art is that it’s all, well, derivative. It can create pictures that look like they were painted by Da Vinci–but we don’t really need more paintings by Da Vinci. It can create music that sounds like Bach–but we don’t need more Bach. What it really can’t do is make something completely new and different, and that’s ultimately what drives the arts forward. We don’t need more Beethoven. We need someone (or something) who can do what Beethoven did: horrify the music industry by breaking music as we know it and putting it back together differently. I haven’t seen that happening with AI. I haven’t yet seen anything that would make me think it might be possible. Not with Stable Diffusion, DALL-E, Midjourney, or any of their kindred.

Until ChatGPT. I haven’t seen this kind of creativity yet, but I can get a sense of the possibilities. I recently heard about someone who was having trouble understanding some software someone else had written. They asked ChatGPT for an explanation. ChatGPT gave an excellent explanation (it is very good at explaining source code), but there was something funny: it referred to a language feature that the user had never heard of. It turns out that the feature didn’t exist. It made sense, it was something that certainly could be implemented. Maybe it was discussed as a possibility in some mailing list that found its way into ChatGPT’s training data, but was never implemented? No, not that, either. The feature was “hallucinated,” or imagined. This is creativity–maybe not human creativity, but creativity nonetheless.

What if we viewed an an AI’s “hallucinations” as the precursor of creativity? After all, when ChatGPT hallucinates, it is making up something that doesn’t exist. (And if you ask it, it is very likely to admit, politely, that it doesn’t exist.) But things that don’t exist are the substance of art. Did David Copperfield exist before Charles Dickens imagined him? It’s almost silly to ask that question (though there are certain religious traditions that view fiction as “lies”). Bach’s works didn’t exist before he imagined them, nor did Thelonious Monk’s, nor did Da Vinci’s.

We have to be careful here. These human creators didn’t do great work by vomiting out a lot of randomly generated “new” stuff. They were all closely tied to the histories of their various arts. They took one or two knobs on the control panel and turned it all the way up, but they didn’t disrupt everything. If they had, the result would have been incomprehensible, to themselves as well as their contemporaries, and would lead to a dead end. That sense of history, that sense of extending art in one or two dimensions while leaving others untouched, is something that humans have, and that generative AI models don’t. But could they?

What would happen if we trained an AI like ChatGPT and, rather than viewing hallucination as error and trying to stamp it out, we optimized for better hallucinations? You can ask ChatGPT to write stories, and it will comply. The stories aren’t all that good, but they will be stories, and nobody claims that ChatGPT has been optimized as a story generator. What would it be like if a model were trained to have imagination plus a sense of literary history and style? And if it optimized the stories to be great stories, rather than lame ones? With ChatGPT, the bottom line is that it’s a language model. It’s just a language model: it generates texts in English. (I don’t really know about other languages, but I tried to get it to do Italian once, and it wouldn’t.) It’s not a truth teller; it’s not an essayist; it’s not a fiction writer; it’s not a programmer. Everything else that we perceive in ChatGPT is something we as humans bring to it. I’m not saying that to caution users about ChatGPT’s limitations; I’m saying it because, even with those limitations, there are hints of so much more that might be possible. It hasn’t been trained to be creative. It has been trained to mimic human language, most of which is rather dull to begin with.

Is it possible to build a language model that, without human interference, can experiment with “that isn’t great, but it’s imaginative. Let’s explore it more”? Is it possible to build a model that understands literary style, knows when it’s pushing the boundaries of that style, and can break through into something new? And can the same thing be done for music or art?

A few months ago, I would have said “no.” A human might be able to prompt an AI to create something new, but an AI would never be able to do this on its own. Now, I’m not so sure. Making stuff up might be a bug in an application that writes news stories, but it is central to human creativity. Are ChatGPT’s hallucinations a down payment on “artificial creativity”? Maybe so.

View Details

This month’s news seems to have been derailed by the three-ring circus: Musk and Twitter, Musk and Tesla, and SBF and FTX. That said, there are a lot of important things happening. We usually don’t say much about computing hardware, but RISC-V is gathering steam. I’m excited by Ion Stoica’s vision of “sky computing,” which is cloud-independent. A similar but even more radical project is Petals, which is a system for running the BLOOM large language model across a large number of volunteer hosts: cloud-free cloud computing, which the authors liken to Bittorrent. There’s been a lot of talk about decentralization; this is the real thing. That model for large-scale computation is more interesting, at least to me, than the ability to run one specific language model.

Artificial Intelligence* Adversarial learning tries to confuse machine learning systems by giving them altered input data, tricking them into giving incorrect answers. It is an important technique for improving AI security and accuracy. * We all know about AI-generated text, voices, and art; what about handwriting? Calligrapher.ai is a handwriting generator. It’s nowhere near as flexible as tools like Stable Diffusion, but it means that ChatGPT can not only write letters, it can sign them. * ChatGPT has been shown to be good at explaining code. It’s also good at re-writing code that has been intentionally obfuscated in a clear, human-readable version. There are clear applications (not all of them ethical) for this ability. * Who needs a database for an app’s backend? For that matter, who needs a backend at all? Just use GPT-3. * Reinforcement learning from human feedback (RLHF) is a machine learning training technique that integrates humans into the training loop. Humans provide additional rewards, in addition to automated rewards. RLHF, which was used in ChatGPT, could be a good way to build AI systems that are less prone to hate speech and similar problems. * Demis Hassabis, founder of DeepMind, advises that humans be careful in adopting AI. Don’t move fast and break things. * A group of researchers from Google has published a Deep Learning Tuning Playbook on Github. It recommends a procedure for hyperparameter tuning to optimize the performance of Deep Learning models. * Anthropic, a startup founded by former OpenAI researchers, has created a chatbot named Claude with capabilities similar to ChatGPT. Claude appears to be somewhat less prone to “hallucination” and hate speech, though they are still issues. * Satya Nadella has tweeted that Microsoft will offer ChatGPT as part of Azure’s OpenAI service. It isn’t clear how this (paid) service relates to other talk about monetizing ChatGPT. * One application for ChatGPT is writing documentation for developers, and providing a conversational search engine for the documentation and code. Writing internal documentation is an often omitted part of any software project. * AllenAI (aka AI2) has developed a language model called ACCoRD for generating descriptions of scientific concepts. It is unique in that it rejects the idea of a “best” description, and instead creates several descriptions of a concept, intended for different audiences. * A researcher trained a very small neural network to do binary addition, and had some fascinating observations about how the network works. * OpenAI is considering a paid, “pro” version of ChatGPT. It’s not clear what additional features the Pro version might have, what it would cost, or whether a free public version with lower performance will remain. The answers no doubt depend on Microsoft’s plans for further integrating ChatGPT into its products. * ChatGPT can create a text adventure game, including a multi-user dungeon (MUD) in which the other players are simulated. That’s not surprising in itself. The important question is whether these games have finite boundaries or extend for as long as you keep playing. * A startup has built a truth checker for ChatGPT. It filters ChatGPT’s output to detect “hallucinations,” using its own AI that has been trained for a specific domain. They claim to detect 90% of ChatGPT’s errors in a given domain. Users can add their own corrections. * Andrej Karpathy has written nanoGPT, a very small version of the GPT language models that can run on small systems–possibly even on a laptop. * Petals is a system for running large language models (specifically, BLOOM-176B, roughly the size of GPT-3) collaboratively. Parts of the computation run on different hosts, using compute time donated by volunteers who receive higher priority for their jobs. * Having argued that we would eventually see formal languages for prompting natural language text generators, I’m proud to say that someone has done it. * DoNotPay has developed an AI “lawyer” that is helping a defendant make arguments in court. The lawyer runs on a cell phone, through which it hears the proceedings. It tells the defendant what to say through Bluetooth earbuds. DoNotPay’s CEO notes that this is illegal in almost all courtrooms. (After receiving threats from Bar Associations, DoNotPay has abandoned this trial.) * Perhaps prompted by claims that Google’s AI efforts have fallen behind OpenAI and others, Google has announcedMuse, which generates images from text prompts. They claim that Muse is significantly faster and more accurate than DALL-E 2 and Stable Diffusion. * Microsoft has developed an impressive speech synthesis (text-to-speech) model named VALL-E. It is a zero-shot model that can imitate anyone’s voice using only a three-second sample. * Amazon has introduced Service Cards for several of their pre-built models (Rekognition, Textract, and Transcribe). Service cards describe the properties of models: how the model was trained, where the training data came from, the model’s biases and weaknesses. They are an implementation of Model Cards, proposed in Model Cards for Model Reporting. * The free and open source BLOOM language model can be run on AWS. Getting it running isn’t trivial, but there are instructions that describe how to get the resources you need.

Data* How do you use the third dimension in visualization? Jeffrey Heer (one of the creators of D3) and colleagues are writing about “cinematic visualization.” * SkyPilot is an open source platform for running data science jobs on any cloud: it is cloud-independent, and a key part of Ion Stoica’s vision of “sky computing” (provider-independent cloud computing).

Security* An annotated field guide to detecting phishing attack might help users to detect phishes before they do damage. According to one study from 2020, most cyber attacks begin with a phish. * Docker security scanning tools inspect Docker images for vulnerabilities and other issues. They could become an important part of software supply chain security. * Browser-in-browser phishing attacks are becoming more common, and are difficult to detect. In these attacks, a web site pops up a replica of a single sign-on window from Google, Facebook, or some other SSO provider to capture the user’s login credentials. * We’re again seeing an increase in advertisements delivering malware or attracting unwary users to web sites that install malware. Ad blockers provide some protection. * Amazon has announced that AWS automatically encrypts all new objects stored in S3. Encrypted by default is a big step forward in cloud data security. * The Python Package Index (PyPI) continues to suffer from attacks that cause users to install packages infected with malware. Most notably, the PyTorch nightly build was linked to a version that would steal system information. Software supply chain problems continue to plague us. * Messaging provider Slack and continuous integration provider CircleCI were both victims of attacks and thefts of software and data. The companies haven’t been forthcoming with details, but it seems likely that CircleCI has lost all customer secrets.

Programming* GPU.js is a JavaScript library that transpiles and compiles simple JavaScript functions to run on a GPU. * Libsodium is being used to benchmark WebAssembly, which is gradually becoming a mainstream technology. * Julia Evans (@b0rk, @b0rk@mastodon.social) has an excellent discussion of the problems that arise from using floating point arithmetic carelessly. * Platform engineering may be the latest buzzword, but building reliable pipelines and tools for self-service development and deployment delivers important benefits for programmers and their companies. * Codeium is an open source code completion engine, like Copilot, that plugs into Vim. It isn’t clear what kind of language model Codeium uses. * YouPlot is a terminal-based plotting tool: no fancy graphics, just your standard terminal window. Quick and easy. * Tetris can be used to implement a general purpose digital computer that, among other things, is capable of running Tetris.

Chips and Chip Design* A new generation of processors could use vibration to generate a flow of air through the chip, providing cooling without the need for fans. The developers are collaborating with Intel and targeting high-end laptops. * Google wants RISC-V to become a “tier-1” chip architecture for Android phones, giving it the same status as ARM. There is already a riscv64 branch in the source repository, though it’s far from a finished product. * Ripes is a visual computer architecture simulator for the RISC-V. You can watch your code execute (slowly). It’s primarily a tool for teaching, but it’s fun to play with.

Things* Boston Dynamics’ humanoid robot Atlas now has the ability to grab and toss things (including awkward and heavy objects). This is a big step towards a robot that can do industrial or construction work. * Matter, a standard for smart home connectivity, appears to be gaining momentum. Among other things, it allows devices to interact with a common controller, rather than an app (and possibly a hub) for each device. * Science fiction alert: Researchers have created a tractor beam! While it’s very limited, it is capable of pulling specially constructed macroscopic objects. * A new catalyst has enabled a specialized solar cell to achieve 9% efficiency in generating hydrogen from water. This is a factor of 10 better than other methods, and approaches the efficiency needed to make “green hydrogen” commercially viable.

Web* A not-so private metaverse: Someone has built a “private metaverse” (hosted on a server somewhere for about $12/month) to display his art and to demonstrate that a metaverse can be open, and doesn’t have to be subject to land-grabs and rent-taking by large corporations. * Twitter has cut off API access for third party apps. This was a big mistake the first time (a decade ago); it’s an even bigger mistake now. * GoatCounter is an alternative to Google Analytics. It provides “privacy-friendly” web analytics. It can be self-hosted, or used as a service (free to non-commercial users). * Google is developing a free tool that websites can use to detect and remove material associated with terrorism, as an aid to help moderators.

Biology* Where do we go next with mRNA vaccines? Flu, Zika, HIV, cancer treatments? The vaccines are relatively easy to design and to manufacture.

View Details

What would you say is the job of a software developer? A layperson, an entry-level developer, or even someone who hires developers will tell you that job is to … well … write software. Pretty simple.

An experienced practitioner will tell you something very different. They’d say that the job involves writing some software, sure. But deep down it’s about the purpose of software. Figuring out what kinds of problems are amenable to automation through code. Knowing what to build, and sometimes what not to build because it won’t provide value.

They may even summarize it as: “my job is to spot for() loops and if/then statements in the wild.”

I, thankfully, learned this early in my career, at a time when I could still refer to myself as a software developer. Companies build or buy software to automate human labor, allowing them to eliminate existing jobs or help teams to accomplish more. So it behooves a software developer to spot what portions of human activity can be properly automated away through code, and then build that.

This mindset has followed me into my work in ML/AI. Because if companies use code to automate business rules, they use ML/AI to automate decisions.

Given that, what would you say is the job of a data scientist (or ML engineer, or any other such title)?

I’ll share my answer in a bit. But first, let’s talk about the typical ML workflow.

Building ModelsA common task for a data scientist is to build a predictive model. You know the drill: pull some data, carve it up into features, feed it into one of scikit-learn’s various algorithms. The first go-round never produces a great result, though. (If it does, you suspect that the variable you’re trying to predict has mixed in with the variables used to predict it. This is what’s known as a “feature leak.”) So now you tweak the classifier’s parameters and try again, in search of improved performance. You’ll try this with a few other algorithms, and their respective tuning parameters–maybe even break out TensorFlow to build a custom neural net along the way–and the winning model will be the one that heads to production.

You might say that the outcome of this exercise is a performant predictive model. That’s sort of true. But like the question about the role of the software developer, there’s more to see here.

Collectively, your attempts teach you about your data and its relation to the problem you’re trying to solve. Think about what the model results tell you: “Maybe a random forest isn’t the best tool to split this data, but XLNet is.” If none of your models performed well, that tells you that your dataset–your choice of raw data, feature selection, and feature engineering–is not amenable to machine learning. Perhaps you need a different raw dataset from which to start. Or the necessary features simply aren’t available in any data you’ve collected, because this problem requires the kind of nuance that comes with a long career history in this problem domain. I’ve found this learning to be a valuable, though often understated and underappreciated, aspect of developing ML models.

Second, this exercise in model-building was … rather tedious? I’d file it under “dull, repetitive, and predictable,” which are my three cues that it’s time to automate a task.

• Dull: You’re not here for the model itself; you’re after the results. How well did it perform? What does that teach me about my data?
• Repetitive: You’re trying several algorithms, but doing roughly the same thing each time.
• Predictable: The scikit-learn classifiers share a similar interface, so you can invoke the same train() call on each one while passing in the same training dataset.

Yes, this calls for a for() loop. And data scientists who came from a software development background have written similar loops over the years. Eventually they stumble across GridSearchCV, which accepts a set of algorithms and parameter combinations to try. The path is the same either way: setup, start job, walk away. Get your results in a few hours.

Building a Better for() loop for MLAll of this leads us to automated machine learning, or autoML. There are various implementations–from the industrial-grade AWS SageMaker Autopilot and Google Cloud Vertex AI, to offerings from smaller players–but, in a nutshell, some developers spotted that same for() loop and built a slick UI on top. Upload your data, click through a workflow, walk away. Get your results in a few hours.

If you’re a professional data scientist, you already have the knowledge and skills to test these models. Why would you want autoML to build models for you?

• It buys time and breathing room. An autoML solution may produce a “good enough” solution in just a few hours. At best, you’ll get a model you can put in production right now (short time-to-market), buying your team the time to custom-tune something else (to get better performance). At worst, the model’s performance is terrible, but it only took a few mouse clicks to determine that this problem is hairier than you’d anticipated. Or that, just maybe, your training data is no good for the challenge at hand.
• It’s convenient. Damn convenient. Especially when you consider how Certain Big Cloud Providers treat autoML as an on-ramp to model hosting. It takes a few clicks to build the model, then another few clicks to expose it as an endpoint for use in production. (Is autoML the bait for long-term model hosting? Could be. But that’s a story for another day.) Related to the previous point, a company could go from “raw data” to “it’s serving predictions on live data” in a single work day.
• You have other work to do. You’re not just building those models for the sake of building them. You need to coordinate with stakeholders and product managers to suss out what kinds of models you need and how to embed them into the company’s processes. And hopefully they’re not specifically asking you for a model, but asking you to use the company’s data to address a challenge. You need to spend some quality time understanding all of that data through the lens of the company’s business model. That will lead to additional data cleaning, feature selection, and feature engineering. Those require the kind of context and nuance that the autoML tools don’t (and can’t) have.

Software Is Hungry, May as Well Feed ItRemember the old Marc Andreessen line that software is eating the world?

More and more major businesses and industries are being run on software and delivered as online services — from movies to agriculture to national defense. Many of the winners are Silicon Valley-style entrepreneurial technology companies that are invading and overturning established industry structures. Over the next 10 years, I expect many more industries to be disrupted by software, with new world-beating Silicon Valley companies doing the disruption in more cases than not.

This was the early days of developers spotting those for() loops and if/then constructs in the wild. If your business relied on a hard-and-fast rule, or a predictable sequence of events, someone was bound to write code to do the work and throw that on a few dozen servers to scale it out.

And it made sense. People didn’t like performing the drudge work. Getting software to take the not-so-fun parts separated duties according to ability: tireless repetition to the computers, context and special attention to detail to the humans.

Andreessen wrote that piece more than a decade ago, but it still holds. Software continues to eat the world’s dull, repetitive, predictable tasks. Which is why software is eating AI.

(Don’t feel bad. AI is also eating software, as with GitHub’s Copilot. Not to mention, some forms of creative expression. Stable Diffusion, anyone? The larger lesson here is that automation is a hungry beast. As we develop new tools for automation, we will bring more tasks within automation’s reach.)

Given that, let’s say that you’re a data scientist in a company that’s adopted an autoML tool. Fast-forward a few months. What’s changed?

Your Team Looks DifferentIntroducing autoML into your workflows has highlighted three roles on your data team. The first is the data scientist who came from a software development background, someone who’d probably be called a “machine learning engineer” in many companies. This person is comfortable talking to databases to pull data, then calling Pandas to transform it. In the past they understood the APIs of TensorFlow and Torch to build models by hand; today they are fluent in the autoML vendor’s APIs to train models, and they understand how to review the metrics.

The second is the experienced ML professional who really knows how to build and tune models. That model from the autoML service is usually good, but not great, so the company still needs someone who can roll up their sleeves and squeeze out the last few percentage points of performance. Tool vendors make their money by scaling a solution across the most common challenges, right? That leaves plenty of niches the popular autoML solutions can’t or won’t handle. If a problem calls for a shiny new technique, or a large, branching neural network, someone on your team needs to handle that.

Closely related is the third role, someone with a strong research background. When the well-known, well-supported algorithms no longer cut the mustard, you’ll need to either invent something whole cloth or translate ideas out of a research paper. Your autoML vendor won’t offer that solution for another couple of years, so, it’s your problem to solve if you need it today.

Notice that a sufficiently experienced person may fulfill multiple roles here. It’s also worth mentioning that a large shop probably needed people in all three roles even before autoML was a thing.

(If we twist that around: aside from the FAANGs and hedge funds, few companies have both the need and the capital to fund an ongoing ML research function. This kind of department provides very lumpy returns–the occasional big win that punctuates long stretches of “we’re looking into it.”)

That takes us to a conspicuous omission from that list of roles: the data scientists who focused on building basic models. AutoML tools are doing most of that work now, in the same way that the basic dashboards or visualizations are now the domain of self-service tools like AWS QuickSight, Google Data Studio, or Tableau. Companies will still need advanced ML modeling and data viz, sure. But that work goes to the advanced practitioners.

In fact, just about all of the data work is best suited for the advanced folks. AutoML really took a bite out of your entry-level hires. There’s just not much for them to do. Only the larger shops have the bandwidth to really bring someone up to speed.

That said, even though the team structure has changed, you still have a data team when using an autoML solution. A company that is serious about doing ML/AI needs data scientists, machine learning engineers, and the like.

You Have Refined Your Notion of “IP”The code written to create most ML models was already a commodity. We’re all calling into the same Pandas, scikit-learn, TensorFlow, and Torch libraries, and we’re doing the same “convert data into tabular format, then feed to the algorithm” dance. The code we write looks very similar across companies and even industries, since so much of it is based on those open-source tools’ call semantics.

If you see your ML models as the sum total of algorithms, glue code, and training data, then the harsh reality is that your data was the only unique intellectual property in the mix anyway. (And that’s only if you were building on proprietary data.) In machine learning, your competitive edge lies in business know-how and ability to execute. It does not exist in the code.

AutoML drives this point home. Instead of invoking the open-source scikit-learn or Keras calls to build models, your team now goes from Pandas data transforms straight to … the API calls for AWS AutoPilot or GCP Vertex AI. The for() loop that actually builds and evaluates the models now lives on someone else’s systems. And it’s available to everyone.

Your Job Has ChangedBuilding models is still part of the job, in the same way that developers still write a lot of code. While you called it “training an ML model,” developers saw “a for() loop that you’re executing by hand.” It’s time to let code handle that first pass at building models and let your role shift accordingly.

What does that mean, then? I’ll finally deliver on the promise I made in the introduction. As far as I’m concerned, the role of the data scientist (and ML engineer, and so on) is built on three pillars:

• Translating to numbers and back. ML models only see numbers, so machine learning is a numbers-in, numbers-out game. Companies need people who can translate real-world concepts into numbers (to properly train the models) and then translate the models’ numeric outputs back into a real-world context (to make business decisions). Your model says “the price of this house should be $542,424.86”? Great. Now it’s time to explain to stakeholders how the model came to that conclusion, and how much faith they should put in the model’s answer.
• Understanding where and why the models break down: Closely related to the previous point is that models are, by definition, imperfect representations of real-world phenomena. When looking through the lens of your company’s business model, what is the impact of this model being incorrect? (That is: what model risk does the company face?)

My friend Roger Magoulas reminded me of the old George Box quote that “all models are wrong, but some are useful.” Roger emphasized that we must consider the full quote, which is:

Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.

• Spotting ML opportunities in the wild: Machine learning does four things well: prediction (continuous outputs), classification (discrete outputs), grouping things (“what’s similar?”), and catching outliers (“where’s the weird stuff?”). In the same way that a developer can spot for() loops in the wild, experienced data scientists are adept at spotting those four use cases. They can tell when a predictive model is a suitable fit to augment or replace human activity, and more importantly, when it’s not. Sometimes this is as straightforward as seeing where a model could guide people. Say you overhear the sales team describing how they lose so much time chasing down leads that don’t work. The wasted time means they miss leads that probably would have panned out. “You know … Do you have a list of past leads and how they went? And are you able to describe them based on a handful of attributes? I could build a model to label a deal as a go/no-go. You could use the probabilities emitted alongside those labels to prioritize your calls to prospects.”

Other times it’s about freeing people from mind-numbing work, like watching security cameras. “What if we build a model to detect motion in the video feed? If we wire that into an alerts system, our staff could focus on other work while the model kept a watchful eye on the factory perimeter.”

And then, in rare cases, you sort out new ways to express ML’s functionality. “So … when we invoke a model to classify a document, we’re really asking for a single label based on how it’s broken down the words and sequences in that block of text. What if we go the other way? Could we feed a model tons of text, and get it to produce text on demand? And what if that could apply to, say, code?”

It Always Has Been From a high level, then, the role of the data scientist is to understand data analysis and predictive modeling, in the context of the company’s use cases and needs. It always has been. Building models was just on your plate because you were the only one around who knew how to do it. By offloading some of the model-building work to machines, autoML tools remove some of that distraction, allowing you to focus more on the data itself.

The data is certainly the most important part of all this. You can consider the off-the-shelf ML algorithms (available as robust, open-source implementations) and unlimited compute power (provided by cloud services) as constants. The only variable in your machine learning work–the only thing you can influence in your path to success–is the data itself. Andrew Ng emphasizes this point in his drive for data-centric AI, and I wholeheartedly agree.

Making the most of that data will require that you understand where it came from, assess its quality, and engineer it into features that the algorithms can use. This is the hard part. And it’s the part we can’t yet hand off to a machine. But once you’re ready, you can hand those features off to an autoML tool–your trusty assistant that handles the grunt work–to diligently use them to train and compare various models.

Software has once again eaten dull, repetitive, predictable tasks. And it has drawn a dividing line, separating work based on ability.

Where to Next?Some data scientists might claim that autoML is taking their job away. (We will, for the moment, skip past the irony of someone in tech complaining that a robot is taking their job.) Is that true, though? If you feel that building models is your job, then, yes.

For the more experienced readers, autoML tools are a slick replacement for their trusty-but-rusty homegrown for() loops. A more polished solution for doing a first pass at building models. They see autoML tools, not as a threat, but as a force multiplier that will test a variety of algorithms and tuning parameters while they tackle the important work that actually requires human nuance and experience. Pay close attention to this group, because they have the right idea.

The data practitioners who embrace autoML tools will use their newfound free time to forge stronger connections to the company’s business model. They’ll look for novel ways to apply data analysis and ML models to products and business challenges, and try to find those pockets of opportunity that autoML tools can’t handle.

If you have entrepreneurship in your blood, you can build on that last point and create an upstart autoML company. You may hit on something the big autoML vendors don’t currently support, and they’ll acquire you. (I currently see an opening for clustering-as-a-service, in case you’re looking for ideas.) Or if you focus on a niche that the big players deem too narrow, you may get acquired by a company in that industry vertical.

Software is hungry. Find ways to feed it.

View Details

Although I don’t subscribe to the idea that history or technology moves in jerky one-year increments, it’s still valuable to take stock at the start of a new year, look at what happened last year, and decide what was important and what wasn’t.

We started the year with many people talking about an “AI winter.” A quick Google search shows that anxiety about an end to AI funding has continued through the year. Funding comes and goes, of course, and with the possibility of a media-driven recession, there’s always the possibility of a funding collapse. Funding aside, 2022 has been a fantastic year for AI. GPT-3 wasn’t new, of course, but ChatGPT made GPT-3 usable in ways people hadn’t imagined. How will we use ChatGPT and its descendants? I don’t believe they put an end to search. When I search, I’m (usually) more interested in the source than I am in an “answer.” But I have a question. Much has been made about ChatGPT’s ability to “hallucinate” facts. I wonder whether that kind of hallucination could be a prelude to “artificial creativity”? I’ll try to have something more to say about that in the coming year.

GitHub CoPilot also wasn’t new in 2022, but in the last year we’ve heard of more and more programmers who are using ChatGPT to write production code. It isn’t just people “kicking the tires”; AI-generated code will inevitably be part of the future. The important questions are: who will it help, and how? Right now, it seems like CoPilot will be less likely to help beginners, and more likely to be a force-multiplier for experienced programmers, allowing them to focus more on what they are trying to do than on remembering details about syntax and libraries. In the longer term, it might bring about a complete change in what “computer programming” means.

DALL-E 2, Stable Diffusion, and Midjourney made it possible for people without artistic skills to generate pictures based on verbal descriptions, with results that are often fantastic. Google and Facebook haven’t released anything to the public, but they have demoed similar applications. All of these tools are raising important questions about intellectual property and copyright. They are already inspiring new startups with new applications, and those companies will inevitably attract investment.

Those tools aren’t without their problems, and if we really want to avoid another AI Winter, we’d do well to think about what those problems are. Intellectual property is one issue: GitHub is already being sued because CoPilot’s output can reproduce code that it was trained on, without regard for the code’s initial license. The art generation programs will inevitably face similar challenges: what happens when you tell an AI system to produce a drawing “in the style of” some artist? What happens when you ask the AI to create an avatar for a woman, and it creates something that’s highly sexualized? ChatGPT’s ability to produce plausible text output is spectacular, but its ability to discriminate fact from non-fact is limited. Will we see a Web that’s flooded with “fake news” and spam? We arguably have that already, but tools like ChatGPT can generate content at a scale that we can’t yet imagine.

At its heart, ChatGPT is really a user interface hack: a chat front end bolted onto an updated version of the GPT-3 language model. “User interface hack” sounds pejorative, but I don’t mean it that way. We now need to start building new applications around these models. UI design is important–and UI design for AI applications is a topic that hasn’t been adequately explored. What can we build with large language and generative art models? How will these models interact with their human users? Exploring those questions will drive a lot of creativity.

After ChatGPT, perhaps the biggest surprise of 2022 was the rise of Mastodon. Mastodon isn’t new, of course; I’ve been looking in from the outside for some time. I’ve never thought it had achieved critical mass, or that it was capable of achieving critical mass. I was proven wrong when Elon Musk’s antics drove thousands of Twitter users to Mastodon (including me). Mastodon is a federated network of communities that are (mostly) pleasant, friendly, and populated by smart people. The sudden influx of Twitter users proved that Mastodon could scale. There were some growing pains, but not as much as I would have expected. I haven’t seen a single “fail whale.”

The growth of Mastodon proved that the federated model worked. It’s important to think about this. Mastodon is a decentralized service based on the ActivityPub protocol. Nobody owns it; nobody controls it, though individuals control specific servers. And there isn’t a blockchain or a token in sight. In the past year, we’ve been treated to a steady diet of noise about Web3, most of which insists that the next step in online interaction must be built on a blockchain, that everything must be owned, everything must be paid for, and that rent collectors (aka “miners”) will have their hands out taking their cut on each transaction. I won’t go so far as to claim that Mastodon is Web3; but I do think that the next generation of the Web, however it evolves, will look much more like Mastodon than like OpenSea, and that it will be based on protocols like ActivityPub.

Which leads us to blockchains and crypto. I’m not going to engage in Schadenfreude here, but I’ve long wondered what can be built with blockchains. At one time, I thought that supply chain management would be the poster child for the Enterprise Blockchain. Unfortunately, IBM and Maersk have abandoned their TradeLens project. NFTs? I have always been skeptical of the connection between NFTs and the art world. NFTs seemed an awful lot like buying a painting and framing the receipt. They existed purely to show that you could spend cryptocurrency at scale, and the people who spent their coins that way have gotten what they deserved. But I’m not willing to say that there’s no value here. NFTs may help us to solve the problem of online identity, a problem that we haven’t yet solved on the Web (though I’m not convinced that NFT advocates have really understood how complex identity is). Are there other applications? A number of companies, including Starbucks and Universal Studios, are using NFTs to build customer loyalty programs and theme park experiences. At this point, NFTs still look like a technology in search of a problem to solve, but I suspect that the appropriate problem isn’t out there.

There was more in 2022, of course. Will we see a Metaverse, or was that just Facebook’s attempt to change the narrative about its actions? Will Europe continue to take the lead in regulating the tech sector, and will other nations follow? Will our daily lives be improved by a flood of interoperable smart devices? In 2023, we shall see.

View Details

Perhaps unsurprisingly, December was a slow month. Blog posts and articles dropped off over the holidays; the antics of Sam Bankman-Fried and Elon Musk created a lot of distractions. While we won’t engage in Schadenfreude over the Twitter exodus, or SBF’s fall from the financial firmament, the most interesting news of the month is the rise of Mastodon. Mastodon isn’t new, and it doesn’t yet challenge the major social media players. But it’s real, it’s scaling, and its federated model presents a different way of thinking about social media, services, and (indeed) Web3. And ChatGPT? Yes, everyone was talking about it. It’s been known to impersonate Linux, help developers learn new programming languages, and even improve traditional college courses (where its ability to make mistakes can be turned into an asset).

AI* One developer has integrated ChatGPT into an IDE, where it can answer questions about the codebase he’s working on. This application promises to be incredibly useful to programmers who are working on large software projects. * While most of the discussion around ChatGPT swirls around errors and hallucinations, one college professor has started to use ChatGPT as a teaching tool. His ideas focus on ChatGPT’s flaws: for example, having it write an essay for students to analyze and correct. * Geoff Hinton proposes forward-forward neural networks, which may be as effective as backpropagation while requiring much less power to train. He also proposes new hardware architectures for artificial intelligence. * Riffusion is a generative model based on Stable Diffusion that creates sound by generating spectrograms. Riffusion doesn’t work with sound itself; it only produces the spectrogram, which can be converted to sound downstream. * A deluge of content generated by AI has the potential to “poison” public sources of training data. What does it mean to train an AI on data that comes from another AI, rather than a human? * DeepMind’s AlphaCode has scored better than 45% of human programmers in a coding competition. Their most important innovation appears to be generating many solutions to a problem and running some simple test cases to select which solutions to submit. * Stability AI has announced that artists may remove their work from the training set used to build Stable Diffusion 3. Opting out requires creating an account on Have I Been Trained and uploading images to be excluded. * The World Cup used an AI “referee” to assist officials in detecting when players are offside. The system incorporates input from (among other things) a “connected ball” that provided position updates 500 times per second. * XetHub is “a collaborative storage platform for managing data at scale.” Essentially, it’s GitHub for data. It appears to be built on top of Git, but with a different approach to minimizing duplication, managing large objects, and supporting different file types. It supports repos up to 1TB, with plans to go to 100TB. * Large language models can be used to understand physician’s notes. While these notes are recorded in electronic health records, they are full of abbreviations, many of which are idiosyncratic and difficult for anyone other than the author to understand. * ChatGPT’s training set included a lot of information about Linux, so you can tell it to act like a Linux terminal. You’ll get a shell prompt, along with a simulated filesystem. Most system commands work, and even some programming–though the output is predicted from the training set, not the result of actually running a program. Is this the future of operating systems? * Simon Willison is using ChatGPT and Copilot to learn Rust by solving problems from Advent of Code. Although ChatGPT occasionally hallucinates answers, it is surprisingly accurate, and capable of explaining what the code it generates is doing. * While ChatGPT’s ability to hold a conversation is impressive, its accuracy is not. StackOverflow has prohibited posts generated by ChatGPT because of incorrect answers. * Diffusion models, the AI models on which generative art tools like DALL-E are based, are being used to design new proteins that have specific properties. It is then possible to synthesize these proteins in a lab. These new proteins could lead to new kinds of drugs. * Adrian Holavaty’s experiments in music generation using ChatGPT are interesting. Adrian isn’t (yet) trying to get ChatGPT to compose new music; it’s more like “Give me Twinkle Twinkle in MusicML.” Still, within limits, the chat server can do it. * OpenAI is continuing to improve GPT-3. A variant of GPT-3 has been trained to admit when it doesn’t know something, and is less prone to generating inappropriate responses. However, there are still many shortcomings. * AI was used to edit swear words out of a movie in production without reshooting any scenes, getting its MPAA rating from R down to PG-13. * Scott Aaronson’s lecture summarizing his work (to date) on AI safety is worth reading.

Programming* Dioxus is a library for write-once-run-anywhere Web and Mobile programing in Rust. * Fission is a web-native (as distinct from cloud native) computing stack that is truly local-first. It was designed to build distributed systems like Mastodon (though Mastodon doesn’t use it at this point) that don’t have central servers, and that can scale. * GitHub requires all users to enable two-factor authentication by the end of 2023. They have also enabled secret scanning for free on all public repositories. Secret scanning inspects code for authentication credentials and other secrets that may have been inadvertently left in code. * Is no code test automation the next trend in software testing? And looking to the future, is it a stepping stone to fully automated testing using artificial intelligence? * JavaScript on the edge? Will JavaScript become the common language for edge computing? That depends in part on what edge computing really means, and that continues to be vague. Is “edge computing” just caching on CDNs? * Stephen O’Grady suggests some heuristics to evaluate an organization’s commitment to developer experience. * Automated reasoning about programs is a useful adjunct to testing. The Halting Problem doesn’t mean that reasoning about errors in code is impossible; it just means that we (occasionally) have to accept “don’t know” as an answer. * Julia Evans (@b0rk) has an excellent set of tips for analyzing logs. Julia has also offered a Debugging Manifesto. * AWS Clean Rooms are a new service that allows organizations to cooperate on data analysis without revealing the underlying data to each other. * WasmEdge is a lightweight Web Assembly runtime that’s built for cloud native applications, edge computing applications, and embedded systems.

Security* A security breach at LastPass, first reported last August, is worse than the company admitted. Customer information was stolen, including customer vaults containing sensitive information. The vaults are (probably) still protected by customers’ master passwords, though it’s possible the attackers have found a back door. * A new wiper malware, called Azov, is spreading rapidly in the wild. Azov is a sophisticated piece of software that is purely destructive: it overwrites files with random data. Recovery is impossible, aside from restoring from backup. * Any new technology has security risks. Here’s a summary of security risks that developers working with WebAssembly should be aware of. * Bettercap is a next-generation tool for exploring networks: scanning and probing WiFi and Bluetooth, in addition to Ethernet, spoofing common network protocols, and many other features. It’s an all-in-one tool for network reconnaissance and attacks.

Biology* In Greenland, scientists have found and sequenced 2 million year old DNA. The DNA comes from a number of different plants and animals (including mastodons), and gives a picture of what Greenland was like when it had a warmer climate.

Metaverse* Nokia argues that the Industrial Metaverse will be centered on digital twins: computer simulations that run in parallel to real-world systems. * Webspaces are a new kind of website that can create 3D worlds, using nothing but static HTML. Webspaces preserve (or reclaim) much of the vision of the early Web: learning by copying and pasting from others’ sites, editing in the browser as an editor, and self-hosting. * Fashion may be the Metaverse’s first killer app. Though it’s fashion that only exists in the Metaverse–a constraint that’s both freeing and limiting. * Build your own Decentralized Twitter is a good introduction (first of three parts) to building federated services. Mastodon is the most prominent example of a federated service, but there are many more applications.

Web* Although compatibility issues remain, the latest release of the Chrome browser supports passkeys, a replacement for passwords and password managers that is much more secure. * danah boyd has published an must-read essay on social media, failure, and Twitter. danah doesn’t draw any conclusions, but gives an excellent analysis of what failure means. * The Brave browser is now showing “privacy preserving” ads in its search results. These ads are currently in a limited beta. Ads will be based only on search query, country, and device type. Brave also plans to release a for-pay ad-free browser.

Web3* The venerable WinAmp MP3 player now supports music NFTs. It can be linked to a Metamask wallet, and can download and play files that have been purchased via NFT.

Regulation* Europe has become the de facto leader in regulating technology; it’s safe to predict that Europe will implement regulations about cybersecurity, algorithmic accountability, and cryptocurrency in the coming year–and that technology companies will have to comply. It’s less clear whether these changes will have any effect outside of Europe. * Privacy regulators in Europe have ruled that it is illegal for Facebook to track user’s activity without explicit consent. This ruling seriously limits Facebook’s ability to use targeted ads.

View Details

The current generation of flashy AI applications, ranging from GitHub Copilot to Stable Diffusion, raise fundamental issues with copyright law. I am not an attorney, but these issues need to be addressed–at least within the culture that surrounds the use of these models, if not the legal system itself.

Copyright protects outputs of creative processes, not inputs. You can copyright a work you produced, whether that’s a computer program, a literary work, music, or an image. There is a concept of “fair use” that’s most applicable to text, but still applicable in other domains. The problem with fair use is that it is never precisely defined. The US Copyright Office’s statement about fair use is a model for vagueness:

Under the fair use doctrine of the U.S. copyright statute, it is permissible to use limited portions of a work including quotes, for purposes such as commentary, criticism, news reporting, and scholarly reports. There are no legal rules permitting the use of a specific number of words, a certain number of musical notes, or percentage of a work. Whether a particular use qualifies as fair use depends on all the circumstances.

We are left with a web of conventions and traditions. You can’t quote another work in its entirety without permission. For a long time, it was considered acceptable to quote up to 400 words without permission, though that number was never codified into law, and has been going down recently–and counting words applies poorly to software as well as works that aren’t written text. Elsewhere the US copyright office states that fair use includes ”transformative” use, though “transformative” has never been defined precisely. It also states that copyright does not extend to ideas or facts, only to particular expressions of those facts–but we have to ask where the “idea” ends and where the “expression” begins. Interpretation of these principles will have to come from the courts, and the body of US case law on software copyright is surprisingly small–only 13 cases, according to the copyright office’s search engine. Although the body of case law for music and other art forms is larger, it’s even less clear how these ideas apply. Just as quoting a poem in its entirety is a copyright violation, you can’t reproduce images in their entirety without permission. But how much of a song or a painting can you reproduce? Counting words isn’t just ill-defined, it is useless for works that aren’t made of words.

These rules of thumb are clearly about outputs, rather than inputs: again, the ideas that go into an article aren’t protected, just the words. That’s where generative models present problems. Under some circumstances, output from Copilot may contain, verbatim, lines from copyrighted code. The legal system has tools to handle this case, even if those tools are imprecise. Microsoft is currently being sued for “software piracy” because of GitHub. The case is based on outputs: code generated by Copilot that reproduces code in its training set, but that doesn’t carry license notices or attribution. It’s about Copilot’s compliance with the license attached to the original software. However, that lawsuit doesn’t address the more important question. Copilot itself is a commercial product that is built a body of training data, even though it is completely different from that data. It’s clearly “transformative.” In any AI application, the training data is at least as important to the final product as the algorithms, if not more important. Should the rights of the authors of the training data be taken into account when a model is built from their work, even if the model never reproduces their work verbatim? Copyright does not adequately address the inputs to the algorithm at all.

We can ask similar questions about works of art. Andy Baio has a great discussion of an artist, Hollie Mengert, whose work was used to train a specialized version of Stable Diffusion. This model enables anyone to produce Mengert-like artworks from a textual prompt. They’re not actual reproductions; and they’re not as good as her genuine artworks–but arguably “good enough” for most purposes. (If you ask Stable Diffusion to generate “Mona Lisa in the style of DaVinci,” you get something that clearly looks like Mona Lisa, but that would embarrass poor Leonardo.) However, users of a model can produce dozens, or hundreds, of works in the time Mengert takes to make one. We certainly have to ask what it does to the value of Mengert’s art. Does copyright law protect “in the style of”? I don’t think anyone knows. Legal arguments over whether works generated by the model are “transformative” would be expensive, possibly endless, and likely pointless. (One hallmark of law in the US is that cases are almost always decided by people who aren’t experts. The Grotesque Legacy of Music as Property shows how this applies to music.) And copyright law doesn’t protect the inputs to a creative process, whether that creative process is human or cybernetic. Should it? As humans, we are always learning from the work of others; “standing on the shoulders of giants” is a quote with a history that goes well before Isaac Newton used it. Are machines also allowed to stand on the shoulders of giants?

Mona Lisa in the style of DaVinci. DaVinci isn’t worried. (Courtesy Hugo Bowne-Anderson)To think about this, we need an understanding of what copyright does culturally. It’s a double-edged sword. I’ve written several times about how Beethoven and Bach made use of popular tunes in their music, in ways that certainly wouldn’t be legal under current copyright law. Jazz is full of artists quoting, copying, and expanding on each other. So is classical music–we’ve just learned to ignore that part of the tradition. Beethoven, Bach, and Mozart could easily have been sued for their appropriation of popular music (for that matter, they could have sued each other, and been sued by many of their “legitimate” contemporaries)–but that process of appropriating and moving beyond is a crucial part of how art works.

J. S. Bach’s 371 Choral Copyright Violations. He would have been in trouble if copyright as we now understand it had existed.We also have to recognize the protection that copyright gives to artists. We lost most of Elizabethan theater because there was no copyright. Plays were the property of the theater companies (and playwrights were often members of those companies), but that property wasn’t protected; there was nothing to prevent another company from performing your play. Consequently, playwrights had no interest in publishing their plays. The scripts were, literally, trade secrets. We’ve probably lost at least one play by Shakespeare (there’s evidence he wrote a play called Love’s Labors Won); we’ve lost all but one of the plays of Thomas Kyd; and there are other playwrights known through playbills, reviews, and other references for whom there are no surviving works. Christopher Marlowe’s Doctor Faustus, the most important pre-Shakespearian play, is known to us through two editions, both published after Marlowe’s death, and one of those editions is roughly a third longer than the other. What did Marlowe actually write? We’ll never know. Without some kind of protection, authors had no interest in publishing at all, let alone publishing accurate texts.

So there’s a finely tuned balance to copyright, which we almost certainly haven’t achieved in practice. It needs to protect creativity without destroying the ability to learn from and modify earlier works. Free and open source software couldn’t exist without the protection of copyright–though without that protection, open source might not be needed. Patents were intended to play a similar role: to encourage the spread of information by guaranteeing that inventors could profit from their invention, limiting the need for “trade secrets.”

Copying works of art has always been (and still is) a part of an artist’s education. Authors write and rewrite each other’s works constantly; whole careers have been made tracing the interactions between John Milton and William Blake. Whether we’re talking about prose or painting, generative AI devalues traditional artistic technique (as I’ve argued), though possibly giving rise to a different kind of technique: the technique of writing prompts that tell the machine what to create. That’s a task that is neither simple nor uncreative. To take Mona Lisa and go a step further than Da Vinci–or to go beyond facile imitations of Hollie Mengert–requires an understanding of what this new medium can do, and how to control it. Part of Google’s AI strategy appears to be building tools that help artists to collaborate with AI systems; their goal is to enable authors to create works that are transformative, that do more than simply reproducing a style or piecing together sentences. This kind of work certainly raises questions of reproducibility: given the output of an AI system, can that output be recreated or modified in predictable ways? And it might cause us to realize that the old cliche “A picture is worth a thousand words” significantly underestimates the number of words it takes to describe a picture.

How do we best protect creative freedom? Is a work of art something that can be “owned,” and what does that mean in an age when digital works can be reproduced perfectly, at will? We need to protect both the original artists, like Hollie Mengert, and those who use their original work as a springboard to go beyond. Our current copyright system does that poorly, if at all. (And the existence of patent trolls demonstrates that patent law hasn’t done much better.) What was originally intended to protect artists has turned into a rent-seeking game in which artists who can afford lawyers monetize the creativity of artists who can’t. Copyright needs to protect the input side of any generative system: it needs to govern the use of intellectual property as training data for machines. But copyright also needs to protect the people who are being genuinely creative with those machines: not just making more works “in the style of,” but treating AI as a new artistic medium. The finely tuned balance that copyright needs to maintain has just become more difficult.

There may be solutions outside of the copyright system. Shutterstock, which previously announced that they were removing all AI-generated images from their catalog, has announced a collaboration with OpenAI that allow the creation of images using a model that has only been trained on images licensed to Shutterstock. Creators of the images used for training will receive a royalty based on images created by the model. Shutterstock hasn’t released any details about the compensation plan, and it’s easy to suspect that the actual payments will be similar to the royalties musicians get from streaming services: microcents per use. But their approach could work with the right compensation plan. Deviant Art has released DreamUp, a model based on Stable Diffusion that allows artists to specify whether models can be trained on their content, along with identifying all of its outputs as computer generated. Adobe has just announced their own set of guidelines for submitting generative art to their Adobe Stock collection, which requiring that AI-generated art be labeled as such, and that the (human) creators have obtained all the licenses that might be required for the work.

These solutions could be taken a step further. What if the models were trained on licenses, in addition to the original works themselves? It is easy to imagine an AI system that has been trained on the (many) Open Source and Creative Commons licenses. A user could specify what license terms were acceptable, and the system would generate appropriate output–including licenses and attributions, and taking care of compensation where necessary. We need to remember that few of the current generative AI tools that now exist can be used “for free.” They generate income, and that income can be used to compensate creators.

Ultimately we need both solutions: fixing copyright law to accommodate works used to train AI systems, and developing AI systems that respect the rights of the people who made the works on which their models were trained. One can’t happen without the other.

View Details

This month’s news has been overshadowed by the implosion of SBF’s TFX and the possible implosion of Elon Musk’s Twitter. All the noise doesn’t mean that important things aren’t happening. Many companies, organizations, and individuals are wrestling with the copyright implications of generative AI. Google is playing a long game: they believe that the goal isn’t to imitate art works, but to build better user interfaces for humans to collaborate with AI so they can create something new. Facebook’s AI for playing Diplomacy is an exciting new development. Diplomacy requires players to negotiate with other players, assess their mental state, and decide whether or not to honor their commitments. None of these are easy tasks for an AI. And IBM now has a 433 Qubit quantum chip–an important step towards making a useful quantum processor.

Artificial Intelligence* Facebook has developed an AI system that plays Diplomacy. Diplomacy is a board game that includes periods for non-binding negotiations between players, leading to collaborations and betrayals. It requires extensive use of natural language, in addition to the ability to understand and maintain relationships with other players. * Shutterstock will be collaborating with OpenAI to build a model based on DALL-E that has been trained only on art that Shutterstock has licensed. They will also put in place a plan for compensating artists whose work was used to train the model. * Facebook’s large language model for scientific research, Galactica, only survived online for three days. It produced scientific papers that sounded reasonable, but the content was often factually incorrect, including “fake research” attributed to real scientists. It was prone to generating hate research directed against almost any minority. * Google has put a Switch Transformers model on HuggingFace. This is a very large Mixture of Experts model (1.6 trillion parameters) that uses many sub-models, routing different tokens to different models. Despite the size, Switch Transformers are relatively fast and efficient. * OneAI has launched a Natural Language Processing-as-a-Service service, based on OpenAI’s Whisper model. Whisper is relatively small, impressively accurate, and supports multiple languages. * AI governance–including the ability to explain and audit results–is a necessity if AI is going to thrive in an era of declining public trust and increasing regulation. * Researches have developed an AI system that learns to identify objects by using a natural language interface to ask humans what they’re seeing. This could be a route towards AI that learns more effectively. * Google is developing a human-in-the-loop tool for their large language model LaMDA, designed to help writers interact with AI to create a story. The Wordcraft Writers Workshop is another project about collaborating with LaMDA. “Using LaMDA to write full stories is a dead end.” * You didn’t really want a never-ending AI-generated discussion between Werner Herzog and Slavoj Žižek, did you? Welcome to the Infinite Conversation. * Code as Policies extends AI code generation to robotics: it uses a large language model to generate Python code for robotic tasks from verbal descriptions. The result is a robot that can perform tasks that it has not been explicitly trained to do. Code is available on GitHub. * AskEdith is a natural language interface for databases that converts English into SQL. Copilot for DBAs. * Facebook has used AI to build an audio CODEC that is 10 times more efficient than MP3. * SetFit is a much smaller language model (1/1600th the size of GPT-3) that allows smaller organizations to build specialized natural language systems with minimal training data. * Wide transformer models with fewer attention layers may be able to reduce the size (and power requirements) of large language models while increasing their performance and interpretability. * Semi-supervised learning is a partially automated process for labeling large datasets. Starting with a small amount of hand-labeled data, you train a model to label data; use that model; check results for accuracy; and retrain.

Programming* DuckDB is a very fast database designed for online analytic processing (OLAP) of small to medium datasets. It runs easily on a laptop and integrates very well with Python. * How do you manage SBOM drift? Building a software bill of materials is one thing; keeping it accurate as a project goes through development and deployment is another. * Who is using Rust? Time for a study. Nearly 200 companies, including Microsoft and Amazon; Azure’s CTO strongly suggests that developers avoid C or C++ in favor of Rust. * What comes after Copilot? Github is looking at voice-to-code: programming without a keyboard. * genv is a tool for managing GPU use, an often neglected part of MLOps. Unlike CPUs, they are usually allocated statically, and can’t be reallocated if they’re underused or unused. * Multidomain service orchestration could be the next step beyond Kubernetes: orchestration between software components that are running in completely different environments. * Rewind, an unreleased product for Macs, claims to record everything you do, see, or hear, so you can look it up later. There are obvious ramifications for privacy and security, though users can start and stop recording. The key technology seems to be extremely effective compression. * Progressive delivery for databases? As James Governor points out, database schemas have been left behind by CI/CD. That may be changing. * Turbopack, a new Rust-based bundler for Next.js, promises greatly improved performance. Unlike Webpack, Turbopack does incremental builds, and is designed for use in both development and production. * Shell scripting never goes out of date. Here are some best practices, starting with “always use bash.”

Security* The US Department of Defense has released their road map towards implementing zero trust by 2027. * A new ransomware attack steals the victim’s Discord account in addition to encrypting files. It’s theorized that the Discord account may be used to launch cryptocurrency and NFT scams. In any case, it’s a sure sign of where cyber criminals see value: not in Facebook or Twitter. * 95% of all web applications have security holes. And that’s an improvement over last year. 77% had a vulnerability listed in OWASP’s top 10: misconfiguration, broken access control, and other basic stuff. The biggest problem in infosec is (still) getting the basics right. * The popularity of cryptojacking (mining cryptocurrency with malware planted in someone else’s applications) continues to rise, as the collapse in cryptocurrency prices makes legitimate mining unprofitable. * A threat group named Worok is using steganography to hide malware within PNG images. * All of the major browsers (Chrome, Firefox, Safari) trust certificates that allow a number of untrustworthy companies to act as certificate authorities. These companies are involved in activities like planting spyware on web sites to collect users’ personal data. * A massive SEO-poisoning campaign has compromised 15,000 WordPress sites, with the aim of causing Google searches to send people to fake Q&A sites. This may be a precursor to using the fake sites for phishing or installing malware. * The British Government has started a scan of all Internet devices located in the UK. Its intent is to detect vulnerabilities. * Cyberattacks are increasingly targeted at small to medium businesses, the vast majority of which don’t have plans for defense or disaster recovery. * Multifactor Fatigue is a new kind of attack against multifactor authentication: bombarding a user with automation requests, hoping that they will accidentally approve one.

Quantum Computing* Scott Aaronson has posted an “extremely compressed” (3-hour) version of his undergraduate course in Quantum Computing on YouTube. It’s an excellent way to get started. * Horizon Quantum Computing is launching a development platform that will let programmers write code in a language like C or C++, and then compile and optimize it for a quantum computer. * IBM has created a 433-qubit quantum chip, and updated the Qiskit runtime with improved error correction. This represents a big step forward, though we are still far from usable quantum computing.

Cryptocurrency and Blockchains* The Australian Stock Exchanged canceled its 6-year-old blockchain experiment, which would have put most of its work onto a Blockchain-like shared distributed ledger. * Vitalik Buterin responds to the FTX failure by hypothesizing about a “proof of solvency” that would be independent of audits and other “fiat” methods. The theme is familiar: can cryptocurrency move closer to trustlessness? * One “selling point” of NFTs has been that royalties can be passed to creators on resale of the NFT. However, many marketplaces do not enforce royalty payments, and building royalties into the smart contracts underlying NFTs is close to impossible. Some marketplaces, including Magic Eden and OpenSea, have developed tools for enforcing royalty payments. * Infrastructure for renewable energy is bound to be less centralized. Is it an application for a blockchain? Or is a blockchain just a tool for recentralization? Is it creepy when Shell is arguing for decentralization?

Metaverse* Can a nation upload itself to the metaverse? At the COP27 climate summit, Tuvalu’s foreign minister proposed, bitterly, that this may be their only solution to global warming, which will put their entire nation underwater. Their geography, culture, and national sovereignty could be preserved in a virtual world. * The Dark Forest is a massive multiplayer online game that is based on a blockchain. It is almost certainly the most complex game based on blockchain technology. There is no central server; it may show a way into building a Metaverse that is truly decentralized. * When is VR too connected to the real world? Palmer Lucky, founder of Oculus, has built a VR headset that will kill you if you die in the game. While he says this is just “office art,” he seems to believe that devices like this will eventually become real products. * The internet developed organically, in ways nobody could have predicted. Ben Evans argues that if the Metaverse happens, it will also develop organically. That isn’t an excuse not to experiment. But it is a reason not to invest too much in conflicting definitions.

Web* The flow of users from Twitter to Mastodon means that the ActivityPub protocol (the protocol behind Mastodon’s federated design) is worth understanding. Mastodon won’t (can’t) make the mistake of disenfranchising developers of new clients and other applications. * Google is imposing a penalty on AI-generated content in its rankings. While a reduction of 20% seems small, that penalty causes a significant reduction in traffic.

Things* Matter is a new standard for interoperability in the Internet of Things. Although there are many possible applications, the most important ones will probably be in energy management. * The winner of this year’s Natural Robotics Contest is Gillbert, an open source fish robot that has been developed to collect microplastic particles from lakes. It can remove particles as small as 2 millimeters, and can be built with a 3D printer. * Data from smartphones could provide better ways to monitor bridge safety. Sensors on phones in moving vehicles can detect vibrations that point to structural problems; they appear to do a better job than either visual inspector or sensors in fixed positions. * A robotic peregrine falcon has been invented to scare flocks of birds away from airports.

View Details

Major catastrophes can transform industries and cultures. The Johnstown Flood, the sinking of the Titanic, the explosion of the Hindenburg, the flawed response to Hurricane Katrina–each had a lasting impact. Even when catastrophes don’t kill large numbers of people, they often change how we think and behave. The financial collapse of 2008 led to tighter […]

View Details

If every company is a technology company, then every healthy company must have a healthy relationship to technology. However, we haven’t seen any discussions of “technical health,” which suggests that industry at large doesn’t know what differentiates a company that’s been through a successful digital transformation from one that’s struggling. To help us understand technological […]

View Details

This summer, we started asking about “technical health.” We don’t see a lot of people asking what it means to use technology in healthy ways, at least not in so many words. That’s understandable because “technical health” is so broad that it’s difficult to think about.  It’s easy to ask a question like “Are you […]

View Details

We’ve all been impressed by the generative art models: DALL-E, Imagen, Stable Diffusion, Midjourney, and now Facebook’s generative video model, Make-A-Video. They’re easy to use, and the results are impressive. They also raise some fascinating questions about programming languages. Prompt engineering, designing the prompts that drive these models, is likely to be a new specialty. […]

View Details

Maintaining a separate category for AI is getting difficult. We’re seeing important articles about AI infiltrating security, programming, and almost everything else; even biology. That sounds like a minor point, but it’s important: AI is eating the world. What does it mean when an AI system can reconstruct what somebody wants to say from their […]

View Details

A recently passed law in New York City requires audits for bias in AI-based hiring systems. And for good reason. AI systems fail frequently, and bias is often to blame. A recent sampling of headlines features sociological bias in generated images, a chatbot, and a virtual rapper. These examples of denigration and stereotyping are troubling […]

View Details

We want to congratulate Dylan Field on his startup Figma, which Adobe recently purchased for $20B. Dylan started his career with O’Reilly Media when he was in high school—not that long ago. With Figma, he’s made the big time. It’s worth thinking about why Figma has been so successful, and why Adobe was willing to […]

View Details

Gartner has anointed “Hyperautomation” one of the top 10 trends for 2022. Should it be? Is it a real trend, or just a collection of buzzwords? As a trend, it’s not performing well on Google; it shows little long-term growth, if any, and gets nowhere near as many searches as terms like “Observability” and “Generative […]

View Details

September was a busy month. In addition to continued fascination over art generation with DALL-E and friends, and the questions they pose for intellectual property, we see interesting things happening with machine learning for low-powered processors: using attention, mechanisms, along with a new microcontroller that can run for a week on a single AA battery. […]

View Details

Projects like OpenAI’s DALL-E and DeepMind’s Gato and LaMDA have stirred up many discussions of artificial general intelligence (AGI). These discussions tend not to go anywhere, largely because we don’t really know what intelligence is. We have some ideas–I’ve suggested that intelligence and consciousness are deeply connected to the ability to disobey, and others have […]

View Details

It’s hardly news to talk about the AI developments of the last month. DALL-E is increasingly popular, and being used in production. Google has built a robot that incorporates a large language model so that it can respond to verbal requests. And we’ve seen a plausible argument that natural language models can be made to […]

View Details

In a recent Radar piece, I explored N-sided marketplaces and the middlemen who bring disparate parties together. One such marketplace is the world of advertising, in which middlemen pair hopeful advertisers with consumer eyeballs. And this market for attention is absolutely huge, with global ad spend weighing in at $763 billion in 2021 revenues. Most […]

View Details

In a previous article, I wrote about how models like DALL-E and Imagen disassociate ideas from technique. In the past, if you had a good idea in any field, you could only realize that idea if you had the craftsmanship and technique to back it up. With DALL-E, that’s no longer true. You can say, […]

View Details

Humans are notoriously poor at judging distances. There’s a tendency to underestimate, whether it’s the distance along a straight road with a clear run to the horizon or the distance across a valley. When ascending toward a summit, estimation is further confounded by false summits. What you thought was your goal and end point turns […]

View Details

The metaphors we use to describe new technology constrain how we think about it, and, like an out-of-date map, often lead us astray. So it is with the metaverse. Some people seem to think of it as a kind of real estate, complete with land grabs and the attempt to bring traffic to whatever bit […]

View Details

The large model train keeps rolling on. This month, we’ve seen the release of Bloom, an open, large language model developed by the BigScience collaboration, the first public access to DALL-E (along with a guide to prompt engineering), a Copilot-like model for generating regular expressions from English-language prompts, and Simon Willison’s experiments using GPT-3 to […]

View Details

Data scientists working in Python or R typically acquire data by way of REST APIs. Both environments provide libraries that help you make HTTP calls to REST endpoints, then transform JSON responses into dataframes. But that’s never as simple as we’d like. When you’re reading a lot of data from a REST API, you need […]

View Details

There’s a puzzling disconnect in the many articles I read about DALL-E 2, Imagen, and the other increasingly powerful tools I see for generating images from textual descriptions. It’s common to read articles that talk about AI having creativity–but I don’t think that’s the case at all.  As with the discussion of sentience, authors are […]

View Details

This month, large models are even more in the news than last month: the open source Bloom model is almost finished, Google’s LaMDA is good enough that it can trick people into thinking it’s sentient, and DALL-E has gotten even better at drawing what you ask. The most important issue facing technology might now be […]

View Details

Last year, our report on cloud adoption concluded that adoption was proceeding rapidly; almost all organizations are using cloud services. Those findings confirmed the results we got in 2020: everything was “up and to the right.” That’s probably still true—but saying “everything is still up and to the right” would be neither interesting nor informative. So rather than […]

View Details

On June 6, Blake Lemoine, a Google engineer, was suspended by Google for disclosing a series of conversations he had with LaMDA, Google’s impressive large model, in violation of his NDA. Lemoine’s claim that LaMDA has achieved “sentience” was widely publicized–and criticized–by almost every AI expert. And it’s only two weeks after Nando deFreitas, tweeting […]

View Details

DeepMind’s new model, Gato, has sparked a debate on whether artificial general intelligence (AGI) is nearer–almost at hand–just a matter of scale.  Gato is a model that can solve multiple unrelated problems: it can play a large number of different games, label images, chat, operate a robot, and more.  Not so many years ago, one […]

View Details

The explosion of large models continues. Several developments are especially noteworthy. DeepMind’s Gato model is unique in that it’s a single model that’s trained for over 600 different tasks; whether or not it’s a step towards general intelligence (the ensuing debate may be more important than the model itself), it’s an impressive achievement. Google Brain’s Imagen […]

View Details

In the previous article, I explored the role of the middleman in a two-sided marketplace.  The term “middleman” has a stigma to it. Mostly because, when you sit between two parties that want to interact, it’s easy to get greedy. Greed will bring you profits in the short term. Probably in the long term, as […]