Three items in today's payload are about evals, which tells you something about where the industry's head is right now. Building the AI is the solved problem. Knowing whether it's working is apparently still a craft.
01
The right way to evaluate an AI agent isn't to check if it got the right answer
Madhu Guru posted part 7 of a running series on eval construction, and this installment is worth the whole thread. The argument: grading a financial analysis agent on whether it recommended the right stock is like grading a surgeon on whether the patient survived, while ignoring everything that happened in the operating room. You miss all the intermediate steps where things went wrong. Good evals measure the jobs-to-be-done along the way, not just the final output. ---
Why it matters: If your team built an AI workflow and validated it by checking a sample of outputs, you have a gut-feel product, not an engineered one. The next time the model updates or the data changes, you won't know what broke until a user tells you. Building evals at each step is more work upfront and the only way to maintain something in production.
There's a type of eval only humans can build, and Claude can't help you with it
Product builder Peter Yang flagged a distinction from researcher Shreya Rajpal: top-down evals (what you'd design from first principles, starting with the task description) versus bottom-up evals (what you discover by staring at actual outputs until your gut tells you something is off). Claude is apparently good at helping with the first kind. The second kind, the one that requires having seen hundreds of real failures and knowing what bad looks like, is entirely on you. ---
Why it matters: This connects directly to the story above. The evals that catch real-world failures tend to come from bottom-up experience, not from theorizing about what could go wrong. If you've been using AI to generate your test cases, you're probably missing your most important ones.
Vercel CEO Guillermo Rauch lays out how he wants agents to compose
Rauch posted the philosophy behind extending `fx`, Vercel's agent framework: open protocols (MCP, Skills, Plugins) plus a Unix-style approach where small programs do one thing and compose by calling each other. The goal is that developers should be able to build their own CLI, background agent, or software factory on top of it, locally or in the cloud. ---
Why it matters: The Unix comparison is doing a lot of work here. Unix composability worked because the interfaces were simple and stable. Agent composition has neither property yet. If Vercel can actually make this stick with a library (`libfx`) that enables embedding, it gives developers a way to build AI workflows that aren't locked to any single cloud provider's abstraction. That's a real differentiator if it holds up.
Following up on the multi-bug story from yesterday's digest: Thibault Sottiaux, who works on Codex at OpenAI, posted Sunday that the account reset has propagated and fixes are live. He says users should feel a positive difference, with more improvements coming and continued communication. ---
Why it matters: Worth watching whether this holds. Three distinct quota accounting bugs in one product suggests the underlying metering is still being figured out. If you were mid-workflow when things broke last week, now is a reasonable time to go back and check whether your limits are back to normal.
Ryo Lu posted a piece of prose poetry about departure, the process of emptying oneself before moving toward something unknown. No AI product news here, but it's a quietly beautiful thing to read on a Tuesday morning.