Saturday, October 17, 2026
The model is a commodity: the harness is the product.
October 17 · 5 videos
Vibes are not a test suite.
Laurie Voss says instrument your traces first.
Trust is the bottleneck in finance.
Eugene Cheah cut costs from $150k to $7k.
The model is frozen: the harness is where you win.
“Looks good to me is not a test suite.”
How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI
Laurie Voss · AI Engineer · 116 min
Watch on YouTube →Laurie Voss from Arize AI explains how to move AI agent development from vibes to engineering. He demonstrates using Arize Phoenix and Claude to build rigorous evaluation loops that replace intuition with empirical evidence.
- Split evaluators into specific dimensions like faithfulness and tone to improve debugging.
- Human experts only agree on complex evaluations 20 to 30 percent of the time.
- Start by instrumenting the agent and manually reading 12 traces to establish a baseline.
- Define success by interviewing non-technical stakeholders to turn vague disappointment into testable criteria.
- Evals allow you to calculate the ROI of model upgrades by comparing performance gains against cost increases.
- The workshop demonstrates a financial agent failing by reporting a 116 percent analyst consensus due to reasoning errors.
AI Safety Fact vs. Fiction | YC Paper Club
Clark Barrett · Y Combinator · 97 min
Watch on YouTube →Stanford professors Clark Barrett and Mykel Kochenderfer discuss shifting AI safety from existential hysteria to formal verification. They argue that AI can be used to mathematically prove code correctness and safety in critical systems.
- Commercial aviation safety standards aim for one catastrophic failure per billion flight hours.
- AI agents verified a 30,000 line Rust crypto library in 11 hours compared to 5 months for humans.
- Simple and interpretable methods are often more robust than complex neural networks for safety-critical tasks.
- Human-in-the-loop systems are vulnerable to a denial of service by AI through sheer volume of generated claims.
- An NTSB for AI model should be non-punitive to encourage honest reporting of system failures.
- Defense-first architectures should focus on independently checkable certificates rather than verbal trust.
Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Francesco Galletta · AI Engineer · 110 min
Watch on YouTube →Francesco Galletta and Felippe Felisola Caso from Monzo explain why trust is the primary bottleneck for AI in finance. They propose building structured harnesses that separate non-deterministic reasoning from deterministic code execution.
- Move calculations from the model's head into version-controlled and unit-tested code.
- The cost of a 1 percent error in finance can represent catastrophic losses.
- Use thin-slicing and gut feelings as verification tools to identify results that do not look right.
- AI impact must be measured by business value and outcome reliability rather than token speed.
- Systems must be designed for human oversight because AI agents face no consequences for errors.
- The 5 Principles of Trust framework includes Shared Domain, Consistency, Evidence, Accountability, and Verification.
The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Eugene Cheah · AI Engineer · 55 min
Watch on YouTube →Eugene Cheah of Featherless AI outlines strategies for reducing AI inference costs by orders of magnitude. He introduces the concept of Tokenomics to help organizations optimize spend through model right-sizing and granular measurement.
- A 2x increase in token generation speed typically results in a 3x increase in organizational cost.
- The slop grenade occurs when agents generate massive volumes of low-quality content for humans to filter.
- Use departmental model labels like Fast or Smart to maintain control over underlying provider switching.
- Organizations spending over $10,000 monthly should move from experimentation to disciplined tokenomics.
- Target costs can reach as low as $0.03 per million tokens using specialized open models and dedicated hardware.
- Incentivize cost-saving by making cheaper models faster because users are often too lazy to switch for price alone.
Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Nacho Martinez · AI Engineer · 51 min
Watch on YouTube →Nacho Martinez from Oracle argues that the agent harness is the primary area for performance gains as frontier models become standardized. He presents a memory-aware architecture for complex tasks like Anti-Money Laundering.
- The agent harness is where developers achieve significant performance gains while models remain standardized.
- Larger context windows often lead to degraded performance due to attention dilution or the lost in the middle phenomenon.
- Multi-agent write conflicts are a solved problem in database engineering from 30 years ago.
- Semantic caching can reuse 80 percent of context to avoid redundant computation costs.
- Agentic systems should flag suspicious activity for human review rather than blocking actions autonomously.
- A multi-tiered memory framework should mimic human cognitive structures including episodic and procedural memory.
References
PeopleLaurie Voss (@seldo) · Clark Barrett · Mykel Kochenderfer · Yann LeCun · Sam Altman · Jacob Coxen · Francesco Galletta · Felippe Felisola Caso · Eugene Cheah (https://x.com/picocreator) · Nacho Martinez · Andrew Ng · Richmond
ToolsArize Phoenix · Claude · CSlib · Lean · Featherless AI · Oracle · MI325 GPUs