# Halyard Labs — Full Content

> Sydney-based AI consultancy helping engineering, product and operations teams implement AI that works. This file contains the canonical content of the marketing site and all published writing in a single document, served at https://halyard.dev/llms-full.txt for AI agents and answer engines.

## About

Halyard Labs is led by Alex Hinds, the founding product and engineering lead at Lorikeet (pre-revenue to Series B) with prior leadership roles at Atlassian, The Guardian and Kayo. We work with engineering, product and operations teams of roughly 10-200 people, primarily in Sydney and across Australia.

## Services

### AI Readiness Audit (1 week, from $5,000 + GST)
A full audit of existing systems, tools and workflows to find where AI will save the most time and money. Output: a written report with a prioritised list of opportunities ranked by effort and impact, plus clear next steps.

### AI Tooling & Training (4-8 weeks, from $20,000 + GST)
We set up the right AI tooling for the team's highest-leverage problems, then train people to run with it. Includes scoping, audit, tool setup and configuration, guided rollout, training and handover.

### Embedded Technical Leadership (3-6 months, from $27,000/month + GST)
Fractional CTO / senior technical leadership, 2-3 days per week, embedded with the team. Includes strategy, architecture, delivery leadership, team capability building and a structured transition.

## Contact

- Website: https://halyard.dev
- Contact form: https://halyard.dev/contact
- Direct booking: https://calendar.notion.so/meet/alhinds/gr3d13sui

---

## Writing

### Context Engineering: The Skill That Separates Good AI Teams from Great Ones

URL: https://halyard.dev/blog/context-engineering-guide
Published: 2026-03-24
Author: Alex Hinds
Category: AI Engineering Fundamentals
Tags: context engineering, CLAUDE.md, prompt engineering, AI best practices, AI infrastructure

> Context engineering—not prompt engineering—determines AI system quality. A practical guide to CLAUDE.md, instruction budgets, and production patterns.

# Context Engineering: The Skill That Separates Good AI Teams from Great Ones

The difference between an AI system that works and one that fails in production often comes down to something unglamorous: how you organize information for the model to read.

This is context engineering—the deliberate discipline of structuring, prioritizing, and distributing information a model needs to perform reliably at scale. It's not prompt engineering (wording), but structural: deciding *what*, *how*, *when*, and *where* to present information so models can apply it across many tasks.

Alex Hinds has spent over a decade scaling AI systems from startups to enterprises, and has seen teams repeatedly stumble here: heavy investment in orchestration and infrastructure, but ad hoc context engineering. The result is predictable—brittle systems that work in demos but fail in production.

## Context vs. Prompt Engineering

Prompt engineering—writing better phrasing to a model—is tactical. Context engineering is structural: deciding what information matters, how to present it, and where to store it for reliable reuse across tasks.

The distinction reshapes the problem. A prompt engineer tweaks wording. A context engineer asks: Should this information be in the context window at all? Could we reference a file? Is this documentation stale and actively making performance worse?

With [Claude's 200,000-token context window](https://docs.anthropic.com), it's tempting to think context is unlimited. It isn't. Your actual usable context—what a model can reliably process and apply—is far smaller. Capacity doesn't translate to practical utility.

This is the first principle: **your context window is a budget, not a blank canvas.**

## The Context Budget: 150-200 Instructions Is Your Real Limit

Research shows performance degrades when you pack too much into context. Models get worse at following dense instructions—they hallucinate more, miss nuance, and optimize for surface patterns rather than actual tasks.

Think in terms of instructions, not tokens. A well-designed instruction might be 50-200 tokens. For a coding agent or complex reasoning task, you're working with a real budget of [150-200 high-quality instructions](https://www.aihero.dev) before saturating the model's ability to reliably apply them.

What belongs in this budget? System architecture, non-standard tooling, project conventions, and things that can't be inferred from code alone. What doesn't: stack traces, entire API documentation, or every edge case you've encountered.

This is where CLAUDE.md becomes essential—ruthlessly minimal documentation that lives in your repo.

## CLAUDE.md: Ruthlessly Minimal Configuration

CLAUDE.md is a single markdown file in your project root containing everything a model needs to navigate the codebase—not everything it could theoretically benefit from, but what it actually *needs*. It's a prose document, typically 200-500 lines, organized by intent.

A good CLAUDE.md includes:

- Two-paragraph project overview and rationale
- Directory structure (prose, not a tree)
- How to run tests, dev server, production builds
- Environment variables and their sources
- Architectural patterns or constraints
- Non-obvious conventions
- What you explicitly don't want changed

The discipline is in ruthless omission. Don't include: auto-generated API docs (link to them instead), stack traces or error logs, stale patterns (outdated documentation actively hurts), or historical one-offs.

When you're tempted to explain something, ask first: should the code be clearer instead? A good CLAUDE.md documents the invisible—context that genuinely can't be inferred. Consider pairing it with [ARCHITECTURE.md for design decisions, TESTING.md for test patterns](/blog/agent-skills-code-quality), and other files distributed by need rather than crammed into one bloated document.

## Progressive Disclosure: Context by Need

One of the highest-leverage decisions is distributing context according to need rather than stuffing everything into one file.

High-level info in CLAUDE.md. Architecture decisions in ARCHITECTURE.md. API documentation in its own system. Testing patterns in TESTING.md. This keeps files lean, enables on-demand loading, and makes maintenance tractable. When a testing pattern changes, you update one file.

This becomes invaluable with AI agents interacting with your codebase repeatedly. An agent that checks ARCHITECTURE.md for domain patterns is demonstrably more reliable than one working from sprawling context. Within the context of the [plan-execute-clear loop](/blog/plan-execute-clear-loop) where agents interact with codebases iteratively, structured, versioned context becomes part of your deployment pipeline.

Version your context like code. Changes go through PRs. You see in git history when documentation shifted. This is critical for production systems.

## What Belongs in Your Context

**Project description and rationale.** Not a sales pitch—why it exists and what problem it solves. "We chose SQLite because we're edge-deployed" shapes better decisions than knowing you use SQLite.

**Package manager and build system.** Explicitly. pnpm vs. npm, custom build scripts, constraints like "no native modules." Models should never guess.

**Non-standard commands.** If your test command is `pnpm test:integration`, document it. This is where CLAUDE.md pays for itself.

**Architectural conventions.** File structure, test colocation patterns, API route conventions, component composition rules. These are invisible until made explicit.

**What you don't want changed.** Complex legacy code, intentional patterns that look wrong, areas teams have decided not to refactor. Models optimize for surface improvements. Being explicit prevents failures.

**Dependency reasoning.** Not a list—context. Why three HTTP clients? Why two database layers? Models that understand reasoning make better decisions.

What doesn't belong: implementation details, full API signatures (link to docs), examples longer than explanations, or anything discoverable by reading the code. When testing patterns change, you might want to reference a [dedicated TDD with AI agents guide](/blog/tdd-with-ai-agents) or measure context quality with [AI evaluation frameworks](/blog/ai-evaluation-testing).

## Anti-Patterns That Kill Production Systems

**The bloated auto-generated dump.** A script ingests your entire codebase and pastes a summary into CLAUDE.md. Works once. By month three, it's stale and actively misleading. Models trust documentation; wrong documentation causes confident hallucinations.

**The everything file.** One CLAUDE.md containing architecture, API docs, testing, deployment, and naming philosophy. Impossible to maintain. Updates break assumptions elsewhere. No one knows what's current.

**Silence.** No CLAUDE.md at all. Fine for small projects. With multiple services, trade-offs, and constraints, models guess, and guesses in production are expensive.

**Stale technical decisions.** You documented three months ago. Major refactors shipped since then. Documentation doesn't reflect reality. You're actively misleading the model about your system.

**Context as sales pitch.** Polished, aspirational documentation that glosses over complications. A model with an honest picture of constraints can work around them. One working from aspirational docs will hit assumptions that don't hold.

## Context Engineering in Production Systems

This matters most when building production AI agents—systems that make decisions, interact with infrastructure, or process customer data autonomously. A coding assistant's mistakes are caught immediately (code doesn't compile). Production agents fail silently.

Context becomes something you test, version, and monitor like code. Changes go through PRs. Integration tests verify agents behave as documented. You measure performance shifts when context changes.

Without ruthlessly minimal context, you lose visibility into what's affecting agent behavior. With bloated documentation, you can't maintain it. Without versioning, you can't debug changes.

Production requires thinking of context as infrastructure. When you're working with reusable agent patterns across projects, [agent skills](/blog/agent-skills-code-quality) become a framework for documenting context consistently. Pair this with [measured evaluation of context quality](/blog/ai-evaluation-testing) to iterate reliably.

## Getting Started

Start simple. Create a CLAUDE.md in your project root. Keep it to one page: what the project does, how to run it, non-standard tooling, and architectural constraints.

Make it ruthlessly minimal. If a sentence doesn't convey something obvious from reading code or official docs, cut it.

Reference it explicitly when working with AI models. Iterate based on gaps you notice. Some things you documented won't matter. Others you'll realize should be clearer.

For multiple projects or codebases touched by multiple agents, layer in ARCHITECTURE.md, TESTING.md, or DEPLOYMENT.md as needed. Start lean.

The constraint is the feature. Limited context forces hard decisions about what's essential. Those decisions are where craft emerges. As [Simon Willison has written](https://simonwillison.net), context engineering is a discipline unto itself—not a byproduct of using models, but a deliberate practice that separates teams that iterate reliably from those that patch fires.

Reference frameworks like [OpenAI's prompt engineering guide](https://platform.openai.com/docs/guides/prompt-engineering) for broader context, but remember: the prompt is ephemeral. Your CLAUDE.md is version-controlled infrastructure. Treat it accordingly.

Ruthlessly minimal context. Honest documentation. Versioned like code. That's context engineering.

---

### The Plan-Execute-Clear Loop: How to Actually Use AI Coding Agents

URL: https://halyard.dev/blog/plan-execute-clear-loop
Published: 2026-03-20
Author: Alex Hinds
Category: AI Development Practices
Tags: AI coding, Claude Code, vibe coding, developer productivity, AI workflow

> Stop vibe coding. The plan-execute-clear loop is the workflow that separates teams shipping real AI-assisted code from those generating throwaway snippets.

You know the feeling. You ask an AI to build something, it works great, then you ask it to do something slightly different and suddenly the output falls apart. The model's gotten worse, right? No—you've lost your grip on what matters. Without structure, AI agents drift. They hallucinate requirements. They ignore constraints. They produce code that looks good in the moment but crumbles under the weight of real systems.

The difference between teams shipping production AI-assisted code and those generating throwaway snippets isn't intelligence or luck. It's workflow. Specifically, it's the plan-execute-clear loop—a discipline for breaking work into properly scoped tasks, executing them with full context, and clearing accumulated complexity before moving forward.

This isn't cargo cult methodology. It's how you translate vague human intent into machine-actionable specifications, then translate machine outputs into artifacts you can actually ship.

## The Plan Phase: Precision Beats Inspiration

Planning isn't about predicting the future. It's about making the agent's job mechanically simpler.

When you hand an AI a task, you're really handing it a context window and a goal. The quality of the output depends almost entirely on how precisely you've scoped the problem. A vague request like "improve our email system" produces vague, scattered output. The agent guesses at priorities, invents requirements, explores tangents. You end up with fifty half-baked ideas instead of one solid implementation.

The plan phase flips this. You write constraints. You define inputs and outputs explicitly. You break large problems into smaller tasks that each have exactly one clear exit condition.

Good planning looks like this: "Add a retry mechanism to the email queue. The function should accept a failed message ID and return true on success, false on failure. It should respect the existing backoff strategy defined in config/email.ts. Do not modify existing database schemas. The change must integrate with the test harness in __tests__/email.queue.test.ts." Notice the specificity. You've bounded the scope. You've named the integration points. You've said what not to do. The agent has a map.

This is where [context engineering](/blog/context-engineering-guide) becomes critical. You're not trying to be poetic. You're trying to be precise. The best plans include a one-sentence description of the why, then shift immediately to the what and how. You state your assumptions. You link to relevant code. You acknowledge constraints the agent might otherwise miss.

Plans also establish the standard for success before execution. What does done look like? Is it passing tests? Is it a working demo? Is it a full feature ready for production? The clearer you are, the less rework happens downstream.

## The Execute Phase: Context Does the Work

Once you've planned properly, execution becomes almost mechanical.

When you give an AI agent a well-scoped task with clear boundaries, the output quality jumps dramatically. The agent isn't scrambling to guess what you want. It's not exploring adjacent features that seemed related. It's executing a defined specification.

This is where tools like [Claude Code](https://docs.anthropic.com/en/docs/claude-code) shine. You can give the agent access to your codebase, environment, and test suite, and it can verify its own work. It can run tests. It can check for type errors. It can see whether the implementation actually solves the problem you scoped in the plan phase.

The execute phase also benefits from [TDD with AI agents](/blog/tdd-with-ai-agents). If you've written failing tests as part of your plan, the agent can see them immediately. It knows exactly what to build. It can iterate on its own implementation until the tests pass. You get verification without having to switch contexts repeatedly.

Good agents produce code that works because the task is clear. Bad agent output usually means the task wasn't clear enough—not that the agent failed at execution.

## The Clear Phase: Why Context Bloat Kills Quality

This is the phase most teams skip, and it costs them.

As you execute task after task, context accumulates. You've built feature A. You've fixed bugs in feature B. You've researched three different approaches to feature C. The agent's context window is now full of decision history, experimental code, discarded designs, and half-baked explorations. The next task you give it lands in this soup of noise.

What happens? The agent makes worse decisions because it's not sure what matters anymore. It might reference deprecated patterns. It might miss constraints it would have seen if the context were clean. It might invent solutions that conflict with earlier decisions because those decisions aren't clear anymore.

The clear phase is where you reset. You archive experimental branches. You update documentation with what you learned. You summarize key decisions in one place. You close the loop on context. Then you start fresh on the next task.

This doesn't mean deleting code. It means organizing it. It means being explicit about what's active and what's historical. It means ensuring the agent always starts a new task in a clean, organized context rather than a pile of accumulated experiments.

Think of it like cleaning your desk before starting a new project. The work happens faster because you're not constantly searching through clutter.

## A Practical Loop in Action

Imagine you're building an API. Your plan phase identifies four distinct tasks: add an authentication middleware, build the user endpoints, implement rate limiting, and write integration tests. Each is bounded. Each has clear inputs and outputs.

You start with task one: the middleware. You scope it precisely. "Create an authentication middleware that validates JWT tokens against the keys endpoint. Return 401 on invalid tokens. Pass valid claims to the request object. Use the existing middleware pattern in lib/middleware.ts." The agent builds it. Tests pass. You merge it.

Before task two, you clear. You update your project README with what you've built. You archive the branch. You close any notes about what didn't work. You start fresh on user endpoints with a clean context.

This approach—planning precisely, executing with full context access, clearing between tasks—produces code that actually ships. It's not flashy. It doesn't require sophisticated prompting tricks. It's just a structure that keeps both human and agent aligned on what matters.

## When to Break the Loop

That said, the loop isn't dogma. Some work requires exploration. Prototyping. Messy research. If you're evaluating whether a library will work, or exploring design patterns, or investigating whether something is even possible, the loop gets in the way.

In those moments, give yourself permission to enter exploration mode. Dump constraints. Accept vague goals. Iterate rapidly. But when exploration is done, return to the loop. Plan the implementation. Execute it cleanly. Clear before you move on.

The teams that ship reliably aren't the ones who never prototype. They're the ones who know when to explore and when to execute. They use the plan-execute-clear loop when precision matters. They break it when discovery does.

You can also think of reusable patterns—what we call [agent skills](/blog/agent-skills-code-quality)—as a way to compress planning. If you've planned and executed a pattern successfully before, you can describe it once, save it as a skill, then invoke it in future tasks. The plan phase becomes shorter because the pattern is proven.

For teams adopting this at scale, it's worth reading about how [AI works for technical leaders](/blog/ai-engineering-technical-leaders)—the loop scales differently depending on your team's structure. Matt Pocock has written about this concept as part of [AI Hero](https://www.aihero.dev), and his thinking on [subagents](https://x.com/mattpocockuk/status/1976313665407099187) extends the loop to multi-agent workflows where each agent owns its own plan-execute-clear cycle.

The mechanical truth is simple: the better your plan, the better your output. The cleaner your context, the more reliable your agent. And the tighter your loop, the faster you ship. Stop vibe coding. Start looping.

---

### Agent Skills: Encoding Engineering Excellence into Reusable AI Workflows

URL: https://halyard.dev/blog/agent-skills-code-quality
Published: 2026-03-18
Author: Alex Hinds
Category: AI Development Practices
Tags: agent skills, AI coding, code quality, Claude Code, engineering processes

> Agent skills turn ad-hoc AI prompting into repeatable engineering processes. Here's how to build skills that actually improve code quality.

When you ask an AI agent to do something once, you might get a passable result. Ask it to do the same thing ten times without a defined process, and you'll get ten different outputs of varying quality. This inconsistency is the core problem agent skills solve.

Agent skills aren't just prompts. They're concise, structured instruction sets that encode repeatable engineering processes into reusable workflows. The difference is subtle but crucial: a prompt tells an agent what to do right now. A skill tells an agent how to do something consistently, correctly, and at scale.

## The Consistency Problem

AI agents lack persistent memory. Each interaction starts fresh. Without explicit structure, you're asking an agent to improvise the same high-quality decision-making process repeatedly, which works against how these models operate best. They excel at following clear, defined procedures.

This is where [Matt Pocock's 5 Agent Skills concept](https://www.aihero.dev/5-agent-skills-i-use-every-day) becomes practical. Rather than treating AI interaction as ad-hoc prompting, you encode your best practices into discrete, chainable skills. You're essentially building muscle memory for your AI workflow.

## What a Skill Actually Is

A skill is a self-contained instruction set that guides an agent through a specific process. It typically includes context setup, procedural steps, expected outputs, and quality gates. The best skills are surprisingly concise—often just a few hundred words of carefully structured guidance. This precision matters more than length. A tight, well-defined skill beats a rambling 2,000-word prompt every time.

[Anthropic's Claude Code documentation](https://docs.anthropic.com/en/docs/claude-code) provides the technical foundation for skill implementation. But the real power emerges when you structure these skills to encode your domain's best practices.

## The Five Skills That Matter

In consulting, I've found five core skills form the backbone of quality AI engineering workflows:

**Grill-me** extracts requirements through rigorous questioning. Rather than accepting a vague feature request, this skill forces systematic discovery. It asks: What does success look like? What are the edge cases? Who are the users? What constraints exist? This skill prevents the most common failure mode—building the wrong thing very efficiently.

**Write-a-prd** takes answers from grill-me and structures them into a proper product requirements document. This isn't busywork. A solid PRD becomes the input for everything downstream. It clarifies scope, prevents scope creep, and gives the agent a reference point when decisions need making later.

**Prd-to-issues** converts the PRD into vertical task slices—what some call tracer bullets. This is critical: most teams break work into horizontal layers (database schema, API endpoints, UI components). Vertical slices, by contrast, cut through the entire stack for a single user capability. Each issue should be shippable and testable in isolation. This skill teaches the agent to think in deployable increments, not architectural layers.

**TDD** enforces red-green-refactor cycles. Rather than asking an agent to write code that happens to be correct, this skill requires test-first thinking. Write the test first, watch it fail, make it pass, refactor. This creates code that's provably correct and documented by tests. For details on TDD with AI agents, see [TDD with AI agents](/blog/tdd-with-ai-agents).

**Improve-codebase-architecture** schedules regular health checks. As codebases grow, entropy increases. This skill guides systematic review of code structure, dependency graphs, and architectural decisions. It prevents the slow accumulation of technical debt that eventually kills velocity.

## Why Shorter Often Beats Longer

I've seen teams spend weeks crafting the perfect 3,000-word mega-prompt, only to find it underperforms a crisp 400-word skill. Why? Precision of language matters more than comprehensiveness. A skill should be specific enough to eliminate ambiguity but flexible enough to apply across variations.

The best skills use this pattern: state the goal clearly, provide the decision framework, define the output format, include 2-3 examples, done. Anything beyond that often creates confusion rather than clarity. The agent gets lost in nuance when what it needs is simplicity.

## Chaining Skills Into Workflows

Individual skills are useful. Chained together, they become powerful. A typical workflow might flow from grill-me into write-a-prd into prd-to-issues, at which point you hand off to development skills. The [plan-execute-clear loop](/blog/plan-execute-clear-loop) describes how these skills fit into a complete AI engineering cycle. The loop reinforces iteration: plan with skills, execute against that plan, clear the context, and repeat.

This chaining is where skills truly shine. Each skill becomes a gate that forces clarity before moving to the next stage. You catch misaligned requirements at the PRD stage, not in code review. You discover missing acceptance criteria in the issues, not in production.

## Context Engineering and Skill Effectiveness

Skills work best when paired with excellent [context engineering](/blog/context-engineering-guide). A skill tells the agent how to proceed. Context tells it what domain knowledge, codebase patterns, and constraints exist. A well-crafted skill operating in a thin context will fail. The same skill operating in rich, structured context becomes powerful.

This means maintaining good context: code samples, architectural diagrams, decision logs, metrics. Not voluminous context—curated context. The skill uses that context as input, and output quality depends heavily on input quality.

## The Garbage In, Garbage Out Reality

Here's the uncomfortable truth: your codebase quality is the input signal. If you're starting from messy, inconsistent code, even the best skill won't fix that instantly. Agent skills improve the process going forward, but they don't retroactively repair legacy systems.

That said, [improve-codebase-architecture](/blog/ai-evaluation-testing) can systematically raise your baseline. Run it regularly, implement suggested changes, and over time your codebase becomes a better input signal for future work.

## Measuring Skill Effectiveness

How do you know if a skill actually improves code quality? This is where [AI evaluation](/blog/ai-evaluation-testing) becomes critical. Define metrics before deploying a skill: deployment frequency, test coverage, bug escape rates, code review comment density. Run the skill on a few tasks, measure the results, compare against baseline. Good skills show measurable improvement in these areas.

The reality is that most one-off AI interactions produce varying quality. Skills produce consistent quality. That consistency is worth documenting and protecting.

## Building Your First Skills

Start with the highest-leverage skill for your context. If requirements are consistently misunderstood, build grill-me first. If code quality is the constraint, start with TDD. Build one skill, use it on real work, refine based on results, then add the next.

The skills work best when they reflect your team's actual practices. If your best engineers follow a specific thinking pattern, encode that into a skill. You're not imposing process from above—you're capturing excellence and making it repeatable.

Agent skills turn AI from a question-answering tool into an engineering partner. They don't replace human judgment, but they do eliminate the inconsistency that comes from relying on humans to manually maintain discipline across dozens of AI interactions. That's where the real value lies.

## Further Reading

Explore the foundational work: [Matt Pocock's skills repository](https://github.com/mattpocock/skills) contains concrete implementations of many of these concepts. The code-first examples there translate directly into your own workflow design.

---

### AI Engineering for Technical Leaders: What Actually Matters in 2026

URL: https://halyard.dev/blog/ai-engineering-technical-leaders
Published: 2026-03-16
Author: Alex Hinds
Category: AI Engineering Fundamentals
Tags: AI strategy, technical leadership, CTO, AI implementation, engineering management

> Cut through AI hype. What CTOs and engineering leaders need to know about AI implementation, team structure, and infrastructure decisions.

# AI Engineering for Technical Leaders: What Actually Matters in 2026

Most organizations that started their AI journey 18 to 24 months ago are stuck in proof-of-concept limbo. They've built demos that work. They've hired someone with "AI" in their title. They still can't articulate what winning with AI looks like for their business.

This isn't a technology problem. Claude and GPT-4 are genuinely capable. The problem is the gap between "I can call an API to generate text" and "we have a production AI system that reliably solves a business problem." That gap is almost entirely engineering.

Organizations fail at AI adoption because they don't have the discipline to build reliable systems with probabilistic components. They lack evaluation infrastructure. They can't observe what's happening inside the system. They have no feedback loops. No way to iterate when things break. This requires treating AI as an engineering problem, not a technology novelty.

## The Build vs. Buy Decision Framework

One of the most consequential decisions you'll make is which AI capabilities you build versus buy. Most leaders optimize for the wrong dimension.

The instinct is to buy. Third-party solutions are faster and someone else owns reliability. But you accept their evaluation criteria, their fine-tuning decisions, their constraints.

The opposite mistake is building too much. Building your own foundation model is nearly always wrong unless you're operating at serious scale. Building a chatbot when Claude via API works is throwing away capital.

The real framework: ask whether the capability is differentiating. If your competitive advantage comes from how you apply AI to a problem unique to your domain or data, you may need to build. If the capability is table-stakes—something every player in your market does—buying is faster and smarter.

Then ask whether you have the people. Building production AI systems requires different engineering than most teams practice. If you don't have that expertise and the capability is worth building, hiring is a multi-month commitment.

Most failures happen because teams underestimated the building part. They thought they were buying a model. They didn't account for the infrastructure, evals, integration complexity, feedback loops, and retraining process. That's where the engineering work actually lives.

## What Production AI Infrastructure Actually Requires

This separates teams that talk about AI from teams that ship with AI.

You need evaluation systems. Most organizations skip this or do it at a hobby level. You should run your AI system against test cases and get a numerical score. This requires defining success clearly—not "this looks good" but measurable criteria. Build test datasets covering edge cases and failure modes. Run evals before deploying new prompts or model versions. This is a blocking gate for serious teams.

You need observability. See what's happening inside the black box: what prompts are generated, what context is retrieved, what the model outputs, how often it fails. This is harder than observability in deterministic software, but non-negotiable for production.

See the [AI evaluation and testing guide](/blog/ai-evaluation-testing) for concrete implementation details.

You need feedback loops that route production issues back into your training and evaluation process. When your AI system fails in production, it should generate a signal that helps you improve. Teams that automate this scale faster than teams waiting for someone to notice problems.

You need a strong foundation in context engineering. Most value lives not in the model itself but in how carefully you control what context it sees. See the [context engineering guide](/blog/context-engineering-guide) for detailed patterns.

## Team Structure: Who You Actually Need

Most CTOs wonder if they need to hire ML engineers. The answer is probably "fewer than you think."

If you're using Claude, GPT-4, or specialized LLM vendors, you're not training models. You're engineering systems with models as components. You don't need deep learning experts. You need software engineers who can build reliable systems with probabilistic components—engineers with testing discipline, observability instincts, strong incident response habits.

The stronger play is usually upskilling. Senior engineers with strong fundamentals can learn the AI piece. The fundamentals are the expensive part to teach.

You probably need specialized talent in one case: if you're building something requiring custom model work or specialized ML infrastructure. Otherwise, invest in teaching existing teams to think systematically about AI.

## Concrete First Steps

Start with something your team is frustrated with—not a chatbot. Pick something currently boring, manual, and repetitive. Code review boilerplate. Documentation writing. Logs analysis. Something where 60% accuracy is useful and being wrong doesn't break anything.

Build it using an existing LLM API. Build evaluation and feedback loops from the start, not as an afterthought. Measure actual time saved and reliability.

If it works, you have a success case, a team practiced at building with AI, and reusable infrastructure. If not, you learned fast without burning a year.

Then expand. Start small enough to learn on but real enough to prove value. See [building AI products](/blog/ai-product-validation) for validation frameworks and [scaling AI products](/blog/scaling-ai-products) for infrastructure patterns.

## The Competitive Advantage

The model is commoditized. Differentiation lives in three things: how quickly you integrate AI into your systems, how reliably it performs in your specific context, and how fast you iterate when it breaks.

That's all engineering. It's boring, hard, and doesn't make headlines. It's also where value gets built.

Your job as a technical leader is to navigate hype, pick the right things to build, and create the infrastructure and culture that lets your team move fast with these capabilities. You've done this with other technologies. AI just means doing it more systematically, earlier, and with higher stakes.

Organizations understanding this are already pulling ahead. See the [a16z AI infrastructure market map](https://a16z.com/emerging-architectures-for-llm-applications/) for landscape context and [Anthropic's enterprise documentation](https://docs.anthropic.com) for Claude implementation details. The [Latent Space podcast](https://www.latent.space) covers leading AI engineering practices.

The ones that don't are still stuck wondering why AI pilots aren't turning into businesses.

---

### TDD with AI Agents: Why Red-Green-Refactor Still Matters

URL: https://halyard.dev/blog/test-driven-development-ai-agents
Published: 2026-03-14
Author: Alex Hinds
Category: AI Development Practices
Tags: TDD, AI coding, code quality, testing, red-green-refactor

> Test-driven development makes AI-generated code dramatically better. Here's how to apply red-green-refactor cycles when working with coding agents.

When working with AI code generation, I noticed something troubling: developers write all tests upfront, the AI implements everything at once, tests pass immediately. No friction. No real constraints. Without a failing test to guide the AI, it has too much freedom—it can take shortcuts, make assumptions, optimize for passing mocks instead of solving the real problem.

This isn't how TDD works. [Kent Beck's red-green-refactor cycle](https://www.amazon.com/Test-Driven-Development-Kent-Beck/dp/0321146530) is still the most powerful approach for AI-assisted development. The difference is understanding how to apply it when your collaborator is a language model.

## The Horizontal Slicing Problem

Most developers default to what I call "horizontal slicing" with AI: write all tests, generate implementation, done. The tests look like this:

```javascript
test('processRefund should succeed for valid request', () => {
  const mockService = { reverseCharge: jest.fn() };
  const processor = new RefundProcessor(mockService);
  const result = processor.processRefund({ orderId: '123', amount: 50 });
  expect(result.success).toBe(true);
});

test('processRefund should fail if payment service throws', () => {
  const mockService = {
    reverseCharge: jest.fn().mockRejectedValue(new Error('failed'))
  };
  const processor = new RefundProcessor(mockService);
  const result = processor.processRefund({ orderId: '123', amount: 50 });
  expect(result.success).toBe(false);
  expect(result.error).toBe('failed');
});
```

Then you ask Claude to implement `RefundProcessor` and make all tests pass. The AI does it in one go. But what you've created is weak. Without real constraint during implementation, the AI can satisfy mocks without actually implementing behavior. You're not testing behavior; you're testing whether the AI matched your implementation expectations.

## Vertical Slicing: One Test, One Implementation

The solution is vertical slicing. Write one test. Watch it fail. Show that failing test to the AI and ask it to make it pass. Review the code. Move to the next test.

This works because a failing test is the clearest specification you can give an AI. It's not ambiguous. It's measurable. The AI either satisfies it or doesn't.

Here's the workflow:

1. Write a single failing test
2. Show it to Claude with: "Make this test pass, implement only what's necessary"
3. Review the minimal implementation
4. Refactor together if needed
5. Write the next test

Each iteration is focused. Each implementation is constrained. You get continuous feedback rather than one large implementation pass.

## Strong Tests vs Implementation Details

Not all tests are equal. A weak test verifies that `processRefund()` calls `mockService.reverseCharge()` with specific arguments. A strong test verifies that `processRefund()` with valid input succeeds and with invalid input fails with the right error.

Weak tests couple your tests to implementation. Strong tests verify the actual contract your code provides.

This matters more with AI because language models naturally pattern-match against common testing structures. You must be explicit: tests verify behavior through public interfaces, not internal implementation. [Martin Fowler's testing guide](https://martinfowler.com/testing/) covers this principle thoroughly.

The difference in practice:

```javascript
// Weak: Testing implementation details
expect(mockService.reverseCharge).toHaveBeenCalledWith('123', 50);

// Strong: Testing actual behavior
expect(result.success).toBe(true);
expect(result.orderId).toBe('123');
```

Strong tests let you refactor internals without breaking tests. With AI, this becomes your safety net as the codebase evolves.

## A Concrete TDD Cycle

Let me walk through an actual workflow. I'm building a refund processor. First, I define the interface: a `RefundProcessor` class with a `processRefund(request)` method that takes injected dependencies.

Test one:

```javascript
test('processRefund returns success for valid request', () => {
  const mockService = { reverseCharge: jest.fn().mockResolvedValue({ id: 'charge-123' }) };
  const processor = new RefundProcessor(mockService);

  const result = processor.processRefund({ orderId: '123', amount: 50 });

  expect(result.success).toBe(true);
  expect(result.transactionId).toBe('charge-123');
});
```

I show this failing test to Claude. Claude implements:

```javascript
class RefundProcessor {
  constructor(paymentService) {
    this.paymentService = paymentService;
  }

  async processRefund(request) {
    const result = await this.paymentService.reverseCharge(request.orderId, request.amount);
    return { success: true, transactionId: result.id };
  }
}
```

Minimal. Specific. Test passes. Next test:

```javascript
test('processRefund returns error when payment fails', () => {
  const mockService = {
    reverseCharge: jest.fn().mockRejectedValue(new Error('Card declined'))
  };
  const processor = new RefundProcessor(mockService);

  const result = processor.processRefund({ orderId: '123', amount: 50 });

  expect(result.success).toBe(false);
  expect(result.error).toBe('Card declined');
});
```

Claude adds error handling. Test passes. Continue iteratively. Each test adds one behavior. Each implementation is minimal and focused. By completion, you have well-tested code that evolved through explicit requirements.

This approach works because each failing test is a concrete specification. Claude isn't guessing; it's satisfying clear constraints.

## Fitting TDD Into Your Workflow

TDD with AI is a reusable [agent skill](/blog/agent-skills-code-quality). You can extract the pattern and apply it across projects. It fits naturally into the [plan-execute-clear loop](/blog/plan-execute-clear-loop)—plan interfaces upfront, execute tests one at a time with AI, clear assumptions as you go.

You can also think of TDD as [AI evaluation](/blog/ai-evaluation-testing). Each failing test is an evaluation criterion. The code either passes or fails. This makes TDD more rigorous than free-form code review.

When documenting patterns in CLAUDE.md files for team AI workflows, use [context engineering](/blog/context-engineering-guide) to specify test-first discipline explicitly. Show examples. Make it clear that vertical slicing is expected.

## When TDD Doesn't Apply

TDD isn't always the right tool. During exploration and prototyping, you might need loose constraints. When learning a new domain, horizontal slicing lets you iterate faster. TDD becomes essential once you move from "what should this do?" to "this must work reliably."

Know the difference. Use TDD for production code. Use exploration for learning.

## Conclusion

The failing test is the most powerful constraint you can give an AI. Not requirements documents. Not architectural diagrams. Not vague descriptions. A test that fails, clearly, unambiguously, until the code is correct.

Red-green-refactor still matters. It matters more with AI because it provides the clarity language models need to generate trustworthy code.

Write one test. Watch it fail. Ask Claude to make it pass. Review. Refactor. Repeat. This is how you build code you can trust.

**About the author:** Alex Hinds builds AI-assisted development workflows and engineering practices across teams.

---

### AI Product Validation: Why Most AI Launches Fail and How to Avoid It

URL: https://halyard.dev/blog/ai-product-validation
Published: 2026-03-12
Author: Alex Hinds
Category: AI Product Development
Tags: AI products, product validation, startup, product-market fit, AI strategy

> Most AI products fail because teams build technology before validating the problem. Here's the validation framework that took us from pre-revenue to Series B.

The graveyard of AI products is littered with impressive technology that nobody wanted. You've probably seen them: startups that spent eighteen months building sophisticated language model pipelines, fine-tuned on proprietary datasets, deployed across multiple cloud providers—only to launch to crickets. The pattern is so common it's become a cliché, yet teams keep repeating it.

The fundamental mistake is the same across nearly all of them: they built the technology before validating that the problem was real or that their solution actually solved it. In traditional SaaS, this is already a problem. In AI, it's catastrophic, because AI projects can consume enormous resources while remaining fundamentally misaligned with what users actually need.

Most founders I talk to understand product-market fit in theory. They've read Lenny's advice on finding PMF, they know [Y Combinator's startup principles](https://www.ycombinator.com/library) emphasize talking to users. But AI feels different. When your entire value proposition is "we have access to advanced models," the temptation is to optimize for model performance rather than user outcomes. This creates a tragic misalignment: your team celebrates a 2-point improvement in test accuracy while your product sits unused.

## The Validation Framework: Three Stages Before You Ship

At Lorikeet, we learned to compress validation into three discrete stages, each one more expensive than the last but requiring proof of concept before advancing. This framework kept us from building the wrong thing at scale.

**Stage 1: Manual, Human-Driven Solution**

Start by doing the work yourself. If you're building an AI system to route customer support tickets, manually route tickets for your early customers. If you're building a content classification system, classify content by hand. This is not a placeholder—it's your control group. You learn whether the problem is actually worth solving by understanding the true cost and complexity of the manual process.

At Lorikeet, we spent three weeks routing tickets manually for a pilot customer. That week taught us more about the actual problem than any requirements document could have. We discovered that ticket complexity varied wildly, that our initial assumptions about routing categories were wrong, and that speed was less important than accuracy because misrouted tickets created more work downstream.

Crucially, this stage validates user need without any AI. If users won't adopt your manual process, they won't adopt your AI version of it either.

**Stage 2: Human-in-the-Loop with AI Assistance**

Once you've confirmed people care about the problem, introduce AI gradually. Your system makes suggestions; humans make decisions. This is where you collect the real data that matters: whether your model actually improves human decision-making in the actual workflow.

For ticket routing, we built a simple Claude integration that suggested a routing category for each incoming ticket. Our human operators accepted or rejected the suggestion. This gave us three critical pieces of information: how often was the model right, how much time did the suggestion save the human, and which types of tickets was the model particularly bad at.

This stage also reveals the gap between offline accuracy and real-world utility. A model that's 92% accurate in your test set might only be correct 65% of the time on genuinely novel edge cases in production. Human-in-the-loop lets you see this gap before you ship.

**Stage 3: Fully Automated (If It Reaches Threshold)**

Only advance to full automation when the human-in-the-loop data proves the AI is genuinely improving outcomes. And "improving outcomes" doesn't mean better benchmarks—it means faster resolution time, fewer escalations, lower cost per transaction, or whatever metric actually matters to your user.

## The Evaluation Trap: Why Your Metrics Lie

Here's where most AI product teams go wrong after validation starts: they optimize for offline metrics instead of online behavior.

You build a classifier and achieve 89% accuracy on your validation set. Fantastic. You ship it. Nobody uses it. What happened?

Offline accuracy measures how often your model produces the "correct" answer according to your test data. Online metrics measure what users actually do. A support ticket routing system with 78% accuracy that saves routing time but requires human verification might drive adoption. A content classifier with 95% accuracy that catches edge cases only a domain expert would catch might languish because the false positives frustrate users faster than the true positives help them.

User behavior is the ultimate ground truth. If your AI system doesn't change user behavior in the direction you wanted, it doesn't matter how well it performs on your test set. At Lorikeet, we learned this distinction the hard way. Our response generation model achieved 85% accuracy on our generated evaluation set—a respectable score. But when we put it in front of users without human-in-the-loop, adoption was mediocre because edge cases created work rather than reducing it.

The solution is to define online success metrics before you ship and monitor them obsessively. Time to resolution. Escalation rate. User satisfaction. Session length. Whatever tells you whether your AI is actually improving the user experience.

## The "Good Enough" Threshold

Here's the uncomfortable truth about AI products: perfect accuracy is often worse than no product at all. The question isn't "How accurate can we make this?" It's "How accurate does this need to be relative to the cost of being wrong?"

For customer support routing at 78% accuracy, the cost of being wrong is moderately high—a customer waits an extra hour and a support agent spends time rerouting. That's recoverable. Users accept it. The time saved by automation outweighs the friction.

For response generation at 85% accuracy, the cost of being wrong is higher—you ship a response that might confuse or frustrate the customer, and the agent has to fix it. That requires more confidence before users trust it. But at 85%, enough of the hard work is automated that agents move faster. The threshold is crossed.

For critical applications like medical diagnosis, the threshold is much higher. For creative writing assistance, much lower.

This calculation changes everything. You're not trying to build the most sophisticated model. You're trying to find the minimum viable accuracy that makes the workflow better for your user. Once you hit that threshold, ship. Scale. Iterate based on real usage. [When you're ready to scale](/blog/scaling-ai-products), you'll face different problems that offline optimization can't predict.

## Validation Is Your Moat

Teams that validate before building scale faster than teams that optimize pure model performance. They ship products people want. They compound learning across users instead of spinning on incremental benchmark improvements. And they stay lean long enough to find product-market fit without burning capital on the wrong solution.

The most successful AI products I've seen shared this pattern: minimal viable technology, obsessive focus on user outcomes, and a clear threshold for when the AI was "good enough" to deploy. The technology improved later, but only after validating that the problem was real and the solution mattered.

[When you're ready to dig deeper into evaluation](/blog/ai-evaluation-testing), you'll need frameworks for measuring what actually matters. And when thinking about how context affects product quality, our [context engineering guide](/blog/context-engineering-guide) breaks down the levers you control. But first: validate that someone wants the answer to the question you're about to spend months optimizing.

---

### Scaling AI Products: What Breaks When You Go from 10 Users to 10,000

URL: https://halyard.dev/blog/scaling-ai-products
Published: 2026-03-10
Author: Alex Hinds
Category: AI Product Development
Tags: scaling AI, AI infrastructure, observability, AI costs, production AI

> Latency, cost, observability, and trust all break differently as AI products scale. Lessons from scaling an AI platform from pre-revenue to Series B.

You've validated product-market fit. Your AI product works for your early users. The contact form is lighting up. Now comes the part nobody warns you about: the infrastructure crumbles the moment you add a zero to your user count.

Before you scale, make sure you've validated—I wrote about [AI product validation](/blog/ai-product-validation) because the biggest waste is scaling something nobody wants. But if you've cleared that bar, the next phase reveals four breaking points that look minor until they're not.

## Latency Breaks First

A 2-second response time feels fine with 50 concurrent users. With 5,000, it's a crisis.

This happens in layers. Your initial setup probably calls the API, waits for a response, and sends it back. Each user adding 200ms of network latency is tolerable. But at scale, you're queuing. The API server gets 100 requests at once. Half of them wait for compute resources. Now that 2-second response is 12 seconds, and users are leaving.

The fix isn't obvious because latency improvements compound. You need to reduce both the AI model latency and your application latency. Smaller models respond faster—calling a 50B-token model when a 8B model works is throwing away seconds per user. But you also need to batch requests, cache aggressively, and design your UX to not depend on instant responses. Can users see incremental results? Can you give them something to do while the AI processes? That's not a band-aid; it's how production AI products work.

## Cost Becomes the Business Model

At 10 users, API costs are noise. At 10,000, they're 60% of your revenue or more.

Here's where most teams get surprised: if you didn't measure cost per transaction from day one, you now have no visibility into what's actually profitable. You're serving customers who may be costing you more than they pay you. If your LLM calls are unoptimized, every feature improvement is a cost improvement problem. Shorter prompts, reusing context, filtering irrelevant data before sending to the API—this is no longer optimization theater. It's survival.

Review [Anthropic's API pricing](https://www.anthropic.com/pricing) and the cost structure of your chosen model family. Understand what you're paying for: are you paying for input tokens, output tokens, or both? Can you batch requests? Will switching to a cheaper model break your product quality? These aren't questions for the CFO. They're questions for the product team during sprint planning.

The uncomfortable truth: the cheapest infrastructure decision you can make is building with cost-awareness from the start. Every AI product at scale uses a cost optimization framework, whether they admit it or not.

## Observability Collapses Into Darkness

With 10 users, you can read logs. With 10,000, you're drowning.

AI products are particularly opaque. Your model might start hallucinating subtly—correct 95% of the time instead of 98%, but you won't notice without proper instrumentation. You need to log what went into the model, what came out, whether the user found it useful, and which requests were anomalous. Without this, you're flying blind.

This is where [AI evaluation and testing](/blog/ai-evaluation-testing) becomes infrastructure. You need automated tests that catch degradation, dashboards that surface cost anomalies, and alerting on latency percentiles—not averages. Tools like [Datadog's AI observability](https://www.datadoghq.com/product/llm-observability/) exist precisely because this problem is unsolved at most companies.

The deeper issue: you can't debug what you can't measure. Early-stage AI products often skip this entirely, then panic when something breaks in production affecting thousands of users.

## Trust Erodes Under Scrutiny

At 10 users, one hallucination is a funny story. At 10,000, it's a support ticket that cascades.

Users at scale have higher expectations and lower tolerance for failure. They're integrating your AI into workflows that matter—their business, their customers, their time. When the AI says something confidently wrong, the damage compounds. One user loses trust in your system. They tell five others. Your NPS collapses.

Building trust at scale means being honest about what the AI can and can't do. It means gracefully degrading when you're uncertain. It means building human-in-the-loop workflows where users verify critical outputs, and the system learns from those verifications. You can't fix trust with better prompts alone. You fix it with product design: always show the source, let users correct the AI, make reversals easy, and never pretend certainty where there isn't any.

## The Human Question: Where Do Humans Belong?

As you scale, you have to decide: where does human judgment remain, and where does the AI run alone?

Early products often default to "humans verify everything," which doesn't scale. Later products sometimes default to "full automation," which breaks trust. The answer is usually: humans verify what matters most. You need a tiered system. High-stakes outputs get human review. Medium-stakes outputs get automated checks plus sampling for auditing. Low-stakes outputs run fully automated. This requires clear definitions of what "matters," which only your product can determine.

## Series B-Ready Infrastructure

By the time you're fundraising for Series B, investors expect you to have built infrastructure, not just features. This means:

- A cost model that ties feature usage to API spend
- Monitoring and alerting on latency, error rates, and quality metrics
- A strategy for which requests use which models (don't call GPT-4 for everything)
- Caching and batching where architecturally sound
- A framework for human-in-the-loop workflows on critical paths
- Decisions on where to use [context engineering](/blog/context-engineering-guide) vs. fine-tuning vs. retrieval

If you're using vector databases or [RAG systems](/blog/rag-systems-best-practices), they need monitoring too. Retrieval quality degrades subtly. You need to know when it does.

## The Honest Advice: Invest Earlier Than You Think

Every founder scaling an AI product learns this the hard way: infrastructure decisions made at 100 users become architecture problems at 10,000. The time to build observability is before you need it. The time to think about cost per transaction is when you're designing the feature. The time to define human-in-the-loop workflows is before they're required by your customers.

This doesn't mean perfection. It means being intentional. It means measuring things you think don't matter yet. It means treating infrastructure investment as part of product development, not as a separate tax you pay after you've grown.

The products that scale cleanly aren't the ones with the most features. They're the ones built by teams that understood, early, that scale breaks different things. And they prepared.

---

**About the Author:** Alex Hinds builds AI products and infrastructure at Halyard Labs, where he leads technical strategy on scaling AI platforms from pre-revenue to institutional adoption.

---

### RAG Systems in Production: What Most Teams Get Wrong

URL: https://halyard.dev/blog/rag-systems-best-practices
Published: 2026-03-08
Author: Alex Hinds
Category: AI Engineering Fundamentals
Tags: RAG, retrieval augmented generation, AI architecture, vector search, production AI

> Most RAG implementations fail silently. Here are the retrieval, chunking, and architecture mistakes teams make and how to fix them.

# RAG Systems in Production: What Most Teams Get Wrong

Most RAG failures aren't dramatic. They're slow burns. Retrieval degrades. Hallucinations increase. Users complain. Teams investigate and discover the system was never retrieving the right documents in the first place.

The core problem is straightforward: RAG is fundamentally limited by retrieval quality. A better generation model cannot fix poor retrieval. Yet teams obsess over which LLM to use while treating retrieval as solved. It's not. This guide covers what actually works in production.

## The Retrieval Quality Problem

Teams typically fail at RAG by ignoring retrieval quality as a measurement problem. They don't know if their system is actually retrieving relevant documents because they never built an evaluation framework.

Build one early. Start with 20-50 representative queries paired with documents that should be retrieved. Measure precision at k (what percentage of top-k results are relevant), mean reciprocal rank, and end-to-end generation quality. See [AI evaluation and testing](/blog/ai-evaluation-testing) for how to set this up rigorously.

Once you measure, you'll discover the real problem: most teams use generic embedding models selected from MTEB leaderboards. These rank well on general-purpose benchmarks but perform poorly on your specific data. Test embedding models against your evaluation set. A smaller, domain-specific model almost always outperforms a larger general-purpose one, with the bonus of being faster and cheaper.

## Chunking Strategies That Actually Work

Fixed-size chunks (512 or 1024 tokens) are convenient but destructive. They cut sentences mid-concept, fragment related information, and force arbitrary boundaries that don't align with your data structure.

Three better approaches:

**Semantic chunking**: Measure the distance between consecutive sentences using embeddings. Split when distance exceeds a threshold. This keeps related information together but adds preprocessing cost.

**Structure-aware chunking**: Align boundaries with your domain. Code repositories chunk by function. Medical records by encounter. Customer support by issue thread. Technical documentation by section and code block.

**Hierarchical chunking**: Create chunks at multiple abstraction levels. Short chunks for fine-grained retrieval, longer summary chunks for context. This helps the retriever find specific answers while understanding broader scope.

Include metadata in every chunk—source, document type, date, author, category. Make this metadata searchable so you can filter results by context. See [Pinecone's chunking guide](https://www.pinecone.io/learn/chunking-strategies/) for implementation patterns.

## Hybrid Search: When Vector-Only Fails

Semantic search alone misses domain-specific terminology, proper nouns, and exact matches. A query for "error code ORA-12514" won't retrieve well using embeddings alone.

Hybrid search combines vector search (for semantic similarity) with BM25 keyword search (for exact matches). For each query, retrieve candidates from both, combine results intelligently, and rerank. The implementation is straightforward: normalize scores from both retrieval methods and weight them (try 0.6 semantic / 0.4 keyword, then tune to your data).

The benefit is substantial. Keyword search catches terminology and names. Vector search catches conceptual matches. Together they improve recall without adding latency.

## Reranking and Two-Stage Retrieval

After retrieval, you have 50-100 candidates. Most are relevant-ish. You need the best ones for your LLM.

This is reranking. Run your initial retrieval with a fast bi-encoder. Rerank the top candidates with a cross-encoder (a model specifically trained to score query-document relevance). Keep the top 5-10. Apply metadata filters. Pass to the LLM.

Cross-encoder reranking consistently outperforms bi-encoder retrieval alone. The latency cost (100-300ms) is justified by precision gains. This two-stage pattern is current best practice for production systems.

## Context Length and Latency Tradeoffs

Modern LLMs have 100k+ token context windows. Resist the urge to use them. Most question-answering tasks see diminishing returns after 3,000-5,000 tokens of context. More tokens mean higher latency and cost with minimal quality improvement. Measure this on your evaluation set.

Be deliberate about what goes into context. Format clearly. Include metadata and source information. A well-formatted 4,000 tokens outperforms unstructured 8,000.

## When RAG Fails and Alternatives

RAG works best for dynamic data where you want current information. It fails when:

**Fine-tuning is better**: You have a large, high-quality dataset and want permanent domain knowledge. More expensive upfront but potentially cheaper long-term.

**Prompt engineering suffices**: Your task is well-defined and the model's base knowledge is enough. Structured data extraction often falls here.

**Long context replaces it**: With [Anthropic's 200k context windows](https://docs.anthropic.com/en/docs/build-with-claude/context-windows), sometimes dumping all documents into context and letting the model find what matters is faster and cheaper than building retrieval infrastructure.

Before investing in RAG, measure the cost-quality tradeoff against alternatives. See [scaling AI products](/blog/scaling-ai-products) for architecture decisions at scale.

## Data Access and MCP

The [Model Context Protocol](https://modelcontextprotocol.io) provides a standard way to expose retrieval as discrete tools. Instead of baking retrieval into application logic, define MCP-compatible search and retrieve operations. This enables multi-step agent workflows where agents can iteratively retrieve documents, reason, and request more specific information.

Structure your retrieval operations as clear, reusable tools and you're ready for agent-based applications as MCP adoption grows. See [context engineering](/blog/context-engineering-guide) for how RAG fits into broader context strategy.

## Production Implementation

Start with proven tools. [LangChain's RAG tutorial](https://python.langchain.com/docs/tutorials/rag/) walks the pattern. Use a vector database (Weaviate, Qdrant, Pinecone) for similarity search. Implement two-stage retrieval: fast bi-encoder retrieval, then cross-encoder reranking.

Monitor retrieval quality in production. Track precision at k, mean reciprocal rank, and generation quality. Set alerts for degradation. When metrics slip, investigate chunking staleness, embedding drift, or data distribution shifts.

Optimize cost: embeddings are computed once. Generation is ongoing. Reduce context length and batch queries when possible. At scale, see [AI for technical leaders](/blog/ai-engineering-technical-leaders) for architectural choices.

## Summary

The teams shipping high-quality RAG systems treat retrieval as seriously as generation. They measure from day one. They test embedding models on actual data. They implement hybrid search and reranking. They chunk thoughtfully, aligned with domain structure. They optimize context length empirically, not aspirationally. They monitor production metrics continuously.

RAG feels simple—embed documents, retrieve similar ones, generate answers. Production RAG is harder. The difference between systems that work in demos and systems that scale is taking retrieval seriously.

---

### AI Evaluation and Testing: How to Build Confidence in Non-Deterministic Systems

URL: https://halyard.dev/blog/ai-evaluation-testing
Published: 2026-03-06
Author: Alex Hinds
Category: AI Engineering Fundamentals
Tags: AI testing, LLM evaluation, AI quality, regression testing, production AI

> Traditional testing assumes determinism. AI systems break that assumption. Here's how to build evaluation pipelines that actually catch regressions.

# AI Evaluation and Testing: How to Build Confidence in Non-Deterministic Systems

Traditional software testing assumes a clean contract: given input X, produce output Y consistently. Your tests pass or fail. Your CI/CD pipeline catches regressions. AI systems obliterate this assumption. The same prompt fed to the same model produces different outputs. A change that improves accuracy on one case might degrade it on another. You need a fundamentally different approach.

The solution isn't to abandon testing—it's to build evaluation systems specifically designed for non-determinism. This requires measurement and statistical thinking. Instead of "does this work," you ask "how well does this work compared to the alternative?" This article covers the practical architecture of a production AI evaluation system: building test datasets, choosing your evaluation layers, detecting regressions, and closing the feedback loop between production and development.

Reference: [Anthropic's eval guide](https://docs.anthropic.com/en/docs/build-with-claude/develop-tests) covers foundational methodology. [Hamel Husain's practical guide](https://hamel.dev/blog/posts/evals/) and [Braintrust](https://www.braintrustdata.com) show industry-standard tooling.

## Three Layers of Evaluation

Build your evaluation system in layers, each catching different problems.

**Layer 1: Rule-based checks.** These are deterministic tests you run first: valid JSON, required fields present, response length within bounds, forbidden content absent. Fast, cheap, and catch obvious failures. Use these as your first gate.

**Layer 2: Model-graded scoring.** Ask another LLM to evaluate your system's output on dimensions like factual grounding, helpfulness, or adherence to a rubric. This scales well but your evaluation model is itself non-deterministic and can develop biases. Always triangulate with other methods.

**Layer 3: Human review.** The gold standard for subjective quality (tone, style, nuance) and for calibrating your automated systems. Keep human evaluation costs down by sampling strategically—review random samples daily, and 100% of failures or borderline cases.

## Building an Evaluation Pipeline

Start with a test dataset—a curated collection of inputs with reference outputs or evaluation rubrics. Minimum thirty cases per scenario, ideally more. Cover common cases, edge cases, and your known failure modes. Quality matters more than quantity. A dataset of 100 carefully labeled, representative cases beats a thousand cases with uncertain labels.

Build this as code: reproducible scripts, version-controlled, integrated into your CI/CD. A single model update should trigger automatic evaluation against your full test set. Track metrics across dimensions: accuracy, latency, cost, hallucination rate. Store results with timestamps so you can track trends over weeks and months.

Use [Anthropic's eval framework](https://docs.anthropic.com/en/docs/build-with-claude/develop-tests) as your starting point. For RAG systems, reference the [RAG evaluation guide](/blog/rag-systems-best-practices). Tools like [Braintrust](https://www.braintrustdata.com) and the [OpenAI eval framework](https://github.com/openai/evals) provide scaffolding for visualization and reporting. Customize to your domain rather than building from scratch.

The critical detail: maintain your dataset as a living artifact. When you find a failure in production, add it to your eval set immediately. When you discover a new edge case, add it. When you build new capabilities, add test cases for them. Historical evaluation allows you to track whether recent changes introduced regressions and to compare different model versions retrospectively.

## Offline Metrics vs. Online Truth

Your evaluation dataset measures offline—what your system could do on curated test cases. But production is where the truth lives. The critical split:

Offline metrics guide development. Accuracy, hallucination rate, latency, cost—measure these against your test set. They're fast and cheap to compute. But they're estimates. Your test set is curated. Production brings new cases you didn't anticipate.

Online metrics measure real behavior. Track user satisfaction, task completion on actual queries, and cost per successful interaction. An online change that improves accuracy by 2% on your eval set but reduces user engagement isn't a win.

Run A/B tests to bridge the gap. Deploy two versions to different user cohorts and measure which one users prefer. This is expensive but reliable. For most AI systems, online metrics ultimately matter more than offline scores.

See [AI product validation](/blog/ai-product-validation) for measuring user impact at scale.

## Regression Detection and Continuous Evaluation

Before pushing any change to production, run your full evaluation suite. Compare metrics against the current version. Look for both aggregate changes and breakdowns by scenario, input length, or domain.

Set regression thresholds. If accuracy can't drop more than 2%, codify it. If hallucination rate can't increase by 0.5%, enforce it. Hard limits prevent subtle degradations from accumulating.

When context changes—new retrieval results, different [prompt engineering](/blog/context-engineering-guide), model updates—run evaluation first. It's easy to optimize one dimension while accidentally breaking another.

For [RAG systems](/blog/rag-systems-best-practices), track both retrieval quality and generation quality separately. A prompt change might hurt grounding even if overall accuracy looks unchanged.

Production monitoring complements offline evaluation. Log accuracy on real requests (through human feedback, implicit signals, or explicit user ratings). When production accuracy diverges from your eval set, your test data has a gap. Add those failure cases to your dataset.

## Practical Priorities

Evaluation is not a one-time effort—it's a continuous practice woven into development from day one. Start simple. You don't need a perfect system before shipping. But you need structure before you scale.

Priority one: build evaluation into your deployment process. Before any production change, run your test suite. Compare against the current version. Track metrics over time, not just on the latest change. This catches regressions that compound.

Priority two: invest in automation early. Manual evaluation doesn't scale. Your engineers shouldn't spend hours reviewing outputs. Layer your evaluation—rule-based checks first (fast, deterministic), model-graded scoring at scale (flexible, fast), human review of failures and samples (expensive, reliable).

Priority three: make evaluation reports a standard part of code review. Engineers should see how their changes affect system metrics. This creates accountability and builds evaluation literacy into your culture.

Practical mistakes to avoid: testing only on training data instead of production cases, changing too many variables at once so you can't tell what actually improved, ignoring latency and cost until late in development (then discovering the system isn't viable), and deploying without checking for regressions on critical subgroups of users.

Track both offline metrics (accuracy, latency, cost on your test set) and online metrics (user satisfaction, task completion on real traffic). Offline estimates guide development. Online metrics reveal truth. A/B testing bridges the gap when changes are significant enough to warrant user exposure.

---

*Alex Hinds is Principal Consultant at Halyard Labs, where he advises companies building AI systems.*