What is a harness, actually?

Work Musings · General AI

I started digging into harnesses after spending time with a team of ML engineers at Canva.

They’re doing work I find inspiring around our support functions: customer feedback and questions, regression testing, evals, and custom harnesses for their team. My time with them made me want to understand the whole setup much more deeply and, honestly, made me feel like I was a little bit behind.

In the simplest terms, a harness is just a new sexy term for the environment around an AI that lets it do work. The tools, context, permissions, yada yada.

What interests me most is what happens when a team runs its work through that environment. You can see what the AI did, check the quality, change part of the process and measure whether the change helped. For a go-to-market team, that creates a way to improve the whole sales motion.

Imagine asking AI to prepare a brief for a customer call. It needs CRM notes and a recent company announcement. The model can request a tool; software carries out the allowed request and brings the result back. The model then decides whether to investigate further, ask a question or finish. That repeated exchange is the agent loop.

The model interprets the task. A skill supplies a reusable method. A connector or tool makes information or actions available. The harness runs the exchange. Together, they let an agent work toward a goal. Anthropic describes the runtime and surrounding components here.

If you’ve given an AI workspace instructions, reusable skills and access to tools, you’re already configuring parts of this environment. You don’t need to build your own to get useful work done.

The reason to own more of it would be a specific gap: context the workflow cannot reach, actions it cannot reliably finish, or a process the team cannot inspect and improve.

We’re developing a custom harness for our go-to-market work. The direction is a shared view of an account, connected to the tools people use, with checks and feedback around the work. It’s still in development.

Take the question, “Who should I contact at this account?” A useful answer needs the account’s product usage, previous conversations and the sales play we’re running. It should explain why a particular role fits that situation, using evidence we can inspect. A familiar job title on its own tells us very little.

I see a few reasons to build around that:

Those capabilities can also exist in hosted products. The custom work earns its place when it gives the team a better fit or more control than its current setup provides.

The part I find most valuable is having a place to launch, track and improve these workflows together.

For a prospecting run, I’d want one record that links:

Later, that same record should connect to the CRM outcome. Tool failures and abandoned runs belong in the picture too.

That makes different problems easier to distinguish. Did we choose the wrong person? Was the message unsupported? Did the rep have to rewrite it? Did an approved action fail? Or did everything work as intended and the offer still get no response?

This is where evals come in. Evals, short for evaluations, are repeatable checks that hold the work to the same standard every time.

For picking a contact, I’d check for a current role, a reason the person fits the play and evidence behind the recommendation. For a message, supported claims and a relevant next step.

Rep feedback catches what the checks miss. Someone still has to read that feedback before changing the workflow.

A better eval score tells us something about the work. We still need to find out whether that work produces a better business result.

The longer-term idea is to give prospecting the same kind of experimentation loop a growth team uses. We’re not there yet, but this is the direction I find exciting.

Say we think an enablement lead will respond to an offer about making better sales decks. I’d run that against the current play on the same kind of accounts, assign them at random where we can, and keep each account in one version.

Before anything goes out, evals check the recipient, the claims and the handoff. Then I’d look past whether a rep liked the draft. Qualified replies and meetings are the early signal, and opportunities, revenue and the cost of running the motion take longer.

I’d count rep review and repair time next to the model and tool costs. More replies aren’t much of a win if they create more work without better opportunities.

The output I want is a decision: expand the play, change the weak part or stop. I’d give it enough volume and time before calling anything an improvement.

Tracking shows what happened. The comparison shows what difference the change made.

More control also means more to maintain: integrations, access rules, checks and the data linking work to outcomes.

The number I’d watch is cost per accepted task, including model and tool costs, human review, fixes and engineering. For a sales motion, I’d also follow that through to the cost and quality of the opportunities it creates.

A cheap run that needs twenty minutes of cleanup is not a cheap result. You just moved the bill to someone’s calendar.

I’d keep using the tools I have until a real limitation justifies owning more of the process.

If your team already runs an AI workflow, I’d start small. Pick one recurring task and link it to its downstream result.

That gives the next experiment something concrete to improve.

The reason to build is being able to finish useful work, understand what happened and make the next version measurably better.

Robert