What it costs an agent to translate a resource file

The question

A coding agent asked to translate an i18n resource file can do the work itself, or it can hand the file to a translator running on the same machine. The first spends API tokens on every value. The second spends none on the values, and BetterTranslator can do it with no agent in the loop at all. The benchmark measures what each way costs, on one file, in tokens, in approximate dollars, in requests and in wall clock.

Three arms were measured, named as the artifacts name them.

A fourth path, BetterTranslator's own window or bt translate with no agent involved, spends zero API tokens by construction and is carried in the projection as the zero row.

How the test was built

A value qualifies for a slice when it is a string of 20 to 180 characters carrying no pluralisation pipe, HTML tag or line break, and still holding at least 4 words and 20 ASCII letters once placeholder tokens are removed. Every selected value is prose a translator has to change. Each slice is a prefix of the next larger one. The full source hash is a8fdce5aba678743b2aac57a9b8c06405e41b836cd174051fc09357da648a633.

Accounting note: this Claude Code build emits no OpenTelemetry metrics or events, so per-request accounting is read from Claude's stream-json usage blocks and the print mode result object. 4 exporter configurations were tried and none produced output. As a cross-check the harness compares both sources per run: 22 of 33 runs agree within 2 percent, and the 11 that disagree carry both numbers in summary.csv.

Per slice

The third arm, for completeness. It was not run at 500 keys by instruction.

What delegating changes

The delegated arm against the agent translating the file itself. The 500 key row is not comparable, because the delegated arm produced no file.

Tokens and money do not rank these arms the same way. In tokens there is no crossover anywhere between 10 and 500 keys: the delegated path costs more at every measured slice. In approximate dollars the two lines cross at about 87 keys, because the delegated path swaps fresh input for cache reads and cache reads bill far below it. At 200 keys the delegated path is the cheaper of the two in money while spending nearly six times the tokens.

Where the tokens go: the payload really does stay out of the context. On the smallest slice the whole MCP exchange is 306 bytes of tool arguments and 707 bytes of tool results against a 1,205 byte input file, and the entire tool surface adds 743 tokens of cache creation over the agent path. The cost is request count. Every job_status poll is a full API request that replays the whole conversation, so the arm's bill is set by how often it asks whether the local model has finished: 7.0 requests at 10 keys, 26.5 at 200. The fitted floor with an empty file is 244,502 tokens. Hold everything else constant and give the arm the agent path's request count, as a blocking call or a wait tool would, and the same 200 key work lands near 183,575 tokens against 195,574. That last figure is arithmetic on measured per-request cost, not a measured run.

The whole file

Projected, not measured: the 4,666 key file does not fit one context window, so the projection repeats the largest measured slice, 200 keys, as 24 chunks. It is arithmetic over measured runs.

Two cells are withheld. The mcp-value request count has no gate-passing basis at 200 keys, since both repetitions fell back to another model. The local wall clock is withheld because the artifacts disagree with each other.

The mcp-path row is cheaper in money than the agent path only above the 87 key break-even, and it reaches that price with 27,361,836 tokens against 4,693,764. The mcp-value projection rests on runs the gate excluded and should be read as an upper bound rather than a result.

The fairness gate

A run passes only when its output file is a JSON object whose key set and key ordering match the slice exactly, whose values are all non-empty strings, whose placeholder multiset per value matches the source, and none of whose values equal their source. The last check has one allowance: a value may equal its source when every alphabetic word in it is an allowlisted brand, protocol or locale token or begins with a capital letter, which covers product names and title-case proper nouns, or when the value is a command line carrying an executable name or a long flag.

Across all 33 recorded runs, summary.csv reports 0 key-set failures, 0 key-order failures and 0 placeholder failures. 29 of the 33 runs delivered a file at all. Every exclusion below is an untranslated value, a model fallback, a session limit or a missing file.

Of 33 runs, 33 billed a request, 1 ended on the account session limit and 3 fell back from the requested model, which makes those uncomparable.

Limitations

The MCP server costs more tokens, not fewer. At every measured slice it spent more than the agent translating the file itself: 137.1 percent more at 10 keys, 337.2 percent at 50, 482.9 percent at 200. Its supported advantage is money above the 87 key break-even, and that advantage exists because cache reads bill below fresh input. It is not a token saving and does not become one at any size measured here.

No quality judgement exists. Nothing in this run scores adequacy, fluency or meaning. The only value-level measurement is whether an output value differs from its source. On the slices it was measured on, the local engine left 2.3 percent of values in English. The agent path passed the gate at every slice. That comparison covers untranslated values and nothing else. Any claim about how well either side translates is outside these artifacts.

The delegated path produced no file at 500 keys. All 3 repetitions billed requests and delivered nothing. The agent stopped waiting while the local job was still running, and the MCP server dies with the session that started it, so the finished work was thrown away. Those runs are a delivery failure, not a cheap result, which is why they are excluded from the difference table and why the largest slice has no head to head figure.

The whole file numbers are projections. No run has ever translated all 4,666 keys under measurement. The projection multiplies the 200 key slice by 24. Chunking differently, or hitting a context or session limit part way through, changes the result.

Thin repetitions. Two comparable repetitions per cell, because repetition 1 is excluded as cache warm-up. The report marks the cells whose spread is wide: the agent path at 500 keys (15.0 percent), mcp-value at 10 keys (32.7 percent) and at 50 keys (48.0 percent).

One file, one pair, one machine. A single en.json translated English to Czech, on one Windows host, with one local model at one quantization, against one CLI version and one Claude model. Both sides of this comparison move over time.

Reproducing this

btbench prepare --source <path to en.json> --model claude-opus-5 --sizes 10,50,200,500
btbench run --run <run-id> --arms direct,mcp-path --reps 3
btbench collect --run <run-id>
btbench compare --run <run-id>

prepare probes the MCP server, hashes the source and cuts the slices. run executes every arm and repetition in a fresh session. collect re-derives summary.csv from the transcripts. compare writes compare.md. The orchestrator is a .NET project that builds to btbench.

The artifacts

Every figure above resolves to one of these files, at the section, table row and column the registry records. The four the figures cite travel with the site so the check runs offline.