The Running Cost of AI

This paper explains where the running cost of an AI system comes from and how to keep it low.

AppsTech Labs calls its approach Resource-Efficient Engineering. For every system, our engineers measure what it costs to process one transaction, such as one invoice.

Growth in use. A pilot usually starts in one team, such as accounts payable. Once it works, other teams ask for it, and new tasks are added, such as reading contracts or answering customer emails. Each new use adds requests, and the AI model provider bills every request. As opposed to a software license, which has a fixed annual price, pay-per-use pricing moves with volume. A tenfold rise in requests brings a tenfold rise in the bill, and the increase reaches the budget without passing through a purchase approval.

Oversized requests. A typical request carries standing instructions, several examples, whole documents and the history of the conversation. The company pays for every part of it on each request. Longer requests also reduce the reliability of the answers. A July 2025 study of 18 leading models found that their performance declined as the input grew longer[4].

Unmeasured cost. In a Deloitte survey of 200 North American chief financial officers in mid-2026, 46% named cost uncertainty or lack of transparency as their biggest internal concern about their company's use of AI[5]. In the same survey, 43% pointed to insufficient visibility into AI tools or their use[5]. Without a figure for the cost of one transaction, the monthly invoice from the AI model provider is the first sign that costs have risen.

AI model providers can store the fixed part of a request, such as the instructions and examples, and reuse it for a few minutes or up to an hour. This is called prompt caching. A prompt is the full text sent to the model with a request. Anthropic, Google and OpenAI all bill stored text at about a tenth of the normal input price on their main models[1][2][3]. The discount applies when the fixed part comes first and the document comes last. Our engineers therefore treat the order of a prompt as a cost decision.

A second set of savings comes from keeping bulk data away from the model. A system may retrieve ten thousand rows from a database to answer a single question. In the costly design, the model receives all ten thousand rows. In the efficient design, a short program filters the rows first and passes the model the result. In a worked example published in November 2025, Anthropic reports that this change reduced one task from 150,000 tokens to 2,000, a saving of 98.7%[7]. Cloudflare reported a similar effect in February 2026 for the descriptions of the services a model can use. Giving a model access to all of Cloudflare's services in this way took about 1,000 tokens of description, against 1.17 million by the usual method[6].

The test is written first, as a set of real cases from the customer's own work, including the difficult ones, each with its correct answer. The engineers try the smallest model that could pass, and move to a larger one when the smaller model fails.

For high-volume sorting, such as classifying documents or directing messages to the right team, a small model trained for that single task often gives the best result. An independent comparison published in January 2026 ran 32 experiments on tasks of this kind. A small model trained for the task processed 277 items per second, against 12 for a small general-purpose model that writes out its answers. The trained models were also more accurate on three of the four tasks tested[8].

For models that AppsTech Labs runs on servers it manages, the same logic applies to the hardware. When a server processes many requests together, energy per token can fall by a factor of three to five, according to measurements across 46 models published by ML.ENERGY in January 2026[9]. A model installed on the customer's own servers carries no fee per question, and the customer's data stays inside its network.

These methods remain useful for the models we run ourselves. Their real benefit depends on how many requests the server handles at the same time, so it has to be measured under real working conditions. The January 2026 study shows this with quantization. It compared the same models in two versions, one storing its numbers in 16 bits, the usual format, and the other in 8 bits. The 8-bit version is expected to use less energy. Yet when the server handled 8 to 16 requests at a time, the 8-bit version used up to 56% more energy than the 16-bit version. The 8-bit version began to save energy once the server handled more than about 64 requests at a time, by 11% in the typical case[9].

A useful answer to each question is a figure or a named design decision, given in writing so that the finance team can check it.