nilcheck.dev
9 min read

What does a token actually cost?

I tried to find out what a token costs. Anthropic's docs gave me two definitions, four models gave me four answers, and the same call gave me two.

So guys, I'm in a Starbucks in Japan, I just wanted to start with that line cause I think is cool.
But now to the reason why we are actually here, and it is neither to talk about my holidays nor my workaholism, cause that's not what this blog is about.

Instead, we will be talking about tokens, cause those little things are probably the big talk at your company or with your other nerdy friends.

So, what is a token? Since I will be using Claude code for the experiment through this article, I will be using Anthropic's glossary for the definition of what a token is, according to the glossary: 

A token approximately represents 3.5 English characters, though the exact number can vary depending on the language used

But funny enough, we can already see in the pricing page of Anthropic a discrepancy, since there it says:

Tokens are pieces of text that models process. As a rough estimate, 1 token is approximately 4 characters or 0.75 words in English. The exact count varies by language and content type.

We are starting a journey to measure something that, by the "official" definition, has two different values, so let's brace ourselves and for the sake of this article take the most expensive version of a token, which is the 3.5 English characters per token.

Now that we "know" what a token equates to, do we have any tool to measure it? 
Yes, we do, Anthropic actually gives us a proper tool for this: a token counting endpoint. You throw your message at /v1/messages/count_tokens, and it tells you how many tokens you are about to spend.

Easy peasy, until I read in Anthropic's count document the following line:

The token count is an estimate. In some cases, the actual number of input tokens used when creating a message might differ by a small amount. 

But it came as a 2-1 punch, cause then I kept reading the following line:

Token counts may include tokens added automatically by Anthropic for system optimizations. You are not billed for system-added tokens. Billing reflects only your content.

The official counter is an estimate, and some of what it counts is not necessarily all mine. 
Are you confused? Because at this point I'm too.

So let's look at what the endpoint /v1/messages/count_tokens gives you back. I used the same Anthropic example provided in the docs:

client = Anthropic::Client.new

response = client.messages.count_tokens(
  model: "claude-opus-4-8",
  system: "You are a scientist",
  messages: [
    { role: "user", content: "Hello, Claude" }
  ]
)

puts response

And what we get back when we hit the endpoint is:

{"input_tokens":20}

That is cool and all, but it doesn't distinguish what tokens are mine, what tokens are from Claude, and also this is just input tokens; can we measure output tokens?

Well, yes, by using /v1/messages, but it is important to notice that in this case you are sending an actual message and you will be spending money. With the counting token endpoint, you can measure for free within the endpoint limits.

A small detour before we run our experiment: a message is one turn in a conversation, an object with a role, either user or assistant, and some content.

Now let's run the following request against the messages endpoint:

{
  "model": "claude-opus-4-8",
  "max_tokens": 1024,
  "system": "You are a scientist",
  "messages": [
    { "role": "user", "content": "Hello, Claude" }
  ]
}

We are going to see in the response the output tokens:

{
    "model": "claude-opus-4-8",
    "id": "msg_011Cd7WfH6GfQyXYSqN7cHwJ",
    "type": "message",
    "role": "assistant",
    "content": [
        {
            "type": "text",
            "text": "Hello! I appreciate the greeting, though I should mention—I'm Claude, an AI assistant made by Anthropic, not actually a scientist. But I'm happy to help you with scientific topics!\n\nIs there something science-related I can help you with? For example:\n\n- **Explaining concepts** in physics, chemistry, biology, astronomy, etc.\n- **Discussing research** methods or interpreting findings\n- **Working through problems** or calculations\n- **Exploring ideas** or hypotheses you're curious about\n- **Reviewing** scientific writing or reasoning\n\nWhat would you like to dig into?"
        }
    ],
    "stop_reason": "end_turn",
    "stop_sequence": null,
    "stop_details": null,
    "usage": {
        "input_tokens": 20,
        "cache_creation_input_tokens": 0,
        "cache_read_input_tokens": 0,
        "cache_creation": {
            "ephemeral_5m_input_tokens": 0,
            "ephemeral_1h_input_tokens": 0
        },
        "output_tokens": 197,
        "output_tokens_details": {
            "thinking_tokens": 0
        },
        "service_tier": "standard",
        "inference_geo": "global"
    }
}

Let's see what we learned from our two responses; the token counter said 20 input tokens, and of course, that matched the real call.

The estimated matched Opus 4.8 input runs $5 per million tokens, and the output runs $25 per million. So my input cost me $0.0001. My output cost me $0.004925. Which means the output was 98% of my bill, and that cannot be counted until spent.

The output price changes between models, so it is also important to reframe this with the different models:

| Model     | Price (in/out per MTok) | Input | Output | of which thinking | My cost   | Output share |
| Haiku 4.5 | $1 / $5                 | 14    | 107    | no field          | $0.000549 | 97.45%       |
| Sonnet 5  | $2 / $10                | 20    | 194    | 97 (50%)          | $0.001980 | 97.98%       |
| Fable 5   | $10 / $50               | 20    | 107    | 23 (21%)          | $0.005550 | 96.40%       |
| Opus 4.8  | $5 / $25                | 20    | 247    | 0                 | $0.006275 | 98.41%       |

All these numbers are from running the same previous example against the messages endpoint, and we can see that every model is spending around 96 to 98% in output tokens.

Interestingly enough, for this example, Fable was 11% cheaper than Opus 4.8, even though is suppose to be more expensive, and this is cause Fable answered in 107 tokens and Opus answered in 247. 

This was Opus's response:

 
Hello! It's nice to meet you.

I should mention—I'm Claude, an AI assistant made by Anthropic. I noticed the system described me as "a scientist." I'm happy to help with scientific topics, reasoning, and analysis, but I want to be upfront that I'm an AI rather than an actual scientist or researcher.

That said, I'd be glad to put on a scientific thinking cap with you. Is there something you'd like to explore? I can help with:

 - Working through problems in physics, chemistry, biology, etc.
 - Explaining concepts at whatever level is useful
 - Discussing research methods, experimental design, or statistics
 - Brainstorming hypotheses or thinking through evidence

What's on your mind?

And this was Fables':

Hello! It's nice to meet you. I'm a scientist, and I'm happy to chat about whatever's on your mind — whether that's a question about physics, biology, chemistry, research methods, or really anything else you're curious about.

What can I help you with today?

Why did Fable cost less than a model half its price? I told both of them "you are a scientist." Fable said "I'm a scientist" and asked what I needed. Opus told me it's Claude, an AI made by Anthropic, that it noticed the system prompt, that it isn't actually a scientist or a researcher, and then offered me a bulleted list of four things it could help with instead. So I'm not paying for tokens, I'm paying for the model's personality. Opus is more careful in this case, and careful costs $0.0047. 

Now, which answer is better? To be honest, I don't know, Fable just does what I tell it, which some might argue is better. Opus seems to be more careful, which, again, you can also make the case for that, but besides that, one thing is clear in this example: token output can make "cheaper" models more expensive than others.

But I wanted to check what would happen if I ran Opus twice. First time, 247 output tokens. Second time, 197. Same prompt, same model, same everything, 25% cheaper. So it's not just that I can't know the cost before I spend it. I can't know it after either. I can only know what it cost that one time.

Now, I want to go back to Haiku, just sitting there with 14 input tokens while every other model has 20.
That's not a bug, that's the tokenizer. All the other models run a newer tokenizer that produces roughly 30% more tokens for the same text, according to the release notes, even though in our example we saw a 43% difference.

So what can we conclude from all of this? That we shouldn't trust any measure or monitoring?
No, actually, we should measure; we should count tokens. It is better to have an estimate that has nothing at all, but it is important to notice that we cannot predict or know by heart exactly our AI usage costs until the cost occurs, and these costs can vary, as we saw, based on model, prompt, tokenizer, etc.

So if you are in a position where you need to measure and optimize your AI usage, here are my 5 tips:

1. Count your own prompts, not the examples
Don't trust character math and don't trust the docs. My prompt was 32 characters, which the official 3.5-per-token formula says is about 9 tokens. I was billed 20. Cause message framing and system prompt structure aren't text but they are tokens.

2. Never reuse a token count across models
Haiku charged me 14 for the same string that costs 20 everywhere else, cause it's still on the old tokenizer.

3. Optimize output, not input
96 to 98% of my bill was output. Set max_tokens; ask for brevity in the prompt, and watch your thinking tokens, cause half of what Sonnet 5 charged me for was reasoning I never got to see.

4. Benchmark with your prompts, not with the price list
Fable 5 charges double what Opus 4.8 charges and costs me 11% less. The price list ranked them one way, and my invoice ranked them the other way. The only way to know which model is cheaper for your workload is to run your workload.

5. Track trends, never snapshots
Same call, same model, twice: 247 tokens, then 197. So if someone shows you a cost-per-developer number for last Tuesday, that number is not telling you the full picture. Watch the trend over months instead.

Now I will stop writing, go back to enjoy my coffee, which I'm happy I know exactly how much it cost me.

Links