# Character.AI describes how it reduces inference costs

[Canonical reading](https://escapemeridian.com/readings/character-ai-describes-how-it-reduces-inference-costs-16f44073)

Character.AI’s article, published June 21, 2024, argues that efficient inference—the process by which large language models generate replies—is essential to serving those models at scale. Character.AI says it reduces the memory needed for attention through changes to its model architecture, reuses cached attention data between chat turns, and trains and serves models using int8 quantization. It reports that its system achieves a 95% cache rate and that serving costs have fallen by a factor of 33 compared with late 2022. Character.AI argues that these changes make it easier and less costly to serve large language models to a global audience, connecting this work to its aim of integrating AI into daily life. The reported cost reduction has no stated measurement method or detailed comparison, and the article does not provide the evaluation results behind its claim about attention-horizon changes.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

Character.AI · Article

Original source date: Jun 21, 2024 (2024-06-21T00:13:00.000Z).

Date basis: Date stated on the publisher’s page.

Added to Escape Meridian: 2026-11-18T12:18:13.524Z. First observed: 2026-10-30T23:04:17.309Z. Retrieved: 2026-10-30T23:06:07.325Z.

- [Optimizing AI Inference at Character.AI](https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/)

## Limits

Text as retrieved on October 30, 2026; originally published June 21, 2024. The article reports Character.AI’s own serving-cost reduction but gives no measurement method, baseline details, or comparison results for that figure. The article reports no significant impact on evaluation metrics but does not provide the evaluation results. The article describes its systems and reported costs but does not establish how the reported efficiencies affect users in practice.


## Open question

How would the reported cost reduction compare under a stated, independently checkable measurement method?


## Source-stated

Character.AI says it reduces the KV cache size by more than 20X using techniques in its serving stack.

> We use the following techniques to reduce KV cache size by more than 20X without regressing quality.

- [Supporting source](https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/)

## Source-stated

Character.AI says its inter-turn caching system achieves a 95% cache rate.

> Our system achieves a 95% cache rate, further reducing inference cost.

- [Supporting source](https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/)

## Source-stated

Character.AI reports that it reduced serving costs by a factor of 33 compared with late 2022.

> We have reduced serving costs by a factor of 33 compared to when we began in late 2022.

- [Supporting source](https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/)

## Source-stated

Character.AI says its evaluation metrics were not significantly affected by reducing the attention horizon on most attention layers.

> We found that reducing attention horizon to 1024 on most attention layers does not have a significant impact on evaluation metrics, including the long context needle-in-haystack benchmark.

- [Supporting source](https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/)

## Source and contributor context

Character.AI · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Model inference

Using a trained model to produce a prediction or response to new input.

- [Definition reference](https://developers.google.com/machine-learning/glossary#inference)

## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Quantization

Storing a model's numbers at lower precision so it uses less memory and runs faster, sometimes at a cost in accuracy.

- [Definition reference](https://developers.google.com/machine-learning/glossary#quantization)

## Figure

Figure develops humanoid robots and publishes its own tests of how they perform physical tasks.

- [Explore this context](/entities/figure)
- [Reference](https://www.figure.ai/news)
- [Reference](https://www.figure.ai/news/helix-2-5-zero-shot-30-home-generalization)

## Editorial provenance

Version 1. Publication ID: 2288d8bf-5716-938e-17c4-3817659a2748.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T12:13:49.747Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical.

Observation: 928310c6b870242579b363056c8b6124ab735456749968d6fc4dac9d89f7a122. Source text hash: d005739845061d7a53d23bd23c2583240259b6ea65e5eaee132c31e2c200584e.


## Revision and source-check history

### Version 1 — 2026-11-18T12:18:13.524Z

Character.AI’s article, published June 21, 2024, argues that efficient inference—the process by which large language models generate replies—is essential to serving those models at scale. Character.AI says it reduces the memory needed for attention through changes to its model architecture, reuses cached attention data between chat turns, and trains and serves models using int8 quantization. It reports that its system achieves a 95% cache rate and that serving costs have fallen by a factor of 33 compared with late 2022. Character.AI argues that these changes make it easier and less costly to serve large language models to a global audience, connecting this work to its aim of integrating AI into daily life. The reported cost reduction has no stated measurement method or detailed comparison, and the article does not provide the evaluation results behind its claim about attention-horizon changes.

## Original source

Character.AI · Article

Original source date: Jun 21, 2024 (2024-06-21T00:13:00.000Z).

Date basis: Date stated on the publisher’s page.

Added to Escape Meridian: 2026-11-18T12:18:13.524Z. First observed: 2026-10-30T23:04:17.309Z. Retrieved: 2026-10-30T23:06:07.325Z.

- [Optimizing AI Inference at Character.AI](https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/)

## Limits

Text as retrieved on October 30, 2026; originally published June 21, 2024. The article reports Character.AI’s own serving-cost reduction but gives no measurement method, baseline details, or comparison results for that figure. The article reports no significant impact on evaluation metrics but does not provide the evaluation results. The article describes its systems and reported costs but does not establish how the reported efficiencies affect users in practice.


## Open question

How would the reported cost reduction compare under a stated, independently checkable measurement method?


## Source-stated

Character.AI says it reduces the KV cache size by more than 20X using techniques in its serving stack.

> We use the following techniques to reduce KV cache size by more than 20X without regressing quality.

- [Supporting source](https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/)

## Source-stated

Character.AI says its inter-turn caching system achieves a 95% cache rate.

> Our system achieves a 95% cache rate, further reducing inference cost.

- [Supporting source](https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/)

## Source-stated

Character.AI reports that it reduced serving costs by a factor of 33 compared with late 2022.

> We have reduced serving costs by a factor of 33 compared to when we began in late 2022.

- [Supporting source](https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/)

## Source-stated

Character.AI says its evaluation metrics were not significantly affected by reducing the attention horizon on most attention layers.

> We found that reducing attention horizon to 1024 on most attention layers does not have a significant impact on evaluation metrics, including the long context needle-in-haystack benchmark.

- [Supporting source](https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/)

## Source and contributor context

Character.AI · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media retrieval status: unchecked.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Model inference

Using a trained model to produce a prediction or response to new input.

- [Definition reference](https://developers.google.com/machine-learning/glossary#inference)

## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Quantization

Storing a model's numbers at lower precision so it uses less memory and runs faster, sometimes at a cost in accuracy.

- [Definition reference](https://developers.google.com/machine-learning/glossary#quantization)

## Figure

Figure develops humanoid robots and publishes its own tests of how they perform physical tasks.

- [Explore this context](/entities/figure)
- [Reference](https://www.figure.ai/news)
- [Reference](https://www.figure.ai/news/helix-2-5-zero-shot-30-home-generalization)

## Editorial provenance

Version 1. Publication ID: 2288d8bf-5716-938e-17c4-3817659a2748.

Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-18T12:13:49.747Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: bridge\_admitted\_by\_model, second\_opinion\_categorical.

Observation: 928310c6b870242579b363056c8b6124ab735456749968d6fc4dac9d89f7a122. Source text hash: d005739845061d7a53d23bd23c2583240259b6ea65e5eaee132c31e2c200584e.


### uncertain — 2026-11-18T12:13:49.747Z — anthropic/claude-opus-5.5

Publication: 2288d8bf-5716-938e-17c4-3817659a2748

attribution — pass: The text is a single Character.AI blog post that makes its own argument about its inference stack. The reading consistently attributes the architecture, caching, quantization and cost claims to Character.AI as its own statements.

> In this blog post, we share some of the techniques and optimizations we have developed over the past two years and recently employed.

summary — uncertain: Most of the reading is clear, and inference, LLM and quantization are defined. However, the central mechanism rests on terms a non-specialist cannot follow: attention, the KV cache (its first claim), cache rate and attention horizon. None is explained or defined. Smallest repair: add term-library entries for attention, KV cache and attention horizon. Alternatively, add one plain clause saying the KV cache stores intermediate data from earlier text so the model need not recompute it. The 13.5X comparison with commercial APIs is omitted, but that omission is acceptable.

> The key bottleneck of LLM inference throughput is the size of the cache of attention keys and values (KV).

date — pass: The originalPublished instant 2024-06-21T00:13Z falls on the UTC calendar day June 21, 2024, which matches the reading. The evidence is a publisher page, not an archive copy, so no archive disclosure is needed. The page was modified on 2025-08-07, but the cited passages are integral to the article's dated argument, such as the past two years since late 2022. There is no hindsight framing, and no forecast is presented as an outcome.

> We have reduced serving costs by a factor of 33 compared to when we began in late 2022.

support — pass: Each claim matches its verbatim excerpt and is framed as source-stated: the 20X KV reduction, the 95% cache rate, the 33X cost reduction, and no significant metric impact from the shorter attention horizon. The limitations accurately note that no measurement method or evaluation results are given. The question follows from that gap. A later page edit exists, but these passages appear plainly part of the original article.

> Efficient inference is crucial for scaling AI systems and integrating them seamlessly into our daily lives.

media — pass: No media was selected, and no verified candidate media items were supplied, so leaving the reading without media is appropriate.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/16f440734294b4a6df0a448880023514f5580a037fb1f6cc13eb0c3eb26aaefe/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/character-ai-describes-how-it-reduces-inference-costs-16f44073.json)
