{"id":"16f440734294b4a6df0a448880023514f5580a037fb1f6cc13eb0c3eb26aaefe","title":"Character.AI describes how it reduces inference costs","argument":"Character.AI’s article, published June 21, 2024, argues that efficient inference—the process by which large language models generate replies—is essential to serving those models at scale. Character.AI says it reduces the memory needed for attention through changes to its model architecture, reuses cached attention data between chat turns, and trains and serves models using int8 quantization. It reports that its system achieves a 95% cache rate and that serving costs have fallen by a factor of 33 compared with late 2022. Character.AI argues that these changes make it easier and less costly to serve large language models to a global audience, connecting this work to its aim of integrating AI into daily life. The reported cost reduction has no stated measurement method or detailed comparison, and the article does not provide the evaluation results behind its claim about attention-horizon changes.","topicIds":["investment-booms-and-busts"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 30, 2026; originally published June 21, 2024. The article reports Character.AI’s own serving-cost reduction but gives no measurement method, baseline details, or comparison results for that figure. The article reports no significant impact on evaluation metrics but does not provide the evaluation results. The article describes its systems and reported costs but does not establish how the reported efficiencies affect users in practice.","speaker":{"name":"Character.AI","roleAtSource":"Publisher"},"source":{"title":"Optimizing AI Inference at Character.AI","publisher":"Character.AI","format":"Article","url":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","publishedAt":"2024-06-21T00:13:00.000Z","dateBasis":"Date stated on the publisher’s page."},"evidence":[{"text":"Character.AI says it reduces the KV cache size by more than 20X using techniques in its serving stack.","excerpt":"We use the following techniques to reduce KV cache size by more than 20X without regressing quality.","url":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","relationship":"Source-stated"},{"text":"Character.AI says its inter-turn caching system achieves a 95% cache rate.","excerpt":"Our system achieves a 95% cache rate, further reducing inference cost.","url":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","relationship":"Source-stated"},{"text":"Character.AI reports that it reduced serving costs by a factor of 33 compared with late 2022.","excerpt":"We have reduced serving costs by a factor of 33 compared to when we began in late 2022.","url":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","relationship":"Source-stated"},{"text":"Character.AI says its evaluation metrics were not significantly affected by reducing the attention horizon on most attention layers.","excerpt":"We found that reducing attention horizon to 1024 on most attention layers does not have a significant impact on evaluation metrics, including the long context needle-in-haystack benchmark.","url":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would the reported cost reduction compare under a stated, independently checkable measurement method?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Character.AI describes how it reduces inference costs","summary":"Character.AI’s article, published June 21, 2024, argues that efficient inference—the process by which large language models generate replies—is essential to serving those models at scale. Character.AI says it reduces the memory needed for attention through changes to its model architecture, reuses cached attention data between chat turns, and trains and serves models using int8 quantization. It reports that its system achieves a 95% cache rate and that serving costs have fallen by a factor of 33 compared with late 2022. Character.AI argues that these changes make it easier and less costly to serve large language models to a global audience, connecting this work to its aim of integrating AI into daily life. The reported cost reduction has no stated measurement method or detailed comparison, and the article does not provide the evaluation results behind its claim about attention-horizon changes.","question":"How would the reported cost reduction compare under a stated, independently checkable measurement method?","claims":[{"text":"Character.AI says it reduces the KV cache size by more than 20X using techniques in its serving stack.","excerpt":"We use the following techniques to reduce KV cache size by more than 20X without regressing quality.","support":"source-stated"},{"text":"Character.AI says its inter-turn caching system achieves a 95% cache rate.","excerpt":"Our system achieves a 95% cache rate, further reducing inference cost.","support":"source-stated"},{"text":"Character.AI reports that it reduced serving costs by a factor of 33 compared with late 2022.","excerpt":"We have reduced serving costs by a factor of 33 compared to when we began in late 2022.","support":"source-stated"},{"text":"Character.AI says its evaluation metrics were not significantly affected by reducing the attention horizon on most attention layers.","excerpt":"We found that reducing attention horizon to 1024 on most attention layers does not have a significant impact on evaluation metrics, including the long context needle-in-haystack benchmark.","support":"source-stated"}],"limitations":["Text as retrieved on October 30, 2026; originally published June 21, 2024.","The article reports Character.AI’s own serving-cost reduction but gives no measurement method, baseline details, or comparison results for that figure.","The article reports no significant impact on evaluation metrics but does not provide the evaluation results.","The article describes its systems and reported costs but does not establish how the reported efficiencies affect users in practice."],"mediaId":null,"correctionReason":null,"id":"16f440734294b4a6df0a448880023514f5580a037fb1f6cc13eb0c3eb26aaefe","publicationId":"2288d8bf-5716-938e-17c4-3817659a2748","version":1,"sourceUrl":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","sourceTitle":"Optimizing AI Inference at Character.AI","publisher":"Character.AI","publishedAt":"2024-06-21T00:13:00.000Z","originalPublished":{"value":"2024-06-21T00:13:00.000Z","precision":"instant","basis":"backfill:article_published"},"firstObservedAt":"2026-10-30T23:04:17.309Z","createdAt":"2026-11-18T12:18:13.524Z","historical":true,"entityIds":[],"topicIds":["investment-booms-and-busts"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"0c2fd3a6746ad8aaf1fa8555a4ea18ec14699bb506d47d98f1ba9e786a0a7b3f","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-30T23:04:14.227Z"},"provenance":{"textHash":"d005739845061d7a53d23bd23c2583240259b6ea65e5eaee132c31e2c200584e","observationId":"928310c6b870242579b363056c8b6124ab735456749968d6fc4dac9d89f7a122","retrievedAt":"2026-10-30T23:06:07.325Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"d255f87e-9904-e4b2-2664-ef7c45c6052f"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"6a2a7fa3-c336-c812-dddb-554c28a200ce"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T12:13:49.747Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/character-ai-describes-how-it-reduces-inference-costs-16f44073","formats":{"html":"/readings/character-ai-describes-how-it-reduces-inference-costs-16f44073","markdown":"/readings/character-ai-describes-how-it-reduces-inference-costs-16f44073.md","json":"/readings/character-ai-describes-how-it-reduces-inference-costs-16f44073.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-inference","name":"Model inference","explanation":"Using a trained model to produce a prediction or response to new input.","referenceUrl":"https://developers.google.com/machine-learning/glossary#inference"},{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-quantization","name":"Quantization","explanation":"Storing a model's numbers at lower precision so it uses less memory and runs faster, sometimes at a cost in accuracy.","referenceUrl":"https://developers.google.com/machine-learning/glossary#quantization"}],"entities":[{"id":"figure","name":"Figure","kind":"organization","context":"Figure develops humanoid robots and publishes its own tests of how they perform physical tasks.","referenceUrls":["https://www.figure.ai/news","https://www.figure.ai/news/helix-2-5-zero-shot-30-home-generalization"],"path":"/entities/figure","json":"/entities/figure.json","markdown":"/entities/figure.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"16f440734294b4a6df0a448880023514f5580a037fb1f6cc13eb0c3eb26aaefe","revision":"2288d8bf-5716-938e-17c4-3817659a2748","revisionKind":"publication-id","title":"Character.AI describes how it reduces inference costs","url":"https://escapemeridian.com/readings/character-ai-describes-how-it-reduces-inference-costs-16f44073"},"source":{"type":"source","id":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","revision":"d005739845061d7a53d23bd23c2583240259b6ea65e5eaee132c31e2c200584e","revisionKind":"sha256-source-text","title":"Optimizing AI Inference at Character.AI","url":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"16f440734294b4a6df0a448880023514f5580a037fb1f6cc13eb0c3eb26aaefe","title":"Character.AI describes how it reduces inference costs","argument":"Character.AI’s article, published June 21, 2024, argues that efficient inference—the process by which large language models generate replies—is essential to serving those models at scale. Character.AI says it reduces the memory needed for attention through changes to its model architecture, reuses cached attention data between chat turns, and trains and serves models using int8 quantization. It reports that its system achieves a 95% cache rate and that serving costs have fallen by a factor of 33 compared with late 2022. Character.AI argues that these changes make it easier and less costly to serve large language models to a global audience, connecting this work to its aim of integrating AI into daily life. The reported cost reduction has no stated measurement method or detailed comparison, and the article does not provide the evaluation results behind its claim about attention-horizon changes.","topicIds":["investment-booms-and-busts"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 30, 2026; originally published June 21, 2024. The article reports Character.AI’s own serving-cost reduction but gives no measurement method, baseline details, or comparison results for that figure. The article reports no significant impact on evaluation metrics but does not provide the evaluation results. The article describes its systems and reported costs but does not establish how the reported efficiencies affect users in practice.","speaker":{"name":"Character.AI","roleAtSource":"Publisher"},"source":{"title":"Optimizing AI Inference at Character.AI","publisher":"Character.AI","format":"Article","url":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","publishedAt":"2024-06-21T00:13:00.000Z","dateBasis":"Date stated on the publisher’s page."},"evidence":[{"text":"Character.AI says it reduces the KV cache size by more than 20X using techniques in its serving stack.","excerpt":"We use the following techniques to reduce KV cache size by more than 20X without regressing quality.","url":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","relationship":"Source-stated"},{"text":"Character.AI says its inter-turn caching system achieves a 95% cache rate.","excerpt":"Our system achieves a 95% cache rate, further reducing inference cost.","url":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","relationship":"Source-stated"},{"text":"Character.AI reports that it reduced serving costs by a factor of 33 compared with late 2022.","excerpt":"We have reduced serving costs by a factor of 33 compared to when we began in late 2022.","url":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","relationship":"Source-stated"},{"text":"Character.AI says its evaluation metrics were not significantly affected by reducing the attention horizon on most attention layers.","excerpt":"We found that reducing attention horizon to 1024 on most attention layers does not have a significant impact on evaluation metrics, including the long context needle-in-haystack benchmark.","url":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would the reported cost reduction compare under a stated, independently checkable measurement method?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Character.AI describes how it reduces inference costs","summary":"Character.AI’s article, published June 21, 2024, argues that efficient inference—the process by which large language models generate replies—is essential to serving those models at scale. Character.AI says it reduces the memory needed for attention through changes to its model architecture, reuses cached attention data between chat turns, and trains and serves models using int8 quantization. It reports that its system achieves a 95% cache rate and that serving costs have fallen by a factor of 33 compared with late 2022. Character.AI argues that these changes make it easier and less costly to serve large language models to a global audience, connecting this work to its aim of integrating AI into daily life. The reported cost reduction has no stated measurement method or detailed comparison, and the article does not provide the evaluation results behind its claim about attention-horizon changes.","question":"How would the reported cost reduction compare under a stated, independently checkable measurement method?","claims":[{"text":"Character.AI says it reduces the KV cache size by more than 20X using techniques in its serving stack.","excerpt":"We use the following techniques to reduce KV cache size by more than 20X without regressing quality.","support":"source-stated"},{"text":"Character.AI says its inter-turn caching system achieves a 95% cache rate.","excerpt":"Our system achieves a 95% cache rate, further reducing inference cost.","support":"source-stated"},{"text":"Character.AI reports that it reduced serving costs by a factor of 33 compared with late 2022.","excerpt":"We have reduced serving costs by a factor of 33 compared to when we began in late 2022.","support":"source-stated"},{"text":"Character.AI says its evaluation metrics were not significantly affected by reducing the attention horizon on most attention layers.","excerpt":"We found that reducing attention horizon to 1024 on most attention layers does not have a significant impact on evaluation metrics, including the long context needle-in-haystack benchmark.","support":"source-stated"}],"limitations":["Text as retrieved on October 30, 2026; originally published June 21, 2024.","The article reports Character.AI’s own serving-cost reduction but gives no measurement method, baseline details, or comparison results for that figure.","The article reports no significant impact on evaluation metrics but does not provide the evaluation results.","The article describes its systems and reported costs but does not establish how the reported efficiencies affect users in practice."],"mediaId":null,"correctionReason":null,"id":"16f440734294b4a6df0a448880023514f5580a037fb1f6cc13eb0c3eb26aaefe","publicationId":"2288d8bf-5716-938e-17c4-3817659a2748","version":1,"sourceUrl":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","sourceTitle":"Optimizing AI Inference at Character.AI","publisher":"Character.AI","publishedAt":"2024-06-21T00:13:00.000Z","originalPublished":{"value":"2024-06-21T00:13:00.000Z","precision":"instant","basis":"backfill:article_published"},"firstObservedAt":"2026-10-30T23:04:17.309Z","createdAt":"2026-11-18T12:18:13.524Z","historical":true,"entityIds":[],"topicIds":["investment-booms-and-busts"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"0c2fd3a6746ad8aaf1fa8555a4ea18ec14699bb506d47d98f1ba9e786a0a7b3f","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-30T23:04:14.227Z"},"provenance":{"textHash":"d005739845061d7a53d23bd23c2583240259b6ea65e5eaee132c31e2c200584e","observationId":"928310c6b870242579b363056c8b6124ab735456749968d6fc4dac9d89f7a122","retrievedAt":"2026-10-30T23:06:07.325Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"d255f87e-9904-e4b2-2664-ef7c45c6052f"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"6a2a7fa3-c336-c812-dddb-554c28a200ce"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T12:13:49.747Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"2288d8bf-5716-938e-17c4-3817659a2748","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The text is a single Character.AI blog post that makes its own argument about its inference stack. The reading consistently attributes the architecture, caching, quantization and cost claims to Character.AI as its own statements.","excerpt":"In this blog post, we share some of the techniques and optimizations we have developed over the past two years and recently employed."},{"dimension":"summary","status":"uncertain","explanation":"Most of the reading is clear, and inference, LLM and quantization are defined. However, the central mechanism rests on terms a non-specialist cannot follow: attention, the KV cache (its first claim), cache rate and attention horizon. None is explained or defined. Smallest repair: add term-library entries for attention, KV cache and attention horizon. Alternatively, add one plain clause saying the KV cache stores intermediate data from earlier text so the model need not recompute it. The 13.5X comparison with commercial APIs is omitted, but that omission is acceptable.","excerpt":"The key bottleneck of LLM inference throughput is the size of the cache of attention keys and values (KV)."},{"dimension":"date","status":"pass","explanation":"The originalPublished instant 2024-06-21T00:13Z falls on the UTC calendar day June 21, 2024, which matches the reading. The evidence is a publisher page, not an archive copy, so no archive disclosure is needed. The page was modified on 2025-08-07, but the cited passages are integral to the article's dated argument, such as the past two years since late 2022. There is no hindsight framing, and no forecast is presented as an outcome.","excerpt":"We have reduced serving costs by a factor of 33 compared to when we began in late 2022."},{"dimension":"support","status":"pass","explanation":"Each claim matches its verbatim excerpt and is framed as source-stated: the 20X KV reduction, the 95% cache rate, the 33X cost reduction, and no significant metric impact from the shorter attention horizon. The limitations accurately note that no measurement method or evaluation results are given. The question follows from that gap. A later page edit exists, but these passages appear plainly part of the original article.","excerpt":"Efficient inference is crucial for scaling AI systems and integrating them seamlessly into our daily lives."},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified candidate media items were supplied, so leaving the reading without media is appropriate.","excerpt":null}],"createdAt":"2026-11-18T12:13:49.747Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}