# BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

[Canonical reading](https://escapemeridian.com/readings/bigcodebench-benchmarking-code-generation-with-diverse-function-8e29b0d2)

The authors of BigCodeBench argue that many existing evaluations miss a hard part of practical code generation: using multiple tools while following complex instructions. They introduce a benchmark in which language models must call functions from 139 libraries across 7 domains to complete 1,140 fine-grained tasks. Each task has an average of 5.6 test cases and 99% branch coverage, and BigCodeBench-Instruct presents the tasks as short natural-language instructions rather than original docstrings. In their evaluation of 60 language models, scores reach 60%, compared with reported human performance of 97%. The authors interpret this gap as evidence that models still struggle to follow complex instructions and use function calls precisely. The benchmark makes those combined demands more visible than evaluations limited to short algorithmic problems or standalone function calls. Its results concern performance on these benchmark tasks; the supplied abstract does not establish how that performance translates to software work outside the benchmark. The paper’s central unresolved issue is what capabilities or task conditions account for the reported gap between model and human performance.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Jun 22, 2024 (2024-06-22).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-16T06:33:23.533Z. First observed: 2026-11-16T05:03:22.917Z. Retrieved: 2026-11-16T06:32:31.221Z.

- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://arxiv.org/abs/2406.15877)

## Limits

The supplied evidence is the arXiv page text and abstract; it does not provide the evaluation methodology or per-model results needed to examine how the reported scores were obtained. The abstract reports benchmark scores and human performance, but does not establish how performance on these tasks translates to software work outside the benchmark.


## Open question

What capabilities or task conditions account for the reported gap between model and human performance on BigCodeBench?


## Source-stated

The authors introduce BigCodeBench to evaluate language models on tasks requiring multiple function calls from diverse libraries and domains.

> we introduce BigCodeBench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks.

- [Supporting source](https://arxiv.org/abs/2406.15877)

## Source-stated

The authors report that, in their evaluation, language-model scores reached 60%, below the human performance of 97%.

> Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%.

- [Supporting source](https://arxiv.org/abs/2406.15877)

## Source-stated

The authors argue that the results indicate continued difficulty following complex instructions while using function calls precisely.

> LLMs are not yet capable of following complex instructions to use function calls precisely

- [Supporting source](https://arxiv.org/abs/2406.15877)

## Escape Meridian interpretation

Escape Meridian interpretation: BigCodeBench brings together instruction-following and tool use, making a combined challenge visible beyond evaluations limited to short algorithmic tasks or standalone calls.

> the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls.

- [Supporting source](https://arxiv.org/abs/2406.15877)

## Source and contributor context

Zhuo, Terry Yue, Vu, Minh Chien, Chim, Jenny, Hu, Han, Yu, Wenhao, Widyasari, Ratnadira, Yusuf, Imam Nur Bani, Zhan, Haolan, He, Junda, Paul, Indraneil, Brunner, Simon, Gong, Chen, Hoang, Thong, Zebaze, Armel Randy, Hong, Xiaoheng, Li, Wen-Ding, Kaddour, Jean, Xu, Ming, Zhang, Zhihan, Yadav, Prateek, Jain, Naman, Gu, Alex, Cheng, Zhoujun, Liu, Jiawei, Liu, Qian, Wang, Zijian, Hui, Binyuan, Muennighoff, Niklas, Lo, David, Fried, Daniel, Du, Xiaoning, de Vries, Harm, Von Werra, Leandro · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: 43a40f94-9562-47d6-8a94-5ebb24ffc4d0.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T06:33:23.533Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 726611484a9977d573edd21dd8bc88ab57f1240bdf0d6969c2b8a3aa1e1c6e42. Source text hash: d44616b70f3f8d697aab8c01931102c7d163d4e7467a05e730cc19310f1b6d8f.


## Revision and source-check history

### Version 1 — 2026-11-16T06:33:23.533Z

The authors of BigCodeBench argue that many existing evaluations miss a hard part of practical code generation: using multiple tools while following complex instructions. They introduce a benchmark in which language models must call functions from 139 libraries across 7 domains to complete 1,140 fine-grained tasks. Each task has an average of 5.6 test cases and 99% branch coverage, and BigCodeBench-Instruct presents the tasks as short natural-language instructions rather than original docstrings. In their evaluation of 60 language models, scores reach 60%, compared with reported human performance of 97%. The authors interpret this gap as evidence that models still struggle to follow complex instructions and use function calls precisely. The benchmark makes those combined demands more visible than evaluations limited to short algorithmic problems or standalone function calls. Its results concern performance on these benchmark tasks; the supplied abstract does not establish how that performance translates to software work outside the benchmark. The paper’s central unresolved issue is what capabilities or task conditions account for the reported gap between model and human performance.

## Original source

arXiv cs.AI · Article

Original source date: Jun 22, 2024 (2024-06-22).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-16T06:33:23.533Z. First observed: 2026-11-16T05:03:22.917Z. Retrieved: 2026-11-16T06:32:31.221Z.

- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://arxiv.org/abs/2406.15877)

## Limits

The supplied evidence is the arXiv page text and abstract; it does not provide the evaluation methodology or per-model results needed to examine how the reported scores were obtained. The abstract reports benchmark scores and human performance, but does not establish how performance on these tasks translates to software work outside the benchmark.


## Open question

What capabilities or task conditions account for the reported gap between model and human performance on BigCodeBench?


## Source-stated

The authors introduce BigCodeBench to evaluate language models on tasks requiring multiple function calls from diverse libraries and domains.

> we introduce BigCodeBench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks.

- [Supporting source](https://arxiv.org/abs/2406.15877)

## Source-stated

The authors report that, in their evaluation, language-model scores reached 60%, below the human performance of 97%.

> Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%.

- [Supporting source](https://arxiv.org/abs/2406.15877)

## Source-stated

The authors argue that the results indicate continued difficulty following complex instructions while using function calls precisely.

> LLMs are not yet capable of following complex instructions to use function calls precisely

- [Supporting source](https://arxiv.org/abs/2406.15877)

## Escape Meridian interpretation

Escape Meridian interpretation: BigCodeBench brings together instruction-following and tool use, making a combined challenge visible beyond evaluations limited to short algorithmic tasks or standalone calls.

> the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls.

- [Supporting source](https://arxiv.org/abs/2406.15877)

## Source and contributor context

Zhuo, Terry Yue, Vu, Minh Chien, Chim, Jenny, Hu, Han, Yu, Wenhao, Widyasari, Ratnadira, Yusuf, Imam Nur Bani, Zhan, Haolan, He, Junda, Paul, Indraneil, Brunner, Simon, Gong, Chen, Hoang, Thong, Zebaze, Armel Randy, Hong, Xiaoheng, Li, Wen-Ding, Kaddour, Jean, Xu, Ming, Zhang, Zhihan, Yadav, Prateek, Jain, Naman, Gu, Alex, Cheng, Zhoujun, Liu, Jiawei, Liu, Qian, Wang, Zijian, Hui, Binyuan, Muennighoff, Niklas, Lo, David, Fried, Daniel, Du, Xiaoning, de Vries, Harm, Von Werra, Leandro · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Benchmark

A defined set of tests used to compare systems. Results depend on the tasks and measures selected.

- [Definition reference](https://arxiv.org/abs/2211.09110)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: 43a40f94-9562-47d6-8a94-5ebb24ffc4d0.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-16T06:33:23.533Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 726611484a9977d573edd21dd8bc88ab57f1240bdf0d6969c2b8a3aa1e1c6e42. Source text hash: d44616b70f3f8d697aab8c01931102c7d163d4e7467a05e730cc19310f1b6d8f.


### uncertain — 2026-11-16T06:33:23.533Z — anthropic/claude-opus-5.5

Publication: 43a40f94-9562-47d6-8a94-5ebb24ffc4d0

attribution — pass: The draft attributes the benchmark, scores and interpretation to the BigCodeBench authors, matching the arXiv page. The 60% and 97% figures are framed as the authors' reported results, not as established fact.

> Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely

summary — uncertain: The core reading is faithful and broadly understandable. However, the closing sentence says this is the paper's central unresolved issue, but the source only says the results 'underscore the need for further advancements.' Smallest repair: present it as an open question raised by the results, not one the paper names. The test-case phrasing also shifts slightly: the source gives 5.6 test cases per task with an average branch coverage of 99%. 'Branch coverage' and 'docstrings' are left unexplained; they need term-library tooltips, but they do not block understanding.

> The results underscore the need for further advancements in this area.

date — pass: originalPublished 2024-06-22 matches v1 submission on 22 Jun 2024, not the 13 Nov 2026 v5 revision or the feed date.

> \[Submitted on 22 Jun 2024 (v1), last revised 13 Nov 2026 (this version, v5)\]

support — uncertain: The claims list labels the 'combined demands more visible' point as Escape Meridian interpretation. In the summary, though, the same sentence ('The benchmark makes those combined demands more visible...') appears as an unlabeled assertion. It is close to the authors' motivation, but the benchmark's effect on visibility is an inference, not a demonstrated result. Smallest repair: label that sentence as interpretation or attribute it to the authors' aim. The other claims and limitations are grounded in the source and avoid overstating real-world reliability.

> the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls.

media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct way to access the source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/8e29b0d2fd7092dd52c2cb984c088032af0dccc51d374fb7a9838657dbfe4cc3/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/bigcodebench-benchmarking-code-generation-with-diverse-function-8e29b0d2.json)
