{"id":"f18f11f498f501a664b0545ee60388d7470194f4d8502164a0f4daa171ee4245","title":"arXiv cs.AI: RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks","argument":"The authors of RealEmbedBench argue that LLMs’ ability to handle application-layer embedded software tasks remains underexplored, despite their performance on general coding tasks. They introduce a benchmark of 5,142 samples covering code generation and completion, state machine modeling and implementation, and cross-OS code translation, and evaluate six representative LLMs. Their results vary substantially across tasks and models: project context, especially the local code window, significantly affects code generation and completion, while models that perform reasonably well at state machine modeling struggle to turn those models into executable code. Translation performance also depends on direction; the average pass rate for translation from Harmony to RT-Thread is 0.1667. The authors say developers should consider local code context and choose models according to translation direction. They also argue that more real-world state machine implementation data in training could narrow the gap between modeling and executable code. The abstract reports benchmark findings, not how well they predict performance on embedded software work outside the benchmark.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract gives no numeric estimate of how much local code context affects code generation and completion. The abstract describes results from six representative LLMs but does not name them or establish how well the benchmark results generalize to embedded software work outside the benchmark.","speaker":{"name":"Huang, Yujun, Zhang, Yuwei, Zhou, Yi, Wu, Shanshan, Chen, Xiang, Gong, Lina","roleAtSource":"Article author"},"source":{"title":"RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.11469","publishedAt":"2026-11-08","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors present RealEmbedBench as a benchmark with 5,142 samples across three application-layer embedded software tasks.","excerpt":"We introduce RealEmbedBench, a benchmark of 5,142 samples covering three application-layer tasks: code generation and completion, state machine modeling and implementation, and cross-OS code translation.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"},{"text":"The authors evaluated six representative LLMs and report substantial differences in performance across tasks and models.","excerpt":"We evaluate six representative LLMs and find substantial performance differences across tasks and models.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"},{"text":"The authors report that project context, especially the local code window, significantly affects code generation and completion; they do not quantify the effect in the abstract.","excerpt":"Project context, especially the local code window, significantly affects code generation and completion.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"},{"text":"The authors report a gap between state machine modeling and implementing those models as executable code.","excerpt":"Models perform reasonably well at state machine modeling but struggle to translate these models into executable code.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"},{"text":"The authors report that cross-OS translation performance varies by direction, with an average pass rate of 0.1667 from Harmony to RT-Thread.","excerpt":"Translation from Harmony to RT-Thread is particularly challenging, with an average pass rate of 0.1667.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"},{"text":"The authors say their benchmark findings suggest developers should consider local code context and select models according to translation direction.","excerpt":"These findings suggest that developers should consider local code context when using LLMs for code generation and completion and select models according to the translation direction.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"},{"text":"The authors propose that more real-world state machine implementation data in LLM training could narrow the gap between modeling and executable code.","excerpt":"They also motivate the use of more real-world state machine implementation data in LLM training to narrow the gap between modeling and executable code.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.11469","checkedAt":"2026-11-10T15:22:09.764Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much does the local code window change code generation and completion performance in RealEmbedBench?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks","summary":"The authors of RealEmbedBench argue that LLMs’ ability to handle application-layer embedded software tasks remains underexplored, despite their performance on general coding tasks. They introduce a benchmark of 5,142 samples covering code generation and completion, state machine modeling and implementation, and cross-OS code translation, and evaluate six representative LLMs. Their results vary substantially across tasks and models: project context, especially the local code window, significantly affects code generation and completion, while models that perform reasonably well at state machine modeling struggle to turn those models into executable code. Translation performance also depends on direction; the average pass rate for translation from Harmony to RT-Thread is 0.1667. The authors say developers should consider local code context and choose models according to translation direction. They also argue that more real-world state machine implementation data in training could narrow the gap between modeling and executable code. The abstract reports benchmark findings, not how well they predict performance on embedded software work outside the benchmark.","question":"How much does the local code window change code generation and completion performance in RealEmbedBench?","claims":[{"text":"The authors present RealEmbedBench as a benchmark with 5,142 samples across three application-layer embedded software tasks.","excerpt":"We introduce RealEmbedBench, a benchmark of 5,142 samples covering three application-layer tasks: code generation and completion, state machine modeling and implementation, and cross-OS code translation.","support":"source-stated"},{"text":"The authors evaluated six representative LLMs and report substantial differences in performance across tasks and models.","excerpt":"We evaluate six representative LLMs and find substantial performance differences across tasks and models.","support":"source-stated"},{"text":"The authors report that project context, especially the local code window, significantly affects code generation and completion; they do not quantify the effect in the abstract.","excerpt":"Project context, especially the local code window, significantly affects code generation and completion.","support":"source-stated"},{"text":"The authors report a gap between state machine modeling and implementing those models as executable code.","excerpt":"Models perform reasonably well at state machine modeling but struggle to translate these models into executable code.","support":"source-stated"},{"text":"The authors report that cross-OS translation performance varies by direction, with an average pass rate of 0.1667 from Harmony to RT-Thread.","excerpt":"Translation from Harmony to RT-Thread is particularly challenging, with an average pass rate of 0.1667.","support":"source-stated"},{"text":"The authors say their benchmark findings suggest developers should consider local code context and select models according to translation direction.","excerpt":"These findings suggest that developers should consider local code context when using LLMs for code generation and completion and select models according to the translation direction.","support":"source-stated"},{"text":"The authors propose that more real-world state machine implementation data in LLM training could narrow the gap between modeling and executable code.","excerpt":"They also motivate the use of more real-world state machine implementation data in LLM training to narrow the gap between modeling and executable code.","support":"source-stated"}],"limitations":["The abstract gives no numeric estimate of how much local code context affects code generation and completion.","The abstract describes results from six representative LLMs but does not name them or establish how well the benchmark results generalize to embedded software work outside the benchmark."],"mediaId":"ff8d9e19b5cbd5c1cd4f06c243a82485e17bbabf5ea1c60bc7ab3e15a68b584f","correctionReason":null,"id":"f18f11f498f501a664b0545ee60388d7470194f4d8502164a0f4daa171ee4245","publicationId":"7a913a01-b6af-4c62-8582-dde7db0af81d","version":1,"sourceUrl":"https://arxiv.org/abs/2611.11469","sourceTitle":"RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-08","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:36:44.603Z","createdAt":"2026-11-10T14:18:03.308Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"ff8d9e19b5cbd5c1cd4f06c243a82485e17bbabf5ea1c60bc7ab3e15a68b584f","kind":"article","url":"https://arxiv.org/abs/2611.11469","sourceUrl":"https://arxiv.org/abs/2611.11469","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-10T13:31:56.313Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.11469","checkedAt":"2026-11-10T15:22:09.764Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"ff8d9e19b5cbd5c1cd4f06c243a82485e17bbabf5ea1c60bc7ab3e15a68b584f","confidence":0.81,"provider":"typesafe","model":"jev-1.13.0","inputHash":"b0cef31d3c6f4b65063aa4b0ee50c3e9e33e3c1046c4feae5d4a46832758184f","decidedAt":"2026-11-10T13:31:58.284Z"},"provenance":{"textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observationId":"5c3d1505e702e8763bf10511f1544abfd4f5c667d4c3efb02957bdaa267d7790","retrievedAt":"2026-11-10T13:31:56.002Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"3040849d-4c01-4bb3-863f-ace7c07ed688"},"reviewer":{"model":"gpt-6-luna","runId":"fbda59e5-f7fb-4a71-8abd-6ca410d04426"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-10T14:18:03.308Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Huang, Yujun","url":null,"kind":"unknown"},{"name":"Zhang, Yuwei","url":null,"kind":"unknown"},{"name":"Zhou, Yi","url":null,"kind":"unknown"},{"name":"Wu, Shanshan","url":null,"kind":"unknown"},{"name":"Chen, Xiang","url":null,"kind":"unknown"},{"name":"Gong, Lina","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Huang, Yujun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Huang, Yujun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11469","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:22:09.764Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","sourceUrl":"https://arxiv.org/abs/2611.11469","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z"}},{"actor":{"name":"Zhang, Yuwei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Yuwei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11469","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:22:09.764Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","sourceUrl":"https://arxiv.org/abs/2611.11469","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z"}},{"actor":{"name":"Zhou, Yi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Yi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11469","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:22:09.764Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","sourceUrl":"https://arxiv.org/abs/2611.11469","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z"}},{"actor":{"name":"Wu, Shanshan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Shanshan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11469","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:22:09.764Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","sourceUrl":"https://arxiv.org/abs/2611.11469","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z"}},{"actor":{"name":"Chen, Xiang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chen, Xiang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11469","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:22:09.764Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","sourceUrl":"https://arxiv.org/abs/2611.11469","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z"}},{"actor":{"name":"Gong, Lina","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gong, Lina","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11469","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:22:09.764Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","sourceUrl":"https://arxiv.org/abs/2611.11469","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.11469","snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/f18f11f498f501a664b0545ee60388d7470194f4d8502164a0f4daa171ee4245/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-realembedbench-benchmarking-llms-on-real-world-f18f11f4","formats":{"html":"/readings/arxiv-cs-ai-realembedbench-benchmarking-llms-on-real-world-f18f11f4","markdown":"/readings/arxiv-cs-ai-realembedbench-benchmarking-llms-on-real-world-f18f11f4.md","json":"/readings/arxiv-cs-ai-realembedbench-benchmarking-llms-on-real-world-f18f11f4.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"f18f11f498f501a664b0545ee60388d7470194f4d8502164a0f4daa171ee4245","revision":"7a913a01-b6af-4c62-8582-dde7db0af81d","revisionKind":"publication-id","title":"arXiv cs.AI: RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks","url":"https://escapemeridian.com/readings/arxiv-cs-ai-realembedbench-benchmarking-llms-on-real-world-f18f11f4"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.11469","revision":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","revisionKind":"sha256-source-text","title":"RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks","url":"https://arxiv.org/abs/2611.11469"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"f18f11f498f501a664b0545ee60388d7470194f4d8502164a0f4daa171ee4245","title":"arXiv cs.AI: RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks","argument":"The authors of RealEmbedBench argue that LLMs’ ability to handle application-layer embedded software tasks remains underexplored, despite their performance on general coding tasks. They introduce a benchmark of 5,142 samples covering code generation and completion, state machine modeling and implementation, and cross-OS code translation, and evaluate six representative LLMs. Their results vary substantially across tasks and models: project context, especially the local code window, significantly affects code generation and completion, while models that perform reasonably well at state machine modeling struggle to turn those models into executable code. Translation performance also depends on direction; the average pass rate for translation from Harmony to RT-Thread is 0.1667. The authors say developers should consider local code context and choose models according to translation direction. They also argue that more real-world state machine implementation data in training could narrow the gap between modeling and executable code. The abstract reports benchmark findings, not how well they predict performance on embedded software work outside the benchmark.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract gives no numeric estimate of how much local code context affects code generation and completion. The abstract describes results from six representative LLMs but does not name them or establish how well the benchmark results generalize to embedded software work outside the benchmark.","speaker":{"name":"Huang, Yujun, Zhang, Yuwei, Zhou, Yi, Wu, Shanshan, Chen, Xiang, Gong, Lina","roleAtSource":"Article author"},"source":{"title":"RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.11469","publishedAt":"2026-11-08","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors present RealEmbedBench as a benchmark with 5,142 samples across three application-layer embedded software tasks.","excerpt":"We introduce RealEmbedBench, a benchmark of 5,142 samples covering three application-layer tasks: code generation and completion, state machine modeling and implementation, and cross-OS code translation.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"},{"text":"The authors evaluated six representative LLMs and report substantial differences in performance across tasks and models.","excerpt":"We evaluate six representative LLMs and find substantial performance differences across tasks and models.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"},{"text":"The authors report that project context, especially the local code window, significantly affects code generation and completion; they do not quantify the effect in the abstract.","excerpt":"Project context, especially the local code window, significantly affects code generation and completion.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"},{"text":"The authors report a gap between state machine modeling and implementing those models as executable code.","excerpt":"Models perform reasonably well at state machine modeling but struggle to translate these models into executable code.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"},{"text":"The authors report that cross-OS translation performance varies by direction, with an average pass rate of 0.1667 from Harmony to RT-Thread.","excerpt":"Translation from Harmony to RT-Thread is particularly challenging, with an average pass rate of 0.1667.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"},{"text":"The authors say their benchmark findings suggest developers should consider local code context and select models according to translation direction.","excerpt":"These findings suggest that developers should consider local code context when using LLMs for code generation and completion and select models according to the translation direction.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"},{"text":"The authors propose that more real-world state machine implementation data in LLM training could narrow the gap between modeling and executable code.","excerpt":"They also motivate the use of more real-world state machine implementation data in LLM training to narrow the gap between modeling and executable code.","url":"https://arxiv.org/abs/2611.11469","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.11469","checkedAt":"2026-11-10T15:22:09.764Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much does the local code window change code generation and completion performance in RealEmbedBench?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks","summary":"The authors of RealEmbedBench argue that LLMs’ ability to handle application-layer embedded software tasks remains underexplored, despite their performance on general coding tasks. They introduce a benchmark of 5,142 samples covering code generation and completion, state machine modeling and implementation, and cross-OS code translation, and evaluate six representative LLMs. Their results vary substantially across tasks and models: project context, especially the local code window, significantly affects code generation and completion, while models that perform reasonably well at state machine modeling struggle to turn those models into executable code. Translation performance also depends on direction; the average pass rate for translation from Harmony to RT-Thread is 0.1667. The authors say developers should consider local code context and choose models according to translation direction. They also argue that more real-world state machine implementation data in training could narrow the gap between modeling and executable code. The abstract reports benchmark findings, not how well they predict performance on embedded software work outside the benchmark.","question":"How much does the local code window change code generation and completion performance in RealEmbedBench?","claims":[{"text":"The authors present RealEmbedBench as a benchmark with 5,142 samples across three application-layer embedded software tasks.","excerpt":"We introduce RealEmbedBench, a benchmark of 5,142 samples covering three application-layer tasks: code generation and completion, state machine modeling and implementation, and cross-OS code translation.","support":"source-stated"},{"text":"The authors evaluated six representative LLMs and report substantial differences in performance across tasks and models.","excerpt":"We evaluate six representative LLMs and find substantial performance differences across tasks and models.","support":"source-stated"},{"text":"The authors report that project context, especially the local code window, significantly affects code generation and completion; they do not quantify the effect in the abstract.","excerpt":"Project context, especially the local code window, significantly affects code generation and completion.","support":"source-stated"},{"text":"The authors report a gap between state machine modeling and implementing those models as executable code.","excerpt":"Models perform reasonably well at state machine modeling but struggle to translate these models into executable code.","support":"source-stated"},{"text":"The authors report that cross-OS translation performance varies by direction, with an average pass rate of 0.1667 from Harmony to RT-Thread.","excerpt":"Translation from Harmony to RT-Thread is particularly challenging, with an average pass rate of 0.1667.","support":"source-stated"},{"text":"The authors say their benchmark findings suggest developers should consider local code context and select models according to translation direction.","excerpt":"These findings suggest that developers should consider local code context when using LLMs for code generation and completion and select models according to the translation direction.","support":"source-stated"},{"text":"The authors propose that more real-world state machine implementation data in LLM training could narrow the gap between modeling and executable code.","excerpt":"They also motivate the use of more real-world state machine implementation data in LLM training to narrow the gap between modeling and executable code.","support":"source-stated"}],"limitations":["The abstract gives no numeric estimate of how much local code context affects code generation and completion.","The abstract describes results from six representative LLMs but does not name them or establish how well the benchmark results generalize to embedded software work outside the benchmark."],"mediaId":"ff8d9e19b5cbd5c1cd4f06c243a82485e17bbabf5ea1c60bc7ab3e15a68b584f","correctionReason":null,"id":"f18f11f498f501a664b0545ee60388d7470194f4d8502164a0f4daa171ee4245","publicationId":"7a913a01-b6af-4c62-8582-dde7db0af81d","version":1,"sourceUrl":"https://arxiv.org/abs/2611.11469","sourceTitle":"RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-08","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:36:44.603Z","createdAt":"2026-11-10T14:18:03.308Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"ff8d9e19b5cbd5c1cd4f06c243a82485e17bbabf5ea1c60bc7ab3e15a68b584f","kind":"article","url":"https://arxiv.org/abs/2611.11469","sourceUrl":"https://arxiv.org/abs/2611.11469","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-10T13:31:56.313Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.11469","checkedAt":"2026-11-10T15:22:09.764Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"ff8d9e19b5cbd5c1cd4f06c243a82485e17bbabf5ea1c60bc7ab3e15a68b584f","confidence":0.81,"provider":"typesafe","model":"jev-1.13.0","inputHash":"b0cef31d3c6f4b65063aa4b0ee50c3e9e33e3c1046c4feae5d4a46832758184f","decidedAt":"2026-11-10T13:31:58.284Z"},"provenance":{"textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observationId":"5c3d1505e702e8763bf10511f1544abfd4f5c667d4c3efb02957bdaa267d7790","retrievedAt":"2026-11-10T13:31:56.002Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"3040849d-4c01-4bb3-863f-ace7c07ed688"},"reviewer":{"model":"gpt-6-luna","runId":"fbda59e5-f7fb-4a71-8abd-6ca410d04426"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-10T14:18:03.308Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Huang, Yujun","url":null,"kind":"unknown"},{"name":"Zhang, Yuwei","url":null,"kind":"unknown"},{"name":"Zhou, Yi","url":null,"kind":"unknown"},{"name":"Wu, Shanshan","url":null,"kind":"unknown"},{"name":"Chen, Xiang","url":null,"kind":"unknown"},{"name":"Gong, Lina","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Huang, Yujun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Huang, Yujun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11469","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:22:09.764Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","sourceUrl":"https://arxiv.org/abs/2611.11469","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z"}},{"actor":{"name":"Zhang, Yuwei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Yuwei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11469","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:22:09.764Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","sourceUrl":"https://arxiv.org/abs/2611.11469","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z"}},{"actor":{"name":"Zhou, Yi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Yi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11469","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:22:09.764Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","sourceUrl":"https://arxiv.org/abs/2611.11469","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z"}},{"actor":{"name":"Wu, Shanshan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Shanshan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11469","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:22:09.764Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","sourceUrl":"https://arxiv.org/abs/2611.11469","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z"}},{"actor":{"name":"Chen, Xiang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chen, Xiang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11469","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:22:09.764Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","sourceUrl":"https://arxiv.org/abs/2611.11469","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z"}},{"actor":{"name":"Gong, Lina","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gong, Lina","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.11469","locator":"head/meta[name=author|citation_author]"},"assertionId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:22:09.764Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","sourceUrl":"https://arxiv.org/abs/2611.11469","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.11469","snapshotId":"e8bece7e9aa3036b472c4758ee286eebcfa417abe1c789ff5eb720b68c90315a","contentSha256":"da05af6dc85c7a92c974232ea21c78556f293b427d48d18837a65112223c8e71","textHash":"84e9fb6243329b145b638e9b798713647c6e19eb0d8697db02670a3c7945ec06","observedAt":"2026-11-10T15:22:09.764Z","basis":"publisher-page-metadata","sourceLinks":45,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/f18f11f498f501a664b0545ee60388d7470194f4d8502164a0f4daa171ee4245/source"}}}],"audits":[{"publicationId":"7a913a01-b6af-4c62-8582-dde7db0af81d","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading consistently attributes the benchmark, results and recommendations to the RealEmbedBench authors and does not present them as established fact or as Escape Meridian's view. The arXiv cs.AI attribution matches the source page.","excerpt":"We introduce RealEmbedBench, a benchmark of 5,142 samples covering three application-layer tasks"},{"dimension":"summary","status":"uncertain","explanation":"The summary is faithful, but its central terms stay unexplained for a non-specialist: application-layer embedded software (software running on devices), state machine modeling, local code window, pass rate, and Harmony and RT-Thread as operating systems. It also never says plainly why this matters to a reader. Smallest repair: add short plain-language glosses, or term-library tooltips, for embedded software, state machine, Harmony and RT-Thread, without changing any claims. The question is a legitimate open issue that the abstract leaves unquantified.","excerpt":"Translation from Harmony to RT-Thread is particularly challenging, with an average pass rate of 0.1667."},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-11-08, which matches the v1 submission date on arXiv. The feed date of Nov 10 is correctly not used.","excerpt":"[Submitted on 8 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Every claim is quoted verbatim and labeled source-stated. The training-data and developer recommendations stay attributed as the authors' suggestions. The limitations correctly note that the abstract does not quantify the effect of local code context and does not name the six models. No benchmark score is turned into real-world reliability.","excerpt":"They also motivate the use of more real-world state machine implementation data in LLM training to narrow the gap between modeling and executable code."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper. It is the most direct way to reach the source.","excerpt":null}],"createdAt":"2026-11-10T14:18:03.308Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}