Burdisso, Sergio, Villatoro-Tello, Esaú, Sánchez-Cortés, Dairazalia, Labrak, Yanis, Luque, Jordi, Cernak, Milos, Motlicek, Petr · Oct 15, 2026
DialFork evaluates procedural hallucination in conversational agents
Markdown · JSON · Original source
The authors of DialFork argue that conversational agents can stay fluent and factually plausible while departing from the intended process of a conversation. They call this failure procedural , distinguishing it from errors of factual content. Their forks conformant dialogues—dialogues that follow the intended process—and injects one of four types of procedural hallucination. The authors report 36,278 dialogues across 24 tasks in 12 domains, with construction validated by 30 human annotators. They set out three evaluation tasks: comparing sets of dialogues, assessing a single dialogue, and identifying the first turn that goes off-track. For the first two, they propose FlowKL and FlowScore, metrics intended to compare how dialogues unfold rather than just what they contain. The authors report that FlowScore significantly outperforms all baselines across seven encoders on the dialogue-level task. Their central point is that semantic similarity—the similarity of meaning or content—does not stand in for procedural similarity: a response may sound relevant yet fail to follow the required conversational sequence. That distinction could matter when monitoring deployed agents or designing real- guardrails, though the abstract does not establish how well the benchmark or metrics work in deployment. An open question is how FlowScore performs on real dialogues
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 15, 2026 (2026-10-15).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T00:41:27.050Z. First observed: 2026-10-16T04:24:52.325Z. Retrieved: 2026-11-03T04:31:09.864Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the abstract; it does not provide detailed construction, metric definitions, or baseline results. The abstract reports benchmark results but does not establish how the metrics perform on real-world deployed conversations.
Open question
How does FlowScore perform on real dialogues, beyond the ?
Source-stated
The authors define procedural hallucination as fluent, factually plausible utterances that deviate from the intended conversational process.
procedural hallucination, in which an agent produces individually fluent and factually plausible utterances while deviating from the intended conversational process
Source-stated
DialFork is built by forking conformant dialogues and injecting one of four hallucination types; it contains 36,278 dialogues across 24 tasks in 12 domains, and 30 human annotators validate its construction.
yielding 36,278 dialogues across 24 tasks in 12 domains; 30 human annotators validate the construction
Source-stated
The authors define three evaluation tasks: measuring deviation across sets of dialogues, measuring deviation in one dialogue, and locating the first turn where a dialogue goes off-track.
how far a set of dialogues deviates from conformant ones (monitoring deployed agents), how far a single dialogue does (procedural retrieval), and at which turn a dialogue first goes off-track (real-time guardrails)
Source-stated
For the first two tasks, the authors propose FlowKL and FlowScore, which compare dialogues by how they unfold rather than their content.
we propose dialog-flow-aware metrics, FlowKL and FlowScore, that compare dialogues by how they unfold rather than their content
Source-stated
The authors report that FlowScore significantly outperforms all baselines across seven encoders at the dialogue level.
At the dialogue level, FlowScore significantly outperforms all baselines across seven encoders
Source-stated
The authors argue that semantic similarity is not a substitute for procedural similarity.
semantic similarity is no substitute for procedural similarity.
Source and contributor context
Burdisso, Sergio, Villatoro-Tello, Esaú, Sánchez-Cortés, Dairazalia, Labrak, Yanis, Luque, Jordi, Cernak, Milos, Motlicek, Petr · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Hallucination
When an AI system states something false or made up as if it were true.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 1479c9a3-1f1b-40d5-8a8e-bfa9c8088329.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T00:41:27.050Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 4393ab5641ccd2cd9c71dab8aa32aadb4fe929e04e43db1d3027d9d486497356. Source text hash: 41e179c05fce172375a838318cba3837457df65f26cff02e82d1cf4631bd478a.
Revision and source-check history
Version 1 · 2026-11-11T00:41:27.050Z
DialFork evaluates procedural hallucination in conversational agents
The authors of DialFork argue that conversational agents can stay fluent and factually plausible while departing from the intended process of a conversation. They call this failure procedural , distinguishing it from errors of factual content. Their forks conformant dialogues—dialogues that follow the intended process—and injects one of four types of procedural hallucination. The authors report 36,278 dialogues across 24 tasks in 12 domains, with construction validated by 30 human annotators. They set out three evaluation tasks: comparing sets of dialogues, assessing a single dialogue, and identifying the first turn that goes off-track. For the first two, they propose FlowKL and FlowScore, metrics intended to compare how dialogues unfold rather than just what they contain. The authors report that FlowScore significantly outperforms all baselines across seven encoders on the dialogue-level task. Their central point is that semantic similarity—the similarity of meaning or content—does not stand in for procedural similarity: a response may sound relevant yet fail to follow the required conversational sequence. That distinction could matter when monitoring deployed agents or designing real- guardrails, though the abstract does not establish how well the benchmark or metrics work in deployment. An open question is how FlowScore performs on real dialogues
Original source
cs.AI · Article
Source details
Original source date: Oct 15, 2026 (2026-10-15).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T00:41:27.050Z. First observed: 2026-10-16T04:24:52.325Z. Retrieved: 2026-11-03T04:31:09.864Z.
Limits
The supplied evidence is the abstract; it does not provide detailed construction, metric definitions, or baseline results. The abstract reports benchmark results but does not establish how the metrics perform on real-world deployed conversations.
Open question
How does FlowScore perform on real dialogues, beyond the ?
Source-stated
The authors define procedural hallucination as fluent, factually plausible utterances that deviate from the intended conversational process.
procedural hallucination, in which an agent produces individually fluent and factually plausible utterances while deviating from the intended conversational process
Source-stated
DialFork is built by forking conformant dialogues and injecting one of four hallucination types; it contains 36,278 dialogues across 24 tasks in 12 domains, and 30 human annotators validate its construction.
yielding 36,278 dialogues across 24 tasks in 12 domains; 30 human annotators validate the construction
Source-stated
The authors define three evaluation tasks: measuring deviation across sets of dialogues, measuring deviation in one dialogue, and locating the first turn where a dialogue goes off-track.
how far a set of dialogues deviates from conformant ones (monitoring deployed agents), how far a single dialogue does (procedural retrieval), and at which turn a dialogue first goes off-track (real-time guardrails)
Source-stated
For the first two tasks, the authors propose FlowKL and FlowScore, which compare dialogues by how they unfold rather than their content.
we propose dialog-flow-aware metrics, FlowKL and FlowScore, that compare dialogues by how they unfold rather than their content
Source-stated
The authors report that FlowScore significantly outperforms all baselines across seven encoders at the dialogue level.
At the dialogue level, FlowScore significantly outperforms all baselines across seven encoders
Source-stated
The authors argue that semantic similarity is not a substitute for procedural similarity.
semantic similarity is no substitute for procedural similarity.
Source and contributor context
Burdisso, Sergio, Villatoro-Tello, Esaú, Sánchez-Cortés, Dairazalia, Labrak, Yanis, Luque, Jordi, Cernak, Milos, Motlicek, Petr · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Hallucination
When an AI system states something false or made up as if it were true.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 1479c9a3-1f1b-40d5-8a8e-bfa9c8088329.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T00:41:27.050Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 4393ab5641ccd2cd9c71dab8aa32aadb4fe929e04e43db1d3027d9d486497356. Source text hash: 41e179c05fce172375a838318cba3837457df65f26cff02e82d1cf4631bd478a.
uncertain · 2026-11-11T00:41:27.050Z · anthropic/claude-opus-5.5
Publication: 1479c9a3-1f1b-40d5-8a8e-bfa9c8088329
attribution — pass: The reading consistently attributes the claims, definitions, benchmark and results to the DialFork authors as an arXiv paper. It does not present the authors' assertions as established fact. Names are omitted, but that does not misattribute anything.
Authors:Sergio Burdisso, Esaú Villatoro-Tello, Dairazalia Sánchez-Cortés, Yanis Labrak, Jordi Luque, Milos Cernak, Petr Motlicek
summary — uncertain: The core meaning is faithful, but the summary's final sentence is cut off ('An open question is how FlowScore performs on real dialogues') with no ending, and it duplicates the question. Smallest repair: complete or remove that sentence. 'Seven encoders' and 'baselines' are left unexplained; add plain-language glosses or term-library tooltips. 'Rather than just what they contain' adds 'just', softening the source's 'rather than their content'; restore the source wording.
compare dialogues by how they unfold rather than their content
date — pass: originalPublished 2026-10-15 matches the v1 submission date. The v5 revision on 30 Oct and the 2 Nov feed date are later events and are correctly not used.
[Submitted on 15 Oct 2026 (v1), last revised 30 Oct 2026 (this version, v5)]
support — pass: The figures (36,278 dialogues, 24 tasks, 12 domains, 30 annotators, four hallucination types, seven encoders) match the abstract. The monitoring and guardrail uses come from the source's own task framing. The reading explicitly says deployment performance is not established. The open question about real-world performance follows from the evidence and is not a disguised claim.
how far a set of dialogues deviates from conformant ones (monitoring deployed agents)
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
When an AI system states something false or made up as if it were true.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceWhen an AI system states something false or made up as if it were true.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference