Choudhuri, Rudrajit, Bird, Christian, Badea, Carmen, Sarma, Anita · Nov 3, 2026
arXiv cs.AI: TACT: A Pragmatic Approach to Qualitative Analysis with an AI Council
Markdown · JSON · Original source
The authors present TACT, a procedure for using AI to analyze qualitative responses while leaving teams in control of interpretation and judgment. They frame it as an alternative to either leaving large collections of responses unanalyzed or accepting a ’s interpretation without that control. TACT-Lite uses the core AI council procedure; TACT-Full adds adversaries—models from other model families that critique each model’s output—and a decision model that surfaces high-uncertainty code assignments. In an evaluation on 256 Stack Overflow threads coded by other researchers, TACT’s candidate codebook achieved an F1 of 0.75–0.77, and its final coding achieved an F1 of 0.71–0.72 against those labels. On a held-out sample, the authors’ manual coding and TACT’s coding had a Cohen’s κ of 0.79–0.85. The authors also report model-call costs of $13.77 for TACT-Lite and $33.59 for TACT-Full per 100 threads. They say the paper identifies pitfalls and safeguards, but the abstract does not describe those safeguards in detail.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 3, 2026 (2026-11-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T07:32:14.445Z. First observed: 2026-11-05T05:03:28.124Z. Retrieved: 2026-11-09T07:25:22.326Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evaluation described in the abstract uses an independent dataset of 256 Stack Overflow threads; it does not establish how TACT performs on other qualitative datasets. The abstract reports that the authors identify pitfalls and safeguards, but does not describe them in detail. The abstract does not provide further details about the held-out sample or the evaluation procedure.
Open question
How would TACT’s reported coding results compare on qualitative datasets beyond the 256 Stack Overflow threads used in this evaluation?
Source-stated
The authors present TACT as a procedure for analyzing qualitative responses at scale while keeping teams in control of interpretation and judgment.
a procedure that enables such analysis at scale while keeping teams in control of interpretation and judgment
Source-stated
TACT-Full adds models from other model families that critique each model’s output, and a decision model that surfaces high-uncertainty code assignments.
adds adversaries (models from other model families that critique each model's output) and a decision model that surfaces high-uncertainty code assignments
Source-stated
On an independent dataset of 256 Stack Overflow threads, the authors report candidate codebook F1 of 0.75–0.77 and final coding F1 of 0.71–0.72 against other researchers’ labels.
Against their labels, TACT's candidate codebook achieved an F1 of 0.75-0.77 and its final coding an F1 of 0.71-0.72.
Source-stated
The authors report Cohen’s κ of 0.79–0.85 for agreement between their own manual coding and TACT’s coding on a held-out sample, and model-call costs of $13.77 to $33.59 per 100 threads.
On a held-out sample, agreement between our own manual coding and TACT's coding was Cohen's $\kappa$ of 0.79-0.85. TACT is also economical: model calls cost \$13.77 (TACT-Lite) to \$33.59 (TACT-Full) per 100 threads.
Source and contributor context
Choudhuri, Rudrajit, Bird, Christian, Badea, Carmen, Sarma, Anita · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 704aefb4-ab56-4cc2-8e12-80ee0d8d33b4.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T07:32:14.445Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 47bbbc7cf47a176ebe10699adc871df1c9d226d417596cb6882a6fd6816f05d2. Source text hash: c0c580fd4de0c6c3ec5c7f995b474d308ff2871a47ce866c26b2cfa74f5ec94d.
Revision and source-check history
Version 1 · 2026-11-09T07:32:14.445Z
arXiv cs.AI: TACT: A Pragmatic Approach to Qualitative Analysis with an AI Council
The authors present TACT, a procedure for using AI to analyze qualitative responses while leaving teams in control of interpretation and judgment. They frame it as an alternative to either leaving large collections of responses unanalyzed or accepting a ’s interpretation without that control. TACT-Lite uses the core AI council procedure; TACT-Full adds adversaries—models from other model families that critique each model’s output—and a decision model that surfaces high-uncertainty code assignments. In an evaluation on 256 Stack Overflow threads coded by other researchers, TACT’s candidate codebook achieved an F1 of 0.75–0.77, and its final coding achieved an F1 of 0.71–0.72 against those labels. On a held-out sample, the authors’ manual coding and TACT’s coding had a Cohen’s κ of 0.79–0.85. The authors also report model-call costs of $13.77 for TACT-Lite and $33.59 for TACT-Full per 100 threads. They say the paper identifies pitfalls and safeguards, but the abstract does not describe those safeguards in detail.
Original source
cs.AI · Article
Source details
Original source date: Nov 3, 2026 (2026-11-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T07:32:14.445Z. First observed: 2026-11-05T05:03:28.124Z. Retrieved: 2026-11-09T07:25:22.326Z.
Limits
The evaluation described in the abstract uses an independent dataset of 256 Stack Overflow threads; it does not establish how TACT performs on other qualitative datasets. The abstract reports that the authors identify pitfalls and safeguards, but does not describe them in detail. The abstract does not provide further details about the held-out sample or the evaluation procedure.
Open question
How would TACT’s reported coding results compare on qualitative datasets beyond the 256 Stack Overflow threads used in this evaluation?
Source-stated
The authors present TACT as a procedure for analyzing qualitative responses at scale while keeping teams in control of interpretation and judgment.
a procedure that enables such analysis at scale while keeping teams in control of interpretation and judgment
Source-stated
TACT-Full adds models from other model families that critique each model’s output, and a decision model that surfaces high-uncertainty code assignments.
adds adversaries (models from other model families that critique each model's output) and a decision model that surfaces high-uncertainty code assignments
Source-stated
On an independent dataset of 256 Stack Overflow threads, the authors report candidate codebook F1 of 0.75–0.77 and final coding F1 of 0.71–0.72 against other researchers’ labels.
Against their labels, TACT's candidate codebook achieved an F1 of 0.75-0.77 and its final coding an F1 of 0.71-0.72.
Source-stated
The authors report Cohen’s κ of 0.79–0.85 for agreement between their own manual coding and TACT’s coding on a held-out sample, and model-call costs of $13.77 to $33.59 per 100 threads.
On a held-out sample, agreement between our own manual coding and TACT's coding was Cohen's $\kappa$ of 0.79-0.85. TACT is also economical: model calls cost \$13.77 (TACT-Lite) to \$33.59 (TACT-Full) per 100 threads.
Source and contributor context
Choudhuri, Rudrajit, Bird, Christian, Badea, Carmen, Sarma, Anita · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 704aefb4-ab56-4cc2-8e12-80ee0d8d33b4.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T07:32:14.445Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 47bbbc7cf47a176ebe10699adc871df1c9d226d417596cb6882a6fd6816f05d2. Source text hash: c0c580fd4de0c6c3ec5c7f995b474d308ff2871a47ce866c26b2cfa74f5ec94d.
uncertain · 2026-11-09T07:32:14.445Z · anthropic/claude-opus-5.5
Publication: 704aefb4-ab56-4cc2-8e12-80ee0d8d33b4
attribution — pass: The reading attributes TACT, its framing, the evaluation results and the cost figures to the paper's authors (Choudhuri, Bird, Badea, Sarma). Results stay as the authors' reports, not established facts. The 'arXiv cs.AI' label in the title matches the publisher field, though the paper's primary listing is cs.SE.
Authors:Rudrajit Choudhuri, Christian Bird, Carmen Badea, Anita Sarma
summary — uncertain: The reading is accurate but hard for a non-specialist to follow. Central terms are never explained in plain language or flagged for tooltips: coding and codebook (qualitative labeling categories), code assignments, F1, Cohen's κ, and what the 'AI council' procedure is. It also drops the source's motivation that rigorous analysis takes expert time few teams can afford or justify, and that the data tells software teams what stakeholders need. Smallest repair: add one sentence restoring that stake, and give brief explanations or tooltips for coding, F1 and κ.
Qualitative data can tell software teams what their stakeholders need, yet rigorously analyzing thousands of qualitative responses takes expert time that few teams can afford or justify.
date — pass: originalPublished is 2026-11-03, which matches the arXiv v1 submission date. The feed date of Nov 5 is later and was correctly not used.
[Submitted on 3 Nov 2026]
support — pass: Every claim matches the abstract: the TACT variants, F1 0.75-0.77 and 0.71-0.72, κ 0.79-0.85, and costs of $13.77 to $33.59 per 100 threads. The limitations and the question about generalizing beyond 256 Stack Overflow threads follow from the evidence without overclaiming real-world reliability.
We evaluated TACT on an independent dataset of 256 Stack Overflow threads qualitatively coded by other researchers.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv