Rezaei, Mohammad Reza, Wang, Dingxian, Levert, Brett, Fejzo, Celeste, Rabinovich, Andrew, Chen, Zhao · Nov 3, 2026
arXiv cs.AI: PAYWORTHY—Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?
Markdown · JSON · Original source
The authors of PAYWORTHY test whether frontier language models can forecast, before an attempt, which paid freelance jobs an will complete. Their contains 1,494 real job postings sampled from 18.8 million postings made between 2020 and 2026. For each job, an agent tries the work from the posting alone in a single run, and an judge acting as the client decides whether to accept it. The researchers compare models’ forecasts with those acceptance decisions using AUC, a measure of how well a model ranks accepted jobs above rejected ones. The striking result depends on what the models are told about the agent. When given information about what it cannot do, all models rank jobs well; when described only as a generic agent with a browser and code execution, their rankings are about as good as random. The authors report that this pattern also holds with judges from four other model families, and that informed forecasts rank the verdicts of two human raters. The result suggests that a forecast may depend less on a broad impression of an AI agent than on a specific account of its capabilities and limits. The abstract does not establish how well these forecasts would guide decisions outside this benchmark and evaluation setup.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 3, 2026 (2026-11-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T00:54:26.981Z. First observed: 2026-11-05T05:06:35.985Z. Retrieved: 2026-11-10T00:53:48.315Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported results come from the described and evaluation setup; the abstract does not report how forecasts perform outside it. The abstract says agents attempt each job from the posting alone in a single run, with acceptance decided by an judge acting as the client.
Open question
Would the informed forecasts still rank accepted jobs well when agents attempt jobs beyond this ’s single-run, posting-only setup?
Source-stated
The paper tests whether frontier language models can forecast an AI agent’s success on paid freelance jobs before the agent attempts them.
We test whether frontier language models can make this call before any attempt.
Source-stated
PAYWORTHY includes 1,494 real freelance job postings sampled from 18.8 million postings made between 2020 and 2026.
1{,}494 real freelance job postings, sampled from 18.8M postings made between 2020 and 2026
Source-stated
Models ranked jobs well when told what the agent cannot do, but performed about as well as random when told it was a generic agent with browser and code execution.
Told what the agent cannot do, every model ranks the jobs well (AUC 0.66--0.86). Told instead that it is a generic agent with a browser and code execution, the same models rank the same jobs about as well as a random ordering (0.45--0.56).
Source-stated
The abstract describes a specific evaluation setup: agents attempt each job from its posting alone in a single run, and an LLM judge decides whether to accept the work.
An agent attempts each job from the posting alone in a single run, and an LLM judge acting as the client decides whether to accept the work.
Source and contributor context
Rezaei, Mohammad Reza, Wang, Dingxian, Levert, Brett, Fejzo, Celeste, Rabinovich, Andrew, Chen, Zhao · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 414fbf41-ad0c-4b2f-8af3-c7138912eaf3.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T00:54:26.981Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: aef9f910dc3890efb0df20d8bd4930bc141c52e3ac5c2112d47fbac954822029. Source text hash: c47d76b681ba3e2133c123ee354ea2ebde0a866decff65bd7e6d44bbaa368175.
Revision and source-check history
Version 1 · 2026-11-10T00:54:26.981Z
arXiv cs.AI: PAYWORTHY—Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?
The authors of PAYWORTHY test whether frontier language models can forecast, before an attempt, which paid freelance jobs an will complete. Their contains 1,494 real job postings sampled from 18.8 million postings made between 2020 and 2026. For each job, an agent tries the work from the posting alone in a single run, and an judge acting as the client decides whether to accept it. The researchers compare models’ forecasts with those acceptance decisions using AUC, a measure of how well a model ranks accepted jobs above rejected ones. The striking result depends on what the models are told about the agent. When given information about what it cannot do, all models rank jobs well; when described only as a generic agent with a browser and code execution, their rankings are about as good as random. The authors report that this pattern also holds with judges from four other model families, and that informed forecasts rank the verdicts of two human raters. The result suggests that a forecast may depend less on a broad impression of an AI agent than on a specific account of its capabilities and limits. The abstract does not establish how well these forecasts would guide decisions outside this benchmark and evaluation setup.
Original source
cs.AI · Article
Source details
Original source date: Nov 3, 2026 (2026-11-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T00:54:26.981Z. First observed: 2026-11-05T05:06:35.985Z. Retrieved: 2026-11-10T00:53:48.315Z.
Limits
The reported results come from the described and evaluation setup; the abstract does not report how forecasts perform outside it. The abstract says agents attempt each job from the posting alone in a single run, with acceptance decided by an judge acting as the client.
Open question
Would the informed forecasts still rank accepted jobs well when agents attempt jobs beyond this ’s single-run, posting-only setup?
Source-stated
The paper tests whether frontier language models can forecast an AI agent’s success on paid freelance jobs before the agent attempts them.
We test whether frontier language models can make this call before any attempt.
Source-stated
PAYWORTHY includes 1,494 real freelance job postings sampled from 18.8 million postings made between 2020 and 2026.
1{,}494 real freelance job postings, sampled from 18.8M postings made between 2020 and 2026
Source-stated
Models ranked jobs well when told what the agent cannot do, but performed about as well as random when told it was a generic agent with browser and code execution.
Told what the agent cannot do, every model ranks the jobs well (AUC 0.66--0.86). Told instead that it is a generic agent with a browser and code execution, the same models rank the same jobs about as well as a random ordering (0.45--0.56).
Source-stated
The abstract describes a specific evaluation setup: agents attempt each job from its posting alone in a single run, and an LLM judge decides whether to accept the work.
An agent attempts each job from the posting alone in a single run, and an LLM judge acting as the client decides whether to accept the work.
Source and contributor context
Rezaei, Mohammad Reza, Wang, Dingxian, Levert, Brett, Fejzo, Celeste, Rabinovich, Andrew, Chen, Zhao · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 414fbf41-ad0c-4b2f-8af3-c7138912eaf3.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T00:54:26.981Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: aef9f910dc3890efb0df20d8bd4930bc141c52e3ac5c2112d47fbac954822029. Source text hash: c47d76b681ba3e2133c123ee354ea2ebde0a866decff65bd7e6d44bbaa368175.
uncertain · 2026-11-10T00:54:26.981Z · anthropic/claude-opus-5.5
Publication: 414fbf41-ad0c-4b2f-8af3-c7138912eaf3
attribution — pass: The reading attributes the benchmark, method and findings to the PAYWORTHY authors on arXiv cs.AI. Reported results stay framed as the authors' claims, including the extra judge families and the human-rater check.
These results hold under judges from four other model families (Claude, GLM, Kimi and Qwen), and the informed forecasts also rank the verdicts of two human raters.
summary — pass: The reading leads with the actor and the question being tested. It explains the benchmark setup, defines AUC plainly and states the central contrast between informed and generic agent descriptions. The question follows from the stated single-run, posting-only setup. A non-specialist can follow it. Omitting the count of 18 models does not distort the meaning.
The answer depends on how the agent is described.
date — pass: The displayed original publication date of 2026-11-03 matches the arXiv v1 submission date. The feed date of 2026-11-05 is correctly not used.
[Submitted on 3 Nov 2026]
support — uncertain: The factual claims match the abstract. However, the sentence 'The result suggests that a forecast may depend less on a broad impression... than on a specific account of its capabilities and limits' is an inference that is not labeled as Escape Meridian interpretation. It also slightly broadens the source condition, which concerns telling models what the agent cannot do, to 'capabilities and limits'. Smallest repair: label it as EM interpretation and replace 'capabilities and limits' with 'what the agent cannot do'.
Told what the agent cannot do, every model ranks the jobs well (AUC 0.66--0.86).
media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
An AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceAn AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference