{"id":"42577e4753dc0fc097e6053dbac617fbc6342b9c13e83c2dea4eab33bc609467","title":"arXiv cs.AI: PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems","argument":"In their paper, Sven Kleinkop and coauthors argue that deployed LLM judges need ongoing maintenance: their agreement with an intended human evaluation policy may degrade as user interactions and score distributions change. They call this problem “judge drift” and introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge systems. The framework separates decisions about when to optimize, which method and feedback records to use, and how to update the judge prompt. The authors instantiate it with five prompt optimization methods and report that iterative optimization can recalibrate judges and recover performance after controlled shifts in the reference score distribution. But they stress that maintenance depends on context: neither the algorithm choice nor update timing works universally across settings. Their argument is that maintaining deployed LLM judges is an iterative problem, rather than a one-shot prompt optimization task. The abstract describes controlled shifts, so it does not establish how the methods perform across other kinds of change or in particular deployments.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The available evidence is the paper’s abstract; it does not provide experiment details or results beyond the reported recovery after controlled shifts in the reference score distribution. The abstract says maintenance depends on context but does not specify which contextual factors determine effective methods or update timing.","speaker":{"name":"Kleinkop, Sven, Janusko, Tamás, Bochnia, Ricardo, Hempel, Gunnar, Tomschke, Steffen, Anke, Jürgen, Thiele, Maik","roleAtSource":"Article author"},"source":{"title":"PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.07123","publishedAt":"2026-11-04","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors say agreement between deployed LLM judges and the intended human evaluation policy may degrade as user interactions and score distributions change.","excerpt":"their agreement with the intended human evaluation policy may degrade as user interactions and score distributions change.","url":"https://arxiv.org/abs/2611.07123","relationship":"Source-stated"},{"text":"The authors introduce PromptOps as a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge systems.","excerpt":"introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge (LLMaaJ) systems.","url":"https://arxiv.org/abs/2611.07123","relationship":"Source-stated"},{"text":"PromptOps separates decisions about when to optimize, which procedure and feedback records to use, and how to update the judge prompt.","excerpt":"PromptOps separates four operational decisions: when to optimize, which optimization procedure to apply, which feedback records to use, and how to update the judge prompt.","url":"https://arxiv.org/abs/2611.07123","relationship":"Source-stated"},{"text":"The authors report that iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the reference score distribution.","excerpt":"iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the referencescore distribution.","url":"https://arxiv.org/abs/2611.07123","relationship":"Source-stated"},{"text":"The authors say optimization method and update timing do not generalize universally across settings.","excerpt":"neither optimization algorithm choice nor update timing generalizes universally across settings.","url":"https://arxiv.org/abs/2611.07123","relationship":"Source-stated"},{"text":"The authors frame maintaining deployed LLM judges as an iterative maintenance problem.","excerpt":"prompt optimization for deployed LLM judges as an inherently iterative maintenance problem","url":"https://arxiv.org/abs/2611.07123","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07123","checkedAt":"2026-11-08T17:16:07.043Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Which features of a deployment determine when to optimize and which prompt optimization method works best?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems","summary":"In their paper, Sven Kleinkop and coauthors argue that deployed LLM judges need ongoing maintenance: their agreement with an intended human evaluation policy may degrade as user interactions and score distributions change. They call this problem “judge drift” and introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge systems. The framework separates decisions about when to optimize, which method and feedback records to use, and how to update the judge prompt. The authors instantiate it with five prompt optimization methods and report that iterative optimization can recalibrate judges and recover performance after controlled shifts in the reference score distribution. But they stress that maintenance depends on context: neither the algorithm choice nor update timing works universally across settings. Their argument is that maintaining deployed LLM judges is an iterative problem, rather than a one-shot prompt optimization task. The abstract describes controlled shifts, so it does not establish how the methods perform across other kinds of change or in particular deployments.","question":"Which features of a deployment determine when to optimize and which prompt optimization method works best?","claims":[{"text":"The authors say agreement between deployed LLM judges and the intended human evaluation policy may degrade as user interactions and score distributions change.","excerpt":"their agreement with the intended human evaluation policy may degrade as user interactions and score distributions change.","support":"source-stated"},{"text":"The authors introduce PromptOps as a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge systems.","excerpt":"introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge (LLMaaJ) systems.","support":"source-stated"},{"text":"PromptOps separates decisions about when to optimize, which procedure and feedback records to use, and how to update the judge prompt.","excerpt":"PromptOps separates four operational decisions: when to optimize, which optimization procedure to apply, which feedback records to use, and how to update the judge prompt.","support":"source-stated"},{"text":"The authors report that iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the reference score distribution.","excerpt":"iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the referencescore distribution.","support":"source-stated"},{"text":"The authors say optimization method and update timing do not generalize universally across settings.","excerpt":"neither optimization algorithm choice nor update timing generalizes universally across settings.","support":"source-stated"},{"text":"The authors frame maintaining deployed LLM judges as an iterative maintenance problem.","excerpt":"prompt optimization for deployed LLM judges as an inherently iterative maintenance problem","support":"source-stated"}],"limitations":["The available evidence is the paper’s abstract; it does not provide experiment details or results beyond the reported recovery after controlled shifts in the reference score distribution.","The abstract says maintenance depends on context but does not specify which contextual factors determine effective methods or update timing."],"mediaId":"c8d7e925447125961a884cf59bdad31b291c5e2a492be56f99ccd675fafc1574","correctionReason":null,"id":"42577e4753dc0fc097e6053dbac617fbc6342b9c13e83c2dea4eab33bc609467","publicationId":"33c6e948-b172-4106-8177-6e5dd1a63ccc","version":1,"sourceUrl":"https://arxiv.org/abs/2611.07123","sourceTitle":"PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-04","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:01:02.545Z","createdAt":"2026-11-08T17:01:34.920Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"c8d7e925447125961a884cf59bdad31b291c5e2a492be56f99ccd675fafc1574","kind":"article","url":"https://arxiv.org/abs/2611.07123","sourceUrl":"https://arxiv.org/abs/2611.07123","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T16:46:49.609Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07123","checkedAt":"2026-11-08T17:16:07.043Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"c8d7e925447125961a884cf59bdad31b291c5e2a492be56f99ccd675fafc1574","confidence":0.81,"provider":"typesafe","model":"jev-1.13.0","inputHash":"e5ea001f9e926f05843e832ae21abf6ff6617e9c91571820684c3f3821f1a490","decidedAt":"2026-11-08T16:46:51.386Z"},"provenance":{"textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observationId":"2165f276c9784bd3e001a6a145efc8538950de98bc8ab01f69e4a152d50b6dd7","retrievedAt":"2026-11-08T16:46:49.351Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"144fd6ef-76eb-4239-8202-7e5fa1c0d45c"},"reviewer":{"model":"gpt-6-luna","runId":"83fc6863-bebc-48a1-80e7-be6d564bdbe3"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-08T17:01:34.920Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Kleinkop, Sven","url":null,"kind":"unknown"},{"name":"Janusko, Tamás","url":null,"kind":"unknown"},{"name":"Bochnia, Ricardo","url":null,"kind":"unknown"},{"name":"Hempel, Gunnar","url":null,"kind":"unknown"},{"name":"Tomschke, Steffen","url":null,"kind":"unknown"},{"name":"Anke, Jürgen","url":null,"kind":"unknown"},{"name":"Thiele, Maik","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Kleinkop, Sven","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kleinkop, Sven","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}},{"actor":{"name":"Janusko, Tamás","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Janusko, Tamás","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}},{"actor":{"name":"Bochnia, Ricardo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bochnia, Ricardo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}},{"actor":{"name":"Hempel, Gunnar","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hempel, Gunnar","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}},{"actor":{"name":"Tomschke, Steffen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tomschke, Steffen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}},{"actor":{"name":"Anke, Jürgen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Anke, Jürgen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}},{"actor":{"name":"Thiele, Maik","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Thiele, Maik","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.07123","snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/42577e4753dc0fc097e6053dbac617fbc6342b9c13e83c2dea4eab33bc609467/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-promptops-iterative-prompt-optimization-for-llm-as-42577e47","formats":{"html":"/readings/arxiv-cs-ai-promptops-iterative-prompt-optimization-for-llm-as-42577e47","markdown":"/readings/arxiv-cs-ai-promptops-iterative-prompt-optimization-for-llm-as-42577e47.md","json":"/readings/arxiv-cs-ai-promptops-iterative-prompt-optimization-for-llm-as-42577e47.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"42577e4753dc0fc097e6053dbac617fbc6342b9c13e83c2dea4eab33bc609467","revision":"33c6e948-b172-4106-8177-6e5dd1a63ccc","revisionKind":"publication-id","title":"arXiv cs.AI: PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems","url":"https://escapemeridian.com/readings/arxiv-cs-ai-promptops-iterative-prompt-optimization-for-llm-as-42577e47"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.07123","revision":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","revisionKind":"sha256-source-text","title":"PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems","url":"https://arxiv.org/abs/2611.07123"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"42577e4753dc0fc097e6053dbac617fbc6342b9c13e83c2dea4eab33bc609467","title":"arXiv cs.AI: PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems","argument":"In their paper, Sven Kleinkop and coauthors argue that deployed LLM judges need ongoing maintenance: their agreement with an intended human evaluation policy may degrade as user interactions and score distributions change. They call this problem “judge drift” and introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge systems. The framework separates decisions about when to optimize, which method and feedback records to use, and how to update the judge prompt. The authors instantiate it with five prompt optimization methods and report that iterative optimization can recalibrate judges and recover performance after controlled shifts in the reference score distribution. But they stress that maintenance depends on context: neither the algorithm choice nor update timing works universally across settings. Their argument is that maintaining deployed LLM judges is an iterative problem, rather than a one-shot prompt optimization task. The abstract describes controlled shifts, so it does not establish how the methods perform across other kinds of change or in particular deployments.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The available evidence is the paper’s abstract; it does not provide experiment details or results beyond the reported recovery after controlled shifts in the reference score distribution. The abstract says maintenance depends on context but does not specify which contextual factors determine effective methods or update timing.","speaker":{"name":"Kleinkop, Sven, Janusko, Tamás, Bochnia, Ricardo, Hempel, Gunnar, Tomschke, Steffen, Anke, Jürgen, Thiele, Maik","roleAtSource":"Article author"},"source":{"title":"PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.07123","publishedAt":"2026-11-04","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors say agreement between deployed LLM judges and the intended human evaluation policy may degrade as user interactions and score distributions change.","excerpt":"their agreement with the intended human evaluation policy may degrade as user interactions and score distributions change.","url":"https://arxiv.org/abs/2611.07123","relationship":"Source-stated"},{"text":"The authors introduce PromptOps as a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge systems.","excerpt":"introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge (LLMaaJ) systems.","url":"https://arxiv.org/abs/2611.07123","relationship":"Source-stated"},{"text":"PromptOps separates decisions about when to optimize, which procedure and feedback records to use, and how to update the judge prompt.","excerpt":"PromptOps separates four operational decisions: when to optimize, which optimization procedure to apply, which feedback records to use, and how to update the judge prompt.","url":"https://arxiv.org/abs/2611.07123","relationship":"Source-stated"},{"text":"The authors report that iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the reference score distribution.","excerpt":"iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the referencescore distribution.","url":"https://arxiv.org/abs/2611.07123","relationship":"Source-stated"},{"text":"The authors say optimization method and update timing do not generalize universally across settings.","excerpt":"neither optimization algorithm choice nor update timing generalizes universally across settings.","url":"https://arxiv.org/abs/2611.07123","relationship":"Source-stated"},{"text":"The authors frame maintaining deployed LLM judges as an iterative maintenance problem.","excerpt":"prompt optimization for deployed LLM judges as an inherently iterative maintenance problem","url":"https://arxiv.org/abs/2611.07123","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07123","checkedAt":"2026-11-08T17:16:07.043Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Which features of a deployment determine when to optimize and which prompt optimization method works best?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems","summary":"In their paper, Sven Kleinkop and coauthors argue that deployed LLM judges need ongoing maintenance: their agreement with an intended human evaluation policy may degrade as user interactions and score distributions change. They call this problem “judge drift” and introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge systems. The framework separates decisions about when to optimize, which method and feedback records to use, and how to update the judge prompt. The authors instantiate it with five prompt optimization methods and report that iterative optimization can recalibrate judges and recover performance after controlled shifts in the reference score distribution. But they stress that maintenance depends on context: neither the algorithm choice nor update timing works universally across settings. Their argument is that maintaining deployed LLM judges is an iterative problem, rather than a one-shot prompt optimization task. The abstract describes controlled shifts, so it does not establish how the methods perform across other kinds of change or in particular deployments.","question":"Which features of a deployment determine when to optimize and which prompt optimization method works best?","claims":[{"text":"The authors say agreement between deployed LLM judges and the intended human evaluation policy may degrade as user interactions and score distributions change.","excerpt":"their agreement with the intended human evaluation policy may degrade as user interactions and score distributions change.","support":"source-stated"},{"text":"The authors introduce PromptOps as a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge systems.","excerpt":"introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge (LLMaaJ) systems.","support":"source-stated"},{"text":"PromptOps separates decisions about when to optimize, which procedure and feedback records to use, and how to update the judge prompt.","excerpt":"PromptOps separates four operational decisions: when to optimize, which optimization procedure to apply, which feedback records to use, and how to update the judge prompt.","support":"source-stated"},{"text":"The authors report that iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the reference score distribution.","excerpt":"iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the referencescore distribution.","support":"source-stated"},{"text":"The authors say optimization method and update timing do not generalize universally across settings.","excerpt":"neither optimization algorithm choice nor update timing generalizes universally across settings.","support":"source-stated"},{"text":"The authors frame maintaining deployed LLM judges as an iterative maintenance problem.","excerpt":"prompt optimization for deployed LLM judges as an inherently iterative maintenance problem","support":"source-stated"}],"limitations":["The available evidence is the paper’s abstract; it does not provide experiment details or results beyond the reported recovery after controlled shifts in the reference score distribution.","The abstract says maintenance depends on context but does not specify which contextual factors determine effective methods or update timing."],"mediaId":"c8d7e925447125961a884cf59bdad31b291c5e2a492be56f99ccd675fafc1574","correctionReason":null,"id":"42577e4753dc0fc097e6053dbac617fbc6342b9c13e83c2dea4eab33bc609467","publicationId":"33c6e948-b172-4106-8177-6e5dd1a63ccc","version":1,"sourceUrl":"https://arxiv.org/abs/2611.07123","sourceTitle":"PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-04","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:01:02.545Z","createdAt":"2026-11-08T17:01:34.920Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"c8d7e925447125961a884cf59bdad31b291c5e2a492be56f99ccd675fafc1574","kind":"article","url":"https://arxiv.org/abs/2611.07123","sourceUrl":"https://arxiv.org/abs/2611.07123","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T16:46:49.609Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07123","checkedAt":"2026-11-08T17:16:07.043Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"c8d7e925447125961a884cf59bdad31b291c5e2a492be56f99ccd675fafc1574","confidence":0.81,"provider":"typesafe","model":"jev-1.13.0","inputHash":"e5ea001f9e926f05843e832ae21abf6ff6617e9c91571820684c3f3821f1a490","decidedAt":"2026-11-08T16:46:51.386Z"},"provenance":{"textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observationId":"2165f276c9784bd3e001a6a145efc8538950de98bc8ab01f69e4a152d50b6dd7","retrievedAt":"2026-11-08T16:46:49.351Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"144fd6ef-76eb-4239-8202-7e5fa1c0d45c"},"reviewer":{"model":"gpt-6-luna","runId":"83fc6863-bebc-48a1-80e7-be6d564bdbe3"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-08T17:01:34.920Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Kleinkop, Sven","url":null,"kind":"unknown"},{"name":"Janusko, Tamás","url":null,"kind":"unknown"},{"name":"Bochnia, Ricardo","url":null,"kind":"unknown"},{"name":"Hempel, Gunnar","url":null,"kind":"unknown"},{"name":"Tomschke, Steffen","url":null,"kind":"unknown"},{"name":"Anke, Jürgen","url":null,"kind":"unknown"},{"name":"Thiele, Maik","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Kleinkop, Sven","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kleinkop, Sven","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}},{"actor":{"name":"Janusko, Tamás","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Janusko, Tamás","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}},{"actor":{"name":"Bochnia, Ricardo","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bochnia, Ricardo","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}},{"actor":{"name":"Hempel, Gunnar","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Hempel, Gunnar","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}},{"actor":{"name":"Tomschke, Steffen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tomschke, Steffen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}},{"actor":{"name":"Anke, Jürgen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Anke, Jürgen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}},{"actor":{"name":"Thiele, Maik","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Thiele, Maik","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07123","locator":"head/meta[name=author|citation_author]"},"assertionId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T17:16:07.043Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","sourceUrl":"https://arxiv.org/abs/2611.07123","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z"}}],"participationCoverage":{"total":7,"returned":7,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.07123","snapshotId":"2adc35e5f54fb5c66a95aad8e35f2f4e05900a6fa5616d5552c9481935489a66","contentSha256":"f903ca2bc750710809ae0d3d86408dd5dcfa40408a1c310657201971c64a0c70","textHash":"b9f923c7ddbc99350b422b3aa8ea350bab145e0702e45a0de442a4f23370f0fc","observedAt":"2026-11-08T17:16:07.043Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/42577e4753dc0fc097e6053dbac617fbc6342b9c13e83c2dea4eab33bc609467/source"}}}],"audits":[{"publicationId":"33c6e948-b172-4106-8177-6e5dd1a63ccc","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the work to Sven Kleinkop and coauthors, matching the listed authors. It keeps the findings framed as the authors' claims and reports rather than established facts.","excerpt":"Authors:Sven Kleinkop, Tamás Janusko, Ricardo Bochnia, Gunnar Hempel, Steffen Tomschke, Jürgen Anke, Maik Thiele"},{"dimension":"summary","status":"uncertain","explanation":"The reading faithfully conveys the argument, but it never says what an LLM judge is: a large language model used to automatically evaluate the behavior of chatbots and other AI systems. Without that, a general reader cannot see what drift threatens or why it matters. Prompt optimization is also left unexplained. Smallest repair: add one opening clause stating that LLMs are increasingly used as automated judges of chatbots and other AI systems. The title also repeats the paper title without plain-language context.","excerpt":"Large language models (LLMs) are increasingly used as automated judges for evaluating the behavior of chatbots and other AI systems."},{"dimension":"date","status":"pass","explanation":"The originalPublished date, 2026-11-04, matches the v1 submission date on arXiv. It is not the later feed date of 2026-11-06.","excerpt":"[Submitted on 4 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim is supported by the abstract. The contrast with one-shot optimization follows from the abstract's description of existing methods. The limitation correctly notes that the results come from controlled shifts. The question follows from the stated context-dependence and is not a leading claim.","excerpt":"neither optimization algorithm choice nor update timing generalizes universally across settings."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the original arXiv abstract page. It is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-08T17:01:34.920Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}