# Li, Liang and Raj on Autonomous ML Engineering for Recommendation Systems

[Canonical reading](https://escapemeridian.com/readings/li-liang-and-raj-on-autonomous-ml-engineering-for-c007152f)

Matthew Li, Albert Liang and Dilip Raj describe human-on-the-loop machine learning engineer (MLE) agents that propose, implement, train and evaluate changes to large-scale search rankers. Engineers retain control over objective design, search direction and final-model review. The authors say the work addresses three challenges: experiments can take hours, offline metrics do not always predict online impact, and improvements can saturate after early wins. Their approach uses parallel training runs, an objective calibrated to match online impact, and careful context engineering. In one campaign, the system completed 1,097 candidate experiments and accepted 325, a 29.6% acceptance rate; the earlier campaign had a 15.6% rate, but used a different objective and search constraint. The authors also report that the best model increased search volume by 0.23% in a large-scale A/B test. They say some improvements were architectural changes the team had not tried before, drawn from industry and literature, suggesting agents can explore parts of the design space that human intuition overlooks. Their central argument is that the design of the loop around the model matters as much as the underlying model in their experience. More broadly, they argue that autonomous agents can do real machine-learning engineering in industry-scale systems.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Nov 18, 2026 (2026-11-18).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-19T05:03:53.319Z. First observed: 2026-11-19T05:01:18.979Z. Retrieved: 2026-11-19T05:02:27.446Z.

- [MLE Agents: Autonomous ML Engineering in Large-scale Recommendation Systems](https://arxiv.org/abs/2611.23910)

## Limits

The evidence is an abstract describing the authors’ work; it does not provide further experimental detail or results beyond those summarized here. The acceptance-rate comparison is not a controlled comparison: the authors say the campaigns used different objectives and search constraints. The abstract reports results from one campaign and an A/B test, and identifies saturation as a challenge; it does not report how gains hold up over continued experimentation.


## Open question

How do improvements from the agents hold up as experiments continue, given the authors’ observation that gains can saturate after early wins?


## Source-stated

The authors built human-on-the-loop MLE agents for large-scale search rankers, while engineers retained control over key decisions and final review.

> Engineers retain control over objective design, search direction, and final-model review.

- [Supporting source](https://arxiv.org/abs/2611.23910)

## Source-stated

The authors identify slow feedback, imperfect alignment between offline metrics and online impact, and saturation after early wins as challenges.

> feedback takes hours per experiment, offline metrics do not always predict online impact, and improvements saturate after the early wins.

- [Supporting source](https://arxiv.org/abs/2611.23910)

## Source-stated

In one campaign, 325 of 1,097 completed candidate experiments were accepted; the reported acceptance rate was higher than in an earlier campaign with a different objective and search constraint.

> 1,097 completed candidate experiments and accepted 325, a 29.6\\% acceptance rate compared with 15.6\\% in the earlier campaign under a different objective and search constraint.

- [Supporting source](https://arxiv.org/abs/2611.23910)

## Source-stated

The authors report that their best model increased search volume in a large-scale A/B test.

> Its best model increases search volume by 0.23\\% in an A/B test of large scale.

- [Supporting source](https://arxiv.org/abs/2611.23910)

## Source-stated

The authors argue that designing the agent’s surrounding loop is as important as relying on a stronger underlying model, based on their experience.

> the design of the loop around it is just as critical.

- [Supporting source](https://arxiv.org/abs/2611.23910)

## Source and contributor context

Li, Matthew, Liang, Albert, Raj, Dilip · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Machine learning

Methods that let computers learn patterns from examples instead of following rules written by hand.

- [Definition reference](https://developers.google.com/machine-learning/glossary#machine-learning)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: 574dd985-6f79-44e9-85cd-f258fe33de79.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-19T05:03:53.319Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 0068442d4ec951c16c734e4ebc7b649d9a47ea2257e779ce9dadb839855ab40d. Source text hash: a531a488536c8075d78107fa6ba31a3619181062169fe2fb8c3a31dc9f30dc16.


## Revision and source-check history

### Version 1 — 2026-11-19T05:03:53.319Z

Matthew Li, Albert Liang and Dilip Raj describe human-on-the-loop machine learning engineer (MLE) agents that propose, implement, train and evaluate changes to large-scale search rankers. Engineers retain control over objective design, search direction and final-model review. The authors say the work addresses three challenges: experiments can take hours, offline metrics do not always predict online impact, and improvements can saturate after early wins. Their approach uses parallel training runs, an objective calibrated to match online impact, and careful context engineering. In one campaign, the system completed 1,097 candidate experiments and accepted 325, a 29.6% acceptance rate; the earlier campaign had a 15.6% rate, but used a different objective and search constraint. The authors also report that the best model increased search volume by 0.23% in a large-scale A/B test. They say some improvements were architectural changes the team had not tried before, drawn from industry and literature, suggesting agents can explore parts of the design space that human intuition overlooks. Their central argument is that the design of the loop around the model matters as much as the underlying model in their experience. More broadly, they argue that autonomous agents can do real machine-learning engineering in industry-scale systems.

## Original source

arXiv cs.AI · Article

Original source date: Nov 18, 2026 (2026-11-18).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-19T05:03:53.319Z. First observed: 2026-11-19T05:01:18.979Z. Retrieved: 2026-11-19T05:02:27.446Z.

- [MLE Agents: Autonomous ML Engineering in Large-scale Recommendation Systems](https://arxiv.org/abs/2611.23910)

## Limits

The evidence is an abstract describing the authors’ work; it does not provide further experimental detail or results beyond those summarized here. The acceptance-rate comparison is not a controlled comparison: the authors say the campaigns used different objectives and search constraints. The abstract reports results from one campaign and an A/B test, and identifies saturation as a challenge; it does not report how gains hold up over continued experimentation.


## Open question

How do improvements from the agents hold up as experiments continue, given the authors’ observation that gains can saturate after early wins?


## Source-stated

The authors built human-on-the-loop MLE agents for large-scale search rankers, while engineers retained control over key decisions and final review.

> Engineers retain control over objective design, search direction, and final-model review.

- [Supporting source](https://arxiv.org/abs/2611.23910)

## Source-stated

The authors identify slow feedback, imperfect alignment between offline metrics and online impact, and saturation after early wins as challenges.

> feedback takes hours per experiment, offline metrics do not always predict online impact, and improvements saturate after the early wins.

- [Supporting source](https://arxiv.org/abs/2611.23910)

## Source-stated

In one campaign, 325 of 1,097 completed candidate experiments were accepted; the reported acceptance rate was higher than in an earlier campaign with a different objective and search constraint.

> 1,097 completed candidate experiments and accepted 325, a 29.6\\% acceptance rate compared with 15.6\\% in the earlier campaign under a different objective and search constraint.

- [Supporting source](https://arxiv.org/abs/2611.23910)

## Source-stated

The authors report that their best model increased search volume in a large-scale A/B test.

> Its best model increases search volume by 0.23\\% in an A/B test of large scale.

- [Supporting source](https://arxiv.org/abs/2611.23910)

## Source-stated

The authors argue that designing the agent’s surrounding loop is as important as relying on a stronger underlying model, based on their experience.

> the design of the loop around it is just as critical.

- [Supporting source](https://arxiv.org/abs/2611.23910)

## Source and contributor context

Li, Matthew, Liang, Albert, Raj, Dilip · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Machine learning

Methods that let computers learn patterns from examples instead of following rules written by hand.

- [Definition reference](https://developers.google.com/machine-learning/glossary#machine-learning)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: 574dd985-6f79-44e9-85cd-f258fe33de79.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-19T05:03:53.319Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 0068442d4ec951c16c734e4ebc7b649d9a47ea2257e779ce9dadb839855ab40d. Source text hash: a531a488536c8075d78107fa6ba31a3619181062169fe2fb8c3a31dc9f30dc16.


### passed — 2026-11-19T05:03:53.319Z — anthropic/claude-opus-5.5

Publication: 574dd985-6f79-44e9-85cd-f258fe33de79

attribution — pass: The reading attributes the work and its claims to Matthew Li, Albert Liang and Dilip Raj, the listed authors. Results and interpretations stay with them through phrasing like 'the authors report' and 'they argue'.

> Authors:Matthew Li, Albert Liang, Dilip Raj

summary — pass: The summary faithfully covers the actor, the human-on-the-loop setup, the three obstacles, the fixes, the numbers with the comparison caveat, the A/B result, the unfamiliar architectural changes and the broader claim. Calling the harness point the 'central argument' is a mild editorial framing, but it stays hedged with 'in their experience'. Terms such as A/B test and context engineering are kept exact and can be explained by tooltips. The question follows from the stated saturation obstacle.

> We find that the design of the harness around the model matters: intuition suggests that a stronger underlying model is enough, but in our experience, the design of the loop around it is just as critical.

date — pass: The originalPublished date of 2026-11-18 matches the v1 arXiv submission date. The feed entry dated Nov 19 is later.

> \[Submitted on 18 Nov 2026\]

support — pass: Every claim is directly supported by an exact excerpt. The limitations correctly note that the acceptance-rate comparison is not controlled and that the evidence is only an abstract. The reading does not turn the 0.23% result into a broader benefit claim.

> a 29.6\\% acceptance rate compared with 15.6\\% in the earlier campaign under a different objective and search constraint

media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct way to reach the source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/c007152f8ed7f7a25f57d3793689e817cacc5cf7e7ab248eb0ff15a249a61891/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/li-liang-and-raj-on-autonomous-ml-engineering-for-c007152f.json)
