Escape Meridian

· Nov 18, 2026

Dharma-AI: Why your fine-tuning job takes longer than the FLOPs say

Markdown · JSON · Original source

The Dharma-AI team argues that -based estimates miss much of the a job spends outside its training loop. They compared estimates on 272 production supervised fine-tuning (SFT) runs using . In 5-fold cross-validation, their CatBoost regressor, using job details known before dispatch, had a mean absolute error of 1.71 hours; the FLOP baseline at 30% MFU had an error of 12.41 hours. The authors attribute the gap to configuration signals and interactions that a compute-only estimate misses, including evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime. Feature importance placed evaluation volume and checkpoint-related variables among prominent inputs, but the authors caution that feature importance is not a causal claim. Their practical point is that setup, evaluation, checkpoint uploads and shared network and storage I/O can make jobs with the same nominal compute finish hours apart. They say better estimates could help a scheduler plan GPU assignments and when capacity will become available. The study covers SFT on H200s, and the authors report wider errors for runs over 60 hours, where data is sparse. They say validation on other GPU types and more long-run data are still needed.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

Hugging FaceArticle · huggingface.coOpen original source

Original source

· Article

Source details

Original source date: Nov 18, 2026 (2026-11-18T18:53:43.959Z).

Date basis: Date stated by the publisher.

Added to Escape Meridian: 2026-11-18T19:03:08.837Z. First observed: 2026-11-18T18:55:12.408Z. Retrieved: 2026-11-18T19:01:53.595Z.

Referenced by

Recorded citations in public readings. A reference alone does not establish agreement or influence.

No public references recorded here yet.

Reference data (JSON)

Limits

The study covers SFT only; it does not include other training and alignment strategies such as , DPO or RLVR. Every run used . The authors say the fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other generations without retraining. The authors report only 13 runs above 60 hours and say errors widen in this sparse long-duration range. The authors describe feature importance as non-causal and caution that GPU allocation can act as a proxy for job scale in their data.

Open question

How well would the predictor perform on other types, once validated there as the authors plan?

Source-stated

The authors compared FLOP-based estimates and learned predictors using 272 production SFT jobs on NVIDIA H200 GPUs.

So we took 272 real supervised fine-tuning (SFT) jobs from our production pool of NVIDIA H200 GPUs

Source-stated

In 5-fold cross-validation, the CatBoost regressor had a mean absolute error of 1.71 hours; the 30% MFU FLOP baseline had a mean absolute error of 12.41 hours.

FLOP baseline (MFU 30%) 0.31 20.45 12.41 FLOP baseline (MFU 45%)

Source-stated

The predictor uses job configuration and model metadata available before dispatch, rather than runtime logs or partial progress.

Everything is known before dispatch. Every input comes from the submitted configuration: hyperparameters, dataset sizes, model identifiers, GPU allocation. No runtime logs, no partial progress.

Source-stated

The authors say the model uses training-specific signals, including evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and learns nonlinear interactions between them.

The regressor sees more training-specific signals, such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and it learns the nonlinear interactions between them from past runs.

Source-stated

The authors say feature importance is not causal evidence, while noting that evaluation and checkpoint choices shape wall-clock time in their data.

Feature importance is not a causal claim. Still, the pattern lines up with how a job is actually spent.

Source-stated

The authors say errors widen above 60 hours and regard the small number of such runs as a data limitation.

The long tail is where the model struggles, and with only 13 runs above 60 hours we read that as a data limitation.

Source-stated

The authors say the fitted model is calibrated to H200 infrastructure and should not be expected to transfer numerically to other GPU generations without retraining.

The fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other GPU generations without retraining.

Source and contributor context

Hugging Face · Publisher

The publisher is identified here. No individual author has been inferred.

Source media

Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Fine-tuning

Further training of an existing model to adapt it to a particular task or use.

Floating-point operations

A count of basic arithmetic steps; used to measure how much computation went into training a model.

NVIDIA H200

An NVIDIA GPU built on the Hopper architecture with 141 gigabytes of memory at 4.8 terabytes per second: nearly double the H100's memory capacity, with 1.4 times its memory bandwidth.

Graphics processing unit

A chip that performs many calculations at once; the main hardware used to train and run AI models.

Reinforcement learning from human feedback

Training that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.

Hugging Face

Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.

TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

Editorial provenance

Version 1. Publication ID: 360aa19b-416c-4316-84b9-3eae3cf88201.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T19:03:08.837Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: a904ab71a4e15bd1cb03ca95cd0514e5b2736659c122d8fcb1acb5419600bfeb. Source text hash: 7b0e97443219999a4981d5dad8c4a6c777cec1d7f86e7e5c2a0371e03939a8b8.

Revision and source-check history

Version 1 · 2026-11-18T19:03:08.837Z

Dharma-AI: Why your fine-tuning job takes longer than the FLOPs say

The Dharma-AI team argues that -based estimates miss much of the a job spends outside its training loop. They compared estimates on 272 production supervised fine-tuning (SFT) runs using . In 5-fold cross-validation, their CatBoost regressor, using job details known before dispatch, had a mean absolute error of 1.71 hours; the FLOP baseline at 30% MFU had an error of 12.41 hours. The authors attribute the gap to configuration signals and interactions that a compute-only estimate misses, including evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime. Feature importance placed evaluation volume and checkpoint-related variables among prominent inputs, but the authors caution that feature importance is not a causal claim. Their practical point is that setup, evaluation, checkpoint uploads and shared network and storage I/O can make jobs with the same nominal compute finish hours apart. They say better estimates could help a scheduler plan GPU assignments and when capacity will become available. The study covers SFT on H200s, and the authors report wider errors for runs over 60 hours, where data is sparse. They say validation on other GPU types and more long-run data are still needed.

Original source

· Article

Source details

Original source date: Nov 18, 2026 (2026-11-18T18:53:43.959Z).

Date basis: Date stated by the publisher.

Added to Escape Meridian: 2026-11-18T19:03:08.837Z. First observed: 2026-11-18T18:55:12.408Z. Retrieved: 2026-11-18T19:01:53.595Z.

Limits

The study covers SFT only; it does not include other training and alignment strategies such as , DPO or RLVR. Every run used . The authors say the fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other generations without retraining. The authors report only 13 runs above 60 hours and say errors widen in this sparse long-duration range. The authors describe feature importance as non-causal and caution that GPU allocation can act as a proxy for job scale in their data.

Open question

How well would the predictor perform on other types, once validated there as the authors plan?

Source-stated

The authors compared FLOP-based estimates and learned predictors using 272 production SFT jobs on NVIDIA H200 GPUs.

So we took 272 real supervised fine-tuning (SFT) jobs from our production pool of NVIDIA H200 GPUs

Source-stated

In 5-fold cross-validation, the CatBoost regressor had a mean absolute error of 1.71 hours; the 30% MFU FLOP baseline had a mean absolute error of 12.41 hours.

FLOP baseline (MFU 30%) 0.31 20.45 12.41 FLOP baseline (MFU 45%)

Source-stated

The predictor uses job configuration and model metadata available before dispatch, rather than runtime logs or partial progress.

Everything is known before dispatch. Every input comes from the submitted configuration: hyperparameters, dataset sizes, model identifiers, GPU allocation. No runtime logs, no partial progress.

Source-stated

The authors say the model uses training-specific signals, including evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and learns nonlinear interactions between them.

The regressor sees more training-specific signals, such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and it learns the nonlinear interactions between them from past runs.

Source-stated

The authors say feature importance is not causal evidence, while noting that evaluation and checkpoint choices shape wall-clock time in their data.

Feature importance is not a causal claim. Still, the pattern lines up with how a job is actually spent.

Source-stated

The authors say errors widen above 60 hours and regard the small number of such runs as a data limitation.

The long tail is where the model struggles, and with only 13 runs above 60 hours we read that as a data limitation.

Source-stated

The authors say the fitted model is calibrated to H200 infrastructure and should not be expected to transfer numerically to other GPU generations without retraining.

The fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other GPU generations without retraining.

Source and contributor context

Hugging Face · Publisher

The publisher is identified here. No individual author has been inferred.

Source media

Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Fine-tuning

Further training of an existing model to adapt it to a particular task or use.

Floating-point operations

A count of basic arithmetic steps; used to measure how much computation went into training a model.

NVIDIA H200

An NVIDIA GPU built on the Hopper architecture with 141 gigabytes of memory at 4.8 terabytes per second: nearly double the H100's memory capacity, with 1.4 times its memory bandwidth.

Graphics processing unit

A chip that performs many calculations at once; the main hardware used to train and run AI models.

Reinforcement learning from human feedback

Training that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.

Hugging Face

Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.

TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

Editorial provenance

Version 1. Publication ID: 360aa19b-416c-4316-84b9-3eae3cf88201.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T19:03:08.837Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: a904ab71a4e15bd1cb03ca95cd0514e5b2736659c122d8fcb1acb5419600bfeb. Source text hash: 7b0e97443219999a4981d5dad8c4a6c777cec1d7f86e7e5c2a0371e03939a8b8.

passed · 2026-11-18T19:03:08.837Z · anthropic/claude-opus-5.5

Publication: 360aa19b-416c-4316-84b9-3eae3cf88201

attribution — pass: The reading attributes the argument and results to the Dharma-AI team. The source byline lists Dharma-AI authors, and the post is a team article. The findings are consistently framed as the authors' own claims.

A preview of our upcoming paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models.

summary — pass: The reading leads with the authors' thesis that compute-only estimates miss time spent outside the training loop. It then gives the evidence, mechanism, scheduling use and limits. FLOP and MFU are left to tooltips, but the surrounding text conveys the meaning. The question about performance on other GPU types follows from the authors' stated next steps. Not mentioning that this is a preview of a not-yet-posted paper is a minor omission, not a distortion.

Two jobs with the same nominal compute can finish hours apart.

date — pass: The originalPublished value of 2026-11-18 from the publisher page matches the post's stated publication date and the feed entry.

Published November 18, 2026

support — pass: All figures are source-stated: 272 runs, H200 GPUs, MAE of 1.71 h for CatBoost versus 12.41 h for the 30% MFU baseline, the before-dispatch inputs, the non-causal caveat, sparse runs over 60 h and the transfer limit. The scheduler benefit is framed as the authors' claim. The claim-2 excerpt shows only the baseline row, but CatBoost's 1.71 h appears elsewhere in the source.

A CatBoost regressor that only sees information available before dispatch reaches R² = 0.98, with a mean absolute error of 1.71 h and a median error of 0.51 h in 5-fold cross-validation.

media — pass: The selected media is the verified link to the original Hugging Face blog post itself. This is the most direct access to the source.

Challenge or correct this reading

Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.