· Nov 18, 2026
Dharma-AI: Why your fine-tuning job takes longer than the FLOPs say
Markdown · JSON · Original source
The Dharma-AI team argues that -based estimates miss much of the a job spends outside its training loop. They compared estimates on 272 production supervised fine-tuning (SFT) runs using . In 5-fold cross-validation, their CatBoost regressor, using job details known before dispatch, had a mean absolute error of 1.71 hours; the FLOP baseline at 30% MFU had an error of 12.41 hours. The authors attribute the gap to configuration signals and interactions that a compute-only estimate misses, including evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime. Feature importance placed evaluation volume and checkpoint-related variables among prominent inputs, but the authors caution that feature importance is not a causal claim. Their practical point is that setup, evaluation, checkpoint uploads and shared network and storage I/O can make jobs with the same nominal compute finish hours apart. They say better estimates could help a scheduler plan GPU assignments and when capacity will become available. The study covers SFT on H200s, and the authors report wider errors for runs over 60 hours, where data is sparse. They say validation on other GPU types and more long-run data are still needed.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Nov 18, 2026 (2026-11-18T18:53:43.959Z).
Date basis: Date stated by the publisher.
Added to Escape Meridian: 2026-11-18T19:03:08.837Z. First observed: 2026-11-18T18:55:12.408Z. Retrieved: 2026-11-18T19:01:53.595Z.
Referenced by
Recorded citations in public readings. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The study covers SFT only; it does not include other training and alignment strategies such as , DPO or RLVR. Every run used . The authors say the fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other generations without retraining. The authors report only 13 runs above 60 hours and say errors widen in this sparse long-duration range. The authors describe feature importance as non-causal and caution that GPU allocation can act as a proxy for job scale in their data.
Open question
How well would the predictor perform on other types, once validated there as the authors plan?
Source-stated
The authors compared FLOP-based estimates and learned predictors using 272 production SFT jobs on NVIDIA H200 GPUs.
So we took 272 real supervised fine-tuning (SFT) jobs from our production pool of NVIDIA H200 GPUs
Source-stated
In 5-fold cross-validation, the CatBoost regressor had a mean absolute error of 1.71 hours; the 30% MFU FLOP baseline had a mean absolute error of 12.41 hours.
FLOP baseline (MFU 30%) 0.31 20.45 12.41 FLOP baseline (MFU 45%)
Source-stated
The predictor uses job configuration and model metadata available before dispatch, rather than runtime logs or partial progress.
Everything is known before dispatch. Every input comes from the submitted configuration: hyperparameters, dataset sizes, model identifiers, GPU allocation. No runtime logs, no partial progress.
Source-stated
The authors say the model uses training-specific signals, including evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and learns nonlinear interactions between them.
The regressor sees more training-specific signals, such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and it learns the nonlinear interactions between them from past runs.
Source-stated
The authors say feature importance is not causal evidence, while noting that evaluation and checkpoint choices shape wall-clock time in their data.
Feature importance is not a causal claim. Still, the pattern lines up with how a job is actually spent.
Source-stated
The authors say errors widen above 60 hours and regard the small number of such runs as a data limitation.
The long tail is where the model struggles, and with only 13 runs above 60 hours we read that as a data limitation.
Source-stated
The authors say the fitted model is calibrated to H200 infrastructure and should not be expected to transfer numerically to other GPU generations without retraining.
The fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other GPU generations without retraining.
Source and contributor context
Hugging Face · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
Floating-point operations
A count of basic arithmetic steps; used to measure how much computation went into training a model.
NVIDIA H200
An NVIDIA GPU built on the Hopper architecture with 141 gigabytes of memory at 4.8 terabytes per second: nearly double the H100's memory capacity, with 1.4 times its memory bandwidth.
Graphics processing unit
A chip that performs many calculations at once; the main hardware used to train and run AI models.
Reinforcement learning from human feedback
Training that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.
Hugging Face
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 360aa19b-416c-4316-84b9-3eae3cf88201.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T19:03:08.837Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: a904ab71a4e15bd1cb03ca95cd0514e5b2736659c122d8fcb1acb5419600bfeb. Source text hash: 7b0e97443219999a4981d5dad8c4a6c777cec1d7f86e7e5c2a0371e03939a8b8.
Revision and source-check history
Version 1 · 2026-11-18T19:03:08.837Z
Dharma-AI: Why your fine-tuning job takes longer than the FLOPs say
The Dharma-AI team argues that -based estimates miss much of the a job spends outside its training loop. They compared estimates on 272 production supervised fine-tuning (SFT) runs using . In 5-fold cross-validation, their CatBoost regressor, using job details known before dispatch, had a mean absolute error of 1.71 hours; the FLOP baseline at 30% MFU had an error of 12.41 hours. The authors attribute the gap to configuration signals and interactions that a compute-only estimate misses, including evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime. Feature importance placed evaluation volume and checkpoint-related variables among prominent inputs, but the authors caution that feature importance is not a causal claim. Their practical point is that setup, evaluation, checkpoint uploads and shared network and storage I/O can make jobs with the same nominal compute finish hours apart. They say better estimates could help a scheduler plan GPU assignments and when capacity will become available. The study covers SFT on H200s, and the authors report wider errors for runs over 60 hours, where data is sparse. They say validation on other GPU types and more long-run data are still needed.
Original source
· Article
Source details
Original source date: Nov 18, 2026 (2026-11-18T18:53:43.959Z).
Date basis: Date stated by the publisher.
Added to Escape Meridian: 2026-11-18T19:03:08.837Z. First observed: 2026-11-18T18:55:12.408Z. Retrieved: 2026-11-18T19:01:53.595Z.
Limits
The study covers SFT only; it does not include other training and alignment strategies such as , DPO or RLVR. Every run used . The authors say the fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other generations without retraining. The authors report only 13 runs above 60 hours and say errors widen in this sparse long-duration range. The authors describe feature importance as non-causal and caution that GPU allocation can act as a proxy for job scale in their data.
Open question
How well would the predictor perform on other types, once validated there as the authors plan?
Source-stated
The authors compared FLOP-based estimates and learned predictors using 272 production SFT jobs on NVIDIA H200 GPUs.
So we took 272 real supervised fine-tuning (SFT) jobs from our production pool of NVIDIA H200 GPUs
Source-stated
In 5-fold cross-validation, the CatBoost regressor had a mean absolute error of 1.71 hours; the 30% MFU FLOP baseline had a mean absolute error of 12.41 hours.
FLOP baseline (MFU 30%) 0.31 20.45 12.41 FLOP baseline (MFU 45%)
Source-stated
The predictor uses job configuration and model metadata available before dispatch, rather than runtime logs or partial progress.
Everything is known before dispatch. Every input comes from the submitted configuration: hyperparameters, dataset sizes, model identifiers, GPU allocation. No runtime logs, no partial progress.
Source-stated
The authors say the model uses training-specific signals, including evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and learns nonlinear interactions between them.
The regressor sees more training-specific signals, such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and it learns the nonlinear interactions between them from past runs.
Source-stated
The authors say feature importance is not causal evidence, while noting that evaluation and checkpoint choices shape wall-clock time in their data.
Feature importance is not a causal claim. Still, the pattern lines up with how a job is actually spent.
Source-stated
The authors say errors widen above 60 hours and regard the small number of such runs as a data limitation.
The long tail is where the model struggles, and with only 13 runs above 60 hours we read that as a data limitation.
Source-stated
The authors say the fitted model is calibrated to H200 infrastructure and should not be expected to transfer numerically to other GPU generations without retraining.
The fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other GPU generations without retraining.
Source and contributor context
Hugging Face · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Fine-tuning
Further training of an existing model to adapt it to a particular task or use.
Floating-point operations
A count of basic arithmetic steps; used to measure how much computation went into training a model.
NVIDIA H200
An NVIDIA GPU built on the Hopper architecture with 141 gigabytes of memory at 4.8 terabytes per second: nearly double the H100's memory capacity, with 1.4 times its memory bandwidth.
Graphics processing unit
A chip that performs many calculations at once; the main hardware used to train and run AI models.
Reinforcement learning from human feedback
Training that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.
Hugging Face
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 360aa19b-416c-4316-84b9-3eae3cf88201.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T19:03:08.837Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: a904ab71a4e15bd1cb03ca95cd0514e5b2736659c122d8fcb1acb5419600bfeb. Source text hash: 7b0e97443219999a4981d5dad8c4a6c777cec1d7f86e7e5c2a0371e03939a8b8.
passed · 2026-11-18T19:03:08.837Z · anthropic/claude-opus-5.5
Publication: 360aa19b-416c-4316-84b9-3eae3cf88201
attribution — pass: The reading attributes the argument and results to the Dharma-AI team. The source byline lists Dharma-AI authors, and the post is a team article. The findings are consistently framed as the authors' own claims.
A preview of our upcoming paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models.
summary — pass: The reading leads with the authors' thesis that compute-only estimates miss time spent outside the training loop. It then gives the evidence, mechanism, scheduling use and limits. FLOP and MFU are left to tooltips, but the surrounding text conveys the meaning. The question about performance on other GPU types follows from the authors' stated next steps. Not mentioning that this is a preview of a not-yet-posted paper is a minor omission, not a distortion.
Two jobs with the same nominal compute can finish hours apart.
date — pass: The originalPublished value of 2026-11-18 from the publisher page matches the post's stated publication date and the feed entry.
Published November 18, 2026
support — pass: All figures are source-stated: 272 runs, H200 GPUs, MAE of 1.71 h for CatBoost versus 12.41 h for the 30% MFU baseline, the before-dispatch inputs, the non-causal caveat, sparse runs over 60 h and the transfer limit. The scheduler benefit is framed as the authors' claim. The claim-2 excerpt shows only the baseline row, but CatBoost's 1.71 h appears elsewhere in the source.
A CatBoost regressor that only sees information available before dispatch reaches R² = 0.98, with a mean absolute error of 1.71 h and a median error of 0.51 h in 5-fold cross-validation.
media — pass: The selected media is the verified link to the original Hugging Face blog post itself. This is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceA count of basic arithmetic steps; used to measure how much computation went into training a model.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEFurther training of an existing model to adapt it to a particular task or use.
Definition referenceAn NVIDIA GPU built on the Hopper architecture with 141 gigabytes of memory at 4.8 terabytes per second: nearly double the H100's memory capacity, with 1.4 times its memory bandwidth.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceHugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceTraining that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.
Definition referenceAn NVIDIA GPU built on the Hopper architecture with 141 gigabytes of memory at 4.8 terabytes per second: nearly double the H100's memory capacity, with 1.4 times its memory bandwidth.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceA count of basic arithmetic steps; used to measure how much computation went into training a model.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEFurther training of an existing model to adapt it to a particular task or use.
Definition referenceAn NVIDIA GPU built on the Hopper architecture with 141 gigabytes of memory at 4.8 terabytes per second: nearly double the H100's memory capacity, with 1.4 times its memory bandwidth.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceHugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceTraining that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.
Definition referenceAn NVIDIA GPU built on the Hopper architecture with 141 gigabytes of memory at 4.8 terabytes per second: nearly double the H100's memory capacity, with 1.4 times its memory bandwidth.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition referenceA chip that performs many calculations at once; the main hardware used to train and run AI models.
Definition reference