# Dharma-AI: Why your fine-tuning job takes longer than the FLOPs say

[Canonical reading](https://escapemeridian.com/readings/dharma-ai-why-your-fine-tuning-job-takes-longer-than-the-flops-cb624136)

The Dharma-AI team argues that FLOP-based estimates miss much of the time a fine-tuning job spends outside its training loop. They compared estimates on 272 production supervised fine-tuning (SFT) runs using NVIDIA H200 GPUs. In 5-fold cross-validation, their CatBoost regressor, using job details known before dispatch, had a mean absolute error of 1.71 hours; the FLOP baseline at 30% MFU had an error of 12.41 hours. The authors attribute the gap to configuration signals and interactions that a compute-only estimate misses, including evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime. Feature importance placed evaluation volume and checkpoint-related variables among prominent inputs, but the authors caution that feature importance is not a causal claim. Their practical point is that setup, evaluation, checkpoint uploads and shared network and storage I/O can make jobs with the same nominal compute finish hours apart. They say better estimates could help a scheduler plan GPU assignments and when capacity will become available. The study covers SFT on H200s, and the authors report wider errors for runs over 60 hours, where data is sparse. They say validation on other GPU types and more long-run data are still needed.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

Hugging Face · Article

Original source date: Nov 18, 2026 (2026-11-18T18:53:43.959Z).

Date basis: Date stated by the publisher.

Added to Escape Meridian: 2026-11-18T19:03:08.837Z. First observed: 2026-11-18T18:55:12.408Z. Retrieved: 2026-11-18T19:01:53.595Z.

- [Why your fine-tuning job takes longer than the FLOPs say](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Limits

The study covers SFT only; it does not include other training and alignment strategies such as RLHF, DPO or RLVR. Every run used H200s. The authors say the fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other GPU generations without retraining. The authors report only 13 runs above 60 hours and say errors widen in this sparse long-duration range. The authors describe feature importance as non-causal and caution that GPU allocation can act as a proxy for job scale in their data.


## Open question

How well would the predictor perform on other GPU types, once validated there as the authors plan?


## Source-stated

The authors compared FLOP-based estimates and learned predictors using 272 production SFT jobs on NVIDIA H200 GPUs.

> So we took 272 real supervised fine-tuning (SFT) jobs from our production pool of NVIDIA H200 GPUs

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source-stated

In 5-fold cross-validation, the CatBoost regressor had a mean absolute error of 1.71 hours; the 30% MFU FLOP baseline had a mean absolute error of 12.41 hours.

> FLOP baseline (MFU 30%)
> 0.31
> 20.45
> 12.41
> FLOP baseline (MFU 45%)

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source-stated

The predictor uses job configuration and model metadata available before dispatch, rather than runtime logs or partial progress.

> Everything is known before dispatch. Every input comes from the submitted configuration: hyperparameters, dataset sizes, model identifiers, GPU allocation. No runtime logs, no partial progress.

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source-stated

The authors say the model uses training-specific signals, including evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and learns nonlinear interactions between them.

> The regressor sees more training-specific signals, such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and it learns the nonlinear interactions between them from past runs.

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source-stated

The authors say feature importance is not causal evidence, while noting that evaluation and checkpoint choices shape wall-clock time in their data.

> Feature importance is not a causal claim. Still, the pattern lines up with how a job is actually spent.

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source-stated

The authors say errors widen above 60 hours and regard the small number of such runs as a data limitation.

> The long tail is where the model struggles, and with only 13 runs above 60 hours we read that as a data limitation.

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source-stated

The authors say the fitted model is calibrated to H200 infrastructure and should not be expected to transfer numerically to other GPU generations without retraining.

> The fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other GPU generations without retraining.

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source and contributor context

Hugging Face · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Fine-tuning

Further training of an existing model to adapt it to a particular task or use.

- [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning)

## Floating-point operations

A count of basic arithmetic steps; used to measure how much computation went into training a model.

- [Definition reference](https://en.wikipedia.org/wiki/FLOPS)

## NVIDIA H200

An NVIDIA GPU built on the Hopper architecture with 141 gigabytes of memory at 4.8 terabytes per second: nearly double the H100's memory capacity, with 1.4 times its memory bandwidth.

- [Definition reference](https://www.nvidia.com/en-us/data-center/h200/)

## Graphics processing unit

A chip that performs many calculations at once; the main hardware used to train and run AI models.

- [Definition reference](https://en.wikipedia.org/wiki/Graphics_processing_unit)

## Reinforcement learning from human feedback

Training that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.

- [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-from-human-feedback-rlhf)

## Hugging Face

Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.

- [Explore this context](/entities/hugging-face)
- [Reference](https://huggingface.co/blog)

## TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

- [Explore this context](/entities/time)
- [Reference](https://time.com/)

## Editorial provenance

Version 1. Publication ID: 360aa19b-416c-4316-84b9-3eae3cf88201.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T19:03:08.837Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: a904ab71a4e15bd1cb03ca95cd0514e5b2736659c122d8fcb1acb5419600bfeb. Source text hash: 7b0e97443219999a4981d5dad8c4a6c777cec1d7f86e7e5c2a0371e03939a8b8.


## Revision and source-check history

### Version 1 — 2026-11-18T19:03:08.837Z

The Dharma-AI team argues that FLOP-based estimates miss much of the time a fine-tuning job spends outside its training loop. They compared estimates on 272 production supervised fine-tuning (SFT) runs using NVIDIA H200 GPUs. In 5-fold cross-validation, their CatBoost regressor, using job details known before dispatch, had a mean absolute error of 1.71 hours; the FLOP baseline at 30% MFU had an error of 12.41 hours. The authors attribute the gap to configuration signals and interactions that a compute-only estimate misses, including evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime. Feature importance placed evaluation volume and checkpoint-related variables among prominent inputs, but the authors caution that feature importance is not a causal claim. Their practical point is that setup, evaluation, checkpoint uploads and shared network and storage I/O can make jobs with the same nominal compute finish hours apart. They say better estimates could help a scheduler plan GPU assignments and when capacity will become available. The study covers SFT on H200s, and the authors report wider errors for runs over 60 hours, where data is sparse. They say validation on other GPU types and more long-run data are still needed.

## Original source

Hugging Face · Article

Original source date: Nov 18, 2026 (2026-11-18T18:53:43.959Z).

Date basis: Date stated by the publisher.

Added to Escape Meridian: 2026-11-18T19:03:08.837Z. First observed: 2026-11-18T18:55:12.408Z. Retrieved: 2026-11-18T19:01:53.595Z.

- [Why your fine-tuning job takes longer than the FLOPs say](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Limits

The study covers SFT only; it does not include other training and alignment strategies such as RLHF, DPO or RLVR. Every run used H200s. The authors say the fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other GPU generations without retraining. The authors report only 13 runs above 60 hours and say errors widen in this sparse long-duration range. The authors describe feature importance as non-causal and caution that GPU allocation can act as a proxy for job scale in their data.


## Open question

How well would the predictor perform on other GPU types, once validated there as the authors plan?


## Source-stated

The authors compared FLOP-based estimates and learned predictors using 272 production SFT jobs on NVIDIA H200 GPUs.

> So we took 272 real supervised fine-tuning (SFT) jobs from our production pool of NVIDIA H200 GPUs

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source-stated

In 5-fold cross-validation, the CatBoost regressor had a mean absolute error of 1.71 hours; the 30% MFU FLOP baseline had a mean absolute error of 12.41 hours.

> FLOP baseline (MFU 30%)
> 0.31
> 20.45
> 12.41
> FLOP baseline (MFU 45%)

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source-stated

The predictor uses job configuration and model metadata available before dispatch, rather than runtime logs or partial progress.

> Everything is known before dispatch. Every input comes from the submitted configuration: hyperparameters, dataset sizes, model identifiers, GPU allocation. No runtime logs, no partial progress.

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source-stated

The authors say the model uses training-specific signals, including evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and learns nonlinear interactions between them.

> The regressor sees more training-specific signals, such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and it learns the nonlinear interactions between them from past runs.

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source-stated

The authors say feature importance is not causal evidence, while noting that evaluation and checkpoint choices shape wall-clock time in their data.

> Feature importance is not a causal claim. Still, the pattern lines up with how a job is actually spent.

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source-stated

The authors say errors widen above 60 hours and regard the small number of such runs as a data limitation.

> The long tail is where the model struggles, and with only 13 runs above 60 hours we read that as a data limitation.

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source-stated

The authors say the fitted model is calibrated to H200 infrastructure and should not be expected to transfer numerically to other GPU generations without retraining.

> The fitted model is calibrated to that infrastructure and should not be expected to transfer numerically to other GPU generations without retraining.

- [Supporting source](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time)

## Source and contributor context

Hugging Face · Publisher

The publisher is identified here. No individual author has been inferred.


## Source media

Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Fine-tuning

Further training of an existing model to adapt it to a particular task or use.

- [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning)

## Floating-point operations

A count of basic arithmetic steps; used to measure how much computation went into training a model.

- [Definition reference](https://en.wikipedia.org/wiki/FLOPS)

## NVIDIA H200

An NVIDIA GPU built on the Hopper architecture with 141 gigabytes of memory at 4.8 terabytes per second: nearly double the H100's memory capacity, with 1.4 times its memory bandwidth.

- [Definition reference](https://www.nvidia.com/en-us/data-center/h200/)

## Graphics processing unit

A chip that performs many calculations at once; the main hardware used to train and run AI models.

- [Definition reference](https://en.wikipedia.org/wiki/Graphics_processing_unit)

## Reinforcement learning from human feedback

Training that rewards a model for answers people rate highly, so it learns the kind of answers people prefer.

- [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-from-human-feedback-rlhf)

## Hugging Face

Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.

- [Explore this context](/entities/hugging-face)
- [Reference](https://huggingface.co/blog)

## TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

- [Explore this context](/entities/time)
- [Reference](https://time.com/)

## Editorial provenance

Version 1. Publication ID: 360aa19b-416c-4316-84b9-3eae3cf88201.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T19:03:08.837Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: a904ab71a4e15bd1cb03ca95cd0514e5b2736659c122d8fcb1acb5419600bfeb. Source text hash: 7b0e97443219999a4981d5dad8c4a6c777cec1d7f86e7e5c2a0371e03939a8b8.


### passed — 2026-11-18T19:03:08.837Z — anthropic/claude-opus-5.5

Publication: 360aa19b-416c-4316-84b9-3eae3cf88201

attribution — pass: The reading attributes the argument and results to the Dharma-AI team. The source byline lists Dharma-AI authors, and the post is a team article. The findings are consistently framed as the authors' own claims.

> A preview of our upcoming paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models.

summary — pass: The reading leads with the authors' thesis that compute-only estimates miss time spent outside the training loop. It then gives the evidence, mechanism, scheduling use and limits. FLOP and MFU are left to tooltips, but the surrounding text conveys the meaning. The question about performance on other GPU types follows from the authors' stated next steps. Not mentioning that this is a preview of a not-yet-posted paper is a minor omission, not a distortion.

> Two jobs with the same nominal compute can finish hours apart.

date — pass: The originalPublished value of 2026-11-18 from the publisher page matches the post's stated publication date and the feed entry.

> Published
November 18, 2026

support — pass: All figures are source-stated: 272 runs, H200 GPUs, MAE of 1.71 h for CatBoost versus 12.41 h for the 30% MFU baseline, the before-dispatch inputs, the non-causal caveat, sparse runs over 60 h and the transfer limit. The scheduler benefit is framed as the authors' claim. The claim-2 excerpt shows only the baseline row, but CatBoost's 1.71 h appears elsewhere in the source.

> A CatBoost regressor that only sees information available before dispatch reaches R² = 0.98, with a mean absolute error of 1.71 h and a median error of 0.51 h in 5-fold cross-validation.

media — pass: The selected media is the verified link to the original Hugging Face blog post itself. This is the most direct access to the source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/cb62413601224081ebc27349bcc9f6414daf164016f5a84bae6a9730a80db640/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/dharma-ai-why-your-fine-tuning-job-takes-longer-than-the-flops-cb624136.json)
