{"id":"fae170555331c6939199c89f647845f45818bf4b3f04018e44bc4b8e008d08e7","title":"Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass","argument":"The paper’s authors propose Layerwise Repacking for variable-length training of hybrid-attention models, where full-attention and sparse-attention layers have different parallelism needs. Their method keeps wide sequence-parallel (SP) groups for full attention, then repacks samples into smaller, node-local groups for sparse-attention layers. The change is combined with existing data transfers, rather than adding a separate redistribution pass; an auxiliary parent solver adjusts which samples share a parent group because that affects how well the smaller groups can balance work.\n\nThe authors report results from all 22 steps with SP16/32 parents among the first 200 training steps of a 1F3S hybrid-attention model. Repacking alone reduced mean end-to-end time by 5.95% and raised model FLOPs utilization (MFU) by 3.75 percentage points compared with a baseline without repacking. With the improved solver, they report a 7.61% time reduction and a 4.83 percentage-point MFU increase. These results describe the reported training run; the abstract does not establish whether the gains carry over to other models, workloads, or longer training runs. How does Layerwise Repacking perform beyond the reported steps and model?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported results cover all 22 steps with SP16/32 parents among the first 200 training steps of one 1F3S hybrid-attention model; the abstract does not report results for other models, workloads, or longer runs.","speaker":{"name":"Zhang, Jiahao, Feng, Junwei, Huang, Dongliang, Zhou, Xiaoling, Liao, Yiqiao, Song, Chengru","roleAtSource":"Article author"},"source":{"title":"Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.35989","publishedAt":"2026-10-26","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors say full-attention and sparse-attention layers have different parallelism needs in variable-length training, motivating separate group sizes.","excerpt":"Variable-length training creates conflicting parallelism requirements across the layers of a hybrid-attention model.","url":"https://arxiv.org/abs/2610.35989","relationship":"Source-stated"},{"text":"Layerwise Repacking reuses existing full-layer Ulysses sites to combine head exchange and sample repacking in one variable-size all-to-all, avoiding an added redistribution pass.","excerpt":"at each, one variable-size all-to-all performs both the head exchange and the sample repacking, so no redistribution pass is added.","url":"https://arxiv.org/abs/2610.35989","relationship":"Source-stated"},{"text":"The authors report that, on all 22 steps with SP16/32 parents among the first 200 training steps of a 1F3S model, repacking alone reduced mean end-to-end time by 5.95% and raised MFU by 3.75 percentage points over the baseline.","excerpt":"repacking alone reduces the mean end-to-end time by 5.95% and raises model FLOPs utilization (MFU) by 3.75 percentage points","url":"https://arxiv.org/abs/2610.35989","relationship":"Source-stated"},{"text":"With the improved solver, the authors report a 7.61% reduction in end-to-end time and a 4.83 percentage-point increase in MFU.","excerpt":"with the improved solver, end-to-end time falls by 7.61% and MFU rises by 4.83 percentage points.","url":"https://arxiv.org/abs/2610.35989","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.35989","checkedAt":"2026-11-14T00:26:08.721Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How does Layerwise Repacking perform beyond the reported steps and model?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass","summary":"The paper’s authors propose Layerwise Repacking for variable-length training of hybrid-attention models, where full-attention and sparse-attention layers have different parallelism needs. Their method keeps wide sequence-parallel (SP) groups for full attention, then repacks samples into smaller, node-local groups for sparse-attention layers. The change is combined with existing data transfers, rather than adding a separate redistribution pass; an auxiliary parent solver adjusts which samples share a parent group because that affects how well the smaller groups can balance work.\n\nThe authors report results from all 22 steps with SP16/32 parents among the first 200 training steps of a 1F3S hybrid-attention model. Repacking alone reduced mean end-to-end time by 5.95% and raised model FLOPs utilization (MFU) by 3.75 percentage points compared with a baseline without repacking. With the improved solver, they report a 7.61% time reduction and a 4.83 percentage-point MFU increase. These results describe the reported training run; the abstract does not establish whether the gains carry over to other models, workloads, or longer training runs. How does Layerwise Repacking perform beyond the reported steps and model?","question":"How does Layerwise Repacking perform beyond the reported steps and model?","claims":[{"text":"The authors say full-attention and sparse-attention layers have different parallelism needs in variable-length training, motivating separate group sizes.","excerpt":"Variable-length training creates conflicting parallelism requirements across the layers of a hybrid-attention model.","support":"source-stated"},{"text":"Layerwise Repacking reuses existing full-layer Ulysses sites to combine head exchange and sample repacking in one variable-size all-to-all, avoiding an added redistribution pass.","excerpt":"at each, one variable-size all-to-all performs both the head exchange and the sample repacking, so no redistribution pass is added.","support":"source-stated"},{"text":"The authors report that, on all 22 steps with SP16/32 parents among the first 200 training steps of a 1F3S model, repacking alone reduced mean end-to-end time by 5.95% and raised MFU by 3.75 percentage points over the baseline.","excerpt":"repacking alone reduces the mean end-to-end time by 5.95% and raises model FLOPs utilization (MFU) by 3.75 percentage points","support":"source-stated"},{"text":"With the improved solver, the authors report a 7.61% reduction in end-to-end time and a 4.83 percentage-point increase in MFU.","excerpt":"with the improved solver, end-to-end time falls by 7.61% and MFU rises by 4.83 percentage points.","support":"source-stated"}],"limitations":["The reported results cover all 22 steps with SP16/32 parents among the first 200 training steps of one 1F3S hybrid-attention model; the abstract does not report results for other models, workloads, or longer runs."],"mediaId":"f68066d3bee4b71ed289ec28cff1f95b09a044e8eba0a5720365152ca779fecd","correctionReason":null,"id":"fae170555331c6939199c89f647845f45818bf4b3f04018e44bc4b8e008d08e7","publicationId":"d3abeaf1-306d-416e-838e-31e8a0fb083f","version":1,"sourceUrl":"https://arxiv.org/abs/2610.35989","sourceTitle":"Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-26","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T07:08:40.477Z","createdAt":"2026-11-13T22:19:58.626Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"f68066d3bee4b71ed289ec28cff1f95b09a044e8eba0a5720365152ca779fecd","kind":"article","url":"https://arxiv.org/abs/2610.35989","sourceUrl":"https://arxiv.org/abs/2610.35989","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T23:18:33.681Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.35989","checkedAt":"2026-11-14T00:26:08.721Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"f68066d3bee4b71ed289ec28cff1f95b09a044e8eba0a5720365152ca779fecd","confidence":0.9,"provider":"typesafe","model":"jev-1.13.0","inputHash":"a1fc59848da0aa6b54086508a686fa6f188b85a9f3cb4136ef202ecc76aca705","decidedAt":"2026-10-30T23:18:36.504Z"},"provenance":{"textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observationId":"ccf231adf9fe12da31371e7f52b4fdd49493fd8a71fc06484e45ab77ce309c19","retrievedAt":"2026-10-30T23:18:33.256Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"5023ae35-431d-4b85-81a7-6eeeb5732655"},"reviewer":{"model":"gpt-6-luna","runId":"62516c5a-d969-497e-859e-4cea11614bc1"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-13T22:19:58.626Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Zhang, Jiahao","url":null,"kind":"unknown"},{"name":"Feng, Junwei","url":null,"kind":"unknown"},{"name":"Huang, Dongliang","url":null,"kind":"unknown"},{"name":"Zhou, Xiaoling","url":null,"kind":"unknown"},{"name":"Liao, Yiqiao","url":null,"kind":"unknown"},{"name":"Song, Chengru","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Zhang, Jiahao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Jiahao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35989","locator":"head/meta[name=author|citation_author]"},"assertionId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:08.721Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","sourceUrl":"https://arxiv.org/abs/2610.35989","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z"}},{"actor":{"name":"Feng, Junwei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Feng, Junwei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35989","locator":"head/meta[name=author|citation_author]"},"assertionId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:08.721Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","sourceUrl":"https://arxiv.org/abs/2610.35989","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z"}},{"actor":{"name":"Huang, Dongliang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Huang, Dongliang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35989","locator":"head/meta[name=author|citation_author]"},"assertionId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:08.721Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","sourceUrl":"https://arxiv.org/abs/2610.35989","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z"}},{"actor":{"name":"Zhou, Xiaoling","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Xiaoling","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35989","locator":"head/meta[name=author|citation_author]"},"assertionId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:08.721Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","sourceUrl":"https://arxiv.org/abs/2610.35989","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z"}},{"actor":{"name":"Liao, Yiqiao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liao, Yiqiao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35989","locator":"head/meta[name=author|citation_author]"},"assertionId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:08.721Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","sourceUrl":"https://arxiv.org/abs/2610.35989","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z"}},{"actor":{"name":"Song, Chengru","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Song, Chengru","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35989","locator":"head/meta[name=author|citation_author]"},"assertionId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:08.721Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","sourceUrl":"https://arxiv.org/abs/2610.35989","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.35989","snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/fae170555331c6939199c89f647845f45818bf4b3f04018e44bc4b8e008d08e7/source"}},"canonicalUrl":"https://escapemeridian.com/readings/layerwise-repacking-variable-length-hybrid-attention-training-fae17055","formats":{"html":"/readings/layerwise-repacking-variable-length-hybrid-attention-training-fae17055","markdown":"/readings/layerwise-repacking-variable-length-hybrid-attention-training-fae17055.md","json":"/readings/layerwise-repacking-variable-length-hybrid-attention-training-fae17055.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-flops","name":"Floating-point operations","explanation":"A count of basic arithmetic steps; used to measure how much computation went into training a model.","referenceUrl":"https://en.wikipedia.org/wiki/FLOPS"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"time","name":"TIME","kind":"organization","context":"TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.","referenceUrls":["https://time.com/"],"path":"/entities/time","json":"/entities/time.json","markdown":"/entities/time.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"fae170555331c6939199c89f647845f45818bf4b3f04018e44bc4b8e008d08e7","revision":"d3abeaf1-306d-416e-838e-31e8a0fb083f","revisionKind":"publication-id","title":"Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass","url":"https://escapemeridian.com/readings/layerwise-repacking-variable-length-hybrid-attention-training-fae17055"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.35989","revision":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","revisionKind":"sha256-source-text","title":"Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass","url":"https://arxiv.org/abs/2610.35989"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"fae170555331c6939199c89f647845f45818bf4b3f04018e44bc4b8e008d08e7","title":"Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass","argument":"The paper’s authors propose Layerwise Repacking for variable-length training of hybrid-attention models, where full-attention and sparse-attention layers have different parallelism needs. Their method keeps wide sequence-parallel (SP) groups for full attention, then repacks samples into smaller, node-local groups for sparse-attention layers. The change is combined with existing data transfers, rather than adding a separate redistribution pass; an auxiliary parent solver adjusts which samples share a parent group because that affects how well the smaller groups can balance work.\n\nThe authors report results from all 22 steps with SP16/32 parents among the first 200 training steps of a 1F3S hybrid-attention model. Repacking alone reduced mean end-to-end time by 5.95% and raised model FLOPs utilization (MFU) by 3.75 percentage points compared with a baseline without repacking. With the improved solver, they report a 7.61% time reduction and a 4.83 percentage-point MFU increase. These results describe the reported training run; the abstract does not establish whether the gains carry over to other models, workloads, or longer training runs. How does Layerwise Repacking perform beyond the reported steps and model?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported results cover all 22 steps with SP16/32 parents among the first 200 training steps of one 1F3S hybrid-attention model; the abstract does not report results for other models, workloads, or longer runs.","speaker":{"name":"Zhang, Jiahao, Feng, Junwei, Huang, Dongliang, Zhou, Xiaoling, Liao, Yiqiao, Song, Chengru","roleAtSource":"Article author"},"source":{"title":"Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.35989","publishedAt":"2026-10-26","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors say full-attention and sparse-attention layers have different parallelism needs in variable-length training, motivating separate group sizes.","excerpt":"Variable-length training creates conflicting parallelism requirements across the layers of a hybrid-attention model.","url":"https://arxiv.org/abs/2610.35989","relationship":"Source-stated"},{"text":"Layerwise Repacking reuses existing full-layer Ulysses sites to combine head exchange and sample repacking in one variable-size all-to-all, avoiding an added redistribution pass.","excerpt":"at each, one variable-size all-to-all performs both the head exchange and the sample repacking, so no redistribution pass is added.","url":"https://arxiv.org/abs/2610.35989","relationship":"Source-stated"},{"text":"The authors report that, on all 22 steps with SP16/32 parents among the first 200 training steps of a 1F3S model, repacking alone reduced mean end-to-end time by 5.95% and raised MFU by 3.75 percentage points over the baseline.","excerpt":"repacking alone reduces the mean end-to-end time by 5.95% and raises model FLOPs utilization (MFU) by 3.75 percentage points","url":"https://arxiv.org/abs/2610.35989","relationship":"Source-stated"},{"text":"With the improved solver, the authors report a 7.61% reduction in end-to-end time and a 4.83 percentage-point increase in MFU.","excerpt":"with the improved solver, end-to-end time falls by 7.61% and MFU rises by 4.83 percentage points.","url":"https://arxiv.org/abs/2610.35989","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.35989","checkedAt":"2026-11-14T00:26:08.721Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How does Layerwise Repacking perform beyond the reported steps and model?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass","summary":"The paper’s authors propose Layerwise Repacking for variable-length training of hybrid-attention models, where full-attention and sparse-attention layers have different parallelism needs. Their method keeps wide sequence-parallel (SP) groups for full attention, then repacks samples into smaller, node-local groups for sparse-attention layers. The change is combined with existing data transfers, rather than adding a separate redistribution pass; an auxiliary parent solver adjusts which samples share a parent group because that affects how well the smaller groups can balance work.\n\nThe authors report results from all 22 steps with SP16/32 parents among the first 200 training steps of a 1F3S hybrid-attention model. Repacking alone reduced mean end-to-end time by 5.95% and raised model FLOPs utilization (MFU) by 3.75 percentage points compared with a baseline without repacking. With the improved solver, they report a 7.61% time reduction and a 4.83 percentage-point MFU increase. These results describe the reported training run; the abstract does not establish whether the gains carry over to other models, workloads, or longer training runs. How does Layerwise Repacking perform beyond the reported steps and model?","question":"How does Layerwise Repacking perform beyond the reported steps and model?","claims":[{"text":"The authors say full-attention and sparse-attention layers have different parallelism needs in variable-length training, motivating separate group sizes.","excerpt":"Variable-length training creates conflicting parallelism requirements across the layers of a hybrid-attention model.","support":"source-stated"},{"text":"Layerwise Repacking reuses existing full-layer Ulysses sites to combine head exchange and sample repacking in one variable-size all-to-all, avoiding an added redistribution pass.","excerpt":"at each, one variable-size all-to-all performs both the head exchange and the sample repacking, so no redistribution pass is added.","support":"source-stated"},{"text":"The authors report that, on all 22 steps with SP16/32 parents among the first 200 training steps of a 1F3S model, repacking alone reduced mean end-to-end time by 5.95% and raised MFU by 3.75 percentage points over the baseline.","excerpt":"repacking alone reduces the mean end-to-end time by 5.95% and raises model FLOPs utilization (MFU) by 3.75 percentage points","support":"source-stated"},{"text":"With the improved solver, the authors report a 7.61% reduction in end-to-end time and a 4.83 percentage-point increase in MFU.","excerpt":"with the improved solver, end-to-end time falls by 7.61% and MFU rises by 4.83 percentage points.","support":"source-stated"}],"limitations":["The reported results cover all 22 steps with SP16/32 parents among the first 200 training steps of one 1F3S hybrid-attention model; the abstract does not report results for other models, workloads, or longer runs."],"mediaId":"f68066d3bee4b71ed289ec28cff1f95b09a044e8eba0a5720365152ca779fecd","correctionReason":null,"id":"fae170555331c6939199c89f647845f45818bf4b3f04018e44bc4b8e008d08e7","publicationId":"d3abeaf1-306d-416e-838e-31e8a0fb083f","version":1,"sourceUrl":"https://arxiv.org/abs/2610.35989","sourceTitle":"Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-26","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T07:08:40.477Z","createdAt":"2026-11-13T22:19:58.626Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"f68066d3bee4b71ed289ec28cff1f95b09a044e8eba0a5720365152ca779fecd","kind":"article","url":"https://arxiv.org/abs/2610.35989","sourceUrl":"https://arxiv.org/abs/2610.35989","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T23:18:33.681Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.35989","checkedAt":"2026-11-14T00:26:08.721Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"f68066d3bee4b71ed289ec28cff1f95b09a044e8eba0a5720365152ca779fecd","confidence":0.9,"provider":"typesafe","model":"jev-1.13.0","inputHash":"a1fc59848da0aa6b54086508a686fa6f188b85a9f3cb4136ef202ecc76aca705","decidedAt":"2026-10-30T23:18:36.504Z"},"provenance":{"textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observationId":"ccf231adf9fe12da31371e7f52b4fdd49493fd8a71fc06484e45ab77ce309c19","retrievedAt":"2026-10-30T23:18:33.256Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"5023ae35-431d-4b85-81a7-6eeeb5732655"},"reviewer":{"model":"gpt-6-luna","runId":"62516c5a-d969-497e-859e-4cea11614bc1"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-13T22:19:58.626Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Zhang, Jiahao","url":null,"kind":"unknown"},{"name":"Feng, Junwei","url":null,"kind":"unknown"},{"name":"Huang, Dongliang","url":null,"kind":"unknown"},{"name":"Zhou, Xiaoling","url":null,"kind":"unknown"},{"name":"Liao, Yiqiao","url":null,"kind":"unknown"},{"name":"Song, Chengru","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Zhang, Jiahao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Jiahao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35989","locator":"head/meta[name=author|citation_author]"},"assertionId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:08.721Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","sourceUrl":"https://arxiv.org/abs/2610.35989","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z"}},{"actor":{"name":"Feng, Junwei","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Feng, Junwei","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35989","locator":"head/meta[name=author|citation_author]"},"assertionId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:08.721Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","sourceUrl":"https://arxiv.org/abs/2610.35989","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z"}},{"actor":{"name":"Huang, Dongliang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Huang, Dongliang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35989","locator":"head/meta[name=author|citation_author]"},"assertionId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:08.721Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","sourceUrl":"https://arxiv.org/abs/2610.35989","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z"}},{"actor":{"name":"Zhou, Xiaoling","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhou, Xiaoling","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35989","locator":"head/meta[name=author|citation_author]"},"assertionId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:08.721Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","sourceUrl":"https://arxiv.org/abs/2610.35989","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z"}},{"actor":{"name":"Liao, Yiqiao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liao, Yiqiao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35989","locator":"head/meta[name=author|citation_author]"},"assertionId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:08.721Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","sourceUrl":"https://arxiv.org/abs/2610.35989","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z"}},{"actor":{"name":"Song, Chengru","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Song, Chengru","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.35989","locator":"head/meta[name=author|citation_author]"},"assertionId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-14T00:26:08.721Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","sourceUrl":"https://arxiv.org/abs/2610.35989","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.35989","snapshotId":"2331134ea26036c28183d14566a8a2a835215140466e4fb8a7954f38652692c9","contentSha256":"1868a9867b0ac98d57c3c1b3cd4d7b6799d3d00ecbaaca0f133b999a8f72c073","textHash":"49c8050c20bc0be9a87eaed5e473cfe6f9bfeb12ee549ad18bf6431ae22b3c67","observedAt":"2026-11-14T00:26:08.721Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/fae170555331c6939199c89f647845f45818bf4b3f04018e44bc4b8e008d08e7/source"}}}],"audits":[{"publicationId":"d3abeaf1-306d-416e-838e-31e8a0fb083f","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the method and results to the paper's authors with 'propose' and 'report', and does not present the speedup and MFU figures as established facts.","excerpt":"Authors:Jiahao Zhang, Junwei Feng, Dongliang Huang, Xiaoling Zhou, Yiqiao Liao, Chengru Song"},{"dimension":"summary","status":"uncertain","explanation":"Accurate, but hard for a non-specialist. Central terms are unexplained: hybrid-attention, sequence-parallel, node-local, MFU, and 1F3S. The source itself defines 1F3S. The reading also drops the core reason for the method: cross-node all-to-all communication was observed to be 7.0x slower, so smaller node-local groups avoid that cost. It never says why faster training matters. Smallest repair: add the 7.0x cross-node rationale, gloss 1F3S as the source does, and add term-library tooltips for SP, MFU, and hybrid attention. The final paragraph also repeats the question verbatim.","excerpt":"Reusing the wide groups for these layers pushes their all-to-all communication across nodes, where the observed effective rate is 7.0x lower."},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-10-26, which matches the arXiv v1 submission date. The feed date of Oct 29 is later and is correctly not used.","excerpt":"[Submitted on 26 Oct 2026]"},{"dimension":"support","status":"pass","explanation":"All claims and figures match the abstract: 22 steps, the first 200 steps, 5.95% and 3.75 pp, and 7.61% and 4.83 pp. The mechanism of a single variable-size all-to-all at the existing Ulysses sites is also accurate. The limitation and the question are appropriately conditional and do not invent benefits or costs.","excerpt":"with the improved solver, end-to-end time falls by 7.61% and MFU rises by 4.83 percentage points."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper, which is the most direct way to reach the source.","excerpt":null}],"createdAt":"2026-11-13T22:19:58.626Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}