Sun, Xiaofei, Wang, Peng, Liu, Huayao, Xu, Xiaoxiao, Song, Jun, Li, Deqing, Qu, Lin · Oct 20, 2026
arXiv cs.AI: When Does Sharing Work?
Markdown · JSON · Original source
In their paper, Xiaofei Sun and coauthors present Omni-Encoder, a encoder that processes image, video, audio, and audio-visual inputs using a single SigLIP-2 Vision Transformer with fully shared parameters. They argue that, rather than training encoders independently and aligning them afterward, the system jointly optimizes all modalities end-to-end. Its design maps each signal into a shared embedding space, places visual frames and audio segments in a shared timeline using timestamps, and interleaves batches from different modalities during training. The authors identify this alternating-modality training as central to stable joint optimization. They report competitive results across for all four modalities, and say joint training improves each modality over its separately trained counterpart. The authors say they will publicly release the code and models.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 20, 2026 (2026-10-20).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T03:09:09.812Z. First observed: 2026-11-05T05:00:30.526Z. Retrieved: 2026-11-10T03:04:08.694Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied source is the paper’s abstract; it gives no names, individual scores, or evaluation details for assessing the reported results. The abstract says code and models will be publicly released, but does not provide them or specify when they will be available.
Open question
What results across the reported support the authors’ claim that joint training improves each modality over its separately trained counterpart?
Source-stated
The authors present Omni-Encoder, which processes four kinds of input in one SigLIP-2 Vision Transformer with shared parameters.
a general multimodal encoder that processes image, video, audio, and audio-visual inputs within a single SigLIP-2 Vision Transformer with fully shared parameters
Source-stated
The authors jointly optimize the modalities end-to-end rather than relying on separate encoders followed by post-hoc alignment.
all modalities are optimized jointly end-to-end
Source-stated
The system maps each signal into a shared embedding space and uses timestamps to place visual and audio elements in a shared spatiotemporal coordinate system.
A modality-native input stage first maps each signal into the shared embedding space
Source-stated
The authors identify alternating-modality training as central to stable joint optimization.
which we find central to stable joint optimization
Source-stated
The authors report competitive results across benchmarks for all four modalities and improvements over separately trained counterparts.
Omni-Encoder achieves competitive results on 16 benchmarks across all four modalities with one shared parameter set, and joint training improves each modality over its separately trained counterpart.
Escape Meridian interpretation
If the reported gains hold beyond the evaluations, shared parameters could reduce the need for separate modality-specific encoders while preserving performance across inputs.
joint training improves each modality over its separately trained counterpart
Source-stated
The authors say code and models will be publicly released.
Code and models will be publicly released.
Source and contributor context
Sun, Xiaofei, Wang, Peng, Liu, Huayao, Xu, Xiaoxiao, Song, Jun, Li, Deqing, Qu, Lin · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Multimodal model
A model that works with more than one kind of input or output, such as text, images and audio.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 90a7fa87-274d-4ab0-8b08-dd4cf92fd4cc.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T03:09:09.812Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 225bac46eee476bdb339ec90e0da9bf71d633f021de09734047c4c538925668c. Source text hash: ac6a4181f8960dc6c9bd23112f52a35ea73de48cd2bdeb695310aa4e77017f1d.
Revision and source-check history
Version 1 · 2026-11-10T03:09:09.812Z
arXiv cs.AI: When Does Sharing Work?
In their paper, Xiaofei Sun and coauthors present Omni-Encoder, a encoder that processes image, video, audio, and audio-visual inputs using a single SigLIP-2 Vision Transformer with fully shared parameters. They argue that, rather than training encoders independently and aligning them afterward, the system jointly optimizes all modalities end-to-end. Its design maps each signal into a shared embedding space, places visual frames and audio segments in a shared timeline using timestamps, and interleaves batches from different modalities during training. The authors identify this alternating-modality training as central to stable joint optimization. They report competitive results across for all four modalities, and say joint training improves each modality over its separately trained counterpart. The authors say they will publicly release the code and models.
Original source
cs.AI · Article
Source details
Original source date: Oct 20, 2026 (2026-10-20).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T03:09:09.812Z. First observed: 2026-11-05T05:00:30.526Z. Retrieved: 2026-11-10T03:04:08.694Z.
Limits
The supplied source is the paper’s abstract; it gives no names, individual scores, or evaluation details for assessing the reported results. The abstract says code and models will be publicly released, but does not provide them or specify when they will be available.
Open question
What results across the reported support the authors’ claim that joint training improves each modality over its separately trained counterpart?
Source-stated
The authors present Omni-Encoder, which processes four kinds of input in one SigLIP-2 Vision Transformer with shared parameters.
a general multimodal encoder that processes image, video, audio, and audio-visual inputs within a single SigLIP-2 Vision Transformer with fully shared parameters
Source-stated
The authors jointly optimize the modalities end-to-end rather than relying on separate encoders followed by post-hoc alignment.
all modalities are optimized jointly end-to-end
Source-stated
The system maps each signal into a shared embedding space and uses timestamps to place visual and audio elements in a shared spatiotemporal coordinate system.
A modality-native input stage first maps each signal into the shared embedding space
Source-stated
The authors identify alternating-modality training as central to stable joint optimization.
which we find central to stable joint optimization
Source-stated
The authors report competitive results across benchmarks for all four modalities and improvements over separately trained counterparts.
Omni-Encoder achieves competitive results on 16 benchmarks across all four modalities with one shared parameter set, and joint training improves each modality over its separately trained counterpart.
Escape Meridian interpretation
If the reported gains hold beyond the evaluations, shared parameters could reduce the need for separate modality-specific encoders while preserving performance across inputs.
joint training improves each modality over its separately trained counterpart
Source-stated
The authors say code and models will be publicly released.
Code and models will be publicly released.
Source and contributor context
Sun, Xiaofei, Wang, Peng, Liu, Huayao, Xu, Xiaoxiao, Song, Jun, Li, Deqing, Qu, Lin · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Multimodal model
A model that works with more than one kind of input or output, such as text, images and audio.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 90a7fa87-274d-4ab0-8b08-dd4cf92fd4cc.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T03:09:09.812Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 225bac46eee476bdb339ec90e0da9bf71d633f021de09734047c4c538925668c. Source text hash: ac6a4181f8960dc6c9bd23112f52a35ea73de48cd2bdeb695310aa4e77017f1d.
uncertain · 2026-11-10T03:09:09.812Z · anthropic/claude-opus-5.5
Publication: 90a7fa87-274d-4ab0-8b08-dd4cf92fd4cc
attribution — pass: The reading consistently attributes the design, the finding about alternating-modality training, the benchmark results and the release plan to Xiaofei Sun and coauthors. It does not present these source claims as established fact.
Authors:Xiaofei Sun, Peng Wang, Huayao Liu, Xiaoxiao Xu, Jun Song, Deqing Li, Lin Qu
summary — uncertain: The summary is faithful, but central terms are not explained for a general reader: encoder, SigLIP-2 Vision Transformer, shared parameters, embedding space and end-to-end. Nothing says what an encoder does or why one model for four input types matters; that significance appears only in the claims list. 'Shared timeline using timestamps' paraphrases the source term 'multimodal rotary position embedding'. The count of 16 benchmarks is dropped. Smallest repair: add one plain sentence on what an encoder is and why one shared model matters, keep the exact technical term (with a tooltip), and restore '16 benchmarks'.
A multimodal rotary position embedding parameterized by absolute timestamps then places visual frames and audio segments in a unified spatiotemporal coordinate system.
date — pass: The displayed original publication date, 2026-10-20, matches the v1 submission date. The later feed date of 2026-11-05 is correctly not used.
[Submitted on 20 Oct 2026]
support — pass: The source-stated claims match the abstract. The interpretation is explicitly conditional ('If the reported gains hold'), is labeled as interpretation, and follows from the source's claim of one shared parameter set with per-modality gains. The limitations are accurate. The question asks for missing evidence rather than assuming a result.
with one shared parameter set, and joint training improves each modality over its separately trained counterpart
media — pass: The selected media is the verified arXiv abstract page for this paper, which is the direct original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A model that works with more than one kind of input or output, such as text, images and audio.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA model that works with more than one kind of input or output, such as text, images and audio.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference