OFASys: Enabling Multitask Learning with One Line of Code!

Training a single model that handles text, images, speech, video and motion sounds great on paper. In practice it's a nightmare of glue code: different preprocessing for every modality, brittle batching logic, and training runs that fall over the moment you add a new task. The team behind OFA, the earlier unified pretrained model from this group, ran into exactly that wall while trying to scale their own generalist ambitions. So they built OFASys, a framework meant to strip away the plumbing and let researchers focus on what tasks and modalities they actually want to combine.

The core trick is something they call an Instruction — basically a short template that describes a task's inputs and outputs in plain tags. Want an image captioning task? You write something like an image tag pointing to a data column, an arrow, and the text column you're predicting. Want a natural language inference task with two text inputs and a closed label set? Same idea, just swap the tags. OFASys parses that single line into a full task plan, handling preprocessing, model wiring, and training setup behind the scenes. It's a small syntax doing a lot of hidden work.

Under the hood, each task plan runs through modality-specific adapters that convert raw data into representation sequences, feeds them into a shared universal model, then converts outputs back with matching postprocessors. Because everything funnels through the same sequence-based interface, the universal model itself doesn't need to know or care whether it's looking at pixels, waveforms, or motion capture data. When you're running multiple tasks at once, OFASys shares parameters across those adapters and the universal model by default, and a task scheduler juggles which tasks train when, while a separate scheduler handles distributing the work across GPUs.

The team put this to the test with OFA+, a generalist model trained across 23 tasks spanning seven modalities — text, image, speech, video and motion combined for the first time in one system, according to their claims. They also built a mixture-of-experts variant, OFA+ Generalist MoE, and compared both against specialist versions of OFA fine-tuned individually on each task. The generalist setup retained more than 95 percent of specialist-level performance despite juggling all those tasks simultaneously, which is a solid result if it holds up under independent scrutiny — multitask training usually costs you something on individual benchmarks, and losing under 5 percent while gaining that much task breadth is a real trade-off worth making.

What's notable here isn't the model itself but the infrastructure bet. Plenty of labs are chasing generalist multimodal systems — Gato, Unified-IO, and others show the appetite is real. But tooling has lagged behind ambition, with everyone hand-rolling their own data pipelines. If OFASys's one-line Instruction approach actually generalizes to messier, less curated datasets than the ones in this paper, it could shave real months off getting a multimodal multitask project off the ground.