How a global fintech scaled coding agent traffic with Dedicated Model Inference
Together AI
A global fintech moved its coding assistant onto Together’s Dedicated Model Inference. Now its own engineers can scale and test it without waiting on tickets.
Based on reporting by Together AI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
A global fintech has moved its coding assistant onto Together’s Dedicated Model Inference, and the reason is pretty simple: its traffic does not behave like a neat monthly forecast. It comes in spikes during engineering hours, grows as more teams adopt agents, and needs capacity decisions to happen in the same flow as the work itself.
The assistant runs on GLM-5.2, a mixture-of-experts model used for long-horizon coding and agentic tasks. Together says the workload sits on the critical path of shipping financial products to millions of users across dozens of markets, so waiting around for capacity requests was becoming the wrong bottleneck.
That is where DMI changes the shape of the problem. The customer’s engineers can now create endpoints, adjust scaling, roll out custom weights, and run blue/green tests without filing a request and waiting for Together. They also get programmatic access to usage and performance data, which matters when a burst is really a burst of concurrency, not a clean jump in throughput.
The numbers in the source tell the story. The traffic pattern shows ISL p50/p90/p95 of roughly 81K/163K/178K tokens and RPS p50/p90/p95 of 3/6/7, so the pain point is headroom, not raw token output. Together says the production setup ran 56 B200s across 14 replicas at 256K context, and when a later migration pushed the endpoint close to 100% prefill capacity, requests started queuing for one to three minutes and decode throughput fell to about 5 tokens per second.
That problem was diagnosed through Together’s metrics API. Support traced a 192-second slow request and found it had spent nearly the whole time waiting behind a 2.3M-token prefill backlog, not burning compute on its own 250K-token prompt. The fix was a live config change: restore a tuned routing policy and widen the max-inflight-per-worker threshold. No redeploy, no ticket loop, no drama.
The relationship is already moving past the first endpoint. The customer is now scoping a dedicated GLM 5.1 node in a new region for a different workload, this time chat-style customer support rather than coding. That is the useful part here: the infrastructure stops being a one-off project and starts acting like a platform teams can actually use.
My take — AI-written commentary, not fact-checked reporting
This is the part of AI infrastructure that matters and the hype crowd keeps skipping: control beats slogans. When the workload is spiky, low-TPS, and operationally urgent, self-serve scaling and real observability are the product. Everything else is just a nice brochure with a GPU bill attached.
Read more about this at: Together AI