New in Together GPU Clusters: Reliability and control for production GPU clusters

Anyone who has babysat a multi-day training job knows the quiet dread of a GPU dropping off the PCIe bus or a loss curve that flattens because thermal throttling snuck in unnoticed. Together AI is targeting exactly that pain with a fresh round of updates to Together GPU Clusters, split into two buckets: keeping hardware healthy and giving teams more control as they scale past the single-admin-kubeconfig phase.

On the health side, the company already ran active health checks — synthetic tests run at provisioning time or on idle nodes. Now it's added passive health checks that run continuously against live workloads, watching for GPUs falling off the bus, Xid errors, thermal throttling, and Slurm node drains, with near-zero overhead on running jobs. Paired with that is auto node repair, which flags a recommended fix — reboot, reprovision, failover, or remove-and-RMA — but still puts a human in the loop to approve it before anything happens to a checkpoint or inference replica. Together says the goal is trading hours of support tickets for minutes of in-product workflow.

The Slurm-on-Kubernetes stack got a rebuild too, based on the company's fork of the open-source Slinky project. The new version supervises worker daemons so they restart in place after transient failures, reaps orphaned processes that used to clog PID tables, and moves job accounting onto durable, PVC-backed storage so a pod restart no longer wipes the accounting database. It also rebuilds Slurm's GPU view fresh on every node start, fixing a drift problem where stale GPU identifiers would silently knock hardware out of the schedulable pool. New Slurm clusters run on this stack by default, and existing managed clusters can be migrated during a scheduled maintenance window.

The operational-control half of the release is aimed at teams that have outgrown sharing one admin kubeconfig. External OIDC support lets clusters authenticate against an existing identity provider — Google, Okta, Auth0, or Microsoft Entra ID among them — so each team member gets their own identity, audit trail, and Kubernetes RBAC permissions instead of one shared login. It has to be set up at cluster creation time, and support for Slurm clusters is still coming. Alongside that, startup scripts let teams bake in custom setup — package installs, scratch-space prep, Slack notifications on job completion — at node boot, job start, or job end, configured once in the console instead of filed as a support ticket or run by hand over SSH.

There's also a new cluster details view built around three plain questions: is it healthy, is it being used, and what happened recently. It shows node health at a glance, live utilization and network metrics with Grafana drill-down, and an event timeline for node state changes, plus new Nodes, Health checks, and Repair tabs that replace what Together describes as incident retros that used to happen over Slack. And acceptance testing — the battery that validates GPU health, networking, and storage before a cluster goes live — stays opt-in by default for smaller or short-lived clusters, but Together recommends switching it on for larger or long-running training jobs, where catching a bad node at provisioning beats catching it on epoch 47.