This week I published Test-Time Compute Orchestration: Applying Business Policies to AI Agents on the HPC Gridware blog. It is about running AI agents at scale as ordinary cluster jobs, and about the policies a business needs once there are hundreds of them. There is also a new video that shows it running.

Hundreds of AI Agents as Cluster Jobs with NVIDIA OpenShell

The video is hosted on YouTube. Nothing is loaded from YouTube until you click the button. After that, the video is loaded from youtube-nocookie.com and Google processes your data according to its privacy policy.

Or watch it on YouTube

Here are the key points in short.

Agents mostly wait

An agent reasons, calls a tool, looks at the result, and starts over. Most of that time it waits: for the model API, for a tool, for a sandbox to start. CPU usage is low, memory comes in bursts. So agents are a scheduling problem, not a CPU problem. Asking for "80 nodes with 4 GPUs each" fits training. It does not fit thousands of small, mostly idle processes.

What the scheduler brings

Open Cluster Scheduler and Gridware Cluster Scheduler have solved this kind of problem for decades. Chip design and life science run millions of small, I/O heavy jobs, and to the scheduler an agent looks much the same.

  • Scale. The scheduler scales well with the number of nodes and with the workload. Hundreds of thousands of jobs a day run without issues, and with one agent per job that means hundreds of thousands of agent runs a day.
  • Slots, not cores. A host can offer more slots than it has cores. Agents that use a fraction of a CPU can oversubscribe the cores, while memory stays a managed resource. During business hours your interactive jobs deserve one core per slot. After hours the agents take over: a queue calendar switches on an agent queue with several slots per core.
  • Quotas that are easy to express. The model API is just a consumable with a seat limit, exactly like a software license. One resource quota rule caps the agents:
$ qconf -srqs agent_cap | grep limit; qconf -se global | grep complex_values
   limit        queues agent.q to slots=96
complex_values        anthropic_api=96
  • Fair-share between teams. When two teams want more agents than the cluster can run, the share tree decides who goes next based on past usage. An urgent agent still starts right away.
  • Resource reservation and advance reservation. A large simulation is not starved by a flood of small agent jobs, because the scheduler reserves resources for it. With an advance reservation you can book capacity for an agent campaign or a deadline ahead of time.
  • Accounting with agent metrics. qacct shows what each team or project used. With flexible accounting you add your own metrics to the accounting record, such as the cost per run, token usage, or the model that was used.

Every agent in its own sandbox

An agent runs code it wrote itself, so it should not run with the full rights of a user. Our OpenShell integration for Gridware Cluster Scheduler starts every agent in its own NVIDIA OpenShell sandbox. A sandbox goes further than a plain container. Its policies are declared in YAML and are configurable:

  • Files. Which paths the agent can read, and which it can write. Everything else is out of reach.
  • Network. Outbound traffic is denied by default. Each allowed endpoint is listed, per binary, down to which requests are allowed, for example reading from an API but not writing to it. These rules can even change while the sandbox runs.
  • Process. The agent runs as an unprivileged user, and seccomp blocks the usual escalation paths.
  • Credentials. Secrets are resolved only at endpoints the policy allows, so the agent never sees them in plain text.

Inside the cluster OpenShell starts the sandboxes as Podman containers, and each sandbox is a regular job. So all the flexibility of job submission comes with it: resource requests, queues, projects, priorities. When a sandbox ends, its accounting details are read out and forwarded to the scheduler, and they show up in qacct next to every other job. The scheduler decides where, when, and how much an agent gets. OpenShell decides what it may touch.

Agents next to your existing work

Agents don't need their own platform with separate storage and separate users. They can run on the same cluster as EDA regressions, life science work from Posit Workbench sessions to Nextflow pipelines, engineering simulations, or AI training, fine-tuning and inference, under one set of quotas and one accounting. And they can work on the results. In the blog post a simulation sweep runs as a classic array job, then one agent per run reads the output and evaluates it.

With Gridware Cluster Scheduler the configuration is at your fingertips in Qontrol, and its MCP server gives AI assistants a read-only view of the cluster configuration. So you can ask why an agent is still pending, and get an answer.

The full blog post has the setup with NVIDIA OpenShell sandboxes, the numbers, and the references. Your cluster resources are too valuable to run just one job type.