Job Orchestration

Running a Python script on one box is easy; running thousands across a shared fleet, enforcing timeouts, capturing artifacts, and preserving logs for audit is where ad-hoc scripts collapse. Stout orchestrates jobs with a BullMQ-backed queue that respects resource locks, capability requirements, and per-job timeouts. Submit a job through the dashboard, the REST API, or a CI webhook, and the control plane places it in a queue keyed to the right box or box group.

When a suitable box is free, the dispatcher opens an authenticated channel to the Lager runtime, uploads the script bundle, and begins streaming stdout, stderr, and the custom output channel back to the control plane. Clients — the dashboard, the CLI, or your own tooling — subscribe over Server-Sent Events or WebSocket and see lines appear in real time with sub-second latency. Long-running jobs send a keepalive every 20 seconds so reverse proxies do not time out.

Artifacts land in object storage. When the job generates output files, Stout creates a presigned S3 upload URL, the box streams the artifact directly to storage, and the control plane records the checksum, size, and storage key. Artifacts are scoped to the job, the organization, and the role-based access control policy that applied when the job ran, so a revoked team member cannot retrieve historical outputs.

Terminal states are where ad-hoc scripts hurt most, so Stout keeps them distinct. Every job carries a timeout (default one hour, configurable per submission); a job that exceeds it is marked timed_out, its resource lock is freed, and a webhook event fires — a hung test releases the bench instead of holding it until someone notices. A job that failed, a job that was cancelled, and a job that never found a suitable box are three different outcomes, not one red X, so alerting rules can act on the signal that actually matters instead of paging on all of them.