Sign in

Training runs

Watching queued work execute — statuses, tasks, and logs.

View as Markdown

A run is the execution of a training request. Queueing an approved request creates a run; the run picks up the specified compute target, clones your code, executes the workflow stages, and publishes what it produced.

Statuses

A run moves through a lifecycle:

StatusMeaning
QueuedCreated, waiting for a compute target to pick it up.
AssignedMatched to a target; the agent hasn't claimed it yet.
AcquiredThe agent has claimed it and is preparing — cloning, building the environment.
In ProgressWorkflow stages are executing.
CompletedEvery stage finished successfully.
FailedA stage failed, or the run couldn't be prepared.
StoppedCancelled before it finished.

Completed, Failed, and Stopped are terminal.

A run sitting in Queued for a long time usually means no target is available: a dedicated target whose agent is offline, or an on-demand target that can't launch — often an invalid provider credential. Check the compute target's status first.

Tasks

Each workflow stage becomes a task, with its own status: Pending, Running, Completed, Failed, or Skipped. Tasks run in the order the workflow lists them, and when one fails the rest are Skipped rather than attempted.

This is the fastest way to read a failed run — find the failed task, and you know which stage broke.

TReqs CLI
treqs jobs tasks <job-id>

Logs

A run's log has more than one producer, which matters when you're debugging.

TReqs CLI
treqs jobs logs <job-id>              # the workload's own output
treqs jobs logs <job-id> --follow     # keep polling until complete
treqs jobs logs <job-id> --agent      # what the agent recorded
treqs jobs logs <job-id> --task <id>  # narrow to one stage

--agent is the one people forget. If a task fails before its command runs — a missing interpreter, a checkout that didn't resolve, an environment that wouldn't build — the workload log is empty and the explanation is in the agent log.

To follow a run from the terminal:

TReqs CLI
treqs jobs watch <job-id>   # lifecycle on stderr, logs on stdout
treqs jobs wait <job-id>    # block until terminal; non-zero unless Completed

treqs jobs wait exits non-zero unless the run reaches Completed, which makes it usable in a script. Ctrl-C during watch detaches without cancelling the run.

Stopping a run

TReqs CLI
treqs jobs stop <job-id>
treqs jobs cancel <job-id>  # `cancel` is just an alias for `stop`

A stopped run lands in Stopped and keeps its logs and task history.

© 2026 T-Reqs Inc. All rights reserved.