Run Harbor tasks in a Promigence sandbox

Select Promigence in Harbor with one flag to run Terminal-Bench and other Harbor tasks in cloud sandboxes, each trial in its own copy.

Also as markdown: /docs/guides/harbor.md

Harbor runs agent tasks such as Terminal-Bench. With the Promigence environment installed, every trial gets its own sandbox instead of a local container, so you can run many trials at once without Docker on the machine that drives them. No patch to Harbor is needed: you select the environment by its import path.

What you need#

Bring a Promigence API key in PROMIGENCE_API_KEY, or sign in once with promigence auth login. You need Python 3.12 or newer. The adapter was written and tested against harbor 0.23.0 and pins harbor>=0.23,<0.24, because Harbor's environment interface is not a stable API yet.

Install#

Terminal

pip install promigence-harbor
export PROMIGENCE_API_KEY=...
export PROMIGENCE_SPEND_CAP=5
export HARBOR_TELEMETRY=0     # optional: Harbor sends usage telemetry unless you turn it off

Tasks that build from a Dockerfile also need PROMIGENCE_REGISTRY set to a registry you can push to. Those images are built on your machine with docker buildx, pushed there, and pulled by Promigence. Tasks that name a docker_image are pulled by Promigence directly.

A minimal run#

Terminal

harbor run -p path/to/dataset -a claude-code -m anthropic/claude-sonnet-4-5 \
  -e promigence_harbor:PromigenceEnvironment \
  --ek spend_cap=5 --ek sandbox_timeout_sec=3600 -n 32

To try it without a model key, run a single task with Harbor's oracle agent, which executes the task's own solution/solve.sh:

Terminal

harbor run --path ./hello-task --agent oracle \
  --env promigence_harbor:PromigenceEnvironment \
  --ek spend_cap=1 --ek sandbox_timeout_sec=900 --n-concurrent 1

The reward lands in jobs/<job>/<trial>/result.json as usual.

How trials map to sandboxes#

Each Harbor trial gets one sandbox. Trials of the same task share one snapshot: a snapshot is a saved, verified environment, and each sandbox starts as a fresh copy of it, so one trial never sees another trial's files. An image tag is pinned to its digest before the snapshot is made, which keeps reruns on the same bytes. That build is billed per second while it runs, whether it verifies or not, up to $0.17 at the default small size (quickstart), and the trials that reuse the snapshot pay no further build charge.

Pass environment options with --ek key=value. The ones you will reach for first:

option	what it does
`spend_cap`	dollar cap per sandbox run
`sandbox_timeout_sec`	sandbox lifetime (default 3600)
`tier`	`small`, `medium` or `large`; otherwise picked from the task's `cpus` and `memory_mb`
`setup`	commands baked into the snapshot
`verify`	the snapshot is kept only if this command exits 0
`registry`, `registry_secret`	where Dockerfile images go, and a stored secret for private pulls

Every --ek value ends up in the job's config.json, so the adapter refuses --ek api_key=.... Keep the key in the environment.

A task with a docker-compose.yaml that defines exactly one service is read and run on the sandbox. Its image or build, environment, working directory, user, resources and TCP ports carry over. Compose files with two or more services are refused by name before anything starts.

Telling environment failures from task failures#

Next to each trial's result you get promigence-<session_id>.json. It records the sandbox id, the snapshot id, the image digest, whether the snapshot was verified, and any failures, each classed as env (the sandbox side failed), env_definition (the task's image could not be built or verified), config or task. Environment failures raise their own exception type, so Harbor's exception_info keeps them apart from an agent that got the task wrong.

Spend cap#

Every sandbox run needs a cap. Set it with --ek spend_cap=5, with PROMIGENCE_SPEND_CAP, or rely on your organization's default. A run with no cap from any of those is refused.

Cleaning up#

Harbor's stop() always ends the sandbox, so finished trials do not linger. To look inside a running one, attach() opens promigence sandbox shell <sandbox_id>, which needs the CLI (npm i -g promigence).

What is not supported#

Compose tasks with more than one service are refused, and so are --extra-docker-compose overlays. Network allowlists, and changing network policy between the agent and verifier phases, are not supported; Harbor itself refuses those tasks during validation. There are no GPUs or Windows images. Images must be linux/amd64 and contain /bin/sh, plus tar for directory transfers. Output is not live: Harbor's callbacks receive it when the command finishes.

Next: read the agent briefing, or start from the quickstart.