Run Inspect evals in a Promigence sandbox

Inspect (inspect_ai) runs tool-using evals inside a sandbox per sample. The inspect-promigence package adds a provider named promigence, so each sample runs in a Promigence sandbox and many samples can run at once without a Docker daemon on your machine. Tasks written for sandbox="docker" with a Dockerfile or a single-service compose file also run unchanged.

What you need#

You need a Promigence API key in PROMIGENCE_API_KEY, or a one-time promigence auth login, and Python 3.10 or newer. The provider was written and tested against inspect_ai 0.3.264 and requires at least that version.

Install#

pip install inspect-promigence openai
export PROMIGENCE_API_KEY=...
export PROMIGENCE_SPEND_CAP=1

Installing is the whole registration step. Inspect discovers the provider through its plugin entry point, so sandbox="promigence" works as soon as the package is present. If your task builds from a Dockerfile, also set PROMIGENCE_REGISTRY to a registry you can push to: the image is built on your machine with docker buildx and pushed there.

A minimal eval#

from inspect_ai import Task, task
from inspect_ai.agent import react
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.tool import bash

from inspect_promigence import PromigenceSandboxConfig

SANDBOX = (
    "promigence",
    PromigenceSandboxConfig(
        image="python:3.12-slim",
        verify="python3 -c 'import sys; assert sys.version_info >= (3, 12)'",
        sandbox_timeout_sec=600,
    ),
)

@task
def hello_bash_agent() -> Task:
    return Task(
        dataset=[Sample(input="Use bash to print the Python version, then answer with it.", target="3.12")],
        solver=react(tools=[bash(timeout=60)]),
        scorer=includes(),
        sandbox=SANDBOX,
        message_limit=12,
    )

Save it as hello_eval.py. The model here is OpenAI's, so set OPENAI_API_KEY before you run it:

inspect eval hello_eval.py@hello_bash_agent --model openai/gpt-4o-mini

Other accepted forms are sandbox=("promigence", "compose.yaml"), sandbox=("promigence", "Dockerfile") and plain sandbox="promigence", which picks up a compose.yaml, Dockerfile or promigence.yaml next to the task. For an existing Docker task, try inspect eval task.py --sandbox promigence.

How samples map to sandboxes#

At task start the image is pinned to its digest and turned into one snapshot. A snapshot is a saved, verified environment; each sample's sandbox starts as a fresh copy of it, so samples never share files. If verify fails, the snapshot is not stored and the task stops before any sample runs, which keeps a broken environment from being scored as a model failure. The build is billed per second while it runs, whether it verifies or not, up to $0.17 at the default small size (quickstart), and samples and later runs reuse the snapshot with no further build charge.

PromigenceSandboxConfig also takes setup (commands baked into the snapshot), start (a background process that is already running when a sample begins), tier or cpus/memory_mb, allow_internet, working_dir, user, env, registry and registry_secret. You can keep the same settings in a promigence.yaml file instead.

The sandbox config goes into every .eval log that Inspect writes. Values under credential-looking env keys (names containing KEY, TOKEN, SECRET, PASSWORD, AUTH or CREDENTIAL) are written as a ${KEY} reference or as <redacted>, never in the clear.

Each sample's transcript carries an info event from promigence with the sandbox id, snapshot id, image digest, whether the snapshot was verified, and any failures classed as env, env_definition, config or task. The same record is in the sample store, so it travels with the log.

Spend cap#

Set spend_cap on the config, or PROMIGENCE_SPEND_CAP in the environment, or leave both unset to use your organization's default. The cap applies to each sample's sandbox.

Cleaning up#

Each sandbox ends when its sample finishes. At task end the provider ends any that missed their cleanup; with --no-sandbox-cleanup it prints their ids instead, and inspect sandbox cleanup promigence <sandbox_id> ends one later. To open a shell in a live sample, run promigence sandbox shell <sandbox_id> with the CLI (npm i -g promigence).

What is not supported#

• Compose files with more than one service.
• Images without /bin/sh (distroless images cannot exec).
• Incremental output: command output arrives when the command exits.
• Images other than linux/amd64.
• timeout_retry, which exists for Docker daemon hangs, is ignored. write_file writes as root.
• connection() gives the CLI command only, with no port mappings or editor attach.

Next: read the agent briefing, or start from the quickstart.