---
title: "Run Inspect evals in a Promigence sandbox"
description: "Use sandbox=\"promigence\" in Inspect to give every sample its own cloud sandbox, with Docker-style tasks running unchanged."
url: "https://www.promigence.ai/docs/guides/inspect-evals"
site: "Promigence"
---

# Run Inspect evals in a Promigence sandbox

Use sandbox="promigence" in Inspect to give every sample its own cloud sandbox, with Docker-style tasks running unchanged.

Inspect (`inspect_ai`) runs tool-using evals inside a sandbox per sample. The `inspect-promigence`
package adds a provider named `promigence`, so each sample runs in a Promigence sandbox and many
samples can run at once without a Docker daemon on your machine. Tasks written for
`sandbox="docker"` with a Dockerfile or a single-service compose file also run unchanged.

## What you need

You need a Promigence API key in `PROMIGENCE_API_KEY`, or a one-time `promigence auth login`, and
Python 3.10 or newer. The provider was written and tested against inspect_ai 0.3.264 and requires at
least that version.

## Install

```bash
pip install inspect-promigence openai
export PROMIGENCE_API_KEY=...
export PROMIGENCE_SPEND_CAP=1
```

Installing is the whole registration step. Inspect discovers the provider through its plugin entry
point, so `sandbox="promigence"` works as soon as the package is present. If your task builds from a
Dockerfile, also set `PROMIGENCE_REGISTRY` to a registry you can push to: the image is built on your
machine with `docker buildx` and pushed there.

## A minimal eval

```python
from inspect_ai import Task, task
from inspect_ai.agent import react
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.tool import bash

from inspect_promigence import PromigenceSandboxConfig

SANDBOX = (
    "promigence",
    PromigenceSandboxConfig(
        image="python:3.12-slim",
        verify="python3 -c 'import sys; assert sys.version_info >= (3, 12)'",
        sandbox_timeout_sec=600,
    ),
)

@task
def hello_bash_agent() -> Task:
    return Task(
        dataset=[Sample(input="Use bash to print the Python version, then answer with it.", target="3.12")],
        solver=react(tools=[bash(timeout=60)]),
        scorer=includes(),
        sandbox=SANDBOX,
        message_limit=12,
    )
```

Save it as `hello_eval.py`. The model here is OpenAI's, so set `OPENAI_API_KEY` before you run it:

```bash
inspect eval hello_eval.py@hello_bash_agent --model openai/gpt-4o-mini
```

Other accepted forms are `sandbox=("promigence", "compose.yaml")`, `sandbox=("promigence", "Dockerfile")`
and plain `sandbox="promigence"`, which picks up a `compose.yaml`, `Dockerfile` or `promigence.yaml`
next to the task. For an existing Docker task, try `inspect eval task.py --sandbox promigence`.

## How samples map to sandboxes

At task start the image is pinned to its digest and turned into one snapshot. A snapshot is a saved,
verified environment; each sample's sandbox starts as a fresh copy of it, so samples never share
files. If `verify` fails, the snapshot is not stored and the task stops before any sample runs, which
keeps a broken environment from being scored as a model failure. The build is billed per second while
it runs, whether it verifies or not, up to $0.17 at the default small size
([quickstart](https://www.promigence.ai/docs/quickstart)), and samples and later runs reuse the snapshot with no further build
charge.

`PromigenceSandboxConfig` also takes `setup` (commands baked into the snapshot), `start` (a
background process that is already running when a sample begins), `tier` or `cpus`/`memory_mb`,
`allow_internet`, `working_dir`, `user`, `env`, `registry` and `registry_secret`. You can keep the same
settings in a `promigence.yaml` file instead.

The sandbox config goes into every `.eval` log that Inspect writes. Values under credential-looking
`env` keys (names containing KEY, TOKEN, SECRET, PASSWORD, AUTH or CREDENTIAL) are written as a
`${KEY}` reference or as `<redacted>`, never in the clear.

Each sample's transcript carries an info event from `promigence` with the sandbox id, snapshot id,
image digest, whether the snapshot was verified, and any failures classed as `env`, `env_definition`,
`config` or `task`. The same record is in the sample store, so it travels with the log.

## Spend cap

Set `spend_cap` on the config, or `PROMIGENCE_SPEND_CAP` in the environment, or leave both unset to use
your organization's default. The cap applies to each sample's sandbox.

## Cleaning up

Each sandbox ends when its sample finishes. At task end the provider ends any that missed their
cleanup; with `--no-sandbox-cleanup` it prints their ids instead, and
`inspect sandbox cleanup promigence <sandbox_id>` ends one later. To open a shell in a live sample,
run `promigence sandbox shell <sandbox_id>` with the CLI (`npm i -g promigence`).

## What is not supported

- Compose files with more than one service.
- Images without `/bin/sh` (distroless images cannot exec).
- Incremental output: command output arrives when the command exits.
- Images other than `linux/amd64`.
- `timeout_retry`, which exists for Docker daemon hangs, is ignored. `write_file` writes as root.
- `connection()` gives the CLI command only, with no port mappings or editor attach.

Next: read [the agent briefing](https://www.promigence.ai/agents), or start from [the quickstart](https://www.promigence.ai/docs/quickstart).