nemo-gym

packaged by Chainguard

Chainguard Container for nemo-gym

NVIDIA NeMo Gym is a library for building reinforcement learning environments for large language models and serving them for rollout collection, reward computation, and training.

Chainguard Containers are regularly-updated, secure-by-default container images.

Download this Container Image

For those with access, this container image is available on cgr.dev:

docker pull cgr.dev/ORGANIZATION/nemo-gym:latest

Be sure to replace the ORGANIZATION placeholder with the name used for your organization's private repository within the Chainguard Registry.

Compatibility Notes

The nemo-gym image is designed as a drop-in replacement for the CPU-only NeMo Gym worker image that NVIDIA NVFlow builds from its dockerfiles/Dockerfile.nemo-gym (on its feat/v1.1.2/ray-dev branch, tested at commit cfdf630c), at NeMo Gym commit 33ef60369f76557e6a6dd828c0bd5f5529624a92. NVFlow runs it for its Gym-only stages: data preparation, cache prefetching, rollout collection, reward computation, and document-grounded SDG. The following match NVFlow's image, so its recipes run unchanged:

• Layout. The NeMo Gym source tree is at /opt/Gym, the CLI venv at /opt/gym-cli-venv, and one prebuilt venv per component under /opt/gym-venvs. gym is on PATH (also as /usr/local/bin/gym), and /opt/gym-commit records the NeMo Gym commit. These paths link to the Chainguard tree under /opt/chainguard/nemo-gym.
• Prebuilt components. The same seven venvs as NVFlow's image, for the equivalence_llm_judge, finance_sec_search, and format_verification resources servers, the simple_agent and finance_agent agents, and the vllm_model and openai_model model servers. With NVFlow's +uv_venv_dir=/opt/gym-venvs +skip_venv_if_present=true, Gym reuses them with no network access.
• On-demand builds. As in NVFlow's image, Gym builds the venv of any other component at runtime with uv, which needs access to a Python package index. /opt/gym-venvs and its category directories are writable by any user.
• Environment. python3 is the system Python 3.12 (/usr/bin/python3; no venv is on PATH), with orjson and PyYAML from the py3.12-orjson and py3.12-pyyaml packages, for NVFlow's python3 -m ... jobs; NVFlow's image pip installs the same two into its system Python. The image also has curl, procps, and the CA certificates. The container runs as root, with python3 as its command and / as its work directory.

The image differs from NVFlow's in the following ways. Each difference is deliberate:

• One base venv instead of eight copies. NVFlow's image installs all of NeMo Gym and its dependencies again in each component venv. Here the CLI venv is the base, and each component venv holds only what the component adds and puts the base venv's packages behind its own. The equivalence_llm_judge venv, for example, holds only its datasets 4.0.0 pin and the older fsspec, dill, and multiprocess that datasets 4.0.0 requires. This avoids shipping NeMo Gym and its dependencies eight times: the image is 1.1 GB unpacked, against 3.7 GB for NVFlow's.
• Dependency versions come from NeMo Gym's uv.lock, with security fixes. NVFlow resolves fresh versions when it builds its image. So that builds are reproducible, this image installs the versions in NeMo Gym's lock, with fixes that move some past it, including ray 2.56.0, mlflow 3.16.1, wandb 0.30.0, aiohttp, gitpython, pyjwt, pyarrow, and urllib3. Gym pins the ray and openai versions of every venv it builds at runtime to the CLI's, so on-demand components use the same ray.
• No development tools in the venvs. Each component's requirements.txt asks for nemo-gym[dev], so NVFlow's component venvs carry pytest, mypy, ruff, and pre-commit. Nothing at runtime imports them. The components' own unit tests (tests/) and recorded example rollouts aren't shipped either, since they serve only gym env test, a development command that also needs those tools, and some of them trip malware scanners. Gym's own tests/unit_tests are kept.
• No MLflow server in the vllm_model venv. NVFlow's vllm_model venv also installs Flask, SQLAlchemy, Alembic, and the rest of MLflow's tracking server, because that component has its own pyproject.toml with a [tool.uv] table, so uv uses it instead of NeMo Gym's root configuration, and the root's exclude-dependencies doesn't apply. Here it gets the same packages as the other venvs, because NeMo Gym's own lock excludes those server dependencies and Gym's servers don't use them.
• Prebuilt venvs are Gym's default. An env.yaml in /opt/Gym sets skip_venv_if_present: true and points uv_venv_dir at the prebuilt venvs. Without it, gym env start with no flags reinstalls each server's requirements from a package index at every start. Flags on the command line, such as NVFlow's, take precedence, and an env.yaml in the working directory replaces this one. Gym applies env.yaml after the configs passed with +config_paths, so skip_venv_if_present or uv_venv_dir set in such a config file are overridden by the shipped env.yaml; set them on the command line, or in your own env.yaml in the working directory.
• The uv cache is not included. NVFlow's /opt/Gym (1.2 GB) carries the uv cache from its build. This image ships only the source tree (78 MB), because the prebuilt venvs don't need the cache and it would add 1.1 GB.
• RAY_JOB_STOP_SIGNAL=SIGINT is set, so NeMo Gym can shut down its servers when it runs in a Ray Job on this image as a Ray head; see Use with NVFlow.
• xxd is available. NVFlow's rollout scripts check the last byte of a partial results file with xxd before resuming. python:3.12-slim has no xxd; this image has it through BusyBox.
• x86_64 only, because the package is built and tested only for x86_64.

Prerequisites

• An x86_64 host with Docker, or a Slurm cluster with enroot and pyxis as NVFlow uses.
• An OpenAI-compatible endpoint serving the policy model, such as vLLM. NeMo Gym serves environments to it; the image runs no model itself and needs no GPU.
• Network access to a Python package index, only for components that aren't prebuilt. The seven prebuilt components run offline.

Getting Started

Check that the CLI runs:

docker run --rm cgr.dev/ORGANIZATION/nemo-gym:latest gym --help

NeMo Gym serves environments to a policy model behind an OpenAI-compatible API, such as vLLM. The following starts the format_verification environment against a vLLM server at http://vllm:8000/v1 serving Qwen/Qwen3-4B. Replace the address and model name with your own:

docker run -d --name gym --shm-size=1g -p 11000:11000 \
  cgr.dev/ORGANIZATION/nemo-gym:latest \
  bash -c 'cd /opt/Gym && gym env start \
    --resources-server format_verification/freeform_formatting --model-type vllm_model \
    +policy_base_url=http://vllm:8000/v1 +policy_api_key=EMPTY +policy_model_name=Qwen/Qwen3-4B \
    +head_server.host=0.0.0.0 +head_server.port=11000'

• --shm-size gives the Ray instance that NeMo Gym starts room for its object store in /dev/shm.
• The head server on port 11000 lists the server instances. Once docker logs gym shows All 3 / 3 servers ready!, the resources server, its agent, and the model server are up:

curl -s http://localhost:11000/server_instances

Collect rollouts from the example data that ships with the environment, as NVFlow's rollout stage does:

docker exec gym bash -c 'cd /opt/Gym && gym eval run --no-serve \
  +agent_name=freeform_formatting_simple_agent \
  +input_jsonl_fpath=resources_servers/format_verification/data/example.jsonl \
  +output_jsonl_fpath=/tmp/rollouts.jsonl +num_repeats=1 \
  +head_server.host=127.0.0.1 +head_server.port=11000'
docker exec gym head -c 300 /tmp/rollouts.jsonl

Each line of rollouts.jsonl is one rollout, with the model's response and the reward the resources server gave it.

Use with NVFlow

Import the image for the cluster with enroot import and reference it as nemo-gym in NVFlow's my_cluster.yaml, as NVFlow's docker_instructions.md describes for its own image. The recipes need no changes.

For NVFlow's three-cluster GRPO path, the image also runs as the Ray head of the rollout cluster. Start it with NVFlow's scripts/start_ray_on_slurm.sb and RAY_VENV=/opt/gym-cli-venv, as NVFlow's INSTALL-RAY.md describes. Ray lives in the CLI venv. Ray Jobs submitted to the head with a working_dir run against the image's NeMo Gym, and Gym attaches to the head with +ray_head_node_address=auto. The image sets RAY_JOB_STOP_SIGNAL=SIGINT. Ray blocks SIGINT in its processes and unblocks it for a job's command only with this setting, and Gym stops its servers with SIGINT; without it, a rollout job's Gym can't stop its servers when the job ends, and leaves them running on the head. NVFlow's image doesn't need the setting because its /bin/sh (Debian's dash) clears the signal mask Ray passes on. With the setting, ray job stop also sends SIGINT before SIGKILL, which Gym handles as a normal shutdown. To try the head role locally:

docker run --rm --shm-size=1g cgr.dev/ORGANIZATION/nemo-gym:latest bash -c '
  export PATH=/opt/gym-cli-venv/bin:$PATH
  ray start --head --disable-usage-stats --num-gpus=0 --dashboard-host=0.0.0.0 --dashboard-port=8267
  mkdir /tmp/job && echo "import nemo_gym; print(nemo_gym.__version__)" > /tmp/job/job.py
  ray job submit --address http://127.0.0.1:8267 --working-dir /tmp/job -- python job.py
  ray stop'

Configuration

NeMo Gym is configured with Hydra-style +key=value overrides on the command line and YAML configs passed with +config_paths=[...]; see the NeMo Gym documentation linked below.

Two settings decide whether a component starts from a prebuilt venv:

• skip_venv_if_present: reuse a component's venv when it exists. The shipped env.yaml sets it to true.
• uv_venv_dir: where Gym looks for and creates component venvs, as <uv_venv_dir>/<category>/<name>/.venv. NVFlow's recipes pass /opt/gym-venvs; the shipped env.yaml sets the same venvs' Chainguard path.

To run an environment that is not prebuilt, give the container network access to a package index. Gym builds its venvs on first start, and reuses them while the container runs. For example, the blackjack environment:

docker run --rm --shm-size=1g cgr.dev/ORGANIZATION/nemo-gym:latest bash -c 'cd /opt/Gym && \
  gym env start --resources-server blackjack --model-type vllm_model +dry_run=true \
    +policy_base_url=http://vllm:8000/v1 +policy_api_key=EMPTY +policy_model_name=Qwen/Qwen3-4B \
  && ls /opt/chainguard/nemo-gym/gym-venvs/resources_servers'

+dry_run=true builds the venvs and checks that each server starts, then exits. A venv built this way is about 1 GB, since it installs all of NeMo Gym again.

Gym also runs as a non-root user. Hydra writes an outputs/ directory in the working directory, and /opt/Gym/outputs is writable by any user (in NVFlow's image, a non-root run from /opt/Gym fails with PermissionError: [Errno 13] Permission denied: 'outputs').

Documentation and Resources

• NeMo Gym on GitHub
• NeMo Gym documentation
• NVFlow on GitHub
• Custom Assembly on Chainguard Academy

What are Chainguard Containers?

Chainguard's free tier of Starter container images are built with Wolfi, our minimal Linux undistro.

All other Chainguard Containers are built with Chainguard OS, Chainguard's minimal Linux operating system designed to produce container images that meet the requirements of a more secure software supply chain.

The main features of Chainguard Containers include:

• Minimal design, without unnecessary software bloat
• Daily builds to ensure container images are up-to-date with available security patches
• High quality build-time SBOMs attesting to the provenance of all artifacts within the image
• Verifiable signatures provided by Sigstore
• Reproducible builds with Cosign and apko (read more about reproducibility)

For cases where you need container images with shells and package managers to build or debug, most Chainguard Containers come paired with a development, or -dev, variant.

In all other cases, including Chainguard Containers tagged as :latest or with a specific version number, the container images include only an open-source application and its runtime dependencies. These minimal container images typically do not contain a shell or package manager.

Although the -dev container image variants have similar security features as their more minimal versions, they include additional software that is typically not necessary in production environments. We recommend using multi-stage builds to copy artifacts from the -dev variant into a more minimal production image.

Need additional packages?

To improve security, Chainguard Containers include only essential dependencies. Need more packages? Chainguard customers can use Custom Assembly to add packages, either through the Console, chainctl, or API.

To use Custom Assembly in the Chainguard Console: navigate to the image you'd like to customize in your Organization's list of images, and click on the Customize image button at the top of the page.

Learn More

Refer to our Chainguard Containers documentation on Chainguard Academy. Chainguard also offers VMs and Libraries — contact us for access.

Trademarks

This software listing is packaged by Chainguard. The trademarks set forth in this offering are owned by their respective companies, and use of them does not imply any affiliation, sponsorship, or endorsement by such companies.