nemo-skills

packaged by Chainguard

Chainguard Container for nemo-skills

NVIDIA NeMo-Skills is a collection of pipelines for synthetic data generation, evaluation, and training data preparation for large language models. This image is a drop-in worker for NVFlow's nemo-skills image.

Chainguard Containers are regularly-updated, secure-by-default container images.

Download this Container Image

For those with access, this container image is available on cgr.dev:

docker pull cgr.dev/ORGANIZATION/nemo-skills:latest

Be sure to replace the ORGANIZATION placeholder with the name used for your organization's private repository within the Chainguard Registry.

Compatibility Notes

The nemo-skills image is designed as a drop-in replacement for NVFlow's nemo-skills worker image, built from dockerfiles/Dockerfile.nemo-skills on NVFlow's feat/v1.1.2/ray-dev branch (tested at commit cfdf630c). It ships the same NeMo-Skills commit (1fd0a9f, on NeMo-Skills' Ray integration line), unmodified, with the same Python packages resolved for Python 3.12 (see the differences below). The following match NVFlow's image, so NVFlow workflows run on it unchanged:

• Source tree. The NeMo-Skills source is at /opt/NeMo-Skills, which NeMo-Skills' Ray backend looks for when a Ray job's working directory carries only the caller's code. It links to the Chainguard tree under /opt/chainguard/nemo-skills/src.
• Python. Bare python and python3 import nemo_skills and every dependency, including NVFlow's additions: the SEC data-prep packages (edgartools, sec-parser), tiktoken, tavily, model-library, and the rest of NVFlow's list. The ns CLI is on PATH.
• Offline data. The cl100k_base tiktoken encoding is in /opt/tiktoken_cache, with TIKTOKEN_CACHE_DIR set. The nltk data (punkt, punkt_tab, stopwords, averaged_perceptron_tagger_eng) and the spaCy en_core_web_sm model are baked in, so they load with no network.
• Tools. git, git-lfs, ffmpeg, curl, wget, ssh and scp (which ns start_server tunnels and git over SSH use), lscpu, and the timezone database are included.
• User and command. The container runs as root, starts bash, and its work directory is /.

The image differs from NVFlow's in the following ways. Each difference is deliberate:

• Python 3.12. NVFlow's image uses Ubuntu 22.04's Python 3.10. NeMo-Skills supports Python 3.10 and later, and NVFlow's own client image already runs it on 3.12. The interpreter and packages are in a virtual environment at /opt/chainguard/nemo-skills/venv, first on PATH. There is no other python3 in the image.
• No pip or uv is shipped in the venv. A few NeMo-Skills dataset preparation scripts install packages when they run (bfcl_v3 and bfcl_v4 clone and install BFCL); like NVFlow's image, this one does not support the BFCL benchmarks. See Features that install packages at run time.
• Dependencies are resolved for Python 3.12, so packages that NVFlow's image holds back (for Python 3.10, or through pins left by BFCL) are newer here: scipy 1.15.3 → 1.18.1, scikit-learn 1.7.2 → 1.9.1, librosa 0.11.0 → 1.0.0, ipython 8.39.0 → 9.17.1, networkx 3.3 → 3.7, inflect 5.6.2 → 7.5.0, deprecated 1.3.1 → 3.0.0, faiss-cpu 1.11.0 → 1.15.1, gradio 6.15.1 → 6.29.0 (gradio-client 2.5.0 → 2.7.1), websockets 15.0.1 → 16.1.1, isort 5.13.2 → 8.0.1, rpds-py 0.30.0 → 2026.6.3, and httpx-sse 0.4.0 → 0.4.3. torchcodec is the same 0.17.0, from the PyTorch CPU index rather than PyPI. Every other package the two images share is at the same version.
• Security fixes move two dependencies past their declared pins: wandb 0.30.0 (NeMo-Skills pins 0.28.1, whose bundled Go binary has High findings that NVFlow fixes by rebuilding it from source) and anyio 4.14.2 (leptonai declares 4.9.0 or lower). The bundled wandb-core binary is rebuilt from wandb's 0.30.0 release with Go 1.27.2 and golang.org/x/net 0.60.0, which fix Critical and High findings in the binary 0.30.0 ships. starlette and lxml are forced past their caps, as in NVFlow's image.
• Not included:
  • apptainer, which only NeMo-Skills' SWE-bench evaluation uses;
  • the gorilla (BFCL), IFBench, and google-research checkouts, which NVFlow's Dockerfile deletes as well;
  • the packages that installing BFCL and IFBench leaves behind in NVFlow's image, such as cohere, qwen-agent, writer-sdk, and the tree-sitter grammars. Nothing in NeMo-Skills or the NVFlow workflows imports them.
• RAY_JOB_STOP_SIGNAL=SIGINT is set. Without it, Ray starts job commands with SIGINT blocked, and every process in the job inherits the block, so SIGINT and Ctrl-C handlers inside Ray jobs never run. NVFlow's image doesn't need the variable.
• FFmpeg 9, the current FFmpeg stream, which torchcodec's FFmpeg 9 backend links against. NVFlow's image has Ubuntu's FFmpeg 4.4.
• librosa 1.0 no longer uses audioread, so it cannot decode formats such as m4a or AAC through librosa.load. NeMo-Skills does not import librosa, and the sdp processors it uses read FLAC and WAV.
• numpy comes from PyPI. Chainguard Libraries' builds of numpy 2.1.1 to 2.1.3 for Python 3.12 crash in np.nonzero, and compute-eval requires numpy 2.1, so the image uses PyPI's numpy 2.1.3 wheel.
• x86_64 only.

Features that install packages at run time

These NeMo-Skills features install Python packages when they run, so they fail in this image. They also fail in NVFlow's image when the cluster is offline, and none of NVFlow's shipped configurations use them:

• translation BLEU for Japanese and Korean, which needs sacrebleu's MeCab extras;
• local LiveCodeBench evaluation, which installs livecodebench;
• the NV-Embed judge, which installs its own transformers;
• ns generate's requirements option, which creates a venv with uv;
• any installation_command that runs pip.

Getting Started

Run the ns CLI:

docker run --rm cgr.dev/ORGANIZATION/nemo-skills:latest ns --help

Prepare a benchmark whose data is in the source tree, with no network access. The prepared file is written into the source tree:

docker run --rm --network=none cgr.dev/ORGANIZATION/nemo-skills:latest \
  bash -c 'cd /tmp && ns prepare_data aime24 && wc -l /opt/NeMo-Skills/nemo_skills/dataset/aime24/test.jsonl'

The edgartools package's module is edgar, so import edgartools in NVFlow's third sanity check fails in NVFlow's image too. Apart from that module name, NVFlow's sanity checks for this image run unchanged:

docker run --rm --network=none -e HF_HUB_OFFLINE=1 cgr.dev/ORGANIZATION/nemo-skills:latest \
  python3 -c 'import tiktoken; tiktoken.get_encoding("cl100k_base"); print("OK")'
docker run --rm cgr.dev/ORGANIZATION/nemo-skills:latest \
  python3 -c 'import pyarrow as pa; pa.array([], type=pa.timestamp("ns", tz="UTC")); print("OK")'
docker run --rm cgr.dev/ORGANIZATION/nemo-skills:latest bash -c \
  'python3 -c "import jsonlines, tiktoken, markdownify, backoff, func_timeout, tavily, edgar, sec_parser, pandas, pyarrow, requests, bs4; print(\"OK\")"'

Configuration

NVFlow runs this image on air-gapped Slurm clusters through enroot. Import it as NVFlow's docker_instructions.md describes, using # between the registry host and the image path:

enroot import --output nemo-skills-latest.sqsh \
  "docker://cgr.dev#ORGANIZATION/nemo-skills:latest"

Then point the nemo-skills entry of the cluster configuration's containers section at the .sqsh file. Keep NVFlow's air-gap variables in env_vars: HF_HUB_OFFLINE=1, HF_DATASETS_OFFLINE=1, TRANSFORMERS_OFFLINE=1, and TIKTOKEN_CACHE_DIR=/opt/tiktoken_cache, which matches what the image sets.

NeMo-Skills' Slurm launcher runs every container with --no-container-mount-home, and ns creates ~/.nemo-evaluator when it starts. The container's root filesystem must therefore be writable, which is pyxis's default (don't pass --container-readonly), or HOME must point to a writable directory. With a read-only root and no writable home (for example, a plain enroot start without --rw), ns exits with Read-only file system: '<home>/.nemo-evaluator'. To run it there, set HOME to a writable directory, for example -e HOME=/tmp with a tmpfs on /tmp, as the image's read-only-root test does. NVFlow's image behaves the same way.

Documentation and Resources

• NeMo-Skills documentation
• NeMo-Skills on GitHub
• NVFlow on GitHub
• Custom Assembly on Chainguard Academy

What are Chainguard Containers?

Chainguard's free tier of Starter container images are built with Wolfi, our minimal Linux undistro.

All other Chainguard Containers are built with Chainguard OS, Chainguard's minimal Linux operating system designed to produce container images that meet the requirements of a more secure software supply chain.

The main features of Chainguard Containers include:

• Minimal design, without unnecessary software bloat
• Daily builds to ensure container images are up-to-date with available security patches
• High quality build-time SBOMs attesting to the provenance of all artifacts within the image
• Verifiable signatures provided by Sigstore
• Reproducible builds with Cosign and apko (read more about reproducibility)

For cases where you need container images with shells and package managers to build or debug, most Chainguard Containers come paired with a development, or -dev, variant.

In all other cases, including Chainguard Containers tagged as :latest or with a specific version number, the container images include only an open-source application and its runtime dependencies. These minimal container images typically do not contain a shell or package manager.

Although the -dev container image variants have similar security features as their more minimal versions, they include additional software that is typically not necessary in production environments. We recommend using multi-stage builds to copy artifacts from the -dev variant into a more minimal production image.

Need additional packages?

To improve security, Chainguard Containers include only essential dependencies. Need more packages? Chainguard customers can use Custom Assembly to add packages, either through the Console, chainctl, or API.

To use Custom Assembly in the Chainguard Console: navigate to the image you'd like to customize in your Organization's list of images, and click on the Customize image button at the top of the page.

Learn More

Refer to our Chainguard Containers documentation on Chainguard Academy. Chainguard also offers VMs and Libraries — contact us for access.

Trademarks

This software listing is packaged by Chainguard. The trademarks set forth in this offering are owned by their respective companies, and use of them does not imply any affiliation, sponsorship, or endorsement by such companies.