ai/nemotron3 repository overview

⁠Read our How to Run Nemotron 3 Nano Omni Guide!⁠

See Unsloth Dynamic 2.0 GGUFs⁠ for our quantization benchmarks.

⁠ ⁠ ⁠

⁠Model Overview

⁠Description:

NVIDIA Nemotron 3 Nano Omni is a multimodal large language model that unifies video, audio, image, and text understanding to support enterprise-grade Q&A, summarization, transcription, and document intelligence workflows. It extends the Nemotron Nano family with integrated video+speech comprehension, Graphical User Interface (GUI), Optical Character Recognition (OCR), and speech transcription capabilities, enabling end-to-end processing of rich enterprise content such as meeting recordings, M&E assets, training videos, and complex business documents. NVIDIA Nemotron 3 Nano Omni was developed by NVIDIA as part of the Nemotron model family.

This model is available for commercial use.

This model was improved using Qwen3-VL-30B-A3B-Instruct, Qwen3.5-122B-A10B, Qwen3.5-397B-A17B, Qwen2.5-VL-72B-Instruct, and gpt-oss-120b. For more information, please see the Training Dataset section below.

⁠License/Terms of Use

Governing Terms: Use of this model is governed by the NVIDIA Open Model Agreement⁠

⁠Deployment Geography:

Global

⁠Use Case:

This model is designed for enterprise customers requiring multimodal understanding capabilities. Expected users include:

• Customer service applications (e.g., Doordash video of drop-off at a given address via OCR, drive-thru order verification)
• Media and Entertainment (M&E) — video and speech analysis, dense captions, video search and summarization
• Document intelligence for AI assistants (contracts, SOW/MSA, scientific discovery, financial documents)
• GUI automation for AI agentic applications (incident management, agentic search, browser agents, email agents)

⁠Release Date:

Build.Nvidia.com 04/28/2026 via URL⁠
Hugging Face 04/28/2026 via:

• BF16⁠
  
• FP8⁠
  
• NVFP4⁠
  

NGC 04/28/2026 via URL⁠

⁠Model Architecture:

Architecture Type: Mamba2-Transformer Hybrid Mixture of Experts (MoE)

Network Architecture:

• Nemotron 3 Nano LLM (30B A3B)⁠
• CRADIO v4-H vision encoder
• Parakeet speech encoder

Number of model parameters: 3.1 x 10^10 (31B A3B)

⁠Input(s):

Input Type(s): Video, Audio, Image, Text

Input Format(s):

• Video: mp4, up to 2 minutes. For 1080p videos, sample up to 1 FPS / 128 frames. For lower-resolution videos such as 720p, higher temporal sampling such as 2 FPS / 256 frames may be used.
• Audio: wav, mp3 files (up to 1 hour), 8kHz and higher sampling rates
• Image: Red, Green, Blue (RGB) (jpeg, png)
• Text: String
  

Input Parameters:

• Video: Three-Dimensional (3D)
• Audio: One-Dimensional (1D)
• Image: Two-Dimensional (2D)
• Text: One-Dimensional (1D)
  

Other Properties Related to Input:

• Maximum context length up to 256k tokens
• Language support: English only
  

⁠Output(s)

Output Type(s): Text

Output Format(s):

• Text: String
  

Output Parameters:

• Text: One-Dimensional (1D)
  

Other Properties Related to Output:

• Maximum context length up to 256k tokens.
• Supports JSON output format
• Supports reasoning output with chain-of-thought
• Supports tool calling
• Supports word-level timestamps for transcription
  

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

⁠Software Integration:

Runtime Engine(s):

• vLLM
  
• NeMo
  
• Megatron
  
• NeMo-RL
  

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere (A100 80GB SXM/NVLink)
  
• NVIDIA Blackwell (B200 SXM/NVLink, RTX Pro 6000 SE, DGX Spark, Jetson Thor, RTX 5090)
  
• NVIDIA Hopper (H100 SXM/NVLink, H200 SXM/NVLink)
  
• NVIDIA Lovelace (L40S)
  

Preferred/Supported Operating System(s):

• Linux
  

Inference Runtimes:

• vLLM
  
• TensorRT LLM
  
• TensorRT Edge-LLM
  
• llama.cpp
  
• Ollama
  
• SGLang
  

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

This AI model can be embedded as an Application Programming Interface (API) call into the software environment described above.

⁠Model Version(s):

Nemotron-3-Nano-Omni-30B-A3B-Reasoning

⁠Quick Start Guide

⁠Model Parameters

⁠Download Model Weights

⁠Install the HuggingFace CLI

pip install -U "huggingface_hub[hf_xet]"
 
# Log in once; the token is cached at ~/.cache/huggingface/token
hf auth login
 
# Sanity check: should print your username and orgs
hf auth whoami

⁠Download the weights

Pick a target directory on a volume with ≥70 GB free (the model is ~62 GB).

WEIGHTS=/path/to/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16

hf download nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
  --local-dir "$WEIGHTS" \
  --max-workers 8

Notes:

⁠Verify the download

ls "$WEIGHTS" | head
du -sh "$WEIGHTS"  # expect ~62 GB
test -f "$WEIGHTS/config.json" && echo OK

⁠vLLM

⁠Container

docker pull vllm/vllm-openai:v0.20.0

⁠General Invocation (1×GPU, e.g. 1×B200)

# vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
# vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \
  --host 0.0.0.0 \
  --max-model-len 131072 \
  --tensor-parallel-size 1 \
  --trust-remote-code \
  --video-pruning-rate 0.5 \
  --max-num-seqs 384 \
  --allowed-local-media-path / \
  --media-io-kwargs '{"video": {"fps": 2, "num_frames": 256}}' \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --kv-cache-dtype fp8 # Omit this for BF16

⁠Platform-Specific Notes

⁠vLLM on DGX Spark (aarch64 / ARM64)

For everything not covered here (API examples, reasoning mode, video tuning), follow the general instructions.

⁠1. Pull the container image

Use the upstream multi-arch vLLM v0.20.0 docker image. Docker will automatically pull the arm64 variant.

docker pull vllm/vllm-openai:v0.20.0

⁠2. Launch the vLLM server on Spark

WEIGHTS=/path/to/nemotron-3-nano-omni-weights

# The image does not include audio packages so we need to install them with "pip install vllm[audio]" as done in the command below
docker run --rm -it \
  --gpus all \
  --ipc=host -p 8000:8000 \
  --shm-size=16g \
  --name vllm-nemotron-omni \
  -v "${WEIGHTS}:/model:ro" \
  --entrypoint /bin/bash \
  vllm/vllm-openai:v0.20.0 -c  \
  "pip install vllm[audio] && vllm serve /model \
  --served-model-name=nemotron_3_nano_omni \
  --max-num-seqs 8 \
  --max-model-len 131072 \
  --port 8000 \
  --trust-remote-code \
  --gpu-memory-utilization 0.8 \
  --limit-mm-per-prompt '{\"video\": 1, \"image\": 1, \"audio\": 1}' \
  --media-io-kwargs '{\"video\": {\"fps\": 2,  \"num_frames\": 256}}' \
  --allowed-local-media-path=/ \
  --enable-prefix-caching \
  --max-num-batched-tokens 32768 \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder"

In another terminal, verify the server is ready:

curl -sS http://localhost:8000/v1/models | python3 -m json.tool

⁠Key Spark-Specific Flags

⁠Memory Tuning on Spark

Spark uses unified LPDDR5X memory (~128 GB shared between CPU and GPU), not separate system + VRAM pools. Two levers, in order of impact:

1. Lower --gpu-memory-utilization from 0.85 → 0.70 to free ~19 GB back to the OS and re-enable weight prefetch. Cost: smaller KV cache budget.
2. Lower --max-model-len to reduce KV cache allocation (e.g. halving context window halves KV cache at --max-num-seqs=1). Combined override:

  --gpu-memory-utilization=0.70 \
  --max-model-len=32768 \

⁠TensorRT-LLM

This model can also be deployed with TensorRT-LLM - see relevant instructions here⁠.

⁠Platform-Specific Notes

⁠TensorRT Edge-LLM

This model can also be deployed with TensorRT Edge-LLM on NVIDIA Jetson Thor - see the Jetson AI Lab model page⁠ and the TensorRT Edge-LLM Quick Start Guide⁠.

⁠SGLang

The BF16 variant of this model is supported on SGLang, with the following images:

• CUDA 13.0: lmsysorg/sglang:dev-cu13-nemotronh-nano-omni-reasoning-v3⁠
• CUDA 12.9: lmsysorg/sglang:dev-nemotronh-nano-omni-reasoning-v3⁠

librosa must be installed first: pip install librosa --break-system-packages

To serve: sglang serve --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --trust-remote-code

⁠Platform-Specific Notes

⁠SGLang on DGX Spark (aarch64 / ARM64)

For everything not covered here (API examples, reasoning mode, video tuning), follow the general instructions.

⁠1. Pull the container image

Use the upstream multi-arch CUDA 13.0 docker image linked above. Docker will automatically pull the arm64 variant.

docker pull lmsysorg/sglang:dev-cu13-nemotronh-nano-omni-reasoning-v3

⁠2. Launch the SGLang server on Spark

WEIGHTS=/path/to/nemotron-3-nano-omni-weights

# The image does not include audio packages so we need to install them with "pip install librosa" as done in the command below
docker run --gpus all -it --rm \
  -p 30000:30000 \
  -v "${WEIGHTS}:/model:ro" \
  --shm-size 16g \
  lmsysorg/sglang:dev-cu13-nemotronh-nano-omni-reasoning-v3 \
  bash -c "pip install librosa && python3 -m sglang.launch_server --model-path /model \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --mem-fraction-static 0.8 \
  --max-running-requests 8 \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_3"

In another terminal, verify the server is ready:

curl -sS http://localhost:30000/v1/models | python3 -m json.tool

⁠Key Spark-Specific Flags

⁠Memory Tuning on Spark

Spark uses unified LPDDR5X memory (~128 GB shared between CPU and GPU), not separate system + VRAM pools. Two levers, in order of impact:

1. Lower --mem-fraction-static from 0.80 → 0.70 to free ~13 GB back to the OS and re-enable weight prefetch. Cost: smaller KV cache budget.
2. Lower --context-length to reduce KV cache allocation (e.g. halving context window halves KV cache at --max-running-requests=1). Combined override:

  --mem-fraction-static=0.70 \
  --context-length=32768 \

⁠API Client (OpenAI-compatible)

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="")
MODEL = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4"

Image Example

import base64
 
def image_to_data_url(path: str) -> str:
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode("utf-8")
    return f"data:image/jpeg;base64,{b64}"
 
image_url = image_to_data_url("media/example1a.jpeg")
 
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image in detail."},
                {"type": "image_url", "image_url": {"url": image_url}},
            ],
        }
    ],
    max_tokens=1024,
    temperature=1.0,
    extra_body={"top_k": 1, "chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)

Audio Example

from pathlib import Path
 
audio_url = Path("media/2414-165385-0000.wav").resolve().as_uri()
 
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "audio_url", "audio_url": {"url": audio_url}},
                {"type": "text", "text": "Transcribe this audio."},
            ],
        }
    ],
    max_tokens=1024,
    temperature=1.0,
    extra_body={"top_k": 1, "chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)

Video Example

from pathlib import Path
 
video_url = Path("media/demo.mp4").resolve().as_uri()
reasoning_budget = 16384
grace_period = 1024
 
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "video_url", "video_url": {"url": video_url}},
                {"type": "text", "text": "Describe this video."},
            ],
        }
    ],
    max_tokens=20480,
    temperature=0.6,
    top_p=0.95,
    extra_body={
        "thinking_token_budget": reasoning_budget + grace_period,
        "chat_template_kwargs": {
            "enable_thinking": True,
            "reasoning_budget": reasoning_budget,
        },
        "mm_processor_kwargs": {"use_audio_in_video": False},
    },
)
print(response.choices[0].message.content)

Text Example (curl)

curl -sS http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4","messages":[{"role":"user","content":"Hello, what can you do?"}],"temperature":1.0,"top_k":1}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['choices'][0]['message']['content'])"

PDF Example (page-by-page via Python)

The API accepts images, not raw PDF files. The script below renders each page to PNG and sends it as base64. Save as pdf_vlm_chat.py and install dependencies: pip install pymupdf pillow requests.

#!/usr/bin/env python3
"""Send PDF page(s) as images to a vLLM /v1/chat/completions endpoint."""
from __future__ import annotations
 
import argparse, base64, sys
from io import BytesIO
from pathlib import Path
 
import requests
 
try:
    import fitz
    from PIL import Image
except ImportError:
    print("Install: pip install pymupdf pillow requests", file=sys.stderr)
    sys.exit(1)
 
USER_PROMPT = (
    "Summarize this PDF page: main topic, section headings, important facts "
    "or bullets, and a brief note on each figure or table. "
    "Do not invent text you cannot read."
)
API_URL = "http://localhost:8000/v1/chat/completions"
MODEL = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4"
MAX_TOKENS = 32000
DPI = 150
 
 
def page_to_b64(pdf_path: str, idx: int) -> str:
    doc = fitz.open(pdf_path)
    z = DPI / 72.0
    pix = doc.load_page(idx).get_pixmap(matrix=fitz.Matrix(z, z))
    img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
    doc.close()
    buf = BytesIO()
    img.save(buf, format="PNG")
    return base64.b64encode(buf.getvalue()).decode("ascii")
 
 
def chat(url, model, b64, text, max_tokens):
    r = requests.post(url, json={
        "model": model,
        "messages": [{"role": "user", "content": [
            {"type": "text", "text": text},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        ]}],
        "max_tokens": max_tokens,
        "stream": False,
        "temperature": 1.0,
        "chat_template_kwargs": {"enable_thinking": False},
    }, timeout=120)
    r.raise_for_status()
    return r.json()["choices"][0]["message"]["content"]
 
 
def main():
    p = argparse.ArgumentParser()
    p.add_argument("pdf")
    p.add_argument("--page", type=int, default=0)
    p.add_argument("--all-pages", action="store_true")
    p.add_argument("-o", "--output")
    p.add_argument("--url", default=API_URL)
    p.add_argument("--model", default=MODEL)
    p.add_argument("--max-tokens", type=int, default=MAX_TOKENS)
    a = p.parse_args()
 
    doc = fitz.open(a.pdf); n = len(doc); doc.close()
    pages = range(n) if a.all_pages else [a.page]
    parts = [f"# Extracted: {Path(a.pdf).name}\n\n*Pages: {n}*\n"] if a.all_pages else []
 
    for i in pages:
        print(f"Page {i+1}/{n} ...", file=sys.stderr)
        b64 = page_to_b64(a.pdf, i)
        text = chat(a.url, a.model, b64, f"Page {i+1}.\n\n{USER_PROMPT}", a.max_tokens)
        parts.append(f"\n---\n\n## Page {i+1}\n\n{text.strip()}\n" if a.all_pages else text.strip())
 
    out = "\n".join(parts)
    if a.output:
        Path(a.output).write_text(out + "\n", encoding="utf-8")
    else:
        print(out)
 
if __name__ == "__main__":
    main()

Single page:

python3 pdf_vlm_chat.py /path/to/your_document.pdf --page 0

All pages to markdown:

python3 pdf_vlm_chat.py /path/to/your_document.pdf --all-pages -o extracted.md

Edit USER_PROMPT in the script for different tasks (detailed extraction, table parsing, etc.).

⁠Reasoning Mode (enable_thinking)

To disable reasoning on a request, add to the JSON body:

"chat_template_kwargs": {"enable_thinking": false}

In the Python heredoc pattern, use False (Python boolean), not false (invalid Python).

We recommend thinking mode for tasks that involve reasoning and complex understanding. For video, audio, and omni use cases, try both enabling and disabling thinking for best results.

from typing import Any, Dict, List

from openai import OpenAI
from transformers import AutoTokenizer


class ThinkingBudgetClient:
    def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str):
        self.tokenizer = AutoTokenizer.from_pretrained(
            tokenizer_name_or_path, trust_remote_code=True
        )
        self.client = OpenAI(base_url=base_url, api_key=api_key)

    def chat_completion(
        self,
        model: str,
        messages: List[Dict[str, Any]],
        reasoning_budget: int = 512,
        max_tokens: int = 1024,
        **kwargs,
    ) -> Dict[str, Any]:
        assert max_tokens > reasoning_budget, (
            f"reasoning_budget must be less than max_tokens. "
            f"Got {max_tokens=} and {reasoning_budget=}"
        )

        # Step 1: generate only the reasoning trace up to the requested budget.
        response = self.client.chat.completions.create(
            model=model,
            messages=messages,
            max_tokens=reasoning_budget,
            extra_body={
                "top_k": 1,
                "chat_template_kwargs": {
                    "enable_thinking": True,
                },
            },
            **kwargs,
        )
        reasoning_con

…(truncated — see the full README on HuggingFace)