NVIDIA Nemotron 3.5 Lightning: New Open-Source Agent Model, Benchmarks & How to Run It

What You'll Learn

• What Nemotron 3.5 Lightning is and which agent tasks it targets.
• How its active-parameter design, context length and checkpoints affect deployment.
• How to interpret the published benchmarks without treating them as guarantees.
• How to test the model on hosted, self-hosted and local serving paths.

What Is NVIDIA Nemotron 3.5 Lightning?

The NVIDIA Nemotron 3.5 Lightning release is an open 30B mixture-of-experts model with 3B active parameters. NVIDIA introduced it on August 11, 2026, for the execution layer of long-running agents, where a system may make repeated tool calls, validate results and delegate work to subagents.

The model is not positioned as a replacement for every reasoning model. The intended pattern is a model system in which a larger model handles planning or difficult decisions while a smaller active-parameter model handles repeated execution. That division can reduce per-request compute, but the benefit depends on tool-call accuracy, prompt length, concurrency and the cost of the surrounding models.

NVIDIA's public release material includes model weights, post-training data and recipes. The official pages also list BF16 and NVFP4 checkpoints, with additional variants for selected hardware. Check the exact model card and license text before distributing a checkpoint or building a commercial service.

Architecture and Active Parameters

The model uses a hybrid design that combines Mamba-2 layers, mixture-of-experts layers and selected attention layers. A router sends tokens through part of the available expert capacity, so the model has a larger total parameter count than the active computation used for each token.

The 30B total and 3B active figures describe different parts of the same model. Total parameters affect storage and model loading. Active parameters affect the work performed per token. Neither number alone predicts latency because memory bandwidth, context length, quantization, batching and the serving runtime also matter.

The release supports a context window of up to 1M tokens in the published model materials. A long context setting does not mean every deployment can use it at full length. KV cache memory, prompt structure and concurrent requests determine the practical limit.

Key Specifications to Check

The context engineering guide explains why a large context window is useful only when the application manages retrieval, history and cache use deliberately.

What the Published Benchmarks Show

NVIDIA's NIM model material reports results for BF16 and NVFP4 checkpoints across tasks such as MMLU Pro, GPQA Diamond, SWE-bench Verified, PinchBench, Terminal-Bench and IFBench. The figures are useful for identifying the tasks the release team measured, but they are not independent tests of every serving setup.

Benchmark scores should be read with the checkpoint, prompt format, tools, sampling settings and evaluation harness. A score change after quantization may be acceptable for one agent workflow and unacceptable for another. Reproduce the task that matters to your service instead of selecting a checkpoint from one headline number.

Nemotron 3.5 Lightning Compared with Larger Models

The practical comparison is not total parameter count alone. A larger model may produce better decisions on planning or code review, while Nemotron 3.5 Lightning may be a better fit for repeated short actions. The right choice depends on the cost of an error, the number of tokens generated and the time allowed for each tool step.

Claims that Lightning matches or approaches another model should be tied to a named benchmark and evaluation date. Results from Artificial Analysis, NVIDIA, a provider or a community test are not interchangeable. Treat throughput claims as conditional on hardware, precision, context, batch size and runtime.

Teams comparing worker models can start with our AI model comparison guide, then build a local scorecard containing success rate per tool call, retries, latency and cost per completed task.

Hardware and Memory Planning

NVIDIA's published deployment material includes Hopper systems from one H100 to multi-GPU configurations. The developer announcement also discusses local hardware such as NVIDIA Jetson systems, GeForce RTX 5090 and DGX Spark. These references show available paths, not a promise that every configuration supports every context length or checkpoint.

BF16 generally requires more memory than a four-bit NVFP4 checkpoint. The lower-weight format can make a single-GPU test possible, but quantization may change output behavior and kernel availability. Measure model load time, peak memory, first-token latency and steady-state generation on the exact device.

Plan memory for weights, KV cache, runtime buffers, CUDA graphs and the operating system. A model that loads with one request may fail when several long prompts arrive together. Set a maximum context and concurrency policy before exposing a public endpoint.

How to Download and Serve the Model

For self-hosting, start with the official NVIDIA model repositories on Hugging Face and select the checkpoint that matches your hardware and runtime. The official BF16 model page is the correct place to verify files, revisions and license notices rather than relying on a copied command.

For a vLLM deployment, create a clean environment, pin the vLLM and framework versions, then run a short smoke test before sending traffic. Verify tokenizer loading, streaming, structured output, tool-call formatting, cancellation and long-context behavior. The vLLM v0.27.0 upgrade guide covers the dependency checks that matter when a serving runtime changes.

Local users may also encounter Ollama, LM Studio or community GGUF builds. Treat those as separate packaging and runtime paths. Do not transfer a benchmark from the NVIDIA checkpoint to a community conversion without recording the quantization, conversion source and serving arguments.

Hosted Inference and NVIDIA NIM

The NVIDIA Build and NIM pages provide hosted and container-oriented ways to test the model. The NVIDIA NIM getting-started guide should be used for current environment variables, container requirements and supported commands.

Hosted inference reduces local hardware work but adds provider pricing, network latency, rate limits and data-handling questions. Before moving agent traffic, test authentication failures, timeouts, retries, streaming behavior and provider-specific request limits. The cheapest token price is not the same as the lowest cost per completed task.

For a production comparison, run the same prompts through the hosted and self-hosted paths. Record time to first token, total completion time, tool-call success, retries, output length and total spend. Keep provider-specific prompts out of the comparison unless the task requires them.

NeMo Switchyard and Multi-Model Agent Routing

NVIDIA announced NeMo Switchyard alongside Nemotron 3.5 Lightning as a routing layer for systems that use more than one model. A larger model can handle planning while Lightning handles repeated execution, result checking or subagent work. This is a design pattern, not an automatic improvement after installing either project.

Routing adds its own failure modes. A classifier may send a difficult request to the wrong model, shared history may exceed the worker context, and a retry may repeat a tool action. Add request IDs, idempotency keys, model identity, routing decisions and failure reasons to the logs. Our multi-agent protocols guide covers the control-layer contracts that should be tested alongside model routing.

Start with a fixed route for a small traffic sample. Compare it against a single-model baseline, then introduce routing only when the measured reduction in latency or spend is larger than the added control-plane cost.

A Practical Evaluation Plan

Build a test set from real agent tasks after removing secrets and personal data. Include short tool calls, long histories, malformed tool results, structured output, code edits, retries, cancellations and requests that should be refused or escalated.

Run BF16 and NVFP4 on the same hardware when possible. Pin the model revision and serving arguments. Repeat each task enough times to capture variation, then report success rate, time to first token, completion time, tokens per second, peak memory, error rate and cost.

Use a canary before production. Keep the existing worker model available, place the new model behind a traffic limit and define rollback thresholds in advance. A fast model that produces more invalid tool calls is not a service improvement.

Limits and Risks to Review

The model's active-parameter count does not remove memory requirements for the full checkpoint. A 1M-token context does not guarantee affordable long-context serving. Quantization can reduce memory while changing accuracy or kernel behavior. These tradeoffs must be measured on the chosen stack.

License, model provenance and data handling also need review. Confirm that the selected checkpoint is from an official or trusted source, pin its revision and retain the license text used during approval. Hosted providers may apply different retention and logging terms than a local deployment.

Do not use release benchmarks as a substitute for safety testing. Agent systems can call tools, write files or send requests. Apply permissions, allowlists, human review and rollback controls around the model rather than assuming a worker model is harmless because it is smaller.

Our AI fallback guide shows how to keep a second path ready when a model, provider or dependency fails.

Bottom Line

NVIDIA Nemotron 3.5 Lightning is an open 30B MoE model with 3B active parameters, a hybrid Mamba-2 design and a published context length of up to 1M tokens. Its intended role is repeated execution in long-running agents, while larger models can remain responsible for planning or difficult reasoning.

The sensible way to adopt it is to select a checkpoint, pin the runtime, reproduce the relevant benchmark, test real tool calls and compare cost per completed task. Use the official NVIDIA announcement, model pages and NIM guide for release details, then let your own hardware and workload decide whether the model belongs in production.