Skip to content

Infinity V3 ♾️

Documentation for the native v3 server. Quickstart · Python / v2 documentation

Infinity V3 is a high-throughput, low-latency REST API for serving embeddings, reranking, classification and typed decision models. The server is written in Rust and uses LibTorch, the C++ interface of PyTorch, for model inference.

Looking for Python / v2? See the Python server README and v2 documentation. The infinity-emb Python package and infinity_emb v2 command belong to that server.

History

  • 2026/10: Native Rust/LibTorch v3 server for embeddings, reranking, classification and typed decisions.
  • 2025/07: Blackwell support in the Python server.
  • 2024: Python / v2 gained multi-model and multimodal serving, Python clients, and broader GPU, CPU, ONNX and TensorRT deployment options.
  • 2023/10: Initial Infinity release.

See the Python / v2 README for the full earlier updates.

Logo Baseten.co Logo Runpod Logo TrueFoundry Logo Vast Logo DataGuard Logo SAP Logo Nosible Logo FreshWorks Logo Dstack Logo JamAI Logo Alibaba Group Logo BentoML Logo JinaAi Logo Dwarves Foundation Logo HF Logo Gradient.ai

Performance highlights

  • High throughput: token-based dynamic batching and dedicated tokenization workers.
  • Memory-efficient batches: ragged attention avoids padding variable-length inputs, reducing memory waste and OOM risk.
  • Less repeated computation: from the authors of RadixMLP, shared-prefix deduplication within a batch for supported decision models and rerankers.
  • Low latency: BERT-small runs in ~850 µs end-to-end through the Rust/C++ serving path, including ~250 µs HTTP/MessagePack, ~100 µs batching and ~500 µs inference with CUDA graphs.
  • Fast startup: ModernBERT-base 0.92 s, Qwen3-8B 2.51 s on one H100 80 GB.
  • Optimized inference: native LibTorch kernels, FlashAttention-4 and optional dynamic FP8 for supported models.

Infinity aims to outperform vLLM and SGLang across embedding, reranking and decision models. Reported October 2026 benchmark highlights versus vLLM: ~2× throughput on Mammoth, ~5× on ModernBERT and ~1.1× on Qwen3-Embedding-0.6B. Speedups depend on the workload and configuration. Startup timings above use local weights; see the startup measurements.

CUDA support and attention kernels

The NVIDIA image supports Ampere through Blackwell (SM80–SM120), with builds for the architectures below. FlashAttention-4 is enabled for supported FP16/BF16 shapes; other shapes use existing attention kernels. DeBERTa-v2/v3 uses a novel CuTeDSL-based algorithm in FA4 to fuse relative-position lookup without quadratic bucket or attention-bias storage.

Compute capability Example GPUs
SM80 A100, A30
SM86 A10 / A10G, GeForce RTX 3090, RTX A6000
SM89 L4, L40S, RTX 6000 Ada, GeForce RTX 4090
SM90 H100, H200
SM100 B200
SM120 RTX PRO 6000 Blackwell, GeForce RTX 5090

FA4 has runtime validation on H100 and compile/link coverage on the other listed targets. See the backend documentation for supported attention shapes and the INFINITY_TORCH_FA4=0 fallback switch.

FP16 standard BERT CLS embeddings with 384 hidden features / 12 attention heads or 1024 hidden features / 16 attention heads use the existing native attention, LayerNorm and cached cuBLASLt projection paths on supported CUDA GPUs (SM80 and newer), subject to the existing GEMM policy and fallback guards. CUDA graphs remain opt-in with INFINITY_TORCH_CUDA_GRAPHS=1.

Getting started

Launch a compatible Hugging Face model or a local checkpoint:

infinity_v3 --model-id sentence-transformers/all-MiniLM-L6-v2

Check infinity_v3 --help for CLI options and environment variables. Open /docs on the running server for the interactive API, or /api-doc/openapi.json for its OpenAPI schema. See running the server and building Infinity V3.

CLI overview

Serve an embedding model on one NVIDIA GPU:

infinity_v3 \
  --model-id Qwen/Qwen3-Embedding-0.6B \
  --served-model-name embeddings \
  --torch-device cuda --device-id 0 --dtype bfloat16 \
  --max-sequence-length 8192 --max-batch-tokens 32768 \
  --hostname 0.0.0.0 --port 8000

Clients use "model": "embeddings" in requests. For CPU inference, use --torch-device cpu --dtype float32. Omit --device-id to use all visible CUDA GPUs as replicas; each GPU loads a complete model, rather than a shard.

Model, precision and device options
Option Purpose
--model-id Hugging Face model ID or local checkpoint directory. Repeat to serve multiple models.
--revision Pin a Hugging Face commit, tag or branch.
--served-model-name Public model name accepted by the API; defaults to the checkpoint ID.
--torch-device auto, cpu or cuda.
--dtype auto, float32, float16 or bfloat16; CUDA attention requires FP16/BF16.
--device-id Run on one visible accelerator ordinal, such as 0.
--backend-device-ids Select CUDA replicas, such as 0,1, or auto.
--enable-fp8-dynamic Enable experimental dynamic FP8 for supported models and GPUs. Off by default; validate task accuracy before enabling.
Batching, input limits and model behavior
Option Purpose
--max-batch-tokens Total token budget per batch; defaults to 16384. Larger batches need more memory.
--max-sequence-length Token limit for one sequence, including prompts and image tokens; bounded by the model and batch limits.
--max-batch-requests Optional maximum number of individual requests in a batch.
--max-concurrent-requests Admission limit per model; defaults to 512.
--max-client-batch-size Maximum number of inputs in one client batch.
--auto-truncate=false Reject oversized inputs instead of truncating. Truncation is enabled by default; decision and multimodal requests have additional restrictions.
--pooling Override the checkpoint's embedding pooling configuration when the model supports it. Normally leave unset.
--default-prompt-name Select a saved query/document prompt by name.
--default-prompt Use a literal prefix instead; cannot be combined with --default-prompt-name.
--decision-protocol Select onejev, rune, clef, pplx or d3 for a compatible decision checkpoint.

For example, an 8192 sequence limit with a 32768 batch budget allows several sequences in a batch while keeping each individual input at or below 8192 tokens. See sequence limits for details.

HTTP, media and observability options
Option Purpose
--hostname, --port HTTP bind address and port; defaults to 0.0.0.0:3000.
--api-key Require Authorization: Bearer … on API requests. Can also be supplied through INFINITY_API_KEY.
--image-allowed-hosts Comma-separated host allowlist replacing the defaults; --image-allowed-hosts= permits inline media only.
--image-memory-budget-mib Memory budget for encoded images, preprocessing and prepared patches.
--json-output Write structured JSON logs.
--prometheus-port Prometheus metrics port; defaults to 9000.
--huggingface-hub-cache Model download cache directory.
Multiple models and environment variables

Repeat --model-id and align per-model settings in the same order:

infinity_v3 \
  --model-id sentence-transformers/all-MiniLM-L6-v2 \
  --model-id BAAI/bge-reranker-base \
  --served-model-name embeddings --served-model-name reranker \
  --torch-device cuda --dtype bfloat16 \
  --device-id 0 --device-id 1 --port 8000

A single per-model setting applies to every model. Multiple values must match the number of models; semicolon-separated values are also accepted. HTTP bind settings are shared by the server.

Official builds expose INFINITY_* environment variables for CLI options:

INFINITY_MODEL_ID=sentence-transformers/all-MiniLM-L6-v2 \
INFINITY_TORCH_DEVICE=cpu \
INFINITY_DTYPE=float32 \
INFINITY_PORT=8000 \
infinity_v3

Use quotes for lists, such as INFINITY_DEVICE_ID="0;1". CLI arguments take precedence over environment values. Run infinity_v3 --help for the complete option list and the environment names used by your build.

Supported Tasks and Models by Infinity V3

Click on the sections below to find tasks and example models. Support depends on the checkpoint architecture and trained output head. The models below use the LibTorch backend. Dense and Mixture-of-Experts (MoE) architectures are supported, including Qwen3, Qwen3.5 and Gemma4 MoE models. Use checkpoints with a supported task head; each GPU replica must fit the full model, including all experts.

The supported model catalog groups named checkpoints by task with launch settings, input requirements and request examples. The native model coverage lists implemented LibTorch paths and their limitations; it is not a claim that every checkpoint in each family has been validated.

Text Embeddings — BGE, GTE, ModernBERT, Qwen3 and more

Text embeddings represent sentences and documents as vectors for semantic search, clustering and recommendations. Use /v1/embeddings or /embed.

Example models:

Native LibTorch families also include RoBERTa, XLM-RoBERTa, CamemBERT, DistilBERT, ModernBERT, Nomic BERT, Jina BERT/JinaCode, MPNet, Llama, Mistral/Ministral3, Qwen2, Qwen3/MoE and Gemma3 text. Use the checkpoint's query/document instructions and pooling configuration.

Reranking

Given a query and a list of documents, reranking scores the documents for relevance. Use /rerank with a supported sequence-classification checkpoint and its trained scoring head.

Example models:

LibTorch supports architecture-specific encoder and decoder classification heads, including experimental DeBERTa-v2/v3. A generative language-model checkpoint does not automatically become a reranker.

Multi-modal Embeddings — Qwen3-VL, EmbeddingGemma2 and Nemotron

Multi-modal models encode text and media for retrieval. Send images as image_url content parts in user messages on /v1/embeddings.

Models and native LibTorch paths:

  • Qwen3-VL Embedding: 2B and 8B, text/image embeddings with last-token pooling.
  • EmbeddingGemma2 (embedding_gemma2): text embeddings with the checkpoint's trained projection, plus image and audio inputs when the corresponding towers are present.
  • Mistral3 / Ministral 3: Pixtral vision tower and native text/image preprocessing. See the multimodal input recipe.
  • Qwen3.5 dense and MoE: native text/image backbones with full attention, DeltaNet and vision processing.
  • Gemma4 dense and MoE: native text/image backbones with routed experts for MoE checkpoints. Gemma3 support is text-only.

Backbone support requires a checkpoint with an appropriate trained output head; it does not make every instruction model a retrieval embedding model.

Nemotron 3 and 3.5 dense embeddings: the Ministral3/Mistral3 backbones are implemented. Supported checkpoints include Nemotron 3 1B, Nemotron 3 8B, Nemotron 3.5 1B and Nemotron 3.5 8B with mean pooling and the checkpoint's query/document instructions. Nemotron 3 is text-only; 3.5 adds document images. See the model catalog for launch settings.

For token-vector retrieval, see ColBERT, ColQwen3 and Nemotron ColEmbed below. Full architecture details are in native model coverage.

Late-interaction Embeddings — ColBERT, ColQwen3 and Nemotron ColEmbed

Late-interaction models return a matrix of token vectors instead of one pooled vector. Use MaxSim scoring in a compatible retrieval system.

Supported models:

ColBERT returns text token vectors; ColQwen3 and Nemotron ColEmbed also encode images. ColQwen2/2.5 adapters and the Python server's other ColBERT variants are not implied by these native implementations.

Use the checkpoint's query/document formatting. Responses preserve the token matrix through float, base64 or packed tensor encoding. Sentence pooling, dimension overrides and single-vector /similarity scoring are unsupported for these models. See embedding formats and client usage and model-specific input requirements.

Sparse Embeddings and Text Classification

SPLADE checkpoints produce sparse vocabulary vectors through /embed_sparse. LibTorch implements SPLADE heads for BERT and DistilBERT; head support is architecture-specific.

Text classification returns trained label scores through /v1/classify. Supported encoder and decoder families need a compatible classification head and label configuration. See native model coverage.

Typed Decisions — OneJev, Laya, Rune, Clef, Perplexity Decider and D3

Typed decision models answer structured questions through /v1/systemone. These checkpoints require their trained decision heads and protocol metadata.

Supported model families and protocols (LibTorch):

  • Laya: ModernBERT-based decision checkpoints with rl_agent_config.json and trained decision heads. Accepts text or structured state; native conversations are unsupported.
  • OneJev, including Elephant and Mammoth: Qwen3.5 dense/MoE decision checkpoints using --decision-protocol onejev. Supports block-attention question batches and image inputs when the checkpoint includes its vision tower.
  • Perplexity Decider / Pplx: v1.1-27b and flash-v1.2. Flash v1.2 supports shared text/images, joint question readouts and calibrated answers. The short model aliases select the protocol automatically.
  • Rune: Gemma4-based decision checkpoints using --decision-protocol rune.
  • Clef: dense Qwen3.5 checkpoints with joint_head_config.json, using --decision-protocol clef.
  • D3: vllm-sr/d3 and vllm-sr/d3-mini, dense Qwen3.5 decision models with code-readout heads. The d3 and d3-mini aliases select --decision-protocol d3 automatically.

For example:

infinity_v3 --model-id pplx-decider-flash-v1.2 --backend libtorch --dtype bfloat16

Use the request schema for the checkpoint's protocol: OneJev, Laya and Perplexity joint v1.2 do not share an interchangeable payload format. See the running server's /docs, decision inputs and OneJev configuration.

Documentation