Everything you need to ship.

Installation

SwiftLLM provides a smart installer that auto-detects your system, installs dependencies, and builds the engine from source. As of v2.0 it ships portable abi3 wheels for Linux (x86_64, aarch64) and macOS (x86_64, arm64) — a single wheel covers Python 3.8 through 3.12.

Prerequisites

• Python 3.8+ (3.10+ recommended)
• Git and a C toolchain (cc/clang)
• NVIDIA GPU with CUDA 11.8+ — optional; pass --cpu for CPU-only builds
• Rust toolchain — installed automatically if missing

# Clone the repository
git clone https://github.com/infrabrew/swiftllm.git
cd swiftllm

# Run the installer (auto-detects CUDA vs CPU)
bash install.sh

Install Options

Platforms

SwiftLLM ships pre-built wheels for the matrix below. Other platforms fall back to a source build, which the installer handles automatically.

GPU requirements

• NVIDIA: Compute Capability 7.0+ (Volta and later). Tested on A100, H100, L4, RTX 4090, RTX 3090.
• VRAM: 16 GB minimum for 7B models in fp16; 24 GB+ recommended for serving with KV-cache headroom.
• Drivers: NVIDIA driver 525+ for CUDA 12, 470+ for CUDA 11.8.

Quick Start

Basic Inference

from swiftllm import LLM, SamplingParams

llm = LLM("meta-llama/Llama-3-8B")
params = SamplingParams(temperature=0.7, max_tokens=256)

outputs = llm.generate(["What is machine learning?"], params)
for output in outputs:
    print(output.text)

Self-Consistency Voting

from swiftllm import LLM
from swiftllm.config import SelfConsistencyConfig

llm = LLM("meta-llama/Llama-3-8B")
results = llm.generate_with_self_consistency(
    "What is 12 x 15?",
    config=SelfConsistencyConfig(num_samples=8),
)
print(results[0].answer)  # majority-voted answer

TurboQuant KV Cache Compression

from swiftllm import LLM, SamplingParams, EngineConfig, TurboQuantConfig

# Enable 3-5x KV cache compression
engine_cfg = EngineConfig(
    turbo_quant=TurboQuantConfig.quality_neutral(),  # 4-bit K, 3-bit V
)
llm = LLM("meta-llama/Llama-3-8B", engine_config=engine_cfg)
outputs = llm.generate(["Summarize this article..."],
    SamplingParams(max_tokens=512))

Start an API Server

# Start with API key authentication
export SWIFTLLM_API_KEY="your-secret-key"
swiftllm serve --model meta-llama/Llama-3-8B --port 8000

CLI — Advanced Generation

# Self-consistency with 8 chains
swiftllm generate -p "What is 12 x 15?" --self-consistency 8

# Best-of-N dense verification
swiftllm generate -p "Prove the Pythagorean theorem" --best-of-n 4

# Recursive Language Model (depth=3)
swiftllm generate -p "Solve step by step: 2^10 mod 7" --rlm 3

# Multi-round self-refinement
swiftllm generate -p "Write a haiku about Rust" --refinement-rounds 3

Air-Gapped Installation

SwiftLLM supports deployment on isolated networks with no internet access. Create a portable bundle on a connected machine, then install offline. The bundle includes source, Python wheels, rustup-init, and optionally pre-downloaded models with SHA256 checksum verification.

# On a connected machine — basic bundle
./airgap-bundle.sh

# Include a model in the bundle
./airgap-bundle.sh --model "Qwen/Qwen2.5-0.5B-Instruct-GGUF:qwen2.5-0.5b-instruct-q4_k_m.gguf"

# Cross-architecture bundle (x86_64 host → ARM64 target)
./airgap-bundle.sh --arch aarch64 -o swiftllm-bundle-arm64.tar.gz

# CPU-only wheels
./airgap-bundle.sh --cpu -o /mnt/usb/swiftllm-bundle.tar.gz

# macOS Apple Silicon
./airgap-bundle.sh --arch arm64 --platform macosx_11_0_arm64

# On the air-gapped target
tar xzf swiftllm-airgap-bundle.tar.gz
cd swiftllm-airgap-bundle/swiftllm
./install.sh --airgap

# Runtime offline mode (disable all HF downloads)
export SWIFTLLM_OFFLINE=1
swiftllm generate -m /path/to/local/model.gguf -p "Hello"

Update

The update.sh lifecycle script pulls the latest source, rebuilds the wheel, and reinstalls the package — preserving your virtual environment and downloaded models.

# Pull latest and rebuild
./update.sh

# Switch to a specific branch
./update.sh --branch main-hybrid-rd

# Switch to a specific tag
./update.sh --tag v2.0.5

# Clean build artifacts before rebuilding
./update.sh --clean

# Rebuild from current source (skip git pull)
./update.sh --no-pull

# Force CPU-only rebuild
./update.sh --cpu

Update Flags

Uninstall

The uninstall.sh lifecycle script cleanly removes SwiftLLM, with options to preserve models and virtual environments.

# Interactive uninstall (prompts before each step)
./uninstall.sh

# Uninstall but keep downloaded models
./uninstall.sh --keep-models

# Remove everything non-interactively
./uninstall.sh --purge --yes

Uninstall Flags

Inference

SwiftLLM's inference engine is built on PagedAttention with continuous batching, speculative decoding, and KV-cache reuse — designed to serve dozens of concurrent requests on a single GPU without head-of-line blocking. v2.0 adds six research-derived test-time inference enhancements and TurboQuant KV cache compression.

Sync, async, and streaming

• LLM — synchronous engine for single-shot generation. Best for scripts and notebooks.
• AsyncLLM — async engine with async for token streaming. Best for serving and concurrent workloads.
• OpenAI-compatible HTTP — swiftllm serve exposes /v1/completions and /v1/chat/completions with SSE streaming.

Sampling parameters

Streaming example

from swiftllm import AsyncLLM, SamplingParams

llm = AsyncLLM("meta-llama/Llama-3-8B")
params = SamplingParams(temperature=0.7, max_tokens=256)

async for token in llm.stream("Explain PagedAttention in one paragraph.", params):
    print(token, end="", flush=True)

Test-Time Inference Enhancements (Phase 3)

Six research-derived methods for improving output quality at inference time. Each is available as a Python API method and a CLI flag.

TurboQuant KV Cache Compression

ICLR 2026 (Zandieh et al., arXiv 2504.19874) — online vector quantization for KV cache. Random rotation via fast Walsh-Hadamard transform with deterministic sign flips, then scalar quantization using precomputed Beta-distribution Max-Lloyd codebooks at 1-4 bits per channel.

Two quantization variants: TurboQuantMse (minimizes MSE, best for general use) and TurboQuantProd (unbiased inner-product estimation via JL sign sketch, best when attention score preservation matters more than individual vector reconstruction).

Recursive Language Model (RLM)

Bounded recursive self-calling with a REPL sandbox, variable binding table, and complexity-classifier MLP. Four operating modes: DISABLED, SHALLOW (depth=1), REASONING (depth=3, default), AGENTIC (depth=5).

from swiftllm import LLM, RlmConfig, RlmMode, SamplingParams

llm = LLM(model="path/to/model")
results = llm.generate_with_rlm(
    "Prove by induction that 1+2+…+n = n(n+1)/2.",
    config=RlmConfig(mode=RlmMode.REASONING, max_depth=3,
                     enable_repl=True, early_exit_threshold=0.92),
    base_params=SamplingParams(temperature=0.7, max_tokens=768),
)
result = results[0]
print(f"Depth used: {result.recursion_depth_used}")
print(f"REPL steps: {len(result.repl_trace)}")

Dense Verification Layer

Cross-attention scoring of draft tokens against REPL execution trace. Per-token and per-step confidence. Strategies: SCORE_ONLY, GATE, GATE_AND_REGEN.

from swiftllm import LLM, DenseVerificationConfig, VerificationStrategy

llm = LLM(model="path/to/model")
results = llm.generate_with_dense_verification(
    "Explain Gödel's incompleteness theorems.",
    config=DenseVerificationConfig(
        strategy=VerificationStrategy.GATE_AND_REGEN,
        min_confidence=0.80, max_regen_attempts=3),
)
result = results[0]
print(f"Confidence: {result.global_score:.1%}")
print(f"Accepted on attempt: {result.accepted_on_attempt}")

API Server

SwiftLLM includes an OpenAI-compatible HTTP server built with Axum, supporting streaming, API key authentication, CORS, and security hardening. As of v2.0 the server binds to 127.0.0.1 by default (localhost only) to prevent accidental exposure on open networks.

Endpoints

• POST /v1/completions — Text completion
• POST /v1/chat/completions — Chat completion
• GET /v1/models — List loaded models
• GET /health — Health check (unauthenticated)
• GET /metrics — Prometheus metrics (authenticated — requires API key when SWIFTLLM_API_KEY is set)

Security Hardening

v2.0 ships with production-grade security defaults out of the box:

• Default bind 127.0.0.1 — the server listens on localhost only. Set SWIFTLLM_HOST=0.0.0.0 to expose externally.
• Constant-time API key auth — set SWIFTLLM_API_KEY to require Authorization: Bearer <key> on all endpoints except /health. Uses constant-time comparison to prevent timing attacks.
• CORS from config — allowed origins are configured via ServerConfig.cors_origins or SWIFTLLM_CORS_ORIGINS. Defaults to same-origin only.
• Authenticated /metrics — Prometheus metrics are gated behind the same API key, preventing info leakage.
• Rate limiting — configurable via tower-http middleware. Defaults to no limit; set SWIFTLLM_RATE_LIMIT for requests/second cap.

Launch examples

# Basic — localhost only, no auth
swiftllm serve --model meta-llama/Llama-3-8B

# Production — external, with API key and CORS
export SWIFTLLM_API_KEY="sk-your-secret-key"
export SWIFTLLM_CORS_ORIGINS="https://app.example.com"
swiftllm serve --model meta-llama/Llama-3-8B --host 0.0.0.0 --port 8000

# Verify auth is required
curl http://localhost:8000/v1/models \
  -H "Authorization: Bearer sk-your-secret-key"

Training & Fine-Tuning

SwiftLLM supports full training, LoRA, QLoRA fine-tuning, and GRPO reinforcement learning with three optimizer backends: Muon (Newton-Schulz orthogonalization), AdamW (decoupled weight decay), and SGD (optional Nesterov momentum). The CLI exposes subcommands: swiftllm train, swiftllm finetune, and swiftllm grpo.

Supported Methods

LoRA Fine-Tuning

from swiftllm import fine_tune, LoRAConfig, TrainingConfig

lora = LoRAConfig(r=16, alpha=32, dropout=0.05)
config = TrainingConfig(learning_rate=2e-4, num_epochs=3,
                        per_device_batch_size=4)

fine_tune(model="meta-llama/Llama-3-8B",
          train_data="./my_data.jsonl",
          lora_config=lora, training_config=config,
          output_dir="./output")

# CLI — LoRA fine-tuning
swiftllm finetune -m meta-llama/Llama-3-8B \
  --train-data ./data/train.jsonl \
  --lora-r 16 --lora-alpha 32 --learning-rate 2e-4

# CLI — QLoRA (4-bit base + LoRA)
swiftllm train -m meta-llama/Llama-3-8B \
  --train-data ./data/train.jsonl --method qlora \
  --lora-r 16 --mixed-precision bf16

Auto-Ingest from Files or HuggingFace

Pass a directory, file list, or HuggingFace dataset directly to fine_tune() — SwiftLLM automatically converts it to JSONL before training. No manual dataset preparation needed.

# Fine-tune from a directory (auto-ingested to JSONL)
fine_tune(model="meta-llama/Llama-3-8B",
          train_data="./my_corpus/",
          output_dir="./output", lora_r=16)

# Fine-tune from a HuggingFace dataset
fine_tune(model="meta-llama/Llama-3-8B",
          hf_dataset="tatsu-lab/alpaca",
          dataset_format="sft_completion",
          output_dir="./output", lora_r=32)

# Combine local files + HuggingFace in one command
fine_tune(model="meta-llama/Llama-3-8B",
          train_data="./my_docs/",
          hf_dataset="tatsu-lab/alpaca",
          hf_max_samples=10_000,
          dataset_format="sft_completion",
          output_dir="./output", lora_r=16)

GRPO Reinforcement Learning

Group Relative Policy Optimization — RL fine-tuning without a critic model. Includes CGAR depth curriculum (1.71x speedup), Process Reward Models (5 aggregation strategies for step-level feedback), and LongR dense token-level rewards (+9% LongBench v2).

from swiftllm import GrpoTrainer, TrainingConfig
from swiftllm.config import GrpoConfig, CgarConfig, PrmConfig

config = TrainingConfig(
    model="meta-llama/Llama-3-8B",
    train_data="./data/math_prompts.jsonl",
    output_dir="./grpo_output",
    num_layers=32,
    grpo=GrpoConfig(group_size=8, kl_coeff=0.04),
    cgar=CgarConfig(),
    prm=PrmConfig(aggregation="last_step"),
)
trainer = GrpoTrainer(config)
trainer.train()

# GRPO with full research stack: CGAR + PRM + LongR
swiftllm grpo -m meta-llama/Llama-3-8B \
  --train-data ./data/math_prompts.jsonl \
  --group-size 8 --enable-prm --long-reward-weight 0.1 \
  --num-layers 32

Dataset Ingestion

Convert directories of text, code, PDF, DOCX, CSV, HTML, and JSON files into JSONL training data in one command. Supports HuggingFace Hub datasets via HuggingFaceSource. Four output schemas: pretraining, sft_messages, sft_completion, code. SHA-256 deduplication across all sources.

# Local files — code fine-tuning
swiftllm dataset --input ./src/ ./docs/ --output train.jsonl --format code

# HuggingFace dataset
swiftllm dataset --hf-dataset tatsu-lab/alpaca --output alpaca.jsonl --format sft_completion

# Mixed: local files + HuggingFace combined
swiftllm dataset --input ./my_docs/ --hf-dataset tatsu-lab/alpaca \
  --hf-max-samples 10000 --format sft_completion --output combined.jsonl

# Large corpus with streaming (avoids full download)
swiftllm dataset --hf-dataset HuggingFaceFW/fineweb --hf-subset sample-10BT \
  --hf-streaming --hf-max-samples 100000 --format pretraining --output fineweb.jsonl

Training Data Formats

Three accepted input formats for supervised fine-tuning:

Advanced Inference

SwiftLLM v2.0 includes six test-time inference enhancements accessible via Python API and CLI flags. These improve output quality by spending more compute at inference time — no training required.

• Self-Consistency: llm.generate_with_self_consistency() — majority voting over N independent chains (Wang et al., 2022)
• Self-Refinement: llm.generate_with_refinement() — iterative critique→revision cycles (Madaan et al., 2023)
• Best-of-N: llm.generate_best_of_n() — dense scoring and reranking of N candidates
• Disaggregated Serving: DisaggregatedServingConfig — separate prefill/decode worker pools (Splitwise/DistServe)
• RLM: llm.generate_with_rlm() — recursive self-calling with REPL sandbox and variable binding
• Dense Verification: llm.generate_with_dense_verification() — cross-attention token/step confidence scoring
• TurboQuant: Set EngineConfig(turbo_quant=TurboQuantConfig.quality_neutral()) for 3-5x KV cache compression