use-case: ai-inference

open-weight inference outside US export-control gating

Open-weight LLMs (Llama, Mistral, Qwen, Gemma) cleared the bar for production inference on most non-frontier tasks somewhere between mid-2024 and late-2025. The constraint moved from API access to GPU availability, and GPU availability is now a geopolitical question. US export-control classifications around H100/H200-class accelerators have made allocation political; mid-tier consumer / pro GPUs (RTX 4090, RTX A6000) in non-US datacenters sidestep the gating entirely.

Memory budget determines what fits. Rule of thumb: VRAM (GB) ≈ params (B) × 2 for FP16, × 0.5 for 4-bit quantization. RTX 4090 (24 GB) runs 13B FP16 or 30B 4-bit comfortably; A6000 (48 GB) runs 70B 4-bit; H100 (80 GB) runs 70B FP16 with serious context. vLLM batches at the request level for production throughput; Ollama is the right answer for single-user / development workflows; both ship preinstalled with CUDA 12 + PyTorch on every gpu-* tier.

The data-sovereignty argument is real and underweighted. Frontier API providers reserve the right to retain prompts and completions for safety, abuse, or training purposes, depending on the SKU. A self-hosted endpoint inside an Iceland or Netherlands GPU plan keeps the prompt corpus, the model weights, and the inference logs all under one operator. Customer data does not flow into a third-party analytics fabric; the OpenAI-compatible HTTP server vLLM ships closes a customer-facing inference endpoint cleanly.

see also

• vLLM — Efficient Memory Management for LLM Serving (Kwon et al., SOSP 2023)
• Llama 2 / Llama 3 Model Cards (ai.meta.com)
• QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)
• US BIS — Advanced Computing Export Controls (federalregister.gov)