Odysseus can download and serve local AI models through its Cookbook. On the CPU that is slow, and it only gets fast with a GPU. This guide enables NVIDIA support in the Odysseus Docker setup and gets your own models running on a GPU server, without your data or prompts ever leaving the machine.
15 minIntermediateTested on Ubuntu 24.04, NVIDIA, DockerUpdated 2026-06-18
In short
Install the NVIDIA driver and Container Toolkit, enable the GPU overlay in .env, restart the stack, then load models in the Cookbook.
Local models are the heavy part. How large a model you can run depends mostly on your GPU's VRAM. As a rough guide at 4-bit quantization:
Model size
VRAM
Suitable GPU
7B to 14B
approx. 6 to 10 GB
RTX 4000 Ada (20 GB)
32B
approx. 22 GB
RTX 6000 Ada (48 GB)
70B
approx. 42 GB
RTX 6000 Ada (48 GB)
A ComputeBox GPU server delivers exactly that: dedicated NVIDIA cards with 20 or 48 GB of VRAM, full root access, and German data centers, ready in minutes and without hourly cost traps.
Local models with full GPU power: RTX 4000 Ada (20 GB) from €99/month, dedicated.
For Docker to pass the GPU through to the container, you need the NVIDIA Container Toolkit. Odysseus ships a helper script for it. Change into the odysseus folder and run:
The script checks passthrough and installs the toolkit on Ubuntu and Debian.
Open Odysseus and go to the Cookbook. There you get hardware-aware model recommendations, download models, and serve them through Odysseus. Downloads land in ./data/huggingface, the serve engines in ./data/local, both survive a container recreation.
GPU passthrough is not the same as a CUDA build
A successful nvidia-smi inside the container only confirms GPU access. If the Cookbook reports Unable to find cudart or runs on the CPU, the serve engine's CUDA build is missing. Reinstall it via Cookbook → Dependencies to get a CUDA-enabled version.
Alternative: connect Ollama
If an Ollama is already running on the host (started with OLLAMA_HOST=0.0.0.0:11434 ollama serve), just add the endpoint http://host.docker.internal:11434/v1 in the Odysseus settings.
