lupine

LUPINE: GPU-over-IP

LUPINE is a GPU over IP bridge allowing GPUs on remote machines to be attached to CPU-only machines.

Quick Start

Use the published GHCR images. The examples below pin CUDA 13.3.1 on Ubuntu 24.04; other published tags use the same cuda-<cuda-version>-ubuntu<ubuntu-version> format.

Run the server on the GPU machine:

docker run --rm --gpus all -p 14833:14833 \
  ghcr.io/lupinemachines/lupine-server:cuda-13.3.1-ubuntu24.04

Run the client pointing at that server:

docker run --rm -it \
  -e LUPINE_SERVER=<server>:14833 \
  ghcr.io/lupinemachines/lupine-client:cuda-13.3.1-ubuntu24.04 \
  nvidia-smi

Example output from a real run against a remote RTX 4090:

…

Inside the client container, LD_LIBRARY_PATH=/opt/lupine/lib is already set, so CUDA driver users pick up the LUPINE libcuda.so.1 shim and NVML users such as nvidia-smi pick up the LUPINE libnvidia-ml.so.1 shim automatically.

Prometheus Metrics

Linux servers built with CUDA and NVML expose Prometheus metrics on the RPC port without monitoring-specific configuration:

curl http://<server>:14833/metrics

The endpoint reports host GPU memory capacity, memory use and utilization, plus memory and utilization for each connected client process. It also exports the mapping between client identity, the Lupine connection child, and the host PID reported by NVML. Values are collected when /metrics is requested, so the server does no background NVML polling.

Client compatibility

Each production server executable embeds the matching Linux, macOS, and Windows client objects for amd64 and arm64; there is no client-bundle directory to deploy beside it. Python clients fetch the current object from /.well-known/lupine/client/v1/<os>/<arch>, verify its strong ETag, content digest, manifest, and file hashes, and cache it locally. The selected ETag is also asserted when the RPC connection opens, closing the race between discovery and a server upgrade. LUPINE_LIBDIR remains an explicit local override for development.

Linux client objects target the manylinux2014 ABI (glibc 2.17) and statically include their private C++, HTTP/2, and TLS dependencies. They therefore work on newer glibc distributions, including Ubuntu 22.04, without requiring host copies of libstdc++, nghttp2, or OpenSSL.

Graceful Server Checkpoints

On Linux, SIGTERM stops the server from accepting connections, asks every connection child to finish its in-flight CUDA calls, and waits for those children to exit. This graceful drain happens in the open-source server with no extra runtime dependency.

Each connection child looks for liblupinecr.so.0, then liblupinecr.so, and uses the versioned provider ABI in checkpoint_provider.h. A missing or incompatible provider is a no-op; the server still drains and exits normally. The provider is loaded before the child's first CUDA call so it can observe RM/UVM activity needed to discover allocations.

Set LUPINE_SESSION in the client to attach a stable connection identifier. The optional provider receives that identifier to restore the connection before its first CUDA RPC and checkpoint it after shutdown has drained. For an unkeyed connection, restore is skipped and checkpoint receives a null identifier. Providers own storage configuration, file layout, and any fallback policy for unkeyed connections; Lupine does not select a checkpoint directory.

LUPINE_CHECKPOINT_LIBRARY can override the provider library path for a private deployment.

Connection Stability

Each client/server connection is a single long-lived TCP stream. Long-running workloads sit idle for long stretches (between training steps, during host-side data loading, inside long kernels), and stateful middleboxes — cloud load balancers, NAT gateways, conntrack tables, firewalls — silently reap idle flows far sooner than the kernel's default 2-hour keepalive. The next RPC then fails fatally. Lupine keeps these connections alive and resilient without retrying RPCs (which would break CUDA semantics):

RPC request and response bodies require content-encoding: lz4. Compression is applied transparently as one LZ4 frame per HTTP/2 body; peers do not negotiate or fall back to another encoding.

• TCP keepalive is enabled on every connection (client and server) with a 60s idle interval, 15s between probes, and 3 unanswered probes before giving up. Probes are sent only while idle, so active transfers pay no latency cost, and a dead peer is detected in ~105s instead of hanging on the TCP retransmit timer.
• Connect retry rides out a server that is not reachable yet (e.g. still provisioning): a connection is attempted a few times with exponential backoff, and each attempt is bounded by a deadline so a packet-filtered port is detected quickly rather than blocking for the full SYN-retransmit window.

Socket buffer sizes are left to the OS, which auto-tunes on modern kernels.

Trace Logging

Set LUPINE_TRACE on the client, server, or both to enable trace logging. LUPINE_TRACE=0 or an unset value disables tracing. LUPINE_TRACE=1 writes trace output to stdout, LUPINE_TRACE=2 writes it to stderr, and any other non-empty value is treated as a file path opened in append mode.

# trace to stdout
LUPINE_TRACE=1 ./your_cuda_program

# trace to stderr
LUPINE_TRACE=2 ./server

# trace to a file
LUPINE_TRACE=/tmp/lupine.trace ./your_cuda_program

The same LUPINE_TRACE variable controls both client and server tracing; LUPINE_SERVER_TRACE is no longer used.

Device printf Forwarding

LUPINE inspects uploaded PTX and cubin symbol data for vprintf, the CUDA device printf implementation. Until an image that may use device stdout is loaded, synchronization avoids stdout redirection and its process-global lock, allowing independent RPC lanes to synchronize concurrently. Fully opaque compressed fatbins are treated conservatively as potentially using device stdout.

After a device-output-capable image is loaded, context, stream, and event synchronization captures server fd 1 and forwards the bounded CUDA printf buffer to the client's stdout. Capture remains process-global so output from concurrent synchronization lanes is not misattributed.

Multi-GPU Across Multiple Servers

The client accepts a comma-separated LUPINE_SERVER list. Devices are exposed as one local ordinal list in server order: all GPUs from the first server, then all GPUs from the next server, and so on.

Run a server on each GPU machine:

# on gpu-host-a
docker run --rm --gpus all -p 14833:14833 \
  ghcr.io/lupinemachines/lupine-server:cuda-13.3.1-ubuntu24.04

# on gpu-host-b
docker run --rm --gpus all -p 14833:14833 \
  ghcr.io/lupinemachines/lupine-server:cuda-13.3.1-ubuntu24.04

Point the client at both servers:

docker run --rm --network host \
  -e LUPINE_SERVER=gpu-host-a:14833,gpu-host-b:14833 \
  ghcr.io/lupinemachines/lupine-client:cuda-13.3.1-ubuntu24.04 \
  nvidia-smi -L

Expected output lists both remote GPUs:

GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-...)
GPU 1: NVIDIA GeForce RTX 4090 (UUID: GPU-...)

CUDA driver applications use the same LUPINE_SERVER value:

docker run --rm --network host \
  -e LUPINE_SERVER=gpu-host-a:14833,gpu-host-b:14833 \
  ghcr.io/lupinemachines/lupine-client:cuda-13.3.1-ubuntu24.04 \
  ./your_cuda_program

Cross-server device-to-device and peer (cuMemcpyDtoD / cuMemcpyPeer) copies are supported: when the source and destination live on different servers, the client transparently stages the data through itself (device->host on one server, then host->device on the other). Direct server-to-server transfers that avoid that client hop, cross-server peer-access enablement, and cuMemcpy3DPeer are not implemented yet. Same-server operations route by handle ownership.

Prefix an endpoint with https:// when the Lupine server is behind a TLS-terminating proxy. Both CUDA applications and NVML tools such as nvidia-smi use the scheme and verify the proxy certificate against the system trust store. HTTPS defaults to port 443; plain and http:// endpoints default to port 14833.

For a specific CUDA version:

docker pull ghcr.io/lupinemachines/lupine-client:cuda-12.4.1-ubuntu22.04
docker pull ghcr.io/lupinemachines/lupine-server:cuda-12.4.1-ubuntu22.04

Client images contain the CUDA driver, CUDA runtime, cuBLAS, cuBLASLt, cuFFT, cuDNN, cuRAND, cuSPARSE, cuSPARSELt, cuSOLVER, cuSOLVERMg, NVRTC, NCCL, nvJitLink, nvJPEG, NPP, cuFile, CUPTI, nvSHMEM, NVML, and HIP shims, their runtime dependencies, and nvidia-smi. They are based on Ubuntu and contain neither the CUDA nor ROCm SDK. The -slim tags remain available as compatibility aliases with the same SDK-free contents, for example ghcr.io/lupinemachines/lupine-client:cuda-13.3.1-ubuntu24.04-slim.

The server image is also based on Ubuntu. It installs the CUDA compatibility runtime and, on amd64, the ROCm HIP runtime so one image can serve either NVIDIA or AMD GPUs; the older separate HIP server Dockerfile is no longer needed.

Slow Start for the Skeptics

This path derives a small PyTorch client image from the published LUPINE client image and runs the microgpt_train test against a remote GPU. It is intentionally explicit so it is easy to see which side is the CPU-only client and which side owns the GPU.

Create a PyTorch client Dockerfile in the repo root:

…

Build it:

docker build -f Dockerfile.pytorch-lupine -t lupine-pytorch:cuda-13.3 .

Run the server on the GPU machine:

docker run --rm --gpus all -p 14833:14833 \
  ghcr.io/lupinemachines/lupine-server:cuda-13.3.1-ubuntu24.04

Run the PyTorch client from the CPU-only machine:

docker run --rm \
  -e LUPINE_SERVER=<server>:14833 \
  lupine-pytorch:cuda-13.3

Expected success looks like:

microgpt first_loss=... last_loss=...
microgpt_train: PASS

Local development

Building the binaries requires running codegen first. The repository provides a containerized runner so local development and CI use the same CUDA and HIP headers, Python, parser, and formatter versions. Docker is the only host dependency.

Run codegen

./codegen/run.sh

Ensure there are no errors in the output of the codegen.

Run cmake

cmake -S . -B build
cmake --build build

CMake builds the CUDA driver shim at build/libcuda.so.1, the CUDA runtime shim at build/libcudart.so.<major>, the cuBLAS, cuBLASLt, cuFFT, cuRAND, cuSPARSE, cuSOLVER, cuSOLVERMg, NVRTC, nvJitLink and nvJPEG shims at build/libcublas.so.<major>, build/libcublasLt.so.<major>, build/libcufft.so.<major>, build/libcurand.so.<major>, build/libcusparse.so.<major>, build/libcusolver.so.<major>, build/libcusolverMg.so.<major>, build/libnvrtc.so.<major>, build/libnvJitLink.so.<major>, build/libnvjpeg.so.<major>, the NPP shims at build/libnppc.so.<major> and its image and signal libraries (build/libnppial.so.<major> through build/libnpps.so.<major>) (when the toolkit's library headers are present; nvJitLink needs CUDA 12.4 or newer), the cuDNN shim at build/libcudnn.so.9 (when cuDNN 9 headers are found beside the toolkit's or through -DLUPINE_CUDNN_INCLUDE_DIR=<dir>), the NCCL shim at build/libnccl.so.2 on Linux (when NCCL 2.14.3 or newer headers are found beside the toolkit's or through -DLUPINE_NCCL_INCLUDE_DIR=<dir>), the cuFile shim at build/libcufile.so.0 on Linux (when cufile.h is found beside the toolkit's or through -DLUPINE_CUFILE_INCLUDE_DIR=<dir>), the CUPTI shim at build/libcupti.so.<major> on Linux (build/libcupti.so.11.8 on CUDA 11, whose CUPTI carries the minor in its SONAME; when cupti_result.h is found beside the toolkit's or through -DLUPINE_CUPTI_INCLUDE_DIR=<dir>), the NVML toolkit's or through -DLUPINE_CUFILE_INCLUDE_DIR=<dir>), the nvSH