How to Run vLLM on Two NVIDIA DGX Sparks

In last week’s post we connected two NVIDIA DGX Sparks. With this connection in place, we can now continue and run vLLM on our two-machine cluster to access the 256GB RAM of both machines.

The official guide

As with connecting the two Sparks there is an official guide on how to run vLLM on two Sparks that could benefit from more details. While it was much better than the one on the connection, it still cost me a lot of time until I added the HF token to the Docker containers on both Sparks. In this post I cover all the other parts that may be helpful for the first steps.

Get the newest Docker image

We can find the newest version of the vLLM container image on ngc.nvidia.com. Make sure to check the release notes, then sometimes a few specific models may not run on the newest container (like GPT-OSS 120B should run on 26.05 instead of 26.06).

We can pull the image on both Sparks with this command (replace 26.06 with the newest version):

docker pull nvcr.io/nvidia/vllm:26.06-py3
export VLLM_IMAGE=nvcr.io/nvidia/vllm:26.06-py3

The benefit of the Docker image is that we get everything pre-installed. It takes a bit of time to download the container, but that is nothing compared to the compatibility and build errors I run into when I tried to build vLLM on my own.

Find models on Hugging Face

While Docker loads our image, we can go to Hugging Face and search for the model we want to run. While some models can be accessed by everyone, many models require a pre-authorisation. This can take from hours to days; therefore, first check which models you can access directly and start with them while you wait on the approval for the others.

Persist the HF token

To access Hugging Face, we best create a token. This allows us a fine-grained access control, and we do not need to retype our password over and over again.

We will use the Hugging Face token a lot and best put it in a persisted environment variable that stays around when we reboot our Sparks. The simplest way to persist our token is to modify our ~.bashrc file on both Sparks and put this export command with our token at the end:

# HuggingFace token (required)
# Get a token from https://huggingface.co/settings/tokens
export HF_TOKEN="your_huggingface_token"

We can run this command to make our token available in our current session:

source ~.bashrc

We can check if the token is accessible with this command:

echo $HF_TOKEN

If this does not show our HF token, we need to repeat the steps above until that works. Otherwise, we will run into problems in the following steps.

Download the cluster deployment script

We can use the official script to run the cluster with this set of commands:

# Download on both nodes — pinned to a known-good commit so upstream changes
# can't silently break this playbook against the 26.05-py3 image.
wget https://raw.githubusercontent.com/vllm-project/vllm/51c1ee9b7c8acbba4899a8ebffd390685d171946/examples/ray_serving/run_cluster.sh

# Patch the script to pip-install ray inside the container before ray starts.
# The 26.05-py3 NGC image ships without ray (upstream made it an optional CUDA dep);
# the install takes ~10s on first container launch.
sed -i 's|^RAY_START_CMD="ray start|RAY_START_CMD="pip install -q --root-user-action=ignore '\''ray[default]>=2.9'\'' \&\& ray start|' run_cluster.sh

chmod +x run_cluster.sh

At the end we have an executable run_cluster.sh file on both Sparks. If we use a larger model, our Sparks may freeze because they run out of resources. Should this happen, we can limit the number of CPU cores by adding this line to the start script:

...
docker run \
    --entrypoint /bin/bash \
    --network host \
    --name "${CONTAINER_NAME}" \
    --shm-size 10.24g \
    --gpus all \
    --cpus="16" \
    -v "${PATH_TO_HF_HOME}:/root/.cache/huggingface" \
    "${ADDITIONAL_ARGS[@]}" \
    "${DOCKER_IMAGE}" -c "${RAY_START_CMD}"

Start Ray head node

On the node we use as an entry into our cluster, the head node, we can create the script start_head.sh with these commands:

export VLLM_IMAGE=nvcr.io/nvidia/vllm:26.06-py3

# On Node 1, start head node

# Get the IP address of the high-speed interface
# Use the interface that shows "(Up)" from ibdev2netdev (enp1s0f0np0 or enp1s0f1np1)
export MN_IF_NAME=enp1s0f1np1
export VLLM_HOST_IP=$(ip -4 addr show $MN_IF_NAME | grep -oP '(?<=inet\s)\d+(\.\d+){3}')

echo "Using interface $MN_IF_NAME with IP $VLLM_HOST_IP"

bash run_cluster.sh $VLLM_IMAGE $VLLM_HOST_IP --head ~/.cache/huggingface \
  -e VLLM_HOST_IP=$VLLM_HOST_IP \
  -e UCX_NET_DEVICES=$MN_IF_NAME \
  -e NCCL_SOCKET_IFNAME=$MN_IF_NAME \
  -e OMPI_MCA_btl_tcp_if_include=$MN_IF_NAME \
  -e GLOO_SOCKET_IFNAME=$MN_IF_NAME \
  -e TP_SOCKET_IFNAME=$MN_IF_NAME \
  -e RAY_memory_monitor_refresh_ms=0 \
  -e MASTER_ADDR=$VLLM_HOST_IP \
  -e HF_TOKEN="$HF_TOKEN"

We can make it executable and start our head node with these two commands:

chmod +x start_head.sh
./start_head.sh

This script saves us a lot of typing compared to the approach in the official guide.

Start Ray worker node

On the second node, our worker node, we create the script start_worker.sh and set the IP address of our head node:

export VLLM_IMAGE=nvcr.io/nvidia/vllm:26.06-py3

# On other Nodes, join as workers

# Set the interface name (same as Node 1)
export MN_IF_NAME=enp1s0f1np1

# Get Node's own IP address
export VLLM_HOST_IP=$(ip -4 addr show $MN_IF_NAME | grep -oP '(?<=inet\s)\d+(\.\d+){3}')

# IMPORTANT: Set HEAD_NODE_IP to Node 1's IP address
# You must get this value from Node 1 (run: echo $VLLM_HOST_IP on Node 1)
export HEAD_NODE_IP=SET_IP_ADDRESS # <=============================

echo "Worker IP: $VLLM_HOST_IP, connecting to head node at: $HEAD_NODE_IP"

bash run_cluster.sh $VLLM_IMAGE $HEAD_NODE_IP --worker ~/.cache/huggingface \
  -e VLLM_HOST_IP=$VLLM_HOST_IP \
  -e UCX_NET_DEVICES=$MN_IF_NAME \
  -e NCCL_SOCKET_IFNAME=$MN_IF_NAME \
  -e OMPI_MCA_btl_tcp_if_include=$MN_IF_NAME \
  -e GLOO_SOCKET_IFNAME=$MN_IF_NAME \
  -e TP_SOCKET_IFNAME=$MN_IF_NAME \
  -e RAY_memory_monitor_refresh_ms=0 \
  -e MASTER_ADDR=$HEAD_NODE_IP \
  -e HF_TOKEN="$HF_TOKEN"

After we make it executable, we can run it to connect our node to the cluster:

chmod +x start_worker.sh
./start_worker.sh

Verify cluster status

To check if both nodes are active, we can make the script check_cluster.sh and put it on our head node:

# On Node 1 (head node)
# Find the vLLM container name (it will be node-<random_number>)
export VLLM_CONTAINER=$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')
echo "Found container: $VLLM_CONTAINER"

docker exec $VLLM_CONTAINER ray status

We run our sequence of making it executable and running it to see if our cluster runs:

chmod +x check_cluster.sh
./check_cluster.sh

The output should look like this and show two nodes and the combined number of RAM and CPU:

Found container: node-10178
======== Autoscaler status: 2026-07-13 20:37:25.012770 ========
Node status
---------------------------------------------------------------
Active:
 1 node_ea9cda04b870238006c1c204afbe20d289b03be0b1696c64dd15f464
 1 node_18c79f6384c5bc535f1fde0991c5df097d341604bc66a8be74988a9e
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Total Usage:
 0.0/34.0 CPU
 0.0/2.0 GPU
 0B/220.64GiB memory
 0B/19.46GiB object_store_memory

From request_resources:
 (none)
Pending Demands:
 (no resource demands)

Download the model

To test if we can load a model that is larger than a single Spark, I use Kimi-Dev-72B from Moonshot AI. It has a file size of about 145 GB and will need additional space for the model weights and the KV cache. A good calculator to see what model can fit our Sparks is the LLM Inference: VRAM & Performance Calculator.

When we have or model and its name on Hugging Face, we can download it with a script like this one that runs on both Sparks:

export VLLM_CONTAINER=$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')

docker exec -it $VLLM_CONTAINER /bin/bash -c '
  hf auth login
  hf download moonshotai/Kimi-Dev-72B'

Load the model

On our head node we can load the model and its model-specific parameters with a script like this one:

export VLLM_CONTAINER=$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')

docker exec -it $VLLM_CONTAINER /bin/bash -c '
vllm serve moonshotai/Kimi-Dev-72B \
    --host 0.0.0.0 \
    --tensor-parallel-size 2 \
    --kv-cache-dtype fp8 \
    --trust-remote-code \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90'

Each model has different parameters. Sometimes we find all the necessary settings directly on the model page in Hugging Face. If this is not the case, ask Gemini or Chat GPT; both models got me valid configurations without a problem.

Chat with the model

We can now run the interference against our model with a little curl script like this one - use the IP address of the head node that you use over Ethernet and not the cluster cable:

curl http://IP_OF_HEAD_ON_NETWORK:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshotai/Kimi-Dev-72B",
    "prompt": "What is the capital city of Norway?",
    "max_tokens": 320,
    "temperature": 0.7
  }'

This gives us an output that should answer our question and offer us some additional details about Oslo:

{
    "id": "cmpl-9522ebff2d98504a",
    "object": "text_completion",
    "created": 1783975717,
    "model": "moonshotai/Kimi-Dev-72B",
    "choices":
    [
        {
            "index": 0,
            "text": " The capital city of Norway is Oslo. It's a beautiful city located on the southeastern coast of the country. Oslo is not only the largest city in Norway but also serves as the political, economic, and cultural center. The city is known for its rich history, vibrant arts scene, and stunning natural surroundings. It's home to several museums, including the Viking Ship Museum and the Munch Museum, as well as the iconic Akershus Fortress. The city's harbor and parks make it a popular destination for both locals and tourists. Oslo also hosts important international events and is a key player in global politics and business. The city's architecture ranges from historic buildings to modern structures, creating a unique urban landscape. The Oslo Opera House and the Barcode Project are notable examples of contemporary architecture. Additionally, Oslo is surrounded by forests and waterways, offering residents and visitors ample opportunities for outdoor activities like hiking, skiing, and sailing. The city's commitment to sustainability and green spaces makes it a model for urban living. Whether you're interested in history, art, nature, or modern culture, Oslo has something to offer.",
            "logprobs": null,
            "finish_reason": "stop",
            "stop_reason": null,
            "token_ids": null,
            "prompt_logprobs": null,
            "prompt_token_ids": null,
            "routed_experts": null
        }
    ],
    "service_tier": null,
    "system_fingerprint": "vllm-0.22.1+7b9cb5b7.dev-tp2-cb207b3e",
    "usage":
    {
        "prompt_tokens": 8,
        "total_tokens": 232,
        "completion_tokens": 224,
        "prompt_tokens_details": null
    },
    "kv_transfer_params": null
}

When we check each node, we should see that the RAM is nearly fully used. In my configuration it allocated 237 of the 256 GB that the two Sparks offer. There is not much space left to add bigger models than the Kimi-Dev-72B, unless we add another Spark.