How to Connect Two NVIDIA DGX Sparks
The NVIDIA DGX Spark comes in many variations, like the Dell Pro Max with GB10 or the HP ZGX Nano AI Station. They vary in the case, but inside we have basically the same GB10 board with the same AI capabilities. The integrated 128GB RAM is great, but for mid-sized models more RAM would be helpful. Luckily for us, we can connect multiple DGX Sparks together and increase the model size we can load. Let us see how we can do that.
Why not just follow the official guide?
The official guide covers all the important points we need to connect two DGX Sparks. However, as so often the tiny little points that are not in the documentation cost a lot of time. That is why I wrote this extended guide so that you do not waste as much time as I did.
1. Update the Sparks
Before we do any work, make sure that you run the newest version of the packages and the system on both machines. We can use these commands (on both machines) to get all the updates:
This fetches the newest package lists, installs the packages, updates the system and restarts it at the end.
2. Ensure same username on both systems
Make sure that your user has the same name on both Sparks. We can check that with this command:
If this is not the case, create a new user on both systems as the official guide suggests:
# Create nvidia user and add to sudo group
sudo useradd -m nvidia
sudo usermod -aG sudo nvidia
# Set password for nvidia user
sudo passwd nvidia
# Switch to nvidia user
su – nvidia
3. Physical hardware connection
Connect the two Sparks with a QSFP cable. It does not matter what port you use; I used the one closer to the centre because I had more space there. After the cable is plugged in, we can run this command to check if the interfaces show up:
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Down)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Up)
rocep1s0f0 port 1 ==> enp1s0f0np0 (Down)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Up)
Run it on both Sparks! The names should be the same, the order may vary.
4. Network interface configuration
This step cost me the most time. There is an automatic IP assignment that got me an IP address that collided with my network. To prevent that, go with the manual IP assignment with the netplan configuration file.
On the first Spark we can run this command:
# Create the netplan configuration file
sudo tee /etc/netplan/40-cx7.yaml > /dev/null <<EOF
network:
version: 2
ethernets:
enp1s0f0np0:
addresses:
- 192.168.50.11/24
dhcp4: no
enp1s0f1np1:
addresses:
- 192.168.60.13/24
dhcp4: no
enP2p1s0f0np0:
addresses:
- 192.168.50.15/24
dhcp4: no
enP2p1s0f1np1:
addresses:
- 192.168.60.17/24
dhcp4: no
EOF
# Set appropriate permissions
sudo chmod 600 /etc/netplan/40-cx7.yaml
# Apply the configuration
sudo netplan apply
On the second Spark we run this command with even IP addresses:
# Create the netplan configuration file
sudo tee /etc/netplan/40-cx7.yaml > /dev/null <<EOF
network:
version: 2
ethernets:
enp1s0f0np0:
addresses:
- 192.168.50.10/24
dhcp4: no
enp1s0f1np1:
addresses:
- 192.168.60.12/24
dhcp4: no
enP2p1s0f0np0:
addresses:
- 192.168.50.14/24
dhcp4: no
enP2p1s0f1np1:
addresses:
- 192.168.60.16/24
dhcp4: no
EOF
# Set appropriate permissions
sudo chmod 600 /etc/netplan/40-cx7.yaml
# Apply the configuration
sudo netplan apply
5. Reboot and verify IP address
We can now restart our Sparks with this command:
When they are back up, we can check if the IP address is still assigned to our network interface:
6: enP2p1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 84:5c:31:ff:f3:53 brd ff:ff:ff:ff:ff:ff
inet 192.168.60.16/24 brd 192.168.60.255 scope global noprefixroute enP2p1s0f1np1
valid_lft forever preferred_lft forever
inet6 fe80::865c:31ff:feff:f353/64 scope link
valid_lft forever preferred_lft forever
Check that on both Sparks but remember that the IP address must differ between the Sparks and should match your configuration.
6. Set up passwordless SSH authentication
We can download the discover-sparks script and save it on both Sparks. We only need to run it on one device, but if something goes wrong, it is better to have a copy on the other box.
We then can run the script with this command:
Read the output carefully! Check that IP addresses from both devices got detected and that it does not throw an error:
Found: 192.168.60.13 (gn100-936b.local)
Found: 192.168.60.12 (promaxgb10-f3da.local)
Found: 192.168.60.13 (gn100-936b.local)
Found: 192.168.60.16 (promaxgb10-f3da.local)
Setting up shared SSH access across all nodes...
You may be prompted for your password on each node.
Configuring 192.168.60.12...
✓ Successfully configured 192.168.60.12 with shared key
Configuring 192.168.60.13...
✓ Successfully configured 192.168.60.13 with shared key
Configuring 192.168.60.16...
✓ Successfully configured 192.168.60.16 with shared key
Shared SSH setup complete!
All nodes can now SSH to each other using the shared key (id_ed25519_shared).
7. Verify SSH access
As the final step, we can check if both Sparks can connect to each other. For that we can use these commands to connect to the other Spark and the one we work on:
If we can log-in without a password everything is ready for the next part. If this fails, retry the previous steps and make sure that you did not set the same IP address on both Sparks.
Next
With the few extra steps, we can configure our Sparks to talk to each other over the QSFP cable without running in all the traps I had with the official guide. The QSFP cable allows us a much larger bandwidth than the ethernet port we would use on any other computer. If we now restart our Sparks, they will keep the network configuration, and we can skip the setup when we continue our work.
Next week we configure VLLM to run a model on both Sparks.