This series tests four ways to give a Kubernetes pod RDMA for GPU training, on two pairs of A100 nodes.
Badges
Badges come from GitHub once the author claims this feed.
Recent posts
every link opens the author’s post-
Getting RDMA into Kubernetes Pods, Part 4: DRANET
-
Getting RDMA into Kubernetes Pods, Part 3: IPvlan
This series tests four ways to give a Kubernetes pod RDMA for GPU training, on two pairs of A100 nodes.
-
Getting RDMA into Kubernetes Pods, Part 2: SR-IOV VFs
This series tests four ways to give a Kubernetes pod RDMA for GPU training, on two pairs of A100 nodes.
-
Getting RDMA into Kubernetes Pods, Part 1: hostNetwork
There are four common ways to give a Kubernetes pod RDMA for GPU training:
-
Inside Dranet: From Host NIC to Pod
Dranet is a Kubernetes network driver.
-
Why Pods See Zero RoCE GIDs on DOCA-OFED
I looked into B300 nodes where RDMA worked on the host but not in pods.
-
ARP, rp_filter and accept_ra for RDMA Pods on Kubernetes
A GPU node with 16 RoCE rails has 16 network interfaces, often in one subnet.
-
Gang Scheduling and Topology-Aware Scheduling in Upstream Kubernetes
In my Kueue post, Kueue did all the placement work.
-
Why Large RoCE Frames Disappeared on OVS-DOCA 3.5
I looked into GB300 hosts where RDMA broke after an image update.
-
Why torchrun Can't Find Node Two on the Host Network
If you want RDMA inside a Kubernetes pod, the simplest way is the host network.
-
RDMA Network Locality for Multi-Node Jobs with Kueue
The pods of a multi-node training job talk to each other all the time.