1242-GRP Distributed Cycle Splitter
Location
https://campus.kennesaw.edu/colleges-departments/ccse/events/computing-showcase/fa26-cday-program.php
Document Type
Event
Start Date
20-11-2026 2:00 PM
Description
Deep learning systems often permute tensors distributed across GPUs. PyTorch's all_to_all_nd writes each GPU's result into a separate output buffer, doubling the memory each GPU's share occupies. We design Distributed Cycle Splitter (DCS), an in-place alternative to all_to_all_nd for the same permutations. DCS keeps its total auxiliary memory within 5% of the tensor instead of a full output buffer. It follows the permutation's cycles, writes blocks that must move across GPUs into the other GPU's inbox buffer through symmetric memory, and the receiver unpacks them one group at a time. On two PCIe-connected GPUs with 40 MB to 4 GB tensors, DCS stays within this bound on all 271 shapes and uses 94-98% less auxiliary memory per GPU than all_to_all_nd.
1242-GRP Distributed Cycle Splitter
https://campus.kennesaw.edu/colleges-departments/ccse/events/computing-showcase/fa26-cday-program.php
Deep learning systems often permute tensors distributed across GPUs. PyTorch's all_to_all_nd writes each GPU's result into a separate output buffer, doubling the memory each GPU's share occupies. We design Distributed Cycle Splitter (DCS), an in-place alternative to all_to_all_nd for the same permutations. DCS keeps its total auxiliary memory within 5% of the tensor instead of a full output buffer. It follows the permutation's cycles, writes blocks that must move across GPUs into the other GPU's inbox buffer through symmetric memory, and the receiver unpacks them one group at a time. On two PCIe-connected GPUs with 40 MB to 4 GB tensors, DCS stays within this bound on all 271 shapes and uses 94-98% less auxiliary memory per GPU than all_to_all_nd.