Daniele De Sensi

2026

1. CSUR

   The Landscape of GPU-Centric Communication

   Didem Unat, Ilyas Turimbetov, Mohammed Kefah Taha Issa, and 4 more authors

   ACM Comput. Surv., Mar 2026

   Abs arXiv Bib

   In recent years, GPUs have become the preferred accelerators for HPC and ML applications due to their parallelism and fast memory bandwidth. While GPUs boost computation, inter-GPU communication can create scalability bottlenecks, especially as the number of GPUs per node and cluster grows. Traditionally, the CPU managed multi-GPU communication, but advancements in GPU-centric communication now challenge this CPU dominance by reducing its involvement, granting GPUs more autonomy in communication tasks, and addressing mismatches in multi-GPU communication and computation. This paper provides a landscape of GPU-centric communication, focusing on vendor mechanisms and user-level library supports. It aims to clarify the complexities and diverse options in this field, define the terminology, and categorize existing approaches within and across nodes. The paper discusses vendor-provided mechanisms for communication and memory management in multi-GPU execution and reviews major communication libraries, their benefits, challenges, and performance insights. Then, it explores key research paradigms, future outlooks, and open research questions. By extensively describing GPU-centric communication techniques across the software and hardware stacks, we provide researchers, programmers, engineers, and library designers insights on how to exploit multi-GPU systems at their best.

   @article{gpucentriccomm,
     author = {Unat, Didem and Turimbetov, Ilyas and Kefah Taha Issa, Mohammed and Sagbili, Dogan and Vella, Flavio and De Sensi, Daniele and Ismayilov, Ismayil},
     title = {The Landscape of GPU-Centric Communication},
     year = {2026},
     month = mar,
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     journal = {ACM Comput. Surv.},
     dimensions = {true},
   }

2025

1. SC25

   Bine Trees: Enhancing Collective Operations by Optimizing Communication Locality

   Daniele De Sensi, Saverio Pasqualoni, Lorenzo Piarulli, and 5 more authors

   In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC’25), Nov 2025

   Abs arXiv Bib Slides

   As high-performance computing (HPC) systems grow, optimizing communication locality becomes essential for performance. HPC networks are often oversubscribed, consisting of fully connected groups that are sparsely connected. We introduce Binomial Negabinary (Bine) trees, a novel approach to enhance collective operations by reducing inter-group communication. They minimize the distance between communicating ranks, reducing traffic on global links and alleviating congestion. Unlike traditional hierarchical algorithms, Bine trees are topology-agnostic and do not assume a uniform partition of ranks, making them ideal for production supercomputers with irregular process allocations. We design algorithms for eight collectives, achieving up to 5x speedups and 33% less global traffic on four supercomputers with four different topologies. Our results emphasize their effectiveness in improving performance while reducing the load on global links.

   @inproceedings{bine,
     author = {De Sensi, Daniele and Pasqualoni, Saverio and Piarulli, Lorenzo and Bonato, Tommaso and Ba, Seydou and Turisini, Matteo and Domke, Jens and Hoefler, Torsten},
     title = {Bine Trees: Enhancing Collective Operations by Optimizing Communication Locality},
     year = {2025},
     month = nov,
     booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC'25)},
     doi = {To Appear},
     dimensions = {true},
   }

2024

1. CACM

   HammingMesh: A Network Topology for Large-Scale Deep Learning

   Torsten Hoefler, Tommaso Bonato, Daniele De Sensi, and 6 more authors

   Communications of the ACM, Nov 2024

   Online First

   Abs Bib

   Numerous microarchitectural optimizations unlocked tremendous processing power for deep neural networks that in turn fueled the ongoing AI revolution. With the exhaustion of such optimizations, the growth of modern AI is now gated by the performance of training systems, especially their data movement. Instead of focusing on single accelerators, we investigate data-movement characteristics of large-scale training at full system scale. Based on our workload analysis, we design HammingMesh, a novel network topology that provides high bandwidth at low cost with high job scheduling flexibility. Specifically, HammingMesh can support full bandwidth and isolation to deep learning training jobs with two dimensions of parallelism. Furthermore, it also supports high global bandwidth for generic traffic. Thus, HammingMesh will power future large-scale deep learning systems with extreme bandwidth requirements.

   @article{10.1145/3623490,
     author = {Hoefler, Torsten and Bonato, Tommaso and De Sensi, Daniele and Di Girolamo, Salvatore and Li, Shigang and Heddes, Marco and Goel, Deepak and Castro, Miguel and Scott, Steve},
     title = {HammingMesh: A Network Topology for Large-Scale Deep Learning},
     year = {2024},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     issn = {0001-0782},
     url = {https://doi.org/10.1145/3623490},
     doi = {10.1145/3623490},
     note = {Online First},
     journal = {Communications of the ACM},
     month = nov,
     numpages = {15},
     dimensions = {true},
   }

2. NSDI24

   Swing: Short-cutting Rings for Higher Bandwidth Allreduce

   Daniele De Sensi, Tommaso Bonato, David Saam, and 1 more author

   In 21th USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), Apr 2024

   Abs arXiv Bib PDF Video

   The allreduce collective operation accounts for a significant fraction of the runtime of workloads running on distributed systems. One factor determining its performance is the distance between communicating nodes, especially on networks like torus, where a higher distance implies multiple messages being forwarded on the same link, thus reducing the allreduce bandwidth. Torus networks are widely used on systems optimized for machine learning workloads (e.g., Google TPUs and Amazon Trainium devices), as well as on some of the Top500 supercomputers. To improve allreduce performance on torus networks we introduce Swing, a new algorithm that keeps a low distance between communicating nodes by swinging between torus directions. Our analysis and experimental evaluation show that Swing outperforms by up to 3x existing allreduce algorithms for vectors ranging from 32B to 128MiB, on different types of torus and torus-like topologies, regardless of their shape and size.

   @inproceedings{swing,
     author = {De Sensi, Daniele and Bonato, Tommaso and Saam, David and Hoefler, Torsten},
     title = {Swing: Short-cutting Rings for Higher Bandwidth Allreduce},
     booktitle = {21th USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)},
     year = {2024},
     address = {Santa Clara, CA},
     publisher = {USENIX Association},
     month = apr,
     dimensions = {true},
   }

2022

1. SIGMETRICS23

   Noise in the Clouds: Influence of Network Performance Variability on Application Scalability

   Daniele De Sensi, Tiziano De Matteis, Konstantin Taranov, and 3 more authors

   Proc. ACM Meas. Anal. Comput. Syst., Dec 2022

   Abs arXiv Bib PDF Code Video

   Cloud computing represents an appealing opportunity for cost-effective deployment of HPC workloads on the best-fitting hardware. However, although cloud and on-premise HPC systems offer similar computational resources, their network architecture and performance may differ significantly. For example, these systems use fundamentally different network transport and routing protocols, which may introduce network noise that can eventually limit the application scaling. This work analyzes network performance, scalability, and cost of running HPC workloads on cloud systems. First, we consider latency, bandwidth, and collective communication patterns in detailed small-scale measurements, and then we simulate network performance at a larger scale. We validate our approach on four popular cloud providers and three on-premise HPC systems, showing that network (and also OS) noise can significantly impact performance and cost both at small and large scale.

   @article{cloudnoise,
     author = {De Sensi, Daniele and De Matteis, Tiziano and Taranov, Konstantin and Di Girolamo, Salvatore and Rahn, Tobias and Hoefler, Torsten},
     title = {Noise in the Clouds: Influence of Network Performance Variability on Application Scalability},
     year = {2022},
     issue_date = {December 2022},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     volume = {6},
     number = {3},
     doi = {10.1145/3570609},
     journal = {Proc. ACM Meas. Anal. Comput. Syst.},
     month = dec,
     articleno = {49},
     numpages = {27},
     keywords = {cloud; HPC; network noise; scalability;},
     eprint = {2210.15315},
     dimensions = {true},
   }

2021

1. SC21

   Flare: Flexible in-Network Allreduce

   Daniele De Sensi, Salvatore Di Girolamo, Saleh Ashkboos, and 2 more authors

   In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Dec 2021

   Abs arXiv Bib PDF Code Slides Video

   The allreduce operation is one of the most commonly used communication routines in distributed applications. To improve its bandwidth and to reduce network traffic, this operation can be accelerated by offloading it to network switches, that aggregate the data received from the hosts, and send them back the aggregated result. However, existing solutions provide limited customization opportunities and might provide suboptimal performance when dealing with custom operators and data types, with sparse data, or when reproducibility of the aggregation is a concern. To deal with these problems, in this work we design a flexible programmable switch by using as a building block PsPIN, a RISC-V architecture implementing the sPIN programming model. We then design, model, and analyze different algorithms for executing the aggregation on this architecture, showing performance improvements compared to state-of-the-art approaches.

   @inproceedings{flare,
     author = {De Sensi, Daniele and Di Girolamo, Salvatore and Ashkboos, Saleh and Li, Shigang and Hoefler, Torsten},
     title = {Flare: Flexible in-Network Allreduce},
     year = {2021},
     isbn = {9781450384421},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3458817.3476178},
     doi = {10.1145/3458817.3476178},
     booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
     articleno = {35},
     numpages = {16},
     keywords = {allreduce, programmable switch, in-network computing},
     location = {St. Louis, Missouri},
     series = {SC '21},
     dimensions = {true},
   }

2020

1. SC20

   An In-Depth Analysis of the Slingshot Interconnect

   Daniele De Sensi, Salvatore Di Girolamo, Kim H. McMahon, and 2 more authors

   In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Dec 2020

   Abs arXiv Bib PDF Code Slides Video

   The interconnect is one of the most critical components in large scale computing systems, and its impact on the performance of applications is going to increase with the system size. In this paper, we will describe Slingshot, an interconnection network for large scale computing systems. Slingshot is based on high-radix switches, which allow building exascale and hyper-scale datacenters networks with at most three switch-to-switch hops. Moreover, Slingshot provides efficient adaptive routing and congestion control algorithms, and highly tunable traffic classes. Slingshot uses an optimized Ethernet protocol, which allows it to be interoperable with standard Ethernet devices while providing high performance to HPC applications. We analyze the extent to which Slingshot provides these features, evaluating it on microbenchmarks and on several applications from the datacenter and AI worlds, as well as on HPC applications. We find that applications running on Slingshot are less affected by congestion compared to previous generation networks.

   @inproceedings{sc2020,
     author = {De Sensi, Daniele and Di Girolamo, Salvatore and McMahon, Kim H. and Roweth, Duncan and Hoefler, Torsten},
     title = {An In-Depth Analysis of the Slingshot Interconnect},
     year = {2020},
     isbn = {9781728199986},
     url = {https://dl.acm.org/doi/abs/10.5555/3433701.3433747},
     publisher = {IEEE Press},
     booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
     articleno = {35},
     numpages = {14},
     keywords = {dragonfly, exascale, interconnection network, congestion, datacenters},
     location = {Atlanta, Georgia},
     series = {SC '20},
     doi = {10.1109/sc41405.2020.00039},
     dimensions = {true},
   }

2019

1. SC19

   Mitigating Network Noise on Dragonfly Networks Through Application-aware Routing

   Daniele De Sensi, Salvatore Di Girolamo, and Torsten Hoefler

   In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Dec 2019

   Abs arXiv Bib PDF Code Slides Video

   System noise can negatively impact the performance of HPC systems, and the interconnection network is one of the main factors contributing to this problem. To mitigate this effect, adaptive routing sends packets on non-minimal paths if they are less congested. However, while this may mitigate interference caused by congestion, it also generates more traffic since packets traverse additional hops, causing in turn congestion on other applications and on the application itself. In this paper, we first describe how to estimate network noise. By following these guidelines, we show how noise can be reduced by using routing algorithms which select minimal paths with a higher probability. We exploit this knowledge to design an algorithm which changes the probability of selecting minimal paths according to the application characteristics. We validate our solution on microbenchmarks and real-world applications on two systems relying on a Dragonfly interconnection network, showing noise reduction and performance improvement.

   @inproceedings{sc2019,
     author = {De Sensi, Daniele and Di Girolamo, Salvatore and Hoefler, Torsten},
     title = {Mitigating Network Noise on Dragonfly Networks Through Application-aware Routing},
     booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
     series = {SC '19},
     year = {2019},
     isbn = {978-1-4503-6229-0},
     location = {Denver, Colorado},
     pages = {16:1--16:32},
     articleno = {16},
     numpages = {32},
     url = {http://doi.acm.org/10.1145/3295500.3356196},
     doi = {10.1145/3295500.3356196},
     acmid = {3356196},
     publisher = {ACM},
     address = {New York, NY, USA},
     keywords = {dragonfly, network noise, routing},
     openaccess = {https://dl.acm.org/authorize?N690645},
     dimensions = {true},
   }