Daniele De Sensi

I am an Assistant Professor at Sapienza University of Rome. I was previously a PostDoc researcher at ETH Zürich and University of Pisa. I received the Ph.D. in Computer Science from the University of Pisa.

My main reseach interests lie on the fields of Interconnection Networks, High Performance Computing, Green Computing, and Parallel Programming. You can find more about my research by checking the publications page.

If you have anything to discuss, feel free to send me an e-mail.

News

Aug 27, 2024	🎉 I’m happy to announce that the paper Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects have been accepted at SC ‘24. The paper focuses on GPU-GPU interconnect. We analyzed three top-10 supercomputers, on up to 4k NVIDIA and AMD GPUs, offering valuable insights to architects, researchers, practitioners, and developers. The paper is the outcome of a fruitful collaboration with NVIDIA, Cray HPE, ETH Zurich, CINECA, University of Trento, Vrije Universitiet, and University of Antwerp. You can find a preprint here.
May 2, 2024	🎉 I’m happy to announce that the paper OSMOSIS: Enabling Multi-Tenancy in Datacenter SmartNICs have been accepted at USENIX ATC ‘24. OSMOSIS is a resource manager for SmartNICs to support multi-tenancy with low overhead.
Apr 4, 2024	🎉 I’m happy to announce that a new paper have been accepted at HPDC ‘24. The paper introduces a performance model and new algorithms for reduce and allreduce on the Cerebras Wafer-Scale Engine (WSE), a novel architecture optimized for machine learning workloads. The new algorithms outperform the current vendor solutions by more than 3x.
Feb 15, 2024	👨‍🏫 In partnership with the University of Trento and CINI, we will host in June a summer school focused on High-Performance Computing (HPC) (https://hpc-summer-school-24.disi.unitn.it/). I’ll be giving two lectures on interconnection networks and the process of writing HPC papers. If you’re interested, feel free to join.
Dec 11, 2023	🎉 I’m happy to announce that two papers have been accepted at NSDI ‘24. The first one is about a new algorithm for allreduce on multidimensional torus networks that improves performance up to 3x compared to state-of-the-art algorithms (you can find a preprint here). The second one is on the design, implementation, and evaluation of the first-ever Slim Fly network deployment (here).

Selected Publications

2024

NSDI24
Swing: Short-cutting Rings for Higher Bandwidth Allreduce

Daniele De Sensi, Tommaso Bonato, David Saam, and 1 more author

In 21th USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), Apr 2024

Abs arXiv Bib PDF Video

The allreduce collective operation accounts for a significant fraction of the runtime of workloads running on distributed systems. One factor determining its performance is the distance between communicating nodes, especially on networks like torus, where a higher distance implies multiple messages being forwarded on the same link, thus reducing the allreduce bandwidth. Torus networks are widely used on systems optimized for machine learning workloads (e.g., Google TPUs and Amazon Trainium devices), as well as on some of the Top500 supercomputers. To improve allreduce performance on torus networks we introduce Swing, a new algorithm that keeps a low distance between communicating nodes by swinging between torus directions. Our analysis and experimental evaluation show that Swing outperforms by up to 3x existing allreduce algorithms for vectors ranging from 32B to 128MiB, on different types of torus and torus-like topologies, regardless of their shape and size.
@inproceedings{swing, author = {De Sensi, Daniele and Bonato, Tommaso and Saam, David and Hoefler, Torsten}, title = {Swing: Short-cutting Rings for Higher Bandwidth Allreduce}, booktitle = {21th USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)}, year = {2024}, address = {Santa Clara, CA}, publisher = {USENIX Association}, month = apr, dimensions = {true}, }
FGCS
Canary: Congestion-aware in-network allreduce using dynamic trees

Daniele De Sensi, Edgar Costa Molero, Salvatore Di Girolamo, and 2 more authors

Future Generation Computer Systems, Apr 2024

Abs arXiv Bib PDF

The allreduce operation is an essential building block for many distributed applications, ranging from the training of deep learning models to scientific computing. In an allreduce operation, data from multiple hosts is aggregated together and then broadcasted to each host participating in the operation. Allreduce performance can be improved by a factor of two by aggregating the data directly in the network. Switches aggregate data coming from multiple ports before forwarding the partially aggregated result to the next hop. In all existing solutions, each switch needs to know the ports from which it will receive the data to aggregate. However, this forces packets to traverse a predefined set of switches, making these solutions prone to congestion. For this reason, we design Canary, the first congestion-aware in-network allreduce algorithm. Canary uses load balancing algorithms to forward packets on the least congested paths. Because switches do not know from which ports they will receive the data to aggregate, they use timeouts to aggregate the data in a best-effort way. We develop a P4 Canary prototype and evaluate it on a Tofino switch. We then validate Canary through simulations on large networks, showing performance improvements up to 40% compared to the state-of-the-art.
@article{canary, title = {Canary: Congestion-aware in-network allreduce using dynamic trees}, journal = {Future Generation Computer Systems}, volume = {152}, pages = {70-82}, year = {2024}, issn = {0167-739X}, doi = {https://doi.org/10.1016/j.future.2023.10.010}, url = {https://www.sciencedirect.com/science/article/pii/S0167739X23003850}, author = {{De Sensi}, Daniele and {Costa Molero}, Edgar and {Di Girolamo}, Salvatore and Vanbever, Laurent and Hoefler, Torsten}, keywords = {In-network compute, Allreduce, Load balancing}, dimensions = {true}, }

2022

SIGMETRICS23
Noise in the Clouds: Influence of Network Performance Variability on Application Scalability

Daniele De Sensi, Tiziano De Matteis, Konstantin Taranov, and 3 more authors

Proc. ACM Meas. Anal. Comput. Syst., Dec 2022

Abs arXiv Bib PDF Code Video

Cloud computing represents an appealing opportunity for cost-effective deployment of HPC workloads on the best-fitting hardware. However, although cloud and on-premise HPC systems offer similar computational resources, their network architecture and performance may differ significantly. For example, these systems use fundamentally different network transport and routing protocols, which may introduce \textitnetwork noise that can eventually limit the application scaling. This work analyzes network performance, scalability, and cost of running HPC workloads on cloud systems. First, we consider latency, bandwidth, and collective communication patterns in detailed small-scale measurements, and then we simulate network performance at a larger scale. We validate our approach on four popular cloud providers and three on-premise HPC systems, showing that network (and also OS) noise can significantly impact performance and cost both at small and large scale.
@article{cloudnoise, author = {De Sensi, Daniele and De Matteis, Tiziano and Taranov, Konstantin and Di Girolamo, Salvatore and Rahn, Tobias and Hoefler, Torsten}, title = {Noise in the Clouds: Influence of Network Performance Variability on Application Scalability}, year = {2022}, issue_date = {December 2022}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, volume = {6}, number = {3}, doi = {10.1145/3570609}, journal = {Proc. ACM Meas. Anal. Comput. Syst.}, month = dec, articleno = {49}, numpages = {27}, keywords = {cloud; HPC; network noise; scalability;}, eprint = {2210.15315}, dimensions = {true}, }
SC22
HammingMesh: A Network Topology for Large-Scale Deep Learning

Torsten Hoefler, Tommaso Bonato, Daniele De Sensi, and 7 more authors

In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC’22), Nov 2022

Abs arXiv Bib PDF Code Video Best Reproducibility Advancement Award

Numerous microarchitectural optimizations unlocked tremendous processing power for deep neural networks that in turn fueled the AI revolution. With the exhaustion of such optimizations, the growth of modern AI is now gated by the performance of training systems, especially their data movement. Instead of focusing on single accelerators, we investigate data-movement characteristics of large-scale training at full system scale. Based on our workload analysis, we design HammingMesh, a novel network topology that provides high bandwidth at low cost with high job scheduling flexibility. Specifically, HammingMesh can support full bandwidth and isolation to deep learning training jobs with two dimensions of parallelism. Furthermore, it also supports high global bandwidth for generic traffic. Thus, HammingMesh will power future large-scale deep learning systems with extreme bandwidth requirements.
@inproceedings{hxmesh, author = {Hoefler, Torsten and Bonato, Tommaso and De Sensi, Daniele and Di Girolamo, Salvatore and Li, Shigang and Heddes, Marco and Belk, Jon and Goel, Deepak and Castro, Miguel and Scott, Steve}, title = {HammingMesh: A Network Topology for Large-Scale Deep Learning}, year = {2022}, month = nov, booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC'22)}, award = {Best Reproducibility Advancement Award}, doi = {10.1109/sc41404.2022.00016}, eprint = {2209.01346}, dimensions = {true}, }

2021

SC21
Flare: Flexible in-Network Allreduce

Daniele De Sensi, Salvatore Di Girolamo, Saleh Ashkboos, and 2 more authors

In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Nov 2021

Abs arXiv Bib PDF Code Slides Video

The allreduce operation is one of the most commonly used communication routines in distributed applications. To improve its bandwidth and to reduce network traffic, this operation can be accelerated by offloading it to network switches, that aggregate the data received from the hosts, and send them back the aggregated result. However, existing solutions provide limited customization opportunities and might provide suboptimal performance when dealing with custom operators and data types, with sparse data, or when reproducibility of the aggregation is a concern. To deal with these problems, in this work we design a flexible programmable switch by using as a building block PsPIN, a RISC-V architecture implementing the sPIN programming model. We then design, model, and analyze different algorithms for executing the aggregation on this architecture, showing performance improvements compared to state-of-the-art approaches.
@inproceedings{flare, author = {De Sensi, Daniele and Di Girolamo, Salvatore and Ashkboos, Saleh and Li, Shigang and Hoefler, Torsten}, title = {Flare: Flexible in-Network Allreduce}, year = {2021}, isbn = {9781450384421}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3458817.3476178}, doi = {10.1145/3458817.3476178}, booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis}, articleno = {35}, numpages = {16}, keywords = {allreduce, programmable switch, in-network computing}, location = {St. Louis, Missouri}, series = {SC '21}, dimensions = {true}, }

2020

SC20
An In-Depth Analysis of the Slingshot Interconnect

Daniele De Sensi, Salvatore Di Girolamo, Kim H. McMahon, and 2 more authors

In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Nov 2020

Abs arXiv Bib PDF Code Slides Video

The interconnect is one of the most critical components in large scale computing systems, and its impact on the performance of applications is going to increase with the system size. In this paper, we will describe Slingshot, an interconnection network for large scale computing systems. Slingshot is based on high-radix switches, which allow building exascale and hyper-scale datacenters networks with at most three switch-to-switch hops. Moreover, Slingshot provides efficient adaptive routing and congestion control algorithms, and highly tunable traffic classes. Slingshot uses an optimized Ethernet protocol, which allows it to be interoperable with standard Ethernet devices while providing high performance to HPC applications. We analyze the extent to which Slingshot provides these features, evaluating it on microbenchmarks and on several applications from the datacenter and AI worlds, as well as on HPC applications. We find that applications running on Slingshot are less affected by congestion compared to previous generation networks.
@inproceedings{sc2020, author = {De Sensi, Daniele and Di Girolamo, Salvatore and McMahon, Kim H. and Roweth, Duncan and Hoefler, Torsten}, title = {An In-Depth Analysis of the Slingshot Interconnect}, year = {2020}, isbn = {9781728199986}, url = {https://dl.acm.org/doi/abs/10.5555/3433701.3433747}, publisher = {IEEE Press}, booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis}, articleno = {35}, numpages = {14}, keywords = {dragonfly, exascale, interconnection network, congestion, datacenters}, location = {Atlanta, Georgia}, series = {SC '20}, doi = {10.1109/sc41405.2020.00039}, dimensions = {true}, }

2019

SC19
Mitigating Network Noise on Dragonfly Networks Through Application-aware Routing

Daniele De Sensi, Salvatore Di Girolamo, and Torsten Hoefler

In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Nov 2019

Abs arXiv Bib PDF Code Slides Video

System noise can negatively impact the performance of HPC systems, and the interconnection network is one of the main factors contributing to this problem. To mitigate this effect, adaptive routing sends packets on non-minimal paths if they are less congested. However, while this may mitigate interference caused by congestion, it also generates more traffic since packets traverse additional hops, causing in turn congestion on other applications and on the application itself. In this paper, we first describe how to estimate network noise. By following these guidelines, we show how noise can be reduced by using routing algorithms which select minimal paths with a higher probability. We exploit this knowledge to design an algorithm which changes the probability of selecting minimal paths according to the application characteristics. We validate our solution on microbenchmarks and real-world applications on two systems relying on a Dragonfly interconnection network, showing noise reduction and performance improvement.
@inproceedings{sc2019, author = {De Sensi, Daniele and Di Girolamo, Salvatore and Hoefler, Torsten}, title = {Mitigating Network Noise on Dragonfly Networks Through Application-aware Routing}, booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis}, series = {SC '19}, year = {2019}, isbn = {978-1-4503-6229-0}, location = {Denver, Colorado}, pages = {16:1--16:32}, articleno = {16}, numpages = {32}, url = {http://doi.acm.org/10.1145/3295500.3356196}, doi = {10.1145/3295500.3356196}, acmid = {3356196}, publisher = {ACM}, address = {New York, NY, USA}, keywords = {dragonfly, network noise, routing}, openaccess = {https://dl.acm.org/authorize?N690645}, dimensions = {true}, }