Daniele De Sensi

2024

1. NSDI24

   Swing: Short-cutting Rings for Higher Bandwidth Allreduce

   Daniele De Sensi, Tommaso Bonato, David Saam, and 1 more author

   In 21th USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), Apr 2024

   Abs Bib PDF

   The allreduce collective operation accounts for a significant fraction of the runtime of workloads running on distributed systems. One factor determining its performance is the distance between communicating nodes, especially on networks like torus, where a higher distance implies multiple messages being forwarded on the same link, thus reducing the allreduce bandwidth. Torus networks are widely used on systems optimized for machine learning workloads (e.g., Google TPUs and Amazon Trainium devices), as well as on some of the Top500 supercomputers. To improve allreduce performance on torus networks we introduce Swing, a new algorithm that keeps a low distance between communicating nodes by swinging between torus directions. Our analysis and experimental evaluation show that Swing outperforms by up to 3x existing allreduce algorithms for vectors ranging from 32B to 128MiB, on different types of torus and torus-like topologies, regardless of their shape and size.

   @inproceedings{swing,
     author = {De Sensi, Daniele and Bonato, Tommaso and Saam, David and Hoefler, Torsten},
     title = {Swing: Short-cutting Rings for Higher Bandwidth Allreduce},
     booktitle = {21th USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)},
     year = {2024},
     address = {Santa Clara, CA},
     publisher = {USENIX Association},
     month = apr,
     dimensions = {true},
   }

2. FGCS

   Canary: Congestion-aware in-network allreduce using dynamic trees

   Daniele De Sensi, Edgar Costa Molero, Salvatore Di Girolamo, and 2 more authors

   Future Generation Computer Systems, Apr 2024

   Abs arXiv Bib PDF

   The allreduce operation is an essential building block for many distributed applications, ranging from the training of deep learning models to scientific computing. In an allreduce operation, data from multiple hosts is aggregated together and then broadcasted to each host participating in the operation. Allreduce performance can be improved by a factor of two by aggregating the data directly in the network. Switches aggregate data coming from multiple ports before forwarding the partially aggregated result to the next hop. In all existing solutions, each switch needs to know the ports from which it will receive the data to aggregate. However, this forces packets to traverse a predefined set of switches, making these solutions prone to congestion. For this reason, we design Canary, the first congestion-aware in-network allreduce algorithm. Canary uses load balancing algorithms to forward packets on the least congested paths. Because switches do not know from which ports they will receive the data to aggregate, they use timeouts to aggregate the data in a best-effort way. We develop a P4 Canary prototype and evaluate it on a Tofino switch. We then validate Canary through simulations on large networks, showing performance improvements up to 40% compared to the state-of-the-art.

   @article{canary,
     title = {Canary: Congestion-aware in-network allreduce using dynamic trees},
     journal = {Future Generation Computer Systems},
     volume = {152},
     pages = {70-82},
     year = {2024},
     issn = {0167-739X},
     doi = {https://doi.org/10.1016/j.future.2023.10.010},
     url = {https://www.sciencedirect.com/science/article/pii/S0167739X23003850},
     author = {{De Sensi}, Daniele and {Costa Molero}, Edgar and {Di Girolamo}, Salvatore and Vanbever, Laurent and Hoefler, Torsten},
     keywords = {In-network compute, Allreduce, Load balancing},
     dimensions = {true},
   }

2022

1. SIGMETRICS23

   Noise in the Clouds: Influence of Network Performance Variability on Application Scalability

   Daniele De Sensi, Tiziano De Matteis, Konstantin Taranov, and 3 more authors

   Proc. ACM Meas. Anal. Comput. Syst., Dec 2022

   Abs arXiv Bib PDF Code Video

   Cloud computing represents an appealing opportunity for cost-effective deployment of HPC workloads on the best-fitting hardware. However, although cloud and on-premise HPC systems offer similar computational resources, their network architecture and performance may differ significantly. For example, these systems use fundamentally different network transport and routing protocols, which may introduce \textitnetwork noise that can eventually limit the application scaling. This work analyzes network performance, scalability, and cost of running HPC workloads on cloud systems. First, we consider latency, bandwidth, and collective communication patterns in detailed small-scale measurements, and then we simulate network performance at a larger scale. We validate our approach on four popular cloud providers and three on-premise HPC systems, showing that network (and also OS) noise can significantly impact performance and cost both at small and large scale.

   @article{cloudnoise,
     author = {De Sensi, Daniele and De Matteis, Tiziano and Taranov, Konstantin and Di Girolamo, Salvatore and Rahn, Tobias and Hoefler, Torsten},
     title = {Noise in the Clouds: Influence of Network Performance Variability on Application Scalability},
     year = {2022},
     issue_date = {December 2022},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     volume = {6},
     number = {3},
     doi = {10.1145/3570609},
     journal = {Proc. ACM Meas. Anal. Comput. Syst.},
     month = dec,
     articleno = {49},
     numpages = {27},
     keywords = {cloud; HPC; network noise; scalability;},
     eprint = {2210.15315},
     dimensions = {true},
   }

2. SC22

   HammingMesh: A Network Topology for Large-Scale Deep Learning

   Torsten Hoefler, Tommaso Bonato, Daniele De Sensi, and 7 more authors

   In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC’22), Nov 2022

   Abs arXiv Bib PDF Code Video Best Reproducibility Advancement Award

   Numerous microarchitectural optimizations unlocked tremendous processing power for deep neural networks that in turn fueled the AI revolution. With the exhaustion of such optimizations, the growth of modern AI is now gated by the performance of training systems, especially their data movement. Instead of focusing on single accelerators, we investigate data-movement characteristics of large-scale training at full system scale. Based on our workload analysis, we design HammingMesh, a novel network topology that provides high bandwidth at low cost with high job scheduling flexibility. Specifically, HammingMesh can support full bandwidth and isolation to deep learning training jobs with two dimensions of parallelism. Furthermore, it also supports high global bandwidth for generic traffic. Thus, HammingMesh will power future large-scale deep learning systems with extreme bandwidth requirements.

   @inproceedings{hxmesh,
     author = {Hoefler, Torsten and Bonato, Tommaso and De Sensi, Daniele and Di Girolamo, Salvatore and Li, Shigang and Heddes, Marco and Belk, Jon and Goel, Deepak and Castro, Miguel and Scott, Steve},
     title = {HammingMesh: A Network Topology for Large-Scale Deep Learning},
     year = {2022},
     month = nov,
     booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC'22)},
     award = {Best Reproducibility Advancement Award},
     doi = {10.1109/sc41404.2022.00016},
     eprint = {2209.01346},
     dimensions = {true},
   }

2021

1. SC21

   Flare: Flexible in-Network Allreduce

   Daniele De Sensi, Salvatore Di Girolamo, Saleh Ashkboos, and 2 more authors

   In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Nov 2021

   Abs arXiv Bib PDF Code Slides Video

   The allreduce operation is one of the most commonly used communication routines in distributed applications. To improve its bandwidth and to reduce network traffic, this operation can be accelerated by offloading it to network switches, that aggregate the data received from the hosts, and send them back the aggregated result. However, existing solutions provide limited customization opportunities and might provide suboptimal performance when dealing with custom operators and data types, with sparse data, or when reproducibility of the aggregation is a concern. To deal with these problems, in this work we design a flexible programmable switch by using as a building block PsPIN, a RISC-V architecture implementing the sPIN programming model. We then design, model, and analyze different algorithms for executing the aggregation on this architecture, showing performance improvements compared to state-of-the-art approaches.

   @inproceedings{flare,
     author = {De Sensi, Daniele and Di Girolamo, Salvatore and Ashkboos, Saleh and Li, Shigang and Hoefler, Torsten},
     title = {Flare: Flexible in-Network Allreduce},
     year = {2021},
     isbn = {9781450384421},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3458817.3476178},
     doi = {10.1145/3458817.3476178},
     booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
     articleno = {35},
     numpages = {16},
     keywords = {allreduce, programmable switch, in-network computing},
     location = {St. Louis, Missouri},
     series = {SC '21},
     dimensions = {true},
   }

2020

1. SC20

   An In-Depth Analysis of the Slingshot Interconnect

   Daniele De Sensi, Salvatore Di Girolamo, Kim H. McMahon, and 2 more authors

   In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Nov 2020

   Abs arXiv Bib PDF Code Slides Video

   The interconnect is one of the most critical components in large scale computing systems, and its impact on the performance of applications is going to increase with the system size. In this paper, we will describe Slingshot, an interconnection network for large scale computing systems. Slingshot is based on high-radix switches, which allow building exascale and hyper-scale datacenters networks with at most three switch-to-switch hops. Moreover, Slingshot provides efficient adaptive routing and congestion control algorithms, and highly tunable traffic classes. Slingshot uses an optimized Ethernet protocol, which allows it to be interoperable with standard Ethernet devices while providing high performance to HPC applications. We analyze the extent to which Slingshot provides these features, evaluating it on microbenchmarks and on several applications from the datacenter and AI worlds, as well as on HPC applications. We find that applications running on Slingshot are less affected by congestion compared to previous generation networks.

   @inproceedings{sc2020,
     author = {De Sensi, Daniele and Di Girolamo, Salvatore and McMahon, Kim H. and Roweth, Duncan and Hoefler, Torsten},
     title = {An In-Depth Analysis of the Slingshot Interconnect},
     year = {2020},
     isbn = {9781728199986},
     url = {https://dl.acm.org/doi/abs/10.5555/3433701.3433747},
     publisher = {IEEE Press},
     booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
     articleno = {35},
     numpages = {14},
     keywords = {dragonfly, exascale, interconnection network, congestion, datacenters},
     location = {Atlanta, Georgia},
     series = {SC '20},
     doi = {10.1109/sc41405.2020.00039},
     dimensions = {true},
   }

2019

1. SC19

   Mitigating Network Noise on Dragonfly Networks Through Application-aware Routing

   Daniele De Sensi, Salvatore Di Girolamo, and Torsten Hoefler

   In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Nov 2019

   Abs arXiv Bib PDF Code Slides Video

   System noise can negatively impact the performance of HPC systems, and the interconnection network is one of the main factors contributing to this problem. To mitigate this effect, adaptive routing sends packets on non-minimal paths if they are less congested. However, while this may mitigate interference caused by congestion, it also generates more traffic since packets traverse additional hops, causing in turn congestion on other applications and on the application itself. In this paper, we first describe how to estimate network noise. By following these guidelines, we show how noise can be reduced by using routing algorithms which select minimal paths with a higher probability. We exploit this knowledge to design an algorithm which changes the probability of selecting minimal paths according to the application characteristics. We validate our solution on microbenchmarks and real-world applications on two systems relying on a Dragonfly interconnection network, showing noise reduction and performance improvement.

   @inproceedings{sc2019,
     author = {De Sensi, Daniele and Di Girolamo, Salvatore and Hoefler, Torsten},
     title = {Mitigating Network Noise on Dragonfly Networks Through Application-aware Routing},
     booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
     series = {SC '19},
     year = {2019},
     isbn = {978-1-4503-6229-0},
     location = {Denver, Colorado},
     pages = {16:1--16:32},
     articleno = {16},
     numpages = {32},
     url = {http://doi.acm.org/10.1145/3295500.3356196},
     doi = {10.1145/3295500.3356196},
     acmid = {3356196},
     publisher = {ACM},
     address = {New York, NY, USA},
     keywords = {dragonfly, network noise, routing},
     openaccess = {https://dl.acm.org/authorize?N690645},
     dimensions = {true},
   }