Predictable and Deadline-Driven Data Transfers over High-Speed Networks (2019 - )
In many scientific workflows data is generated at remote locations and must be transferred to the distributed HPC nodes over high-speed networks. Increasingly many of these scientific workflows require processing to be completed within a deadline, which, in turn, imposes deadline for the network data transfer. This research focuses on science networks and private WANs in which the sources of traffic (and hence their nature) are well understood although the volume may change. The goal of this proposal is to design a system that consists of 1) a centralized network controller for scheduling (assigning rates to flows) and routing flows with time constraint and 2) an end-system traffic controller that can maintain the rates assigned by the controller with low variance. The goal of the network controller is to determine schedules that are provably correct implying that flows meet their deadlines and predictable implying that the completion time of the data transfer can be tightly bounded despite changes in the request and/or the network condition. Predictable data transfers along with intelligent routing and end-system traffic control will also result in high network utilization. The design and implementation of the network controller will be enabled by a SDN-based softwarized network and high-precision network telemetry.
In this research, we will adopt a Reinforcement Learning (RL) based approach to implement the network controller. We will extend our preliminary study and investigate how Deep Deterministic Policy Gradient (DDPG) approach can be adapted to design a scalable network controller. In order to guarantee predictability, we will formulate the scheduling problem as constraint satisfaction problem and use Satisfiability Modulo Theory (SMT) solvers to determine correct and predictable schedules. We will introduce the notion of $\delta$-satisfiability to capture the notion of predictability, and enable the network controller to synthesize a schedule that meets the deadline in spite of changes in the network state (e.g., utilization). To achieve the goals of the network scheduler we will design and implement a model predictive based (MPC)-based end-system traffic controller that can maintain smooth transmission rates assigned by network scheduler.
SAANet - System and Architecture Aware Networking (NSF, 2015 - 2019)
This research builds on INTIME and focuses on high speed networking using commodity multi-core systems. Network speeds are continuing to climb. 10~Gbps Network Interface Cards (NICs) are common now, 40~Gbps NICs are widely available, and 100~Gbps NICs will be available soon. However, in practice, little of this bandwidth capability is being actively utilized by applications. One of the major reasons for this is the relatively large protocol processing overhead of TCP/IP at these speeds. Increasing clock speeds can no longer be relied upon to ameliorate the problem of protocol processing. Meanwhile, TCP/IP remains the most widely-adopted protocol stack used by distributed applications and supported by widely-available hardware.
Our previous research has been dedicated to attaining high-throughput TCP/IP-based networking from commodity hardware by intelligently exploiting the parallelism of the end-systems using a concept we refer to as affinity. Affinity, or core binding, has to do with deciding which core a particular program or process is executed on in a multicore system. We have characterized the performance and efficiency of the TCP/IP receive process across multiple affinity configurations within modern and widely-deployed commodity end-systems. Through several publications, the results of our research are well-positioned to influence the design of applications, NIC driver and hardware design, and high-speed distributed systems - both directly through collaborations with application developers and network operators, and indirectly through industrial adoptions of affinity best practices.
Our current and future research builds upon this expertise, with the aim to aid in efficiently leveraging commodity end-system hardware by broadening our reach to newly-available technologies, protocols, and platforms. Furthermore, we plan to utilize efficient and highly-effective statistical methods to manage end-to-end performance of high speed flows to the point of delivering predictably efficient performance by monitoring parameters and modifying end-system variables at all times during network activity. Our process of careful characterization of current technology, followed by statistical analysis, and finally, middleware tool development affords us the maximum impact on shaping best practices while minimizing our impact on distributed application development processes.
In this research proposal, we intend to further characterize the end-system bottlenecks that arise during data transfers required in different distributed scientific and business applications. What we learn will drive the development of introspective end-system aware models, in order to auto-tune data transfers. This tuning will consider both latency and throughput requirements of the applications. We will develop flow striping methods that exploit multicore end-systems and adapt to the end-system bottlenecks. This will require addressing many new issues, such as assigning flows to cores while taking into account various (application, cache, and interrupt) affinities. Additionally, the underlying topology of the cache (inclusive vs exclusive), the memory organization (NUMA vs UMA), and the heterogeneity of the cores must also be considered when controlling the end-to-end flows. We will investigate memory-mapped network channels, such as RDMA over Converged Ethernet (RoCE), for data transfers over wide-area networks. Towards this end, we will design and implement memory management, message synchronization and end-to-end flow control to enable remote messaging for different types of network flows. From the end-system architectural perspective, we will propose and study cache architectures that can significantly improve the network I/O performance. The methods developed in this proposed research will be prototyped and tested in ESNet 100Gbps testbed and the UC Davis Research Network.
Project Title and Duration: [NSF 1528087] NeTS: Small: Addressing End-system Bottlenecks in High-speed Networks. October 2015 - September 2019.
INTIME (NSF, 2009 - 2013)
This National Science Foundation (NSF) project investigates optimizing network transfers using introspective modeling of end-systems. The bottleneck for the transferring data at very high speeds often turns out to be the end-system performance. In the absence of definitive knowledge about the workload at the receiving end-system, the sender's transmission rate often overshoots the critical bottleneck rate of the receiver. This typically results in oscillations between the extremes and poor performance. To optimize the performance of the transport protocols and achieve the important flow control functionality, it is important to estimate the receiving end-system effective bottleneck rate. In this project we will use modeling and active analysis of the end-system to estimate this rate. We will develop queueing network models for representing the different multi-core and multiprocessor end-systems running different types of workloads. We will develop a software tool to be integrated with existing transport protocols. We will carry out experimental analysis for different types of end-systems with different configurations and workloads. We will apply and extend methods that have been proposed to address the limitations of queueing network models for performance analysis of computer systems with bursty workloads and correlated service times. The software tool will be made available to the research community to analyze and optimize distributed applications and systems. The research project will provide a framework to train graduate and undergraduate students in both analytical and experimental methods, and develop knowledge and intuition about next generation computer systems and distributed applications.
People (Overall the projects)
*Current participants are bolded
Sambit Shukla (PhD Student)
Taran Lynn (PhD Student)
Nathan Handforf (PhD, 2018)
Jian Wu (Postdoc, 2018)
Joseph Mcgee (MS Student)
Harshvardhan Chauhan (MS, 2018)
Dylan Wang (Undergraduate Researcher, Microsoft, 2015)
Vishal Ahuja (PhD, 2014)
Rennie Archibald (PhD, 2013 )
Amitabha Banerjee (PhD, 2012)
Aditya Thakur (Professor, UCDavis)
Matt Farrens (Professor, UCDavis)
Dipak Ghosal (Professor, UCDavis)
Alex Sim (Staff Scientist, Lawrence Berkeley Labs)
Kesheng (John) Wu (Group Lead, Scientific Data Management, Lawrence Berkeley Labs)
Giuseppe Serazzi (Professor Emeritus, Politecnico di Milano)
Mehmet Balman (VMWare)
Brian Tierney (Staff Scientist and Group Leader, Lawrence Berkeley Labs)
Eric Pouyal (Staff Scientist, Lawrence Berkeley Labs)
- Leh, Goldwayne Smith, Jian Wu, Sambit Shukla, Matthew K. Farrens, and Dipak Ghosal. Model-Driven Joint Optimization of Power and Latency Guarantee in Data center Applications. SN Computer Science 1, no. 1 (2020): 34.
- Taran Lynn, Nathan Hanford, and Dipak Ghosal. Impact of buffer size on a congestion control algorithm based on model predictive control. ACM Workshop on Buer Sizing, 2019.
- Ghosal, Dipak, Sambit Shukla, Alex Sim, Aditya V. Thakur, and Kesheng Wu. A Reinforcement Learning Based Network Scheduler For Deadline-Driven Data Transfers. IEEE Globecom Conference, December 2019.
- Sambit Kumar Shukla, Dipak Ghosal, Kesheng Wu, Alex Sim, Matthew K. Farrens: Co-optimizing Latency and Energy for IoT services using HMP servers in Fog Clusters. IEEE FMEC 2019: 121-12812
- Sambit Kumar Shukla, Dipak Ghosal, Matthew K. Farrens: Tuning Network I/O Processing to Achieve Performance and Energy Objectives of Latency Critical Workloads. HPCC/SmartCity/DSS 2019: 1499-1508.
- Nathan Hanford, Vishal Ahuja, Matthew K. Farrens, Brian Tierney, Dipak Ghosal: A Survey of End-System Optimizations for High-Speed Networks. ACM Comput. Surv. 51(3): 54:1-54:36 (2018)
- Balasubramanian, Sowmya, Dipak Ghosal, Kamala Narayanan Balasubramanian Sharath, Eric Pouyoul, Alex Sim, Kesheng Wu, and Brian Tierney. Auto-Tuned Publisher in a Pub/Sub System: Design and Performance Evaluation. In 2018 IEEE International Conference on Autonomic Computing (ICAC), pp. 21-30. IEEE, 2018.
- Alali, Fatma, Nathan Hanford, Eric Pouyoul, Raj Kettimuthu, Mariam Kiran, Ben Mack-Crane, Brian Tierney, Yatish Kumar, and Dipak Ghosal. Calibers: A bandwidth calendaring paradigm for science workflows. Future Generation Computer Systems 89 (2018): 736-745.
- Nathan Hanford, Vishal Ahuja, Matthew Farrens, Dipak Ghosal, Mehmet Balman, Eric Pouyoul, Brian Tierney, Improving network performance on multicore systems: Impact of core affinities on high throughput flows, Future Generation Computer Systems 56 (2016) 277–283.
- Hanford, Nathan, Brian Tierney, and Dipak Ghosal. Optimizing data transfer nodes using packet pacing. In Proceedings of the Second Workshop on Innovating the Network for Data-Intensive Science, p. 4. ACM, 2015.
- Nathan Hanford, Vishal Ahuja, Matthew Farrens, Dipak Ghosal, Mehmet Balman, Eric Pouyoul, Brian Tierney, Analysis of the Effect of Core Affinity on High-Throughput Flows, 2014 Fourth International Workshop on Network-Aware Data Management (NDM), Nov. 2014
- Dylan Wang, Abhinav Bhatele, Dipak Ghosal, Performance Variability due to Job Placement on Edison, Poster presented in SC14, The International Conference for High Performance Computing, Networking, Storage and Analysis, New Orleans, LA, November 2014.
- Nathan Hanford, Vishal Ahuja, Matthew Farrens, and Dipak Ghosal, Mehmet Balman, Eric Pouyoul, and Brian Tierney, Impact of the End-System and Affinities on the Throughput of High-Speed Flows, Proceedings of The Tenth ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS) ANCS’14, October 20–21, 2014, Los Angeles, CA, USA.
- Nathan Hanford, Vishal Ahuja, Mehmet Balman, Matthew K. Farrens, Dipak Ghosal, Eric Pouyoul, and Brian Tierney. 2013. Characterizing the impact of end-system affinities on the end-to-end performance of high-speed flows. In Proceedings of the Third International Workshop on Network-Aware Data Management (NDM '13). ACM, New York, NY, USA.
- Dipak Ghosal, Optimizing Transport of Big Data over Dedicated Networks, Invited Talk, (NDM 2012 @ SC'12) The 2nd International Workshop on Network-aware Data Management, November 11th, 2012, Salt Lake City, Utah.
- Vishal Ahuja, Matthew Farrens, Dipak Ghosal, Cache-aware affinitization on commodity multicores for high-speed network flows, ANCS '12: Proceedings of the eighth ACM/IEEE symposium on Architectures for networking and communications systems, October 2012.
- Vishal Ahuja, Dipak Ghosal, Matthew Farrens, Minimizing the Data Transfer Time Using Multicore End-System Aware Flow Bifurcation, CCGRID '12: Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012), May 2102.
- V. Ahuja, A. Banerjee, M. Farrens, G. Serazzi, D. Ghosal, Introspective End-system Modeling to Optimize the Transfer Time of Rate Based Protocols, In Proceedings of the 20th International ACM Symposium on High Performance Parallel and Distributed Computing, San Jose, CA, June 8-11, 2011.
- V. Ahuja, R. Archibald, A. Banerjee, M. Farrens, and D. Ghosal, Active End-System Analysis to Estimate the Network I/O Bottleneck Rate, Workshop on The Influence of I/O on Microprocessor Architecture (IOM-2009) Raleigh, North Carolina, February 15, 2009
- A. Banerjee, D. Ghosal, B. Mukherjee, and W. Feng, Algorithms for Integrated Routing and Scheduling for Aggregating Data from Distributed Resources on a Lambda Grid, IEEE Transaction on Parallel and Distributed Systems, 2008, vol. 19, Issue 1, pp. 24-34.
- A. Banerjee, D. Ghosal, and B. Mukherjee, Modeling and Analysis to Estimate the End-System Performance Bottleneck Rate for High-Speed Data Transfer, Fifth International Workshop on Protocols for Fast Long-Distance Networks (PFLDNet) 2007, Los Angeles.
- A. Banerjee, W. Feng, B. Mukherjee, and D. Ghosal, RAPID: An End-System Aware Protocol for Intelligent Data-Transfer over Lambda-Grids, in the Proceedings of the IEEE/ACM International Parallel and Distributed Processing Symposium (IPDPS 2006), Rhode Island, Greece, April 2006.
- A. Banerjee, W. Feng, B. Mukherjee, and D. Ghosal, End-system Performance Aware Transport over Optical Circuit-Switched Connections, IEEE INFOCOM High-Speed Networking Workshop: The Terabits Challenge, April 2006.
- N. Rao, Q. Wu, S. Carter, W. Wing, A. Banerjee, D. Ghosal, and B. Mukherjee, Control Plane for Advance Bandwidth Scheduling in Ultra High-Speed Networks, IEEE INFOCOM High-Speed Networking Workshop: The Terabits Challenge, April 2006.
- A. Banerjee, W.-C. Feng, B. Mukherjee, and D. Ghosal, Routing and Scheduling Large File Transfers over Lambda Grids, Third International Workshop on Protocols for Fast Long-Distance Networks (PFLDNet) 2005, February 3,4 2005,Lyon France.
- INTME - The modified RBUDP protocol based on INTIME, FBM and CAAD.