In many scientific workflows data is generated at remote locations and must be transferred to the distributed HPC nodes over high-speed networks. Increasingly many of these scientific workflows require processing to be completed within a deadline, which, in turn, imposes deadline for the network data transfer. This research focuses on science networks and private WANs in which the sources of traffic (and hence their nature) are well understood although the volume may change. The goal of this proposal is to design a system that consists of 1) a centralized network controller for scheduling (assigning rates to flows) and routing flows with time constraint and 2) an end-system traffic controller that can maintain the rates assigned by the controller with low variance. The goal of the network controller is to determine schedules that are provably correct implying that flows meet their deadlines and predictable implying that the completion time of the data transfer can be tightly bounded despite changes in the request and/or the network condition. Predictable data transfers along with intelligent routing and end-system traffic control will also result in high network utilization. The design and implementation of the network controller will be enabled by a SDN-based softwarized network and high-precision network telemetry.
In this research, we will adopt a Reinforcement Learning (RL) based approach to implement the network controller. We will extend our preliminary study and investigate how Deep Deterministic Policy Gradient (DDPG) approach can be adapted to design a scalable network controller. In order to guarantee predictability, we will formulate the scheduling problem as constraint satisfaction problem and use Satisfiability Modulo Theory (SMT) solvers to determine correct and predictable schedules. We will introduce the notion of $\delta$-satisfiability to capture the notion of predictability, and enable the network controller to synthesize a schedule that meets the deadline in spite of changes in the network state (e.g., utilization). To achieve the goals of the network scheduler we will design and implement a model predictive based (MPC)-based end-system traffic controller that can maintain smooth transmission rates assigned by network scheduler.
This research builds on INTIME and focuses on high speed networking using commodity multi-core systems. Network speeds are continuing to climb. 10~Gbps Network Interface Cards (NICs) are common now, 40~Gbps NICs are widely available, and 100~Gbps NICs will be available soon. However, in practice, little of this bandwidth capability is being actively utilized by applications. One of the major reasons for this is the relatively large protocol processing overhead of TCP/IP at these speeds. Increasing clock speeds can no longer be relied upon to ameliorate the problem of protocol processing. Meanwhile, TCP/IP remains the most widely-adopted protocol stack used by distributed applications and supported by widely-available hardware.
Our previous research has been dedicated to attaining high-throughput TCP/IP-based networking from commodity hardware by intelligently exploiting the parallelism of the end-systems using a concept we refer to as affinity. Affinity, or core binding, has to do with deciding which core a particular program or process is executed on in a multicore system. We have characterized the performance and efficiency of the TCP/IP receive process across multiple affinity configurations within modern and widely-deployed commodity end-systems. Through several publications, the results of our research are well-positioned to influence the design of applications, NIC driver and hardware design, and high-speed distributed systems - both directly through collaborations with application developers and network operators, and indirectly through industrial adoptions of affinity best practices.
Our current and future research builds upon this expertise, with the aim to aid in efficiently leveraging commodity end-system hardware by broadening our reach to newly-available technologies, protocols, and platforms. Furthermore, we plan to utilize efficient and highly-effective statistical methods to manage end-to-end performance of high speed flows to the point of delivering predictably efficient performance by monitoring parameters and modifying end-system variables at all times during network activity. Our process of careful characterization of current technology, followed by statistical analysis, and finally, middleware tool development affords us the maximum impact on shaping best practices while minimizing our impact on distributed application development processes.
In this research proposal, we intend to further characterize the end-system bottlenecks that arise during data transfers required in different distributed scientific and business applications. What we learn will drive the development of introspective end-system aware models, in order to auto-tune data transfers. This tuning will consider both latency and throughput requirements of the applications. We will develop flow striping methods that exploit multicore end-systems and adapt to the end-system bottlenecks. This will require addressing many new issues, such as assigning flows to cores while taking into account various (application, cache, and interrupt) affinities. Additionally, the underlying topology of the cache (inclusive vs exclusive), the memory organization (NUMA vs UMA), and the heterogeneity of the cores must also be considered when controlling the end-to-end flows. We will investigate memory-mapped network channels, such as RDMA over Converged Ethernet (RoCE), for data transfers over wide-area networks. Towards this end, we will design and implement memory management, message synchronization and end-to-end flow control to enable remote messaging for different types of network flows. From the end-system architectural perspective, we will propose and study cache architectures that can significantly improve the network I/O performance. The methods developed in this proposed research will be prototyped and tested in ESNet 100Gbps testbed and the UC Davis Research Network.
Project Title and Duration: [NSF 1528087] NeTS: Small: Addressing End-system Bottlenecks in High-speed Networks. October 2015 - September 2019.
This National Science Foundation (NSF) project investigates optimizing network transfers using introspective modeling of end-systems. The bottleneck for the transferring data at very high speeds often turns out to be the end-system performance. In the absence of definitive knowledge about the workload at the receiving end-system, the sender's transmission rate often overshoots the critical bottleneck rate of the receiver. This typically results in oscillations between the extremes and poor performance. To optimize the performance of the transport protocols and achieve the important flow control functionality, it is important to estimate the receiving end-system effective bottleneck rate. In this project we will use modeling and active analysis of the end-system to estimate this rate. We will develop queueing network models for representing the different multi-core and multiprocessor end-systems running different types of workloads. We will develop a software tool to be integrated with existing transport protocols. We will carry out experimental analysis for different types of end-systems with different configurations and workloads. We will apply and extend methods that have been proposed to address the limitations of queueing network models for performance analysis of computer systems with bursty workloads and correlated service times. The software tool will be made available to the research community to analyze and optimize distributed applications and systems. The research project will provide a framework to train graduate and undergraduate students in both analytical and experimental methods, and develop knowledge and intuition about next generation computer systems and distributed applications.
*Current participants are bolded
Sambit Shukla (PhD Student)
Taran Lynn (PhD Student)
Nathan Handforf (PhD, 2018)
Jian Wu (Postdoc, 2018)
Joseph Mcgee (MS Student)
Harshvardhan Chauhan (MS, 2018)
Dylan Wang (Undergraduate Researcher, Microsoft, 2015)
Vishal Ahuja (PhD, 2014)
Rennie Archibald (PhD, 2013 )
Amitabha Banerjee (PhD, 2012)
Aditya Thakur (Professor, UCDavis)
Matt Farrens (Professor, UCDavis)
Dipak Ghosal (Professor, UCDavis)
Alex Sim (Staff Scientist, Lawrence Berkeley Labs)
Kesheng (John) Wu (Group Lead, Scientific Data Management, Lawrence Berkeley Labs)
Giuseppe Serazzi (Professor Emeritus, Politecnico di Milano)
Mehmet Balman (VMWare)
Brian Tierney (Staff Scientist and Group Leader, Lawrence Berkeley Labs)
Eric Pouyal (Staff Scientist, Lawrence Berkeley Labs)