1. End-System Aware Networking

SAANet - System and Architecture Aware Networking

This research builds on INTIME and focuses on  high speed networking using commodity multi-core systems. Network speeds are continuing to climb. 10~Gbps Network Interface Cards (NICs) are common now, 40~Gbps NICs are widely available, and 100~Gbps NICs will be available soon. However, in practice, little of this bandwidth capability is being actively utilized by applications. One of the major reasons for this is the relatively large protocol processing overhead of TCP/IP at these speeds. Increasing clock speeds can no longer be relied upon to ameliorate the problem of protocol processing. Meanwhile, TCP/IP remains the most widely-adopted protocol stack used by distributed applications and supported by widely-available hardware.

Our previous research has been dedicated to attaining high-throughput TCP/IP-based networking from commodity hardware by intelligently exploiting the parallelism of the end-systems using a concept we refer to as affinity. Affinity, or core binding, has to do with deciding which core a particular program or process is executed on in a multicore system. We have characterized the performance and efficiency of the TCP/IP receive process across multiple affinity configurations within modern and widely-deployed commodity end-systems. Through several publications, the results of our research are well-positioned to influence the design of applications, NIC driver and hardware design, and high-speed distributed systems - both directly through collaborations with application developers and network operators, and indirectly through industrial adoptions of affinity best practices.

Our current and future research builds upon this expertise, with the aim to aid in efficiently leveraging commodity end-system hardware by broadening our reach to newly-available technologies, protocols, and platforms. Furthermore, we plan to utilize efficient and highly-effective statistical methods to manage end-to-end performance of high speed flows to the point of delivering predictably efficient performance by monitoring parameters and modifying end-system variables at all times during network activity. Our process of careful characterization of current technology, followed by statistical analysis, and finally, middleware tool development affords us the maximum impact on shaping best practices while minimizing our impact on distributed application development processes.

In this research proposal, we intend to further characterize the end-system bottlenecks that arise during data transfers required in different distributed scientific and business applications. What we learn will drive the development of introspective end-system aware models, in order to auto-tune data transfers. This tuning will consider both latency and throughput requirements of the applications. We will develop flow striping methods that exploit multicore end-systems and adapt to the end-system bottlenecks. This will require addressing many new issues, such as assigning flows to cores while taking into account various (application, cache, and interrupt) affinities. Additionally, the underlying topology of the cache (inclusive vs exclusive), the memory organization (NUMA vs UMA), and the heterogeneity of the cores must also be considered when controlling the end-to-end flows.  We will investigate memory-mapped network channels, such as RDMA over Converged Ethernet (RoCE), for data transfers over wide-area networks. Towards this end, we will design and implement memory management, message synchronization and end-to-end flow control to enable remote messaging for different types of network flows. From the end-system architectural perspective, we will propose and study cache architectures that can significantly improve the network I/O performance. The methods developed in this proposed research will be prototyped and tested in ESNet 100Gbps testbed and the UC Davis Research Network.


This National Science Foundation (NSF) project investigates optimizing network transfers using introspective modeling of end-systems. The bottleneck for the transferring data at very high speeds often turns out to be the end-system performance. In the absence of definitive knowledge about the workload at the receiving end-system, the sender's transmission rate often overshoots the critical bottleneck rate of the receiver. This typically results in oscillations between the extremes and poor performance. To optimize the performance of the transport protocols and achieve the important flow control functionality, it is important to estimate the receiving end-system effective bottleneck rate. In this project we will use modeling and active analysis of the end-system to estimate this rate.  We will develop queueing network models for representing the different multi-core and multiprocessor end-systems running different types of workloads. We will develop a software tool to be integrated with existing transport protocols. We will carry out experimental analysis for different types of end-systems with different configurations and workloads. We will apply and extend methods that have been proposed to address the limitations of queueing network models for performance analysis of computer systems with bursty workloads and correlated service times.  The software tool will be made available to the research community to analyze and optimize distributed applications and systems. The research project will provide a framework to train graduate and undergraduate students in both analytical and experimental methods, and develop knowledge and intuition about next generation computer systems and distributed applications.

Project Title and Duration: [0917315/08-004661] Estimating the End-system Network I/O Bottleneck Rate to Optimize Transport Layer Performance. September 2009 - August 2013. Funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5).


Sambit Shukla (PhD Student) 
Ross Gegan (PhD Student) 
Nathan Handforf (PhD Student) 
Saeedeh Komijani ((PhD Student) 
Vishal Ahuja (Research Associate) 
Rennie Archibald (Alumni, AT&T Labs) 
Amitabha Banerjee (Alumni, VMWare)
Matt Farrens (Professor, UCDavis) 
Dipak Ghosal
Giuseppe Serazzi (Professor Emeritus, Politecnico di Milano) 
Mehmet Balman (VMWare) 
Brian Tierney (Staff Scientist and Group Leader, Lawrence Berkeley Labs) 
Eric Pouyal (Staff Scientist, Lawrence Berkeley Labs) 


  1. Nathan Hanford, Vishal Ahuja, Matthew Farrens, Dipak Ghosal, Mehmet Balman, Eric Pouyoul, Brian Tierney, "Analysis of the Effect of Core Affinity on High-Throughput Flows," 2014 Fourth International Workshop on Network-Aware Data Management (NDM), Nov. 2014
  2. Dylan Wang, Abhinav Bhatele, Dipak Ghosal, Performance Variability due to Job Placement on Edison, Poster presented in SC14, The International Conference for High Performance Computing, Networking, Storage and Analysis, New Orleans, LA, November 2014.
  3. Nathan Hanford, Vishal Ahuja, Matthew Farrens, and Dipak Ghosal, Mehmet Balman, Eric Pouyoul, and Brian Tierney, " Impact of the End-System and Affinities on the Throughput of High-Speed Flows, Proceedings of The Tenth ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS) ANCS’14, October 20–21, 2014, Los Angeles, CA, USA.
  4. Nathan Hanford, Vishal Ahuja, Mehmet Balman, Matthew K. Farrens, Dipak Ghosal, Eric Pouyoul, and Brian Tierney. 2013. Characterizing the impact of end-system affinities on the end-to-end performance of high-speed flows. In Proceedings of the Third International Workshop on Network-Aware Data Management (NDM '13). ACM, New York, NY, USA.
  5. Dipak Ghosal,Optimizing Transport of Big Data over Dedicated Networks," Invited Talk, (NDM 2012 @ SC'12) The 2nd International Workshop on Network-aware Data Management, November 11th, 2012, Salt Lake City, Utah. 
  6. Vishal Ahuja, Matthew Farrens, Dipak Ghosal, Cache-aware affinitization on commodity multicores for high-speed network flows, ANCS '12: Proceedings of the eighth ACM/IEEE symposium on Architectures for networking and communications systems, October 2012.
  7. Vishal Ahuja, Dipak Ghosal, Matthew Farrens, Minimizing the Data Transfer Time Using Multicore End-System Aware Flow Bifurcation,” CCGRID '12: Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012), May 2102.
  8.  V. Ahuja, A. Banerjee, M. Farrens, G. Serazzi, D. Ghosal, "Introspective End-system Modeling to Optimize the Transfer Time of Rate Based Protocols", In Proceedings of the 20th International ACM Symposium on High Performance Parallel and Distributed Computing, San Jose, CA, June 8-11, 2011. 
  9. V. Ahuja,  R. Archibald, A. Banerjee, M. Farrens, and D. Ghosal, "Active End-System Analysis to Estimate the Network I/O Bottleneck Rate,"  Workshop on The Influence of I/O on Microprocessor Architecture (IOM-2009) Raleigh, North Carolina, February 15, 2009
  10. A. Banerjee, D. Ghosal, B. Mukherjee, and W. Feng, Algorithms for Integrated Routing and Scheduling for Aggregating Data from Distributed Resources on a Lambda Grid"  IEEE Transaction on Parallel and Distributed Systems, 2008, vol. 19, Issue 1, pp. 24-34.
  11. A. Banerjee, D. Ghosal, and B. Mukherjee, Modeling and Analysis to Estimate the End-System Performance Bottleneck Rate for High-Speed Data Transfer, Fifth International Workshop on Protocols for Fast Long-Distance Networks (PFLDNet) 2007, Los Angeles. 
  12. A. Banerjee, W. Feng, B. Mukherjee, and D. Ghosal, RAPID: An End-System Aware Protocol for Intelligent Data-Transfer over Lambda-Grids, in the Proceedings of the IEEE/ACM International Parallel and Distributed Processing Symposium (IPDPS 2006), Rhode Island, Greece, April 2006.
  13. A. Banerjee, W. Feng, B. Mukherjee, and D. Ghosal, "End-system Performance Aware Transport over Optical Circuit-Switched Connections," IEEE INFOCOM High-Speed Networking Workshop: The Terabits Challenge, April 2006.
  14. N. Rao, Q. Wu, S. Carter, W. Wing, A. Banerjee, D. Ghosal, and B. Mukherjee, "Control Plane for Advance Bandwidth Scheduling in Ultra High-Speed Networks," IEEE INFOCOM High-Speed Networking Workshop: The Terabits Challenge, April 2006.
  15. A. Banerjee, W.-C. Feng, B. Mukherjee, and D. Ghosal, Routing and Scheduling Large File Transfers over Lambda Grids, Third International Workshop on Protocols for Fast Long-Distance Networks (PFLDNet) 2005, February 3,4 2005,Lyon France.

Thesis and Dissertations

  1. Vishal Ahuja, End-System Optimizations for High-Speed Networks, 2013
  2. Amitabha Bannerjee, Improving data transport over high-speed networks, 2007


  1. INTME - The modified RBUDP protocol based on INTIME, FBM and CAAD.