MRNet

News and Recent Developments

  • March 2014 - Version 4.1.0 has been released. See below for details.
  • April 2012 - Version 4.0.0 has been released.
  • June 2011 - Version 3.1.0 has been released.
  • A recent paper on using MRNet on Cray XT systems: Michael J. Brim, Luiz DeRose, Barton P. Miller, Ramya Olichandran, and Philip C. Roth, "MRNet: A Scalable Infrastructure for the Development of Parallel Tools and Applications", Cray User Group 2010, Edinburgh, Scotland, May 2010. [PDF]
  • A recent paper on lightweight TBON infrastructure: Emily R. Jacobson, Michael J. Brim, and Barton P. Miller, "A Lightweight Library for Building Scalable Tools", Para 2010: State of the Art in Scientific and Parallel Computing, Reykjavik, Iceland, June 2010. [PDF]
  • A recent paper on scalable TBON reliability: Dorian C. Arnold and Barton P. Miller, "Scalable Failure Recovery for High-performance Data Aggregation" , International Parallel and Distributed Processing Symposium (IPDPS), Atlanta, April 2010. [PDF]

Overview

MRNet is a software overlay network that provides efficient multicast and reduction communications for parallel and distributed tools and systems. MRNet uses a tree of processes between the tool's front-end and back-ends to improve group communication performance. These internal processes are also used to distribute many important tool activities, reducing data analysis time and keeping tool front-end loads manageable.

MRNet-based tool components communicate across logical channels called streams. At MRNet internal processes, filters are bound to these streams to synchronize and aggregate dataflows. Using filters, MRNet can efficiently compute averages, sums, and other more complex aggregations and analyses on tool data. MRNet also supports facilities that allow tool developers dynamically load new tool-specific filters into the system.

Key Features:

  • Flexible Organization: MRNet process tree organization is specified in a configuration file that can specify common network layouts like k-ary and k-nomial trees, or custom layouts tailored to the system(s) running the tool.
  • Scalable, Flexible Data Aggregation: MRNet's built-in filters provide efficient computation of averages, sums, concatenation, and other common data reductions. Custom filters can be loaded dynamically into the network to perform tool-specific aggregation operations.
  • High-bandwidth Communication: MRNet transfers data within the tool system using an efficient, packed binary representation. Zero-copy data paths are used whenever possible to reduce the cost of transferring data through internal processes.
  • Scalable Multicast: MRNet supports efficient message multicast to reduce the cost of issuing control requests from the tool front-end to its back-ends.
  • Multiple Concurrent Data Channels: MRNet supports multiple logical streams of data between tool components. Data aggregation and message multicast takes place within the context of a data stream, and multiple operations (both upward and downward) can be active simultaneously.
  • Open Source Licensing.

You can download MRNet here.