CTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> DataSys: Projects

DataSys: Data-Intensive Distributed Systems LaboratoryData-Intensive Distributed Systems Laboratory

Illinois Institute of Technology
Department of Computer Science

MATRIX: MAny-Task computing execution fabRIc at eXascale 

Proposed Work: MATRIX will be a distributed data-aware execution fabric supporting both high-performance computing (HPC) and many-task computing (MTC) workloads. The execution fabric is fault tolerant by having all compute nodes participate in the job submission and handling process; work stealing is used to achieve efficient distributed load balancing. The fabric guarantees job execution and dependencies, and relies on an underlying scalable distributed storage system for inter-process communication (e.g. FusionFS). Data-aware scheduling maximizes data locality by scheduling computational tasks close to the data. Computations are overlapped with I/O to reduce wasted resources and hide latencies. The fabric is elastic, allowing it to grow and shrink in resource usage based on the application demand. The fabric also support compact task representation to alleviate task submission bottlenecks for common patterns (e.g. “for each x do y”). The execution fabric is integrated with several other projects, including FusionFS(a distributed file system), D^3 (direct distributed data-structure), and Swift (parallel programming system). The work will be evaluated with many applications (e.g. bioinformatics, medicine, pharmaceuticals, astronomy, physics, climate modeling, economics, and analytics) through the Swift project collaboration on the largest high-end computing (HEC) systems.

Current State: MATRIX utilizes an adaptive work stealing algorithm for distributed load balancing and distributed hash tables (ZHT) for managing task metadata. MATRIX supports both high-performance computing (HPC) and many-task computing (MTC) workloads, as well as task dependencies efficiently supporting the execution of complex large-scale workflows. We have evaluated it using synthetic workloads up to 4K-cores on an IBM Blue Gene/P supercomputer, and have shown high efficiency rates (e.g. 85%+) are possible with certain workloads with task granularities as low as 64ms. MATRIX has shown throughput rates as high as 13K tasks/sec at 4K-core scales (one to two orders of magnitude higher than existing centralized systems). We study the performance of MATRIX in depth, including understanding the network traffic generated by the work stealing algorithm. We also use simulations to explore the feasibility of adaptive work stealing up to 1M-node scales (the expected scales at exascale).