A Lightweight Execution Framework for Massive Independent Tasks
Li Hui, Shenzhen Graduate School of Peking University
Yu Huashan, Institute of Network Computing and Distributed System, Peking University
Li Xiaoming, State Key Laboratory of Advanced Optical Communication & Network, Peking University
Location and Time: November 17th, 2008, Room 11AB, 10:30AM
Abstract
This paper presents a lightweight execution framework for executing massive independent tasks efficiently on computational grids. It dynamically partitions a set of tasks of different granularities and dispatches tasks onto distributed computational resources concurrently. Three optimization strategies have been devised to improve computation efficiency and resource utilization. One strategy is to pack up to thousands of tasks into one request. Another is to share the effort in resource discovery and allocation among requests by separating resource allocations from request submissions. The third strategy is to pack variable numbers of tasks into different requests, where the task number is a function of the destination resource’s computability. This framework has been implemented in Gracie, a computational grid software platform developed by Peking University, and used for executing bioinformatics tasks. We describe its architecture, evaluate its strategies, and compare its performance with GRAM. Analyzing the experiment results, we found that Gracie outperforms GRAM significantly for execution of sets of small tasks, which is aligned with the intuitive advantage of our approaches built in Gracie.