Data-Intensive Distributed Systems LaboratoryFalkon: Fast and Light-weight tasK executiON framework
Falkon enables the rapid and efficient execution of many independent jobs on large compute clusters. Falkon combines three techniques to achieve this goal: (1) multi-level scheduling to enable dynamic resource provisioning; (2) a streamlined task dispatcher able to achieve order-of-magnitude higher task dispatch rates than conventional schedulers; and (3) performs data caching and uses a data-aware scheduler to co-locate computational and storage resources.
-
Period: 12/2006 - Present
-
Languages: Java (20K lines), C (1K lines), Bash scripts (1K lines)
-
Features: Web Services, TCP, UDP, Notifications, Threads, JNI, Queues, Maps, Caches, GC tunning
-
Technologies: Globus, GRAM, GridFTP, Condor, PBS, SGE, Cobalt
-
OS: Linux, ZeptOS
-
Testbeds: TeraGrid, SiCortex SC5832, IBM BlueGene/L, IBM Blue Gene/P, Sun Constellation (Sun Blade x6420), Cray XT5, Amazon EC2
-
Scalability: 160K processors, millions of queued or tasks, billions of executed tasks
-
Performance: 15K+ tasks/sec throughputs, 170Gb/s I/O rates
-
Funding: NASA, NSF/CRA