Incubator/Falkon
From Globus
Contents |
FALKON: A FAST AND LIGHT-WEIGHT TASK EXECUTION FRAMEWORK
Falkon aims to enable the rapid and efficient execution of many tasks on large compute clusters. Falkon integrates (1) multi-level scheduling to separate resource acquisition from task dispatch, and (2) a streamlined dispatcher. Falkon’s integration of multi-level scheduling and streamlined dispatchers delivers performance not provided by any other system. Microbenchmarks show that Falkon throughput (ranging from 100s to 1000s of tasks/sec) and scalability (to 54K executors and 2M queued tasks) are several orders of magnitude better than other systems used in production Grids. Large-scale astronomy and medical applications executed under Falkon by the Swift parallel programming system achieve up to 90% reduction in end-to-end run time, relative to versions that execute tasks via separate scheduler submissions.
Goals
- Reducing task dispatch time by using a streamlined dispatcher that eliminates support for features such as multiple queues, priorities, accounting, etc.
- Using an adaptive provisioner to acquire and/or release resources as application demand varies.
- Improve application performance and scalability through data diffusion and data-aware scheduling to leverage the co-located computational and storage resources offloading the shared file systems I/O with local disk I/O.
Project Branches
- Efficient Task Dispatch and Execution
- Dynamic Resource Provisioning
- Enabling Data Diffusion (Data Caching, Data Management, and Data-Aware Scheduling) in Falkon
- Running Swift Applications over Falkon
- Moving from Batch-Scheduled Grids to Economic Driven Resources: Falkon and the Amazon Elastic Computing Clound (EC2)
- Enabling Serial Job Execution on the IBM BlueGene Supercomputer via Falkon
- AstroPortal Image Stacking Service
NEWS
- "Falkon Brochure", November 2007.
- Presentation on Falkon at IEEE/ACM Supercomputing 2007, November 15th, 2007 at 11:30AM.
- Presentation on Falkon at IEEE/ACM Supercomputing 2007 at the ANL MCS booth, November 14th, 2007 at 5:00PM.
Documents
Publications
- Yong Zhao, Mihael Hategan, Ioan Raicu, Mike Wilde, Ian Foster. “Swift: Realizing Fast, Reliable, Large Scale Scientific Computation”, under review.
- Ioan Raicu, Yong Zhao, Ian Foster, Alex Szalay. “A Data Diffusion Approach to Large Scale Scientific Exploration”, Microsoft Research eScience Workshop 2007.
- Ioan Raicu, Yong Zhao, Catalin Dumitrescu, Ian Foster, Mike Wilde. “Falkon: a Fast and Light-weight tasK executiON framework”, IEEE/ACM SuperComputing 2007.
- Ioan Raicu, Catalin Dumitrescu, Ian Foster. Dynamic Resource Provisioning in Grid Environments, TeraGrid Conference 2007.
- Yong Zhao, Mihael Hategan, Ben Clifford, Ian Foster, Gregor von Laszewski, Ioan Raicu, Tiberiu Stef-Praun, Mike Wilde. “Swift: Fast, Reliable, Loosely Coupled Parallel Computation”, IEEE Workshop on Scientific Workflows 2007.
- Alex Szalay, Julian Bunn, Jim Gray, Ian Foster, Ioan Raicu. “The Importance of Data Locality in Distributed Computing Applications”, NSF Workflow Workshop 2006.
Presentations
- "Towards Urgency Solutions in the Globus Toolkit", April 27th, 2007.
- "Falkon: a Fast and Light-weight tasK executiON framework for Grid Environments", IEEE/ACM SuperComputing 2007, November 15th, 2007.
- "Accelerating Large Scale Scientific Exploration with Falkon", IEEE/ACM SuperComputing 2007, Argonne National Laboratory Booth, November 14th, 2007.
- "A Data Diffusion Approach to Large Scale Scientific Exploration", 2007 Microsoft eScience Workshop at RENCI, October 21st, 2007.
- "A Data Diffusion Approach to Large Scale Scientific Exploration", University of Chicago, CS Department, DSL Seminar, October 24th, 2007.
- "Falkon: a Fast and Light-weight tasK executiON framework for Grid Environments", DSL Workshop 2007, University of Chicago, April 30th, 2007.
- "Storage and Compute Resource Management via DYRE, 3DcacheGrid, and CompuStore", University of Chicago, CS Department, DSL Seminar, November 1st, 2006.
Other Documents
- "Falkon Brochure", November 2007.
- Ioan Raicu, Ian Foster, "Harnessing Grid Resources with Data-Centric Task Farms", Technical Report, Department of Computer Science, University of Chicago, November 2007.
- Ioan Raicu, Yong Zhao, Ian Foster, Alex Szalay. “Accelerating Large Scale Scientific Exploration through Data Diffusion”, Technical Report, Department of Computer Science, University of Chicago, November 2007.
- Ioan Raicu, Ian Foster. "Harnessing Grid Resources to Enable the Dynamic Analysis of Large Astronomy Datasets: Year 1 Status and Year 2 Proposal", NASA GSRP Year 1 Progress Report and Year 2 Proposal, Ames Research Center, NASA, February 2007.
- Ioan Raicu, Ian Foster. "Harnessing Grid Resources to Enable the Dynamic Analysis of Large Astronomy Datasets", NASA GSRP Proposal, Ames Research Center, NASA, February 2006.
Mailing Lists
| Developer discussion | archive | subscribe | unsubscribe |
| User discussion | archive | subscribe | unsubscribe |
| Commit notifications | archive | subscribe | unsubscribe |
How to subscribe
How to unsubscribe
Search the email archives
Bugs Reporting
FAQ
Source Code
Code
To download the entite Falkon source tree, type:
svn co https://svn.globus.org/repos/falkon
If you don't have SVN and would like to download the source tree archive (revision 3), click here.
Prerequisites
- Java 1.4+
- SVN: only needed for source control
- Apache Ant: only needed to compile
- Ploticus: included; only needed to generate graphs
- Globus Toolkit 4: included
- gcc or g++: only needed for a specialized C-based component that was build to run on the IBM BlueGene
Instructions
Committers
If you would like to become a committer, guidelines are here:
- Ioan Raicu, Computer Science Dept. The University of Chicago
- Yong Zhao, Microsoft
- Catalin Dumitrescu, Computer Science Dept. The University of Chicago
- Ian Foster, Math and Computer Science Div. Argonne National Laboratory & Computer Science Dept. The University of Chicago
- Mike Wilde, Computation Institute, University of Chicago & Argonne National Laboratory
- Ben Clifford, Computational Institute, The University of Chicago
- Mihael Hategan, Computational Institute, The University of Chicago
- Zhao Zhang, Computer Science Dept. The University of Chicago
Contributors
The Falkon project gratefully acknowledges the following contributions:
- Alex Szalay, John Hopkins University
- NASA Ames Research Center GSRP Grant Number NNA06CB89H
- U.S. Dept. of Energy, Office of Advanced Scientific Computing Research, Office of Science, Mathematical, Information, and Computational Sciences Division Contract DE-AC02-06CH11357
Miscellaneous
Status
Newly accepted Incubator Project 11/06/2007, as defined by the Incubator Process Guidelines found at http://dev.globus.org/wiki/Incubator/Incubator_Process .
Roadmap
We plan to have an official incremental code snapshots every few months. The current version is v0.9, and we plan to release v1.0 in the next few months. The transition from 0.9 to 1.0 will mostly address ease of installation/use, reliability, and robustness; we are not planning to add any new major features to include in v1.0. Check back soon for more details on the release schedule.
Policies
The Falkon project adheres to the following guidelines: Globus Alliance Project Guidelines.
