Incubator/HOC-SA

From Globus

HOC-SA is an Incubator project. HOC-SA stands for Higher-Order Components-Service Architecture. Higher-Order Components (HOCs) have the aim of simplifying Grid application development. Our objective is to provide Globus users, who do not want to know about all the details of the Globus middleware (GRAM RSL documents, Web services and resource configuration etc.), with HOCs that provide a higher-level interface to the Grid than the core Globus Toolkit. Thus, typical HOC users might be non-computer scientist, but rather applied scientists, e.g., physicists or biologists, who want to use Grid technology without becoming experts in setting it up for their purposes.

HOCs are implementations of recurring schemata of parallel processing which are deployed to Grid servers and pre-packaged with the required configuration files making them accessible and customizable via Web services.

Each HOC implements a generic pattern of parallel behavior with a specific communication structure. A HOC can be customized for a particular application by providing it with arguments which may be both data and application-specific code.

The HOC-SA represents a specialized Web service-based Globus Resource Allocation Manager implementation for the Globus Toolkit 4 (WS-GRAM).

Like WS-GRAM, HOCs make use of Web services for submitting code and executing it on geographically dispersed Grid resources.

In contrast to WS-GRAM, the submitted code is not necessarily a fully self-contained program including all the coordination and communication code for its parallel execution on the Grid. HOC-SA users can submit code pieces, describing only application-specific tasks, to a HOC, which is a ready-made software component handling the execution of the submitted code within a high-level parallel processing schema.

The HOC-SA provides users with the required tools for working with HOCs. Concretely, HOC-SA consists of the Remote Code Loader and the Code Service enabling the exchange of code pieces among Web service providers and consumers. Moreover, the HOC-SA includes a client API, example applications, a tutorial and basic HOCs, which can be extended into more complex components, by HOC-SA users.

HOC-SA Project Metadata

Status

The status of HOC-SA is: Newly accepted project (November 2, 2006), as defined by the Incubator Process Guidelines found at http://dev.globus.org/wiki/Incubator/Incubator_Process .

Roadmap

Development roadmap for HOC-SA

Committers

If you would like to become a committer, guidelines are [here].

Jan Duennweber
Philipp Luedeking
Eduardo Argollo
Catalin Dumitrescu
Sergei Gorlatch
Michael Classen
Philipp Classen
Juergen Voerding
Johannes Tomasoni

Mailing Lists

How to subscribe
How to unsubscribe
Search the email archives

Installation

In the following, we describe how to install HOC-SA in 3 steps.
(more extensive installation instructions, which include the neccessary commands to install the required tools, such as Java, Globus and Ant, are available on the HOC Web Site)

(1) Set your environment variables JAVA_HOME, ANT_HOME, GLOBUS_LOCATION, OGSADAI_HOME and JALIGNER_HOME to point to the respective installation directories.

(2) Install OGSA-DAI and set up your database management system to work with the HOC-SA Code Service: the most comfortable way to do this is running the installService.xml and manageDB.xml ant scripts inside the scripts directory by issuing the following commands:
ant -f installService
ant -f manageDB CreateDatabase CreateTable
hint: if this script fails, check if the default settings in file scripts/database.properties comply with your system.

(3) Include the HOC-SA build directory into your Java CLASSPATH

HOC-SA is now ready and you can start deploying and using HOCs.

The example HOCs, Worker-HOC and Alignment-HOC, can be built using the HOC-SA scripts as follows:
ant WorkerService builds the Worker-HOC and packages it into a GAR (Grid application archive)
ant AlignmentService builds and packages the Alignment-HOC
For deploying the *.gar files into a Globus container, change to your Globus installation directory and run
globus-deploy-gar workerService.gar and globus-deploy-gar alignmentService.gar

The Alignment-HOC allows the outsourcing of work units to worker servers for processing input in parallel and increasing the performance of the alignment application.
By running ant ServerJar, a binary server.jar is created, which must be executed on each worker server by running java -jar server.jar.
(note that the worker servers do not require their own installation of Globus but port 1099 must be open for RMI communication. Thus, your firewall must be configured appropriately).

ant StartClient starts a graphical (Swing-based) Client, which allows to administer the Code Service, the worker servers, upload and edit code parameters, and to process data using the Worker-HOC and the Alignment-HOC.
Using the graphical client should be self-explanatory.
Please report any problem to hoc-sa-user@globus.org.

FAQ

• What software is required for working with the HOC-SA?

The HOC-SA requires the Globus WS-Core
+ Java SDK 5
+ Apache Ant Version 1.6.5 or higher
+ OGSA-DAI Version 2.1
+ any database system supported by OGSA-DAI, e.g., MySQL

if you intend to run the DNA sequence alignment example application
+ The JAligner Open Source Library

• Is HOC-SA a replacement for WS-GRAM?

No, a (legacy) script or binary which is...
(a)...not intended to be reused multiple times
(b)...directly executable from the UNIX command line
(c)...readily described by existing GRAM-RSL Files
can still be submitted using GRAM, even if a the HOC-SA should be used in the same project.
The HOC-SA and GRAM can both be used in conjunction.

Code which is provided as a reusable component (a HOC) uses the tools provided by the HOC-SA, which enables the required exchange of the code-carrying parameters (code mobility) between Web service providers and consumers.

For any new Grid application that accords to a common pattern of parallel processing, one should consider if either
(a) an existing HOC can be used for processing this application
(b) a new HOC should be developed for this purpose.

If ...
(a) ...a HOC exists, programmers will only deal with application-specific concerns and shielded them from all the required setup work for running their applications on top of the Globus middleware.
(b) ...a new HOC is developed, all future applications that share the same processing pattern do not need to be developed form scratch anymore. Note that the HOC-SA already provides the required means for transferring code and data parameters from clients to the HOC via a Web Service, and when the Worker HOC code in the HOC-SA repository gives an easy example showing how a HOC is developed. So, developing a new HOC does not require more effort than writing a new hard-wired Globus application, but using a HOC much more prospective.

• What HOCs are available and what is their purpose?

Currently, there are two HOCs available:

The basic Worker HOC that processes its input data in a user-defined manner. Since the full processing code must be provided by the user, the Worker HOC provides almost the same functionality as Globus GRAM. The only difference is the separation between code and input data, leading to a reduction of network traffic and improved reusability: when the same procedure should be applied to different data multiple times, The Worker HOC requires only the data to be transferred anew for each application, while the processing code is only transferred once and reused whenever needed.

Grid developers who want to create their own HOCs for more complex applications can refer to the Worker HOC source code as a starting point.

The Alignment HOC, an example for a more complex HOC that runs Sequence Alignment by executing the JAligner code in parallel on multiple Grid servers. Sequence alignment computations are required in many different applications of computation molecular biology. Using the Alignment HOC, DNA sequence alignments can be computed on the Grid and users can customize the HOC by specifying the traceback and pattern recognition routines required in their particular applications.

Both, the Worker HOC and the Alignment HOC are documented in the HOC-SA tutorial. At the University of Muenster, there are some more experimental HOC implementations available which run, e.g., material deformation simulations or image processing applications. The code for these HOCs can be requested and questions about it can be sent to Jan Duennweber

Bugzilla & Browse CVS

• http://bugzilla.globus.org/globus/buglist.cgi?product=HOC-SA
• http://viewcvs.globus.org/viewcvs.cgi/hoc-sa

Latest Sources

• HOC-SA: cvs -d :pserver:anonymous@cvs.globus.org:/home/globdev/CVS/globus-packages co hoc-sa

For each individual application/service, you could use:

• WokerService (simple HOC-SA example): cvs -d :pserver:anonymous@cvs.globus.org:/home/globdev/CVS/globus-packages co hoc-sa/WorkerService
• AlignmentService (protein sequence alignment application based on the FARM pattern): cvs -d :pserver:anonymous@cvs.globus.org:/home/globdev/CVS/globus-packages co hoc-sa/AlignmentService
• SyntheticService (crash study application based on the FARM pattern): cvs -d :pserver:anonymous@cvs.globus.org:/home/globdev/CVS/globus-packages co hoc-sa/SyntheticService

For a specific version, you could add:

• default (implicitly v1.0): cvs checkout hoc-sa
• v1.0 (tagged as 1.0): cvs checkout -d HOC-SA.1.0 -r HOC-SA_1_0_branch hoc-sa
• v2.0 (tagged as 2.0): cvs checkout -d HOC-SA.2.0 -r HOC-SA_2_0_branch hoc-sa

Policies

In addition to the Globus Alliance Project Guidelines, the HOC-SA adheres to the following policies:

Guidelines for committers

• Rules for HOC-SA: http://dev.globus.org/wiki/Guidelines#Committers

Guidelines for individual contributors

• How individuals can contribute: http://dev.globus.org/wiki/Guidelines#Contributors

References

• From Grid Middleware to Grid Applications: Bridging the Gap with HOCs, Sergei Gorlatch and Jan Duennweber - In Future Generation Grids, Springer Verlag, 2005
• Simplifying Grid application programming using Web-enabled Code Transfer tools, Catalin Dumitrescu, Jan Duennweber, Philipp Luedeking, Sergei Gorlatch, Ioan Raicu, Ian Foster - CoreGRID'07 Symposium
• HOC-SA: Higher-Order Components-Service Architecture, a Status Report for the Globus Committers Meeting, May 22-23, 2007 pdf

Contributors

The HOC-SA project gratefully acknowledges the following contributors

• Waldemar Gorus (University of Muenster, Germany)
• Peter G. Lane (Argonne National Laboratory, Illinois)
• Ioan Raicu (University of Chicago, Illinois)