Cook Book

From Informatics

Goal

Tools

This section provides links to all the tools we used in this project. It is best to continue reading this wiki to get an idea of how we used these tools before coming back for more information on the tools themselves.

caGrid

General

Security

Communicating with TeraGrid

Introduce with gRAVi Plugin

TeraGrid

General

Logging On

Submitting Jobs

Moving Files

gridFTP
- Data Staging
- Plain old moving files

Additional Information

WS-GRAM

Grid Resource Allocation and Management
This is a Globus protocol used for remote job submission and control.

TeraGrid GRAM Resources

Condor

There is talk of using Condor to submit jobs to TeraGrid. Condor will use the GRAM protocol for this job submission. Condor has support for java jobs.

Setup

This section references the following caGrid services:

You can set up your own Dorian, GTS, CDS, and Grid Grouper services by following the links above. In this wiki, we use the services available on the caGrid training grid.

Security for caGrid-TeraGrid Communication

Before diving into the setup process, bring up the caGrid GAARDS UI:

Go to the directory with caGrid 1.1 installed (we will refer to this directory as CAGRID_HOME, for example C:\java\apps\caGrid). If you don't have caGrid 1.1 installed, please do so.
Under CAGRID_HOME, type

ant security

Obtain caGrid user account for the application user

On the machine that the application is expected to be used,

Synchronize user credentials with caGrid Trust Fabric

On the machine that the application is expected to be used, go to CAGRID_HOME, and type

ant -Dtarget.grid=<grid name> configure  (Possible grids names are: nci_prod,nci_qa,nci_stage,nci_dev,osu_dev,training,custom_grid.)

to specify the caGrid with which you wish to work. (For more information on configuring caGrids, see how to change target grids.)

Next, go to CAGRID_HOME/projects/syncgts

ant syncWithTrustFabric

For more information on synchronizing with caGrid Trust Fabric, see caGrid Wiki on GTS.

Include user ID in the caGrid Grid Grouper

Go to the machine running the application and obtain the user proxy:

Click on "Login" and ask to authenticate with your username and password. Use the appropriate dorian uri, for example, if you obtained your user account on the training grid, get its proxy from the training Dorian. Also set the proxy lifetime to 4 hrs and delegation length to 2. Click "Authenticate".

Note: The 4 hours of proxy lifetime is required to coordinate with the delegation lifetime that is set programmatically in the "Obtaining the Delegation Reference" section.

A "Proxy Manager" dialog pops up --> click on the certificate you just generated listed under "Certificate Chain" --> click on "Set Default"

Add the user id for the application user to the grid grouper:

In the GAARDS UI, click on Group Management menu --> Group Browser to bring up the grid grouper UI. Click on "Add Grid Grouper". This brings up the Add Grid Grouper dialog. In the dialog, specify the appropriate grid grouper URL and the application user credential. Click "Add".

After the grid grouper service loads successfully, go to the service explorer on the left. Choose the grid group for this project, and double click on the group name. This brings up the stem details for that group. Click on the "Members" tab.

Click on "Add Member" at the bottom. This brings up the add member dialog. Choose member type "user" and specify user name in "Member Identity". Click "Add Member".

User name should show up in the members list.

caBIG community account for TeraGrid

Developers of gateway services will use the caBIG Community account to access Teragrid resources. They will not have to obtain individual Teragrid accounts but if interested, these steps can be followed:

Verfiy that you have access to the caBIG community account: Log onto a login node (for example, the San Diego cluster login node)

ssh -X <userNameForSanDiego>@dslogin.sdsc.edu
password:<SanDiegoPassword>

Perform Single Sign On

<userNameForSanDiego>@tg-grid1:~> myproxy-logon -T -l <userNameForUserPortal> 
Enter MyProxy pass phrase:<userPortalPassword>
A credential has been received for user <userNameForUserPortal> in /tmp/x509up_u510.
Trust roots have been installed in /home/<userNameForUserPortal>/.globus/certificates/.

<userNameForUserPortal>@tg-grid1:~> grid-proxy-info
subject  : /C=US/O=National Center for Supercomputing Applications/ CN=<Name of User>
issuer   : /C=US/O=National Center for Supercomputing Applications/ 
OU=Certificate Authorities/CN=MyProxy
identity : /C=US/O=National Center for Supercomputing Applications/ CN=<Name of User>
type     : end entity credential
strength : 1024 bits
path     : /tmp/x509up_u510
timeleft : 11:59:37

<userNameForUserPortal>@tg-grid1:~> gsissh tg-login.ncsa.teragrid.org NCSA Teragrid Cluster (MERCURY) --In Production with 868 nodes--

For addtional information on available clusters and node, see TeraGrid Resources (Click on "Resources" tab and "Systems Monitor" sub-menu to get a list of host nodes.)

Synchronize caBIG community credentials for TeraGrid with caGrid Trust Fabric

Synchronize caBIG community credentials for TeraGrid with TeraGrid Trust Fabric

Associate the appropriate grid group with the gateway service

Assuming the gateway service has been created -- load it into Introduce (via "Modify Service" button) and go to the "Security" Tab. The authorization sub-tab allows you to specify the grid grouper membership that is allowed to access this gateway service. (If the gateway service has not been created, please skip this step and follow instructions under section #Creating the caGrid Gateway Service.)

In the "Authorization" tab, first specify the authorization mechanism to be "Grid Grouper". Then specify the grid grouper service URL and click "Load" to bring up the list of groups available. Click on "TeragridSample" group and then click on "Add Group". That will add the TeragridSample group to the list of authorized users to invoke the gateway service. For more information, see caGrid Grid Grouper.

Binary Staging on TeraGrid

Make sure you have access to the caBIG community account for TeraGrid or some other valid TeraGrid account. If not, see step 7 from the section above.

Stage your binary in the community software area. Argonne is a good machine to use. To transfer your binary files, log in using the caBIG account, and use scp or ftp to transfer the jar file.
Make sure the node you plan to stage the binaries on contains the correct version of the software your binaries will need (e.g. does Argonne has JDK 1.5?).
First, create a command line interface and test it out locally.
Next, test your binary on the Argonne cluster.
- Log onto to a login node (e.g. sdsc) as detailed in step 4 from the section above.
- scp your binary to the login machine.
- Perform single sign on as detailed in step 4 from the section above.
- scp your binary from the login machine to your chosen cluster machine (e.g. tg-grid1.uc.teragrid.org).
- Verify that the necessary software is available to you.
- Run your binary again.
- The log output should be the same as the local run.

Creating the caGrid Gateway Service

This section details how to create a caGrid gateway service.

Introduce and gRAVi

Use Introduce with the gRAVi plugin to create the basis for the gateway service -- TeragridSample. The generated code should have the following structure:

In Introduce, the data types imported are:

The specified operations method is:

In this case the one method the gateway client calls is execute().

The security options should be:

with "Custom" option, and all three communications security options checked. (All three communications methods should show "Privacy".

The authorization tab allows you to specify the grid grouper membership that is allowed to access this gateway service.

In the "Authorization" tab, first specify the authorization mechanism to be "Grid Grouper". Then specify the grid grouper service URL and click "Load" to bring up the list of groups available. Click on "TeragridSample" group and then click on "Add Group". That will add the TeragridSample group to the list of authorized users to invoke the gateway service. For more information, see caGrid Grid Grouper.

Under the "Service Credentials" tab, choose to import credentials from the file system and run as the system.

Gateway Client

org.cagrid.introduce.sample.teragrid.client.TeragridSampleClient.java

The gateway client passes the application data, and any user selected parameters, to the gateway service hosted at GATEWAY_SERVICE_URL. Both the data and the parameters are should be caDSR compliant data structures.

TeragridSampleClient client = new TeragridSampleClient(GATEWAY_SERVICE_URL);
<ApplicationResult app> = client.execute(getData(), getParameters());

Gateway Service

org.cagrid.introduce.sample.teragrid.context.service.TeragridSampleImpl.java

Make sure the following jars are on the classpath:

GLOBUS_LOCATION/lib/axis.jar
GLOBUS_LOCATION/lib/gram-client.jar
GLOBUS_LOCATION/lib/gram-stubs.jar
GLOBUS_LOCATION/lib/gram-util.jar
GLOBUS_LOCATION/lib/gram-monitoring.jar
GLOBUS_LOCATION/lib/wsrf-core.jar
GLOBUS_LOCATION/lib/cog-jglobus.jar
GLOBUS_LOCATION/lib/jgss.jar

Import the following classes:

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.RandomAccessFile;
import java.net.URL;
import java.rmi.RemoteException;
import java.util.Date;
import org.apache.axis.components.uuid.UUIDGen;
import org.apache.axis.components.uuid.UUIDGenFactory;
import org.apache.axis.message.addressing.EndpointReferenceType;
import org.globus.exec.client.GramJob;
import org.globus.exec.generated.JobDescriptionType;
import org.globus.exec.generated.StateEnumeration;
import org.globus.exec.utils.ManagedJobFactoryConstants;
import org.globus.exec.utils.client.ManagedJobFactoryClientHelper;
import org.globus.wsrf.impl.security.authentication.Constants;
import org.globus.wsrf.impl.security.authorization.Authorization;
import org.globus.wsrf.impl.security.authorization.HostAuthorization;
import org.gridforum.jgss.ExtendedGSSCredential;
import org.gridforum.jgss.ExtendedGSSManager;
import org.ietf.jgss.GSSCredential;
import org.globus.exec.client.GramJobListener;
import org.globus.ftp.DataChannelAuthentication;
import org.globus.ftp.DataSink;
import org.globus.ftp.DataSource;
import org.globus.ftp.FileRandomIO;
import org.globus.ftp.GridFTPClient;
import org.globus.ftp.GridFTPSession;
import org.globus.gsi.GlobusCredential;
import org.globus.gsi.GlobusCredentialException;
import org.globus.gsi.gssapi.GlobusGSSCredentialImpl;

Make sure the class implements GramJobListener interface.

public class TeragridSampleImpl extends TeragridSampleImplBase  implements GramJobListener {

This interface has one method:

public void stateChanged(GramJob job) {
StateEnumeration jobState = job.getState();
boolean holding = job.isHolding();
printMessage("========== State Notification ==========");
printJobState(jobState, holding);
printMessage("========================================");
synchronized (this) {
if ( jobState.equals(StateEnumeration.Done)|| jobState.equals(StateEnumeration.Failed)) {
printMessage("Exit Code: " + Integer.toString(job.getExitCode()));
// also call code to start processing results, for example gridFTP them from TeraGrid back to gateway service
this.jobCompleted = true;
}
notifyAll();
}
}

Authenticate with TeraGrid using caBIG community credentials

GlobusCredential globusCred = new GlobusCredential(PROXY_LOCATION);
GlobusGSSCredentialImpl cred = new GlobusGSSCredentialImpl(globusCred, GSSCredential.INITIATE_AND_ACCEPT);

where PROXY_LOCATION is the location of the TeraGrid user certificate.

set up gridFTP info

String DEST_HOST = <name of the host in TeraGrid with the gridFTP service>
int GRIDFTP_PORT = 2811;

gridFTP input data and parameters

String FILE_TO_FTP = <full path to the input data file/parameter file>;

GridFTPClient client = new GridFTPClient(DEST_HOST, GRIDFTP_PORT);
client.authenticate(creds);
client.setProtectionBufferSize(16384);
client.setType(GridFTPSession.TYPE_IMAGE);
client.setMode(GridFTPSession.MODE_EBLOCK);
client.setDataChannelAuthentication(DataChannelAuthentication.NONE);
client.setDataChannelProtection(GridFTPSession.PROTECTION_SAFE);
client.setPassive();
client.setLocalActive();
DataSource source = new FileRandomIO(new RandomAccessFile(new File(FILE_TO_FTP), "r"));
client.extendedPut(FILE_TO_FTP, source, null);
client.close();

Submitting the TeraGrid job (also see TeraGrid wiki on submitting GRAM jobs via Java).

Break up the original command line to invoke the staged binary to

String [] arguments = new String [...];
arguments [0] = "...";
arguments [1] = "...";
...

Allow delegeation from client -> gRAVi service -> GRAM

boolean limitedDelegation = true;	
boolean delegationEnabled = true;

Set service dates

Date serviceDuration = <set service duration>
Date serviceTermination= <set service termination date>

Submit the job

JobDescriptionType jobDescription = createJobDescription(PATH_TO_JAVA_BIN + "/java", arguments);
GramJob job = new GramJob(jobDescription);
job.setTimeOut(GramJob.DEFAULT_TIMEOUT);
job.setAuthorization(HostAuthorization.getInstance());
job.setMessageProtectionType(Constants.ENCRYPTION);
job.setDelegationEnabled(delegationEnabled);
job.setDuration(serviceDuration);
job.setTerminationTime(serviceTermination);
// GRAMContact and GRAMType are both specified in rave.properties
EndpointReferenceType factoryEndpoint = getFactoryEPR(GRAMContact, GRAMType);
ExtendedGSSManager manager = (ExtendedGSSManager) ExtendedGSSManager.getInstance();
// proxyPath is specified in rave.properties
String handle = "X509_USER_PROXY=" + proxyPath.toString();  
GSSCredential proxy = manager.createCredential(handle.getBytes(), ExtendedGSSCredential.IMPEXP_MECH_SPECIFIC, GSSCredential.DEFAULT_LIFETIME, null, GSSCredential.INITIATE_AND_ACCEPT);
job.setCredentials(proxy);
String submissionID = "uuid:" + uuidgen.nextUUID();
job.addListener(this);  
job.submit(factoryEndpoint, false, limitedDelegation, submissionID);

Note: for details on rave.properties, see wiki on gRAVi pluging.

gridFTP to retrieve results

String FILE_TO_FTP = <full path to the result data file>;

GridFTPClient client = new GridFTPClient(DEST_HOST, GRIDFTP_PORT);
client.authenticate(cred);
client.setProtectionBufferSize(16384);
client.setType(GridFTPSession.TYPE_IMAGE);
client.setMode(GridFTPSession.MODE_EBLOCK);
client.setDataChannelAuthentication(DataChannelAuthentication.SELF);
client.setDataChannelProtection(GridFTPSession.PROTECTION_SAFE);
client.setLocalPassive();
client.setActive();
DataSink sink = new FileRandomIO(new RandomAccessFile(new File(FILE_TO_FTP), "rw"));
client.get(FILE_TO_FTP, sink, null);
client.close();

Deploying the Gateway Service

A gateway services must be deployed in a secured caGrid container. You can either configure tomcat to be a secured container or install globus as the secured container.

In Introduce's Deploy dialog, click "Deploy".

Simply use the URL to the gateway service when you instantiate your gateway client. (See #Gateway Client).

caGrid-TeraGrid Security

Also see #Security for caGrid-TeraGrid Communication.