GQSUB

Lets start with an example submission script, and show it in action. Firstly, here's the script in Example 4.1, “Trivial submission script”.

#!/bin/sh

#PBS -l cput=0:30:00
#PBS -l walltime=0:30:00
#GQSUB -q UKI-SCOTGRID-GLASGOW

echo in user script
hostname
pwd

And here's the submission and monitoring in use.

-bash-3.00$ ls
gqdel  gqoutput  gqstat  gqsub  gqsubconfig.py  gqsubconfig.pyc  gqsubproxy.py  gqsubproxy.pyc  test.sh
-bash-3.00$ gqsub test.sh 
File stagein requested: on a shared path, so no staging needed.  Note that this captures files at time of execution.

Couldn't find a valid proxy.

Proxy renewal required - looking for 36:45:00
Enter GRID pass phrase:
Submitting job 320 as: https://svr023.gla.scotgrid.ac.uk:9000/SJ7Ch-mRrEn4dD9oAgHecw

-bash-3.00$ gqstat 320
Getting status for 320
Job id Name User                                                    WallTime State     Queue                                               
------ ---- ------------------------------------------------------- -------- --------- ----------------------------------------------------
320         /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=stuart_purdie          Scheduled dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q7d
-bash-3.00$ ./gqstat 320
Getting status for 320
Job id Name User                                                    WallTime State   Queue
------ ---- ------------------------------------------------------- -------- ------- ----------------------------------------------------
320         /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=stuart_purdie 00:02:14 Running dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q7d
-bash-3.00$ ./gqstat 320
Getting status for 320
Job id Name User                                                    WallTime State Queue                                               
------ ---- ------------------------------------------------------- -------- ----- ----------------------------------------------------
320         /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=stuart_purdie 00:26:32 Done  dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q7d
-bash-3.00$ ls
gqdel  gqoutput  gqstat  gqsub  gqsubconfig.py  gqsubconfig.pyc  gqsubproxy.py  gqsubproxy.pyc  test.sh  test.sh.e320  test.sh.o320
-bash-3.00$ more test.sh.o320 
in user script
node296.beowulf.cluster
/cluster/share/gla058/gqsub
-bash-3.00$ 

The details of the output from gqstat will be covered in detail later; for the moment note that it shows the job going through the relevent stages. In this case, the job was run on a shared path, so the output appeared automatically as it was generated. The job script executed, test.sh, is the one shown in Example 4.1, “Trivial submission script”.

The prompt to generate the proxy was due to the current proxy having expired. Note that there was no VO specified explicitly anywhere - it was assumed that the VO to use matched the last VO used. If there had been no proxy at all ( e.g. voms-proxy-destroy had been run, then it would have drawn the VO name from the default VO in .gqsubrc.

POSIX defines many command line options, and many more are present in vendor extensions. Here we have a complete survey of the command line options available to gqsub.

In most cases, the most important part of a job is the output. This is what the aim of the exercise is, and is the whole point. A few jobs will, as a natutal consequence of the desired effect, write the output to some external system. Most of the time, however, the person who submitted the job wants to see some output back. gqsub has a few tricks available to assist with this.

Fundementally, Grid systems are desigined with the idea of disconnected operation in mind - that is, the machine where the job runs knows and cares about nothing that relates to the machine that submitted the job. This is part of what allows it to scale to such a large system, and prevents problems in one machine affecting others. However, this is not the way the people want to work. In most cases, local batch systems are used with a shared filesystem - shared between the headnode and the worker node. Although it introduces risks of race conditions, and other oddities, if used incorrectly, it is at it's core much easier to understand what is happening - everything is written to one place. Given the familiarity with shared filesystems that is prevelent, the theoretical downsides do not appear to cause problems under real use conditions.

By building the illusion of a shared filesystem, the aim is to couple the scability and distributed nature of the Grid with an easy to understand way of working.

There are three ways of working that gqsub supports, depending on what's available at the UI. Where more than one is feasable, the first in the list is picked - although it is possible to change this with specific command line options.

In general, the post-hoc output collection is always available, whilst the other require additional support, beyond the basic gLite UI. In particular, as use of a GridFTP server on the UI requires it to be present when the job completes, it is not suitable to use that output mode for use in an 'occasionally connect' manner - i.e. laptops. On the other hand, it's a strong adjunt to an existing cluster, if installed on the head node.

#!/bin/bash

#PBS -l cput=0:30:0

#GQSUB -q UKI-SCOTGRID-GLASGOW

#GQSUB -W stagein=Si_00.usp
#GQSUB -W stagein=si.cell
#GQSUB -W stagein=si.param

#GQSUB -W stageout=si.castep
#GQSUB -W stageout=si.castep_bin
#GQSUB -W stageout=si.bands
#GQSUB -W stageout=si.check
#GQSUB -W stageout=si.cst_esp
#GQSUB -W stageout=si.wvfn.1

# Specific to Glasgow, hence the -q flag
CASTEP=/expsoft/ssp/castep/4.4/castep/castep

$CASTEP si

It's often the case that one wants to do a something very many times, with only slight differences between each run. The traditional method to specifing this is to use the 'Array job' semantics in qsub, where each subjob is given a number, and this number is used to influence the behaviour of the job.

With gqsub, the -t flag indicates the indent to use array jobs. Each job is submitted in batches, and when they are started on the worker node the environment contains a number of values. It is these values that should be used to tune the performance of the job.

#!/bin/bash

#PBS -l cput=0:30:0

#GQSUB -W stagein=input.$PBS_ARRAY_ID
#GQSUB -W stagein=cruncher

#GQSUB -W stageout=output.$PBS_ARRAY_ID

cruncher -i input.$PBS_ARRAY_ID -o output.$PBS_ARRAY_ID

This example would be run with the -t flag, as 'gqsub -t 1-78 runCruncher'. The enrivonemnt variable $PBS_ARRAY_ID is set to the subjob number for that specific job - this is the Torque convention. Also supported in gqsub is the SGE conventions for array jobs. All of these are detailed in the table below.

Although perhaps not the most expected feature, gqstat can be involved in job output. If the job cannot return the output data automatically (either by shared filesystem, or GridFTP), then the output has to be collected after the job is finished. Indeed, this is the default more of operation for gLite middlewares.

With gqsub, the aim is to produce the illusion of a shared file system. Therefore, if the output has to be collected, once gqstat determines the job is complete, (in the 'Done' state), it will collect the output automatically. This has the effect that one gqstat reports the job is done, you can work with it's output, and don't have to be concerned with the details of collecting jobs or proxy certs.