gqsub is a package that offers the familiar POSIX qsub interface to grid jobs. It is intented to reduce the cognitive gap the exists between local cluster use, and Grid job submission. Depending on the configuration of the local cluster, it may be possible to run submission scripts unmodified on the Grid, using gqsub. More commonly, it takes a few small modifications to produce a submission script that can run on either the local cluster, or remotely on the Grid.
This gives a mechanism for introducing experienced cluster users to the Grid, without a large barrier to entry.
It depends on the gLite 3.1 or 3.2 UI packages, but can be installed by an end user, or for the whole machine. 'Installation' is a simple matter of putting the files somewhere.
With the 1.3.0 release, gqsub offers end to end (submission, monitoring and output retrival) job management. Use gqsub to submit the job, and gqstat to monitor. Once gqstat shows that a job is done, the output is back in the directory the job was launched from. There's a few tricks it uses to optimise this, but they are not required.
1.4.0 splits up the internals a bit, and allows for plugable credential systems and submission engines. The only submission engine currently supported is gLite. Credential support covers simple VOMS proxies, and the use of MyProxy servers for credential renewal. Put 'credentials = MyProxy' in your .gqsubrc file, and away you go. Also new is the /etc/gqsubrc file, for system wide options.
1.5.0 gives data handling - staging data in and out of Storage Elements. The general method of use is a generalisation of the stagein / stageout syntax. File are specified as localName@remoteLocation, where if the localName and the @ are ommited, it defaults to the same filename as the remote host. Support remoteLocation specifiations are:
Efficency is not quite optimal, but it works. That means that occasionally a file with multiple replicas might be pulled not from the closest host; if there are multiple files to be pulled from Storage Elements. Efficency improvements will occur over the next few releases
1.5.2 release fixes a problem with gqdel, and disables deep resubmission for jobs doing direct stage out to gridFTP. There is a repo at http://www.scotgrid.ac.uk/gqsub/repo/
There is a blog to cover development and other chatter
Key features planned are listed, along with (sometimes) an expected date for release. Release dates are not (sadly) guaranteed, but a best estimate.
There are some features that are readily identifiable as interesting, but also not clear that there is user demand for them. These are listed as 'unscheduled'. Should you find one of these particularly interesting, do let me know (s dot purdie at physics dot gla dot ac dot uk), and I'll amend the plan.
| Key milestone | Version | Estimated Release date | Notes |
|---|---|---|---|
| Whole directory submission [0] | 1.6.0 | Late 2011 | |
| Handle ARC submission | 1.7.0 | Late 2011 | |
| Extension to handle MPI jobs | 1.8.0 | ||
| Direct OGSA-BES submission | Unscheduled | ||
| Handle GRAM submission | Unscheduled | ||
| Job sequencing directives | Unscheduled | Blog post |
[0]: Allow users to use explicit file staging, without worrying over the details. Ship everything in the current directory (and subdirs) to the worker node. After the job, ship back everything that's changed. Perhaps not hte most efficent, but for many users this is much simpler to handle. With thanks to Jan Just Keijser for the idea.
| Version | Release date | Tarball | Online manual | Notes |
|---|---|---|---|---|
| 1.0.0 | 7 Sep 2009 | gqsub-1.0.0.tgz | manual | |
| 1.1.0 | 21 Sep 2009 | gqsub-1.1.0.tgz | manual | Removed dependance on python32, and removed assumptions about environment variables. |
| 1.2.0 | 24 Sep 2009 | gqsub-1.2.0.tgz | manual | Auto return of stage out data, via GridFTP |
| 1.3.1 | 26 Oct 2009 | gqsub-1.3.1.tgz | manual | Auto collection of stage out data, when gqstat identifies the job is complete |
| 1.3.2 | 16 Dec 2009 | gqsub-1.3.2.tgz | manual | JDL escape valve, for JDL features not supported nativly |
| 1.4.1 | 4 Feb 2010 | gqsub-1.4.1.tgz | manual | Mutlti credentials use - allows for proper myproxy support |
| 1.5.2 | 6 Dec 2010 | gqsub-1.5.2.tar.gz | manual | Data mangement - stage in and out of data to SE's |