gqsub is a package that offers the familiar POSIX qsub interface to grid jobs. It is intented to reduce the cognitive gap the exists between local cluster use, and Grid job submission. Depending on the configuration of the local cluster, it may be possible to run submission scripts unmodified on the Grid, using gqsub. More commonly, it takes a few small modifications to produce a submission script that can run on either the local cluster, or remotely on the Grid.
This gives a mechanism for introducing experienced cluster users to the Grid, without a large barrier to entry.
It depends on the gLite 3.1 or 3.2 UI packages, but can be installed by an end user, or for the whole machine. 'Installation' is a simple matter of putting the files somewhere.
With the 1.3.0 release, gqsub offers end to end (submission, monitoring and output retrival) job management. Use gqsub to submit the job, and gqstat to monitor. Once gqstat shows that a job is done, the output is back in the directory the job was launched from. There's a few tricks it uses to optimise this, but they are not required.
1.3.2 lets you specify two things for direct use of JDL - --cerequirements which is &&'d with any other generated CERequirements, and --jdl which should be a complete JDL line (although the terminating semi-colon can be skipped), which is dropped in verbatim. This allows for things like selecting (or preventing) a Cream CE with CERequirements, or using --jdl MyProxyServer="lcgrbp01.gridpp.rl.ac.uk". Note that the string should not be escaped, should not have spaces in it, and doesn't need a semi colon.
1.4.0 splits up the internals a bit, and allows for plugable credential systems and submission engines. The only submission engine currently supported is gLite. Credential support covers simple VOMS proxies, and the use of MyProxy servers for credential renewal. Put 'credentials = MyProxy' in your .gqsubrc file, and away you go. Also new is the /etc/gqsubrc file, for system wide options.
There is a blog to cover development and other chatter
Key features planned are listed, along with (sometimes) an expected date for release. Release dates are not (sadly) guaranteed, but a best estimate.
There are some features that are readily identifiable as interesting, but also not clear that there is user demand for them. These are listed as 'unscheduled'. Should you find one of these particularly interesting, do let me know (s dot purdie at physics dot gla dot ac dot uk), and I'll amend the plan.
| Key milestone | Version | Estimated Release date |
|---|---|---|
| Data management interface | 1.5.0 | Mar 2010 |
| Whole directory submission [0] | 1.6.0 | Apr 2010 |
| Handle ARC submission | 1.7.0 | May 2010 |
| Extension to handle MPI jobs | 1.8.0 | |
| Direct OGSA-BES submission | Unscheduled | |
| Handle GRAM submission | Unscheduled |
[0]: Allow users to use explicit file staging, without worrying over the details. Ship everything in the current directory (and subdirs) to the worker node. After the job, ship back everything that's changed. Perhaps not hte most efficent, but for many users this is much simpler to handle. With thanks to Jan Just Keijser for the idea.
| Version | Release date | Tarball | Online manual | Notes |
|---|---|---|---|---|
| 1.0.0 | 7 Sep 2009 | gqsub-1.0.0.tgz | manual | |
| 1.1.0 | 21 Sep 2009 | gqsub-1.1.0.tgz | manual | Removed dependance on python32, and removed assumptions about environment variables. |
| 1.2.0 | 24 Sep 2009 | gqsub-1.2.0.tgz | manual | Auto return of stage out data, via GridFTP |
| 1.3.1 | 26 Oct 2009 | gqsub-1.3.1.tgz | manual | Auto collection of stage out data, when gqstat identifies the job is complete |
| 1.3.2 | 16 Dec 2009 | gqsub-1.3.2.tgz | manual | JDL escape valve, for JDL features not supported nativly |
| 1.4.1 | 4 Feb 2010 | gqsub-1.4.1.tgz | manual | Mutlti credentials use - allows for proper myproxy support |