Glasgow GLite Torque Tips

From ScotGrid

Check if a node is running jobs (from svr001):

checkjobs nodexxx

Full listing of queue status:

qstat -f

Useful queue grepping for host and user:

qstat -f |  egrep -i "exec_host|job_owner"

Suspending all running jobs - be careful of the value used for colrm so that you don't crop the job numbers

qstat | grep " R "  | colrm 9 | xargs qsig -s SUSPEND

Small listing in columns:

qstat -n

List who has been running jobs over the course of many log files:

grep "owner=gla" /var/spool/pbs/server_priv/accounting/2008* | grep Exit_status | awk '{print $10}' | sort -g -k 1 | uniq -c

If you want to know how many jobs are running on, say, node123:

svr016:/var/spool/pbs/server_priv/accounting# qstat -f | grep node125
     exec_host = node125/3

or, to see who's running on that server

node=node123; qstat `qstat -f | grep -v resources_used | grep -B 10 ${node} | grep Job\ Id | cut -d' ' -f3`

What one user is doing

user=pheno047; qstat -f `qstat | grep $user | cut -d'.' -f1` | egrep "Job Id|resources_used.wall|exec_host|queue |euser"

Scrape the WMS id's from the qstat output

user=gla029; qstat -f1 | grep EDG_WL_JOBID | grep $user | cut -d , -f 18 | colrm 1 13 > current_jobs

Submit Filter Exampe

dev011:/usr/local/sbin# cat torque_submit_filter.sh
#!/bin/sh

while read i
  do
    if [[ $i =~ "^#PBS -l nodes=[0-9]" ]]
    then
      export NODES=`echo $i | cut -d = -f 2`
      export i="#PBS -l nodes=$NODES:SL5"
    fi
    echo $i
  done

dev011:/var/spool/pbs# cat torque.cfg
SUBMITFILTER /usr/local/sbin/torque_submit_filter.sh