Rules

Grid usage

Getting access

To use the grid computing environment, you need to send securely (SMS to Alain Mobile: +32 497 701764 ) :

• a standard UNIX login name
• a good password (mixture of lower/uppercases, digits, special characters)
• a passphrase (a few words, easy to remember but very difficult to guess or to break by brute force) for internal ssh use.

Accessing

When you have an account on the grid,

• you may log in the Rocks front-end, aka 'RFE' (139.165.108.214)
• from this RFE, you may submit your jobs to the Sun Grid Engine, aka 'SGE'
• your account is also set on every computing node ('ssh compute-O-5' for ex, using your passphrase)
• the list of active compute-nodes is given by 'qhost'

Tip : if you want to often use ssh or scp, it is possible to give your passphrase once :
* start a new shell under the protection of ssh-agent :
     ssh-agent bash
* then inject your passphrase :
     ssh-add 
     Please enter your passphrase : ***********
  and you are done : scp and ssh will then work with no explicit passphrase requirements.

IMPORTANT REMARKS

• All computational work (i.e. tasks running for more than 1h) are submitted through advanced qsub system
• The grid is not infinite (2011 : roughly 400 cores), so everyone is invited to be kind with the other. Check available resources by running qhost
• When you start a large batch of jobs, please

Check the grid state

The state of the grid is reported on this page

• Make sure you are asking for a "reasonnable" amount of ressources
• Make sure that your jobs are not crashing the grid

To test a script by direct connection to a node, prefer the less loaded one (whenever possible).

Do not send an email for each job

• 20000 emails in your box is not a good idea, and anyway due to the email flood, the University will block our site for a while... : to suppress the email mechanism, use the switch -m n (for 'no')

• You can also choose about which submitted job events you will be notified via email by specifying in the switch -m following options (more than one option could be specified, for example -m base )

 'b'     Mail is sent at the beginning of the job.
 'e'     Mail is sent at the end of the job.
 'a'     Mail is sent when the job is aborted or rescheduled. (i.e. killed by sys admin)
 's'     Mail is sent when the job is suspended.
 'n'     No mail is sent.

Do not start every job at the same time but use :

• the job array mechanism -t from-to (ex: -t 1-1000)
• with the additional condition to start the whole work 'chunk by chunk' -tc small-chunk-size (ex: -tc 10)

By the way, you will have access to the variable $SGE_TASK_ID which is an incremental variable corresponding to the job number within the array: it goes from start to stop in this example. $SGE_TASK_FIRST and $SGE_TASK_LAST are also available.

Example

To start a big batch of 200000 jobs, 100 by 100, you have simply to insert

#$ -M billy.thekid@ulg.ac.be   # to identify you, anyway
#$ -m n                        # no email 
#$ -t 1-200000                 # declare a big batch of 200 k jobs'
#$ -tc  100                    # use chunks of 100 jobs at a time'
echo "This job is the $SGE_TASK_ID th"

MANDATORY requirements

To give you the best chance to access and to share the right resources, you MUST explicitly define some MEM and TIME requirements : three requirements are absolutely required : h_vmem, mem_free and h_rt , for instance :

#$ -l h_vmem=5500M                                                                                             
#$ -l mem_free=5500M
#$ -l h_rt=99:00:00

If you prefer a more concise writing, or if you invoke SGE's options directly from a terminal. You can separate resource request by a comma. The previous instruction will then be : -l h_vmem=5G,mem_free=5G,h_rt=99:00:00

In order to adjust the requested resources to the best values, you can check previous work summaries. To do so you can use :

qacct -j jobid

where jobid is a previous job with similar resources requirement. You'll find at the end of the report a line containing maxvmem=iiiM with iii being half of the highest value of memory consumption of the job. (The report being half of the value we are interested in, due to a known bug in SGE).

alternatively, you can use

qstat -j jobid

this command work only for a running job and report the real-time memory resource taken by the job

Last, if you request an e-mail for each submitted job, you should also have in it some hints on the resources needed by your job.

More informations

• Gridengine
• [SGE job arrays : Introduction]
• [Job arrays : Task dependencies]
• Variables
• Options
• Job status : qstat

Storage

Three central locations are provided :

• your 'natural' home on RFE : please use it with care, no huge data or application !
• 12 TB NAS for stable, long term storage : /home/safe/youraccount
• 11 TB NAS for short-term work data, used heavily by the computing nodes : /home/volatile/youraccount; old and non active files may be erased when a disk full is imminent.

For AIS :

• 19 TB NAS : /home/safeais

For STATGEN :

• 11 TB : /home/storage

Two local storages

• /tmp on every computing node
• /scratch on every computing node (bigger partition : 50 - 200 GB, depending on the nodes)

Please take care to discard your temporary files !