A scratch folder usually refers to a folder that is fast for reading and writing. If you use a supercomputer, like those in XSEDE or NERSC, you will find out that the SCRATCH folder are the best place to run your simulation (though it is not suggested to store the data there). There may be multiple reseaons that the scratch folder is faster than normal disk. In our own in-house cluster, Fractal, the scratch folder is fast, simply because it is local and large.

Fractal has one main node and multiple compute nodes, where most of the data are stored at four hard disks at the main node. During a simulation, the compute node mounts the hard disks in the main node as a remote device and read/write data via internet. Therefore, the simulation can be limited by the internet band width, if it has heavy reading and writing. For example, VASP simulation with LWAVE=T and Quantum Espresso simulation with verbosity=high.

At the day it was installed, each Fractal compute node has already had a /scratch partition locally. But these partitions have never been used... I thought it is a good idea to make use of them. That's why I twisted the access of these folders in the following steps and open the scratch folders to users.

For the admin (given that the /scratch folder already exist):

rocks run host compute command="chmod 755 /scratch"
for name in $(grep 100 /etc/passwd|grep home|cut -d: -f1)
do
 rocks run host compute command="mkdir /scratch/$name; chown $name:users /scratch/$name; chmod 700 /scratch/$name"
done

These steps create folders for each user in the /scratch for all compute nodes.

I also added the following line to the "addClusterUser" routine, so that in the future this will be automatically done for new users

echo
echo ..... build scratch folder at compute nodes .....
echo
rocks run host compute command="mkdir $1; chown $1:users /scratch/$1; chmod 700 /scratch/$1"

For the user build up a symbolic link in your home folder.

ssh compute-0-22 cd ~/ ln -s /scratch/$USER scratch exit

revise your job script. add the below lines to the beginning

cd /scratch/$USER
echo "entering scratch"
mkdir $JOB_ID
cd $JOB_ID
cp $SGE_CWD_PATH/* . -rf
echo "run the job at" $(pwd) $(date)

and the below to the end of your job script

echo "start copying all files back to data folder" $(date)
cp -rf * $SGE_CWD_PATH/
echo "finish copying" $(date) "and remove the file"
rm * -rf
cd $SGE_CWD_PATH