FSL Storage

FSL data storage offerings are to be used in conjunction with high performance compute processing performed on FSL compute resources. Data storage needs for other purposes should be held elsewhere.

FSL Storage Resources Summary:

Name Path Default Quota Backed Up Purpose
Home Directory /fslhome/username or ~/ 100 GB Yes Critical data to be retained, source codes, small file processing
Scratch or Compute ~/compute None No Temporary files, in processing data, large file processing
Group ~/fsl_groups/groupname 40 GB Yes Shared data sets and results, group small file processing
Group Scratch ~/fsl_groups/groupname/compute None No Temporary files, in processing data, and large file processing of group shared data
Archive ~/fsl_archive 15 TB No Holding area for large data sets that may be needed again soon. A storage area to relieve more expensive scratch storage.
Group Archive ~/fsl_groups/groupname/fsl_archive 15 TB No Holding area for large shared data that may be needed again soon. Relieves more expensive group scratch storage.
tmp or local scratch /tmp N/A No Limited storage on compute nodes for in processing scratch data

Home Directories

  • Home directories should be used to store codes and data deemed critical and worth being backed up
  • Home directories are accessible from compute nodes and may be used for storing small files being processed against, but should not be used to store temporary scratch data
  • The default quota is 100 GB, but requests may be made for more if appropriate
  • Data in home directories is backed up daily

Scratch, or Compute, Storage

  • Scratch storage is expensive high performance storage to store large data and results
  • Users should take the time to periodically clean up scratch storage and free up space for others
  • There is no default quota on scratch but in some cases quotas are implemented to control some users' processes that diverge
  • Data in scratch is not backed up

Group Storage

  • Group Storage should be used like home directories, with the added benefit of being able to share files with other users
  • The default quota for a group is 40 GB
  • Data in group storage is backed up daily
  • See the Group File Sharing page for more information

Group Scratch Storage

  • Group scratch storage should be used like scratch storage, with the added benefit of being able to share files with other users
  • There is no quota on group scratch storage
  • Group scratch storage is not backed up

Archive Storage

  • Archive storage serves as a holding place:
    1. to get data off of the more expensive scratch and home directory storage
    2. for data that is typically too large to transfer across the campus network efficiently
    3. for data that may be needed in the near future for reference or ongoing computing
  • Archive storage is only available on the interactive nodes, not on compute nodes
  • Users have to manually migrate data on and off of archive storage
  • There is a 15TB total quota on archive storage, which encompasses both personal and group files
  • Archive storage is not backed up at this time, but backup is being considered. Drives are protected with RAID 6, which is not a remedy for accidental deletion.

Group Archive Storage

  • Group archive storage should be used like archive storage, with the added benefit of being able to share files with other users
  • Group archive storage is only available on the interactive nodes, not on compute nodes
  • Users have to manually migrate data on and off of group archive storage
  • There is a 15TB total quota on archive storage, which encompasses both personal and group files
  • Archive storage is not backed up at this time, but backup is being considered. Drives are protected with RAID 6, which is not a remedy for accidental deletion.

tmp, or Local Scratch Storage

  • tmp storage uses the local hard drive on the node the job is running on
  • Users may create temporary files and directories in /tmp for local scratch file needs
  • Users are responsible for and should automate cleaning up of /tmp after every job
  • Each compute node has limited space. This may vary from about 30 GB to over 100 GB
  • We recommend using the path /tmp/\$PBS_JOBID to make sure that the directory is unique for each job.
  • An example script to manage the cleanup follows:
#!/bin/bash
#PBS -l nodes=1:ppn=8,walltime=3:00:00
#PBS -N test_localscratch
 
TMPDIR=/tmp/$PBS_JOBID
 
#prepare function to clean up directory if job is killed, etc
#NOTE:  THIS IS NOT EXECUTED UNTIL THE SIGNAL IS CALLED.
cleanup_scratch()
	echo "Cleaning up temporary directory inside signal handler, meaning I either hit the walltime, or deliberately deleted this job using qdel"
	rm -rfv $TMPDIR
	echo "Signal handler ended at:"
	date
	exit 1
}
 
#Now, associate this function with the signal that is called when the job is killed, or hits its walltime
trap 'cleanup_scratch' TERM
 
#Now, initial setup of temporary scratch directory
echo "Creating scratch directory at $TMPDIR"
mkdir -pv $TMPDIR 2>&1
 
#PUT CODE TO COPY YOUR DATA INTO $TMPDIR HERE IF NECESSARY
 
#DO WHATEVER YOU NEED TO DO TO GET YOUR SOFTWARE TO USE $TMPDIR.  THIS WILL DEPEND ON THE SOFTWARE BEING USED
 
#PUT CODE TO RUN YOUR JOB HERE
 
cd $PBS_O_WORKDIR
 
echo "Cleaning up temporary directory at end of script, meaning that the job exited cleanly"
rm -rfv $TMPDIR

To Top

Please feel free to contact us with your questions about FSL storage.