Python is well suited to do preparation and post-processing for jobs, but is not as suited for high-performance computing as a compiled language like C or Fortran. We do not recommend that users use it for a major component of their HPC jobs. We do recommend using it for preparation, post-processing, and proof-of-concept work for defining algorithms.
Versions on the Supercomputer
The version of Python shipped with the operating system is Python 2.6.6 and has few packages. Using modules we have several more full-featured versions available. A current list of Python versions (and other software) is maintained at /documentation/apps/softwareModuleList. You can also run this command to see which Python versions are available:
module avail python
To load Python using modules:
# load Python 2.7.x module load python/2/7 # switch to Python 3.6.x module switch python/3/6
TensorFlow and Keras
At the time of this writing TensorFlow 1.2 and Keras 2.0 are installed for the
python/2/7 module only. In order to use TensorFlow or Keras you will also need these three modules (loading them in this order is important):
A GPU is also required. Add
--gres=gpu:n to your
salloc command where n is the number of GPUs you need per node. At the moment we have up to 4 GPUs per node.
If you need to install a library that you anticipate being the only user of, we recommend that it be installed in your own directory somewhere.
With pip you can do this by running the following command after you have loaded the relevant environment module for Python (eg
pip install --user package
You can also install everything relative to an alternate prefix directory, such as one within a group directory:
pip install --prefix=directory-to-install-to package
For example if you have a top level directory:
~/fsl_groups/fslg_somegroup/.local you can use pip to install packages to
~/fsl_groups/fslg_somegroup/.local/lib. You will then need to add the following two lines to your
export PATH="$HOME/fsl_groups/fslg_somegroup/.local/bin:$PATH" export PYTHONPATH="$HOME/fsl_groups/fslg_somegroup/.local/lib/python2.7/site-packages"
python2.7 with the appropriate version number for Python, like
You can run
pip help install for more options and help.
If the project does not use pip, you can tell Python to use your home directory:
python setup.py install --user package
If it is a library that a large set of users will use or if it requires some specialized compiling, please open a support ticket.
The following packages are available for each version of Python in the module list: