Updated Operating System Image

We have prepared an updated Operating System image for Marylou4, and have installed it on a portion of the cluster. This image, based on RedHat 5.0, will provide updated libraries, compilers, etc., and we hope will provide a cleaner, easier-to-use environment for our users. However, this means that there were some changes to the environment that we need to make you aware of. There are currently 60 nodes available with Ethernet connectivity. We will be moving to this new OS version everywhere by the end of the summer, so we encourage you to make sure your code works as soon as possible

We fully expect this new image to have some problems. This is why we are asking the users' help in isolating what those problems are. Please give your code a try, and contact us for additional help.

Interactive Node

We have provided one interactive node with the new configuration, with the hostname m4new.et.byu.edu. This node is only available from on-campus. Since there is only one of these nodes, we ask that you refrain from running too many processes on that node. This is intended for the purpose of re-compiling software, and submitting jobs.

Operating System

The new operating system is based on RedHat Linux v5.0 with the associated libraries. Since these libraries are different then the old library versions, most compiled software will need to be recompiled with the new libraries. If you have questions about specific software, please contact us, and we will do our best to help you through the problems.

Batch Commands

The new system image includes the traditional qsub and qdel commands for queueing and removing jobs from the queue. However, we now recommend two new commands instead, as shown below. These will become the only way to queue and remove jobs in the future.

• msub - Use this to submit jobs. The syntax is exactly the same as qsub. The output may be slightly different, but shouldn't cause many problems.
• mjobctl -c <jobid> - Use this to cancel/remove jobs from the queue.

In order to request nodes with the new Operating System image, you need to do one of the following:

• Add "#PBS -q newos" to your job script
• Add "-q newos" as a parameter when you call msub to submit the job

Compilers

The new image has the following compilers available:

• GNU Compilers v4.1.2 - includes gcc, g++, and gfortran - Note that gfortran handles both Fortran 77 and Fortran 90, and has replaced the old g77 compiler
• Intel Compilers v10.1.015 - includes icc for C, icpc for C++, and ifort for Fortran
• Portland Group Compilers v7.1-6 - includes pgcc for C, pgCC for C++, and pgf77 and pgf90 for Fortran 77 and Fortran 90 respectively

All these compilers should be available in your default path. If you cannot find them by simply typing the short name, please contact us for help

MPI Implementation

The default MPI implementation has moved from MPICH to OpenMPI. This new implementation has several advantages, including supporting both Ethernet and Infiniband directly. This means that once we migrate this OS image everywhere, you won't have to recompile your code for Infiniband anymore. The same compiled program will work over either transport. In fact, if Infiniband is available on all the nodes your job is assigned to, then OpenMPI will use it by default, even if you didn't request Infiniband.

Since MPI is a standard, any program that uses MPI should work with any implementation, so we don't anticipate too many problems with switching to OpenMPI. The MPI programs will almost certianly need to be recompiled against the new libraries, though.

MPI Environment

With the new MPI implementation, we have attempted to make chosing the MPI compilers easier to use. The default login environment should put all the appropriate compilers like mpicc and job launchers like mpirun and mpiexec in your path. For most users there should be no reason to change this. Unlike the previous system, there is no difference between using mpirun and mpiexec. You can use either one inside a job with minimal syntax. Simply call the launcher you want to use, and pass it your command. For more syntax options, please log into the interactive node and type man mpirun or man mpiexec.

By default, the system is set up to use OpenMPI with GCC compilers. If you want to use either Intel or Portland Group compilers, you will have to make a small change using a new tool called mpi-selector. This tool lets you query your current settings, list available modules, and choose one to use. Whenever you make a change using mpi-selector, you will have to log out and back in for the changes to take place.

In order to use mpi-selector, you will need to use one of the following options:

• mpi-selector --query - lists the current module chosen, and whether it is a system-wide setting or a user-specific setting.
• mpi-selector --list - lists all the modules available - we ask that you choose one of the modules that start with the fsl prefix. Operation is not guaranteed with any of the others.
• mpi-selector --set modulename - sets a user's chosen module to the module name specified. Again, you must log out and back in to have the changes take place
• mpi-selector --unset - unsets a user-chosen setting, dropping back to the system default. Again, you must log out and back in to have the changes take place