Illegal Instruction errors occur when you try to run a program that was compiled with some processor-specific optimizations, and is then running on a processor that fails to meet those requirements.

What are the symptoms of this problem?

You've encountered this situation when one of the following occurs:

• You try to run an MPI program, and you get an error like this one:

Illegal Instruction

• You try to run a serial or MPI program, and you get an error like this one:

Fatal Error: This program was not built to run on the processor in your system.
The allowed processors are: Intel(R) processors with SSE4.2 and POPCNT instructions support.

What causes this?

Most often this occurs when the following occurs:

• The program is compiled using Intel compilers
• The program uses some processor-specific optimizations, like -xHost, or -xSSE4.2
• The program runs on processors that don't support the enhancements you requested. For example, the Intel Harpertown processors do not support the SSE 4.2 instructions, while the Intel Nehalem and Intel Westmere processors do.

But it worked before. What changed?

Recently (the week of February 14, 2011), we changed the order in which nodes in our cluster are allocated, putting the Intel Harpertown processors first, instead of last like they were previously. If your program uses optimizations that the Intel Harpertown processors don't support, your job is much more likely now to have problems, than you were previously.

It's important to note that it was always possible that a job would end up on these nodes. It just used to be much less likely.

Okay. What do I do about it?

In general, you can choose to do one of two things:

• Avoid using those optimizations

You can simply turn of the optimizations you're using, or even explicitly include the unoptimized code. For example, if you use -axSSE4.2 as an option to the Intel compilers, rather than -xSSE4.2, then the compiler will build two code paths: An optimized path, and an unoptimized path. Then when your program runs on a processor that doesn't support the optimized path, it will fall back to the unoptimized one. It will still run, although slower.

• Request resources for your job that support the optimizations you're using

You can also just request the resources you need for your program. For example, if you want to request nodes that are either Nehalem or Westmere nodes, you could use a resource request like this:

#PBS -l nodes=2:ppn=8,walltime=30:00,pmem=1gb,feature='nehalem|westmere'

Alternatively, if you wanted to request any intel processor except Harpertown processors, you could do it like this:

#PBS -l nodes=2:ppn=8,walltime=30:00,pmem=1gb,feature='intel:!harpertown'