Docker

Unfortunately Docker is still a gaping security hole on multi-user systems. It must run as root and there's nothing we can do about it as far as we know. Conceptually it would be simple to do the initial setup as root then drop privileges to the user who is requesting to run the container, but they haven't made that a priority (or even added it to the TODO list, as far as we know).

https://fosterelli.co/privilege-escalation-via-docker.html
https://docs.docker.com/articles/security/#docker-daemon-attack-surface

There has been progress (maybe completed by now?) on allowing a user in a container to look like they are root but they are not actually root on the host. This is an orthogonal issue. Docker is set up to allow anyone who uses it to get root on the host. When launching a Docker instance you can still get root access on the host even if you then later want a user inside the container to look like it's root even when it isn't.

Anyone added to the docker group essentially has root access, thus making Docker unusable here.

Aside from security issues, code optimization is still an issue. We're guessing that it's desirable to you and others since it would make for easy rpm/deb package installation. Those packages are often compiled to the lowest-supported common denominator, often a CPU architecture that is many, many years old. Due to Docker's design goals (the reason Docker exists), the containers don't use external libraries. This means they won't link with any of the optimized libraries that we have installed.

We wouldn't be surprised to see anywhere from 20-70% slowdowns for most code. Using some test code we have, we can easily come up with a likely scenario where the unoptimized compilation (which is what an rpm/deb would usually have) is half the speed of the optimized compilation. Even recompiling code targeted for m6 to target m7 will often result in a 10-30% speedup, and those clusters were only purchased two years apart.