“Build, Ship, and Run Any App, Anywhere” is a bold statement to make but it sure does live up to it. For the past few weeks the team has been working on incorporating Docker into the UCT eResearch HPC Cluster as a proof of concept. Our base HPC OS is SLES11 and works well for us. Docker is supported in SLES12 and as such decided to update the CLOUDQ:series700 cluster nodes to SLES12. Upgrading the nodes brought about a few challenges and caveats which we noted. The new cluster for 2016 will run CentOS 7 and SLURM as its workload manager. Andrew has been working hard on understanding and documenting some of the new features and functionality associated with SLURM. Further information can be found in our blog or by using the search button located at the top right of this website.
Reasons for containerization within HPC:
Okay, lets get down to the reason for this blog entry. The team compile and install applications almost daily and what we have noticed is that every application compiles differently on every system or sometimes not at all due to the lack of certain versioned dependencies. As I understand it, containerization assists with the ability to compartmentalize a specific set of resources (libraries / binaries / versions of resources) on a host which would not neccessarly have the resources to execute it. Containers are portable and repeatable. Meaning that what you have installed in a container will always execute on any Linux host that has docker installed and will always execute with the configuration in which it initially was built. For an HPC environment it would be perfect to submit a PBS job as a HPC user which includes the ability to “ docker pull “ the application of your choice and then “ docker run application:version “. These would be short lived docker instances and would only run for as long as the application inside the container is scheduled to run for, after which the application is shutdown.
Security:
In our opinion the Docker framework was initially built with single-user features in mind. It’s intention was to only focus on developers / system administrators and not for a multi-user environment, for example HPC. So some of the security concerns associated with multi-user environments have not been implemented and therefore pose a significant risk to the host. Information about docker security can be found here – https://docs.docker.com/articles/security/
Storage:
What good is it to run a container with no data. Containers would usually mount their volumes via “ docker run application:version -v /data:/data “. This option is not recommended for a multi-user environment because of the security implications. The implication being that once the volume is mounted as user root, the container root inherits the priviledge of the host root user. It is possible to execute a container with the “docker run application:version -u nobody -v /data:/data “ which will launch a container as the user nobody. This is secure and the user nobody will only have access to the data on a readonly basis. However, your container needs the ability to write its data somewhere. The team is currently working on a docker wrapper script which will only incorporate the ability to run certain syntatical options via sudo. Will update with another blog entry once we have that running well.
The storage solution we proposed to our docker users was Ceph Object Storage using the RADOS gateway and Amazon’s S3 Toolset – http://s3tools.org/s3cmd . The docker HPC user is given access to the object storage via access key and secret key. These two keys are injected into the docker container using docker run. It would have been great to make use of “ docker cp “ but during our conceptual design you could only copy from the container to the host and not from the host to the container. Even if we have the ability to copy from the host to the container these containers are short-lived. So the keys are injected into the .s3cfg and immediately the container has the ability to push and pull data from the object storage. We then configure the scientific application to get the dataset from the object store, compute against the dataset and push the output back to the S3 bucket.
HPC docker job submission script example:
Example of our docker submission script. I am sure it can be improved lots but this is how we have done it.
#PBS -N DockerTest #PBS -l nodes=1:series700:ppn=1 #PBS -q CLOUDQ JOB=$(docker run image:version /bin/sh -c \ 'S3ACCESSKEY=<insert_access_key> && \ S3SECRETKEY=<insert_secret_key> && \ S3CFGTEMPL=<make the default s3cfg template available for download>&& \ S3DLAPP=http://za.archive.ubuntu.com/ubuntu/pool/universe/s/s3cmd/s3cmd_1.5.2-4_all.deb && \ S3CMD=s3cmd.deb && \ CEPHMON=ceph_hostname:6789 && \ CEPHSRV=ceph_hostname && \ wget $S3CFGTEMPL -O ~/.s3cfg && \ wget $S3DLAPP -O ~/$S3CMD && \ apt-get -y install python-dateutil python-magic && \ dpkg -i ~/$S3CMD && \ sed -i -e "/access_key =/ s/= .*/= $S3ACCESSKEY/" ~/.s3cfg && \ sed -i -e "/secret_key =/ s/= .*/= $S3SECRETKEY/" ~/.s3cfg && \ sed -i -e "/host_bucket =/ s/= .*/= $CEPHMON/" ~/.s3cfg && \ sed -i -e "/host_base =/ s/= .*/= $CEPHSRV/" ~/.s3cfg && \ s3cmd get <input_filename> s3://data/input/ && \ <Science application information goes here > && \ s3cmd put <job_output_filename> s3://data/output/')
Just a note that some of the commands which exist in the job could have been included during the docker build process. Submit the job to the cluster using the standard qsub command and off runs your job.
Additional options we will be working on and writing about:
- Docker MPI runs
- cgroups to ensure that the requested resources from the scheduler match up with what the container is requested todo.
- Currently this is only deployed to a small group of trusted users. Additional users will be asked to execute the docker wrapper script.
Caveats:
- The Torque PBS does not support Docker. It would have been great to have Docker options as PBS directives to submit to the cluster.
Thus far its been working well for us. Looking forward to many more tweaking and dockering.