The UCT HPC cluster has a wide range of software installed. Occasionally these need to be extended to add additional functionality by adding packages or libraries. We maintain ‘base’ versions of R and Python (miniconda) which can be activated with the module command. We also have extended versions of R and Python into which we will install libraries for you. Module versions of R and Python followed by -usr have additional libraries. For example python/miniconda3-py3.12 is the base version of miniconda with Python 3.12 while python/miniconda3-py3.12-usr has additional libraries. Modules python/miniconda3-py39-picrust2 and python/qiime2-metagenome are examples dedicated to a specific package.
Do not copy\paste these instructions, they are examples only. Please contact us before attempting any of the instructions below if you are unsure how to proceed.
Important
The methods outlined below induce high load on the server they are run on due to the fact that code compilation occurs. Please do not do this on the head node, you must start an interactive job and then exit the job once the packages are installed. An example of starting an interactive job:
sintx --ntasks=4
Don’t forget to type exit to end your interactive job once the installation is complete.
R
To install packages locally the following method can be used:
Start an interactive job and activate the version of R you require with module
module load software/R-3.6.0 or on the new cluster module load R/R-4.3.3
Start R and install the package
install.packages("XYZ")
You will immediately be challenged by the installer as you do not have write rights to the software location
Warning in install.packages("XYZ") :
'lib = "/opt/exp_soft/R-3.6.0/lib64"' is not writable
Would you like to use a personal library instead? (yes/No/cancel)
Enter yes, the package will then download, compile and install. You can then load the library.
library(XYZ)
Once this is confirmed as working you no longer need to install the package each time you start R, just load the library.
To install packages locally from github use the following:
install.packages("devtools")
library(devtools)
withr::with_libpaths(new="/home/myusername/R/x86_64-pc-linux-gnu-library/4.2", devtools::install_github('AUTHOR/PACKAGE') )
…where myusername is your username and 4.2 is the version of R you’re using. To check the version number look in your R/x86_64-pc-linux-gnu-library folder.
Python pip
The pip package management tool can be used to install packages locally.
module load python/miniconda3-py310 # NB THIS IS AN EXAMPLE, DON'T JUST COPY AND PASTE! pip install --user PackageName
If a version of the package you’re trying to install exists in /opt/exp_soft already then pip will not install locally as the package already exists from its point of view. To get around this add the –ignore-installed argument.
pip install --user PackageName --ignore-installed
Python conda virtual environment
Python has a specific tool used to install packages, conda. Python also comes with a mechanism to set up virtual environments. Below is an example of setting up the short read aligner bowtie:
Start an interactive job, then:
module load python/miniconda3-py310 # NB THIS IS AN EXAMPLE, DON'T JUST COPY AND PASTE! conda create -y -n RelevantName source activate RelevantName conda config --add channels defaults conda config --add channels bioconda conda config --add channels conda-forge conda install bowtie conda deactivate exit
The above only needs to be done once. The term RelevantName should be a short text word describing your project\environment. Additionally the channel names are for bowtie specifically. Your package may require different channels. Now when you launch jobs your script should contain:
module load python/miniconda3-py310 # NB THIS IS AN EXAMPLE, DON'T JUST COPY AND PASTE! source activate RelevantName
I want a very specific version of Python
Occasionally a Python library requires one specific version of Python. It is possible to set up a conda environment that overwrites the base version of Python. When creating a conda virtual environment as shown above, just add the required Python version to the create command:
conda create -y -n RelevantName python=3.6.5
Python conda Singularity Containers
I’m using a Singularity container and can’t add python libraries as conda\pip was not added to the Singularity container by the person who created it. (#21stCenturyProblems)
You’re going to need to make pip available. Start an interactive job and then run:
module load [NameOfModuleForSingularityContainer] wget https://bootstrap.pypa.io/get-pip.py python get-pip.py --user
You have now installed your own version of pip which you can use with the singularity version of python. Now you can add your own libraries, for example to install Pandas run:
python -m pip install Pandas --user
Now you can run python and import the library:
python Python [VER] >>> import pandas as pd >>>
The libraries will be installed into $HOME/.local/lib/python[VER]/site-packages
where [VER] is the version of python in the Singularity container.