{"id":3735,"date":"2019-10-21T09:02:04","date_gmt":"2019-10-21T07:02:04","guid":{"rendered":"http:\/\/ucthpc.uct.ac.za\/?page_id=3735"},"modified":"2025-06-16T14:42:18","modified_gmt":"2025-06-16T12:42:18","slug":"adding-libraries","status":"publish","type":"page","link":"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/","title":{"rendered":"Adding libraries"},"content":{"rendered":"
\r\n\t
\r\n\t\t
\r\n\t\t\t

The UCT HPC cluster has a wide range of software installed. Occasionally these need to be extended to add additional functionality by adding packages or libraries. We maintain ‘base’ versions of R and Python (miniconda) which can be activated with the module command<\/a>. We also have extended versions of R and Python into which we will install libraries for you. Module versions of R and Python followed by -usr<\/em> have additional libraries. For example python\/miniconda3-py3.12<\/em> is the base version of miniconda with Python 3.12 while python\/miniconda3-py3.12-usr<\/em> has additional libraries.\u00a0 Modules python\/miniconda3-py39-picrust2<\/em> and python\/qiime2-metagenome<\/em>\u00a0are examples dedicated to a specific package.<\/p>\n

Do not copy\\paste these instructions, they are examples<\/span> only.\u00a0 <\/strong><\/span><\/p>\n

Do NOT<\/span> run installs on the head node, installs must be run as jobs, interactive<\/a> if needed.\u00a0<\/strong><\/span><\/p>\n

Please contact us before attempting any of the instructions below if you are unsure how to proceed.<\/strong><\/span><\/p>\n

 <\/p>\n

Important<\/strong><\/em>
\nThe methods outlined below induce high load on the server they are run on due to the fact that code compilation occurs. Please do not do this on the head node<\/strong><\/span><\/span>, you must start an interactive job<\/a> and then exit the job once the packages are installed. An example of starting an interactive job:<\/p>\n

sintx --ntasks=4<\/pre>\n
Don’t forget to type exit to end your interactive job once the installation is complete.<\/p>\n
 <\/p>\n
R<\/strong><\/p>\n
To install packages locally the following method can be used:<\/p>\n
Start an interactive job<\/a> and activate the version of R you require with module<\/p>\n
module load software\/R-3.6.0       or on the new cluster     module load R\/R-4.3.3<\/pre>\n
Start R and install the package<\/p>\n
install.packages(\"XYZ\")<\/pre>\n
You will immediately be challenged by the installer as you do not have write rights to the software location<\/p>\n
Warning in install.packages(\"XYZ\") :\r\n'lib = \"\/opt\/exp_soft\/R-3.6.0\/lib64\"' is not writable\r\nWould you like to use a personal library instead? (yes\/No\/cancel)<\/pre>\n
Enter yes, the package will then download, compile and install. You can then load the library.<\/p>\n
library(XYZ)<\/pre>\n
Once this is confirmed as working you no longer need to install the package each time you start R, just load the library.<\/p>\n
 <\/p>\n
To install packages locally from github use the following:<\/p>\n
install.packages(\"devtools\")\r\nlibrary(devtools)\r\nwithr::with_libpaths(new=\"\/home\/myusername\/R\/x86_64-pc-linux-gnu-library\/4.2\", devtools::install_github('AUTHOR\/PACKAGE') )<\/pre>\n
…where myusername is your username and 4.2 is the version of R you’re using.\u00a0 To check the version number look in your R\/x86_64-pc-linux-gnu-library folder.<\/p>\n
 <\/p>\n
\n
 <\/p>\n
Python pip<\/strong><\/p>\n
The pip package management tool can be used to install packages locally.<\/p>\n
module load python\/miniconda3-py310    # NB THIS IS AN EXAMPLE, DON'T JUST COPY AND PASTE!\r\npip install --user PackageName<\/pre>\n
If a version of the package you’re trying to install exists in \/opt\/exp_soft already then pip will not install locally as the package already exists from its point of view. To get around this add the –ignore-installed argument.<\/p>\n
pip install --user PackageName --ignore-installed<\/pre>\n
 <\/p>\n
Python conda virtual environment<\/strong><\/p>\n
Python has a specific tool used to install packages, conda. Python also comes with a mechanism to set up virtual environments. Below is an example of setting up the short read aligner bowtie:<\/p>\n
Start an interactive job<\/a>, then:<\/p>\n
module load python\/miniconda3-py3.12       # NB THIS IS AN EXAMPLE, DON'T JUST COPY AND PASTE!\r\nconda create -y -n RelevantName\r\nsource activate RelevantName\r\nconda config --add channels defaults\r\nconda config --add channels bioconda\r\nconda config --add channels conda-forge\r\nconda install bowtie\r\nconda deactivate\r\nexit<\/pre>\n
The above only needs to be done once. The term RelevantName should be a short text word describing your project\\environment. Additionally the channel names are for bowtie specifically. Your package may require different channels. Now when you launch jobs your script should contain:<\/p>\n
module load python\/miniconda3-py310 # NB THIS IS AN EXAMPLE, DON'T JUST COPY AND PASTE! \r\nsource activate RelevantName<\/pre>\n
 <\/p>\n
I want a very specific version of Python<\/strong><\/p>\n
Occasionally a Python library requires one specific version of Python. It is possible to set up a conda environment that overwrites the base version of Python. When creating a conda virtual environment as shown above, just add the required Python version to the create command:<\/p>\n
conda create -y -n RelevantName python=3.6.5<\/pre>\n
 <\/p>\n
Python conda Singularity Containers<\/strong><\/p>\n
I’m using a Singularity container and can’t add python libraries as conda\\pip was not added to the Singularity container by the person who created it. (#21stCenturyProblems)<\/p>\n
You’re going to need to make pip available. Start an interactive job<\/a> and then run:<\/p>\n
module load [NameOfModuleForSingularityContainer]\r\nwget https:\/\/bootstrap.pypa.io\/get-pip.py\r\npython get-pip.py --user<\/pre>\n
You have now installed your own version of pip which you can use with the singularity version of python. Now you can add your own libraries, for example to install Pandas run:<\/p>\n
python -m pip install Pandas --user<\/pre>\n
Now you can run python and import the library:<\/p>\n
python\r\nPython [VER]\r\n>>> import pandas as pd\r\n>>><\/pre>\n
The libraries will be installed into $HOME\/.local\/lib\/python[VER]\/site-packages<\/p>\n
where [VER] is the version of python in the Singularity container.<\/p>\n\r\n\t\t<\/div>\r\n\t<\/div>\r\n<\/div><\/div><\/div><\/div><\/div><\/section>\n","protected":false},"excerpt":{"rendered":"The UCT HPC cluster has a wide range of software installed. Occasionally these need to be extended to add additional functionality by adding packages or libraries. We maintain 'base' versions of R and Python (miniconda) which can be activated with the module command. We also have extended versions of R and Python into which we...","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"yoast_head":"\nAdding libraries - UCT HPC<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Adding libraries - UCT HPC\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/\" \/>\n<meta property=\"og:site_name\" content=\"UCT HPC\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-16T12:42:18+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/\",\"name\":\"Adding libraries - UCT HPC\",\"isPartOf\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\"},\"datePublished\":\"2019-10-21T07:02:04+00:00\",\"dateModified\":\"2025-06-16T12:42:18+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/#breadcrumb\"},\"inLanguage\":\"en-ZA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ucthpc.uct.ac.za\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Adding libraries\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#website\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"name\":\"UCT HPC\",\"description\":\"University of Cape Town High Performance Computing\",\"publisher\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-ZA\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#organization\",\"name\":\"University of Cape Town High Performance Computing\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-ZA\",\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"contentUrl\":\"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png\",\"width\":450,\"height\":423,\"caption\":\"University of Cape Town High Performance Computing\"},\"image\":{\"@id\":\"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Adding libraries - UCT HPC","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/","og_locale":"en_US","og_type":"article","og_title":"Adding libraries - UCT HPC","og_url":"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/","og_site_name":"UCT HPC","article_modified_time":"2025-06-16T12:42:18+00:00","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/","url":"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/","name":"Adding libraries - UCT HPC","isPartOf":{"@id":"https:\/\/ucthpc.uct.ac.za\/#website"},"datePublished":"2019-10-21T07:02:04+00:00","dateModified":"2025-06-16T12:42:18+00:00","breadcrumb":{"@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/#breadcrumb"},"inLanguage":"en-ZA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ucthpc.uct.ac.za\/index.php\/adding-libraries\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ucthpc.uct.ac.za\/"},{"@type":"ListItem","position":2,"name":"Adding libraries"}]},{"@type":"WebSite","@id":"https:\/\/ucthpc.uct.ac.za\/#website","url":"https:\/\/ucthpc.uct.ac.za\/","name":"UCT HPC","description":"University of Cape Town High Performance Computing","publisher":{"@id":"https:\/\/ucthpc.uct.ac.za\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ucthpc.uct.ac.za\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-ZA"},{"@type":"Organization","@id":"https:\/\/ucthpc.uct.ac.za\/#organization","name":"University of Cape Town High Performance Computing","url":"https:\/\/ucthpc.uct.ac.za\/","logo":{"@type":"ImageObject","inLanguage":"en-ZA","@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/","url":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","contentUrl":"https:\/\/ucthpc.uct.ac.za\/wp-content\/uploads\/2015\/09\/logocircless.png","width":450,"height":423,"caption":"University of Cape Town High Performance Computing"},"image":{"@id":"https:\/\/ucthpc.uct.ac.za\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/pages\/3735"}],"collection":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/comments?post=3735"}],"version-history":[{"count":28,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/pages\/3735\/revisions"}],"predecessor-version":[{"id":5000,"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/pages\/3735\/revisions\/5000"}],"wp:attachment":[{"href":"https:\/\/ucthpc.uct.ac.za\/index.php\/wp-json\/wp\/v2\/media?parent=3735"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}