Skip to main content

Python Packages with Virtual Environments

In order to be able to install new Python packages and make your work reproducible, please use virtual environments.

There is more than one way to create a private environment in Python.

Create project directory and load Python module​

## Find python version you need
module avail python
## created directory for your project and cd there
mkdir /scratch/$USER/my_project
cd /scratch/$USER/my_project
## load python module (different versions available)
module load  python/intel/3.8.6

Automatic deletion of your files​

This page describes the installation of packages on /scratch. One has to remember, though, that files stored in the HPC scratch file system are subject to the HPC Scratch old file purging policy: Files on the /scratch file system that have not been accessed for 60 or more days will be purged (read more).

Thus you can consider the following options

  • Reinstall your packages if some of the files get deleted
    • You can do this manually
    • You can do this automatically. For example, within a workflow of a pipeline software like Nextflow
  • Pay for "Research Project Space" - read more here
  • Use Singularity and install packages within a corresponding overlay file - read more here

Create virtual environment​

It is advisable to create private environment inside the project directory. This boosts reproducibility and does not use space in /home/$USER

virtualenv​

virtualenv is a tool to create isolated Python environments

Since Python 3.3, a subset of it has been integrated into the standard library under the venv module.

Note: you may need to install virtualenv first, if it is not yet installed (instructions)

Now create new virtual environment in current directory

  • Empty
  • OR
  • inherit all packages from those installed on HPC already (and available in PATH after you load python module)
## created directory for your project and cd there
mkdir /scratch/$USER/my_project
cd /scratch/$USER/my_project

## Create an EMPTY virtual environment
virtualenv venv

## Create an virtual environment that inherits system packages
virtualenv venv --system-site-packages

venv​

venv is package shipped with Python3. It provides subset of options available in virtualenv tool (link).

python3 -m venv venv

Create new virtual environment in current directory

  • Empty
  • OR
  • inherit all packages from those installed on HPC already (and available in PATH after you load python module)
## created directory for your project and cd there
mkdir /scratch/$USER/my_project
cd /scratch/$USER/my_project
##EMPTY
## (use venv command to create environment called "venv")

python3 -m venv venv

## Inhering all packages
python3 -m venv venv --system-site-packages

Install packages. Keep things reproducible​

## activate
source venv/bin/activate
## install packages
pip install <package you need>
## If package was inherited, but you want to install it in your own env anyway
pip install <package you need> --ignore-installed
## export list of packages (to report together with paper and/or to reproduce environment on another computer)
pip freeze > requirements.txt
## restore
pip install -r requirements.txt

Close an Activated Virtual Environment​

If you have activated a virtual environment, you can exit it with the following command:

deactivate

Use with sbatch​

When you use this env in sbatch script, please use

module purge;
source venv/bin/activate;
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK;
python python_script.py

If you use mpi

mpiexec  bash -c "module purge;
                  source venv/bin/activate;
                  export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK;
                  python python_script.py"