Installing your own Python on the Cluster

Many applications on the cluster reply on the stable Python distribution and the Python packages installed with the system. Hence they are usually older than the latest versions available. Updating the system's Python or its Python packages could lead to some applications or system utilities not working so it does not get updated often.

It's possible that your application or code needs a particular Python package,
or one that is more up-to-date or a Python package that is only available in a more recent version of Python, for instance Python 2.7 or even Python 3.3. This article covers installing your own versions of Python and Python packages.

We will be using two small utility programs: virtualenv and scl. See virtualenv --help, scl --help and man scl.

Let's see what version of Python is the default system version.

hpcnode1 ~/$ which python
/usr/bin/python         <-- it's the system Python
hpcnode1 ~/$ 

hpcnode1 ~/$ python -V
Python 2.6.6            <-- it's version 2.6
hpcnode1 ~/$ 

Installing your own Local Python 2.6

This is an example of installing your own local copy of the current default Python 2.6. You might do this if the current Python version is fine but you need additional Python packages just for your own application or code.

hpcnode1 ~/$ virtualenv mypy26            <-- you can use any name you wish  
New python executable in mypy26/bin/python
Installing Setuptools...... done.
Installing Pip............. done.
hpcnode1 ~/$ 

Now you need to activate this new Python environment. Note: you must use source bin/activate from bash, you cannot run activate directly. A shortcut for source filename is . filename.

hpcnode1 ~/$ . mypy26/bin/activate
(mypy26)hpcnode1 ~/$ 

(mypy26)hpcnode1 ~/$ which python
(mypy26)hpcnode1 ~/$ 

This will have added the path to this new Python i.e. mypy26/bin to the start of your existing PATH. It will change your PROMPT so you will know you are in the new Python environment. (It will have also added VIRTUAL_ENV environment variable.)

Pip is a simple Python packaging system. You can use it to search, list, install and uninstall Python packages. Another way to install Python packages is using setuptools. This supplies an executable called easy_install. In this example we will just use pip.

$ pip list   <-- list python packages installed 
pip (1.4.1)
setuptools (0.9.8)
$ pip search fft   <-- let's search for FFT packages 
pyFFTW          - A pythonic wrapper around FFTW, the FFT library, presenting
                  a unified interface for all the supported transforms.
pyfft           - FFT library for PyCuda and PyOpenCL
FFTresize       - FFTresize resizes images using zero-padding in the frequency domain.
FFT-tools       - Wrap NumpPy's FFT routines to reduce clutter.
pyNFFT          - A pythonic wrapper around NFFT
pynfftls        - Fast Lomb-Scargle periodogram using Non-equispaced 
                  Fast Fourier Transform (NFFT) by B. Leroy
PyFFTW3         - PyFFTW: Python bindings to the FFTW3 C-library

$ pip install pyFFTW    <-- we'll install pyFFTW
... lots of compilation guff ...
Successfully installed pyFFTW
Cleaning up...

/$ python
>>> import pyfftw       <-- it's now installed and usable

The above procedure using virtualenv does a clean install of Python. There are few, if any, python packages installed. If you want a more complete Python then you can use the option --system-site-packages which will add the global site-packages directory to the virtual environment when it is created.

$ virtualenv --system-site-packages mypy26

$. mypy26/bin/activate
(mypy26)hpcnode1 ~/$

(mypy26)hpcnode1 ~/$ pip list
... lots more packages will be displayed ...
(mypy26)hpcnode1 ~/$ 

To exit this local python use deactivate.

(mypy26)hpcnode1 ~$ deactivate
hpcnode1 ~$ which python

Note: The deactivate command is not a binary, nor a script that you source; it is a shell alias that gets defined dynamically in your current shell by the activate script.

If you don't need the local Python 2.6 anymore then just delete the py26 directory.

Installing your own Local Python 2.7 or Python 3.3

Many packages need a later version that 2.6 so let's create a new Python 2.7 environment. We will first need to use a small utility called scl
("Setup and run software from Software Collection environment").

hpcnode1 ~/$ scl enable python27 bash
hpcnode1 ~/$ 

hpcnode1 ~/$ which python
/opt/rh/python27/root/usr/bin/python <-- scl has set a Python 2.7 environment for us
hpcnode1 ~/$ 

hpcnode1 ~/$ virtualenv mypy27       <-- this will clone this Python 2.7 env for us
New python executable in mypy27/bin/python
hpcnode1 ~/$ 

hpcnode1 ~/$ . mypy27/bin/activate   <-- activate this new Python environment
(mypy27)hpcnode1 ~/$ 

(mypy27)hpcnode1 ~/$ which python
(mypy27)hpcnode1 ~/$            <-- Now we could do a `pip install foo` at this stage.

Now its the same as the notes above for using virtualenv.

Possible Errors


If you try to run "pip search" and get an error like this: error while loading shared libraries: "" ... No such file or directory

Then here is how to diagnose and fix the problem.

(mypy27)ermdc06 ~/$ locate

(mypy27)ermdc06 ~/$ LD_LIBRARY_PATH=/opt/rh/python27/root/usr/lib64/
(mypy27)ermdc06 ~/$ export LD_LIBRARY_PATH

You can add these lines to your .bash_profile file so that the LD_LIBRARY_PATH will be set when you login.