Installing your own Python on the Cluster
Many applications on the cluster reply on the stable Python distribution and the Python packages installed with the system. Hence they are usually older than the latest versions available. Updating the system's Python or its Python packages could lead to some applications or system utilities not working so it does not get updated often.
It's possible that your application or code needs a particular Python package,
or one that is more up-to-date or a Python package that is only available in a more recent version of Python, for instance Python 2.7 or even Python 3.3. This article covers installing your own versions of Python and Python packages.
We will be using two small utility programs: virtualenv and scl. See
scl --help and
Let's see what version of Python is the default system version.
hpcnode1 ~/$ which python /usr/bin/python <-- it's the system Python hpcnode1 ~/$ hpcnode1 ~/$ python -V Python 2.6.6 <-- it's version 2.6 hpcnode1 ~/$
Installing your own Local Python 2.6
This is an example of installing your own local copy of the current default Python 2.6. You might do this if the current Python version is fine but you need additional Python packages just for your own application or code.
hpcnode1 ~/$ virtualenv mypy26 <-- you can use any name you wish New python executable in mypy26/bin/python Installing Setuptools...... done. Installing Pip............. done. hpcnode1 ~/$
Now you need to activate this new Python environment. Note: you must use
source bin/activate from bash, you cannot run activate directly. A shortcut for
source filename is
hpcnode1 ~/$ . mypy26/bin/activate (mypy26)hpcnode1 ~/$ (mypy26)hpcnode1 ~/$ which python ~/mypy26/bin/python (mypy26)hpcnode1 ~/$
This will have added the path to this new Python i.e.
mypy26/bin to the start of your existing PATH. It will change your PROMPT so you will know you are in the new Python environment. (It will have also added VIRTUAL_ENV environment variable.)
Pip is a simple Python packaging system. You can use it to search, list, install and uninstall Python packages. Another way to install Python packages is using setuptools. This supplies an executable called easy_install. In this example we will just use pip.
$ pip list <-- list python packages installed pip (1.4.1) setuptools (0.9.8) $ $ pip search fft <-- let's search for FFT packages pyFFTW - A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms. pyfft - FFT library for PyCuda and PyOpenCL FFTresize - FFTresize resizes images using zero-padding in the frequency domain. FFT-tools - Wrap NumpPy's FFT routines to reduce clutter. pyNFFT - A pythonic wrapper around NFFT pynfftls - Fast Lomb-Scargle periodogram using Non-equispaced Fast Fourier Transform (NFFT) by B. Leroy PyFFTW3 - PyFFTW: Python bindings to the FFTW3 C-library $ $ pip install pyFFTW <-- we'll install pyFFTW ... lots of compilation guff ... Successfully installed pyFFTW Cleaning up... $ /$ python >>> import pyfftw <-- it's now installed and usable >>>
The above procedure using virtualenv does a clean install of Python. There are few, if any, python packages installed. If you want a more complete Python then you can use the option
--system-site-packages which will add the global site-packages directory to the virtual environment when it is created.
$ virtualenv --system-site-packages mypy26 $ $. mypy26/bin/activate (mypy26)hpcnode1 ~/$ (mypy26)hpcnode1 ~/$ pip list ... lots more packages will be displayed ... (mypy26)hpcnode1 ~/$
To exit this local python use
(mypy26)hpcnode1 ~$ deactivate hpcnode1 ~$ which python /usr/bin/python $
deactivate command is not a binary, nor a script that you
source; it is a shell alias that gets defined dynamically in your current shell by the
If you don't need the local Python 2.6 anymore then just delete the
Installing your own Local Python 2.7 or Python 3.3
Many packages need a later version that 2.6 so let's create a new Python 2.7 environment. We will first need to use a small utility called scl
("Setup and run software from Software Collection environment").
hpcnode1 ~/$ scl enable python27 bash hpcnode1 ~/$ hpcnode1 ~/$ which python /opt/rh/python27/root/usr/bin/python <-- scl has set a Python 2.7 environment for us hpcnode1 ~/$ hpcnode1 ~/$ virtualenv mypy27 <-- this will clone this Python 2.7 env for us New python executable in mypy27/bin/python hpcnode1 ~/$ hpcnode1 ~/$ . mypy27/bin/activate <-- activate this new Python environment (mypy27)hpcnode1 ~/$ (mypy27)hpcnode1 ~/$ which python ~/mypy27/bin/python (mypy27)hpcnode1 ~/$ <-- Now we could do a `pip install foo` at this stage.
Now its the same as the notes above for using virtualenv.
If you try to run "pip search" and get an error like this:
error while loading shared libraries: "libpython2.7.so.1.0" ... No such file or directory
Then here is how to diagnose and fix the problem.
(mypy27)ermdc06 ~/$ locate libpython2.7.so.1.0 /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0 (mypy27)ermdc06 ~/$ LD_LIBRARY_PATH=/opt/rh/python27/root/usr/lib64/ (mypy27)ermdc06 ~/$ export LD_LIBRARY_PATH
You can add these lines to your
.bash_profile file so that the LD_LIBRARY_PATH will be set when you login.
LD_LIBRARY_PATH=/opt/rh/python27/root/usr/lib64/ export LD_LIBRARY_PATH