Pythonbrew, pip and github – Circumventing Issues on CentOS 5.8

Python

I wrote a post a little while ago raving about the benefits of using Perlbrew to manage your perl installation and, crucially, allow you to install additional modules without the need of administer access to the machine.

Well, the Perlbrew project inspired a group of people to work on Pythonbrew, which is broadly similar but for python. It comes with pip, a really easy to use installer module that allows you to grab packages from PyPI (Python Package Index). Naturally, it working well on my system (Ubuntu 12.04) led me to recommend it to some of my colleagues for working with Python. And, naturally, it had issues. And, just to make things even more frustrating, the machine that they were using runs on an old version of CentOS (5.8) with what feels like nothing installed. Oh, and they didn’t have administrator privileges, so I couldn’t install anything without configuring it all for their home directory! URGH

Some modules, like matplotlib installed really quickly and cleanly:

$pip install matplotlib
...
Successfully installed matplotlib
Cleaning up...

The real issues came with ‘pip install numpy’ (and scipy), which seemed to fail on patching together some C libraries, with an end result:

Command /home/USER/.pythonbrew/pythons/Python-2.7.2/bin/python -c "import setuptools;__file__='/home/USER/build/numpy/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-2RdvJ1-record/install-record.txt failed with error code 1 in /home/USER/build/numpy
Storing complete log in /home/USER/.pip/pip.log

Short to say that the compiler “Failed with exit status 1′. On the machine that I was working on, this was a major pain in the arse. Plus, I had no way of knowing if it was merely that the version of libc was old or that I hadn’t included the correct library path. However, someone on a message forum suggested that I use the ‘bleeding edge’ github version of numpy as the issue might have been fixed. Low and behold:

$pip install git+git://github.com/numpy/numpy.git
...
Successfully installed numpy
Cleaning up...
$pip install git+git://github.com/scipy/scipy.git
...
Successfully installed scipy
Cleaning up...

Naice. I wouldn’t make a habit of using the github version of a project, but if you’re finding that you’re having issues with installing from PyPI then maybe give it a go!

Finally, MySQLdb also played up on the system that I was working on (again because of C libraries not linking). However, I came across PyMySQL, which is a pure python implementation of a MySQL interface, which is designed to be a ‘drop-in’ replacement to MySQLdb. As the project git wiki states:

“MySQLdb, is a C extension module that has a reputation of being difficult to compile, especially if you’re on a Mac. Additionally, end-users need to wait for new binaries to be compiled for each new release of Python [...]“

“It [PyMySQL] is also a drop-in replacement for the standard way of connecting to MySQL in Python, MySQLdb. Just import pymysql; pymysql.install_as_MySQLdb() in any existing project before MySQLdb is imported, and it’ll work.”

As it’s pure python, you’re already having issues with C linker libraries then perhaps try something written in pure python. ‘pip install pymysql’ installed it nicely on the ‘problem’ system nicely.

8 Responses

  1. Adam Zienkiewicz says:

    Big thanks for that Adam!

    I got stuck at the same place trying to pip install numpy with the same failures. I get to easily lost on the forums and don’t really have the patience.

    Why on earth are these computers not set up / are we given rights to install the software we NEED to undertake our work. I’d rather they were all standalone boxes without any network connection!

    Anyway, thanks very much for that!

    Ad

    • Adam says:

      It is a perennial problem! However, with pythonbrew on there you should be able to progress with your research without the need of root access. Fingers crossed anyway!

  2. Adam Zienkiewicz says:

    One major problem however – git is not installed on these computers either it seems!!

    Is this easy to rectify??

    Ad

    • Adam says:

      If you go to here: https://github.com/git/git

      And download the ‘zip’ of the file. It will go into ‘Downloads’ by default. Then:

      cd ~/Downloads *Or cd into wherever you downloaded it
      unzip git-git-v1.7.12-rc2-9-ge5acacf.zip *NB, the file name might have changed, depending on whether they update the github files
      cd git-git-e5acacf
      make && make install

      This will install git into your ~/bin/ folder

  3. Adam Zienkiewicz says:

    Hallo again! Hilarious Friday in Gloucester by the way – loved it! haaa!

    Wondering if you might have a similar trick up your sleeve for another install I’m having trouble with.. Open MPI.

    I’ve tried the following advice at: https://igppwiki.ucsd.edu/groups/publichelpwiki/wiki/e509a/Installing_OpenMPI_15_in_Your_Home_Directory.html

    but for some reason the configure still doesn’t seem to be making the right MAKEFILES as when I do make all install, it quits straight away as if the correct files aren’t there..

    All this guff is really starting to get on my tits – give me a standalone 8core box any day!!

    Ad

    • Adam says:

      In honesty I have not. A quick google seems to imply that it’s not dissimilar to the mac ports project (which I do use) and often works fine.

      However, installing a package with a package manager (such as aptitude or yum or whatever) means that you are playing with the system version of python, which means that you need root access. Pythonbrew allows you, the user, to have an install of python that is separate to the system version – so you can’t mess up the core workings of your machine!

Leave a Reply to Adam Zienkiewicz Cancel reply

*