py2app, MacPorts, Snow Leopard, and PIL: a match made in Hell

For a while, I’ve been needing to create a little app out of some python scripts I’d been hacking on so I could have others use them easily. For those of you not familiar with it, py2app generates the equivalent of a statically-linked binary using python files and the transitive closure of all the libraries they require. This feature is pretty much standard for most compiled languages, and its presence makes python a pretty compelling development platform. (Actually, making a statically-linked binary on Mac OS is also really difficult, for reasons that are a bit outside the scope of this blog post).

It seems as though this shouldn’t be difficult; like most development efforts on the Mac, one is easily lulled into a false sense of security (“It looks just like Linux; I bet everything works the same way!”). This is one of those times when you’re rudely jolted back to the sad, disastrous reality.

I’d recently installed Snow Leopard on both my Macs, so I was already familiar with things randomly failing for no obvious reason, but I was not prepared for challenges of this magnitude! For those of you in a hurry, I’m going to just explain what you have to do to get this stuff to work properly. The curious (or should I say brazen?) can hear some more details later on in the post.

How to get py2app to work with PIL, numpy, etc on Snow Leopard

Here are some things that I’ve spent a lot of time figuring out:

  • 64-bit python with Snow Leopard breaks pretty much everything.
  • You can’t use system Python (see below for what this is) because py2app refuses to build a standalone version with it, and because of the whole 64-bit problem. If you try, you’ll probably get errors like this:

    File "build/bdist.macosx-10.6-intel/egg/macholib/", line
    48, in from_str
    return cls.from_tuple(struct.unpack(endian + cls._format_, s), **kw)
    error: unpack requires a string argument of length 8
    > /Users/vijayp/src/build/py2app/examples/simple/build/bdist.macosx-10.6-intel/egg/macholib/
    -> return cls.from_tuple(struct.unpack(endian + cls._format_, s), **kw)

    or this

    4/6/10 5:07:10 PM[186] ([0x0-0x1d51d5].org.pythonmac.unspecified.hello[54121]) Exited with exit code: 255

    or a bunch of other things.
  • Forget about mac ports. Without a huge number of patches, altered configuration files, and changes you have to make yourself to source files, it will never work.

The solution is to use one of the pre-compiled binaries built without 64-bit support. Here’s how:

  1. To be safe, eliminate all traces of your various python installations to ensure that things don’t get confused. Here are the Mac Ports Uninstall instructions.
  2. You might also want to blow away everything under /Library/Frameworks/Python.framework/ if you’re paranoid.
  3. If you want to use object libraries like PIL, you’re going to have to use python 2.5, so if you’re hoping for something newer, you’re kind of out of luck. Get
    the Universal python 2.5 install image from the official python site. Install it.
  4. Close and re-open your terminal (to refresh your profile and bashrc), then ensure that your python is set up correctly:

    terrance:~ vijayp$ python
    Python 2.5.4 (r254:67917, Dec 23 2008, 14:57:27)
    [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    terrance:~ vijayp$ which python
    terrance:~ vijayp$ file -L `which python`
    /Library/Frameworks/Python.framework/Versions/Current/bin/python: Mach-O universal binary with 2 architectures
    /Library/Frameworks/Python.framework/Versions/Current/bin/python (for architecture i386): Mach-O executable i386
    /Library/Frameworks/Python.framework/Versions/Current/bin/python (for architecture ppc): Mach-O executable ppc
    terrance:~ vijayp$ openssl sha1 `which python`
    SHA1(/Library/Frameworks/Python.framework/Versions/Current/bin/python)= 675b28e55955734aba9e4df5d1ddd7d48ff9c83b

  5. Pull some stuff from svn

    export BASE_DIR=/tmp
    mkdir $BASE_DIR/build
    cd $BASE_DIR/build
    svn co py2app
    svn co setuptools

  6. Build it

    cd $BASE_DIR/build/setuptools
    python ./ build
    sudo python ./ install
    cd ../py2app/trunk
    python ./ build
    sudo python ./ install

  7. Now, you should have everything you need to build a little simple app without much trouble. First, verify that python and setuptools and py2app are installed correctly:

    terrance:simple vijayp$ python
    Python 2.5.4 (r254:67917, Dec 23 2008, 14:57:27)
    [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import setuptools

  8. Next up, try to get the example to work:

    cd examples/simple
    rm -rf build dist *.app
    python ./ py2app
    open dist/

    If you didn’t get any errors, but an app just flashed in and out, you’re in business. If you really care, you can open up the console log (in the Console app). You should have a bunch of environment variables printed.
  9. Building the 3rd party libraries from source shouldn’t be that difficult, but luckily, py25 libraries have mostly been built. Try installing PIL, then building the PIL example. It should work correctly.

Hopefully this little tutorial was helpful. Please let me know if it worked, or if it didn’t.

Gory, ugly details

Mac OS comes with a built-in “System Python,” located in /usr/local/bin. As you can see, it’s compiled for three platforms, 64-bit, 32-bit, and ppc:

terrance:~ vijayp$ file /usr/bin/python
/usr/bin/python: Mach-O universal binary with 3 architectures
/usr/bin/python (for architecture x86_64): Mach-O 64-bit executable x86_64
/usr/bin/python (for architecture i386): Mach-O executable i386
/usr/bin/python (for architecture ppc7400): Mach-O executable ppc

py2app requires a couple of generic system libraries contained in the setuptools package, especially macholib, this library that can interact with OsX object files. There are many instructions online about how to install these; I tried a number of different things, and a combination of easy_install, compiling source, and port install packages sort of worked. Unfortunately, I was never able to compile or install a PIL that would embed its .so into the python archive. I didn’t spend too much time on it, though, since I really needed a standalone python archive. Apparently, a Mr. Brant Faircloth (cool name!) figured this out at some point, but I wasn’t able to get it to work

Next up, I tried to use Mac Ports to install python. I tried following Aral Balkan’s instructions, and a number of other random ones. I tried compiling from source, using python-25 ports, python-26 ports, and even python-24 ports, and all failed with one of the two errors I quoted above. I tried installing universal versions of them using sudo port install python-25 +universal, and also by setting the architecture to i386 in the macports configuration file.

The most common error I saw was this:

File "build/bdist.macosx-10.6-intel/egg/macholib/", line
48, in from_str
return cls.from_tuple(struct.unpack(endian + cls._format_, s), **kw)
error: unpack requires a string argument of length 8
> /Users/vijayp/src/build/py2app/examples/simple/build/bdist.macosx-10.6-intel/egg/macholib/
-> return cls.from_tuple(struct.unpack(endian + cls._format_, s), **kw)

I spent a couple of hours perusing the source — it helped me get a clearer picture of the issue. It appears to be the result of parts of the macholib code having trouble pulling information from object files that have both 64- and 32-bit versions in them. I’m not sure why; perhaps it’s because both architectures are viable?

Sadly, figuring this out fully would take me a week, and my time is limited. I’m sure an expert could figure this out soon. I worked with a bunch of incredible python gurus at Google, and some of you guys read my blog! Hint hint!

(While we’re on this subject, the lack of strace has been bothering me for some time. It turns out dtruss is a reasonable replacement.)






5 responses to “py2app, MacPorts, Snow Leopard, and PIL: a match made in Hell”

  1. […] macholib will fail and cause errors while building your app. They will also tell you to give up and fall back to Python 2.5. No need for that. The latest trunk of py2app and macholib have been fixed to work on both i386 and […]

  2. Iacopo Papalini Avatar
    Iacopo Papalini

    Wonderful yet simple: I think I owe you a lot of hours of research.
    The only shame is the old version of Python you have to use this way, but I think I can survive with it 🙂

  3. Matt Avatar

    I’ve got a simple wxPython app that I packaged with py2app, which worked fine on Leopard (10.5) but not Snow Leopard.

    The fix I ended up using — I don’t remember where I got this, but it’s jammed in the top of the makefile where I invoke “python py2app”:

    PYTHON = python2.5 # force use of 32-bit version on Snow Leopard

    So I can agree with your comment “64-bit python with Snow Leopard breaks pretty much everything”, but I found a simple way to use an acceptable 32-bit version of Python included with Snow Leopard.

  4. Stephen Avatar

    Thanks – after a day of trying to fight this on my own, this article helped me get my app up and running. Love the title – hit the nail on the head there.

  5. Adrian Avatar

    Yeap, it works. After several hours of work, but it works 😀

    Thank you

Leave a Reply

Your email address will not be published. Required fields are marked *