py2app, MacPorts, Snow Leopard, and PIL: a match made in Hell

For a while, I’ve been needing to create a little app out of some python scripts I’d been hacking on so I could have others use them easily. For those of you not familiar with it, py2app generates the equivalent of a statically-linked binary using python files and the transitive closure of all the libraries they require. This feature is pretty much standard for most compiled languages, and its presence makes python a pretty compelling development platform. (Actually, making a statically-linked binary on Mac OS is also really difficult, for reasons that are a bit outside the scope of this blog post).

It seems as though this shouldn’t be difficult; like most development efforts on the Mac, one is easily lulled into a false sense of security (“It looks just like Linux; I bet everything works the same way!”). This is one of those times when you’re rudely jolted back to the sad, disastrous reality.

I’d recently installed Snow Leopard on both my Macs, so I was already familiar with things randomly failing for no obvious reason, but I was not prepared for challenges of this magnitude! For those of you in a hurry, I’m going to just explain what you have to do to get this stuff to work properly. The curious (or should I say brazen?) can hear some more details later on in the post.

How to get py2app to work with PIL, numpy, etc on Snow Leopard

Here are some things that I’ve spent a lot of time figuring out:

• 64-bit python with Snow Leopard breaks pretty much everything.
• You can’t use system Python (see below for what this is) because py2app refuses to build a standalone version with it, and because of the whole 64-bit problem. If you try, you’ll probably get errors like this:
  
File "build/bdist.macosx-10.6-intel/egg/macholib/ptypes.py", line
48, in from_str
return cls.from_tuple(struct.unpack(endian + cls._format_, s), **kw)
error: unpack requires a string argument of length 8
> /Users/vijayp/src/build/py2app/examples/simple/build/bdist.macosx-10.6-intel/egg/macholib/ptypes.py(48)from_str()
-> return cls.from_tuple(struct.unpack(endian + cls._format_, s), **kw)

  or this
  
4/6/10 5:07:10 PM com.apple.launchd.peruser.501[186] ([0x0-0x1d51d5].org.pythonmac.unspecified.hello[54121]) Exited with exit code: 255

  or a bunch of other things.
• Forget about mac ports. Without a huge number of patches, altered configuration files, and changes you have to make yourself to source files, it will never work.

The solution is to use one of the pre-compiled binaries built without 64-bit support. Here’s how:

1. To be safe, eliminate all traces of your various python installations to ensure that things don’t get confused. Here are the Mac Ports Uninstall instructions.
2. You might also want to blow away everything under /Library/Frameworks/Python.framework/ if you’re paranoid.
3. If you want to use object libraries like PIL, you’re going to have to use python 2.5, so if you’re hoping for something newer, you’re kind of out of luck. Get
   the Universal python 2.5 install image from the official python site. Install it.
4. Close and re-open your terminal (to refresh your profile and bashrc), then ensure that your python is set up correctly:
   
terrance:~ vijayp$ python
Python 2.5.4 (r254:67917, Dec 23 2008, 14:57:27)
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
terrance:~ vijayp$ which python
/Library/Frameworks/Python.framework/Versions/Current/bin/python
terrance:~ vijayp$ file -L `which python`
/Library/Frameworks/Python.framework/Versions/Current/bin/python: Mach-O universal binary with 2 architectures
/Library/Frameworks/Python.framework/Versions/Current/bin/python (for architecture i386): Mach-O executable i386
/Library/Frameworks/Python.framework/Versions/Current/bin/python (for architecture ppc): Mach-O executable ppc
terrance:~ vijayp$ openssl sha1 `which python`
SHA1(/Library/Frameworks/Python.framework/Versions/Current/bin/python)= 675b28e55955734aba9e4df5d1ddd7d48ff9c83b


5. Pull some stuff from svn
   
export BASE_DIR=/tmp
mkdir $BASE_DIR/build
cd $BASE_DIR/build
svn co http://svn.pythonmac.org/py2app/py2app/trunk py2app
svn co http://svn.python.org/projects/sandbox/trunk/setuptools setuptools


6. Build it
   
cd $BASE_DIR/build/setuptools
python ./setup.py build
sudo python ./setup.py install
cd ../py2app/trunk
python ./setup.py build
sudo python ./setup.py install


7. Now, you should have everything you need to build a little simple app without much trouble. First, verify that python and setuptools and py2app are installed correctly:
   
terrance:simple vijayp$ python
Python 2.5.4 (r254:67917, Dec 23 2008, 14:57:27)
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>>


8. Next up, try to get the example to work:
   
cd examples/simple
rm -rf build dist *.app
python ./setup.py py2app
open dist/hello.app

   If you didn’t get any errors, but an app just flashed in and out, you’re in business. If you really care, you can open up the console log (in the Console app). You should have a bunch of environment variables printed.
9. Building the 3rd party libraries from source shouldn’t be that difficult, but luckily, py25 libraries have mostly been built. Try installing PIL, then building the PIL example. It should work correctly.

Hopefully this little tutorial was helpful. Please let me know if it worked, or if it didn’t.

Mac OS comes with a built-in “System Python,” located in /usr/local/bin. As you can see, it’s compiled for three platforms, 64-bit, 32-bit, and ppc:

terrance:~ vijayp$ file /usr/bin/python
/usr/bin/python: Mach-O universal binary with 3 architectures
/usr/bin/python (for architecture x86_64): Mach-O 64-bit executable x86_64
/usr/bin/python (for architecture i386): Mach-O executable i386
/usr/bin/python (for architecture ppc7400): Mach-O executable ppc


py2app requires a couple of generic system libraries contained in the setuptools package, especially macholib, this library that can interact with OsX object files. There are many instructions online about how to install these; I tried a number of different things, and a combination of easy_install, compiling source, and port install packages sort of worked. Unfortunately, I was never able to compile or install a PIL that would embed its .so into the python archive. I didn’t spend too much time on it, though, since I really needed a standalone python archive. Apparently, a Mr. Brant Faircloth (cool name!) figured this out at some point, but I wasn’t able to get it to work

Next up, I tried to use Mac Ports to install python. I tried following Aral Balkan’s instructions, and a number of other random ones. I tried compiling from source, using python-25 ports, python-26 ports, and even python-24 ports, and all failed with one of the two errors I quoted above. I tried installing universal versions of them using sudo port install python-25 +universal, and also by setting the architecture to i386 in the macports configuration file.

The most common error I saw was this:

File "build/bdist.macosx-10.6-intel/egg/macholib/ptypes.py", line
48, in from_str
return cls.from_tuple(struct.unpack(endian + cls._format_, s), **kw)
error: unpack requires a string argument of length 8
> /Users/vijayp/src/build/py2app/examples/simple/build/bdist.macosx-10.6-intel/egg/macholib/ptypes.py(48)from_str()
-> return cls.from_tuple(struct.unpack(endian + cls._format_, s), **kw)


I spent a couple of hours perusing the source — it helped me get a clearer picture of the issue. It appears to be the result of parts of the macholib code having trouble pulling information from object files that have both 64- and 32-bit versions in them. I’m not sure why; perhaps it’s because both architectures are viable?

Sadly, figuring this out fully would take me a week, and my time is limited. I’m sure an expert could figure this out soon. I worked with a bunch of incredible python gurus at Google, and some of you guys read my blog! Hint hint!

(While we’re on this subject, the lack of strace has been bothering me for some time. It turns out dtruss is a reasonable replacement.)