Getting BerkeleyDB working with Python on OSX Lion

Come on Apple. Why do you have to do this to me? Every other unix installation is just fine, but you have to go and be super insane with your default modules. So many things are secretly broken with the default python installation, it's recommended that you install a 'real version' before doing anything. So I'll just go ahead and follow these instructions and...

Oh.

Come on.

Okay, it looks like the solution might be to first install Xcode 4.3 and then to OH MY GOD I AM SICK OF THIS, I JUST WANT BDB IN PYTHON. EFFF YOUUUU APPLE.

Alright, I got that out of my system, let's get to solving the problem. The solution turns out to be really easy, but completely undocumented. If you don't have the intestinal fortitude to go reading through install scripts, or if you haven't dealt with good old-fashioned *nix library installations from a decade ago, you probably won't ever find it on google. If you can figure out the magic search terms that do bring up the solution, please let me know.

To start, first wipe your hard drive, repartition it to ext4 volumes, and install ubuntu or any other debian variant, and tadaa, you're done.

No? Fine. If you don't want to use the grandness of apt, then we'll have to settle for homebrew. Homebrew is a package manager along the same vein of apt or yum, but specifically for macs. The recipes included in homebrew have acceptable coverage of most of the popular packages I've tried to install, so it gets my stamp of approval. Install homebrew before moving on.

If you're like me, you'll have learned to really hate IDEs, and Xcode in particular. If you don't have it installed, take a moment to grab the gcc-without-xcode installer from the venerable Kenneth Reitz.

Now that you have brew and gcc installed, time to get BerkeleyDB installed. Or at least, one we can manipulate. For all I know, Apple may have included one somewhere, but I'll be damned if I can find it.

brew install berkeley-db

What's that? I have to install the JDK because Apple chose to not include it? I have to sign up for an Apple developer account to get it?! I give up. I'll just go back to writing my key-value pairs on this cave wall.

Wait, what's this? If we look at the brew formula for BerkeleyDB, you'll notice an interesting "without-java" option listed. Let's try that.

brew install berkeley-db --without-java

Aha, success! Now we're getting somewhere. Homebrew installs all of its packages to /usr/local/Cellar. Take note of that, it'll be important later. The end must be near. It should be a simple matter of running pip and...

Huh, okay, it wants the bdb path. Okay, no problem.

Hmmm, if we do 'pip help install', we find that to pass options to the installation process, we need to use the --install-option option of pip, which will then feed it to the package's setup.py. So we do it like so:

Still nothing. If we look at the output, we can see that the install is failing on running 'python setup.py egg_info'. If we go into the build directory and run that manually, it suggests we add the --berkeley-db option. When we do that, it seems to work, but how do we get pip to call it with that option?

It turns out, we can't. At least, not that I can figure out. You can pass parameters to 'setup.py build' through pip by the --install-option flag, but not the 'setup.py egg_info'. It's just not baked in there. Horrible.

At this point, our options are, fix pip, or fix the bsddb3 setup.py process. Well, neither are especially appealing options. If I ever need to do this again, or do it on another machine, or whatever, that would be a mess. There's gotta be a better way.

Thinking back to the 90s (I know, right?), when you tried to build a library that had a dependency on another library, you could specify the location of that dependency through a similar flag, like --my-library-path=/blah. This was almost always a convenience shortcut for setting an environment variable, MY_LIBRARY_PATH=/blah. Let's see if that's the case here. If we go into the egg and grep around for 'berkeley-db', we find the following segment:

It looks like all we have to do is set the environment variable BERKELEYDB_DIR and point it at our bsddb installation.

Finally. I'm aggravated that it was annoying as it was. I figured we had basically solved all of these problems 12 years ago. Every time I'm asked to install java or gcc on a mac, I realize how obnoxious OSX is for doing old-timey non-Apple development. It's not impossible, but some things are just absurd.

To recap (or in the modern parlance, tl;dr):

OSX ships with broken bsddb python bindings. To fix this:

  1. Install gcc
  2. Install homebrew
  3. Install berkeley-db through brew
  4. Set the BERKELEY_DB environment variable to point to the brew installation before running pip install bsddb3.
12 responses
in bash, and w/sudo $ sudo BERKELEYDB_DIR=/usr/local/Cellar/berkeley-db/5.3.15/ pip install bsddb3 not: $ BERKELEYDB_DIR=/usr/local/Cellar/berkeley-db/5.3.15/ sudo pip install bsddb3 Thanks very much for this!!! I wound up going through all the steps you did, but at least, thanks to you, I knew where I was going.
Thanks, Chris. Thanks, Jon.
This is excellent! Thank you for this post!
Thanks, your post saved me a lot of time.
Hi! Thanks for your post, it saved me a lot of time I just followed your steps and got it working ... almost. After successfully installing the module and trying to import it i got: ImportError: dlopen(/Library/Python/2.7/site-packages/bsddb3/_pybsddb.so, 2): Library not loaded: /usr/local/opt/berkeley-db/lib/libdb-5.3.dylib Referenced from: /Library/Python/2.7/site-packages/bsddb3/_pybsddb.so Reason: image not found To fix this, i just created a symbolic link pointing to the brew installed berkeley-db and got it finally working :) : ➜ ~ cd /usr/local/opt/ ➜ opt git:(master) sudo ln -s /usr/local/Cellar/berkeley-db/5.3.28 berkeley-db
Thanks for the writing. And this would work too: BERKELEYDB_DIR=/usr/local/ pip install bsddb3 Since there is sim_link for both include and lib from cellar folder.
6 visitors upvoted this post.