Unpacking bundler

This post is the final part of our 4 part series on loading code in Ruby. You can find the other parts here: Part 1, Part 2, Part 3.

With the addition of RubyGems, we’ve solved the issue of direct dependencies. These are now specified with named versions. However, we still have a problem with our indirect dependencies (in other words: our dependencies’ dependencies).

We have no say in these. They have already been determined by the gem developer. But what happens if they’re incompatible?

In the cartoon above, both gem_pink and gem_green need to use gem_yellow.

# Gem Green is loaded first, and runs this code
gem "gem_yellow", "> 3.0"
# RubyGems finds gem_yellow v3.1 on the machine and activates it

# Gem Pink is loaded second, and runs this code
gem "gem_yellow", "< 2.0"
# RubyGems finds gem_yellow v1.6 on the machine and tries to activate it

When RubyGems comes across this second statement it will throw an “Activation Error”.

"Gem::LoadError (can't activate gem_yellow-1.6, 
already activated gem_yellow-3.1.)"

It’s a pretty reasonable complaint. A single Ruby process can’t load two different versions of the same gem¹.

So somehow, we need to agree on which version of Gem Yellow we’re going to use in the app.

Solving the problem

We can’t solve this by changing our indirect dependencies. They are already set in stone for each version of each gem. We can only do this by changing our direct dependencies. Sure, the latest version of Gem Green needs the latest version of Gem Yellow. But do we need the latest version of Gem Green? Can we downgrade and still make the app work?

Sorting this out used to be the job of the developer. If you added a new gem which upset the existing gems, it was your job to upgrade or downgrade gems until everything worked again.

There wasn’t a particularly scientific way of doing this - it was mostly trial and error.

But this was nuts! This is exactly the sort of problem that computers are really good at, and humans are pretty bad at.

Software to the rescue

And so it was that a new piece of software was developed to solve this problem - Bundler. Bundler is essentially a negotiator. She wants a solution that works for everyone.

Bundler required developers to create a new kind of file - a Gemfile - for each project. This set out two things:

• the dependencies that are needed to make the project work
• the range of versions of each gem that would be acceptable

Here’s an example Gemfile (along with the implied range of acceptable versions for each gem). This is a super small Gemfile with just two gems.

gem 'http', '~> 5.1' # works with: 5.1.0, 5.1.1, 5.2.0
gem 'phonelib', '~> 0.9' # works with: 0.9.0, 0.9.1, 0.9.2   

It uses the same RubyGems syntax that we’re now familiar with. But Bundler is actually doing something quite different with it.

RubyGems looks at each gem statement in isolation, and uses it to find a gem which satisfies the constraints on the existing machine.

Bundler doesn’t care about the gems that you have installed on your machine. Instead, she looks at all of the requirements of the project as a whole, and works out which set of versions will satisfy all these constraints at the same time.

Once she has a set of gems that work together, Bundler spits out a new file - a Gemfile.lock. This tells us the exact versions of each gem that should be used in the project.

Here’s the Gemfile.lock from our example Gemfile.

GEM
remote: https://rubygems.org/
specs:
  addressable (2.8.7)
    public_suffix (>= 2.0.2, < 7.0)
  base64 (0.2.0)
  capybara (3.40.0)
    addressable
    matrix
    mini_mime (>= 0.1.3)
    nokogiri (~> 1.11)
    rack (>= 1.6.0)
    rack-test (>= 0.6.3)
    regexp_parser (>= 1.5, < 3.0)
    xpath (~> 3.2)
  domain_name (0.6.20240107)
  ffi (1.17.0-arm64-darwin)
  ffi-compiler (1.3.2)
    ffi (>= 1.15.5)
    rake
  http (5.2.0)
    addressable (~> 2.8)
    base64 (~> 0.1)
    http-cookie (~> 1.0)
    http-form_data (~> 2.2)
    llhttp-ffi (~> 0.5.0)
  http-cookie (1.0.7)
    domain_name (~> 0.5)
  http-form_data (2.3.0)
  llhttp-ffi (0.5.0)
    ffi-compiler (~> 1.0)
    rake (~> 13.0)
  matrix (0.4.2)
  mini_mime (1.1.5)
  nokogiri (1.16.7-arm64-darwin)
    racc (~> 1.4)
  public_suffix (6.0.1)
  racc (1.8.1)
  rack (3.1.7)
  rack-test (2.1.0)
    rack (>= 1.3)
  rake (13.2.1)
  regexp_parser (2.9.2)
  xpath (3.2.0)
    nokogiri (~> 1.8)

You’ll see that it’s already pretty long and complicated, with just two gems.

If you graph out the relationships between the gems, as in the image below², you’ll see some of the potential issues.

So this isn’t a simple “tree” with a clear hierarchy, this is a crazy graph with links all over the place.

• our indirect dependencies also have dependencies
• sometimes our indirect dependencies will have depdendencies on our direct dependencies
• some dependencies have multiple “parents”

Bear in mind that in a typical Rails project, there will be hundreds of gems and connections between them. Not an easy task.

How does it work?

Unlike a hapless developer using trial and error and hoping for the best, Bundler has a systematic approach.

Bundler uses a ‘dependency resolution algorithm’ to find a set of direct and indirect dependencies that satisfy all the constraints³.

We can think of the algorithm as Bundler heading to shops with a shopping list. The requirements are her shopping list, and the basket represents the gems she believes will satisfy those requirements activate.

She goes though each item on the list in turn, looking for a gem version that satisfies the stated requirements. If she finds one that works, she puts it in her basket.

However, if she later finds a requirement that conflicts with one of the gems that she has already put in the basket, she “backtracks”. This means taking out of the basket every gem that she has put in since she added the conflicting gem, and starting over from that point. She then tries to find a new version of that original gem that satisfies both the original requirement, and the new conflicting requirement, before carrying on again.

It’s pretty laborious stuff⁴. Once she’s finished, she’ll download any new versions that are needed.

To ensure that the gems in your Gemfile.lock are the ones that are activated, you’ll need to add bundle exec before your Ruby command. If you forget to do this, RubyGems will just do it’s normal thing and use the latest versions of the gems installed on your system.

An unexpected advantage

Using a Gemfile.lock also has another advantage. Not only do we know that the dependencies will work together, it also means that everyone working on the project is using the exact same set of dependencies.

If we don’t commit the Gemfile.lock to our repository, but instead rely on each developer running bundle install, then it’s very possible that different machines would end up with slightly different versions of some gems⁵.

If we’re all running exactly the same code, we’ve just eliminated a whole class of bugs. 🎉

Hooray for Bundler

So Bundler is pretty great. Compared to the node and python ecosystems, I think the way manages Ruby dependencies is pretty good.

Of course, if you make an impossible request in your Gemfile, and specify two gems that have fundamentally incompatible dependencies - Bundler can’t help you. In this case Bundler will stop and let you know.

However, if there is a way to make your Gemfile requirements work, Bundler will find it.