Do not upgrade your Rails project to ruby 2 before you read this

17 Dec 2013

Update (03-10-2014)

After some great debugging from @mislav, turns out that this is not actually a Ruby issue or even a Koala/Faraday issue.

This is actually the right_aws gem and/or the right_http_connection gem that are monkey-patching the class and causing the issue.

The github issue on Koala has all the discussion

Thanks @mislav

The story

One of the most interesting project I did recently was to completely revamp the underlying infrastructure that runs Gogobot.

From Apache-Passenger pair, I converted to Nginx+Unicorn while upgrading the Ruby version to 2.0 (from 1.9.3).

I did everything using Chef (many thanks to my colleague @kesor6 for the nights of pair programming), it was definitely one of the most interesting projects I had done, Chef is awesome but that’s for a different post.

The problem

When I upgraded the project to Ruby 2.0, things started to break, really weird things like the Facebook API (using the Koala gem), AWS api (using right-aws) and others.

Everything that broke was related to some sort of an external API, and it was really weird since everything was working super smoothly before the upgrade.

Now, the bug was really obscure and hard to find, and it required quite a bit of digging, but here are the results.

The code that was breaking was pretty standard, nothing out of the ordinary

graph = Koala::Facebok::API.new(token)
graph.get_object 'me'

When this code ran on Ruby 2.0, it threw an exception:

MultiJson::DecodeError: 399: unexpected token at ''

Now, this API did not change, I did not upgrade the gem, ruby 1.9 version of the site was running the same code with no problem.

At first, I tried upgrading the Koala gem, the JSON gem, replacing the JSON backend and more, nothing worked and I was getting into the dependency limbo we all know and hate.

Then, I started digging through the code, the line that was breaking was inside Koala:

body = MultiJson.decode("[#{result.body.to_s}]")[0]

The problem was that result.body was actually Gzipped, and trying to parse it Gzipped just exploded all over the place.

Now, I did not ask for it to be Gzipped, no where in the source am I passing the header that I am accepting Gzip, so it felt odd, I started digging through the Ruby STDlib.

I found the culprit here: https://github.com/ruby/ruby/blob/v2_0_0_353/lib/net/http/generic_request.rb#L39

	initheader["accept-encoding"] =
  		"gzip;q=1.0,deflate;q=0.6,identity;q=

You can read the code on your own, but what you can easily see is that if you don’t pass a header, by default Ruby will pass the header that will tell the external service that your app is ok with a Gzipped response (which is obviously wrong)

The fix

Digging through some Stackoverflow answers (which were 99% wrong), I found the one that hinted to the solution and it was that the response class had an option to decode the content, and if you pass this variable, everything will be ok.

(The source code is almost identical to the proposed solution)

module DecodeHttpResponseOverride
  def initialize(h,c,m)
    super(h,c,m)
    @decode_content = true
  end

  def body
    res = super
    if self['content-length']
      self['content-length']= res.bytesize
    end
    res
  end
end

module Net
  class HTTPResponse
    prepend DecodeHttpResponseOverride
  end
end

This solved the problem, for all classes, AWS, Koala and all other places that had bugs.

Summing up

I posted an issue to Koala here: https://github.com/arsduo/koala/issues/346, to be clear, this is not a Koala bug, but I figured the guys maintaining it should know about the bug.

Before you upgrade your project, make sure you have a test suite that covers those situations, luckily we did so the damage wasn’t big.

Hopefully, this post and the Koala bug report will save you some time when you are upgrading.