Ruby on Rails: Why Upgrading Matters – Part 1

Ruby on Rails (or Rails) is a web development framework that gives Rails developers an optimized experience to write their (Ruby) code. Rails are one of many web frameworks in the world of app programming and web development. These frameworks are collections of code libraries that give app and web developers off-the-shelf solutions for time-consuming, repetitive tasks—such as building menus, tables, or forms on a website.

Because Rails is a web development framework all security aspects around it need to be taken very seriously.  This creates the need to be on (or near) the latest release in order to take advantage of the most updated functionality and of course, the latest security patches.

Why did we upgrade Rails?

Our app was running on an older Rails version which meant:

• Security support had ended. This meant that the existing Rails version was no longer receiving updates for bug fixes for critical security issues.
• We could not directly upgrade to the latest ruby version — here is why.
• Newer features from newer Rails versions were not available
• We could not use the latest gems versions. Older gems also lacked functionality and suffered from security flaws.

How we started the upgrade journey:

Whenever you’re upgrading software, the most critical points are:

• Upgrading without blocking feature development
• Affecting as few stakeholders as possible
• Making sure changes are backward compatible

To fulfill all the above critical points we did the upgrade process as follows:

Plan: 

Our aim was to upgrade our existing application to the latest stable version.

Upgrading directly from the current version to the desired stable version is not recommended and also it would be a huge change for us to make in one go. So, we decided to first upgrade Rails to the next available higher version, and then subsequently upgrade to the latest desired Rails version.

So, we followed a staggered upgrade approach where we updated Rails to the desired version in 2 steps.

We started by first doing a small POC by changing rails and many other gem versions and checking if the application is booting properly. There were many deprecations and changes needed for the POC and we did all that in a week.

After a week, we were able to start the rails console and server.

Here is a list of critical changes that we did when upgrading Rails:

• All view files, especially partials should be named with underscores and not with hyphens.
• Factorygirl gem has to be updated for RSpec tests and the new version has syntax changes for defining factories. Use Factory.define instead of FactoryGirl.define.
• In models, while defining scopes or relationships, use Procs instead of where conditions.
• Routes should be written with proper HTTP methods i.e. GET, POST, etc. instead of MATCH.
• Use attr_accessor or decorators for assigning non-column attributes to an ActiveRecord object to avoid MissingAttributeError exceptions.

You can also find the critical changes in different Rails versions here.

Dual boot:

We made changes in our Gemfile to make the app dual boot. So that the current development work can be done simultaneously with our upgrade work. This also helped us during the rollout because it acts as a single switch to change the app’s rails version.

What we did for dual boot:

• Defined a kernel method named rails_next? the method in gemfile so that new Rails version-specific changes can be isolated under it.
• This was an indispensable part of the Rails upgrade as all the changes like bundle install, rails console, rails server, and rake assets: precompile can work independently of the Rails version.
• We took help from this article
• We created a different gemfile.lock for the new Rails version and named it Gemfile_next.lock. This file was used to keep the gem versions for the newer Rails version.

Gems compatibility: 

We tried to upgrade the gems which were incompatible with newer Rails versions so that they can work with both Rails versions. But approx 23 gems couldn’t be upgraded to the newer Rails version. Some of them were critical to the app like devise, activeadmin, etc.

These gems came under the rails_next? the method we had defined in the gemfile.

We added some more gems for compatibility.

Green unit tests: 

You have to depend on unit tests to check if things are working fine. We had around 3.5k tests failing out of 6k tests. We had to incorporate changes in such a way that the unit tests should work with both Rails versions.

We followed the following steps to achieve that:

• Upgraded rspec to a newer version
• Identify the issue in the failed unit test; Fix; Repeat. (Across full codebase)
• Added a bitbucket pipeline that ran tests in both Rails versions for every Pull-Request. (First, we added a pipeline for the new Rails version bootup and then for unit tests along with existing Rails version unit tests)

The biggest challenge for rollout:
The biggest challenge that we faced was the cache incompatibility between the Rails version and between 2 releases.

Environment Cache setup:
We have around 5 major prod environments and around 3 staging/internal envs.

We use Memcached and Redis as application cache in these environments.

The Problem:
We had multiple instances in code where we were caching objects directly in the cache(eg. ActiveRecord objects).
Due to this, in the newer Rails version, we were unable to read the cache written via the older Rails version and vice versa.

Also, since we follow a blue-green deployment there was a difference in the code in the 2 releases which needed to be compatible to avoid any issues.

There are only two hard things in Computer Science: cache invalidation and naming things.

— Phil Karlton

The First Solution (Using Openstruct object in place of Rails objects):

We decided to use Openstruct in place of Rails objects since it’s a Ruby class and available in both Rails versions and accessible with both dot and bracket notation. We also monkey-patched the Dalli gem to handle this incompatibility. The fix was used to invalidate the old cache from the older Rails version and replace it with the new cache.

For Redis, we thought of clearing the old cache and letting the new cache be created since the prod env cache was very small.

The patch had changes to handle NameError and to check if the object retrieved from Memcached can be converted to an object which Rails understand.

With the following changes, we handled the following scenarios for production environments since both releases will be serving traffic while accounts transition from one release to another:

Scenario 1: Cache value is the older Rails version like an active record:

1. Reading from the existing Rails version will work fine in both releases
2. Reading from the newer Rails version will give nil which will result in creating the cache again. This will force the app to override the cache value with the newer ones (i.e. with OpenStruct).

Scenario 2: Cache value is a newer Rails version. OpenStruct object:

1. Reading from the existing Rails version will work fine in both releases as we cache the OpenStruct object which is accessible with both dot and bracket notation.
2. Reading from the newer Rails version will work fine.

All was good and our changes went well in 3 smaller environments. We were ecstatic until we hit a performance issue in the second-biggest production environment.

The command execution delay had increased after the Rails upgrade. We found that the execution delay has increased from 3-3.5 to 4.5-5s in all the environments where we have done Rails upgrades.

Cause of delay :

OpenStruct was being used to wrap the ActiveRecord object in the application cache.

OpenStruct uses Ruby’s method_missing and define_method to simulate “normal” objects and hence is slower. Hence the deserialization of an OpenStruct object takes time.

Prior to the newer Rails upgrade changes, we were reading and writing Rail’s ActiveRecord objects directly into the cache. To tackle this we had made some changes.

Some of them were easier like just converting into Ruby hash while others were trickier since the retrieved cache object(s) were subsequently being used with dot notation.

To fix it with minimal code change, we wrapped the Hash(of the AR object) in OpenStruct since both bracket and dot notation are supported in OpenStruct.

That said, there is performance degradation in using OpenStruct.

This was profusely being used in feature check which is done numerous times throughout the whole app. This was the cause of this delay.

Following are some of the numbers to verify the same:

Benchmarking with respect to Feature cache retrieval:

Env: QA_ENV_1
cache_store: memecached
cache_key = "dummy_account_feature_cache_key"
 
#This cache key's value returns a hash of 35 items where the key is the feature_name and the value is a hash of db records wrapped in OpenStruct.
 
# latency if the cached Object is OpenStruct
latency_openStruct = Benchmark.realtime do
Rails.cache.read(cache_key)
end * 1000
12.446482 ms
 
# latency if the cached Object is ActiveRecord
latency_with_activeRecord = Benchmark.realtime do
 Rails.cache.read(cache_key)
end * 1000
8.524003 ms
 
# latency if the cached Object is a Ruby hash
latency_hash =Benchmark.realtime do
Rails.cache.read(cache_key)
end * 1000
5.614253 ms

General Benchmarking:

require 'ostruct'
require 'benchmark'
 
COUNT = 100_000
NAME = "Qubole Test"
EMAIL = "[email protected]"
RANK = 100
 
class Person
  attr_accessor :name, :email, :rank
end
 
Benchmark.bm(13) do |obj|
  obj.report("Ruby Hash:") do
    COUNT.times do
      p = {name: NAME, email: EMAIL, rank: RANK}
    end
  end
 
  obj.report("OpenStruct:") do
    COUNT.times do
     p = OpenStruct.new({name: NAME, email: EMAIL, rank: RANK})
   end
  end
 
  obj.report("Ruby Class:") do
    COUNT.times do
      p = Person.new
      p.name = NAME
      p.email = EMAIL
      p.rank = RANK
    end
  end
end

Results:

                    user     system      total        real
Ruby Hash:      0.060000   0.000000   0.060000 (  0.062216)
OpenStruct:     1.620000   0.010000   1.630000 (  1.643002)
Ruby Class:     0.030000   0.000000   0.030000 (  0.024538)

The Solution that worked (Using CustomStruct object in place of Rails objects):

We implemented our own implementation of OpenStruct with a simple Class(CustomStruct) whose object can be accessed with both dot and bracket notation and replaced all the instances of OpenStruct.

Following is the implementation of our custom class CustomStruct :

Benchmarking (CustomStruct vs ActiveRecord vs OpenStruct):

Measurement unit: milliseconds

Note: Our application cache is Read Heavy

MEMCACHED (QA_ENV_1)

1. CustomStruct (new implementation)

cache_key = "cache:Benchmark:test1"
Benchmark.realtime do
    type_features = Rails.cache.fetch(cache_key) do
    type_features = Hash.new
    Feature.last(1000).each do |type_feature|
      type_features[type_feature.name] = CustomStruct.new(type_feature.attributes)
    end
    type_features
  end
end
 
write : ~130
 read : ~7.1

2. ActiveRecord (previous implementation)

cache_key2 = "cache:Benchmark:test2"
Benchmark.realtime do
    type_features = Rails.cache.fetch(cache_key2) do
    type_features = Hash.new
    Feature.last(1000).each do |type_feature|
      type_features[type_feature.name] = type_feature
    end
    type_features
  end
end
 
write : ~51
 read : ~12.5

3. OpenStruct (current implementation)

cache_key3 = "cache:Benchmark:test3"
  Benchmark.realtime do
      type_features = Rails.cache.fetch(cache_key3) do
      type_features = Hash.new
      Feature.last(1000).each do |type_feature|
        type_features[type_feature.name] = OpenStruct.new(type_feature.attributes)
      end
      type_features
    end
  end
 
write : ~218
 read : ~21

REDIS (QA_ENV_2)

1. CustomStruct (new implementation)

cache_key1 = "cache:redis:Benchmark:test1"
Benchmark.realtime do
    type_features = Rails.cache.fetch(cache_key1) do
    type_features = Hash.new
    Feature.last(1000).each do |type_feature|
      type_features[type_feature.name] = CustomStruct.new(type_feature.attributes)
    end
    type_features
  end
end * 1000
 
write : ~122
 read : ~7.5

2. ActiveRecord (previous implementation)

cache_key2 = "cache:redis:Benchmark:test2"
Benchmark.realtime do
   type_features = Rails.cache.fetch(cache_key2) do
   type_features = Hash.new
   Feature.last(1000).each do |type_feature|
     type_features[type_feature.name] = type_feature
   end
   type_features
 end
end * 1000
 
write : ~52
 read : ~8.4

3. OpenStruct (current implementation)

cache_key3 = "cache:redis:Benchmark:test3"
  Benchmark.realtime do
      type_features = Rails.cache.fetch(cache_key3) do
      type_features = Hash.new
      Feature.last(1000).each do |type_feature|
        type_features[type_feature.name] = OpenStruct.new(type_feature.attributes)
      end
      type_features
    end
  end * 1000
 
write : ~270
 read : ~21.7

Result: We could see from the benchmarks that CustomStruct was way faster than OpenStruct which gave us the requisite performance.

Some other important tasks:

• Add Rails version tag and request-id for better debuggability. This was very essential for QA, support, and other devs so that they can deterministically debug any issue.
• Make appropriate changes so that devs can easily bring up their dev environment in any of the Rails versions without fail. This was necessary for us to test any changes that we do in both versions so that the code was backward compatible.
• Fix custom code that was strictly adhering to the existing Rails version’s APIs.

We will publish part 2 shortly. Watch this space for more.