Coverband 3 03 October 2018

Coverband 3

Coverband gives deep insight into the production usage of every line of code in your Ruby app. Find dead code, find hotspots, see code paths that might be unexpectedly in high usage.

This release marks deep dives into Ruby performance, benchmarking, and driving almost all features based on improving performance. The release is a majory performance improvement, resolving some issues were Coverband 2 could have outliers that performed worse than the original Coverband 1 gem.

Bugs

Me first and the Gimme Gimmes, a punk cover band, image from Wikimedia

Want to see it in action, visit the Coverband Demo Site, visit a few pages and watch the coverage report change as you use the site.

What breaking changes are there?

  • drops Tracepoint
  • drops Ruby <= 2.3.0
  • drops JSON Gem dependency
  • drops various other features not needed without Tracepoint
    • memory cache, sampling, restricted to app folders, etc

What is new?

  • standardizes on Coverage array format vs sparse hash
  • rewrites store methods, for 60X perf!
    • implemented for Redis and File store
  • improved mountable web interface

What exactly is faster?

tldr;

Storing Coverage data to a data store. While this didn’t happen often when it did it could generate thousands of Redis requests on large apps, which could make for a very slow outlier request…

Long Version

The primary improvements come down to changing the storage format and moving from method of storage that required one call per file and assummed a small list of files, to a 2 pass stoage for all data. Basically, the previous storage mechanism created an N+1 type issue. Coverband 2 had solved the issue of capturing Coverage data, which was no longer a performance concern, in fact it captured all files by default as there wasn’t really a reason to filter for performance anymore… This created a new problem when trying to store coverage data. Eventually a very specific benchmark was added that simulates trying to store coverage reports for apps with 2934 files tracking coverage. In Coverband 3 by changing the storage format and the storage plan the storage benchmark improved by nearly 60X!

Simulate repeatedly storing coverage reports with Coverband 2.0.3:

rake benchmarks:redis_reporting
runs benchmarks on reporting large sets of files to redis
Warming up --------------------------------------
       store_reports     1.000  i/100ms
Calculating -------------------------------------
       store_reports      0.814  (± 0.0%) i/s -     13.000  in  15.982911s

Simulate repeatedly storing coverage reports with Coverband 3.0.0:

rake benchmarks:redis_reporting
runs benchmarks on reporting large sets of files to redis
Warming up --------------------------------------
       store_reports     4.000  i/100ms
Calculating -------------------------------------
       store_reports     47.111  (± 4.2%) i/s -    708.000  in  15.066259s

How did this impact Rails Apps

In general, depending on various settings the outliers weren’t that noticable or it made Coverband unusable for folks. I configured a Rails app with non ideal, but not terrible options, to show what this could look like and how Coverband 3 resolves the issue. If configured idealy even Coverband 2 would mostly only show in outliers, so this test is showing how even in a bad setup V3 performs with nearly no impact.

The benchmarks below are a further iteration of my AB Benchmarking Rails apps process. Now much easier to read and compare, by graphing by following this Apache Bench via Gnuplot guide.

No Coverband Coverband 2 Coverband 3
Bugs Bugs Bugs
mean: 27.272 [ms] mean: 35.762 [ms] mean: 28.460 [ms]
36.67 [#/sec] (mean) 27.96 [#/sec] (mean) 35.14 [#/sec] (mean)

Notice that with no Coverband and Coverband 3, the majority of requests show little variation, while Coverband 2 has frequent outliers pushing request timing all over the place. See full sized images of the benchmarks below.

What is next?

I am glad you asked, as we introduced a roadmap and set of feature discussions for upcoming Coverband improvements. Check out the changes.md for the latest, but in short here is some of what we have in the works.

Please give Coverband a shot and add any issue you have to the repo.

Footnotes

For those crazy few that really want all the data on the benchmarks…

No Coverband

No Coverband benchmark graph
rake benchmarks:coverband_demo_graph
Benchmarking coverband-demo.herokuapp.com (be patient)

Concurrency Level:      10
Time taken for tests:   54.543 seconds
Complete requests:      2000
Failed requests:        0
Requests per second:    36.67 [#/sec] (mean)
Time per request:       272.717 [ms] (mean)
Time per request:       27.272 [ms] (mean, across all concurrent requests)

Percentage of the requests served within a certain time (ms)
  50%    262
  66%    269
  75%    273
  80%    276
  90%    288
  95%    302
  98%    328
  99%    350
 100%   1400 (longest request)

Coverband 2

Coverband 2 benchmark graph
rake benchmarks:coverband_demo_graph
Benchmarking coverband-demo.herokuapp.com (be patient)

Concurrency Level:      10
Time taken for tests:   71.524 seconds
Complete requests:      2000
Failed requests:        0
Requests per second:    27.96 [#/sec] (mean)
Time per request:       357.619 [ms] (mean)
Time per request:       35.762 [ms] (mean, across all concurrent requests)

Percentage of the requests served within a certain time (ms)
  50%    325
  66%    353
  75%    378
  80%    398
  90%    489
  95%    545
  98%    622
  99%    682
 100%   1442 (longest request)

Coverband 3

Coverband 3 benchmark graph

rake benchmarks:coverband_demo_graph
Benchmarking coverband-demo.herokuapp.com (be patient)

Concurrency Level:      10
Time taken for tests:   56.919 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      16565120 bytes
HTML transferred:       14974000 bytes
Requests per second:    35.14 [#/sec] (mean)
Time per request:       284.597 [ms] (mean)
Time per request:       28.460 [ms] (mean, across all concurrent requests)

Percentage of the requests served within a certain time (ms)
  50%    273
  66%    282
  75%    290
  80%    296
  90%    319
  95%    344
  98%    381
  99%    410
 100%   1358 (longest request)
comments

Lead Developer London 18 July 2018

A Unique Conference

I attended #LeadDevLondon this year, it was a nice and different conference. Generally, I have attended more language-specific conferences. There were basically no language-specific talks at all during the conf, as everything was a bit more general about building and working well with technology teams. Given that these days a large part of what I do is manage teams of devs, it was good to hear other folks talk through some of the challenges and solutions they have gone through during their journey.

leaddev london stickers

My swag bag full of stickers

Some Favorite Slides

For my very favorite slide see the section with my tweets below ;) Otherwise, enjoy some favorite slides with little to no context ;) You can generally find the full presentation slides on the talks page and the full videos of the #LeadDevLondon talks on youtube.

leaddev london slides

@alicegoldfuss explaining containers

leaddev london slides

@alicegoldfuss explaining what else ops does ;)

leaddev london slides

@alexhillphd going over the art of code reviews, this talk had my favorite slide of the conf

leaddev london slides

@cmccarrick how to scale yourself and make important decisions

leaddev london slides

@tara_ojo on how to help Jr’s with good 1:1s

leaddev london slides

@nmeans lessons learn from 3 mile island

leaddev london slides

@ClareSudbery

leaddev london slides

@dbussink shows how distributed his team is

leaddev london slides

@dbussink explains how to make that an advantage

leaddev london slides

@WebDevBev on how to run awesome internships

leaddev london slides

@jqiu25 the journey to safe ways to edit production data

Four Slides From The Legacy Code Talk

I guess I really liked what @ramtop had to say about legacy code

leaddev london slides

leaddev london slides

leaddev london slides

leaddev london slides

A Talk Pattern I Want To See

One thing I noticed and really came away with from the talks is that every piece of advice and best practice really fits into part of a companies growth curve. Even the most agreed on advice can’t and shouldn’t apply the same to a single person startup and a company of thousands of people and hundreds of engineers. This made me crave a talk that opposed to saying how some team approaches a challenge and the ways it works for them, but how a best practice or process changes and adapts to different growth stages of a company…

As a single talk, an example let’s consider continuous delivery of the growth of a team. While the end best practices involve style checkers, peer code review, automated test suites, deployment to one or more staging, and feature flag or percentage based releases to production, before/after metric analysis on the impact of deploys… It would not be possible to start there nor would it be sensible to set up all that complexity when just getting started… I feel like the flow would naturally look something more like this.

  • 1 person startup -> manually run CLI deployment task from the dev machine
  • 2-3 people -> CI test suite, manual CLI deployment
  • 3-8 people -> static style checker, CI test suite, code review, staging deployment with stakeholder / QA review, push-button deployments and rollbacks automated on CI
  • 8-16 -> all the above, with multiple staging environments, some integrated with 3rd party sandboxes, feature flag based releases, metrics tied to specific deployed releases, and deployment queue to help schedule deployments
  • etc, etc, etc…

I think it would be great to see detailed talks on tech org structures, testing, monitoring, alerting, etc..

My Tweets

Some of the tweets I shared during the conf.

The Most Important Part, Stickers

Seriously, thanks to all the sponsors and friends handing out stickers. I love collecting stickers at conferences to come back and cover my climbing wall. I am building a large collage covering the entire climbing wall in layers of stickers. After returning from a conference I always have a huge collection of stickers to use, which is great. In this case, I basically was able to make an entire tile of my climbing wall a #leaddevlondon tile full of stickers I picked up during the trip.

leaddev london stickers

leaddev london stickers

leaddev london stickers

If you don’t know what to do with all the stickers you have collected at various conferences, feel free to send them my way ;)

comments

Benchmarks Bugs 10 June 2018

Benchmarking Bugs

On my previous post I covered benchmarking Rubygems… I was working on some follow up to improve the benchmarks and discovered a bug rendering one of the benchmark comparisons invalid. In this post we will cover what I had missed, how to avoid some gotchas, and ways to improve the readability of your benchmarks.

Bugs

image from pixabay

Spot The Bug

This is a simplified example from the source code. The bug is hard to notice without knowing the internals of the library, but if you understand Ruby’s Coverage library you might spot it.

namespace :benchmarks do
  desc 'set up coverband with coverage redis'
  task :setup_coverage do
    clone_classifier
    $LOAD_PATH.unshift(File.join(classifier_dir, 'lib'))
    require 'benchmark'
    require 'classifier-reborn'

    Coverband.configure do |config|
      config.root               = Dir.pwd
      config.percentage         = 100.0
      config.logger             = $stdout
      config.collector          = 'coverage'
      config.memory_caching     = ENV['MEMORY_CACHE'] ? true : false
      config.store              = Coverband::Adapters::RedisStore.new(Redis.new)
    end
  end
  
    def run_work
    puts "benchmark for: #{Coverband.configuration.inspect}"
    puts "store: #{Coverband.configuration.store.inspect}"
    Benchmark.bm(15) do |x|
      x.report 'coverband' do
        SAMPLINGS.times do
          Coverband::Collectors::Base.instance.sample do
            work
          end
        end
      end

      x.report 'no coverband' do
        SAMPLINGS.times do
          work
        end
      end
    end
    Coverband::Collectors::Base.instance.stop
    Coverband::Collectors::Base.instance.reset_instance
  end

  desc 'runs benchmarks coverage'
  task run_coverage: :setup_coverage do
    puts 'Coverband Coverage configured with to use default redis store'
    SAMPLINGS = 5
    run_work
  end
end

The issue is that the benchmark is trying to compare running Coverband with Ruby’s Coverage lib against code, which doesn’t collect usage data. When using Ruby’s Coverage is loaded and started, it changes how ALL other code is loaded and interpreted. Which means once the Coverage library is loaded both benchmarks will run with the performance impact. The only difference is once is collecting and reporting the coverage to Redis, via Coverband. Since the Coverage library effect the Ruby runtime it is best to run the benchmarks as entirely isolated processes to avoid blending the impacts of one benchmark into the other.

Spot the Bug in the Stats

Even if one doesn’t notice the issue in the code, the output from the benchmarks should make the issue stand out. In my previous post I noticed that I could no longer detect performance impacts of Coverband, that seemed to good to be true.

                      user     system      total        real
coverband         0.320000   0.010000   0.330000 (  0.322387)
no coverband      0.320000   0.000000   0.320000 (  0.321767)

It also didn’t match with real world Rails benchmarks. At the time in that post, I believed my benchmarks just weren’t sufficient to capture the small impact. Wanting to make my benchmarks easier to read and to be able to detect the known performance impact is what caused me to find the bug.

Making Benchmarks More Readable

As I was working to detail the the performance impact of Coverband, I decided I should port my benchmarks from the std-lib benchmark, to the excellent benchmark-ips gem by @evanphx. The gem has output in the format below. The new format in comparison to the default format, makes it much easier to understand performance differences. In this case because of the bug to see clearly a missing and expected difference. The improved Gem also helped me find better iterations to detect changes on my existing and working benchmarks.

Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
           coverband     1.000  i/100ms
        no coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband     14.690  (±27.2%) i/s -    149.000  in  12.045429s
        no coverband     15.112  (±33.1%) i/s -    151.000  in  12.240970s

Comparison:
        no coverband:       15.1 i/s
           coverband:       14.7 i/s - same-ish: difference falls within error

The improved format along with increased iterations (5 seconds warmup and 12 seconds runtime) made it clear that there was truly ZERO difference between the Coverage benchmarks, which was wrong. This makes it very clear a there was a bug in my methodology.

Fixing the Issue

Once I realized Coverage was loaded for both runs, it was pretty clear what was wrong. I needed to do a couple things.

  • The Coverage lib needs to be both required and started before any code that we plan to track (this also wasn’t done in the previous benchmark, so actual coverage data wasn’t being collected correctly.)
  • Since Coverage changes how Ruby actually interprets the code when loaded. I needed to run the two timed tasks independently and finally compare the data.

It was pretty easy to port my old benchmark code to benchmark-ips, and to clean things up along the way. As well as handle both of the mentioned issues. See the new benchmark code below (simplified here, see Github for the full src code details).

namespace :benchmarks do
  desc 'setup standard benchmark'
  task :setup do
    clone_classifier
    $LOAD_PATH.unshift(File.join(classifier_dir, 'lib'))
    require 'benchmark'
    require 'benchmark/ips'

    # NOTE: When we require files is what makes performance impact of Coverage interesting (moving this above or below Coverage.start gives detectable vs not detectable performance impacts
    require 'classifier-reborn'
    if ENV['COVERAGE']
      puts 'Coverage library loaded and started'
      require 'coverage'
      ::Coverage.start
    end
    require 'redis'
    require 'coverband'
    require File.join(File.dirname(__FILE__), 'dog')
  end

  desc 'set up coverband with coverage Redis'
  task :setup_coverage do
    Coverband.configure do |config|
      config.root               = Dir.pwd
      config.percentage         = 100.0
      config.logger             = $stdout
      config.collector          = 'coverage'
      config.memory_caching     = ENV['MEMORY_CACHE'] ? true : false
      config.store              = Coverband::Adapters::RedisStore.new(Redis.new)
    end
  end
  
  def run_work(hold_work = false)
    suite = GCSuite.new
    #puts "benchmark for: #{Coverband.configuration.inspect}"
    #puts "store: #{Coverband.configuration.store.inspect}"
    Benchmark.ips do |x|
      x.config(:time => 12, :warmup => 5, :suite => suite)
      x.report 'coverband' do
        Coverband::Collectors::Base.instance.sample do
          work
        end
      end
      Coverband::Collectors::Base.instance.stop
      x.report 'no coverband' do
        work
      end
      x.hold! 'temp_results' if hold_work
      x.compare!
    end
    Coverband::Collectors::Base.instance.reset_instance
  end
  
  desc 'runs benchmarks coverage'
  task run_coverage: [:setup, :setup_coverage] do
    puts 'Coverband Coverage configured with to use default Redis store'
    run_work(true)
  end

  desc 'compare Coverband Ruby Coverage with normal Ruby'
  task :compare_coverage do
    puts 'comparing with Coverage loaded and not, this takes some time for output...'
    puts `COVERAGE=true rake benchmarks:run_coverage`
    puts `rake benchmarks:run_coverage`
  end
end

As you can see to handle the hard part of running the benchmarks independently and comparing them I used a cool feature of benchmark-ips. Loading the Coverage library or not is now controlled by ENV['COVERAGE'] making it easy to run with or without a change to how Ruby interprets the code.

Using benchmark-ips hold!

As I was starting to look at how to compare data on two runs, I checked to see if the new benchmarking gem I was using could help me. The documentation covered having a hold! feature, which seemed exactly what I needed.

If you are comparing multiple implementations of a piece of code you may want to benchmark them in separate invocations of Ruby so that the measurements are independent of each other. You can do this with the hold! command.
benchmark-ips, independent-benchmarking

Sounds great, but the hold! wasn’t detailed very well in the documentation. In fact another user had created an issue for the project trying to figure out how to properly use the hold!. I decided to read through the source code to figure it out and verify that it could do what I needed.

PR with example benchmark-ips hold! Usage

Well long as I take the time to figure it out, and I know others have struggled to understand the usage. We might as well try to make the usage clear to everyone. The project has an examples folder detailing several usages. Adding a small PR with a hold usage example, gave a simple usage scenario. Which was a simplified version of the approach I used to benchmark the Ruby Coverage library performance impacts in Coverband.

Updated Benchmark Conclusions

After fixing the bug, benchmarking Coverage is still very interesting. Even small changes in when Coverage is started and which files are required, can have a massive impact on the overall performance. Take a look at diff below, which shows moving a single line require 'classifier-reborn' to be above or below when we require and start Coverage, which means the libraries code is either included or not in the Coverage collection.

-    require 'classifier-reborn'
     if ENV['COVERAGE']
       puts 'Coverage library loaded and started'
       require 'coverage'
       ::Coverage.start
     end
+    require 'classifier-reborn'
     require 'redis'
     require 'coverband'
     require File.join(File.dirname(__FILE__), 'dog')

The first benchmark shows when classifier-reborn is required prior to setting up Coverage and therefor it’s data is excluded from the results.

rake benchmarks:compare_coverage
comparing with Coverage loaded and not, this takes some time for output...
Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
           coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband     11.950  (±16.7%) i/s -    140.000  in  12.041935s

Pausing here -- run Ruby again to measure the next benchmark...
Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
        no coverband     1.000  i/100ms
Calculating -------------------------------------
        no coverband     14.497  (±20.7%) i/s -    161.000  in  12.020487s

Comparison:
        no coverband:       14.5 i/s
           coverband:       12.0 i/s - same-ish: difference falls within error

In this second benchmark we ensure the Coverage library is loaded and started prior to requiring classifier-reborn, which means we are capturing all the usage of that library code.

rake benchmarks:compare_coverage
comparing with Coverage loaded and not, this takes some time for output...
Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
           coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband      9.117  (±11.0%) i/s -    109.000  in  12.108073s

Pausing here -- run Ruby again to measure the next benchmark...
Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
        no coverband     1.000  i/100ms
Calculating -------------------------------------
        no coverband     14.184  (±14.1%) i/s -    164.000  in  12.017139s

Comparison:
        no coverband:       14.2 i/s
           coverband:        9.1 i/s - 1.56x  slower

In these examples the only difference is that the library classifier-reborn is either included in the recorded coverage or ignored by the Coverage library, based on when we require it. The improved benchmark code:

  • is easier to read than it was previously (both code and benchmark output)
  • it clearly captures and allows for easy performance tests of Coverband using Coverage
  • It makes easily clear how much of a win Coverage is over the older TracePoint API
    • Coverage showing either no diff or 1.56x slower (depending on the coverage scope)
    • Tracepoint showing around 3.95x slower

Full Benchmark Output

Here is the full benchmark comparison output across all benchmarked configurations of Coverband.

Coverband tracepoint configured with file store
Warming up --------------------------------------
           coverband     1.000  i/100ms
        no coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband      3.348  (±29.9%) i/s -     39.000  in  12.095922s
        no coverband     13.921  (±21.6%) i/s -    160.000  in  12.048469s

Comparison:
        no coverband:       13.9 i/s
           coverband:        3.3 i/s - 4.16x  slower

Coverband tracepoint configured with default Redis store
Warming up --------------------------------------
           coverband     1.000  i/100ms
        no coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband      3.726  (± 0.0%) i/s -     45.000  in  12.286824s
        no coverband     14.736  (±27.1%) i/s -    150.000  in  12.059519s

Comparison:
        no coverband:       14.7 i/s
           coverband:        3.7 i/s - 3.95x  slower

comparing with Coverage loaded and not, this takes some time for output...
Coverage library loaded and started
Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
           coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband     12.173  (±16.4%) i/s -    143.000  in  12.017026s

Pausing here -- run Ruby again to measure the next benchmark...
Coverband Coverage configured with to use default Redis store
Warming up --------------------------------------
        no coverband     1.000  i/100ms
Calculating -------------------------------------
        no coverband     15.045  (±13.3%) i/s -    177.000  in  12.024386s

Comparison:
        no coverband:       15.0 i/s
           coverband:       12.2 i/s - same-ish: difference falls within error

Driving Performance Impact Improvements Based on Learnings

Now with an ability to detect performance differences when running Coverband with Ruby’s Coverage library, I can build features and benchmarks to show how to reduce the performance impact on real world use cases.

For example in general with Coverband the goal is to show the coverage of application code ignoring framework and gem code. Knowing that which files are loaded at what time related to Coverage start can impact performance. I can build a feature to ensure that Coverband sets the Coverage library to ensure ONLY the application code is being tracked by Coverage, which will help ensure the lowest possible performance impact.

I will be working on this feature (Coverband Safelist) and new benchmarks to show how much application developers could reduce the performance burden on standard Rails apps by using a Safelist. Opposed to allowing various Rails framework code to be tracked along side their application code which is required by the current version of Coverband, given how Rails loads code. Look forward to another post, when that feature and benchmarks are ready.

comments

RubGems Benchmarks 29 May 2018

RubyGems Benchmarks

This follows up on my previous post focused on Ruby Benchmarking. This post will focus on benchmarking a gem as part of it’s long term maintenance. Building a community standard around Gem Benchmarking has even been suggested as a good way to help Ruby fine tune performance over time.

Stopwatch

image from pixabay

Why Benchmark A Gem

If you maintain a Gem, which could have large performance impacts on folks integrating the gem. It can make sense to try to build performance testing into the Gem, helping to ensure that changes and new features don’t impact the performance in unexpected ways overtime. It also will help for you to publish performance impacts to give confidence to folks adding your gem as a dependency.

For example I maintain a Gem, Coverband, which records production code coverage. Obviously, this can have a major impact on performance as it tracks every line executed on production. During the life of Coverband, nearly every decision about features and Gem release involved some Benchmarking. The project was initially developed to help remove dead code from a large old monolithic rails app, before the first release could be put on production it went through a number of performance tests and only ran on staging until the perf impact could be acceptably controlled.

Coverband Benchmark Timeline

Coverband code has gone through several strategies to mitigate the performance impact. Let’s take a look at quick timeline of major changes. As Coverband has changed, we can check and compare the performance impacts with the Coverband Benchmark Tasks, which are explained later in this post.

  • 2013: initial release
    • performance cost of 100% recorded coverage, around 8X slower Rails requests
    • initial release performance costs weren’t well explained with no released benchmarks
    • Performance costs were mitigated by sampling a percentage of requests
    • As well as safelist & blocklisting files to track
  • 2014: released coverband_ext (C extension for fast access to tracepoint API) in
    • benchmarks for 100% recorded coverage showed this 1.25X slower Rails requests
    • This was the first time I released solid benchmarks on coverband
      • benchmarks were done on a sample Rails app & a large production app at my current place of employment
      • benchmarks were done by hand in a non repeatable process
  • 2015: Various performance improvements introduced
    • Redis pipelining
    • Redis zadd
    • benchmarking was done by hand, in a non repeatable way, to confirm actual performance impacts
  • 2016: performance tests introduced into Coverband repository by @kbaum8
    • The new performance tests were used to propose that with various improvements between Ruby 1.9.x and Ruby 2.1.x that we could drop support for Coverband_ext
    • Moved from set_trace_func to Ruby trace_point
    • The micro-benchmark showed that Ruby trace_point was more than good enough an the C extension no longer provided significant performance improvements
    • See the source for integrated micro-benchmark performance tests
    • At these point most features and changes were checked with the performance tests to ensure the project was always getting faster
    • improved file filtering
    • Adding support for line usage count vs just used or unused for example didn’t incur additional overhead
  • 2017:
    • Multiple backend stores added
    • The benchmark performance tests were refactored, so performance could be compared across multiple backend stores
  • 2018:
    • Attempt to patch the Coverage reentrant Ruby bug
      • in this quest I propose Coverage.pause and Coverage.resume
      • and in response @tenderlove says my goals are supported by peek_results which was added about 3 years ago, and that the performance impacts I had been assuming were not likely correct.
      • I attempt to prove that the performance impact would be significant, and end up proving @tenderlove is correct ;) Which is what started the deeper dive into Ruby Benchmarking
    • The results from the above cause me to update the Coverband benchmarks and use those benchmarks to help prove the significant win of using Coverage vs trace_point
    • Extending the benchmark code to report across different collector methods, shows massive win for the new Coverage based collector, at least a 4X improvement.

Benchmark Code

Below are some selected examples of the Coverband micro-benchmark performance tasks.

There are many ways to setup benchmarks for a Gem. In the Coverband examples are simple Rake tasks. This is opposed to performance tests, which folks have often employed for Rails app benchmarks. Given all the configuration and runtime impacts, the Rake approach worked well, but it requires reviewing the data by hand as nothing is tracked or charted in a machine readable format over time via CI.

The example below has been simplified a bit from the full source linked above.

namespace :benchmarks do
  # leaving out some helper methods
  
    desc 'set up coverband tracepoint collector to redis'
  task :setup do
    clone_classifier
    $LOAD_PATH.unshift(File.join(classifier_dir, 'lib'))
    require 'benchmark'
    require 'classifier-reborn'

    Coverband.configure do |config|
      config.redis              = Redis.new
      config.root               = Dir.pwd
      config.percentage         = 100.0
      config.logger             = $stdout
      config.collector          = 'trace'
      config.memory_caching     = ENV['MEMORY_CACHE'] ? true : false
      config.store              = Coverband::Adapters::RedisStore.new(Redis.new)
    end
  end

  desc 'set up coverband with coverage collector to redis'
  task :setup_coverage do
    clone_classifier
    $LOAD_PATH.unshift(File.join(classifier_dir, 'lib'))
    require 'benchmark'
    require 'classifier-reborn'

    Coverband.configure do |config|
      config.root               = Dir.pwd
      config.percentage         = 100.0
      config.logger             = $stdout
      config.collector          = 'coverage'
      config.memory_caching     = ENV['MEMORY_CACHE'] ? true : false
      config.store              = Coverband::Adapters::RedisStore.new(Redis.new)
    end
  end
  
  def work
    5.times do
      bayes_classification
      lsi_classification
    end

    # simulate many calls to the same line
    10_000.times { Dog.new.bark }
  end

  def run_work
    puts "benchmark for: #{Coverband.configuration.inspect}"
    puts "store: #{Coverband.configuration.store.inspect}"
    Benchmark.bm(15) do |x|
      x.report 'coverband' do
        SAMPLINGS.times do
          Coverband::Collectors::Base.instance.sample do
            work
          end
        end
      end

      x.report 'no coverband' do
        SAMPLINGS.times do
          work
        end
      end
    end
    Coverband::Collectors::Base.instance.stop
    Coverband::Collectors::Base.instance.reset_instance
  end

  desc 'runs benchmarks on default redis setup'
  task run: :setup do
    puts 'Coverband tracepoint configured with default redis store'
    SAMPLINGS = 5
    run_work
  end

  desc 'runs benchmarks coverage'
  task run_coverage: :setup_coverage do
    puts 'Coverband Coverage configured with to use default redis store'
    SAMPLINGS = 5
    run_work
  end
end

desc 'runs all benchmarks'
task benchmarks: ['benchmarks:run', 'benchmarks:run_coverage']

Benchmark Results

Below is a sample of the output generated when all the benchmarks are run. For each test, it configures Coverband and outputs the configuration settings, along with the same code executed with and without Coverband. The key point being the output below for the new Coverband implementation.

                      user     system      total        real
coverband         0.320000   0.010000   0.330000 (  0.322387)
no coverband      0.320000   0.000000   0.320000 (  0.321767)

Stopwatch

User time from benchmark results below graphed

While it is easy to see the performance impact of the previous tracepoint collector, at 1.350000 vs 0.320000 the current benchmark can’t even detect a performance slowdown using the new Coverage collector. While this means, I should further extend the Gems performance tests, the new implementation is significantly and easily proved to be far more performant than the previous implementation.

rake benchmarks
Coverband tracepoint configured with file store
benchmark for: #<Coverband::Configuration:0x007fc069235c68 @root="/Users/danmayer/projects/coverband", @redis=nil, @root_paths=[], @ignore=[], @additional_files=[], @include_gems=false, @percentage=100.0, @verbose=false, @reporter="scov", @collector="trace", @logger=#<IO:<STDOUT>>, @startup_delay=0, @memory_caching=false, @store=#<Coverband::Adapters::FileStore:0x007fc0692359e8 @path="/tmp/benchmark_store.json">, @disable_on_failure_for=nil>
store: #<Coverband::Adapters::FileStore:0x007fc0692359e8 @path="/tmp/benchmark_store.json">
                      user     system      total        real
coverband         1.350000   0.000000   1.350000 (  1.354026)
no coverband      0.310000   0.000000   0.310000 (  0.321970)
Coverband tracepoint configured with default redis store
/Users/danmayer/projects/coverband/test/benchmarks/benchmark.rake:127: warning: already initialized constant SAMPLINGS
/Users/danmayer/projects/coverband/test/benchmarks/benchmark.rake:134: warning: previous definition of SAMPLINGS was here
benchmark for: #<Coverband::Configuration:0x007fc069235c68 @root="/Users/danmayer/projects/coverband", @redis=#<Redis client v3.3.3 for redis://127.0.0.1:6379/0>, @root_paths=[], @ignore=[], @additional_files=[], @include_gems=false, @percentage=100.0, @verbose=false, @reporter="scov", @collector="trace", @logger=#<IO:<STDOUT>>, @startup_delay=0, @memory_caching=false, @store=#<Coverband::Adapters::RedisStore:0x007fc06911c8b8 @redis=#<Redis client v3.3.3 for redis://127.0.0.1:6379/0>>, @disable_on_failure_for=nil>
store: #<Coverband::Adapters::RedisStore:0x007fc06911c8b8 @redis=#<Redis client v3.3.3 for redis://127.0.0.1:6379/0>>
                      user     system      total        real
coverband         1.310000   0.000000   1.310000 (  1.344789)
no coverband      0.320000   0.000000   0.320000 (  0.318863)
Coverband Coverage configured with to use default redis store
/Users/danmayer/projects/coverband/test/benchmarks/benchmark.rake:141: warning: already initialized constant SAMPLINGS
/Users/danmayer/projects/coverband/test/benchmarks/benchmark.rake:127: warning: previous definition of SAMPLINGS was here
benchmark for: #<Coverband::Configuration:0x007fc069235c68 @root="/Users/danmayer/projects/coverband", @redis=#<Redis client v3.3.3 for redis://127.0.0.1:6379/0>, @root_paths=[], @ignore=[], @additional_files=[], @include_gems=false, @percentage=100.0, @verbose=false, @reporter="scov", @collector="coverage", @logger=#<IO:<STDOUT>>, @startup_delay=0, @memory_caching=false, @store=#<Coverband::Adapters::RedisStore:0x007fc0691acfa8 @redis=#<Redis client v3.3.3 for redis://127.0.0.1:6379/0>>, @disable_on_failure_for=nil>
store: #<Coverband::Adapters::RedisStore:0x007fc0691acfa8 @redis=#<Redis client v3.3.3 for redis://127.0.0.1:6379/0>>
                      user     system      total        real
coverband         0.320000   0.010000   0.330000 (  0.322387)
no coverband      0.320000   0.000000   0.320000 (  0.321767)

Additional Resources on Performance Testing Gems

Some other examples of benchmarking a Gem or even single commits.

comments

Ruby Benchmarking 25 March 2018

Ruby Benchmarking

If you work on a large app, publish, gems, or are interested in the Ruby community at some point you will likely want to dig into performance and benchmarking. Often what you think will help performance doesn’t have as much of an impact as you would guess. Which is why being able to set up quick experiments to verify the impact of code changes on performance and memory can be important. Benchmarking is great as it can help prove, your change had the impact you want. Ruby performance and the measurement of it is getting more discussion at the moment because Ruby 3X3 set out a goal to make Ruby 3 three times faster than Ruby 2. Learn more in this talk, Ruby3x3: How are we going to measure 3x?

This post will talk about measuring the performance of Ruby and benchmarking your own code and applications. In a future post, I will dig more into good practices around benchmarking Ruby gems.

Stopwatch

image from pixabay

Ruby Benchmarking Projects

A number of projects have been proposed and have support from the community for benchmarking Ruby… These project all will help the community improve the performance of Ruby, and can be used to help you measure the impact or Ruby or Gem changes you would like to try. If you are interested in contributing to Ruby or helping the community with improved performance or better benchmarking I encourage you to check out these projects. A great way to learn more about benchmarking is to dig into projects that are already out available and looking for more contributors.

Ruby Benchmarking Basics

When I need to quickly benchmark something, I look no further than Ruby’s simple Benchmark Module. I don’t have all the usage memorized off the top of my head, but have ended up at this great blog post, timing Ruby code is easy with Benchmark, a number of times over the years. It is a quick and simple post that will help get basics up and running in no time at all. For some additional Ruby benchmarking support also see the resources below.

Learning by Benchmarking A Proposed Ruby Change

I previously covered how to build Ruby from scratch on OS X. I posted that as I was working on a feature, I was hoping to get into Ruby. I proposed a change to Ruby’s Coverage that I hoped would lead to large performance improvements adding Coverage Pause & Resume support. A little discussion with @tenderlove, lead me to try to prove the value of the feature in terms of performance. In the end, I actually proved my feature idea was unnecessary and that Ruby’s Coverage was significantly faster than Ruby’s Tracepoint. Below, I will walk through some of the steps and code, I used while driving towards better benchmarks and understanding of what was really impacting performance while trying to collect the line of code runtime.

First Attempt: A Micro Benchmark

For some specific changes a micro benchmark, that focuses in very narrowly on the specific changes under test can be all that is needed and make for faster and easier iteration while testing changes. Often these are the easiest to setup and be more easily repeatable with various versions and settings to compare a number of changes.

My first attempt was to use a small repository, that I had used to show an issue with Ruby’s coverage, see my Benchmarking Coverage Example. The code was extended it so it could be run in 3 modes:

  • Ruby without Coverage loaded: ruby example.rb
  • Ruby with Coverage as it exists in the current release: COVERAGE=true ruby example.rb
  • Ruby with my suggested Coverage feature put to use: ENHANCED_COVERAGE=true ruby example.rb
require 'benchmark'
require 'coverage'

WITH_COVERAGE = !!ENV['COVERAGE']
WITH_ENHANCED_COVERAGE = !!ENV['ENHANCED_COVERAGE']

Coverage.start if WITH_COVERAGE || WITH_ENHANCED_COVERAGE
require 'bigdecimal/math'
require './app'
require './app_proxy'
Coverage.pause if WITH_ENHANCED_COVERAGE

ITERATIONS = 2_000
UPTO = 1_000
coverage_data = nil

# warm up
AppProxy.process(App, {iterations: 1, up_to: UPTO})

Benchmark.bm do |bm|
  if WITH_ENHANCED_COVERAGE
    bm.report { coverage_data = AppProxy.process(App, {iterations: ITERATIONS, up_to: UPTO, coverage: false, enhanced_coverage: false}) }
  end
  bm.report { coverage_data = AppProxy.process(App, {iterations: ITERATIONS, up_to: UPTO, coverage: WITH_COVERAGE, enhanced_coverage: WITH_ENHANCED_COVERAGE}) }
end

puts "coverage"
puts coverage_data
puts "done"

The initial results, even with a high number of iterations basically couldn’t show a real difference. In the case, I was trying to show, since it wasn’t a dramatic improvement and wasn’t obviously measurable like the number of objects allocated to memory, proved to not be a good case for a microbenchmark.

Second Attempt: Build On Rails Ruby Bench

As I abandoned the idea of a microbenchmark being able to show the differences of the performance impact when using Ruby’s code coverage. I sought a more realistic example of how Ruby code is frequently used in production. I found @codefolio’s project, rails_ruby_bench, which I mentioned previously intends to help measure performance impacts for Ruby 3X3. Unfortunately, I ran into some issues getting this benchmark to run locally and I would need to do a good deal of work to embed the code I wanted under test into this project and Discourse which the project uses as the Rails app under performance testing.

While I didn’t end up pursuing this route, it did help push me towards a good direction to more realistically measure the performance impacts I wanted to see. By setting up a sample Rails application and testing full request cycles, I would have a much more realistic measure of my changes. I plan to follow up on this project more in the future and think it is an ideal way to test many Ruby or Gem changes that would have performance or memory impacts on standard Rails applications.

Third Attempt: Sample Rails App

Combining the learnings from the microbenchmark and the ideas in Rails Ruby Bench, I wanted to be able to quickly test a number of different scenarios by just setting a few environment variables, and then pull meaningful results on exercising the full Rails app with the changes. To measure the impact on the full Rails stack I turned to Apache’s AB, a simple HTTP benchmarking tool I have used many times over the years.

For each test, the Rails application would be run in a different mode, then benchmarked via AB. I created a new repository coverage_rails_benchmark. The project README covers all the steps for anyone to run their own set of benchmarks and records all the results of my performance tests.

The benchmarking script itself is extremely simple, the script below will make 2000 requests making 5 concurrent requests at a time to the endpoint listed. The code can be found in bin/benchmark.rb

puts `ab -n 2000 -c 5 "http://127.0.0.1:3000/posts"`

For example to get the most basic results of Ruby without Coverage loaded, one would follow the two steps below in two different terminals.

  • start the Rails server in basic mode: IGNORED_COVERAGE=true RAILS_ENV=production bin/rails server
  • execute the benchmark: ruby ./bin/benchmark.rb

This would output a bunch of data about the benchmark results.

Benchmarking 127.0.0.1 (be patient)

Server Software:
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /posts
Document Length:        3631 bytes

Concurrency Level:      5
Time taken for tests:   8.391 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      8572000 bytes
HTML transferred:       7262000 bytes
Requests per second:    238.34 [#/sec] (mean)
Time per request:       20.978 [ms] (mean)
Time per request:       4.196 [ms] (mean, across all concurrent requests)
Transfer rate:          997.58 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0      11
Processing:     6   21  10.8     19     192
Waiting:        6   20  10.7     19     190
Total:          6   21  10.8     19     192

Percentage of the requests served within a certain time (ms)
  50%     19
  66%     23
  75%     25
  80%     26
  90%     30
  95%     33
  98%     38
  99%     42
 100%    192 (longest request)

For my various comparisons, I ended up caring the most for the mean time per request in the above example 20.978 [ms] (mean) was the starting point baseline I comparing with all other results. My final sample benchmark support 7 distinct modes that could be run in two different distinct settings. Unlike with my microbenchmark performance differences were extremely obvious with a full spread of fasted benchmarking mode running at 17.735 [ms] (mean) vs the slowest mode taking 85.069 [ms] (mean). Showing the worst mode was nearly 5X slower in a simple benchmark.

Benchmarking Conclusions

The various modes and options made it clear that I had initially been optimizing for the wrong thing, and that while there was significant performance improvements that could be made, it wouldn’t be related to my suggested Coverage feature proposal to support pause and resume. Instead, it showed that collecting coverage data was always very fast, but pulling and processing that data could be very slow. Opposed to trying to sample data collection the goal should be to reduce as much as possible the frequency one processes the results and to filter it down to the smallest set of results needed. Let’s take a look at a high-level summary below, or see the full benchmark details.

  • No Coverage Support: 17.735
  • Coverage Running (Ignore Coverage): 18.131
  • Coverage Stopped (Ignore Coverage): 18.268
  • Coverage Paused (Ignore Coverage): 18.717
  • Coverband Coverage (Ignore Coverage): 18.759
  • New Pilot Version Coveraband Coverage (Collect Coverage): 19.227
  • Coverage Running (Collect Coverage, but only into memory): 21.141
  • Coverage Resume (Ignore Coverage): 23.930
  • Coverage Resume (Collect Coverage, but only into memory): 26.720
  • Coverband Coverage (Collect Coverage): 39.421
  • Coverband Tracepoint (Collect Coverage): 46.979
  • Coverband Tracepoint (Ignore Coverage): 47.500
  • Coverage (Collect Coverage, send to Rails.logger): 85.069

In the end, my goal of changing Ruby’s Coverage was to be able to significantly reduce the performance overhead of my gem Coverband’s ability to collect runtime data. The results from the benchmark gave me all the data I needed to see that a different approach, where I dropped TracePoint in favor of Coverage but worked to reduce the frequency of checking the results would lead to far better performance improvements than my initially suggested feature. In the end, I pursued that approach, and was able to reduce the overhead with a sample rate of 100% from 2.5X slower to only being 1.08X slower! The details on that, I will cover in another post with some specifics on how to build performance benchmark testing into a Gem.

Ruby Benchmarking Learnings

I came away with a deeper understanding of Ruby performance benchmarking, and a much faster Gem that I will be able to release shortly. Beyond that some additional thoughts on Ruby benchmarking,

  • For some changes microbenchmarks aren’t helpful
  • There are some existing great projects to help folks benchmark changes to the Ruby language
  • Ruby’s Coverage is significantly Faster than collecting line usage via Ruby’s TracePoint functionality
  • Specific to Ruby’s Coverage library
    • The number of files instrumented with coverage has a big impact on performance, making micro benchmarks useless for that library
    • Calling coverage.peek_results to access the data is the biggest cost, much larger than collecting it.
    • Simply logging, processing, or trying to do anything with data is often slower than collecting it
    • Performance wins will come in reducing processing the data vs collecting it
  • Writing an maintaining good benchmarks can be challenging, I didn’t touch on the issues of running benchmarks with “background noise” and running multiple times to tease that out
  • Without measuring performance, guessing what will have a large impact is often wrong

Let’s Go Faster

via GIPHY

In a follow-up post, I will dig into the details of benchmarking a Gem over time, and the specific changes that helped to reduce the performance overhead of Coverband making it an order of magnitude faster. While there are code changes and using different features of Ruby, large performance wins also can come from a fundamentally different approach to solving the problem, which is the case with the changes needed to improve Coverband.

comments

Rack Proxy Tour 26 February 2018

Rack Proxy Tour

I wanted to share a quick tour of a flexible Ruby tool. I have turned to rack-proxy a number of times, throughout the years. It is a tiny and super useful Rack middleware that can quickly be adapted to perform a number of useful functions. While it is a small library it is a tool that has been handy for quick and lasting workarounds a number of times. It is a sharp tool, so be careful with it… Especially because it is severely lacking documentation… Perhaps I should send a documentation PR one of these days.

Job Challenge

image from pixabay

Rack Proxy Examples

Some of the various ways I have used Rack Proxy over the years, while I wouldn’t recommend all of them, sometimes a quick hack is needed and Rack proxy can be a powerful tool for that.

  • subdomain based pass-through to multiple apps
  • useful for handling awkward redirection rules for moved pages
  • fan out a single API request to multiple concurrent backend requests and merging results
  • authentication / authorization prior to proxying requests to a blindly trusting backend
  • avoiding CORs complications by proxying from same domain to another backend

Example Code

In the example below we will have our rack proxy middleware, handle user authentication and authorization then make an authenticated request to another service. In this case, any request to our Rails applications host with the path /example_service/ will pass through to the target service.

require 'rack-proxy'

class ExampleServiceProxy < Rack::Proxy
  def perform_request(env)
    request = Rack::Request.new(env)

	 # path matches our target &&
	 # user auth (devise in this case) found a logged in user
    if request.path =~ %r{^/example_service} &&
      env['warden'] &&
      env['warden'].user

		# have a user but using CanCan check if user has needed permissions
      if env['warden'].user.can?(:access, :access_example_service)
        token = "Bearer #{Settings['service.token']}"
        service_url = Settings['service.url']

        @backend = URI(service_url)
        env['rack.backend'] = @backend
        
        # while documentation says you only need on of these,
        # I needed to set them all to have the expected results
        env['REQUEST_PATH'] = env['REQUEST_URI'] = env['PATH_INFO'] = '/api/target_service_path'
        
        # target service fails on cookies
        env['HTTP_COOKIE'] = ''
        env['HTTP_AUTHORIZATION'] = token
        super(env)
      else
        Rails.logger.info "example_service: 401 user #{env['warden'].user.inspect} denied"
        [401, {}, ['Unauthorized!']]
      end
    else
      @app.call(env)
    end
  end
end

Then you can just hook up the middleware as needed, using something like below (in application.rb for example).

config.middleware.use(ExampleServiceProxy)

If you want to be able to access the devise user env['warden'].user you will need to make sure your middleware is inserted after the devise middleware. You can quickly check the order by printing out the Rails middleware stack.

> rake middleware

use Raven::Rack
use BufferedLoggingMiddleware
...
use Warden::Manager
use ExampleServiceProxy
...
use OtherExamples
comments

Building Your Own Ruby 17 February 2018

Building & Using Your Own Ruby

If you want to work on changes or learn more about the internals of the Ruby language, you can alter the source and build your own from scratch. It isn’t that hard or scary, you will learn a bit more about Ruby just by building it. This is a quick start to building Ruby from scratch on OSX, and using it to run your local apps.

If you want to read a great guide about contributing to Ruby, it also covers how to build even easier via a Dockerfile.

Get the Source

Use git or SVN to pull the source repo.

  • I like git, so I forked the Ruby project.
  • Then clone it: git clone [email protected]:danmayer/ruby.git
  • go into the directory and take a look around.

Build Trunk Before Modifying

Before you make any modifications, I recommend you get the current trunk building and test running your apps with it. Then you can create a branch and see the impact of any modifications you would like to try out.

Building Yourself

I recommend using the Ruby-buil instructions below as I continued to hit issues with building from scatch with my own options related to SSL.

If you want to give it a shot thought, run the 4 commands below and it will compile and install Ruby from src on your system.

aclocal
autoconf
bash -c './configure'
make && make install

If you get this error about OpenSSL:

openssl:
	Could not be configured. It will not be installed.
	Check ext/openssl/mkmf.log for more details.
*** Fix the problems, then remove these directories and try again if you want.

It is a bit hard to resolve on OS X yourself, but luckily we can just use ruby-build to do the work for us, see below.

Building via Ruby-build

Ruby build will detect and use SSL from homebrew and avoid the broken OSX implementation.

  • install ruby-build: brew install ruby-build
  • copy the current dev trunk target: cp /usr/local/Cellar/ruby-build/20171226/share/ruby-build/2.6.0-dev /usr/local/Cellar/ruby-build/20171226/share/ruby-build/2.6.0-mine
  • edit the file (...ruby-build/2.6.0-mine) to point to your fork: ... install_git "ruby-trunk" "https://github.com/danmayer/ruby.git" ...
  • build from your git to the ~/.rubies/ directory: ruby-build 2.6.0-mine ~/.rubies/ruby-2.6.0mine
    • run from the build dir: /usr/local/Cellar/ruby-build/20171226/share/ruby-build
  • make sure to open a new shell so ChRuby (or RbEnv) find the new ruby.
  • you can now reference the new build in your .ruby-version file in any project

If you want to build from your local git to avoid pushing to a remote branch while testing this is how your build file should end up.

install_package "openssl-1.1.0g" "https://www.openssl.org/source/openssl-1.1.0g.tar.gz#de4d501267da39310905cb6dc8c6121f7a2cad45a7707f76df828fe1b85073af"  mac_openssl --if has_broken_mac_openssl
install_git "coverage_pause" "/Users/danmayer/projects/ruby" "feature/coverage_pause" ldflags_dirs autoconf standard_build standard_install_with_bundled_gems verify_openssl

If all is working as expected you should see this.

ruby-build 2.6.0-coverage ~/.rubies/ruby-2.6.0coverage
ruby-build: use openssl from homebrew
Cloning https://github.com/danmayer/ruby.git...
Installing coverage_pause...
ruby-build: use readline from homebrew
Installed coverage_pause to /Users/danmayer/.rubies/ruby-2.6.0coverage

Playing Nice with Other Rubies?

If you are like most Rubyists you have a number of Rubies installed, beyond the default OSX Ruby, using something like RBenv, RVM, or ChRuby.

  • The default OS X Ruby should be: /usr/bin/ruby
  • If you build from scratch the default target will be installed to: /usr/local/bin
    • This can cause some issues with various Ruby environment managers
    • adjust your path to target this Ruby or your normal one
    • Another reason I recommend building via ruby-build, as it will work easier with most Ruby environment managers
  • Ruby build will put ruby into the ~/.rubies
    • with ChRuby it will automatically pick these up

Then you can reference your new custom ruby in a .ruby-version file at the root project directory, for example:

# .ruby-version
ruby-2.6.0coverage

Running an App with your Custom Ruby

After building your own Ruby and setting the .ruby-version you should be good to go. You can verify you are running your Ruby by adding some simple print statement. Pick a favorite Ruby method and just add a print statement like so…

printf( "hello from my method!\n" );

  • push the change to the branch your referenced in your Ruby build steps
    • there are ways to build locally, but I have just targetted git branches
  • rebuild Ruby: ruby-build 2.6.0-coverage ~/.rubies/ruby-2.6.0coverage
  • enter your project iwth the set ruby-version
  • run ruby -v to make sure it matches expectations
    • if you get chruby: unknown Ruby: ruby-2.6.0coverage either refresh chruby or open a new terminal so it picks up the new build
  • now run things as you normally would…
# in Rakefile
desc "call coverage running"
task :call_coverage_running do
  require 'coverage'
  Coverage.running?
end
rake call_coverage_running
hello from Coverage.running

In the above output, I can see that I am runnin my custom Ruby branch as I had added a print messaged to the Coverage.running? method.

Testing your Custom Ruby

If you are trying to do some real work on Ruby other than exploring, you will want to be able to run the tests on files after you modify them, and add your own tests.

Running Tests

For example to run a specific test file, you can run the below command.

make test-all TESTS=test/coverage/test_coverage.rb
Run options: "--ruby=./miniruby -I./lib -I. -I.ext/common  ./tool/runruby.rb --extout=.ext  -- --disable-gems" --excludes-dir=./test/excludes --name=!/memory_leak/

# Running tests:

Finished tests in 1.580204s, 12.0238 tests/s, 74.6739 assertions/s.
19 tests, 118 assertions, 0 failures, 0 errors, 0 skips

ruby -v: ruby 2.6.0dev (2018-01-13 trunk 61811) [x86_64-darwin15]

Running Specs

Below we can see how to run a single specific spec file

make test-spec MSPECOPT=spec/ruby/library/coverage/start_spec.rb
generating x86_64-darwin15-fake.rb
x86_64-darwin15-fake.rb updated
$ /Users/danmayer/projects/ruby/miniruby -I/Users/danmayer/projects/ruby/lib /Users/danmayer/projects/ruby/tool/runruby.rb --archdir=/Users/danmayer/projects/ruby --extout=.ext -- /Users/danmayer/projects/ruby/spec/mspec/bin/mspec-run -B ./spec/default.mspec spec/ruby/library/coverage/start_spec.rb
ruby 2.6.0dev (2018-01-13 trunk 61811) [x86_64-darwin15]
[\ | ==================100%================== | 00:00:00]      0F      0E

Finished in 0.002753 seconds

1 file, 0 examples, 0 expectations, 0 failures, 0 errors, 0 tagged
comments
Dan Mayer Profile Pic
Welcome to Dan Mayer's development blog. I primary write about Ruby development, distributed teams, and dev/PM process. The archives go back to my first CS classes during college when I was first learning programming. I contribute to a few OSS projects and often work on my own projects, You can find my code on github.

Twitter @danmayer

Github @danmayer