A Repo Quick Start

As many organizations with many microservices and employee churn, you can and should expect to often step into a new codebase that needs some smaller updates. It can be really helpful to be able to very quickly go from cloning a repo to being able to run tests and check things in a developer environment. The goal of a number of different helper scripts is to help reduce the total maintenance cost of repositories over their lifetime and accelerate anyone onboarding into the repository.

enter bin/setup

The goal of bin/setup is to go from git clone to being able to run the test suite and boot the app in development mode. The workflow should allow someone to be running any repository very quickly.

> git clone someproject.git
> cd someproject
> bin/setup
> Bootstrapping someproject
> ...
> # checks various depenencies and installs what is missing
> # ... redis, postgres, gems, etc...
> # checks if DBs exist if not creates them and sets them up
> # ... db create, db migrate, db seed, etc...
> ...
> All complete, now you can run `bin/start` for a webserver or `bin/test` to run the tests

The script might have a few assumptions depending on where you work. We have some that if on a Mac will require Homebrew, we have some that might bail if you don’t have Docker. The required dependencies are very small and the script can point you at the documentation to get the minimum requirements up and running.

Language Agnostic and Composable Scripts

While I think bin/setup is the most valuable. You can see I already referenced a few other scripts. Some of the scripts are just one-line aliases, but it still keeps the concept language agnostic. I can get up and running on a Go, Ruby, and Node app on the same day to upgrade a feature touching on the frontend and a number of internal services. In Ruby, for example, the scripts can be very simple.

example bin/test for a Ruby repository:

#!/bin/sh
bundle exec rspec 

example bin/start for a Ruby repository:

#!/bin/sh
PORT=${PORT:-3000} RACK_ENV=development RAILS_ENV=development foreman start -f Procfile.dev --color

The bin/setup script is by far the most complicated, but normally looks something like below:

#!/usr/bin/env ruby

require "fileutils"

APP_ROOT = File.expand_path("..", __dir__)

def system!(*args)
  system(*args) || abort("Command failed: #{args}")
end

def executable?(command)
  ENV["PATH"].split(File::PATH_SEPARATOR).map do |path|
    (ENV["PATHEXT"] ? ENV["PATHEXT"].split(";") : [""]).map do |extension|
      File.executable?(File.join(path, "#{command}#{extension}"))
    end
  end.flatten.any?
end

# WARNING: not portable (macOS and Linux only)
def running?(process_name)
  system("{ ps -ef; docker ps 2>/dev/null; } | grep #{process_name} | grep -v grep > /dev/null 2>&1")
end

def database_exists?(database_name)
  puts %(running database_exists? check for #{database_name})
  system(%(psql -lqt | cut -d \| -f 1 | grep -qw #{database_name}))
end

def create_database!
  system!("bundle exec rails db:create")
end

def check_dependencies
  # Check all dependencies first, and guide the user to the right place
  # to learn more if any are missing.
  #
  # you can add anything you might need here like redis, rabbit, kafka, etc
  # you can either warn or try to detect and install
  unless executable?("psql")
    abort <<~WARNING
      ############################ WARNING ############################
      PostgreSQL is required for using this setup, for tests, the
      server, or the console! For more information, see
      https://internal-wiki.com/psql
    WARNING
  end

  unless running?("postgres")
    abort <<~WARNING
      ############################ WARNING ############################
      PostgreSQL must be running for using this setup, for tests, the
      server, or the console! For more information, see
      https://internal-wiki.com/postgres
    WARNING
  end

  unless require "bundler"
    abort <<~WARNING
      ############################ WARNING ############################
      Bundler is required for using this setup, for tests, the server,
      or the console! For more information, see
      See https://internal-wiki.com/bundler
    WARNING
  end
end

def run_setup
  # This section will go through the basic steps to setup the app to the point specs or server start would work
  puts "Checking dependencies using Bundler"
  system("bundle check") || system!("bundle install")

  puts "Copying example files into place if needed"
  FileUtils.cp "config/database.yml.example", "config/database.yml" unless File.exist?("config/database.yml")

  puts "Creating tmp folder if needed"
  FileUtils.mkdir "tmp" unless File.exist?("tmp")

  create_database!("app_name_development") if !database_exists?("app_name_development")
  if database_exists?("app_name_development")
    puts "Migrating development database"
    system!({"RAILS_ENV" => "development"}, "bin/rails db:migrate")
        puts "seeding development database"
    system!({"RAILS_ENV" => "development"}, "bin/rails db:seed")
  end

  create_database!("app_name_test") if !database_exists?("app_name_test")
  if database_exists?("app_name_test")
    puts "Migrating test database"
    system!({"RAILS_ENV" => "test"}, "RAILS_ENV=test bin/rails db:migrate")
  end

  puts "Installing foreman"
  system! "gem install foreman"
end

FileUtils.chdir APP_ROOT do
  check_dependencies
  run_setup

  puts <<~SUCCESS
    The app is setup
    run tests try:
        bin/test
    To run as a server, try:
        bin/start
  SUCCESS
end

Verify The Scripts on CI

If these scripts aren’t run often and kept up to date, they are bound to break and go un-noticed. It is helpful to verify the scripts on CI. It also generally had the dual purpose of helping keep the script working on both Docker and Macs. Let’s take a look at the things we can run on CI to help verify our app can be setup and run for folks immediately.

While the examples I am giving are for CircleCI, they should easily integrate into any continuous integration toolchain.

Verifying bin/setup & bin/start on CI

  check_bin_setup:
    <<: *base-job
    steps:
      - checkout
      - restore_cache:
          key: *gem_cache_key
      - run:
          name: Run setup
          command: bin/setup
      - run:
          name: Run setup again (idempotency check)
          command: bin/setup
      - run:
          name: Boot dev server in the background
          command: bin/start
          background: true
      - run:
          name: Start the rails server, wait for it to be available, then make a request and verify the response.
          command: |
            dockerize -wait tcp://localhost:3000/ -timeout 1m
            STATUS_CODE=`curl -s -o /dev/null -w "%{http_code}" -H "Accept: application/json;" -H "Content-Type: application/json"' http://localhost:3000/app_name/healthcheck`
            if [ $STATUS_CODE != "200" ]; then
                echo "Server failed to return a 200"
                exit 2
            fi
          environment:
            RAILS_ENV: development
            RACK_ENV: development

Bonus Points Verify Eager Load on CI

Long as we are verifying things work as expected on CI, another thing that can help avoid an incident is verifying the app works correctly with eager loading. We add a Rake task like below and have our CI testing script call it.

desc "Loads the Rails Application with 'eager_loading=true' as our production deployment eagerloads and we don't want to find out about a missing depedency that late."
task :eager_load_check do
  require File.expand_path('./config/environment', File.dirname(__FILE__))
  Rails.application.eager_load!
end

Enable Any Developer to Quickly Jump Into Any Repo

Suppose your teams make it easy to jump into any repository. In that case, you will help reduce siloing and encourage understanding not only the systems that teams own but working in and helping contribute to the systems in the ecosystem they participate in. It will promote collaboration and exploration and reduce the friction when folks move between teams and projects. There are other steps teams can take to reduce the friction of getting started, but maintaining these few scripts is a low-cost and high-value return for teams managing many repositories.