Let’s begin with some numbers about skroutz.gr:
- app/ 21k+ lines of code
- lib/ 15k+ lines of code
- about 380 classes
- spec/ 38k+ lines of code in ~350 files
- 6k+ spec examples
- takes about 7 minutes to run all specs
7 minutes is a LOT of time to wait before you merge and push your experimental feature branch to master and thus most developers don’t bother running the specs and stuff breaks. Our first solution was parallel_tests, a great gem that runs the specs on multiple cores of the same machine. This dropped the time to 4 minutes (about 40% drop). This of course wasn’t enough and so we decided to venture into distributed testing.
##Enter Kowalski!
Kowalski is our re-inventing of the distributed-testing-wheel. It consists of RSpec, Git, Capistrano (which were already part of our workflow) and Spork.
- Git it used to distribute the code (app and specs) to the runners
- Capistrano runs the necessary remote commands to setup and prepare the runners for the specs
- Spork loads up the environment so that running a bunch of spec files decomposes to a simple ssh command of “bundle exec rspec spec/path/to/file_spec.rb”
It is pivotal to state the importance of Spork into this mix. Our app takes about 10-30 seconds to load the environment (depending on the machine’s load). Supposing we have 7 runners, this delay accumulates to about 8 minutes if we send one ssh-rspec command per file and gets smaller if we send them in batches. Splitting the 350 files in equal parts and sending those batches of 50 files in one go is not an option because to get the total results we would have to wait for the slowest runner to finish (speed depending on machine hardware and system load). Spork and small batches allows us to benefit from active load-balancing and thus the fastest and less stressed machines get more specs to run.
This way, the CPU and memory bottleneck was widened. The next bottleneck to widen was our main DB, MySQL. To get around the slow disks issue, we could have switched to SSD. But why mess our hands with hardware when we have perfectly good software? Our solution layed on the hands of tmpfs. Tmpfs can mount a part of RAM somewhere on our filesystem which means we get an immense speed boost. The drawback for real-life applications is that once you unmount it, you lose all data. This drawback though, becomes a benefit when testing where we need a clean DB for running our tests. With tmpfs in place, we get a run time of 2 minutes (70% drop) and because of distribution, we hog no machine. Some details on Kowalski:
- needs a user with its own home directory to ssh in and operate from
- needs ruby and rubygems to be installed system-wide (sudo apt-get install ruby rubygems1.8)
- benefits from a line added to /etc/fstab for the tmpfs (optional)
- utilizes rbenv and bundler to keep its environment separated from the rest of the system
- a sinatra application exposes some commands (like report and up/down) the our webdevs