Running a suite of tests in the same order always can mask unintentional dependencies between tests. You can end up with a test that succeeds in a full run of the suite but fails by itself.
One way to shake out these dependencies is to run tests in a random order. RSpec has an –order option that you can use to randomize test order for each run.
rspec --order rand
Recent versions of RSpec enable random ordering by default in generated spec_helper.rb files.
It’s important though to be able to reproduce a specific run order to track down issues when they pop up. RSpec accepts a seed value to initiate a specific order.
rspec --order rand:1234
At work we wanted to randomize test order on our CI server. We run our projects on TeamCity which does a nice job of parsing and reporting test run results. It does this through a custom Rake wrapper that does not output the order seed into the build logs.
A coworker Mark Starkman pointed us to this blog post with a handy shell one-liner for converting a Git commit hash into a seed numeric value.
$(git rev-parse HEAD | tr -d 'a-z' | cut -b 1-5)
Armed with this, I created a Command Line build step that populates an environment variable at runtime with a commit hash generated number. TeamCity recognizes a special format to populate parameters like environment variables at runtime.
Next up I updated the Rake step that handles RSpec runs to pass the environment variable in as the seed value.
Now each build log on our CI server includes the command needed to reproduce the test run order on any developer system.