The Secrets of Successful Test Automation

by Brad on October 25, 2011

Lisa Crispin presented The Secrets of Successful Test Automation at yesterday’s Agile Denver meeting. I’ve summarized them here with some of my interpretations.

  1. Whole team ownership of and commitment to quality and testing. Leverage everyone’s strengths.
  2. Pair programmers with testers for both unit and acceptance tests. My commentary: do this regularly even if you don’t do it full time.
  3. Use a framework that allows non-programmers to write test cases (like fitnesse)
  4. Use a programming language for tests (or your automated test framework) that the programmers know well
  5. Use continuous integration for fast feedback
  6. Run periodic “engineering sprints” when team can do whatever they want (e.g. to reduce debt or upgrade tools)
  7. Make it a team responsibility to automate all regression tests. Require everyone to run manual regression tests to motive them to automate.
  8. Stop the line to fix failing tests. My commentary: this requires commitment to quality from management and discipline by the development team.
  9. Continually refactor test code, just like production code. Test code deserves same respect as production code.
  10. Under-commit. Plan less than you think you can do so you have time for unexpected things and test automation. My commentary: I don’t like to use the term “under commit” because it’s likely to get managers and stakeholders to push back. What it really comes down to is working at a sustainable pace, with a commitment to quality and discipline.
  11. Expect some time to “get over the hump” before the investment in test automation pays off with reduced testing effort.

There were a few items Lisa didn’t explicitly list as secrets but were implied:

  • passionate people
  • strong teamwork and collaboration

Others in the audience shared some interesting ideas of their own. Here are a few:

  • Programmers write some happy path automated tests first and then testers can expand on them.
  • Periodically set aside a day as testing day for the whole team. (Suggested by Mark Waite)
  • Use Test-driven bug fixing. Require and automated test for every bug fix. This can also be a way to get the team to “dip their toes in the water” of TDD.
  • Budget some amount of team capacity every sprint for debt reduction.
Peter Westlund October 25, 2011 at 6:11 am

Thanks för a great summon up on what makes automated tests successful in Agile. Using cucumber myself, but if developers don’t know ruby or rspec very well it’s a little hard to sell the idea to dev teams. Sure will look into fitnesse.

Peter Westlund (@bastlund)

Charles Bradley October 25, 2011 at 1:41 pm


Thanks so much for the summary. I’m going to give your post a shout out on my blog.

Jon Archer October 25, 2011 at 3:39 pm

Nice summary. I’d planned on making one for my teammates at work, but now I’ll just point them here 🙂

PM Hut October 26, 2011 at 4:11 am

You can eliminate all the other “secrets”and leave just the first one “Whole team ownership of and commitment to quality and testing”. Code ownership is the essence of it all, and it will guarantee successful testing (whether it’s automated or not).

Chris October 26, 2011 at 5:47 pm

Great summary! Would someone mind expanding a bit on this thought? I don’t know if I completely understand what is being said here:

Stop the line to fix failing tests. My commentary: this requires commitment to quality from management and discipline by the development team.

Brad October 26, 2011 at 6:50 pm

“Stop the line” means whenever a test is failing, the team’s top priority is to fix it, even if other work must be delayed. It requires management commitment to support this practice and not to undermine it. It requires team discipline to stop the line when it may be tempting to just ignore the failing test – “we’ll fix it later”. We all no that “later” often means “never”.

Mikalai Alimenkou October 27, 2011 at 2:02 am

Thanks a lot for sharing these notes. Many useful points. I want to add one important for fixing failed test. It must be responsibility of the team member who broke the test. Instead many teams delegate this responsibility to testers themselves. And as a result testers start continuous loop of fixing broken tests. While they try to fix one test somebody may break another one. Especially this issue is visible when development team is unbalanced (for example 5 developers and 1 tester). Fixing test by developer who broke it also helps to archive common responsibility for product quality.

Jonny Reber October 27, 2011 at 3:59 am

Excellent post and summary! I think I’ll print and stick up around our agile boards. We have spent the last year trying to implement numbers 1-11 at a huge UK media company. On the whole we have been successful but the biggest factor in making the above happen is captured in the lines:
– passionate people
– strong teamwork and collaboration

To which I would add ‘discipline – discipline – discipline’

James Moseley November 1, 2011 at 1:31 pm

Thanks for sharing this information Brad. If it wasn’t against company policy, I’d demonstrate my passion and commitment to these secrets with an honorary tattoo of 1-11 🙂

Wei Zhu March 21, 2012 at 2:29 am

Thanks for sharing!

For the point 2,3,4, if UI intensive, I would suggest to consider tools like Jbehave and SpecFlow.

Steve Wilson October 6, 2013 at 2:11 am

I’d like to add some input as a career Test Strategist….

Not all systems, sub-systems, operations and functions can be automated or have an ROI on the automation effort.

Some functions are so fast that factoring tool overheads actually makes the function slower !!!!

The implementation of an automated framework also needs to include a process – making the case for automation. The criteria determine which tests should/ could be automated.

Automated testing should be used to support manual testing effort. Without manual testing, the knowledge sharing and knowledge transfer mechanism breaks, reducing the clients teams EA working knowledge of their organisations.

Some tests, particularly in finance systems require calculations to be verified by a human being and a calculator e.g. Accrued Interest.

Short release cycles have always existed and due to the volume of change in a short period of time have a much higher % possibility of failed build and failure of delivered functionality. You could reduce non testing downstream by stretching your sprints into 4 week cycles. Whoops, this has gone full circle and now becomes planned builds in a waterfall scenario !!!

My message is – Agile is not the future state solution for all.


{ 6 trackbacks }

Previous post:

Next post: