From : http://www.hward.com/end-to-end-testing-with-rspec-integration/

Testing end-to-end? Remember, more could be counter productive

Posted on Posted in Functional Testing, Software Testing

One of the projects I am currently working on had a formal end-to-end testing phase. There are many interrelated systems, and end-to-end testing is a good exercise to ensure that system works as expected.

However, it’s important to remember that executing more tests, especially during end-to-end testing phase could be counter productive. In general, I prefer less because constraints make me focus on right and important things.

So for end-to-end testing, let’s assume that product in question is a web-based e-commerce product such as Amazon. A typical end-to-end test for such a product could be on the lines of

  • User purchases an item
  • An order is generated
  • User gets order confirmation
  • Inventory is adjusted
  • Dispatch system receives order information
  • Dispatch is scheduled
  • Order status is updated
  • User receives notification about dispatch
  • Item is dispatched
  • Order status is updated
  • Item is delivered
  • Order status is updated
  • User gets an email

A simple user journey such as buying an item triggers action in many connected systems. These systems are often independent, and end-to-end testing is performed to ensure that they work together as expected.

So it is important to perform end-to-end testing, but it’s equally important to ensure that scope of end-to-end testing is not increased unnecessarily. Unnecessary scenarios included in this phase can increase the time it would take to execute them. It is important to understand (and educate others if needed) that purpose of end-to-end tests is not regression or functional testing. Unusual cases related to the shopping cart (can add 100 items?) or dispatch/scheduling (can deliver on 29’Th Feb?) etc. are important, but they should be handled during functional testing of the components and not during end-to-end testing.

End to end testing phase should be as small as possible and any request to include scenarios in this phase should be questioned. The value of including a particular scenario should be clear and communicated to everyone. I often ask following questions to see if there are any unnecessary scenarios –

  • What value are we getting by adding/executing this test?
  • How is this test different from other tests?
  • What’s different about this test and does this different matter?
  • Is this test uncovering something we don’t know already?
  • Has it been tested before?
  • What’s the opportunity cost? Is there anything more important to test?

Remember, executing more tests does not add value and especially in the context of end-to-end tests, executing more tests could be counter productive. Think about the time it takes to set-up data, users, coordinating execution with everyone, monitoring all the messages and ensuring that everything works – It’s time-consuming and could be draining.

Always think of the opportunity cost – what else you could be doing if you didn’t have to execute your current test? Is there something else more important, useful, valuable you could do? We are already pressed for time and have plenty of challenges already.

Not doing something which could be more valuable can make end-to-end testing counter productive.

Agree? Have you been involved in end-to-end testing phase? What has been your experience? How have you handled it?

Please follow and like us: