Reliability of Tests in an Execution Plan

Problem

In a continuous integration environment tests are ideally executed at least once per day for testing the daily build and ensuring the quality of your application under test. To understand how reliable your test set is for measuring the quality of your AUT it is inevitable to sometimes have a look at how the results change over time. For example you might have tests in your test set that frequently change status, therefore being no real measure for quality.

Solution

Use the data mart view RV_TestStatusExtended to create a report that lists the results for a specific test in the context of a specific execution plan. This allows you to see how this test’s results have changed over time. For convenience, we will narrow the list of results down to those related to tagged builds, thus looking at specific milestone builds of the application under test only. This report collects test result data for tests in the context of execution plans and builds. In the following query we:

  • Select the columns we want to display from this view.
  • Narrow the result down by the ID of the test we want to investigate and the ID of the execution plan in which the test belongs.
  • Add a constraint to consider tagged builds only.
SELECT TestName, ExecutionPlanName, VersionName, BuildName, TestRunID, 
  PassedCount, FailedCount, NotExecutedCount
FROM RV_TestStatusExtended
WHERE TestID = ${TESTID|1|Test ID} AND ExecutionPlanID = ${EXECUTIONPLANID|1|Execution Plan ID} AND BuildIsTagged = 1
ORDER BY BuildOrderNumber
The result of the SQL query are all test runs for the selected test within the selected execution plan. In the following example you can see that the test was re-run against build 579_Drop2:
TestName Execution PlanName Version Name BuildName TestRunID Passed Count FailedCount NotExecutedCount
UI Tests EN|SQL2012|IE9|IIS 3.0 579_Drop02 7741797 59 5 0
UI Tests EN|SQL2012|IE9|IIS 3.0 579_Drop02 7745078 63 1 0
UI Tests EN|SQL2012|IE9|IIS 3.0 593_Drop03 7787437 63 1 0
UI Tests EN|SQL2012|IE9|IIS 3.0 605_Drop04 7848720 63 1 0