Test outcome

[thekid]

Scope of Change

At the moment, there are three possible outcomes of a unittest: Succeeded,
Skipped and Failed. This RFC suggests adding more kinds of outcomes:
Warnings, Error and NotRun.

Rationale

Be able to distinguish between the different non-success reasons.

Functionality

The new outcomes are represented by these new classes:

<?php
  // Test succeeded but PHP warnings and/or notices were raised
  class TestWarnings implements TestFailure {
  }

  // An exception other than unittest.AssertionFailedError was raised
  class TestError implements TestFailure {
  }

  // The test was not run because an @ignore annotation was present
  class TestNotRun implements TestSkipped {
  }
?>

The existing classes TestSuccess, TestFailure and TestSkipped are turned
into interfaces and the concrete implementations moved to the following
new classes:

<?php
  // Test succeeded
  class TestExpectationMet implements TestSuccess {
  }

  // An AssertionFailedError was thrown - e.g. because one of the 
  // assert*() methods failed or fail() was called explicitely
  class TestAssertionFailed implements TestFailure {
  }

  // A PrerequisitesNotMetError was thrown in setUp() or @beforeClass
  class TestPrerequisitesNotMet implements TestSkipped {
  }
?>

Definitions

These current outcomes are defined:

• "." - Succeeded (Test was successful)
• "S" - Skipped (Test was skipped by either an error inside setUp() or by
  an @ignore annotation)
• "F" - Failed (Test failed due to an assertion error, an expected exception
  not being caught or by an exception raised by the code inside).

Example output:

  [..SSFFF]
  2 succeeded, 2 skipped, 3 failed

This RFC suggests to add the following new outcomes:

• "W" - Warning (One or more PHP errors were raised)
• "E" - Error (An exception other than AssertionFailedError was thrown when
  executing the test and no @expected annotation existed)
• "N" - Not run (Test was ignored by @ignore, as stated above, this is currently
  also reported as "skipped")

Example output:

  [..SNFFE]
  2 succeeded, 2 skipped, 3 failed

Security considerations

n/a

Speed impact

n/a

Dependencies

No BC break: All instanceof checks on TestFailure, TestSuccess and
TestSkipped will continue to work as expected due to the classes being
converted to interfaces.

Related documents

• http://msdn.microsoft.com/en-us/library/ms243180(VS.100).aspx
• http://www.phpunit.de/manual/current/en/textui.html
• http://www.nunit.org/index.php?p=nunit-console&r=2.5.2

Implementing patch:

• http://xp-framework.net/rfc/contrib/rfc0188.diff