XUnit

Important: This is not a stage. This is just a common specification for all testing frameworks which emits XUnit XML format reports.

Weighted Scorable

See Scorable.md for the specification of weighted scorables.

Override:
  score: Double                     # The overriding score of this test case
  joinPolicy: JoinPolicy?           # The boolean operation for joining multiple predicates
  className: Predicate.CharSeq?     # Predicate for the XUnit test case class name
  testName: Predicate.CharSeq?      # Predicate for the XUnit test case name
  displayName: Predicate.CharSeq?   # Predicate for the XUnit test case display name

It is an error if:
- No predicates (i.e. at least one of className, testName, displayName) are defined
- joinPolicy is not defined when multiple predicates are specified

Weighted Scorable Example

The following example is a generic demonstration of how to use the scoreWeighting field. It is not reflective of a specific framework.

xunit:
  # ...
  scoreWeighting:
    default: 1.0                        # All test cases have a default score of 1.0
    limit: null                         # The stage does not have an upper limit for the score of each test case.
    overrides:                          # List of all overriding predicates...
      - score: 2.0                      # Override 1: Test cases matching the predicate will have a score of 2.0;
        className:                      #             Introduce a predicate for the class name of the test case...
          op: EQ                        #             ... that if the class name is equal to...
          value: "DoubleWeightedTests"  #             ... "DoubleWeightedTests".
      - score: 3.0                      # Override 2: Test cases matching the predicate will have a score of 3.0...
        joinPolicy: AND                 #             ... only if all listed predicates are satisfied;
        className:                      #             Introduce the first predicate for the class name of the test case...
          op: EQ                        #             ... that if the class name is equal to...
          value: "TripleWeightedTests"  #             ... "TripleWeightedTests".
        testName:                       #             Introduce the second predicate for the test case name...
          op: REGEX_EQ                  #             ... that if the test case name matches the regular expression of...
          value: "testTriple.+"         #             ... "testTriple.+"

The above configuration will result in the following score-determining logic (in Kotlin-pseudocode):

score = when {
    testcase.className == "DoubleWeightedTests" -> 2.0
    testcase.className == "TripleWeightedTests" && testcase.testname.matches(Regex("testTriple.+")) -> 3.0
    else -> 1.0
}

Report

<stage_name>:
  - hasTimedOut: Boolean          # Whether the stage timed out while executing
    exitCode: Int                 # The exit code of the application.
    stderr: [String]              # Standard error output, if any.
    score: Double?                # Scoring information of this stage.
    report:                       # XUnit report content
      tests: Int                  # Total number of tests executed
      skipped: Int                # Total number of tests skipped
      failures: Int               # Total number of tests failed
      errors: Int                 # Total number of tests which resulted in an error
      testsuites:                 # List of all test suites
      - name: String              # Name of the test suite
        displayName: String       # Display name of the test suite
        tests: Int                # Number of tests executed in the test suite
        failures: Int             # Number of tests failed in the test suite
        skipped: Int              # Number of tests skipped in the test suite
        errors: Int               # Number of tests which resulted in an error in the test suite
        testcases:                # List of all test cases within the test suite
        - name: String            # Name of the test case
          displayName: String     # Display name of the test case
          classname: String       # Class name which this test case belongs to
          failures:               # List of failures encountered while running this test case
          - category: Category    # Category of failure
            message: String?      # Failure message
            type: String?         # Failure type
            context: String       # Failure context

<stage_name> is the name of the stage which emits the XUnit XML report.
Category: ERROR | FAILURE | SKIPPED

Example: JUnit

Example: JUnit via Gradle

Example: GTest

Example: PyLint

Example: PyTest