Score Report

In addition to the Grading Report, a score report may also be generated which only contains scoring information.

The score report is intended to be used as quick way to obtain scoring information for score aggregation, e.g. when generating a spreadsheet containing scores of all submissions, including per test case scores.

Generation

There are two ways a score report can be generated.

As part of the Grading Report
~~As an independent report~~ (TODO)

Embed in Grading Report

To enable embedding the score report into the grading report, add the following line to the Grader configuration file.

report.emitScoreReport=true

This change requires an application restart to take effect. Once the changes have been applied, all grading reports will contain an additional scoreReports key.

Emit an independent report

WIP

Score Report Format

ScoreReports:
  accScore: Double            # The accumulated score of all test cases of the submission
  accTotal: Double            # The accumulated total score of all test cases
  reports: [ScoreReportUnit]  # The array of score information which contributed to the accumulated score.

ScoreReportUnit:
  stageMangledName: String      # The mangled name of the stage which generated the score report
  testCaseMangledName: String   # The mangled name of the test case which generated the score report
  stageReportPath: String       # The JSON Path expression of the stage report, relative to the Grading Report, which 
                                # this score report is a part of
  testCaseReportPath: String    # The JSON Path expression of the test case, relative to the Grading Report, which 
                                # generated the score report
  displayName: String           # The display name for the user
  score: Double?                # The score of this test case achieved by the submission
  total: Double                 # The total score of this test case
  hash: String                  # A unique hash of this test case; Generated using stageMangledName and 
                                # testCaseMangledName

Mangled Name and Report Path

In each score report unit, both mangled names and report paths are generated.

Mangled name is an auto-generated name of a stage to uniquely identify it within a pipeline. The current implementation uses the location of the stage within the pipeline, the type of the stage, and optionally nested attributes to generate the name. The mangled name is currently used to generate the stage hash, and may be useful when tracing the source of a score report to the stage which generated it.

See Name Mangling for more information regarding the mangling scheme.

Report path is a JSON path to the stage report which generated the score report. This field is useful when looking up the “cause” of a score, where implementations can use the JSON path to directly lookup the stage report and/or the test case which led to the score.

Stage and Test Case

In each score report unit, MangledName and ReportPath both contains stage and testCase variations.

The difference between testCase and stage is that some stages generate stage reports which contains scores for more than one test case. Moreover, implementations may opt to lookup the report in either level as they see fit, for example testCase report may be more useful when displaying the cause of failure, whereas stage report may be more useful when displaying the stage information as a whole.

Therefore, names and paths to both the test case and the stage is generated. In general:

stage refers to the stage which the report or name is based on
testCase refers to the specific test case which the report or name is based on

For example, in a pipeline with multiple JUnit test cases:

stage: The stage which the report is base on, e.g. testCaseReportPath = $.stageReports.jUnit[0]
testCase: The test case which the report is based on, e.g. testCaseReportPath = $.stageReports.jUnit[0].report.testsuites[0].testcases[0]

Name Mangling

In order to differentiate different test cases within an assignment configuration, each test case must have a unique name within the pipeline. At the same time, each test case name should also be consistent between different submissions, as score reports of different submissions should be convertible to a spreadsheet, with each column matching a single test case.

Currently, only a small number of pipeline stages implement unique identifiers. To address this issue, a name mangling scheme is implemented to allow for unique name generation between different stages and test cases.

The name mangling lexicon is as follows (using a loose variation of eBNF):

upper case letter = ? Regular Expression "[A-Z]" ? ;
letter = ? Regular Expression "\w" ? ;
non zero digit = ? Regular Expression "[1-9]" ? ;
digit = ? Regular Expression "\d" ? ;
alphanum char = letter | digit ;
identifier char = alphanum character | "_" ;
char = ? Regular Expression "." ? ;

num idx = "0"
        | non zero digit, { digit } ;
alphanum idx = '"' , { char } , '"' ;

(* The index for array or map based lookup *)
idx = num idx
    | alphanum idx ;

(* The stage type *)
type = upper case letter , { alphanum char } ;

(*The stage component identifiers *)
identifier = { identifier char } ;

(* The stage name portion of the mangled name *)
stage = "pipeline[" , idx , "](" , type, ")" , [ "[" , idx , "]" ]    (* Root-level Stage Representation *)
                                                                      (*   References a stage in the root-level pipeline *)
      | stage , "$" , stage ;                                         (* Nested-level Stage Representation *)
                                                                      (*   References a nested stage in a non-root-level pipeline *)

(* The optional components within a stage *)
component = identifier                      (* Qualifies a member within a stage *)
          | identifier , "." , component ;  (* Qualifies a nested member within a stage *)

(* The full mangled name *)
name = stage                        (* Stage-only representation *)
     | stage , "::" , component ;   (* Stage+Component representation *)
                                    (*   For additional memberwise qualification within a stage *)