# Score Report

In addition to the [Grading Report](Report.md), a score report may also be generated which only contains scoring 
information.

The score report is intended to be used as quick way to obtain scoring information for score aggregation, e.g. when 
generating a spreadsheet containing scores of all submissions, including per test case scores.

## Generation

There are two ways a score report can be generated.

1. As part of the Grading Report
2. ~~As an independent report~~ (TODO)

### Embed in Grading Report

To enable embedding the score report into the grading report, add the following line to the Grader configuration file.

```
report.emitScoreReport=true
```

This change requires an application restart to take effect. Once the changes have been applied, all grading reports will
contain an additional `scoreReports` key.

### Emit an independent report

WIP

## Score Report Format

```
ScoreReports:
  accScore: Double            # The accumulated score of all test cases of the submission
  accTotal: Double            # The accumulated total score of all test cases
  reports: [ScoreReportUnit]  # The array of score information which contributed to the accumulated score.

ScoreReportUnit:
  stageMangledName: String      # The mangled name of the stage which generated the score report
  testCaseMangledName: String   # The mangled name of the test case which generated the score report
  stageReportPath: String       # The JSON Path expression of the stage report, relative to the Grading Report, which 
                                # this score report is a part of
  testCaseReportPath: String    # The JSON Path expression of the test case, relative to the Grading Report, which 
                                # generated the score report
  displayName: String           # The display name for the user
  score: Double?                # The score of this test case achieved by the submission
  total: Double                 # The total score of this test case
  hash: String                  # A unique hash of this test case; Generated using stageMangledName and 
                                # testCaseMangledName
```

### Mangled Name and Report Path

In each score report unit, both mangled names and report paths are generated.

Mangled name is an auto-generated name of a stage to uniquely identify it within a pipeline. The current implementation 
uses the location of the stage within the pipeline, the type of the stage, and optionally nested attributes to generate 
the name. The mangled name is currently used to generate the stage hash, and may be useful when tracing the source of a 
score report to the stage which generated it.

See [Name Mangling](#name-mangling) for more information regarding the mangling scheme.

Report path is a JSON path to the stage report which generated the score report. This field is useful when looking up 
the "cause" of a score, where implementations can use the JSON path to directly lookup the stage report and/or the test 
case which led to the score.

### Stage and Test Case

In each score report unit, `MangledName` and `ReportPath` both contains `stage` and `testCase` variations.

The difference between `testCase` and `stage` is that some stages generate stage reports which contains scores for more
than one test case. Moreover, implementations may opt to lookup the report in either level as they see fit, for example
`testCase` report may be more useful when displaying the cause of failure, whereas `stage` report may be more useful 
when displaying the stage information as a whole.

Therefore, names and paths to both the test case and the stage is generated. In general:

- `stage` refers to the stage which the report or name is based on
- `testCase` refers to the specific test case which the report or name is based on

For example, in a pipeline with multiple JUnit test cases:

- `stage`: The stage which the report is base on, e.g. `testCaseReportPath = $.stageReports.jUnit[0]`
- `testCase`: The test case which the report is based on, e.g. 
`testCaseReportPath = $.stageReports.jUnit[0].report.testsuites[0].testcases[0]`

## Name Mangling

In order to differentiate different test cases within an assignment configuration, each test case must have a unique 
name within the pipeline. At the same time, each test case name should also be consistent between different submissions,
as score reports of different submissions should be convertible to a spreadsheet, with each column matching a single 
test case.

Currently, only a small number of pipeline stages implement unique identifiers. To address this issue, a name mangling
scheme is implemented to allow for unique name generation between different stages and test cases.

The name mangling lexicon is as follows (using a loose variation of eBNF):

```
upper case letter = ? Regular Expression "[A-Z]" ? ;
letter = ? Regular Expression "\w" ? ;
non zero digit = ? Regular Expression "[1-9]" ? ;
digit = ? Regular Expression "\d" ? ;
alphanum char = letter | digit ;
identifier char = alphanum character | "_" ;
char = ? Regular Expression "." ? ;

num idx = "0"
        | non zero digit, { digit } ;
alphanum idx = '"' , { char } , '"' ;

(* The index for array or map based lookup *)
idx = num idx
    | alphanum idx ;

(* The stage type *)
type = upper case letter , { alphanum char } ;

(*The stage component identifiers *)
identifier = { identifier char } ;

(* The stage name portion of the mangled name *)
stage = "pipeline[" , idx , "](" , type, ")" , [ "[" , idx , "]" ]    (* Root-level Stage Representation *)
                                                                      (*   References a stage in the root-level pipeline *)
      | stage , "$" , stage ;                                         (* Nested-level Stage Representation *)
                                                                      (*   References a nested stage in a non-root-level pipeline *)

(* The optional components within a stage *)
component = identifier                      (* Qualifies a member within a stage *)
          | identifier , "." , component ;  (* Qualifies a nested member within a stage *)

(* The full mangled name *)
name = stage                        (* Stage-only representation *)
     | stage , "::" , component ;   (* Stage+Component representation *)
                                    (*   For additional memberwise qualification within a stage *)
```