Assignment Report

The report json fully captures important reporting information from a grading pipeline specified by a config yaml.

Here it is presented as YAML for simplicity.

Grammar

configError: ConfigError?
contextError: ContextError?
pipelineError: PipelineError?
stageReports: StageReports?

Errors

Errors that are caught during the entire grading process are reported. There are currently 3 different types of errors serving specific purposes.

ConfigError

These are syntactical and statically thrown errors during parse time of the config yaml.

  • Any caught incorrect syntax, undesired behaviors, unsupported options will be reported.

A config yaml must be free of any config errors in order to proceed to pipeline generation and execution.

configError:
  configFormatError: ConfigFormatError?
  langFormatError: LangFormatError?
  langUnsupportedError: LangUnsupportedError?
  fieldUnsupportedError: FieldUnsupportedError?
  fieldsUndeclaredError: [FieldsUndeclaredError]?
  fieldsIncorrectlyUsedError: [FieldsIncorrectlyUsedError]?
  heredocDelimitingError: [HeredocDelimitingError]?

ConfigFormatError

A Yaml formatting error that result in failure in parsing. The user needs to provide a valid yaml.

  • There is always just zero or one ConfigFormatError because parsing will early exit upon an error.

  configFormatError:
    errorField: String                   # first field that is syntactally incorrect in the yaml
    knownValues: [String]                # list of known legal values for this field instead
    fieldPath: String                    # the location of this field in this yaml tree, as a json path
    throwableSummary: ThrowableSummary?  # detailed information of the exception

LangFormatError

As specified in _settings.lang, the lang follows a specific syntax. Failing to comply will result in this error.

  langFormatError:
    actual: String                       # user erroneous input 
    message: String                      # [default: "$lang[$/compiler]:$version"]

LangUnsupportedError

Currently the support for which _settings.lang is available for use is limited. This error is shown if the user-chosen lang (follows the valid format but) is not supported.

  langUnsupportedError:
    lang: String                         # user input lang
    version: String?                     # user input lang version, if any parsed
    compiler: String?                    # user input lang compiler, if applicable and any parsed

FieldUnsupportedError

Error for unidentified toplevel fields of the config yaml.

  • This currently means a declared <stageName> is not available for use.

  fieldUnsupportedError:
  - field: String                        # user input field

FieldsUndeclaredError

Some stages depend on particular config and keys declared in the config yaml for them to work. This error is thrown if such dependencies are violated.

  fieldsUndeclaredError:
  - stageUsed: String                    # the stage used that has unfulfilled dependencies
    optionUsed: String?                  # the option used that triggers the dependency. This is optional because somtimes the stage itself is the trigger.
    missingAnyOfFields: [String]?        # list of involved fields that should be declared, in which one of them should be used
    missingAllOfFields: [String]?        # list of involved fields that should be declared, in which all should be used
    message: String?                     # optional extra message explaining this situation

FieldsIncorrectlyUsedError

Some stages depend on particular config and keys declared in the config yaml for them to work, but even when the field is present and legal, its value can be incorrect to use; this is when this error is used.

  fieldsIncorrectlyUsedError:
  - stageUsed: String                    # the stage used that has unfulfilled dependencies
    optionUsed: String?                  # the option used that triggers the dependency. This is optional because somtimes the stage itself is the trigger.
    problematicFields: [String]?         # list of involved fields that are specified incorrectly
    message: String?                     # optional extra message explaining this situation

HeredocDelimitingError

This is a special formatting error to cover an edge case.

  • Due to how config parsing is implemented internally, there is a syntactical restriction to the config yaml keys that will be read literally.

  • E.g. keys like list of files for _settings.template and test case standard input and expected values.

  • The restriction is that no line should contain EOF unless otherwise specified. If this is violated and detected, this error will be shown.

  heredocDelimitingError:
  - fieldPath: String                    # the location of this field in this yaml tree, as a json path
    stdin: String                        # the user-provided value for this field, which would be used as some form of standard input
    message: String                      # [default: "Please do not put 'EOF' on its own line within the body of your multiline input"]

ContextError

These are dynamically thrown errors during parse time of the config yaml.

  • Context errors concern the environment or context of the config and its pipeline, any missing prerequisites that may prevent the pipeline from executing successfully shall be noted here.

Unlike config errors, context errors are advisory (currently): having them will not stop the config from generating and executing the pipeline, albeit undesirable results.

contextError:
  fileNotFoundError: [FileNotFoundError]?

FileNotFoundError

If helper files like template, skeleton, provided files are required in some stages, but these files cannot be found uploaded to the file system already, this error will be shown as a warning.

  fileNotFoundError:
  - envHostRoot: String                  # the environment root directory used to search for the helper
    requested: String                    # the helper requested, i.e. template, skeleton, or provided

PipelineError

These are dynamically thrown errors during run time of the config yaml, i.e. pipeline execution.

  • These errors are intended for debugging purposes as they faithfully report error trace and other technical information in general

pipelineError:
  nonFatalError: [NonFatalError]?
  fatalError: FatalError?

NonFatalError

Any stage that failed to execute (call PipelineStage.onExecfailed) at any step before completion will throw a non-fatal error.

  • These are non-fatal, meaning that even if a stage failed to run, the pipeline could still continue from that point.

  • There can be many non-fatal errors.

  nonFatalError:
  - className: String                     # the pipeline stage class that failed to execute
    message: String?                      # optional generated message providing more context

FatalError

Any exceptional behaviour beyond recovery during pipeline execution will result in this error.

  • There is always just zero or one FatalError because pipeline execution can only early exit or finish entirely.

  fatalError:
    type: String                          # the exception class name involved
    message: String?                      # optional generated message providing more context
    stacktrace: [String]                  # the stack trace of this exception

StageReports

Each pipeline stage implements its own stage report unit, so the config structure (json) is up to the designer of the pipeline stage.

Stage reports are outputted as a list of Stage.Report, grouped by the respective stage which emitted the report.

For stages currently available, see document under the pipeline directory.

stageReports:
  <stageName1>: [<StageName1.Report>]
  <stageName2>: [<StageName2.Report>]
  <stageName3>: [<StageName3.Report>]
  # ...

ScoreReports

The grading report may embed a score report which only contains scoring information.

See the documentation for more details.