StdioTest

Stage for grading submissions against standard input/output using declared test cases.

• Kind: Grading Stage

• Behaviour:

  • Runs the executable according to a run command in the terminal, applying standard input and command line arguments

  • The resultant standard output will be compared (diff-ed) against an expected output, in which the correctness of a case will be determined

  • Score for this case will be granted fully only if it is correct

  • If execution exceeds time limit specified in _settings.stage_wait_duration_secs, it will be treated as incorrect

  • cases against the same executable file are grouped as a testCase

• Throws:

  • FieldsIncorrectlyUsedError if _settings.lang does not correspond to a distro

  • HeredocDelimitingError against stdin and expected

  • FileNotFoundError if file specified in file_stdin and file_expected cannot be found

Config

stdioTest:
  testCases:
  - file: String                    # file to run
    id: Int                         # id for this case
    args: [String]?                 # optional command line arguments
    stdin: String?                  # optional standard input
    file_stdin: String?             # optional helper file for providing standard input
    expected: String?               # optional expected output of the case
    file_expected: String?          # optional helper file for providing expected output
    visibility: Visiblity           # visibility of the result of this case
    hide_from_report: [HiddenItem]? # items to be hidden from report
    score: Double?                  # scores for passing this case, if any (if not specified, scoring is disabled for this test case)
    valgrind: Valgrind?             # test case-specific override for Valgrind [default: null (following the `valgrind` key)]
  diff_ignore_flags: [DiffIgnoreFlag]?  # list of diff ignore flags to be used for comparing standard output against expected output
  compat:denyStderr: Boolean?           # whether to treat `stderr` output as a failing condition [default: false]
  additional_packages: [String]?        # list of additional packages to install [default: []]
  additional_pip_packages: [String]?    # list of additional PIP packages to install [default: []]

• Visiblity: ALWAYS_VISIBLE | ALWAYS_HIDDEN | VISIBLE_AFTER_GRADING | VISIBLE_AFTER_GRADING_IF_FAILED

  • The visibility of the test case to the student

  • Individual report elements can be hidden via the hide_from_report flags

  • ALWAYS_VISIBLE: Test case information is always available to students

  • ALWAYS_HIDDEN: Test case information is always hidden from students (except whether the test case has passed, and the score of the test case)

  • VISIBLE_AFTER_GRADING: Test case information is visible after the deadline of the assignment

  • VISIBLE_AFTER_GRADING_IF_FAILED: Test case information is visible after the deadline of the assignment only if the test case has failed

• DiffIgnoreFlag: TRAILING_WHITESPACE | SPACE_CHANGE | ALL_SPACE | BLANK_LINES

  • Additional flags to provide to the diff application to ignore certain types of differences

  • TRAILING_WHITESPACE: Ignores line differences which only differ by trailing whitespace

  • SPACE_CHANGE: Ignores line differences which only differ by the change in number and/or type of whitespace

  • ALL_SPACE: Ignores all types of space changes, including between whitespace and no whitespace

  • BLANK_LINES: Ignores all changes whose lines are blank

• HiddenItem: STDIN | STDOUT | STDERR | DIFF

  • Individual elements of the StdioTest report to hide; Overrides the visibility setting if set

  • STDIN: Hides the standard input from the report

  • STDOUT: Hides the standard output from the report

  • STDERR: Hides the standard error from the report

  • DIFF: Hides the diff between the expected and actual output from the report

• file_stdin and file_expected: These files are expected to be within the provided directory. If these fields are set, specifying their in-file text counterpart will return error

• score: Caveat - If valgrind.score is set to null and stdioTest.score is set to null, the Valgrind stage will use the score value of the global Valgrind configuration. To disable this behavior, use 0 as the stdioTest.score.

• Valgrind: see the config: of Valgrind stage

• compat:denyStderr: If set to true, any stderr output from the executable will be treated as the test case failing.

  • Treating stderr as a failing condition will become the default in the next release of the ZINC Grader. To opt out of this behavior, set this field to false.

• additional_packages: Additional packages to install before running the executable

• additional_pip_packages: Additional PIP packages to install before running the executable

  • If the lang is not python, you must specify python3-pip in additional_packages to first install Pip.

Report

stdioTest:
  - file: String           # file to run
    id: Int                # id for this case
    args: String           # command line arguments concatenated
    stdin: [String]        # standard input, could be an empty list
    stdout: [String]       # the standard output of the submission upon execution
    stderr: [String]       # the standard errors generated during execution
    exitCode: Int          # the first non-zero exit code in the sequence of commands executed, or zero if no command 
                           # exited with a non-zero exit code
    expect: [String]       # expected output of the case
    visibility: Visiblity  # visibility of the result of this case
    diff: [String]         # the diff generated output between expect and stdout (in this order)
    hasTimedOut: Boolean   # whether the execution has timed out
    isCorrect: Boolean     # whether the case is considered correct
    score: Score?          # scores for passing this case and the total score
    exeExitCode: Int       # The exit code of the executable file
    diffExitCode: Int      # The exit code of diff

Example

See Scorable.md for more information on score.