# StdioTest Stage for grading submissions against standard input/output using declared test cases. - Kind: Grading Stage - Behaviour: - Runs the executable according to a run command in the terminal, applying standard input and command line arguments - The resultant standard output will be compared (diff-ed) against an expected output, in which the correctness of a `case` will be determined - Score for this `case` will be granted fully only if it is correct - If execution exceeds time limit specified in `_settings.stage_wait_duration_secs`, it will be treated as incorrect - `cases` against the same executable file are grouped as a `testCase` - Throws: - [`FieldsIncorrectlyUsedError`](../../model/Config.md) if `_settings.lang` does not correspond to a distro - [`HeredocDelimitingError`](../../model/Config.md) against `stdin` and `expected` - [`FileNotFoundError`](../../model/Config.md) if file specified in `file_stdin` and `file_expected` cannot be found ## Config ``` stdioTest: testCases: - file: String # file to run id: Int # id for this case args: [String]? # optional command line arguments stdin: String? # optional standard input file_stdin: String? # optional helper file for providing standard input expected: String? # optional expected output of the case file_expected: String? # optional helper file for providing expected output visibility: Visiblity # visibility of the result of this case hide_from_report: [HiddenItem]? # items to be hidden from report score: Double? # scores for passing this case, if any (if not specified, scoring is disabled for this test case) valgrind: Valgrind? # test case-specific override for Valgrind [default: null (following the `valgrind` key)] diff_ignore_flags: [DiffIgnoreFlag]? # list of diff ignore flags to be used for comparing standard output against expected output compat:denyStderr: Boolean? # whether to treat `stderr` output as a failing condition [default: false] additional_packages: [String]? # list of additional packages to install [default: []] additional_pip_packages: [String]? # list of additional PIP packages to install [default: []] ``` - `Visiblity: ALWAYS_VISIBLE | ALWAYS_HIDDEN | VISIBLE_AFTER_GRADING | VISIBLE_AFTER_GRADING_IF_FAILED` - The visibility of the test case to the student - Individual report elements can be hidden via the `hide_from_report` flags - `ALWAYS_VISIBLE`: Test case information is always available to students - `ALWAYS_HIDDEN`: Test case information is always hidden from students (except whether the test case has passed, and the score of the test case) - `VISIBLE_AFTER_GRADING`: Test case information is visible after the deadline of the assignment - `VISIBLE_AFTER_GRADING_IF_FAILED`: Test case information is visible after the deadline of the assignment *only if* the test case has failed - `DiffIgnoreFlag: TRAILING_WHITESPACE | SPACE_CHANGE | ALL_SPACE | BLANK_LINES` - Additional flags to provide to the `diff` application to ignore certain types of differences - `TRAILING_WHITESPACE`: Ignores line differences which only differ by trailing whitespace - `SPACE_CHANGE`: Ignores line differences which only differ by the change in number and/or type of whitespace - `ALL_SPACE`: Ignores all types of space changes, including between whitespace and no whitespace - `BLANK_LINES`: Ignores all changes whose lines are blank - `HiddenItem: STDIN | STDOUT | STDERR | DIFF` - Individual elements of the StdioTest report to hide; Overrides the visibility setting if set - `STDIN`: Hides the standard input from the report - `STDOUT`: Hides the standard output from the report - `STDERR`: Hides the standard error from the report - `DIFF`: Hides the diff between the expected and actual output from the report - `file_stdin` and `file_expected`: These files are expected to be within the `provided` directory. If these fields are set, specifying their in-file text counterpart will return error - `score`: Caveat - If `valgrind.score` is set to `null` and `stdioTest.score` is set to `null`, the `Valgrind` stage will use the `score` value of the global `Valgrind` configuration. To disable this behavior, use `0` as the `stdioTest.score`. - `Valgrind`: see the [config](Valgrind.md): of Valgrind stage - `compat:denyStderr`: If set to `true`, any `stderr` output from the executable will be treated as the test case failing. - **Treating `stderr` as a failing condition will become the default in the next release of the ZINC Grader. To opt out of this behavior, set this field to `false`.** - `additional_packages`: Additional packages to install before running the executable - `additional_pip_packages`: Additional PIP packages to install before running the executable - If the `lang` is not `python`, you must specify `python3-pip` in `additional_packages` to first install Pip. ## Report ``` stdioTest: - file: String # file to run id: Int # id for this case args: String # command line arguments concatenated stdin: [String] # standard input, could be an empty list stdout: [String] # the standard output of the submission upon execution stderr: [String] # the standard errors generated during execution exitCode: Int # the first non-zero exit code in the sequence of commands executed, or zero if no command # exited with a non-zero exit code expect: [String] # expected output of the case visibility: Visiblity # visibility of the result of this case diff: [String] # the diff generated output between expect and stdout (in this order) hasTimedOut: Boolean # whether the execution has timed out isCorrect: Boolean # whether the case is considered correct score: Score? # scores for passing this case and the total score exeExitCode: Int # The exit code of the executable file diffExitCode: Int # The exit code of diff ``` [Example](examples/stdio_report.yaml) See [Scorable.md](../Scorable.md) for more information on `score`.