Cpp - Meta Stage for StdioTest on C++ code

Meta stage for creating compile and StdioTest stage for grading submissions against standard input/output using declared test cases.

• Kind: Grading Stage

• Behaviour:

  • Construct a pipeline consist of compile stage and a stdiotest stage

  • Takes a list of input files and compiles to one output executable, then the submission is tested with the defined test cases.

  • See Compile and StdioTest

• Throws:

  • See Compile and StdioTest

Config

cpp:
  input: [String]          # list of input files to be compiled
  output: String           # name of output executable
  testCases:
  - id: Int                # id for this case
    args: [String]?        # optional command line arguments
    stdin: String?         # optional standard input
    file_stdin: String?    # optional helper file for providing standard input
    expected: String?      # optional expected output of the case
    file_expected: String? # optional helper file for providing expected output
    visibility: Visiblity  # visibility of the result of this case
    score: Double?         # scores for passing this case, if any (if not specified, scoring is disabled for this test case)
    valgrind: Valgrind?    # test case-specific override for Valgrind [default: null (following the `valgrind` key)]
  diff_ignore_flags: [DiffIgnoreFlag]?  # list of diff ignore flags to be used for comparing standard output against expected output

• Visiblity: ALWAYS_VISIBLE | ALWAYS_HIDDEN | VISIBLE_AFTER_GRADING | VISIBLE_AFTER_GRADING_IF_FAILED

• DiffIgnoreFlag: TRAILING_WHITESPACE | SPACE_CHANGE | ALL_SPACE | BLANK_LINES

• file_stdin and file_expected: These files are expected to be within the provided directory. If these fields are set, specifying their in-file text counterpart will return error

• score: Caveat - If valgrind.score is set to null and stdioTest.score is set to null, the Valgrind stage will use the score value of the global Valgrind configuration. To disable this behavior, use 0 as the stdioTest.score.

• Valgrind: see the config: of Valgrind stage

Report

• See Compile and StdioTest

stdioTest:
  - file: String           # file to run
    id: Int                # id for this case
    args: String           # command line arguments concatenated
    stdin: [String]        # standard input, could be an empty list
    stdout: [String]       # the standard output of the submission upon execution
    stderr: [String]       # the standard errors generated during execution
    exitCode: Int          # the first non-zero exit code in the sequence of commands executed, or zero if no command 
                           # exited with a non-zero exit code
    expect: [String]       # expected output of the case
    visibility: Visiblity  # visibility of the result of this case
    diff: [String]         # the diff generated output between expect and stdout (in this order)
    hasTimedOut: Boolean   # whether the execution has timed out
    isCorrect: Boolean     # whether the case is considered correct
    score: Score?          # scores for passing this case and the total score
    exeExitCode: Int       # The exit code of the executable file
    diffExitCode: Int      # The exit code of diff

Example

See Scorable.md for more information on score.