Valgrind

Stage for memory checking via Valgrind.

  • Kind: Grading Stage

  • Required stages: stdioTest

  • Behavior: Runs Valgrind memory check over all test cases list in stdioTest

  • Throws:

Config

Global Configuration

For configuring Valgrind globally (i.e. for all test cases):

valgrind:
  enabled: Boolean?               # explicit override of using Valgrind or not (especially useful in StdioTest.Config) [default: true]
  args: [String]?                 # Valgrind command line options [default: []]
  checksFilter: [String]?         # list of filters against the kind of Valgrind errors [default: ["*"]]
  visibility: Visibility?         # specifies how this test case should be displayed in report rendering. INHERIT means taken from the visibility of the same test case in StdioTest [default: INHERIT]
  compat:noMatchStdout: Boolean   # whether to compare the `stdout` of the executable when determining the correctness of the test case
  score: Double?                  # Score of this stage. If not defined, uses stdioTest.score first, then cascades to global 
                                  # valgrind.score if still null.
                                  # 0.0 may be used to indicate that this stage should not contribute to the final score.

  • Visiblity: ALWAYS_VISIBLE | ALWAYS_HIDDEN | VISIBLE_AFTER_GRADING | VISIBLE_AFTER_GRADING_IF_FAILED | INHERIT

  • checksFilter takes values: *, Leak_*, Uninit*, *Free

    • which maybe mapped to just ‘all’, ‘leak’, ‘uninitialized’, ‘free’ etc. in the UI

    • there may be hints in the ui to explain what these kinds/classes of bugs mean. e.g.

      • “Leak-class of bugs catches memory leaks within the application”

      • “Uninit-class of bugs catches usages of variables which were not initialized”

  • compat:noMatchStdout: If set to true, the stdout of the executable (within Valgrind) will not be used in determining the correctness of the test case.

    • In other words, the score will be solely determined by whether the executable emits errors in Valgrind.

    • The expected output and diff flags will be inherited from the corresponding stdioTest test case.

    • Including stdout for determining the correctness of the test case will become the default in the next release of the ZINC Grader. To opt out of this behavior, set this field to true.

Local Configuration

For fine-grained override/control in each testCase given that valgrind is defined at the top-level:

stdioTest:
  testCases:
    # ...
    valgrind: Valgrind?    # that is, the above config is nested under the `valgrind` key

Example

stdioTest:
  testCases:
    - file: a.out
      id: 1
      args: [1]
      stdin: ~
      expected: ~
      visibility: ALWAYS_VISIBLE
      score: 50
      valgrind:
        enabled: false  # Disables valgrind for this case
    - file: a.out
      id: 2
      args: [2]
      expected: ~
      visibility: ALWAYS_HIDDEN
      score: 50
      valgrind:
        visibility: INHERIT # Enables valgrind for this case since this case is ALWAYS_VISIBLE
        score: 50           # Sets the score of Valgrind to 50
    - file: a.out
      id: 3
      args: [3]
      expected: ~
      visibility: ALWAYS_HIDDEN
      score: 50
      valgrind:
        visibility: INHERIT # Enables valgrind for this case since this case is ALWAYS_VISIBLE; 
                            # Score is inherited from the test case, so it is 50.

Report

TODO

stageReports:
  valgrind:
    - executable: String     # executable for this test case
      id: Int                # test case id
      args: [String]         # command line arguments of the test case
      isSuccess: Boolean     # Whether this stage executed without errors
      hasTimedOut: Boolean   # whether the execution has timed out
      exitCode: Int          # Valgrind exit code
      stdout: [String]       # All standard output messages from the executable (Valgrind does not output to stdout)
      stderr: [String]       # All standard error messages from Valgrind and the executable
      errors:                # list of valgrind errors
        - kind: String       # kind of error
          what:              # list of bugs as a trace: from call site to bottom-most (system) function
          - whatText: String # bug description
            stack:           # call stack of this bug
            - ip: Int        # instruction pointer
              obj: String?   # object file
              fn: String?    # function called
              dir: String?   # parent directory of the object file
              file: String?  # involved source file ('culprit')
              line: Int?     # line number in source file
            aux: Boolean     # indicates whether or not this error is primary: the root error will have aux=false
      visibility: Visibility # specifies how this test case should be displayed in report rendering after resolution
      exeExitCode: Int       # the exit code from Valgrind
      diffExitCode: Int?     # the exit code from diff, if `compat:noMatchStdout` is set to `false`
      score: Score?          # Score of this stage.

Example