Writing a Python Assignment Configuration

This is a quick-start guide to writing your own Python Assignment Configuration YAML. Note that this document is by no means comprehensive; The Grader is much more extensible than what is advertised here.

Template

A starting template can be found here. The file will contain a one-liner description of which required option does. Alternatively, an unannotated version can be found here.

The following sections will all use the above template to briefly explain the meaning of each configuration option. Always refer to configuration-specific documents for more details on how to use each stage!

Settings

Full Documentation Here.

The _settings block is used to configure global grading options.

_settings:
  lang: python:3.8
  use_template: FILENAMES
  template:
    - do_not_include.txt
  use_skeleton: true
  use_provided: true
  stage_wait_duration_secs: 10
  cpus: 1.0
  mem_gb: 2.0
  enable_features:
    network: false
    gpu_device:
      - NVIDIA

The most important configuration keys are explained below. More info about other configuration keys are available in Writing an Assignment Configuration.

The _settings.enable_features.gpu_device key is only necessary if your pipeline requires access to a GPU, e.g. if your pipeline runs a Machine Learning pipeline.

lang

This key determines the programming language which this grading pipeline is targeted for.

Certain stages, such as Compile and StdioTest, contain custom language-specific logic. By setting this key, it allows those stages to adapt its execution for the specified language.

  • python:<python_version>

    • Supports Python 2 and 3

    • <python_version> can be the full version (3.8.7), major+minor version (3.8), or major version only (3)

  • cuda/python:<cuda_version>

    • Supports CUDA version 7+

    • Uses Python bundled with the distribution, and therefore its version cannot be specified

    • Refer to table in Assignment Configuration for the supported versions matrix and the corresponding Python versions

    • <cuda_version> must be the full version of CUDA (11.5.1), optionally postfixed by -cudnn if cuDNN libraries are needed (11.5.1-cudnn)

Typical Config Structure

fileStructureValidation: {}
diffWithSkeleton: {}
stdioTest:
    # ...
score:
    # ...

To include more than one compilation stage, see Multiple Stages of Same Type.

Pre-Grading Stages

Pre-Grading Stages are pipeline stages which are performed before compilation and/or test execution. These stages are generally language-agnostic in nature.

fileStructureValidation

Full Documentation Here.

File Structure Validation is a stage which checks whether files submitted by the students matches the expected list of files that students should submit. If the student submission does not contain certain files, or the student submission contains additional files, this stage will emit a warning for students.

Note that in order to use this stage, _settings.use_template must not be null.

diffWithSkeleton

Full Documentation Here.

Diff With Skeleton is a stage which checks whether students have inadvertently submitted certain files from the skeleton code. If there are any files in the student submission which are identical to those in the skeleton code, this stage will emit a warning for students.

Note that in order to use this stage, _settings.use_skeleton must be true.

Grading Stages

Grading Stages are pipeline stages which are performed to either prepare or execute grading tasks. Some of these stages have the additional ability to be score-generating, i.e. a score can be generated from the stage based on its correctness.

Note that the following sections are not an exhaustive list of all supported stages. Refer to this page for a comprehensive list of supported stages and their configuration.

Language-Agnostic

StdioTest

Full Documentation Here.

The Stdio Test stage is a language-agnostic stage which is used to perform testing using Standard I/O (i.e. stdout and stderr) streams.

Note

To specify the Python modules required by this stage, use the additional_pip_packages key in the stage configuration.

Python

PyTest

Full Documentation Here.

The PyTest stage is a stage which executes PyTest test cases.

Note

To specify the Python modules required by this stage, use the additional_pip_packages key in the stage configuration.

Post-Grading Stages

Score

Full Documentation Here.

The Score stage performs aggregation on all score-generating stages preceding it.

Note

If this stage is not specified, an overall score will not be generated for submissions.