Assignment Configuration

The config yaml fully characterizes a user-defined grading pipeline:

Grammar

_settings:
  - ...

<stageName1>:
  - ...

<stageName2>:
  - ...

...

See Settings for information regarding the _settings key
See Stage Configuration for information regarding stage configuration keys.

This is likely to be readjusted in the future to further promote extensibility

Settings

_settings is for general configurations across the pipeline.

_settings:
  lang: String                # specify language/compiler/version
  use_template: (PATH | FILENAMES)?      # enable use of template helper files (what students should submit) by uploading. PATH: take from `template` helper file, FILENAMES: take from `_settings.template` list of files, null: disable use of template 
  template: [String]?         # enable use of template helper by listing out the files as paths
  use_skeleton: Boolean?      # enable use of skeleton helper files (what students receive from start) by uploading
  use_provided: Boolean?      # enable use of provided helper files (file teaching staff needs to grade) by uploading
  stage_wait_duration_secs: Long? # maximum time for a stage to execute before recognizing as a timeout in seconds [default: 60 (1 minute)]
  cpus: Double?               # number of CPUs a stage have access to, can be fractional [default: 2.0]
  mem_gb: Double?             # amount of memory a stage have access to in GBs [default: 4]
  early_return_on_throw: Boolean? # whether the pipeline will early return when any stage returns non-zero exit code [default: false]
  enable_features:            # specifies list of features to enable/disable; can be left unspecified to use default values
    network: Boolean?         # whether stages can connect to the internet [default: true]
    gpu_device: ~             # list of GPU vendor names whose GPUs are necessary in the grading process [default: null]

`_settings.lang`

Specifies the language used for grading, which in turn allows the Grader to determine the appropriate image to use for the grading pipeline.

The general format of this key is ${language}[/${compiler}]:${version}, where

language is the language which the grading is conducted in
compiler is the compiler used for the specified language, and may also be used to specify addition features required by the grading environment
version is the version of the language/compiler/runtime that should be used, depending on the language/compiler pair

The following subsections will explain the different languages supported by the Grader.

C

The language of C pipeline is c/${compiler}:${version}.

The ${compiler} must be one of:

gcc: Uses GNU C++ compiler
clang: Uses LLVM Clang compiler
- Note that this option is currently unsupported, as the Docker images have been made obsolete

The ${version} refers to the version of the compiler as set in ${compiler}.

For c/gcc, the version may be the full version (9.3.0), major+minor version (9.3), or the major version (9)
- Refer to here for all available versions
For c/clang, the version must be major-version only (11)
- Refer to here for all available versions

For example, to use GCC 11.2.0, you should specify the value as c/gcc:11.2.0 or c/gcc:11.2 (if you allow patch-level versions, e.g. 11.2.1, to be used).

C++

The language of C++ pipeline is cpp/${compiler}:${version}.

The ${compiler} must be one of:

g++: Uses GNU C++ compiler
clang++: Uses LLVM Clang++ compiler
- Note that this option is currently unsupported, as the Docker images have been made obsolete

The ${version} refers to the version of the compiler as set in ${compiler}.

For cpp/g++, the version may be the full version (9.3.0), major+minor version (9.3), or the major version (9)
- Refer to here for all available versions
For cpp/clang, the version must be major-version only (11)
- Refer to here for all available versions

For example, to use GCC 11.2.0, you should specify the value as cpp/g++:11.2.0 or cpp/g++:11.2 (if you allow patch-level versions, e.g. 11.2.1, to be used).

C++ with Qt

C++ pipelines can additionally request to use Qt libraries. These pipelines have a language of qt.

The version refers to the version of Debian to create the Qt pipeline from. The version must be the name of the Debian version (buster) or the version code in major+minor version format (10.11) or major version only (10).

The version of compiler cannot be specified using this language. The version matrix for each supported version is shown in the table below.

Debian Version	Qt Version	g++ version
10 (buster)	5.11.3	8.3.0

For example, to use the version of Qt bundled with Debian Buster, you should specify the value as qt:buster.

Java

The language of Java pipeline is java/${version}. The ${compiler} section is not supported by this language, and will default to using Oracle OpenJDK.

The ${version} refers to the version of the OpenJDK compiler/runtime.

For Java versions 8 or below, the version may be the full version (8u322) or the major version only (8)
For Java versions 9 or above, the version may be the full version (17.0.2), major+minor version (17.0), or major version only (17)

Refer to here for all available versions.

For example, to use OpenJDK 11, you should specify the value as java:11 (which will default to the latest version of OpenJDK 11 available).

Python

The language of Python is python/${version}. To enable CUDA support for Python pipelines, one should use python/cuda:${version} instead (see here).

The version refers to the version of the Python runtime. The version may be the full version (3.8.7), major+minor version (3.8), or major version only (3). Refer to here for all available versions.

For example, to use Python 3.9, you should specify the value as python:3.9.0 or python:3.9 (if you allow patch-level versions, e.g. 3.9.10, to be used).

Python with CUDA

Python pipelines can additionally request to use CUDA. These pipelines have a language of python/cuda.

The version refers to the version of the CUDA runtime. The version must be the full version of the CUDA runtime (11.5.1), and may additionally have the -cudnn postfix to indicate that cuDNN libraries are required as well. Refer to here for all available versions.

The version of Python cannot be specified using this language. The version matrix for each version is shown in the table below.

CUDA Version	cuDNN Version (if enabled)	Ubuntu Base Version	Python Version
11.x.y	8	20.04.x (Focal)	3.8.x
10.x.y	8	18.04.x (Bionic)	3.6.x
9.2	7	18.04.x (Bionic)	3.6.x
9.x (except 9.2)	7	16.04.x (Xenial)	3.5.x
8.x	7	16.04.x (Xenial)	3.5.x
7.x	6	14.04.x (Trusty)	3.4.x

For example, to use CUDA 11.5.1, you should specify the value as cuda/python:11.5.1, or cuda/python:11.5.1-cudnn if cuDNN is needed.

`_settings.enable_features.gpu`

If specified, requests the Grader to allocate a GPU from any of the specified vendor(s).

This key is useful when grading assignments which require the use of GPUs, such as computer graphics and machine learning.

The recognized values for this key are:

An array containing one or more of:
- NVIDIA: Any NVIDIA GPU
- AMD: Any AMD GPU
- INTEL: Any Intel GPU
ANY: Any GPU, including virtual devices (e.g. exposed by virtual machine hosts)

Stage Configuration

A single <stageName> top-level key corresponds to a pipeline stage.

<stageName>:
  # stage-specific settings...

Declaring the stage configuration implies using it; Not declaring means opting out.
The validity of the configuration will be checked; only those defined in the project (in the search path) will be used.
- An invalid stage name will render the config yaml invalid.
Important: Always declare stages in the order you wish for it to execute in.
- The Grader is not guaranteed to fix bad pipeline stage ordering, meaning that the Grader may accept “execution-before-compilation”!

Advanced Usage: Multiple Stages of Same Type

Multiple stage definitions with the same type can be specified as follows:

<stageName>:<id1>:
  # configuration for 1st stage with type <stageName>

<stageName>:<id2>:
  # configuration for 2nd stage with type <stageName>

<stageName>:<id3>:
  # configuration for 3rd stage with type <stageName>

# ...

<id*> can be composed of any character except :.
- It is recommended to use [A-Za-z0-9]+, since the above requirement may change in the future.
<id*> must be unique within the same <stageName> (as required by YAML)