Writing an Assignment Configuration

This is a quick-start guide to writing your own Assignment Configuration YAML. Note that this document is by no means comprehensive; The Grader is much more extensible than what is advertised here.

Template

A starting template can be found here. The file will contain a one-liner description of which required option does. Alternatively, an unannotated version can be found here.

The following sections will all use the above template to briefly explain the meaning of each configuration option. Always refer to configuration-specific documents for more details on how to use each stage!

Settings

Full Documentation Here.

The _settings block is used to configure global grading options.

_settings:
  lang: cpp/g++:8
  use_template: FILENAMES
  template:
    - do_not_include.txt
  use_skeleton: true
  use_provided: true
  stage_wait_duration_secs: 10
  cpus: 1.0
  mem_gb: 2.0
  enable_features:
    network: false
    gpu_device: ~

The most important configuration keys are explained below.

`lang`

This key determines the programming language which this grading pipeline is targeted for.

Certain stages, such as Compile and StdioTest, contain custom language-specific logic. By setting this key, it allows those stages to adapt its execution for the specified language.

The currently supported languages are:

cpp/g++:<gcc_version>
- Supports any GCC version 8 or above
- <gcc_version> can be the full version (9.3.0), major+minor version (9.3), or major version only (9)
java:<java_version>
- Supports any Java version 7 or above
- For Java versions 8 or below, <java_version> can be the full version (8u282) or major version only (8)
- For Java versions 9 or above, <java_version> can be the full version (11.0.10), major+minor version (11.0), or major version only (11)
python:<python_version>
- Supports Python 2 and 3
- <python_version> can be the full version (3.8.7), major+minor version (3.8), or major version only (3)

We also support additional frameworks like Qt for C++. For such cases, you will need to specify the framework instead of the programming language in this field.

cuda/python:<cuda_version>
- Uses Python bundled with the distribution, therefore its version cannot be specified
- Refer to table in Assignment Configuration for the supported versions matrix and the corresponding Python versions
- <cuda_version> must be the full version of CUDA (11.5.1), optionally postfixed by -cudnn if cuDNN libraries are needed (11.5.1-cudnn)
qt5:<debian_version>
- The Qt version is based on the version shipped by the Debian Linux distribution.
- We officially support the <debian_version> below, other versions are not tested:
  
  buster, with Qt5 5.11.3 and g++ 8.3.0

`use_template` and `template`

This key specifies whether template files or filenames are available for stages to use.

Template files (or filenames) are the list of files which students are expected to submit for an assignment. This list can be provided either as part of the Helper Files package, or listed in this configuration as part of template.

This field should be set to PATH if the template files are provided as part of the Helper Files package; This field should be set to FILENAMES if the template filenames are provided by this configuration, and _settings.template should be populated appropriately. To indicate that template files will not be used, this key should be removed.

_settings.template does not need to be present in the _settings object if use_template is not set to FILENAMES.

`use_skeleton`

This key specifies whether skeleton files are available for stages to use.

Skeleton files are the files provided to students which act as starting code for the assignment. This list can be provided as part of the Helper Files package.

This field should be set to true if the skeleton files are provided. Otherwise, the field can be set to false, or the field can be removed from the _settings object.

`use_provided`

This key specifies whether provided files are available for stages to use.

Provided files are the files used only in the grading process. These files are typically for internal use and are not provided to students. Examples for provided files are driver code for grading student submissions, and test case code which are hidden from students.

This list can be provided as part of the Helper Files package.

This field should be set to true if the provided files are provided. Otherwise, the field can be set to false, or the field can be removed from the _settings object.

`stage_wait_duration_secs`

This key specifies the timeout for each stage in seconds.

The most common purpose of this key is to limit how long stages can run, to allow the grading process to continue if specific test cases take too long (e.g. due to an infinite loop or broken I/O).

Tuning this number can be very difficult, as the optimal value depends on the longest running stage of your grading pipeline and the performance of the server. In general, you should time how long the grading process takes on your local machine, and multiply that by 2-3 times for a good estimate of this value.

Note

Do not set this value too low or too high. Setting this value too low will cause grading stages to be killed before it can start executing; Setting this value too high can cause other grading tasks (including those from other courses) to be choked out, creating a huge backlog of grading tasks!

A good minimum value is 10 seconds, whereas a good maximum value is 120 seconds. Only go above if your stage(s) require it!

`cpus`

This key specifies how many cores each stage can use.

In general, setting this key to 1.0 implies that a single core is available to each stage. More specifically, this value actually sets the amount of CPU-time stages are allocated, and 1.0 implies that one CPU worth of time is available to each stage. If the stage exhausts the allocated CPU-time, the task will be evicted by the OS scheduler until more CPU-time is available.

Note that the setting of this value may have an impact to _settings.stage_wait_duration_secs, as certain stages may be able to use multiple threads to accelerate its workload.

In general, this key should be set to 1.0, unless your grading pipeline contains stages which can exploit parallelism, such as Make (using jobserver mode).

Note

Do not set this value too high, as it may impact the throughput of the Grader with regards with grading other submissions. In fact, setting this value too high may cause your grading task to be rejected, as no online servers can fulfill the amount of CPUs requested.

Please be courteous to other users and only set this value up to what is necessary.

`mem_gb`

This key specifies how much RAM each stage can use.

In general, setting this key to 1.0 implies that 1 GB of RAM is available to each stage. If the stage exhausts the allocated RAM, the stage will be terminated with a SIGKILL (or equivalent) signal.

In general, this key should be set to 1.0, unless your grading pipeline contains stages which require more RAM, such as when using build systems for compilation.

Note

Do not set this value too high, as it may impact the throughput of the Grader with regards with grading other submissions. In fact, setting this value too high may cause your grading task to be rejected, as no online servers can fulfill the amount of RAM requested.

Please be courteous to other users and only set this value up to what is necessary.

`early_return_on_throw`

This key specifies whether to terminate the entire grading task if any non-failable stage has failed.

When certain pipeline stages fail, such as compilation stages, it often makes no sense to continue grading, as the artifacts required by subsequent stages are not generated. Setting this option to true will cause these stages to be marked as “unfailable”, meaning that if the grading task will be aborted if the stage fails.

Warning

This option is experimental, as the Grader has only been tested with this option turned off.

Use at your own risk!

`enabled_features`

This key specifies a list of features which should be enabled/disabled for this assignment configuration. The key accepts the following key-value pairs:

network (Default: true)

If enabled, all stages in the pipeline has access to the Internet.
gpu_device (Default: null)

If specified, the Grader will allocate a GPU from one of the specified vendor(s) and attach it to all pipeline stages.

The list of recognized vendors are:
- NVIDIA: Use any NVIDIA GPUs
- AMD: Use any AMD GPUs
- INTEL: Use any Intel GPUs
For example, to enable the use of NVIDIA GPUs (because, e.g., your pipeline stage requires the use of CUDA), set this key as:
```
gpu_device:
  - NVIDIA
```
To enable the use of either an AMD or Intel GPU (because, e.g., your pipeline stage requires using the Mesa open-source drivers), set this key as:
```
gpu_device:
  - AMD
  - INTEL
```
Additionally, this key also accepts a string ANY, which indicates that the pipeline requires the presence of any GPUs. Note that this option also considers GPUs exposed as virtual machine devices, e.g. VMWare VGA controller, as a viable candidate.

To enable the use of any GPU, set this key as:
```
gpu_device: ANY
```
Note

The Grader can only guarantee that a GPU of any of the listed vendors is allocated. Specific models of GPUs cannot be requested to be allocated, as the hardware backing the Grader may change without prior notice (including upgrades), which may silently break pipelines if a specific GPU is pinned.

Pipeline Stages Overview

To adapt to the different grading requirements for different courses, the grading process is represented as a list of execution stages, forming a grading pipeline. Each student submission will be executed using the grading pipeline.

Grader pipeline stages are generally categorized into three types: Pre-Grading Stages, Grading Stages, and Post-Grading Stages.

Configuration Format

When configuring a pipeline stage, the configuration will generally look like this:

<stageName>:
    key1: value1
    key2: value2
    key3: value3

Each <stageName> will represent one instance of the stage within the pipeline. The stage configurations should be declared in the order you want the stages to execute in.

For example, the following YAML will build a pipeline which runs the stages in the following order:

fileStructureValidation
diffWithSkeleton
compile
stdioTest
score

fileStructureValidation: {}
diffWithSkeleton: {}
compile:
    # ...
stdioTest:
    # ...
score:
    # ...

Note

The Grader may perform some rudimentary reordering to ensure that no illogical pipelines are created. For example, pipelines which execute stdioTest before compile may be reordered to ensure that compilation is always performed before testing.

Multiple Stages of Same Type

Multiple stages with the same type are also supported in the configuration format; Append : with a custom name to allow Grader to differentiate between the different stages.

For example, in a C++ separate compilation pipeline, the following shows an example of compiling two files into their respective object files, before linking together.

compile:main:
  input: ["main.cpp"]
  output: "main.o"
  flags:
    - '-c'

compile:student:
  input: ["student.cpp"]
  output: "student.o"
  flags:
    - '-c'

compile:link:
  input:
    - main.o
    - student.o
  output: a.out

Note that for consecutive stages of the same type, the Grader will execute each stage in the order it is declared. For the above configuration, this means that compile:main will first run, followed by compile:student, and finally compile:link.

Pre-Grading Stages

Pre-Grading Stages are pipeline stages which are performed before compilation and/or test execution. These stages are generally language-agnostic in nature.

`fileStructureValidation`

Full Documentation Here.

File Structure Validation is a stage which checks whether files submitted by the students matches the expected list of files that students should submit. If the student submission does not contain certain files, or the student submission contains additional files, this stage will emit a warning for students.

Note that in order to use this stage, _settings.use_template must not be null.

`diffWithSkeleton`

Full Documentation Here.

Diff With Skeleton is a stage which checks whether students have inadvertently submitted certain files from the skeleton code. If there are any files in the student submission which are identical to those in the skeleton code, this stage will emit a warning for students.

Note that in order to use this stage, _settings.use_skeleton must be true.

Grading Stages

Grading Stages are pipeline stages which are performed to either prepare or execute grading tasks. Some of these stages have the additional ability to be score-generating, i.e. a score can be generated from the stage based on its correctness.

Since different programming languages have different tools for compilation and testing, this section will be split based on the programming language.

Note that the following sections are not an exhaustive list of all supported stages. Refer to this page for a comprehensive list of supported stages and their configuration.

Language-Agnostic

`Compile`

Full Documentation Here.

The Compile stage is a language-agnostic stage which can be used in any pipeline with a supported _settings.lang. This allows you to only need to specify the input, output, and optionally the compilation flags to compile file(s) using the compiler.

`StdioTest`

Full Documentation Here.

The Stdio Test stage is a language-agnostic stage which is used to perform testing using Standard I/O (i.e. stdout and stderr) streams.

C++

`Make`

Full Documentation Here.

Ths Make stage is a stage which invokes GNU Make on Makefile for easier compilation.

`Valgrind`

Full Documentation Here.

The Valgrind stage is a stage which invokes Valgrind on an executable.

Note that the specification for Valgrind does not contain individual test cases. Instead, the Valgrind configuration is designed to be applied to all test cases within all stdioTest blocks. Fine-grained overrides can be configured using the stdioTest.testCases.valgrind key.

`GTest`

Full Documentation Here.

The GTest stage is a stage which compiles files containing GTest test cases and executes them.

Java

`JUnit`

Full Documentation Here.

The JUnit stage is a stage which compiles files containing JUnit test cases and executes them.

Note that this stage uses the JUnit Platform Runner provided by JUnit 5. While JUnit 5 test cases are natively supported, only part of JUnit 3 and JUnit 4’s feature set is supported.

`Gradle`

Full Documentation Here.

The Gradle stage is a stage which executes Gradle task(s) in a Gradle project.

Python

`PyTest`

Full Documentation Here.

The PyTest stage is a stage which executes PyTest test cases.

Note

To specify the Python modules required by this stage, use the additional_pip_packages key in the stage configuration.

Post-Grading Stages

`Score`

Full Documentation Here.

The Score stage performs aggregation on all score-generating stages preceding it.

Note

If this stage is not specified, an overall score will not be generated for submissions.

Writing an Assignment Configuration

Template

Settings

lang

use_template and template

use_skeleton

use_provided

stage_wait_duration_secs

cpus

mem_gb

early_return_on_throw

enabled_features

Pipeline Stages Overview

Configuration Format

Multiple Stages of Same Type

Pre-Grading Stages

fileStructureValidation

diffWithSkeleton

Grading Stages

Language-Agnostic

Compile

StdioTest

C++

Make

Valgrind

GTest

Java

JUnit

Gradle

Python

PyTest

Post-Grading Stages

Score

`lang`

`use_template` and `template`

`use_skeleton`

`use_provided`

`stage_wait_duration_secs`

`cpus`

`mem_gb`

`early_return_on_throw`

`enabled_features`

`fileStructureValidation`

`diffWithSkeleton`

`Compile`

`StdioTest`

`Make`

`Valgrind`

`GTest`

`JUnit`

`Gradle`

`PyTest`

`Score`