Grader - Standalone version
The Grader supports running via a Standalone runner.
Building the Application
The packaged JAR should be distributed alongside this document. If not, run ./gradlew standalone:shadowJar to build
the JAR.
Usage
Before using the Grader Standalone runner, it is advised to enclose all required files in the following directory structure:
- extracted/
- student1/
- student2/
- student3/
- provided/
- skeleton/
- template/
- generated/
- config.properties
- config.yaml
extracted/: Directory containing all extracted student submissions. One student should have one subdirectory.provided/: Directory containing all TA helper files.skeleton/: Optional - Skeleton files provided to the studenttemplate/: Optional - Directory containing all files that should be submittedgenerated: Optional - Directory containing all files that are generated from running the pipeline. Defaults toroot/generated/. This folder will be automatically generated if not exist already.config.properties: Configuration definition for the Grader profileconfig.yaml: Configuration definition for assignment grading
An example config.properties can be found here. An explanation of the relevant keys
can be found here. In general, keys in the configuration do not need to be changed.
An example config.yaml can be found here. An explanation of the format can be
found here, and the pipeline stage definitions can be found under pipeline.
After this is set up, change the working directory to the root of the directory, and execute the following script. Alternatively, you can also copy the JAR into the directory.
java -jar zinc-jvm-staging-standalone-all.jar run \
--profile=$PWD/config.properties \
--config=$PWD/config.yaml
To verify the validity of the Grader profile configuration (config.properties), run the following script.
java -jar zinc-jvm-staging-standalone-all.jar verify-profile \
--profile=$PWD/config.properties
To verify the validity of the assignment configuration (config.yaml), run the following script.
java -jar zinc-jvm-staging-standalone-all.jar verify-config \
--profile=$PWD/config.properties \
--config=$PWD/config.yaml
Grader Profile Configuration
The following is a list of all profile configuration keys relevant to the Grader Standalone version.
context.inPaths.graderHostRoot: Specifies the location of all input files on the host running the Grader. If using relative path and the profile is loaded from the filesystem, it is relative to the location of theconfig.propertiesfile.This key must be changed in conjunction with
context.inPaths.envHostRoot.
context.outPaths.graderHostRoot: Specifies the location of all output files. If using relative path and the profile is loaded from the filesystem, it is relative to the location of theconfig.propertiesfile.Grading reports are all emitted into this directory.
This key must be changed in conjunction with
context.outPaths.envHostRoot.
context.logPaths.graderHostRoot: Specifies the location of all temporary files. If using relative path and the profile is loaded from the filesystem, it is relative to the location of theconfig.propertiesfile.It is recommended to set this path to a high-speed storage, such as in a
tmpfspartition.This key must be changed in conjunction with
context.logPaths.envHostRoot.
context.submission.src: Path to all submissions, relative tocontext.inPaths.graderHostRoot.This key must contain at least one placeholder (
{}) token, which indicates to the grader the submission directories.If multiple placeholder tokens are present, all placeholders will be replaced by the directory name of the first placeholder token. For example,
{}/{}in a directory containingabc/will be resolved toabc/abc, regardless of whether other directories are present underabc/.
context.template.src: Path to all template files, relative tocontext.inPaths.graderHostRoot.context.skeleton.src: Path to all skeleton files, relative tocontext.inPaths.graderHostRoot.context.provided.src: Path to all provided files, relative tocontext.inPaths.graderHostRoot.context.generated.src: Path to all generated files, relative tocontext.inPaths.graderHostRoot. Defaults toroot/generated/docker.daemon.main: URLs of all Docker engines to use as the primary dispatcher.URLs must be comma-separated.
docker.host_reservation.main.cpu: The amount of CPU to reserve for the host system (in terms of CPU time).docker.host_reservation.mem.cpu: The amount of memory to reserve for the host system (in Megabytes).docker.overcommit.cpu: The ratio of CPU to overcommit.For example, if your machine has 4.0 CPU cores after host reservation and the overcommit ratio is 16, the Grader will allocate up to
4.0 x 16 = 64.0cores of CPU time for grading tasks.
docker.overcommit.mem: The ratio of memory to overcommit.For example, if your machine has 6.0 GiB after host reservation and the overcommit ratio is 1.5, the Grader will allocate up to
6.0 x 1.5 = 9.0 GiBof memory for grading tasks.
Example Grader Configuration
## Settings - See docs/model/Config.md
_settings:
lang: cpp/g++:8
use_template: FILENAMES
template:
- consecutiveRNG.cpp
- dictionaryImplementation.h
- randomTextGenerator.cpp
use_skeleton: true
use_provided: true
stage_wait_duration_secs: 10
cpus: 2
mem_gb: 2
early_return_on_throw: true
## Pipeline stages - See files under docs/pipeline
fileStructureValidation:
ignore_in_submission: [ ]
diffWithSkeleton:
exclude_from_provided: true
compile:
input: [ "*.cpp" ]
output: a.out
stdioTest:
testCases:
- file: a.out
id: 1
args: [ 1 ]
stdin: ~
expected: |-
================= Test 1 =================
GOOD.
visibility: ALWAYS_VISIBLE
score: 5
valgrind:
enabled: false
- file: a.out
id: 2
args: [ 2 ]
expected: |-
================= Test 2 =================
0
visibility: ALWAYS_VISIBLE
score: 5
valgrind:
visibility: INHERIT
- file: a.out
id: 3
args: [ 3 ]
expected: |-
================= Test 3 =================
10
90
visibility: VISIBLE_AFTER_GRADING
score: 5
- file: a.out
id: 4
args: [ 4 ]
expected: |-
================= Test 4 =================
m[x][x]=0
m[x][y]=1
m[x][z]=2
m[y][x]=23
m[y][y]=24
m[y][z]=25
m[z][x]=46
m[z][y]=47
m[z][z]=48
m[x][x]=0
m[x][y]=1
m[x][z]=2
m[y][x]=23
m[y][y]=24
m[y][z]=25
m[z][x]=46
m[z][y]=47
m[z][z]=48
Printing m:
{
("x",{ ("x",0),("y",1),("z",2) }),
("y",{ ("x",23),("y",24),("z",25) }),
("z",{ ("x",46),("y",47),("z",48) })
}
visibility: VISIBLE_AFTER_GRADING
score: 5
- file: a.out
id: 5
args: [ 5 ]
expected: |-
================= Test 5 =================
Original from original.txt:
Word1 Word2 Word3 Word4 Word5 Word6 Word7 WordA WordX WordB WordX WordA
visibility: VISIBLE_AFTER_GRADING
score: 5
diff_ignore_flags: [ TRAILING_WHITESPACE ]
valgrind:
args:
- --error-limit=no
- --leak-check=full
- --show-reachable=yes
- --num-callers=20
- --track-origins=yes
- --read-inline-info=yes
visibility: ALWAYS_HIDDEN
score:
normalizedTo: 100.0
Example Assignment Configuration
context.inPaths.graderHostRoot=.
context.inPaths.envHostRoot=.
context.outPathsRoots.graderHostRoot=./grader-out
context.outPathsRoots.envHostRoot=./grader-out
context.logPaths.graderHostRoot=/tmp/grader-log
context.logPaths.envHostRoot=/tmp/grader-log
context.submission.src=extracted/{}
context.template.src=template
context.skeleton.src=skeleton
context.provided.src=provided
docker.daemon.main=unix:///var/run/docker.sock
docker.host_reservation.main.cpu=2
docker.host_reservation.main.mem=2048
docker.overcommit.cpu=16
docker.overcommit.mem=1.5