# Grader - Standalone version The Grader supports running via a Standalone runner. ## Building the Application The packaged JAR should be distributed alongside this document. If not, run `./gradlew standalone:shadowJar` to build the JAR. ## Usage Before using the Grader Standalone runner, it is advised to enclose all required files in the following directory structure: ``` - extracted/ - student1/ - student2/ - student3/ - provided/ - skeleton/ - template/ - generated/ - config.properties - config.yaml ``` - `extracted/`: Directory containing all extracted student submissions. One student should have one subdirectory. - `provided/`: Directory containing all TA helper files. - `skeleton/`: Optional - Skeleton files provided to the student - `template/`: Optional - Directory containing all files that should be submitted - `generated`: Optional - Directory containing all files that are generated from running the pipeline. Defaults to `root/generated/`. This folder will be automatically generated if not exist already. - `config.properties`: Configuration definition for the Grader profile - `config.yaml`: Configuration definition for assignment grading An example `config.properties` can be found [here](#example-grader-configuration). An explanation of the relevant keys can be found [here](#grader-profile-configuration). In general, keys in the configuration do not need to be changed. An example `config.yaml` can be found [here](#example-assignment-configuration). An explanation of the format can be found [here](../model/Config.md), and the pipeline stage definitions can be found under `pipeline`. After this is set up, change the working directory to the root of the directory, and execute the following script. Alternatively, you can also copy the JAR into the directory. ```sh java -jar zinc-jvm-staging-standalone-all.jar run \ --profile=$PWD/config.properties \ --config=$PWD/config.yaml ``` To verify the validity of the Grader profile configuration (`config.properties`), run the following script. ```sh java -jar zinc-jvm-staging-standalone-all.jar verify-profile \ --profile=$PWD/config.properties ``` To verify the validity of the assignment configuration (`config.yaml`), run the following script. ```sh java -jar zinc-jvm-staging-standalone-all.jar verify-config \ --profile=$PWD/config.properties \ --config=$PWD/config.yaml ``` ## Grader Profile Configuration The following is a list of all profile configuration keys relevant to the Grader Standalone version. - `context.inPaths.graderHostRoot`: Specifies the location of all input files on the host running the Grader. If using relative path and the profile is loaded from the filesystem, it is relative to **the location of the `config.properties` file**. - This key must be changed in conjunction with `context.inPaths.envHostRoot`. - `context.outPaths.graderHostRoot`: Specifies the location of all output files. If using relative path and the profile is loaded from the filesystem, it is relative to **the location of the `config.properties` file**. - Grading reports are all emitted into this directory. - This key must be changed in conjunction with `context.outPaths.envHostRoot`. - `context.logPaths.graderHostRoot`: Specifies the location of all temporary files. If using relative path and the profile is loaded from the filesystem, it is relative to **the location of the `config.properties` file**. - It is recommended to set this path to a high-speed storage, such as in a `tmpfs` partition. - This key must be changed in conjunction with `context.logPaths.envHostRoot`. - `context.submission.src`: Path to all submissions, relative to `context.inPaths.graderHostRoot`. - This key must contain at least one placeholder (`{}`) token, which indicates to the grader the submission directories. - If multiple placeholder tokens are present, all placeholders will be replaced by the directory name of the first placeholder token. For example, `{}/{}` in a directory containing `abc/` will be resolved to `abc/abc`, regardless of whether other directories are present under `abc/`. - `context.template.src`: Path to all template files, relative to `context.inPaths.graderHostRoot`. - `context.skeleton.src`: Path to all skeleton files, relative to `context.inPaths.graderHostRoot`. - `context.provided.src`: Path to all provided files, relative to `context.inPaths.graderHostRoot`. - `context.generated.src`: Path to all generated files, relative to `context.inPaths.graderHostRoot`. Defaults to `root/generated/` - `docker.daemon.main`: URLs of all Docker engines to use as the primary dispatcher. - URLs must be comma-separated. - `docker.host_reservation.main.cpu`: The amount of CPU to reserve for the host system (in terms of CPU time). - `docker.host_reservation.mem.cpu`: The amount of memory to reserve for the host system (in Megabytes). - `docker.overcommit.cpu`: The ratio of CPU to overcommit. - For example, if your machine has 4.0 CPU cores after host reservation and the overcommit ratio is 16, the Grader will allocate up to `4.0 x 16 = 64.0` cores of CPU time for grading tasks. - `docker.overcommit.mem`: The ratio of memory to overcommit. - For example, if your machine has 6.0 GiB after host reservation and the overcommit ratio is 1.5, the Grader will allocate up to `6.0 x 1.5 = 9.0 GiB` of memory for grading tasks. ## Example Grader Configuration ```yaml ## Settings - See docs/model/Config.md _settings: lang: cpp/g++:8 use_template: FILENAMES template: - consecutiveRNG.cpp - dictionaryImplementation.h - randomTextGenerator.cpp use_skeleton: true use_provided: true stage_wait_duration_secs: 10 cpus: 2 mem_gb: 2 early_return_on_throw: true ## Pipeline stages - See files under docs/pipeline fileStructureValidation: ignore_in_submission: [ ] diffWithSkeleton: exclude_from_provided: true compile: input: [ "*.cpp" ] output: a.out stdioTest: testCases: - file: a.out id: 1 args: [ 1 ] stdin: ~ expected: |- ================= Test 1 ================= GOOD. visibility: ALWAYS_VISIBLE score: 5 valgrind: enabled: false - file: a.out id: 2 args: [ 2 ] expected: |- ================= Test 2 ================= 0 visibility: ALWAYS_VISIBLE score: 5 valgrind: visibility: INHERIT - file: a.out id: 3 args: [ 3 ] expected: |- ================= Test 3 ================= 10 90 visibility: VISIBLE_AFTER_GRADING score: 5 - file: a.out id: 4 args: [ 4 ] expected: |- ================= Test 4 ================= m[x][x]=0 m[x][y]=1 m[x][z]=2 m[y][x]=23 m[y][y]=24 m[y][z]=25 m[z][x]=46 m[z][y]=47 m[z][z]=48 m[x][x]=0 m[x][y]=1 m[x][z]=2 m[y][x]=23 m[y][y]=24 m[y][z]=25 m[z][x]=46 m[z][y]=47 m[z][z]=48 Printing m: { ("x",{ ("x",0),("y",1),("z",2) }), ("y",{ ("x",23),("y",24),("z",25) }), ("z",{ ("x",46),("y",47),("z",48) }) } visibility: VISIBLE_AFTER_GRADING score: 5 - file: a.out id: 5 args: [ 5 ] expected: |- ================= Test 5 ================= Original from original.txt: Word1 Word2 Word3 Word4 Word5 Word6 Word7 WordA WordX WordB WordX WordA visibility: VISIBLE_AFTER_GRADING score: 5 diff_ignore_flags: [ TRAILING_WHITESPACE ] valgrind: args: - --error-limit=no - --leak-check=full - --show-reachable=yes - --num-callers=20 - --track-origins=yes - --read-inline-info=yes visibility: ALWAYS_HIDDEN score: normalizedTo: 100.0 ``` ## Example Assignment Configuration ``` context.inPaths.graderHostRoot=. context.inPaths.envHostRoot=. context.outPathsRoots.graderHostRoot=./grader-out context.outPathsRoots.envHostRoot=./grader-out context.logPaths.graderHostRoot=/tmp/grader-log context.logPaths.envHostRoot=/tmp/grader-log context.submission.src=extracted/{} context.template.src=template context.skeleton.src=skeleton context.provided.src=provided docker.daemon.main=unix:///var/run/docker.sock docker.host_reservation.main.cpu=2 docker.host_reservation.main.mem=2048 docker.overcommit.cpu=16 docker.overcommit.mem=1.5 ```