CmdBuilder Usage

This is an introduction to using the builder CmdBuilder (Domain-specific language) for constructing commands for a pipeline stage.

Audience and Purposes

This piece of documentation is useful when implementing new custom pipeline stages that are executed in an environment - that is making a new DockerPipelineStage.

The property DockerPipelineStage.cmdSpec is the place to declare what command this stage will execute, so new stages should override it. While DockerPipelineStage ultimately accepts plain text commands (essentially an embedded shell script) as a (multiline) string, it is easier to do the same with a CmdBuilder, which handles many low-level details and provides convenience for common command constructs.

Especially since pipeline stages often need to emit outputs to generate reports - if the stage is Reportable, there are boilerplate commands necessary to store command outputs within the script (by piping to a file, registering to an environment variable, etc.), making the command script cumbersome to read; CmdBuilder will generate and expand the commands into that form on behalf of the author, to let them focus on the command’s core logic.

Getting Started

Let’s walk through an example of writing a command for getting the list of files in the stage’s working directory.

class ListFiles : DockerPipelineStage(/* ... */), Reportable {
    // ... define Config and parsing logic, and other essential overrides.
    
    // 1. Create a variable if we need to collect outputs from commands.
    //      Make one variable per different needed command.
    private val files = Var("files")
    // 2. Implement the command specification using `buildCmd`
    //      `add(variable) { "<command>" }` means append "<command>" to the script,
    //      and store its command output to the `variable`.
    override val cmdSpec = buildCmd {
        add(files) { "ls -al" }
    }
    // 3. After execution as we generate a report unit for this stage, use the `files` variable to retrieve command outputs.
    //      Assume some `Report` data class is defined within this stage.    
    data class Report(val listOfFiles: List<String>, val isSuccess: Boolean)
    override val reportUnit by lazy {
        Report(
            // Var.stdout returns the stdout of the variable as a list of string.
            listOfFiles = files.stdout,
            // Var.stderr works just like Var.stdout to return a list of string, amendable to common list methods.
            // Var.exitcode returns the exit code for this line of command as in integer.
            isSuccess = files.stderr.isEmpty() && files.exitcode == 0
        )   
    }
}

To report more detailed information, we can use more advanced commands:

class ListFilesDetailed : DockerPipelineStage(/* ... */), Reportable {
    // ... define Config and parsing logic, and other essential overrides.
    
    private val fullForm = Var("full")
    private val visibleFiles = Var("visible")
    private val allFiles = Var("all")
    private val numFiles = Var("numFiles")
    override val cmdSpec = buildCmd {
        add(fullForm) { "ls -al" }      
        // `..` is used to concat command strings: a space will be added if both sides are non-empty strings.  
        add(allFiles) { "ls" .. "-a" }
        add(visibleFiles) { "ls" }
        // Reuse the result of `visibleFiles` (the file storing the stdout) for `numFiles`
        add(numFiles) { "cat" .. visibleFiles.Stdout.asFile .. "| wc -c" }
    }

    data class Report(
        val listOfFiles: List<String>, val isSuccess: Boolean,
        val numOfFiles: Int, val hiddenFiles: List<String>
    )
    override val reportUnit by lazy {
        Report(
            listOfFiles = fullForm.stdout, isSuccess = fullForm.stderr.isEmpty() && fullForm.exitcode == 0,
            // Since Var.stdout/stderr/exitcode just return ordinary values, they can be massaged to a form we want to 
            // populate the report.
            numOfFiles = numFiles.stdout.trim(), hiddenFiles = allFiles.stdout - visibleFiles.stdout - listOf(".", "..")
        )   
    }
}

Advanced features

These can be used when specify the cmdSpec:

  • Files that are passed into the stage (e.g. provided files as part of the grading package) can be accessed, so we can use these filenames to construct commands as well.

  • To explicitly exit the command spec using an exit code, use exitWith(num) where e.g. num = 1 for constant exit codes, or exitCode(variable) for using the exit code of the variable to determine the whole command’s exit code.

  • For oneliner commands with no add calls, we can use buildCmdWith { "<command>" } in place of buildCmd.

  • Heredoc can store a multiline string for commands to use within the command spec. Use makeHeredoc or makePipedHeredoc.

  • Import extra command line tools for building custom commands by using DockerPipelineStage.installPacakges to override DockerPipelineStage.dockerfileSpec. Check that the package name is consistent with what’s available in the distro chosen. The default distro is Debian.

  • To save generated files from a stage so that we can read it after execution (e.g. a test case result file emitted by a testing framework), pipe it in the command to /log, so that it can be retrieved by manually walking the graderLogPath.

  • Be default, only the gist of the commands without the outputting constructs (known as the “core” commands) will be logged, use logFullCmd() to enable logging of expanded actual commands.