# `CmdBuilder` Usage

This is an introduction to using the builder `CmdBuilder` (Domain-specific language) for constructing commands for a pipeline stage.

## Audience and Purposes

This piece of documentation is useful when implementing new custom pipeline stages that are executed in an environment -
that is making a new `DockerPipelineStage`.

The property `DockerPipelineStage.cmdSpec` is the place to declare what command this stage will execute, so new stages 
should override it. While `DockerPipelineStage` ultimately accepts plain text commands (essentially an embedded shell script)
as a (multiline) string, it is easier to do the same with a `CmdBuilder`, which handles many low-level details and provides
convenience for common command constructs. 

Especially since pipeline stages often need to emit outputs to generate reports - 
if the stage is `Reportable`, there are boilerplate commands necessary to store command outputs within the script (by piping
to a file, registering to an environment variable, etc.), making the command script cumbersome to read; `CmdBuilder` will
generate and expand the commands into that form on behalf of the author, to let them focus on the command's core logic.

## Getting Started

Let's walk through an example of writing a command for getting the list of files in the stage's working directory.

```kotlin
class ListFiles : DockerPipelineStage(/* ... */), Reportable {
    // ... define Config and parsing logic, and other essential overrides.
    
    // 1. Create a variable if we need to collect outputs from commands.
    //      Make one variable per different needed command.
    private val files = Var("files")
    // 2. Implement the command specification using `buildCmd`
    //      `add(variable) { "<command>" }` means append "<command>" to the script,
    //      and store its command output to the `variable`.
    override val cmdSpec = buildCmd {
        add(files) { "ls -al" }
    }
    // 3. After execution as we generate a report unit for this stage, use the `files` variable to retrieve command outputs.
    //      Assume some `Report` data class is defined within this stage.    
    data class Report(val listOfFiles: List<String>, val isSuccess: Boolean)
    override val reportUnit by lazy {
        Report(
            // Var.stdout returns the stdout of the variable as a list of string.
            listOfFiles = files.stdout,
            // Var.stderr works just like Var.stdout to return a list of string, amendable to common list methods.
            // Var.exitcode returns the exit code for this line of command as in integer.
            isSuccess = files.stderr.isEmpty() && files.exitcode == 0
        )   
    }
}
```

To report more detailed information, we can use more advanced commands:

```kotlin
class ListFilesDetailed : DockerPipelineStage(/* ... */), Reportable {
    // ... define Config and parsing logic, and other essential overrides.
    
    private val fullForm = Var("full")
    private val visibleFiles = Var("visible")
    private val allFiles = Var("all")
    private val numFiles = Var("numFiles")
    override val cmdSpec = buildCmd {
        add(fullForm) { "ls -al" }      
        // `..` is used to concat command strings: a space will be added if both sides are non-empty strings.  
        add(allFiles) { "ls" .. "-a" }
        add(visibleFiles) { "ls" }
        // Reuse the result of `visibleFiles` (the file storing the stdout) for `numFiles`
        add(numFiles) { "cat" .. visibleFiles.Stdout.asFile .. "| wc -c" }
    }

    data class Report(
        val listOfFiles: List<String>, val isSuccess: Boolean,
        val numOfFiles: Int, val hiddenFiles: List<String>
    )
    override val reportUnit by lazy {
        Report(
            listOfFiles = fullForm.stdout, isSuccess = fullForm.stderr.isEmpty() && fullForm.exitcode == 0,
            // Since Var.stdout/stderr/exitcode just return ordinary values, they can be massaged to a form we want to 
            // populate the report.
            numOfFiles = numFiles.stdout.trim(), hiddenFiles = allFiles.stdout - visibleFiles.stdout - listOf(".", "..")
        )   
    }
}
```

## Advanced features

These can be used when specify the `cmdSpec`:

- Files that are passed into the stage (e.g. `provided` files as part of the grading package) can be accessed, so we can
use these filenames to construct commands as well.
- To explicitly exit the command spec using an exit code, use `exitWith(num)` where e.g. `num = 1` for constant exit codes,
 or `exitCode(variable)`
for using the exit code of the `variable` to determine the whole command's exit code.
- For oneliner commands with no `add` calls, we can use `buildCmdWith { "<command>" }` in place of `buildCmd`.
- Heredoc can store a multiline string for commands to use within the command spec. Use `makeHeredoc` or `makePipedHeredoc`.
- Import extra command line tools for building custom commands by using `DockerPipelineStage.installPacakges` to override 
`DockerPipelineStage.dockerfileSpec`. Check that the package name is consistent with what's available in the distro chosen.
The default distro is Debian.
- To save generated files from a stage so that we can read it after execution (e.g. a test case result file emitted by a
 testing framework), pipe it in the command to `/log`, so that it can be retrieved by manually walking the `graderLogPath`.
- Be default, only the gist of the commands without the outputting constructs (known as the "core" commands) will be logged,
use `logFullCmd()` to enable logging of expanded actual commands.


## Related components

`CmdLang`: a command line language abstraction
- so that the builder depends on this interface and dispatches common 
concrete language constructs to shells or languages.
- Examples could be `Bash` and `PowerShell`.
- Currently we are using `Bash`.

 
`CmdOutputHandler`: a command output strategy abstraction
- so that the builder depends on this abstract class and 
dispatches the actual IO behaviors needed to store, transfer, and read outputs from the pipeline execution environment.
- Examples are `Bash.EnvVarMapper` which stores selected command results (stdout, stderr, exitcode) to environment variables
for later retrieval, and `Bash.FileBased` which does the same but with text files; both of them operate under the `Bash` `CmdLang`.
- Currently we are using `Bash.FileBased`.


`CmdUtils`: a set of common utilities used by the above command-building components
- These are language-agnostic tools
including notably, an abstraction for a shell variable `Var`, and a representation of a result (stdout, stderr, exitcode)
of a command stored in a `Var` called `Var.Result`.
- `Var` dispatches concrete variable-related command constructs like how to print out the stored value in a variable using
`CmdLang` and `CmdOutputHandler`.
- As you have seen, `Var` works with `CmdBuilder` to provide a scripting interface for writing commands for pipeline stages.