Configuration Customisation

Default Configuration

The default path for the cluster configuration, is under tests/configs/cluster. This contains a boilerplate layered configuration including cluster, system, operating system, software, and benchmark layers. Here, the layers are associated using partition. Additionally, the scheduler configuration is under tests/configs/scheduler, which contains the master configurations and job templates used for job script generation.

Customisation

To change the default cluster configuration path, use --path when reading the configuration, for example: autobench config read --path="path/to/cluster/config".

Cluster

To create a custom configuration for benchmarks, duplicate the cluster folder from tests/configs/cluster to your chosen directory, where you can make necessary adjustments.

When specifying nodes in cluster configuration, they can be explicitly listed or defined using a node range.

  - partition:
      name: ice
      nodes :
        - ice1
      node_range : false

# sample for node range i.e. sap3, sap4
  - partition:
      name: sap
      node_range : true
      start: 3
      end: 4

Within the systems layer, various hardware components can be integrated, as illustrated below:

    target_components: [icelake_cpu, a100_gpu]

In the operating system layer, you can see the knobs for setting up the frequency for the Ice Lake CPU and the relevant placeholders for scripts to enable or disable it after completing the benchmark.

  icelake_cpu :
    knobs_type: "cartesian"
    knobs :
      freqs : ["800MHz", "900MHZ"]
    script :
      set :
        - common:
          - echo "[freqs]"
      unset :
        - common:
          - echo "os cleanup"

In the software layer, you can define models (idle shown below) for use in benchmarks to facilitate reuse, such as a container with multiple benchmarks that can be used by several benchmarks. Here, you can specify knobs and scripts similar to the operating system layer.

icelake_cpu :
model : ["idle"]
param :
    idle:
    knobs_type: "cartesian"
    knobs:
        skip: [1]
    script:
        set:
        - common:
            - echo "Initiating idle test"
        unset:
        - common:
            - echo "software cleanup"

Lastly, in the benchmark layer, you can define different benchmarks/applications using the model defined in the software layer, such as an idle benchmark using the idle model as shown below. Here, you can specify knobs and scripts similar to the operating system layer.

icelake_cpu :
benchmark : ["idle"]
param :
    idle:
    model: ["idle"]
    knobs_type: "cartesian"
    knobs:
        # replace with actual knob
        skip : [1]
    script:
        set:
        - common:
            - echo "START:" $(($(date +%s%N)/1000000))
            - echo "EXECUTING"
            - sleep 1
            # -  your code
            - sleep 1
            - echo "END:" $(($(date +%s%N)/1000000))
            - echo "DONE"
        unset:
        - common:
            - echo "benchmark cleanup"

For two models intended for the same benchmark or application, if there are overlapping configuration components, you can utilize common to denote these shared part. Meanwhile, specify individual configurations associated with each model using their names under the custom section, as demonstrated below:

a100_gpu:
benchmark : ["babelstream"]
param :
    babelstream:
    model: ["babelstream_omp","babelstream_cuda"]
    knobs_type: "cartesian"
    knobs:
        s : [1024000000]
        n : [100]
    script:
        set:
        - common:
            - git clone https://github.com/UoB-HPC/BabelStream.git [benchmark]
            - rm -rf  [build] // replace with ids
        - custom:
            - babelstream_omp: cmake -B[build] -H./[benchmark] -DCMAKE_BUILD_TYPE=Release -DMODEL=[model_type] -DOFFLOAD=ON -DCMAKE_CXX_COMPILER=[compiler] -DOFFLOAD_FLAGS="-fopenmp -fopenmp-targets=nvptx64 -Xopenmp-target=nvptx64 -march=sm_80"
            - babelstream_cuda: cmake -B[build] -H./[benchmark] -DCMAKE_BUILD_TYPE=Release -DMODEL=[model_type] -DCMAKE_CUDA_COMPILER=[compiler] -DCXX_EXTRA_FLAGS=-fPIE -DCUDA_ARCH=sm_80
        - common:
            - cd [build]
            - make VERBOSE=1
            - cd ..
            - echo "START:" $(($(date +%s%N)/1000000))
            - echo "EXECUTING"
            - sleep 60
        - custom:
            - babelstream_omp: ./[build]/[model_type]-stream -s [s] -n [n]
            - babelstream_cuda: ./[build]/[model_type]-stream -s [s] -n [n]
        - common:
            - sleep 60
            - echo "END:" $(($(date +%s%N)/1000000))
            - echo "DONE"
        unset:
        - common:
            - echo "benchmark cleanup"
            - rm -rf [benchmark]
            - rm -rf [build]

In the configuration, knobs_type: "cartesian" currently denotes the method employed, specifically the cartesian product of the knobs in the setup. However, we plan to include additional options in the future.

Scheduler

The scheduler master configuration can be modified using tests/configs/scheduler/config.yaml. Templates, which are text files, can be modified accordingly. To add a new scheduler, create an entry in the tests/configs/scheduler/config.yaml file similar to the existing scheduler, and add the corresponding template. During the generation of the job script, AutoBench substitutes the placeholder variable names enclosed in square brackets.

Package configurations

The package can be configured using a .env file, which contains file names, paths, scheduler, repetition, etc., which can be configured accordingly. Note: The .env file should be modified before triggering autobench setup, which creates the file called current_config.json that stores the configuration along with other details for the correct benchmark experiment.