This is branch with the new GA CI

It has the .circleci CI stuff as well
for comparison purposes.

.circleci/README.md

@@ -0,0 +1,75 @@
+Chipyard CI
+===========
+
+Website: https://circleci.com/gh/ucb-bar/chipyard
+
+CircleCI Brief Explanation
+---------------------------
+
+CircleCI is controlled by the `config.yml` script.
+It consists of a *workflow* which has a series of *jobs* within it that do particular tasks.
+All jobs in the workflow must pass for the CI run to be successful.
+
+At the bottom of the `config.yml` there is a `workflows:` section that specifies the order in which the jobs of the workflow should run.
+For example:
+
+    - prepare-rocketchip:
+        requires:
+            - install-riscv-toolchain
+
+This specifies that the `prepare-rocketchip` job needs the `install-riscv-toolchain` steps to run before it can run.
+
+All jobs in the CI workflow are specified at the top of `config.yml`
+They specify a docker image to use (in this case a riscv-boom image since that is already available and works nicely) and an environment.
+Finally, in the `steps:` section, the steps are run sequentially and state persists throughout a job.
+So when you run something like `checkout` the next step has the checked out code.
+Caching in the job is done by giving a file to cache on.
+`restore_cache:` loads the cache into the environment if the key matches while `save_cache:` writes to the cache with the key IF IT IS NOT PRESENT.
+Note, if the cache is already present for that key, the write to it is ignored.
+Here the key is built from a string where the `checksum` portion converts the file given into a hash.
+
+.circleci directory
+-------------------
+
+This directory contains all the collateral for the Chipyard CI to work.
+The following is included:
+
+    `build-toolchains.sh`        # build either riscv-tools or esp-tools
+    `create-hash.sh`             # create hashes of riscv-tools/esp-tools so circleci caching can work
+    `do-rtl-build.sh`            # use verilator to build a sim executable (remotely)
+    `config.yml`                 # main circleci config script to enumerate jobs/workflows
+    `defaults.sh`                # default variables used
+    `check-commit.sh`            # check that submodule commits are valid
+    `build-extra-tests.sh`       # build default chipyard tests located in tests/
+    `clean-old-files.sh`         # clean up build server files
+    `do-fpga-rtl-build.sh`       # similar to `do-rtl-build` but using fpga/
+    `install-verilator.sh`       # install verilator on build server
+    `run-firesim-scala-tests.sh` # run firesim scala tests
+    `run-tests.sh                # run tests for a specific set of designs
+    `images/`                    # docker image used in CI
+
+How things are setup for Chipyard
+---------------------------------
+
+The steps for CI to run are as follows.
+1st, build the toolchains in parallel (note: `esp-tools` is currently not used in the run).
+The docker image sets up the `PATH` and `RISCV` variable so that `riscv-tools` is the default (currently the `env.sh` script that is created at tool build is unused).
+2nd, create the simulator binary.
+This requires the `riscv-tools` for `fesvr` and `verilator` to be able to build the binary.
+This stores all collateral for the tests (srcs, generated-srcs, sim binary, etc) to run "out of the gate" in the next job (make needs everything or else it will run again).
+3rd, finally run the desired tests.
+
+Other CI Setup
+--------------
+
+To get the CI to work correctly you need to setup CircleCI environment variables to point to the remote directory to build files and the server user/ip.
+In the project settings, you can find this under "Build Settings" "Environment Variables".
+You need to add two variables like the following:
+
+CI\_DIR = /path/to/where/you/want/to/store/remote/files
+SERVER = username@myserver.coolmachine.berkeley.edu
+
+Additionally, you need to add under the "PERMISSIONS" "SSH Permissions" section a private key that is on the build server that you are using.
+After adding a private key, it will show a fingerprint that should be added under the jobs that need to be run.
+
+Note: On the remote server you need to have the `*.pub` key file added to the `authorized_keys` file.

.circleci/build-extra-tests.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+# turn echo on and error on earliest command
+set -ex
+
+# get shared variables
+SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
+source $SCRIPT_DIR/defaults.sh
+
+make -C $LOCAL_CHIPYARD_DIR/tests clean
+make -C $LOCAL_CHIPYARD_DIR/tests

.circleci/build-toolchains.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+# create the riscv tools/esp tools binaries
+# passed in as <riscv-tools or esp-tools>
+
+# turn echo on and error on earliest command
+set -ex
+
+# get shared variables
+SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
+source $SCRIPT_DIR/defaults.sh
+
+if [ ! -d "$HOME/$1-install" ]; then
+    cd $HOME
+
+    # init all submodules including the tools
+    CHIPYARD_DIR="$LOCAL_CHIPYARD_DIR" NPROC=$CI_MAKE_NPROC $LOCAL_CHIPYARD_DIR/scripts/build-toolchains.sh $1
+fi

.circleci/check-commit.sh

@@ -0,0 +1,152 @@
+#!/bin/bash
+
+# check to see that submodule commits are present on the master branch
+
+# turn echo on and error on earliest command
+set -ex
+
+# get shared variables
+SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
+source $SCRIPT_DIR/defaults.sh
+
+# enter bhd repo
+cd $LOCAL_CHIPYARD_DIR
+
+# ignore the private vlsi submodules
+git config submodule.vlsi/hammer-cadence-plugins.update none
+git config submodule.vlsi/hammer-mentor-plugins.update none
+git config submodule.vlsi/hammer-synopsys-plugins.update none
+
+# initialize submodules and get the hashes
+git submodule update --init
+status=$(git submodule status)
+
+all_names=()
+
+
+search_submodule() {
+    echo "Running check on submodule $submodule in $dir"
+    hash=$(echo "$status" | grep "$dir.*$submodule " | awk '{print$1}' | grep -o "[[:alnum:]]*")
+    for branch in "${branches[@]}"
+    do
+        echo "Searching for $hash in origin/$branch of $submodule"
+        (git -C $dir/$submodule branch -r --contains "$hash" | grep "origin/$branch") && true # needs init'ed submodules
+        if [ $? -eq 0 ]
+        then
+            all_names+=("$dir/$submodule $hash 0")
+            return
+        fi
+    done
+    all_names+=("$dir/$submodule $hash 1")
+    return
+}
+
+search () {
+    for submodule in "${submodules[@]}"
+    do
+        search_submodule
+    done
+}
+
+submodules=("cva6" "boom" "gemmini" "hwacha" "icenet" "nvdla" "rocket-chip" "sha3" "sifive-blocks" "sifive-cache" "testchipip" "riscv-sodor")
+dir="generators"
+if [ "$CIRCLE_BRANCH" == "master" ] || [ "$CIRCLE_BRANCH" == "dev" ]
+then
+    branches=("master")
+else
+    branches=("master" "dev")
+fi
+search
+
+submodules=("riscv-gnu-toolchain" "riscv-isa-sim" "riscv-pk" "riscv-tests")
+dir="toolchains/esp-tools"
+branches=("master")
+search
+
+
+submodules=("riscv-gnu-toolchain" "riscv-isa-sim" "riscv-pk" "riscv-tests")
+dir="toolchains/riscv-tools"
+branches=("master")
+search
+
+# riscv-openocd doesn't use its master branch
+submodules=("riscv-openocd")
+dir="toolchains/riscv-tools"
+branches=("riscv")
+search
+
+submodules=("qemu" "libgloss")
+dir="toolchains"
+branches=("master")
+search
+
+submodules=("coremark" "firemarshal" "nvdla-workload" "spec2017")
+dir="software"
+if [ "$CIRCLE_BRANCH" == "master" ] || [ "$CIRCLE_BRANCH" == "dev" ]
+then
+    branches=("master")
+else
+    branches=("master" "dev")
+fi
+search
+
+submodules=("DRAMSim2" "axe" "barstools" "chisel-testers" "dsptools" "rocket-dsp-utils" "firrtl-interpreter" "torture" "treadle")
+dir="tools"
+if [ "$CIRCLE_BRANCH" == "master" ] || [ "$CIRCLE_BRANCH" == "dev" ]
+then
+    branches=("master")
+else
+    branches=("master" "dev")
+fi
+search
+
+submodules=("dromajo-src")
+dir="tools/dromajo"
+branches=("master")
+search
+
+submodules=("firesim")
+dir="sims"
+if [ "$CIRCLE_BRANCH" == "master" ] || [ "$CIRCLE_BRANCH" == "dev" ]
+then
+    branches=("master")
+else
+    branches=("master" "dev")
+fi
+search
+
+submodules=("hammer")
+dir="vlsi"
+branches=("master")
+search
+
+submodules=("fpga-shells")
+dir="fpga"
+branches=("master")
+search
+
+# turn off verbose printing to make this easier to read
+set +x
+
+# print 0's
+for str in "${all_names[@]}";
+do
+    if [ 0 = $(echo "$str" | awk '{print$3}') ]; then
+        echo "$str"
+    fi
+done
+
+echo ""
+
+# check if there was a non-zero return code and print 1's
+EXIT=0
+for str in "${all_names[@]}";
+do
+    if [ ! 0 = $(echo "$str" | awk '{print$3}') ]; then
+        echo "$str"
+        EXIT=1
+    fi
+done
+
+echo "Done checking all submodules"
+exit $EXIT

.circleci/clean-old-files.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+# clean directories that are older than 14 days
+# argument is used as the directory to look in
+
+age () {
+	local AGE_SEC
+	local CUR_SEC
+	local DIFF_SEC
+	local SEC_PER_DAY
+
+	SEC_PER_DAY=86400
+
+	CUR_SEC=$(date +%s)
+	AGE_SEC=$(stat -c %Y -- "$1")
+	DIFF_SEC=$(expr $CUR_SEC - $AGE_SEC)
+
+	echo $(expr $DIFF_SEC / $SEC_PER_DAY)
+}
+
+for d in $1/*/ ; do
+	DIR_AGE="$(age $d)"
+	if [ $DIR_AGE -ge 14 ]; then
+		echo "Deleting $d since is it $DIR_AGE old"
+		rm -rf $d
+	else
+	    echo "Keep $d since it is $DIR_AGE old"
+	fi
+done

.circleci/config.yml

@@ -0,0 +1,524 @@
+# CircleCI Configuration File
+
+# version of circleci
+version: 2.1
+
+parameters:
+    tools-cache-version:
+        type: string
+        default: "v13"
+
+# default execution env.s
+executors:
+    main-env:
+        docker:
+            - image: ucbbar/chipyard-ci-image:554b436
+        environment:
+            JVM_OPTS: -Xmx3200m # Customize the JVM maximum heap limit
+
+# re-usable commands
+commands:
+    toolchain-build:
+        description: "Build a toolchain"
+        parameters:
+            tools-version:
+                type: string
+                default: "riscv-tools"
+        steps:
+            - checkout
+            - run:
+                name: Create hash of toolchains
+                command: |
+                    .circleci/create-hash.sh
+            - restore_cache:
+                keys:
+                    - << parameters.tools-version >>-installed-<< pipeline.parameters.tools-cache-version >>-{{ checksum "../<< parameters.tools-version >>.hash" }}
+            - run:
+                name: Building << parameters.tools-version >>
+                command: |
+                    .circleci/build-toolchains.sh << parameters.tools-version >>
+                no_output_timeout: 120m
+            - save_cache:
+                key: << parameters.tools-version >>-installed-<< pipeline.parameters.tools-cache-version >>-{{ checksum "../<< parameters.tools-version >>.hash" }}
+                paths:
+                    - "/root/<< parameters.tools-version >>-install"
+
+    ssh-checkout:
+        description: "Add SSH key and checkout code"
+        steps:
+            - add_ssh_keys:
+                fingerprints:
+                    - "3e:c3:02:5b:ed:64:8c:b7:b0:04:43:bc:83:43:73:1e"
+                    - "32:d6:89:d2:97:fa:db:de:a8:2d:2a:f2:70:dd:80:89"
+            - checkout
+
+    setup-tools:
+        description: "Get toolchain"
+        parameters:
+            tools-version:
+                type: string
+                default: "riscv-tools"
+        steps:
+            - ssh-checkout
+            - run:
+                name: Create hash of toolchains
+                command: |
+                    .circleci/create-hash.sh
+            - restore_cache:
+                keys:
+                    - << parameters.tools-version >>-installed-<< pipeline.parameters.tools-cache-version >>-{{ checksum "../<< parameters.tools-version >>.hash" }}
+
+    prepare-rtl:
+        description: "Run the prepare step of RTL"
+        parameters:
+            tools-version:
+                type: string
+                default: "riscv-tools"
+            group-key:
+                type: string
+            timeout:
+                type: string
+                default: "120m"
+            build-script:
+                type: string
+                default: "do-rtl-build.sh"
+            build-type:
+                type: string
+                default: "sim"
+        steps:
+            - setup-tools:
+                tools-version: "<< parameters.tools-version >>"
+            - run:
+                name: Building << parameters.group-key >> subproject using Verilator
+                command: .circleci/<< parameters.build-script >> << parameters.group-key >> << parameters.build-type >>
+                no_output_timeout: << parameters.timeout >>
+            - save_cache:
+                key: << parameters.group-key >>-{{ .Branch }}-{{ .Revision }}
+                paths:
+                    - "/root/project"
+
+    run-tests:
+        description: "Run a set of tests"
+        parameters:
+            tools-version:
+                type: string
+                default: "riscv-tools"
+            group-key:
+                type: string
+            project-key:
+                type: string
+            run-script:
+                type: string
+                default: "run-tests.sh"
+            timeout:
+                type: string
+                default: "25m"
+        steps:
+            - setup-tools:
+                tools-version: "<< parameters.tools-version >>"
+            - restore_cache:
+                keys:
+                    - << parameters.group-key >>-{{ .Branch }}-{{ .Revision }}
+            - run:
+                name: Run << parameters.project-key >> subproject tests
+                command: .circleci/<< parameters.run-script >> << parameters.project-key >>
+                no_output_timeout: << parameters.timeout >>
+
+# set of jobs to run
+jobs:
+    commit-on-master-check:
+        executor: main-env
+        steps:
+            - checkout
+            - run:
+                name: Check commits of each submodule
+                command: |
+                    .circleci/check-commit.sh
+    tutorial-setup-check:
+        executor: main-env
+        steps:
+            - checkout
+            - run:
+                name: Check that the tutorial-setup patches apply
+                command: |
+                    scripts/tutorial-setup.sh
+    documentation-check:
+        executor: main-env
+        steps:
+            - checkout
+            - run:
+                name: Check that documentation builds with no warnings/errors
+                command: |
+                    sudo apt-get update -y
+                    sudo apt-get install -y python3-pip
+                    sudo pip3 install -r docs/requirements.txt
+                    make -C docs html
+
+    install-riscv-toolchain:
+        executor: main-env
+        steps:
+            - toolchain-build:
+                tools-version: "riscv-tools"
+    install-esp-toolchain:
+        executor: main-env
+        steps:
+            - toolchain-build:
+                tools-version: "esp-tools"
+    install-verilator:
+        executor: main-env
+        steps:
+            - ssh-checkout
+            - run:
+                name: Install Verilator to remote
+                command: |
+                    .circleci/install-verilator.sh
+    build-extra-tests:
+        executor: main-env
+        steps:
+            - ssh-checkout
+            - run:
+                name: Create hash of toolchains
+                command: |
+                    .circleci/create-hash.sh
+            - restore_cache:
+                keys:
+                    - riscv-tools-installed-<< pipeline.parameters.tools-cache-version >>-{{ checksum "../riscv-tools.hash" }}
+            - run:
+                name: Build extra tests
+                command: .circleci/build-extra-tests.sh
+                no_output_timeout: 120m
+            - save_cache:
+                key: extra-tests-{{ .Branch }}-{{ .Revision }}
+                paths:
+                    - "/root/project/tests"
+
+    prepare-chipyard-cores:
+        executor: main-env
+        steps:
+            - prepare-rtl:
+                group-key: "group-cores"
+    prepare-chipyard-peripherals:
+        executor: main-env
+        steps:
+            - prepare-rtl:
+                group-key: "group-peripherals"
+    prepare-chipyard-accels:
+        executor: main-env
+        steps:
+            - prepare-rtl:
+                tools-version: "esp-tools"
+                group-key: "group-accels"
+    prepare-chipyard-tracegen:
+        executor: main-env
+        steps:
+            - prepare-rtl:
+                group-key: "group-tracegen"
+    prepare-chipyard-other:
+        executor: main-env
+        steps:
+            - prepare-rtl:
+                group-key: "group-other"
+
+    chipyard-rocket-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-cores"
+                project-key: "chipyard-rocket"
+    chipyard-hetero-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-cores"
+                project-key: "chipyard-hetero"
+                timeout: "20m"
+    chipyard-boom-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-cores"
+                project-key: "chipyard-boom"
+    chipyard-cva6-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-cores"
+                project-key: "chipyard-cva6"
+                timeout: "30m"
+    chipyard-sodor-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-cores"
+                project-key: "chipyard-sodor"
+                timeout: "30m"
+    chipyard-multiclock-rocket-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-cores"
+                project-key: "chipyard-multiclock-rocket"
+    chipyard-dmirocket-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-peripherals"
+                project-key: "chipyard-dmirocket"
+    chipyard-spiflashwrite-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-peripherals"
+                project-key: "chipyard-spiflashwrite"
+    chipyard-spiflashread-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-peripherals"
+                project-key: "chipyard-spiflashread"
+    chipyard-lbwif-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-peripherals"
+                project-key: "chipyard-lbwif"
+
+    chipyard-sha3-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                tools-version: "esp-tools"
+                group-key: "group-accels"
+                project-key: "chipyard-sha3"
+    chipyard-streaming-fir-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-accels"
+                project-key: "chipyard-streaming-fir"
+    chipyard-streaming-passthrough-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-accels"
+                project-key: "chipyard-streaming-passthrough"
+    chipyard-hwacha-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                tools-version: "esp-tools"
+                group-key: "group-accels"
+                project-key: "chipyard-hwacha"
+                timeout: "30m"
+    chipyard-gemmini-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                tools-version: "esp-tools"
+                group-key: "group-accels"
+                project-key: "chipyard-gemmini"
+    chipyard-nvdla-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-accels"
+                project-key: "chipyard-nvdla"
+    tracegen-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-tracegen"
+                project-key: "tracegen"
+    tracegen-boom-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-tracegen"
+                project-key: "tracegen-boom"
+    icenet-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-other"
+                project-key: "icenet"
+                timeout: "30m"
+    testchipip-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "group-other"
+                project-key: "testchipip"
+                timeout: "30m"
+    firesim-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "extra-tests"
+                project-key: "firesim"
+                run-script: "run-firesim-scala-tests.sh"
+                timeout: "20m"
+    fireboom-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "extra-tests"
+                project-key: "fireboom"
+                run-script: "run-firesim-scala-tests.sh"
+                timeout: "45m"
+    firesim-multiclock-run-tests:
+        executor: main-env
+        steps:
+            - run-tests:
+                group-key: "extra-tests"
+                project-key: "firesim-multiclock"
+                run-script: "run-firesim-scala-tests.sh"
+                timeout: "20m"
+    prepare-chipyard-fpga:
+        executor: main-env
+        steps:
+            - prepare-rtl:
+                group-key: "group-fpga"
+                build-type: "fpga"
+
+# Order and dependencies of jobs to run
+workflows:
+    version: 2
+    submodules-on-master:
+        jobs:
+            # Check to make sure submodule commits are on master branches
+            - commit-on-master-check
+        triggers:
+            - schedule:
+                cron: "0 0 * * *"
+                filters:
+                    branches:
+                        only:
+                            - master
+
+    build-and-test-chipyard-integration:
+        jobs:
+            # Make the toolchains
+            - install-riscv-toolchain
+
+            - install-esp-toolchain
+
+            - install-verilator
+
+            - commit-on-master-check
+
+            # Attempt to apply the tutorial patches
+            - tutorial-setup-check
+
+            # Check that documentation builds
+            - documentation-check
+
+            # Build extra tests
+            - build-extra-tests:
+                requires:
+                    - install-riscv-toolchain
+
+            # Prepare the verilator builds
+            - prepare-chipyard-cores:
+                requires:
+                    - install-riscv-toolchain
+                    - install-verilator
+            - prepare-chipyard-peripherals:
+                requires:
+                    - install-riscv-toolchain
+                    - install-verilator
+            - prepare-chipyard-accels:
+                requires:
+                    - install-esp-toolchain
+                    - install-verilator
+            - prepare-chipyard-tracegen:
+                requires:
+                    - install-riscv-toolchain
+                    - install-verilator
+            - prepare-chipyard-other:
+                requires:
+                    - install-riscv-toolchain
+                    - install-verilator
+
+            # Run the example tests
+            - chipyard-rocket-run-tests:
+                requires:
+                    - prepare-chipyard-cores
+            - chipyard-hetero-run-tests:
+                requires:
+                    - prepare-chipyard-cores
+            - chipyard-boom-run-tests:
+                requires:
+                    - prepare-chipyard-cores
+            - chipyard-cva6-run-tests:
+                requires:
+                    - prepare-chipyard-cores
+            - chipyard-sodor-run-tests:
+                requires:
+                    - prepare-chipyard-cores
+            - chipyard-dmirocket-run-tests:
+                requires:
+                    - prepare-chipyard-peripherals
+            - chipyard-spiflashwrite-run-tests:
+                requires:
+                    - prepare-chipyard-peripherals
+            - chipyard-spiflashread-run-tests:
+                requires:
+                    - prepare-chipyard-peripherals
+            - chipyard-lbwif-run-tests:
+                requires:
+                    - prepare-chipyard-peripherals
+
+            - chipyard-sha3-run-tests:
+                requires:
+                    - prepare-chipyard-accels
+            - chipyard-streaming-fir-run-tests:
+                requires:
+                    - prepare-chipyard-accels
+            - chipyard-streaming-passthrough-run-tests:
+                requires:
+                    - prepare-chipyard-accels
+            - chipyard-hwacha-run-tests:
+                requires:
+                    - prepare-chipyard-accels
+            - chipyard-gemmini-run-tests:
+                requires:
+                    - prepare-chipyard-accels
+            - chipyard-nvdla-run-tests:
+                requires:
+                    - prepare-chipyard-accels
+
+            - tracegen-run-tests:
+                requires:
+                    - prepare-chipyard-tracegen
+            - tracegen-boom-run-tests:
+                requires:
+                    - prepare-chipyard-tracegen
+
+            - icenet-run-tests:
+                requires:
+                    - prepare-chipyard-other
+            - testchipip-run-tests:
+                requires:
+                    - prepare-chipyard-other
+
+            # Run the firesim tests
+            - firesim-run-tests:
+                requires:
+                    - install-riscv-toolchain
+                    - install-verilator
+                    - build-extra-tests
+            - firesim-multiclock-run-tests:
+                requires:
+                    - install-riscv-toolchain
+                    - install-verilator
+                    - build-extra-tests
+            - fireboom-run-tests:
+                requires:
+                    - install-riscv-toolchain
+                    - install-verilator
+                    - build-extra-tests
+
+            # Prepare the fpga builds (just Verilog)
+            - prepare-chipyard-fpga:
+                requires:
+                    - install-riscv-toolchain

.circleci/create-hash.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+
+# get the hash of riscv-tools
+
+# turn echo on and error on earliest command
+set -ex
+set -o pipefail
+
+# get shared variables
+SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
+source $SCRIPT_DIR/defaults.sh
+
+# enter bhd repo
+cd $LOCAL_CHIPYARD_DIR
+
+# Use normalized output of git-submodule status as hashfile
+for tools in 'riscv-tools' 'esp-tools' ; do
+    git submodule status "toolchains/${tools}" 'toolchains/libgloss' 'toolchains/qemu' |
+    while read -r line ; do
+        echo "${line#[!0-9a-f]}"
+    done > "${HOME}/${tools}.hash"
+done
+echo "Hashfile for riscv-tools and esp-tools created in $HOME"

.circleci/defaults.sh

@@ -0,0 +1,90 @@
+#!/bin/bash
+
+copy () {
+    rsync -avzp -e 'ssh' --exclude '.git' $1 $2
+}
+
+run () {
+    ssh -o "StrictHostKeyChecking no" -t $SERVER $@
+}
+
+run_script () {
+    ssh -o "StrictHostKeyChecking no" -t $SERVER 'bash -s' < $1 "$2"
+}
+
+clean () {
+    # remove remote work dir
+    run "rm -rf $REMOTE_WORK_DIR"
+}
+
+# make parallelism
+CI_MAKE_NPROC=8
+# chosen based on a 24c system shared with 1 other project
+REMOTE_MAKE_NPROC=4
+
+# verilator version
+VERILATOR_VERSION=v4.034
+
+# remote variables
+REMOTE_PREFIX=$CI_DIR/$CIRCLE_PROJECT_REPONAME-$CIRCLE_BRANCH
+REMOTE_WORK_DIR=$REMOTE_PREFIX-$CIRCLE_SHA1-$CIRCLE_JOB
+REMOTE_RISCV_DIR=$REMOTE_WORK_DIR/riscv-tools-install
+REMOTE_ESP_DIR=$REMOTE_WORK_DIR/esp-tools-install
+REMOTE_CHIPYARD_DIR=$REMOTE_WORK_DIR/chipyard
+REMOTE_SIM_DIR=$REMOTE_CHIPYARD_DIR/sims/verilator
+REMOTE_FIRESIM_DIR=$REMOTE_CHIPYARD_DIR/sims/firesim/sim
+REMOTE_FPGA_DIR=$REMOTE_CHIPYARD_DIR/fpga
+REMOTE_JAVA_OPTS="-Xmx10G -Xss8M"
+# Disable the supershell to greatly improve the readability of SBT output when captured by Circle CI
+REMOTE_SBT_OPTS="-Dsbt.ivy.home=$REMOTE_WORK_DIR/.ivy2 -Dsbt.supershell=false -Dsbt.global.base=$REMOTE_WORK_DIR/.sbt -Dsbt.boot.directory=$REMOTE_WORK_DIR/.sbt/boot"
+REMOTE_VERILATOR_DIR=$REMOTE_PREFIX-$CIRCLE_SHA1-verilator-install
+
+# local variables (aka within the docker container)
+LOCAL_CHECKOUT_DIR=$HOME/project
+LOCAL_RISCV_DIR=$HOME/riscv-tools-install
+LOCAL_ESP_DIR=$HOME/esp-tools-install
+LOCAL_CHIPYARD_DIR=$LOCAL_CHECKOUT_DIR
+LOCAL_SIM_DIR=$LOCAL_CHIPYARD_DIR/sims/verilator
+LOCAL_FIRESIM_DIR=$LOCAL_CHIPYARD_DIR/sims/firesim/sim
+
+# key value store to get the build groups
+declare -A grouping
+grouping["group-cores"]="chipyard-cva6 chipyard-rocket chipyard-hetero chipyard-boom chipyard-sodor chipyard-digitaltop chipyard-multiclock-rocket"
+grouping["group-peripherals"]="chipyard-dmirocket chipyard-blkdev chipyard-spiflashread chipyard-spiflashwrite chipyard-mmios chipyard-lbwif"
+grouping["group-accels"]="chipyard-nvdla chipyard-sha3 chipyard-hwacha chipyard-gemmini chipyard-streaming-fir chipyard-streaming-passthrough"
+grouping["group-tracegen"]="tracegen tracegen-boom"
+grouping["group-other"]="icenet testchipip"
+grouping["group-fpga"]="arty vcu118"
+
+# key value store to get the build strings
+declare -A mapping
+mapping["chipyard-rocket"]=""
+mapping["chipyard-dmirocket"]=" CONFIG=dmiRocketConfig"
+mapping["chipyard-lbwif"]=" CONFIG=LBWIFRocketConfig"
+mapping["chipyard-sha3"]=" CONFIG=Sha3RocketConfig"
+mapping["chipyard-digitaltop"]=" TOP=DigitalTop"
+mapping["chipyard-streaming-fir"]=" CONFIG=StreamingFIRRocketConfig"
+mapping["chipyard-streaming-passthrough"]=" CONFIG=StreamingPassthroughRocketConfig"
+mapping["chipyard-hetero"]=" CONFIG=LargeBoomAndRocketConfig"
+mapping["chipyard-boom"]=" CONFIG=SmallBoomConfig"
+mapping["chipyard-blkdev"]=" CONFIG=SimBlockDeviceRocketConfig"
+mapping["chipyard-hwacha"]=" CONFIG=HwachaRocketConfig"
+mapping["chipyard-gemmini"]=" CONFIG=GemminiRocketConfig"
+mapping["chipyard-cva6"]=" CONFIG=CVA6Config"
+mapping["chipyard-spiflashread"]=" CONFIG=LargeSPIFlashROMRocketConfig"
+mapping["chipyard-spiflashwrite"]=" CONFIG=SmallSPIFlashRocketConfig"
+mapping["chipyard-mmios"]=" CONFIG=MMIORocketConfig verilog"
+mapping["tracegen"]=" CONFIG=NonBlockingTraceGenL2Config"
+mapping["tracegen-boom"]=" CONFIG=BoomTraceGenConfig"
+mapping["chipyard-nvdla"]=" CONFIG=SmallNVDLARocketConfig"
+mapping["chipyard-sodor"]=" CONFIG=Sodor5StageConfig"
+mapping["chipyard-multiclock-rocket"]=" CONFIG=MulticlockRocketConfig"
+
+mapping["firesim"]="SCALA_TEST=firesim.firesim.RocketNICF1Tests"
+mapping["firesim-multiclock"]="SCALA_TEST=firesim.firesim.RocketMulticlockF1Tests"
+mapping["fireboom"]="SCALA_TEST=firesim.firesim.BoomF1Tests"
+mapping["icenet"]="SUB_PROJECT=icenet"
+mapping["testchipip"]="SUB_PROJECT=testchipip"
+
+mapping["arty"]="SUB_PROJECT=arty verilog"
+mapping["vcu118"]="SUB_PROJECT=vcu118 verilog"

.circleci/do-rtl-build.sh

@@ -0,0 +1,95 @@
+#!/bin/bash
+
+# create the different verilator builds
+# usage:
+#   do-rtl-build.sh <make command string> sim
+#     run rtl build for simulations and copy back results
+#   do-rtl-build.sh <make command string> fpga
+#     run rtl build for fpga and don't copy back results
+
+# turn echo on and error on earliest command
+set -ex
+
+# get shared variables
+SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
+source $SCRIPT_DIR/defaults.sh
+
+# call clean on exit
+trap clean EXIT
+
+cd $LOCAL_CHIPYARD_DIR
+./scripts/init-submodules-no-riscv-tools.sh
+./scripts/init-fpga.sh
+
+# replace the workspace dir with a local dir so you can copy around
+sed -i -E 's/(workspace=).*(\/tools)/\1$PWD\2/g' .sbtopts
+
+# set stricthostkeychecking to no (must happen before rsync)
+run "echo \"Ping $SERVER\""
+
+clean
+
+# copy over riscv/esp-tools, and chipyard to remote
+run "mkdir -p $REMOTE_CHIPYARD_DIR"
+copy $LOCAL_CHIPYARD_DIR/ $SERVER:$REMOTE_CHIPYARD_DIR
+
+run "cp -r ~/.ivy2 $REMOTE_WORK_DIR"
+run "cp -r ~/.sbt  $REMOTE_WORK_DIR"
+
+TOOLS_DIR=$REMOTE_RISCV_DIR
+LD_LIB_DIR=$REMOTE_RISCV_DIR/lib
+
+if [ $1 = "group-accels" ]; then
+    export RISCV=$LOCAL_ESP_DIR
+    export LD_LIBRARY_PATH=$LOCAL_ESP_DIR/lib
+    export PATH=$RISCV/bin:$PATH
+    GEMMINI_SOFTWARE_DIR=$LOCAL_SIM_DIR/../../generators/gemmini/software/gemmini-rocc-tests
+    cd $LOCAL_SIM_DIR/../../generators/gemmini/software
+    git submodule update --init --recursive gemmini-rocc-tests
+    cd gemmini-rocc-tests
+    ./build.sh
+
+    TOOLS_DIR=$REMOTE_ESP_DIR
+    LD_LIB_DIR=$REMOTE_ESP_DIR/lib
+    run "mkdir -p $REMOTE_ESP_DIR"
+    copy $LOCAL_ESP_DIR/ $SERVER:$REMOTE_ESP_DIR
+else
+    run "mkdir -p $REMOTE_RISCV_DIR"
+    copy $LOCAL_RISCV_DIR/ $SERVER:$REMOTE_RISCV_DIR
+fi
+
+# choose what make dir to use
+case $2 in
+    "sim")
+        REMOTE_MAKE_DIR=$REMOTE_SIM_DIR
+        ;;
+    "fpga")
+        REMOTE_MAKE_DIR=$REMOTE_FPGA_DIR
+        ;;
+esac
+
+# enter the verilator directory and build the specific config on remote server
+run "export RISCV=\"$TOOLS_DIR\"; \
+     make -C $REMOTE_MAKE_DIR clean;"
+
+read -a keys <<< ${grouping[$1]}
+
+# need to set the PATH to use the new verilator (with the new verilator root)
+for key in "${keys[@]}"
+do
+    run "export RISCV=\"$TOOLS_DIR\"; \
+         export LD_LIBRARY_PATH=\"$LD_LIB_DIR\"; \
+         export PATH=\"$REMOTE_VERILATOR_DIR/bin:\$PATH\"; \
+         export VERILATOR_ROOT=\"$REMOTE_VERILATOR_DIR\"; \
+         export COURSIER_CACHE=\"$REMOTE_WORK_DIR/.coursier-cache\"; \
+         make -j$REMOTE_MAKE_NPROC -C $REMOTE_MAKE_DIR FIRRTL_LOGLEVEL=info JAVA_OPTS=\"$REMOTE_JAVA_OPTS\" SBT_OPTS=\"$REMOTE_SBT_OPTS\" ${mapping[$key]}"
+done
+
+run "rm -rf $REMOTE_CHIPYARD_DIR/project"
+
+# choose to copy back results
+if [ $2 = "sim" ]; then
+    # copy back the final build
+    mkdir -p $LOCAL_CHIPYARD_DIR
+    copy $SERVER:$REMOTE_CHIPYARD_DIR/ $LOCAL_CHIPYARD_DIR
+fi

.circleci/install-verilator.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+# move verilator to the remote server
+
+# turn echo on and error on earliest command
+set -ex
+
+# get shared variables
+SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
+source $SCRIPT_DIR/defaults.sh
+
+# clean older directories (delete prior directories related to this branch also)
+run_script $LOCAL_CHIPYARD_DIR/.circleci/clean-old-files.sh $CI_DIR
+run "rm -rf $REMOTE_PREFIX*"
+
+# set stricthostkeychecking to no (must happen before rsync)
+run "echo \"Ping $SERVER\""
+
+run "git clone http://git.veripool.org/git/verilator $REMOTE_VERILATOR_DIR; \
+     cd $REMOTE_VERILATOR_DIR; \
+     git checkout $VERILATOR_VERSION; \
+     autoconf; \
+     export VERILATOR_ROOT=$REMOTE_VERILATOR_DIR; \
+     ./configure; \
+     make -j$REMOTE_MAKE_NPROC;"

.circleci/run-firesim-scala-tests.sh

@@ -0,0 +1,62 @@
+#!/bin/bash
+
+# create the different verilator builds
+# argument is the make command string
+
+# turn echo on and error on earliest command
+set -ex
+
+# get shared variables
+SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
+source $SCRIPT_DIR/defaults.sh
+
+# call clean on exit
+trap clean EXIT
+
+# Directory locations for handling firesim-local installations of libelf/libdwarf
+# This would generally be handled by build-setup.sh/firesim-setup.sh
+firesim_sysroot=lib-install
+local_firesim_sysroot=$LOCAL_FIRESIM_DIR/$firesim_sysroot
+remote_firesim_sysroot=$REMOTE_FIRESIM_DIR/$firesim_sysroot
+
+cd $LOCAL_CHIPYARD_DIR
+./scripts/init-submodules-no-riscv-tools.sh
+cd $LOCAL_CHIPYARD_DIR/sims/firesim/sim/firesim-lib/src/main/cc/lib
+git submodule update --init elfutils libdwarf
+cd $LOCAL_CHIPYARD_DIR/sims/firesim
+mkdir -p $local_firesim_sysroot
+./scripts/build-libelf.sh $local_firesim_sysroot
+./scripts/build-libdwarf.sh $local_firesim_sysroot
+cd $LOCAL_CHIPYARD_DIR
+
+# replace the workspace dir with a local dir so you can copy around
+sed -i -E 's/(workspace=).*(\/tools)/\1$PWD\2/g' .sbtopts
+
+make -C $LOCAL_CHIPYARD_DIR/tools/dromajo/dromajo-src/src
+
+# set stricthostkeychecking to no (must happen before rsync)
+run "echo \"Ping $SERVER\""
+
+clean
+
+# copy over riscv/esp-tools, and chipyard to remote
+run "mkdir -p $REMOTE_CHIPYARD_DIR"
+run "mkdir -p $REMOTE_RISCV_DIR"
+copy $LOCAL_CHIPYARD_DIR/ $SERVER:$REMOTE_CHIPYARD_DIR
+copy $LOCAL_RISCV_DIR/ $SERVER:$REMOTE_RISCV_DIR
+
+run "cp -r ~/.ivy2 $REMOTE_WORK_DIR"
+run "cp -r ~/.sbt  $REMOTE_WORK_DIR"
+
+TOOLS_DIR=$REMOTE_RISCV_DIR
+
+LD_LIB_DIR=$remote_firesim_sysroot/lib:$REMOTE_RISCV_DIR/lib
+
+# Run Firesim Scala Tests
+run "export RISCV=\"$TOOLS_DIR\"; \
+     export LD_LIBRARY_PATH=\"$LD_LIB_DIR\"; \
+     export FIRESIM_ENV_SOURCED=1; \
+     export PATH=\"$REMOTE_VERILATOR_DIR/bin:\$PATH\"; \
+     export VERILATOR_ROOT=\"$REMOTE_VERILATOR_DIR\"; \
+     export COURSIER_CACHE=\"$REMOTE_WORK_DIR/.coursier-cache\"; \
+     make -C $REMOTE_FIRESIM_DIR JAVA_OPTS=\"$REMOTE_JAVA_OPTS\" SBT_OPTS=\"$REMOTE_SBT_OPTS\" testOnly ${mapping[$1]}"

.circleci/run-tests.sh

@@ -0,0 +1,117 @@
+#!/bin/bash
+
+# run the different tests
+
+# turn echo on and error on earliest command
+set -ex
+
+# get remote exec variables
+SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
+source $SCRIPT_DIR/defaults.sh
+
+run_bmark () {
+    make run-bmark-tests-fast -j$CI_MAKE_NPROC -C $LOCAL_SIM_DIR $@
+}
+
+run_asm () {
+    make run-asm-tests-fast -j$CI_MAKE_NPROC -C $LOCAL_SIM_DIR $@
+}
+
+run_both () {
+    run_bmark $@
+    run_asm $@
+}
+
+run_tracegen () {
+    make tracegen -C $LOCAL_SIM_DIR $@
+}
+
+# TODO BUG: the run-binary command forces a rebuild of the simulator in CI
+#           instead, directly run the simulator binary
+case $1 in
+    chipyard-rocket)
+        run_bmark ${mapping[$1]}
+        ;;
+    chipyard-dmirocket)
+        run_bmark ${mapping[$1]}
+        ;;
+    chipyard-lbwif)
+        run_bmark ${mapping[$1]}
+        ;;
+    chipyard-boom)
+        run_bmark ${mapping[$1]}
+        ;;
+    chipyard-hetero)
+        run_bmark ${mapping[$1]}
+        ;;
+    rocketchip)
+        run_bmark ${mapping[$1]}
+        ;;
+    chipyard-hwacha)
+        export RISCV=$LOCAL_ESP_DIR
+        export LD_LIBRARY_PATH=$LOCAL_ESP_DIR/lib
+        export PATH=$RISCV/bin:$PATH
+        make run-rv64uv-p-asm-tests -j$CI_MAKE_NPROC -C $LOCAL_SIM_DIR ${mapping[$1]}
+        ;;
+    chipyard-gemmini)
+        export RISCV=$LOCAL_ESP_DIR
+        export LD_LIBRARY_PATH=$LOCAL_ESP_DIR/lib
+        export PATH=$RISCV/bin:$PATH
+        GEMMINI_SOFTWARE_DIR=$LOCAL_SIM_DIR/../../generators/gemmini/software/gemmini-rocc-tests
+        rm -rf $GEMMINI_SOFTWARE_DIR/riscv-tests
+        cd $LOCAL_SIM_DIR
+        $LOCAL_SIM_DIR/simulator-chipyard-GemminiRocketConfig $GEMMINI_SOFTWARE_DIR/build/bareMetalC/aligned-baremetal
+        $LOCAL_SIM_DIR/simulator-chipyard-GemminiRocketConfig $GEMMINI_SOFTWARE_DIR/build/bareMetalC/raw_hazard-baremetal
+        $LOCAL_SIM_DIR/simulator-chipyard-GemminiRocketConfig $GEMMINI_SOFTWARE_DIR/build/bareMetalC/mvin_mvout-baremetal
+        ;;
+    chipyard-sha3)
+        export RISCV=$LOCAL_ESP_DIR
+        export LD_LIBRARY_PATH=$LOCAL_ESP_DIR/lib
+        export PATH=$RISCV/bin:$PATH
+        (cd $LOCAL_CHIPYARD_DIR/generators/sha3/software && ./build.sh)
+        $LOCAL_SIM_DIR/simulator-chipyard-Sha3RocketConfig $LOCAL_CHIPYARD_DIR/generators/sha3/software/tests/bare/sha3-rocc.riscv
+        ;;
+    chipyard-streaming-passthrough)
+        make -C $LOCAL_CHIPYARD_DIR/tests
+        $LOCAL_SIM_DIR/simulator-chipyard-StreamingPassthroughRocketConfig $LOCAL_CHIPYARD_DIR/tests/streaming-passthrough.riscv
+        ;;
+    chipyard-streaming-fir)
+        make -C $LOCAL_CHIPYARD_DIR/tests
+        $LOCAL_SIM_DIR/simulator-chipyard-StreamingFIRRocketConfig $LOCAL_CHIPYARD_DIR/tests/streaming-fir.riscv
+        ;;
+    chipyard-spiflashread)
+        make -C $LOCAL_CHIPYARD_DIR/tests
+        make -C $LOCAL_SIM_DIR ${mapping[$1]} BINARY=$LOCAL_CHIPYARD_DIR/tests/spiflashread.riscv SIM_FLAGS="+spiflash0=${LOCAL_CHIPYARD_DIR}/tests/spiflash.img" run-binary
+        ;;
+    chipyard-spiflashwrite)
+        make -C $LOCAL_CHIPYARD_DIR/tests
+        make -C $LOCAL_SIM_DIR ${mapping[$1]} BINARY=$LOCAL_CHIPYARD_DIR/tests/spiflashwrite.riscv SIM_FLAGS="+spiflash0=${LOCAL_CHIPYARD_DIR}/tests/spiflash.img" run-binary
+        [[ "`xxd $LOCAL_CHIPYARD_DIR/tests/spiflash.img  | grep 1337\ 00ff\ aa55\ face | wc -l`" == "6" ]] || false
+        ;;
+    tracegen)
+        run_tracegen ${mapping[$1]}
+        ;;
+    tracegen-boom)
+        run_tracegen ${mapping[$1]}
+        ;;
+    chipyard-cva6)
+        make run-binary-fast -C $LOCAL_SIM_DIR ${mapping[$1]} BINARY=$RISCV/riscv64-unknown-elf/share/riscv-tests/benchmarks/multiply.riscv
+        ;;
+    chipyard-sodor)
+        run_asm ${mapping[$1]}
+        ;;
+    chipyard-nvdla)
+        make -C $LOCAL_CHIPYARD_DIR/tests
+        make -C $LOCAL_SIM_DIR ${mapping[$1]} BINARY=$LOCAL_CHIPYARD_DIR/tests/nvdla.riscv run-binary
+        ;;
+    icenet)
+        make run-binary-fast BINARY=none -C $LOCAL_SIM_DIR ${mapping[$1]}
+        ;;
+    testchipip)
+        make run-binary-fast BINARY=none -C $LOCAL_SIM_DIR ${mapping[$1]}
+        ;;
+    *)
+        echo "No set of tests for $1. Did you spell it right?"
+        exit 1
+        ;;
+esac

.github/workflows/chipyard-run-tests.yml

@@ -3,7 +3,7 @@ name: chipyard-ci-process
 on: [push]
 
 env:
-  tools-cache-version: v7
+  tools-cache-version: v8
   BUILDSERVER: ${{ secrets.BUILDSERVER }}
   BUILDUSER: ${{ secrets.BUILDUSER }}
   SERVER: ${{ secrets.BUILDUSER }}@${{ secrets.BUILDSERVER }}