BOOM Bump w. Fromajo (#523)

* [uart] add uart adapter | add uart + adapter to all configs

* [uart] change pty define name | add uart to all configs that need it

* [uart] default to 115200 baudrate

* [dromajo] first working commit

* [dromajo] bump boom for commit-width > 1 fix

* [dromajo] adjust dromajo commits

* [dromajo] bump boom

* commit dromajo changes

* extra

* [dromajo] add block device to configs

* rebump older modules

* bump firesim

* [chipyard] enable dromajo in midas level simulation

* [testchipip] forgot to bump

* get rid of breaking things

* bump firesim

* bump boom

* Bump BOOM to ifu3 WIP

* bump firesim

* fix how memory is passed to dromajo

* bump boom and firesim

* fix merge issues

* add dromajo cosim bridge in chipyard

* move traceio back into testchipip (#488)

* refer to testchipip traceio in firechip (#490)

* Move TraceIO fragment to chipyard (#492)

* fix chipyard dromajo bridge (#493)

* Sboom dromajo bump (#501)

* [FireChip] Use clock in BridgeBinders

* [firesim] Update TraceGen BridgeBinder

* [Firechip] Add support for Tile <-> Uncore rational division

* [firesim] Update the multiclock test

* [firechip] Commit some Eagle X-related mock configs

* [firechip] Instantiate multiple TracerV bridges

* [Firechip] Include reset in tracerv tokens

* [TracerV] Drop the first token in comparison tests

* [Firechip] Make reverse instruction order in trace printf

* WARNING: Point at a fork of boom @ davidbiancolin

* [firesim] Update ClockBridge API

* Add Gemmini to README [ci skip] (#487)

* [firechip] Isolate all firesim-multiclock stuff in a single file

* add documentation on ring network and system bus

* Bump firesim for CI

* Bump FireSim

* Bump testchipip to dev

[ci skip]

* Bump FireSim

* [make] split up specific make vars/targets into frags (#499)

* [make] split up specific make vars/targets into frags

* [make] move dramsim and max-cycles into SIM_FLAGS

* [misc] move ariane configs to configs/ folder

* [dromajo] add dromajo

* [dromajo] bump for new traceio changes

* bump firesim

* bump firesim

* point to chipyard traceio

* bump boom

Co-authored-by: David Biancolin <david.biancolin@gmail.com>
Co-authored-by: Howard Mao <zhehao.mao@gmail.com>

* Support Dromajo + TracerV configurations

* [docs] add documentation for Dromajo in FireSim + Chipyard

* add a bit more docs

* [docs] bump docs

* [firesim] dump artefacts in firesim

* [firesim] update firesim

* [testchipip] remove extraneous items in testchipip

* [dromajo] prevent dromajo from breaking when params unset

* update firesim, dromajo, and testchipip

* [firesim] bump firesim

* [firesim] bump firesim

* [misc] bump firesim and testchipip for reviewer comments

* remove WithNoGPIO fragment

* bump firesim

* bump dromajo boom config

* bump firesim

* generate artefacts in firesim testsuite

Co-authored-by: abejgonzalez <abe.j.gonza@gmail.com>
Co-authored-by: Abraham Gonzalez <abe.gonzalez@berkeley.edu>
Co-authored-by: David Biancolin <david.biancolin@gmail.com>
Co-authored-by: Howard Mao <zhehao.mao@gmail.com>

.circleci/run-firesim-scala-tests.sh

@@ -22,6 +22,8 @@ cd $LOCAL_CHIPYARD_DIR/sims/firesim
 ./scripts/build-libdwarf.sh
 cd $LOCAL_CHIPYARD_DIR
 
+make -C $LOCAL_CHIPYARD_DIR/tools/dromajo/dromajo-src/src
+
 # set stricthostkeychecking to no (must happen before rsync)
 run "echo \"Ping $SERVER\""
 

.circleci/run-midasexamples-tests.sh

@@ -12,6 +12,10 @@ trap clean EXIT
 
 cd $LOCAL_CHIPYARD_DIR
 ./scripts/init-submodules-no-riscv-tools.sh
+
+# build libdromajo_cosim
+make -C $LOCAL_CHIPYARD_DIR/tools/dromajo/dromajo-src/src
+
 cd sims/firesim/sim/midas
 
 # set stricthostkeychecking to no (must happen before rsync)

.gitmodules

@@ -119,3 +119,6 @@
 [submodule "tools/DRAMSim2"]
 	path = tools/DRAMSim2
 	url = https://github.com/firesim/DRAMSim2.git
+[submodule "tools/dromajo/dromajo-src"]
+	path = tools/dromajo/dromajo-src
+	url = https://github.com/abejgonzalez/dromajo.git

build.sbt

@@ -147,7 +147,7 @@ lazy val hwacha = (project in file("generators/hwacha"))
   .dependsOn(rocketchip)
   .settings(commonSettings)
 
-lazy val boom = (project in file("generators/boom"))
+lazy val boom = conditionalDependsOn(project in file("generators/boom"))
   .dependsOn(rocketchip)
   .settings(commonSettings)
 

common.mk

@@ -14,6 +14,7 @@ SHELL=/bin/bash
 #########################################################################################
 include $(base_dir)/generators/ariane/ariane.mk
 include $(base_dir)/generators/tracegen/tracegen.mk
+include $(base_dir)/tools/dromajo/dromajo.mk
 
 #########################################################################################
 # Prerequisite lists

docs/Advanced-Concepts/Debugging-RTL.rst

@@ -86,9 +86,32 @@ identical program behavior. The torture utility can also be configured to run
 continuously for stress-testing. The torture utility exists within the ``utilities``
 directory.
 
+Dromajo Co-simulation for BOOM designs
+--------------------------------------
+Dromajo co-simulation is setup to work when two config. fragments are added to a BOOM config.
+First, a ``chipyard.config.WithTraceIO`` config. fragment must be added so that BOOM's traceport is enabled.
+Second, a ``chipyard.iobinders.WithSimDromajoBridge`` config. fragment must be added to
+connect the Dromajo co-simulator to the traceport.
+Once both config. fragments are added Dromajo should be enabled.
+
+To build/run Dromajo with a BOOM design, run your configuration the following make commands:
+
+.. code-block:: shell
+
+    # build the default Dromajo BOOM config. without waveform dumps
+    # replace "DromajoBoomConfig" with your particular config
+    make CONFIG=DromajoBoomConfig ENABLE_DROMAJO=1
+
+    # run a simulation with Dromajo
+    make CONFIG=DromajoBoomConfig ENABLE_DROMAJO=1 BINARY=<YOUR-BIN> run-binary
+
+.. warning:: Dromajo currently only works in single-core BOOM systems without accelerators.
+
+.. warning:: Dromajo currently only works in VCS simulation and FireSim.
+
 Firesim Debugging
 ---------------------------
-Chisel printfs, asserts, and waveform generation are also available in FireSim
+Chisel printfs, asserts, Dromajo co-simulation, and waveform generation are also available in FireSim
 FPGA-accelerated simulation. See the FireSim
 `documentation <https://docs.fires.im/en/latest/>`__ for more detail.
 

docs/Chipyard-Basics/Chipyard-Components.rst

@@ -79,6 +79,10 @@ Tools
 **Dsptools**
   A Chisel library for writing custom signal processing hardware, as well as integrating custom signal processing hardware into an SoC (especially a Rocket-based SoC).
 
+**Dromajo**
+  A RV64GC emulator primarily used for co-simulation and was originally developed by Esperanto Technology.
+  See :ref:`Dromajo` for more information.
+
 Toolchains
 -------------------------------------------
 

docs/Tools/Dromajo.rst

@@ -0,0 +1,22 @@
+Dromajo
+===============================
+
+`Dromajo <https://github.com/chipsalliance/dromajo/>`__ is a RV64GC functional simulator designed for co-simulation.
+To use it as a co-simulator, it requires you to pass the committed trace of instructions coming from the core into the tool.
+Within Chipyard, this is done by connecting to the `TracePort`` signals that are piped to the top level of the DUT.
+While the Rocket core does have a `TracePort`, it does not provide the committed write data that Dromajo requires.
+Thus, Dromajo uses the `ExtendedTracePort` only probided by BOOM (BOOM is the only core that supports Dromajo co-simulation).
+An example of a divergence and Dromajo's printout is shown below.
+
+.. code-block:: shell
+
+    [error] EMU PC ffffffe001055d84, DUT PC ffffffe001055d84
+    [error] EMU INSN 14102973, DUT INSN 14102973
+    [error] EMU WDATA 00000000000220d6, DUT WDATA 00000000000220d4
+    [error] EMU MSTATUS a000000a0, DUT MSTATUS 00000000
+    [error] DUT pending exception -1 pending interrupt -1
+
+Dromajo shows the divergence compared to simulation (PC, inst, inst-bits, write data, etc) and also provides the register state on failure.
+It is useful to catch bugs that affect architectural state before a simulation hangs or crashes.
+
+To use Dromajo with BOOM, refer to :ref:`Debugging RTL` section on Dromajo.

docs/Tools/index.rst

@@ -14,4 +14,4 @@ The following pages will introduce them, and how we can use them in order to gen
    Chisel-Testers
    Dsptools
    Barstools
-
+   Dromajo

generators/chipyard/src/main/scala/ConfigFragments.scala

@@ -13,7 +13,7 @@ import freechips.rocketchip.rocket.{RocketCoreParams, MulDivParams, DCacheParams
 import freechips.rocketchip.util.{AsyncResetReg}
 
 import boom.common.{BoomTilesKey}
-
+import ariane.{ArianeTilesKey}
 import testchipip._
 
 import hwacha.{Hwacha}
@@ -153,3 +153,9 @@ class WithControlCore extends Config((site, here, up) => {
     )
   case MaxHartIdBits => log2Up(up(RocketTilesKey, site).size + up(BoomTilesKey, site).size + 1)
 })
+
+class WithTraceIO extends Config((site, here, up) => {
+  case BoomTilesKey => up(BoomTilesKey) map (tile => tile.copy(trace = true))
+  case ArianeTilesKey => up(ArianeTilesKey) map (tile => tile.copy(trace = true))
+  case TracePortKey => Some(TracePortParams())
+})

generators/chipyard/src/main/scala/IOBinders.scala

@@ -351,4 +351,12 @@ class WithTraceGenSuccessBinder extends OverrideIOBinder({
   }
 })
 
+class WithSimDromajoBridge extends ComposeIOBinder({
+   (system: CanHaveTraceIOModuleImp) => {
+     system.traceIO match { case Some(t) => t.traces.map(tileTrace => SimDromajoBridge(tileTrace)(system.p)) }
+     Nil
+   }
+})
+
+
 } /* end package object */

generators/chipyard/src/main/scala/Subsystem.scala

@@ -104,4 +104,29 @@ class SubsystemModuleImp[+L <: Subsystem](_outer: L) extends BaseSubsystemModule
 
   // create file with boom params
   ElaborationArtefacts.add("""core.config""", outer.tiles.map(x => x.module.toString).mkString("\n"))
+
+  // Generate C header with relevant information for Dromajo
+  // THIS IS INCLUDED IN THE `dromajo_params.h` header file
+  var dromajoParams: String = ""
+  dromajoParams += "#ifndef DROMAJO_PARAMS_H"
+  dromajoParams += "\n#define DROMAJO_PARAMS_H"
+  dromajoParams += "\n\n" + "#define DROMAJO_RESET_VECTOR " + "\"" + "0x" + f"${p(BootROMParams).hang}%X" + "\""
+  dromajoParams += "\n" + "#define DROMAJO_MMIO_START " + "\"" + "0x" + f"${p(BootROMParams).address + p(BootROMParams).size}%X" + "\""
+  p(ExtMem) map { eP =>
+    dromajoParams += "\n" + "#define DROMAJO_MMIO_END " + "\"" + "0x" + f"${eP.master.base}%X" + "\""
+    // dromajo memory is in MiB chunks
+    dromajoParams += "\n" + "#define DROMAJO_MEM_SIZE " + "\"" + "0x" + f"${eP.master.size >> 20}%X" + "\""
+  }
+  p(PLICKey) map { pP =>
+    dromajoParams += "\n" + "#define DROMAJO_PLIC_BASE " + "\"" + "0x" + f"${pP.baseAddress}%X" + "\""
+    dromajoParams += "\n" + "#define DROMAJO_PLIC_SIZE " + "\"" + "0x" + f"${PLICConsts.size(pP.maxHarts)}%X" + "\""
+  }
+  p(CLINTKey) map { cP =>
+    dromajoParams += "\n" + "#define DROMAJO_CLINT_BASE " + "\"" + "0x" + f"${cP.baseAddress}%X" + "\""
+    dromajoParams += "\n" + "#define DROMAJO_CLINT_SIZE " + "\"" + "0x" + f"${CLINTConsts.size}%X" + "\""
+  }
+  dromajoParams += "\n\n#endif"
+
+  ElaborationArtefacts.add("""dromajo_params.h""", dromajoParams)
+
 }

generators/chipyard/src/main/scala/config/BoomConfigs.scala

@@ -163,3 +163,24 @@ class LoopbackNICLargeBoomConfig extends Config(
   new freechips.rocketchip.subsystem.WithCoherentBusTopology ++
   new freechips.rocketchip.system.BaseConfig)
 
+class DromajoBoomConfig extends Config(
+  new chipyard.iobinders.WithUARTAdapter ++
+  new chipyard.iobinders.WithTieOffInterrupts ++
+  new chipyard.iobinders.WithBlackBoxSimMem ++
+  new chipyard.iobinders.WithTiedOffDebug ++
+  new chipyard.iobinders.WithSimSerial ++
+  new chipyard.iobinders.WithSimDromajoBridge ++                 // attach Dromajo
+  new testchipip.WithTSI ++
+  new chipyard.config.WithTraceIO ++                             // enable the traceio
+  new chipyard.config.WithBootROM ++
+  new chipyard.config.WithUART ++
+  new chipyard.config.WithL2TLBs(1024) ++
+  new freechips.rocketchip.subsystem.WithNoMMIOPort ++
+  new freechips.rocketchip.subsystem.WithNoSlavePort ++
+  new freechips.rocketchip.subsystem.WithInclusiveCache ++
+  new freechips.rocketchip.subsystem.WithNExtTopInterrupts(0) ++
+  new boom.common.WithSmallBooms ++
+  new boom.common.WithNBoomCores(1) ++
+  new freechips.rocketchip.subsystem.WithCoherentBusTopology ++
+  new freechips.rocketchip.system.BaseConfig)
+

generators/firechip/src/main/scala/BridgeBinders.scala

@@ -13,7 +13,7 @@ import freechips.rocketchip.tile.{RocketTile}
 import sifive.blocks.devices.uart.HasPeripheryUARTModuleImp
 import sifive.blocks.devices.gpio.{HasPeripheryGPIOModuleImp}
 
-import testchipip.{CanHavePeripherySerialModuleImp, CanHavePeripheryBlockDeviceModuleImp, CanHaveTraceIOModuleImp}
+import testchipip.{CanHavePeripherySerialModuleImp, CanHavePeripheryBlockDeviceModuleImp}
 import icenet.CanHavePeripheryIceNICModuleImp
 
 import junctions.{NastiKey, NastiParameters}
@@ -27,7 +27,8 @@ import ariane.ArianeTile
 import boom.common.{BoomTile}
 
 import chipyard.iobinders.{IOBinders, OverrideIOBinder, ComposeIOBinder}
-import chipyard.HasChipyardTilesModuleImp
+import chipyard.{HasChipyardTilesModuleImp}
+import testchipip.{CanHaveTraceIOModuleImp}
 
 object MainMemoryConsts {
   val regionNamePrefix = "MainMemory"
@@ -72,12 +73,20 @@ class WithFASEDBridge extends OverrideIOBinder({
   }
 })
 
-class WithTracerVBridge extends OverrideIOBinder({
+class WithTracerVBridge extends ComposeIOBinder({
   (system: CanHaveTraceIOModuleImp) =>
     system.traceIO.foreach(_.traces.map(tileTrace => TracerVBridge(tileTrace)(system.p))); Nil
 })
 
 
+
+class WithDromajoBridge extends ComposeIOBinder({
+  (system: CanHaveTraceIOModuleImp) => {
+    system.traceIO.foreach(_.traces.map(tileTrace => DromajoBridge(tileTrace)(system.p))); Nil
+  }
+})
+
+
 class WithTraceGenBridge extends OverrideIOBinder({
   (system: HasTraceGenTilesModuleImp) =>
     GroundTestBridge(system.clock, system.success)(system.p); Nil

generators/firechip/src/main/scala/FireSim.scala

@@ -45,6 +45,7 @@ class FireSim(implicit val p: Parameters) extends RawModule {
       }))
       (lazyModule, Module(lazyModule.module))
     }
+
     val peekPokeBridge = PeekPokeBridge(clock, reset)
     // A Seq of partial functions that will instantiate the right bridge only
     // if that Mixin trait is present in the target's LazyModule class instance

generators/firechip/src/main/scala/FireSimMulticlockPOC.scala

@@ -79,6 +79,7 @@ class FiresimMulticlockTop(implicit p: Parameters) extends chipyard.DigitalTop
   override lazy val module = new FiresimMulticlockTopModule(this)
 }
 
+
 class FiresimMulticlockTopModule[+L <: DigitalTop](l: L) extends chipyard.DigitalTopModule(l) with HasFireSimClockingImp
 
 // Harness Definition

generators/firechip/src/main/scala/TargetConfigs.scala

@@ -73,13 +73,6 @@ class WithNIC extends icenet.WithIceNIC(inBufFlits = 8192, ctrlQueueDepth = 64)
 
 
 
-// Enables tracing on all cores
-class WithTraceIO extends Config((site, here, up) => {
-  case BoomTilesKey => up(BoomTilesKey) map (tile => tile.copy(trace = true))
-  case ArianeTilesKey => up(ArianeTilesKey) map (tile => tile.copy(trace = true))
-  case TracePortKey => Some(TracePortParams())
-})
-
 
 // Tweaks that are generally applied to all firesim configs
 class WithFireSimConfigTweaks extends Config(
@@ -92,7 +85,7 @@ class WithFireSimConfigTweaks extends Config(
   // Required*: Removes thousands of assertions that would be synthesized (* pending PriorityMux bugfix)
   new WithoutTLMonitors ++
   // Optional: Adds IO to attach tracerV bridges
-  new WithTraceIO ++
+  new chipyard.config.WithTraceIO ++
   // Optional: Request 16 GiB of target-DRAM by default (can safely request up to 32 GiB on F1)
   new freechips.rocketchip.subsystem.WithExtMemSize((1 << 30) * 16L) ++
   // Required: Adds IO to attach SerialBridge. The SerialBridges is responsible
@@ -143,7 +136,6 @@ class FireSimLargeBoomConfig extends Config(
   new WithFireSimConfigTweaks ++
   new chipyard.LargeBoomConfig)
 
-
 //********************************************************************
 // Heterogeneous config, base off chipyard's LargeBoomAndRocketConfig
 //********************************************************************

generators/firechip/src/test/scala/ScalaTestSuite.scala

@@ -130,6 +130,7 @@ abstract class FireSimTestSuite(
   mkdirs
   elaborate
   generateTestSuiteMakefrags
+  generateArtefacts
   runTest("verilator", "rv64ui-p-simple", false, Seq(s"""EXTRA_SIM_ARGS=+trace-humanreadable0"""))
   //diffTracelog("rv64ui-p-simple.out")
   runSuite("verilator")(benchmarks)

generators/tracegen/src/main/scala/Tile.scala

@@ -165,8 +165,15 @@ class BoomLSUShim(implicit p: Parameters) extends BoomModule()(p)
   io.lsu.rob_pnr_idx := rob_tail
   io.lsu.commit_load_at_rob_head := false.B
 
-  io.lsu.brinfo := DontCare
-  io.lsu.brinfo.valid := false.B
+  io.lsu.brupdate.b1 := (0.U).asTypeOf(new boom.exu.BrUpdateMasks)
+  io.lsu.brupdate.b2.uop := DontCare
+  io.lsu.brupdate.b2.mispredict := false.B
+  io.lsu.brupdate.b2.taken := false.B
+  io.lsu.brupdate.b2.cfi_type := 0.U
+  io.lsu.brupdate.b2.pc_sel := 0.U
+  io.lsu.brupdate.b2.jalr_target := 0.U
+  io.lsu.brupdate.b2.target_offset := 0.S(2.W)
+
   io.lsu.rob_head_idx := rob_head
 
 

scripts/build-toolchains.sh

@@ -130,6 +130,9 @@ SRCDIR="$(pwd)/toolchains" module_all libgloss --prefix="${RISCV}/riscv64-unknow
 
 SRCDIR="$(pwd)/toolchains" module_all qemu --prefix="${RISCV}" --target-list=riscv64-softmmu
 
+# make Dromajo
+make -C $CHIPYARD_DIR/tools/dromajo/dromajo-src/src
+
 cd "$RDIR"
 
 # create specific env.sh

tools/dromajo/dromajo.mk

@@ -0,0 +1,59 @@
+##############################################################
+# extra variables/targets ingested by the chipyard make system
+##############################################################
+
+DROMAJO_DIR = $(base_dir)/tools/dromajo/dromajo-src/src
+DROMAJO_LIB_NAME = dromajo_cosim
+DROMAJO_LIB = $(DROMAJO_DIR)/lib$(DROMAJO_LIB_NAME).a
+
+# Dromajo assumes using the default bootrom
+DROMAJO_ROM = $(base_dir)/bootrom/bootrom.rv64.img
+
+DTS_FILE = $(build_dir)/$(long_name).dts
+DROMAJO_DTB = $(build_dir)/$(long_name).dtb
+
+$(DTS_FILE): $(FIRRTL_FILE)
+
+$(DROMAJO_DTB): $(DTS_FILE)
+	dtc -I dts -O dtb -o $(DROMAJO_DTB) $(DTS_FILE)
+
+DROMAJO_SRCS = $(call lookup_srcs,$(DROMAJO_DIR),cc) $(call lookup_srcs,$(DROMAJO_DIR),h)
+
+$(DROMAJO_LIB): $(DROMAJO_SRCS)
+	$(MAKE) -C $(DROMAJO_DIR)
+
+# depending on where the simulation is done, use the auto-variable or the hardcoded defined one
+ifeq ($(BINARY),)
+DROMAJO_BIN = $(<)
+else
+DROMAJO_BIN = $(BINARY)
+endif
+
+DROMAJO_FLAGS = +drj_dtb=$(DROMAJO_DTB) +drj_rom=$(DROMAJO_ROM) +drj_bin=$(DROMAJO_BIN)
+
+DROMAJO_PARAMS_FILE    = $(build_dir)/$(long_name).dromajo_params.h
+DROMAJO_PARAMS_SYMLINK = $(build_dir)/dromajo_params.h
+
+$(DROMAJO_PARAMS_FILE): $(FIRRTL_FILE)
+
+$(DROMAJO_PARAMS_SYMLINK): $(DROMAJO_PARAMS_FILE)
+	rm -rf $(DROMAJO_PARAMS_SYMLINK)
+	ln -s $(DROMAJO_PARAMS_FILE) $(DROMAJO_PARAMS_SYMLINK)
+
+##################################################################
+# THE FOLLOWING MUST BE += operators
+##################################################################
+
+# simargs needed (i.e. like +drj_test=hello)
+ifdef ENABLE_DROMAJO
+EXTRA_SIM_FLAGS += $(DROMAJO_FLAGS)
+
+# CC flags needed for all simulations
+EXTRA_SIM_CC_FLAGS += -I$(DROMAJO_DIR)
+
+# sourced needed for simulation
+EXTRA_SIM_SOURCES += $(DROMAJO_LIB)
+
+# requirements needed for simulation
+EXTRA_SIM_REQS += $(DROMAJO_PARAMS_SYMLINK) $(DROMAJO_LIB) $(DROMAJO_DTB)
+endif