Merge pull request #849 from ucb-bar/vcu118-update

VCU118 FPGA Updates + FireMarshal on Prototypes

docs/Advanced-Concepts/Chip-Communication.rst

@@ -222,4 +222,4 @@ The following image shows this flow:
 .. image:: ../_static/images/chip-bringup.png
 
 In fact, this exact type of bringup setup is what the following section discusses:
-:ref:`Prototyping/VCU118:Introduction to the Bringup Platform`.
+:ref:`Prototyping/VCU118:Introduction to the Bringup Design`.

docs/Prototyping/VCU118.rst

@@ -47,8 +47,8 @@ After the harness is created, the ``BundleBridgeSource``'s must be connected to
 This is done with harness binders and io binders (see ``fpga/src/main/scala/vcu118/HarnessBinders.scala`` and ``fpga/src/main/scala/vcu118/IOBinders.scala``).
 For more information on harness binders and io binders, refer to :ref:`Customization/IOBinders:IOBinders and HarnessBinders`.
 
-Introduction to the Bringup Platform
-------------------------------------
+Introduction to the Bringup Design
+----------------------------------
 
 An example of a more complicated design used for Chipyard test chips can be viewed in ``fpga/src/main/scala/vcu118/bringup/``.
 This example extends the default test harness and creates new ``Overlays`` to connect to a DUT (connected to the FMC port).
@@ -58,3 +58,108 @@ The TSI Host Widget is used to interact with the DUT from the prototype over a S
 .. Note:: Remember that since whenever a new test harness is created (or the config changes, or the config packages changes, or...), you need to modify the make invocation.
     For example, ``make SUB_PROJECT=vcu118 CONFIG=MyNewVCU118Config CONFIG_PACKAGE=this.is.my.scala.package bitstream``.
     See :ref:`Prototyping/General:Generating a Bitstream` for information on the various make variables.
+
+Running Linux on VCU118 Designs
+-------------------------------
+
+As mentioned above, the default VCU118 harness is setup with a UART and a SPI SDCard.
+These are utilized to both interact with the DUT (with the UART) and load in Linux (with the SDCard).
+The following steps describe how to build and run buildroot Linux on the prototype platform.
+
+Building Linux with FireMarshal
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since the prototype currently does not have a block device setup for it, we build Linux with the rootfs built into the binary (otherwise known as "initramfs" or "nodisk" version of Linux).
+To make building this type of Linux binary easy, we will use the FireMarshal platform (see :ref:`fire-marshal` for more information).
+
+1. Setup FireMarshal (see :ref:`fire-marshal` on the initial setup).
+2. By default, FireMarshal is setup to work with FireSim.
+   Instead, we want to target the prototype platform.
+   This is done by switching the FireMarshal "board" from "firechip" to "prototype" using ``marshal-config.yaml``:
+
+.. code-block:: shell
+
+    # this assumes you do not have a `marshal-config.yaml` file already setup
+    echo "board-dir : 'boards/prototype'" > $PATH_TO_FIREMARSHAL/marshal-config.yaml
+
+.. Note:: Refer to the FireMarshal docs on more ways to set the board differently through environment variables and more.
+
+3. Next, build the workload (a.k.a buildroot Linux) in FireMarshal with the ``nodisk`` option flag.
+   For the rest of these steps, we will assume you are using the base ``br-base.json`` workload.
+   This workload has basic support for GPIO and SPI drivers (in addition to the default UART driver) but you can build off it in different workloads (refer to FireMarshal docs on workload inheritance).
+
+.. code-block:: shell
+
+    ./marshal -v -d build br-base.json # here the -d indicates --nodisk or initramfs
+
+.. Note:: Using the "board" FireMarshal functionality allows any child workload depending on the ``br-base.json`` workload specification to target a "prototype" platform rather than FireChip platform.
+   Thus, you can re-use existing workloads that depend on ``br-base.json`` on the prototype platform by just changing the "board"!
+
+4. The last step to generate the proper binary is to flatten it.
+   This is done by using FireMarshal's ``install`` feature which will produce a ``*-flat`` binary in the ``$PATH_TO_FIREMARSHAL/images`` directory (in our case ``br-base-bin-nodisk-flat``) from the previously built Linux binary (``br-base-bin-nodisk``).
+
+.. code-block:: shell
+
+   ./marshal -v -d install -t prototype br-base.json
+
+Setting up the SDCard
+~~~~~~~~~~~~~~~~~~~~~
+
+These instructions assume that you have a spare uSDCard that can be loaded with Linux and other files using two partitions.
+The 1st partition will be used to store the Linux binary (created with FireMarshal or other means) while the 2nd partition will store a file system that can be accessed from the DUT.
+Additionally, these instructions assume you are using Linux with ``sudo`` privileges and ``gdisk``, but you can follow a similar set of steps on Mac (using ``gpt`` or another similar program).
+
+1. Wipe the GPT on the card using ``gdisk``.
+   Use the `z` command to zap everything.
+   For rest of these instructions, we assume the SDCard path is ``/dev/sdc`` (replace this with the path to your SDCard).
+
+.. code-block:: shell
+
+    sudo gdisk /dev/sdc
+
+2. The VCU118 bootrom assumes that the Linux binary to load into memory will be located on sector 34 of the SDCard.
+   Change the default partition alignment to `1` so you can write to sector `34`.
+   Do this with the `l` command.
+
+3. Create the new GPT with `o`.
+   Click yes on all the prompts.
+
+4. Create a 512MiB partition to store the Linux binary (this can be smaller but it must be larger than the size of the Linux binary).
+   Use `n` and select sector 34, with size `+1048576` (corresponding to 512MiB).
+   For the type, search for the `apfs` type and use the hex number given.
+
+5. Create a second partition to store any other files with the rest of the SDCard.
+   Use `n` and use the defaults for starting sector and overall size (expand the 2nd partition to the rest of the SDCard space).
+   For the type, search for the `hfs` and use the hex number given.
+
+6. Write the changes using `w`.
+
+7. Setup the filesystem on the 2nd partition.
+   Note that the ``/dev/sdc2`` points to the 2nd partition.
+   Use the following command:
+
+.. code-block:: shell
+
+    sudo mkfs.hfs -v "PrototypeData" /dev/sdc2
+
+Transfer and Run Linux from the SDCard
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+After you have a Linux boot binary and the SDCard is setup properly (1st partition at sector 34), you can transfer the binary to the 1st SDCard partition.
+In this example, we generated a ``br-base-bin-nodisk-flat`` from FireMarshal and we will load it using ``dd``.
+Note that ``sdc1`` points to the 1st partition (remember to change the ``sdc`` to your own SDCard path).
+
+.. code-block:: shell
+
+    sudo dd if=$PATH_TO_FIREMARSHAL/br-base-bin-nodisk-flat of=/dev/sdc1
+
+If you want to add files to the 2nd partition, you can also do this now.
+
+After loading the SDCard with Linux and potentially other files, you can program the FPGA and plug in the SDCard.
+To interact with Linux via the UART console, you can connect to the serial port (in this case called ``ttyUSB1``) using something like ``screen``:
+
+.. code-block:: shell
+
+   screen -S FPGA_UART_CONSOLE /dev/ttyUSB1 115200
+
+Once connected, you should see the binary being loaded as well as Linux output (in some cases you might need to reset the DUT).

fpga/src/main/resources/vcu118/sdboot/sd.c

@@ -8,10 +8,12 @@
 #define DEBUG
 #include "kprintf.h"
 
-#define MAX_CORES 8
-
-// A sector is 512 bytes, so ((1 << 11) * 512) = 1 MiB
-#define PAYLOAD_SIZE	(16 << 11)
+// Total payload in B
+#define PAYLOAD_SIZE_B (30 << 20) // default: 30MiB
+// A sector is 512 bytes, so (1 << 11) * 512B = 1 MiB
+#define SECTOR_SIZE_B 512
+// Payload size in # of sectors
+#define PAYLOAD_SIZE (PAYLOAD_SIZE_B / SECTOR_SIZE_B)
 
 // The sector at which the BBL partition starts
 #define BBL_PARTITION_START_SECTOR 34
@@ -168,9 +170,11 @@ static int copy(void)
 	int rc = 0;
 
 	dputs("CMD18");
+
+	kprintf("LOADING 0x%xB PAYLOAD\r\n", PAYLOAD_SIZE_B);
 	kprintf("LOADING  ");
 
-    // John: Let's go slow until we get this working
+	// TODO: Speed up SPI freq. (breaks between these two values)
 	//REG32(spi, SPI_REG_SCKDIV) = (F_CLK / 16666666UL);
 	REG32(spi, SPI_REG_SCKDIV) = (F_CLK / 5000000UL);
 	if (sd_cmd(0x52, BBL_PARTITION_START_SECTOR, 0xE1) != 0x00) {
@@ -182,7 +186,7 @@ static int copy(void)
 		long n;
 
 		crc = 0;
-		n = 512;
+		n = SECTOR_SIZE_B;
 		while (sd_dummy() != 0xFE);
 		do {
 			uint8_t x = sd_dummy();

fpga/src/main/scala/vcu118/Configs.scala

@@ -17,7 +17,7 @@ import sifive.fpgashells.shell.xilinx.{VCU118ShellPMOD, VCU118DDRSize}
 
 import testchipip.{SerialTLKey}
 
-import chipyard.{BuildSystem, ExtTLMem}
+import chipyard.{BuildSystem, ExtTLMem, DefaultClockFrequencyKey}
 
 class WithDefaultPeripherals extends Config((site, here, up) => {
   case PeripheryUARTKey => List(UARTParams(address = BigInt(0x64000000L)))
@@ -26,11 +26,10 @@ class WithDefaultPeripherals extends Config((site, here, up) => {
 })
 
 class WithSystemModifications extends Config((site, here, up) => {
-  case PeripheryBusKey => up(PeripheryBusKey, site).copy(dtsFrequency = Some(site(FPGAFrequencyKey).toInt*1000000))
-  case DTSTimebase => BigInt(1000000)
+  case DTSTimebase => BigInt((1e6).toLong)
   case BootROMLocated(x) => up(BootROMLocated(x), site).map { p =>
     // invoke makefile for sdboot
-    val freqMHz = site(FPGAFrequencyKey).toInt * 1000000
+    val freqMHz = (site(DefaultClockFrequencyKey) * 1e6).toLong
     val make = s"make -C fpga/src/main/resources/vcu118/sdboot PBUS_CLK=${freqMHz} bin"
     require (make.! == 0, "Failed to build bootrom")
     p.copy(hang = 0x10000, contentFileName = s"./fpga/src/main/resources/vcu118/sdboot/build/sdboot.bin")
@@ -41,18 +40,23 @@ class WithSystemModifications extends Config((site, here, up) => {
 
 // DOC include start: AbstractVCU118 and Rocket
 class WithVCU118Tweaks extends Config(
+  // harness binders
   new WithUART ++
   new WithSPISDCard ++
   new WithDDRMem ++
+  // io binders
   new WithUARTIOPassthrough ++
   new WithSPIIOPassthrough ++
   new WithTLIOPassthrough ++
+  // other configuration
   new WithDefaultPeripherals ++
   new chipyard.config.WithTLBackingMemory ++ // use TL backing memory
   new WithSystemModifications ++ // setup busses, use sdboot bootrom, setup ext. mem. size
   new chipyard.config.WithNoDebug ++ // remove debug module
   new freechips.rocketchip.subsystem.WithoutTLMonitors ++
-  new freechips.rocketchip.subsystem.WithNMemoryChannels(1))
+  new freechips.rocketchip.subsystem.WithNMemoryChannels(1) ++
+  new WithFPGAFrequency(100) // default 100MHz freq
+)
 
 class RocketVCU118Config extends Config(
   new WithVCU118Tweaks ++
@@ -64,9 +68,10 @@ class BoomVCU118Config extends Config(
   new WithVCU118Tweaks ++
   new chipyard.MegaBoomConfig)
 
-class WithFPGAFrequency(MHz: Double) extends Config((site, here, up) => {
-  case FPGAFrequencyKey => MHz
-})
+class WithFPGAFrequency(fMHz: Double) extends Config(
+  new chipyard.config.WithPeripheryBusFrequency(fMHz) ++ // assumes using PBUS as default freq.
+  new chipyard.config.WithMemoryBusFrequency(fMHz)
+)
 
 class WithFPGAFreq25MHz extends WithFPGAFrequency(25)
 class WithFPGAFreq50MHz extends WithFPGAFrequency(50)

fpga/src/main/scala/vcu118/TestHarness.scala

@@ -17,12 +17,10 @@ import sifive.blocks.devices.uart._
 import sifive.blocks.devices.spi._
 import sifive.blocks.devices.gpio._
 
-import chipyard.{HasHarnessSignalReferences, HasTestHarnessFunctions, BuildTop, ChipTop, ExtTLMem, CanHaveMasterTLMemPort}
+import chipyard.{HasHarnessSignalReferences, HasTestHarnessFunctions, BuildTop, ChipTop, ExtTLMem, CanHaveMasterTLMemPort, DefaultClockFrequencyKey, HasReferenceClockFreq}
 import chipyard.iobinders.{HasIOBinders}
 import chipyard.harness.{ApplyHarnessBinders}
 
-case object FPGAFrequencyKey extends Field[Double](100.0)
-
 class VCU118FPGATestHarness(override implicit val p: Parameters) extends VCU118ShellBasicOverlays {
 
   def dp = designParameters
@@ -55,7 +53,8 @@ class VCU118FPGATestHarness(override implicit val p: Parameters) extends VCU118S
   harnessSysPLL := sysClkNode
 
   // create and connect to the dutClock
-  val dutClock = ClockSinkNode(freqMHz = dp(FPGAFrequencyKey))
+  println(s"VCU118 FPGA Base Clock Freq: ${dp(DefaultClockFrequencyKey)} MHz")
+  val dutClock = ClockSinkNode(freqMHz = dp(DefaultClockFrequencyKey))
   val dutWrangler = LazyModule(new ResetWrangler)
   val dutGroup = ClockGroup()
   dutClock := dutWrangler.node := dutGroup := harnessSysPLL
@@ -136,4 +135,11 @@ class VCU118FPGATestHarnessImp(_outer: VCU118FPGATestHarness) extends LazyRawMod
   _outer.topDesign match { case d: HasIOBinders =>
     ApplyHarnessBinders(this, d.lazySystem, d.portMap)
   }
+
+  // check the top-level reference clock is equal to the default
+  // non-exhaustive since you need all ChipTop clocks to equal the default
+  _outer.topDesign match {
+    case d: HasReferenceClockFreq => require(d.refClockFreqMHz == p(DefaultClockFrequencyKey))
+    case _ =>
+  }
 }