docs update

docs/cache_subsystem.md

@@ -0,0 +1,70 @@
+# Vortex Cache Subsystem
+
+The Vortex Cache Sub-system has the following main properties:
+
+- High-bandwidth with bank parallelism
+- Snoop protocol to flush data for CPU access
+- Generic design: Dcache, Icache, Shared Memory, L2 cache, L3 cache
+
+### Cache Hierarchy 
+
+![Image of Cache Hierarchy](./assets/img/cache_hierarchy.png)
+
+- Cache can be configured to be any level in the hierarchy
+- Caches communicate via snooping
+- Cache flush from AFU is passed down the hierarchy​
+
+### VX_cache.v (Top Module)
+
+VX.cache.v is the top module of the cache verilog code located in the `/hw/rtl/cache` directory.
+
+![Image of Vortex Cache](./assets/img/vortex_cache_top_module.png)
+
+- Configurable (Cache size, number of banks, bank line size, etc.)
+- I/O signals
+  - Core Request
+  - Core Rsp
+  - DRAM Req
+  - DRAM Rsp
+  - Snoop Rsp
+  - Snoop Rsp
+  - Snoop Forwarding Out
+  - Snoop Forwarding In
+- Bank Select
+  - Assigns valid and ready signals for each bank
+- Snoop Forwarder
+- DRAM Request Arbiter
+  - Prepares cache response for communication with DRAM
+- Snoop Response Arbiter
+  - Sends snoop response
+- Core Response Merge
+  - Cache accesses one line at a time. As a result, each request may not come back in the same response. This module tries to recombine the responses by thread ID. 
+
+### VX_bank.v
+
+VX_bank.v is the verilog code that handles cache bank functionality and is located in the `/hw/rtl/cache` directory.
+
+![Image of Vortex Cache Bank](./assets/img/vortex_bank.png)
+
+- Allows for high throughput​
+- Each bank contains queues to hold requests to the cache​
+- I/O signals
+  - Core request​
+  - Core Response​
+  - DRAM Fill Requests​
+  - DRAM Fill Response​
+  - DRAM WB Requests​
+  - Snp Request​
+  - Snp Response
+- Request Priority: DRAM fill, miss reserve, core request, snoop request​
+- Snoop Request Queue​
+- DRAM Fill Queue​
+- Core Req Arbiter​
+  - Requests to be processed by the bank
+- Tag Data Store​
+  - Registers for valid, dirty, dirtyb, tag, and data​
+  - Length of registers determined by lines in the bank​
+- Tag Data Access:​
+  - I/O: stall, snoop info, force request miss
+  - Writes to cache or sends read response; hit or miss determined here
+  - A missed request goes to the miss reserve if it is not a snoop request or DRAM fill 

docs/codebase.md

@@ -0,0 +1,40 @@
+# Vortex Codebase
+
+The directory/file layout of the Vortex codebase is as followed:
+
+- `hw`:     
+  - `rtl`: hardware rtl sources
+    - `cache`: cache subsystem code
+    - `fp_cores`: floating point unit code
+    - `interfaces`: interfaces for inter-module communication
+    - `libs`: general-purpose RTL modules
+  - `syn`: synthesis directory
+    - `opae`: OPAE synthesis scripts
+    - `quartus`: Quartus synthesis scripts    
+    - `synopsys`: Synopsys synthesis scripts
+    - `modelsim`: Modelsim synthesis scripts
+    - `yosys`: Yosys synthesis scripts
+  - `unit_tests`: unit tests for some hardware components
+- `driver`: host drivers repository
+  - `include`: Vortex driver public headers
+  - `stub`: Vortex stub driver library
+  - `fpga`: software driver that uses Intel OPAE FPGA
+  - `asesim`: software driver that uses Intel ASE simulator
+  - `vlsim`: software driver that uses vlsim simulator
+  - `rtlsim`: software driver that uses rtlsim simulator
+  - `simx`: software driver that uses simX simulator
+- `runtime`: kernel runtime software
+  - `include`: Vortex runtime public headers
+  - `linker`: linker file for compiling kernels
+  - `src`: runtime implementation
+- `sim`: 
+  - `vlsim`: AFU RTL simulator
+  - `rtlsim`: processor RTL simulator
+  - `simX`: cycle approximate simulator for vortex
+- `tests`: tests repository.
+  - `runtime`: runtime tests
+  - `regression`: regression tests
+  - `riscv`: RISC-V standard tests
+  - `opencl`: opencl benchmarks and tests
+- `ci`: continuous integration scripts
+- `miscs`: miscellaneous resources.

docs/debugging.md

@@ -0,0 +1,43 @@
+# Debugging Vortex Hardware
+
+## SimX Debugging
+
+SimX cycle-approximate simulator allows faster debugging of Vortex kernels' execution. 
+The recommended method to enable debugging is to pass the `--debug` flag to `blackbox` tool when running a program.
+
+    // Running demo program on SimX in debug mode
+    $ ./ci/blackbox.sh --driver=simx --app=demo --debug 
+
+A debug trace `run.log` is generated in the current directory during the program execution. The trace includes important states of the simulated processor (decoded instruction, register states, pipeline states, etc..). You can increase the verbosity level of the trace by changing the `DEBUG_LEVEL` variable to a value [1-5] (default is 3).
+
+    // Using SimX in debug mode with verbose level 4
+    $ CONFIGS=-DDEBUG_LEVEL=4 ./ci/blackbox.sh --driver=simx --app=demo --debug
+
+## RTL Debugging
+
+To debug the processor RTL, you need to use VLSIM or RTLSIM driver. VLSIM simulates the full processor including the AFU command processor (using `/rtl/afu/vortex_afu.sv` as top module). RTLSIM simulates the Vortex processor only (using `/rtl/Vortex.v` as top module).
+
+The recommended method to enable debugging is to pass the `--debug` flag to `blackbox` tool when running a program.
+
+    // Running demo program on vlsim in debug mode
+    $ ./ci/blackbox.sh --driver=vlsim --app=demo --debug
+
+    // Running demo program on rtlsim in debug mode
+    $ ./ci/blackbox.sh --driver=rtlsim --app=demo --debug
+
+A debug trace `run.log` is generated in the current directory during the program execution. The trace includes important states of the simulated processor (memory, caches, pipeline, stalls, etc..). A waveform trace `trace.vcd` is also generated in the current directory during the program execution. You can visualize the waveform trace using any tool that can open VCD files (Modelsim, Quartus, Vivado, etc..). [GTKwave] (http://gtkwave.sourceforge.net) is a great open-source scope analyzer that also works with VCD files.
+
+## FPGA Debugging
+
+Debugging the FPGA directly may be necessary to investigate runtime bugs that the RTL simulation cannot catch. We have implemented an in-house scope analyzer for Vortex that works when the FPGA is running. To enable the FPGA scope analyzer, the FPGA bitstream should be built using `SCOPE=1` flag
+
+    & cd /hw/syn/opae
+    $ CONFIGS=-DSCOPE=1 make fpga-4c
+
+When running the program on the FPGA, you need to pass the `--scope` flag to the `blackbox` tool.
+
+    // Running demo program on FPGA with scope enabled
+    $ ./ci/blackbox.sh --driver=fpga --app=demo --scope
+
+
+A waveform trace `trace.vcd` will be generated in the current directory during the program execution. This trace includes a limited set of signals that are defined in `/hw/scripts/scope.json`. You can expand your signals' selection by updating the json file.

docs/execute_opencl_on_vortex.md

@@ -0,0 +1,128 @@
+# Execute OpenCL on Vortex backend
+
+## Requirements
+- [Vortex](https://github.com/vortexgpgpu/vortex)
+- [POCL for Vortex](https://github.com/vortexgpgpu/pocl)
+- [riscv-toolchain](https://github.com/riscv-collab/riscv-gnu-toolchain)
+- [llvm-riscv](https://github.com/llvm-mirror/llvm)
+
+For installation, please see [Build Instructions](../README.md) for more details.
+
+**For Ubuntu18.04 users, you can directly download pre-build toolchains with [toolchain_install.sh](https://github.com/vortexgpgpu/vortex/blob/master/ci/toolchain_install.sh) script.**
+```bash
+# please modify the DESTDIR variable in the script before execution
+bash toolchain_install.sh -all
+```
+Assuming we have installed all dependencies in `/opt` path, we can get the following environment:
+```bash
+tree -L 2 /opt
+'''
+/opt/
+├── llvm-riscv
+│   ├── bin
+│   ├── include
+│   ├── lib
+│   ├── libexec
+│   └── share
+├── pocl
+│   ├── compiler
+│   └── runtime
+├── riscv-gnu-toolchain
+│   ├── bin
+│   ├── drops
+│   ├── include
+│   ├── lib
+│   ├── libexec
+│   ├── riscv32-unknown-elf
+│   ├── share
+│   └── var
+└── verilator
+    ├── bin
+    ├── examples
+    ├── include
+    ├── verilator-config.cmake
+    └── verilator-config-version.cmake
+'''
+```
+## Execute OpenCL on Vortex
+In this tutorial, we show the example of executing a vecadd programs on SIMX backend. 
+To execute a OpenCL program on Vortex, we have the following steps:
+- Compile the [OpenCL kernels](https://github.com/vortexgpgpu/vortex/blob/master/tests/opencl/vecadd/kernel.cl) into risc-v binary by POCL compiler.
+- Compile the [OpenCL host](https://github.com/vortexgpgpu/vortex/blob/master/tests/opencl/vecadd/main.cc) and link with Vortex driver(```-lvortex```).
+- Execute the compiled host programs on a backend.
+
+Thus, we can write a Makefile as following:
+```Makefile
+LLVM_PREFIX ?= /opt/llvm-riscv
+RISCV_TOOLCHAIN_PATH ?= /opt/riscv-gnu-toolchain
+SYSROOT ?= $(RISCV_TOOLCHAIN_PATH)/riscv32-unknown-elf
+POCL_CC_PATH ?= /opt/pocl/compiler
+POCL_RT_PATH ?= /opt/pocl/runtime
+
+OPTS ?= -n64
+
+# please edit these two variable to your environment
+VORTEX_DRV_PATH ?= $(realpath ../../../driver)
+VORTEX_RT_PATH ?= $(realpath ../../../runtime)
+
+K_LLCFLAGS += "-O3 -march=riscv32 -target-abi=ilp32f -mcpu=generic-rv32 -mattr=+m,+f -mattr=+vortex -float-abi=hard -code-model=small"
+K_CFLAGS   += "-v -O3 --sysroot=$(SYSROOT) --gcc-toolchain=$(RISCV_TOOLCHAIN_PATH) -march=rv32imf -mabi=ilp32f -Xclang -target-feature -Xclang +vortex -I$(VORTEX_RT_PATH)/include -fno-rtti -fno-exceptions -ffreestanding -nostartfiles -fdata-sections -ffunction-sections"
+K_LDFLAGS  += "-Wl,-Bstatic,-T$(VORTEX_RT_PATH)/linker/vx_link.ld -Wl,--gc-sections $(VORTEX_RT_PATH)/libvortexrt.a -lm"
+
+CXXFLAGS += -std=c++11 -O2 -Wall -Wextra -Wfatal-errors
+
+CXXFLAGS += -Wno-deprecated-declarations -Wno-unused-parameter
+
+CXXFLAGS += -I$(POCL_RT_PATH)/include
+
+LDFLAGS += -L$(POCL_RT_PATH)/lib -L$(VORTEX_DRV_PATH)/stub -lOpenCL -lvortex
+
+PROJECT = vecadd
+
+SRCS = main.cc
+
+all: $(PROJECT) kernel.pocl
+
+kernel.pocl: kernel.cl
+	LLVM_PREFIX=$(LLVM_PREFIX) POCL_DEBUG=all LD_LIBRARY_PATH=$(LLVM_PREFIX)/lib:$(POCL_CC_PATH)/lib $(POCL_CC_PATH)/bin/poclcc -LLCFLAGS $(K_LLCFLAGS) -CFLAGS $(K_CFLAGS) -LDFLAGS $(K_LDFLAGS) -o kernel.pocl kernel.cl
+ 
+$(PROJECT): $(SRCS)
+	$(CXX) $(CXXFLAGS) $^ $(LDFLAGS) -o $@
+
+run-fpga: $(PROJECT) kernel.pocl   
+	LD_LIBRARY_PATH=$(POCL_RT_PATH)/lib:$(VORTEX_DRV_PATH)/fpga:$(LD_LIBRARY_PATH) ./$(PROJECT) $(OPTS)
+
+run-asesim: $(PROJECT) kernel.pocl   
+	LD_LIBRARY_PATH=$(POCL_RT_PATH)/lib:$(VORTEX_DRV_PATH)/asesim:$(LD_LIBRARY_PATH) ./$(PROJECT) $(OPTS)
+	
+run-vlsim: $(PROJECT) kernel.pocl   
+	LD_LIBRARY_PATH=$(POCL_RT_PATH)/lib:$(VORTEX_DRV_PATH)/vlsim:$(LD_LIBRARY_PATH) ./$(PROJECT) $(OPTS)
+
+run-simx: $(PROJECT) kernel.pocl   
+	LD_LIBRARY_PATH=$(POCL_RT_PATH)/lib:$(VORTEX_DRV_PATH)/simx:$(LD_LIBRARY_PATH) ./$(PROJECT) $(OPTS)
+
+run-rtlsim: $(PROJECT) kernel.pocl   
+	LD_LIBRARY_PATH=$(POCL_RT_PATH)/lib:$(VORTEX_DRV_PATH)/rtlsim:$(LD_LIBRARY_PATH) ./$(PROJECT) $(OPTS)
+
+.depend: $(SRCS)
+	$(CXX) $(CXXFLAGS) -MM $^ > .depend;
+
+clean:
+	rm -rf $(PROJECT) *.o .depend 
+
+clean-all: clean
+	rm -rf *.pocl *.dump
+
+ifneq ($(MAKECMDGOALS),clean)
+    -include .depend
+endif
+```
+
+First, build the host program.
+```bash
+make all
+```
+If we want to execute on SIMX, we can execute the command below.
+```bash
+make run-simx
+```

docs/fpga_setup.md

@@ -0,0 +1,84 @@
+# FPGA Startup and Configuration Guide 
+
+OPAE Environment Setup
+----------------------
+
+    $ source /opt/inteldevstack/init_env_user.sh
+    $ export OPAE_HOME=/opt/opae/1.1.2
+    $ export PATH=$OPAE_HOME/bin:$PATH
+    $ export C_INCLUDE_PATH=$OPAE_HOME/include:$C_INCLUDE_PATH
+    $ export LIBRARY_PATH=$OPAE_HOME/lib:$LIBRARY_PATH
+    $ export LD_LIBRARY_PATH=$OPAE_HOME/lib:$LD_LIBRARY_PATH
+    $ export RISCV_TOOLCHAIN_PATH=/opt/riscv-gnu-toolchain
+    $ export PATH=:/opt/verilator/bin:$PATH
+    $ export VERILATOR_ROOT=/opt/verilator
+
+OPAE Build Configuration
+------------------------
+
+Within the `/hw/syn/opae` directory, there are source text files for each core-option for the fpga build (the 32 and 64 core options are not currently implemented) which have the following parameters that can be configured:
+- NUM_CORES: the number of cores per cluster
+- NUM_CLUSTERS: the number of clusters alotted to the processor
+- L2_ENABLE: enable the use of the L2 cache
+- PERF_ENABLE: enable the use of all profile counters
+
+To enable L3 cache and profile counters for a build, simply uncomment the definition within the respective source file.
+
+OPAE Build
+------------------
+
+The FPGA has to following configuration options:
+- 1 core fpga (fpga-1c)
+- 2 cores fpga (fpga-2c)
+- 4 cores fpga (fpga-4c)
+- 8 cores fpga (fpga-8c)
+- 16 cores fpga (fpga-16c)
+- 32 cores fpga (fpga-32c)
+- 64 cores fpga (fpga-64c)
+
+Command line:
+
+    $ cd hw/syn/opae
+    $ make fpga-<num-of-cores>c
+
+Example: `make fpga-4c`
+
+A new folder (ex: `build_fpga_4c`) will be created and the build will start and take ~30-480 min to complete.
+
+OPAE Build Progress
+-------------------
+
+You could check the last 10 lines in the build log for possible errors until build completion.
+
+    $ tail -n 10 ./build_fpga_<num-of-cores>c/build.log
+
+Check if the build is still running by looking for quartus_sh, quartus_syn, or quartus_fit programs.
+
+    $ ps -u <username>
+
+
+If the build fails and you need to restart it, clean up the build folder using the following command:
+
+    $ make clean-fpga-<num-of-cores>c
+
+Example: `make clean-fpga-4c`
+
+The file `vortex_afu.gbs` should exist when the build is done:
+
+    $ ls -lsa ./build_fpga_<num-of-cores>c/vortex_afu.gbs
+
+
+Signing the bitstream and Programming the FPGA
+----------------------------------------------
+
+    $ cd ./build_fpga_<num-of-cores>c
+    $ PACSign PR -t UPDATE -H openssl_manager -i vortex_afu.gbs -o vortex_afu_unsigned_ssl.gbs
+    $ fpgasupdate vortex_afu_unsigned_ssl.gbs
+
+FPGA sample test running OpenCL sgemm kernel
+--------------------------------------------
+
+Run the following from the Vortex root directory
+
+    $ ./ci/blackbox.sh --driver=fpga --app=sgemm --args="-n64"
+

docs/index.md

@@ -0,0 +1,30 @@
+# Vortex Documentation
+
+## Table of Contents
+
+- [Codebase Layout](codebase.md)
+- [Microarchitecture](microarchitecture.md)
+- [Cache Subsystem](cache_subsystem.md)
+- [Software](software.md)
+- [Simulation](simulation.md)
+- [FPGA Setup Guide](fpga_setup.md)
+- [Debugging](debugging.md)
+- [Useful Links](references.md)
+
+
+## Installation
+
+- Refer to the build instructions in [README](../README.md). 
+
+## Quick Start Scenarios
+
+Running Vortex simulators with different configurations:
+- Run basic driver test with rtlsim driver and Vortex config of 2 clusters, 2 cores, 2 warps, 4 threads
+
+    $ ./ci/blackbox.sh --driver=rtlsim --clusters=2 --cores=2 --warps=2 --threads=4  --app=basic
+- Run demo driver test with vlsim driver and Vortex config of 1 clusters, 4 cores, 4 warps, 2 threads
+
+    $ ./ci/blackbox.sh --driver=vlsim --clusters=1 --cores=4 --warps=4 --threads=2 --app=demo
+- Run dogfood driver test with simx driver and Vortex config of 4 cluster, 4 cores, 8 warps, 6 threads 
+
+    $ ./ci/blackbox.sh --driver=simx --clusters=4 --cores=4 --warps=8 --threads=6  --app=dogfood

docs/microarchitecture.md

@@ -0,0 +1,94 @@
+# Vortex Microarchitecture
+
+### Vortex GPGPU Execution Model
+
+Vortex uses the SIMT (Single Instruction, Multiple Threads) execution model with a single warp issued per cycle.
+
+- **Threads**
+  - Smallest unit of computation
+  - Each thread has its own register file (32 int + 32 fp registers)
+  - Threads execute in parallel
+- **Warps**
+  - A logical clster of threads
+  - Each thread in a warp execute the same instruction
+    - The PC is shared; maintain thread mask for Writeback
+  - Warp's execution is time-multiplexed at log steps
+    - Ex. warp 0 executes at cycle 0, warp 1 executes at cycle 1
+
+### Vortex RISC-V ISA Extension
+
+- **Thread Mask Control**
+  - Control the number of warps to activate during execution
+  - `TMC` *count*: activate count threads
+- **Warp Scheduling**
+  - Control the number of warps to activate during execution
+  - `WSPAWN` *count, addr*: activate count warps and jump to addr location
+- **Control-Flow Divergence**
+  - Control threads to activate when a branch diverges
+    - `SPLIT` *predicate*: apply 'taken' predicate thread mask adn save 'not-taken' into IPDOM stack
+    - `JOIN`: restore 'not-taken' thread mask
+- **Warp Synchronization**
+  - `BAR` *id, count*: stall warps entering barrier *id* until count is reached
+
+### Vortex Pipeline/Datapath
+
+![Image of Vortex Microarchitecture](./assets/img/vortex_microarchitecture_v2.png)
+
+Vortex has a 5-stage pipeline: FI | ID | Issue | EX | WB.
+
+- **Fetch**
+  - Warp Scheduler
+    - Track stalled & active warps, resolve branches and barriers, maintain split/join IPDOM stack
+  - Instruction Cache
+    - Retrieve instruction from cache, issue I-cache requests/responses
+- **Decode**
+  - Decode fetched instructions, notify warp scheduler when the following instructions are decoded:
+    - Branch, tmc, split/join, wspawn
+  - Precompute used_regs mask (needed for Issue stage)
+- **Issue**
+  - Scheduling
+    - In-order issue (operands/execute unit ready), out-of-order commit
+  - IBuffer
+    - Store fetched instructions, separate queues per-warp, selects next warp through round-robin scheduling
+  - Scoreboard
+    - Track in-use registers
+  - GPRs (General-Purpose Registers) stage
+    - Fetch issued instruction operands and send operands to execute unit
+- **Execute**
+  - ALU Unit
+    - Single-cycle operations (+,-,>>,<<,&,|,^), Branch instructions (Share ALU resources)
+  - MULDIV Unit
+    - Multiplier - done in 2 cycles
+    - Divider - division and remainder, done in 32 cycles
+      - Implements serial alogrithm (Stalls the pipeline)
+  - FPU Unit
+    - Multi-cycle operations, uses `FPnew` Library on ASIC, uses hard DSPs on FPGA
+  - CSR Unit
+    - Store constant status registers - device caps, FPU status flags, performance counters
+    - Handle external CSR requests (requests from host CPU)
+  - LSU Unit
+    - Handle load/store operations, issue D-cache requests, handle D-cache responses
+    - Commit load responses - saves storage, Scoreboard tracks completion
+  - GPGPU Unit
+    - Handle GPGPU instructions
+      - TMC, WSPAWN, SPLIT, BAR
+    - JOIN is handled by Warp Scheduler (upon SPLIT response)
+- **Commit**
+  - Commit
+    - Update CSR flags, update performance counters
+  - Writeback
+    - Write result back to GPRs, notify Scoreboard (release in-use register), select candidate instruction (ALU unit has highest priority)
+- **Clustering**
+  - Group mulitple cores into clusters (optionally share L2 cache)
+  - Group multiple clusters (optionally share L3 cache)
+  - Configurable at build time
+  - Default configuration:
+    - #Clusters = 1
+    - #Cores = 4
+    - #Warps = 4
+    - #Threads = 4
+- **FPGA AFU Interface**
+  - Manage CPU-GPU comunication
+    - Query devices caps, load kernel instructions and resource buffers, start kernel execution, read destination buffers
+  - Local Memory - GPU access to local DRAM
+  - Reserved I/O addresses - redirect to host CPU, console output

docs/references.md

@@ -0,0 +1 @@
+# Useful Links

docs/simulation.md

@@ -0,0 +1,50 @@
+# Vortex Simulation Methods 
+
+### RTL Simulation
+
+[Verilator](https://www.veripool.org/projects/verilator/wiki) is a Verilog/SystemVerilog design simulator that converts the Verilog HDL to single- or mult-ithreaded C++/SystemC code to perform the design simulation. An installation guide for Verilator is located [here.](https://www.veripool.org/projects/verilator/wiki/Installing)
+
+### Cycle-Approximate Simulation
+
+SimX is a C++ cycle-level in-house simulator developed for Vortex. The relevant files are located in the `simX` folder.
+
+### FGPA Simulation
+
+The current target FPGA for simulation is the Arria10 Intel Accelerator Card v1.0. The guide to build the fpga with specific configurations is located [here.](https://github.com/vortexgpgpu/vortex-dev/blob/master/doc/FPGA_Startup_Guide.md)
+
+### How to Test
+
+Running tests under specific drivers (rtlsim,simx,fpga) is done using the script named `blackbox.sh` located in the `ci` folder. Running command `./ci/blackbox.sh --help` from the Vortex root directory will display the following command line arguments for `blackbox.sh`:
+
+- *Clusters* - used to specify the number of clusters (collection of processing elements) within a configuration.
+- *Cores* - used to specify the number of cores (processing element containing multiple warps) within a configuration.
+- *Warps* - used to specify the number of warps (collection of concurrent hardware threads) within a configuration.
+- *Threads* - used to specify the number of threads (smallest unit of computation) within a configuration.
+- *L2cache* - used to enable the shard l2cache among the Vortex cores.
+- *L3cache* - used to enable the shared l3cache among the Vortex clusters.
+- *Driver* - used to specify which driver to run the Vortex simulation (either rtlsim, vlsim, fpga, or simx).
+- *Debug* - used to enable debug mode for the Vortex simulation.
+- *Perf* - used to enable the detailed performance counters within the Vortex simulation.
+- *App* - used to specify which test/benchmark to run in the Vortex simulation. The main choices are vecadd, sgemm, basic, demo, and dogfood. Other tests/benchmarks are located in the `/benchmarks/opencl` folder though not all of them work wit the current version of Vortex.
+- *Args* - used to pass additional arguments to the application.
+
+Example use of command line arguments: Run the sgemm benchmark using the vlsim driver with a Vortex configuration of 1 cluster, 4 cores, 4 warps, and 4 threads.
+
+    $ ./ci/blackbox.sh --clusters=1 --cores=4 --warps=4 --threads=4 --driver=vlsim --app=sgemm
+
+Output from terminal:
+```
+Create context
+Create program from kernel source
+Upload source buffers
+Execute the kernel
+Elapsed time: 2463 ms
+Download destination buffer
+Verify result
+PASSED!
+PERF: core0: instrs=90802, cycles=52776, IPC=1.720517
+PERF: core1: instrs=90693, cycles=53108, IPC=1.707709
+PERF: core2: instrs=90849, cycles=53107, IPC=1.710678
+PERF: core3: instrs=90836, cycles=50347, IPC=1.804199
+PERF: instrs=363180, cycles=53108, IPC=6.838518
+```

docs/software.md

@@ -0,0 +1 @@
+# Vortex OpenCL Support