Hansung Kim
5d5a6fbad2
sgemm_tcore: Template-ize kernel code
2024-07-29 20:11:51 -07:00
Hansung Kim
bca53a9c76
sgemm_tcore: Skip load at last k-iter; do DMA by default
2024-07-19 16:37:51 -07:00
Hansung Kim
85cace9524
sgemm_tcore: Fix smem allocation for non-dma
2024-06-15 01:28:27 -07:00
Hansung Kim
cfb6ae4a91
sgemm_tcore: Fix wrong double-buf addr for wmma_load
2024-06-15 00:51:35 -07:00
Hansung Kim
9d6ff196b3
sgemm_tcore: Use old opcodes to match frozen rtl
2024-06-15 00:26:57 -07:00
Hansung Kim
ce4f3a24e3
sgemm_tcore: Replace hardcoded NUM_LANES with NUM_THREADS
2024-06-12 21:01:37 -07:00
Hansung Kim
21452661f2
sgemm_tcore: Fix double-buffered addr for GEMMINI_DMA
2024-06-12 13:36:29 -07:00
Hansung Kim
32e31c51a4
sgemm_tcore: Blocksize 64; Fix kernel launch on larger dim
...
& fix addrgen assembly too large offset error
2024-06-11 22:27:12 -07:00
Hansung Kim
03d1df8f53
sgemm_tcore: Separate transpose control on AS read/write
...
Make separate control flags on transposed AS read/write to make it easy
to model bank-conflict-free GMEM _and_ SMEM access.
2024-06-11 21:16:23 -07:00
Hansung Kim
9febfb9bdc
sgemm_tcore: Move global_dmem_load back to kernel.cpp
2024-06-11 20:12:30 -07:00
Hansung Kim
ca7fd84a83
sgemm_tcore: Split util functions to a header file
2024-06-11 19:06:22 -07:00
Hansung Kim
dab9d7c6fc
sgemm_tcore: Fix kernel launch for smaller TBs than cluster threads
...
E.g. bm32bn32bk32wm16wn8
2024-06-11 14:09:31 -07:00
Hansung Kim
e3c4a4d2f5
sgemm_tcore: Improve agen for !transpose_as smem load
2024-06-10 22:08:37 -07:00
Hansung Kim
dc7bd6b248
sgemm_tcore: Fix warp_row/col calculation bug
2024-06-10 19:52:37 -07:00
Hansung Kim
3b2f5a31de
sgemm_tcore: Improve write_result addr gen
2024-06-10 19:34:00 -07:00
Hansung Kim
a22762db94
sgemm_tcore: Add GEMMINI_DMA to non-warp-specialized mode
...
~63% util for 128x128; ~83% for the k-loop.
FIXME: result is not correct currently. Need to fix the transpose
2024-06-10 19:32:39 -07:00
Hansung Kim
90662484b1
sgemm_tcore: inline left out
2024-06-07 20:34:29 -07:00
Hansung Kim
d61dd85872
sgemm_tcore: Remove unused SIMT core code
2024-06-07 20:32:02 -07:00
Hansung Kim
3a6427a491
sgemm_tcore: Hardcode threadblock id 0
...
this is fine since we're statically dispatching only one threadblock to
the whole cluster.
2024-06-07 16:10:02 -07:00
Hansung Kim
985c5fc0dc
sgemm_tcore: Remove uneffective register asm
2024-06-06 18:50:31 -07:00
Hansung Kim
7c4d850074
sgemm_tcore: Experiment with high K; 48% util
2024-06-06 18:42:14 -07:00
Hansung Kim
062403066e
sgemm_tcore: Bring M/N-loop inside the kernel
...
Instead of spawning multiple threadblocks which comes with stack access
overhead, have 1 threadblock work on the entire M/N-space thru a loop.
Grid size is fixed to the hardware parallelism.
TODO currently only works with 1 cluster in the system.
2024-06-06 15:22:01 -07:00
Hansung Kim
2c50b0cdce
sgemm_tcore: Remove BM_d/BN_d
2024-06-06 14:27:57 -07:00
Hansung Kim
7f6f096191
sgemm_tcore: Use if constexpr
2024-06-06 13:43:57 -07:00
Hansung Kim
deb6e5eba2
sgemm_tcore: Move bank-conflicts to SMEM stores from GMEM loads
2024-06-06 12:43:36 -07:00
Hansung Kim
ab4d525970
sgemm_tcore: More asserts on manual unrolling
2024-06-06 12:26:07 -07:00
Hansung Kim
a42fa6a113
sgemm_tcore: Swap out mul with bitwise ops for addr ping-pong
2024-06-05 19:01:59 -07:00
Hansung Kim
65c653afde
sgemm_tcore: Use arithmetic instead of branch for double-buffered addr
2024-06-05 18:03:08 -07:00
Hansung Kim
c7a6ed03de
sgemm_tcore: Use constant offset to reduce SMEM addr calc
2024-06-05 18:02:34 -07:00
Hansung Kim
95b5719847
sgemm_tcore: Split K-dim loop between consumer/producer
...
... so that you don't have to run (warpgroup_id == 0) condition at every
loop iteration which is expensive due to vx_split/join.
2024-06-05 17:15:30 -07:00
Hansung Kim
150f14af25
sgemm_tcore: Use multiple fp regs for GMEM->SMEM
2024-06-05 16:53:39 -07:00
Hansung Kim
e44173c65e
sgemm_tcore: Deconstruct addr calc for GMEM->SMEM
2024-06-05 15:11:01 -07:00
Hansung Kim
ff6e5bf6dc
sgemm_tcore: Deconstruct smem addr calc to reduce reg alloc
2024-06-05 14:50:36 -07:00
Hansung Kim
d8944db369
sgemm_tcore: Double-buffer over K-dimension
...
TODO: Not completely parameterized with DOUBLE_BUFFER yet.
2024-06-04 18:23:27 -07:00
Hansung Kim
18e3653d31
sgemm_tcore: Increase RF data reuse for WMITER/WNITER
...
... by splitting vx_wmma_load to vx_wmma_load_{a,b} and pulling it
out of the innermost loop.
TODO: there's some duplicate address compute being done in the both
functions.
2024-06-03 21:10:42 -07:00
Hansung Kim
4e723c4655
sgemm_tcore: Support two accumulation reg tiles
2024-06-01 01:12:38 -07:00
Hansung Kim
c8d6c56dd9
sgemm_tcore: Split global DMEM load into a function
2024-05-31 17:37:06 -07:00
Hansung Kim
220ee0aa5e
sgemm_tcore: Unroll around WMITER/WNITER
...
This is within a very tight loop so it's worth unrolling at the risk of
stack spills somewhere else.
2024-05-31 17:35:22 -07:00
Hansung Kim
1e48bad4f9
sgemm_tcore: Fix AS transpose
2024-05-26 13:51:47 -07:00
Hansung Kim
200fd3e08c
sgemm_tcore: Revert to packed smem alloc
2024-05-25 22:47:59 -07:00
Hansung Kim
b892c22f00
sgemm_tcore: Reflect WMITER/WNITER in threadblock size
2024-05-16 23:31:52 -07:00
Hansung Kim
18ecebddc0
sgemm_tcore: Fix round-down error with CORES_PER_CLUSTER
2024-05-16 21:36:24 -07:00
Hansung Kim
78b2a318c1
sgemm_tcore: Implement A transpose for coalesced smem access
2024-05-16 20:22:15 -07:00
Hansung Kim
8f64fae7a7
sgemm_tcore: Addr gen for local_k; add SIMT-only for reference
2024-05-16 14:11:09 -07:00
Hansung Kim
df1aa62916
sgemm_tcore: Add warptiling parameters
...
FIXME: accumulation is done wrong
2024-05-15 15:23:26 -07:00
Hansung Kim
09b23ffe87
sgemm_tg: 1-octet 8-lane kernel
2024-05-13 14:52:33 -07:00
Hansung Kim
d848e88f72
sgemm_tcore: Move C from regF->GMEM directly
2024-05-13 14:00:50 -07:00
Hansung Kim
9e60b1834c
sgemm_tcore: Rewrite with sgemm_Wg parametrization
2024-05-13 13:22:06 -07:00
Hansung Kim
5c298c81df
sgemm_tg: Use reg mapping functions
2024-05-12 22:22:54 -07:00
Hansung Kim
7775830814
Hardcode chipyard device addresses
2024-05-07 16:30:30 -07:00
