wu-arch/kernels
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,571 Commits 6 Branches 0 Tags
443a37be6ca93f22ddafade18349ccee8bdd617d
Commit Graph

65 Commits

Author SHA1 Message Date
Hansung Kim
443a37be6c sgemm_impl: Add DMA_FAST option; fix dbuf offset for dma 2024-09-08 14:58:41 -07:00
Hansung Kim
adcd0a9d49 sgemm_impl: Fix wrong smem address for fp16 
Verified results for fp16 256x256.
 2024-09-08 02:23:51 -07:00
Hansung Kim
7aa0e6cbe4 sgemm_tcore: Fix correctness for GEMMINI_DMA 
Remap the logical SMEM row/col coordinates to the DMA's two-level
block-row-major layout.
 2024-09-02 23:46:50 -07:00
Hansung Kim
dd1b408f56 sgemm_tcore: Add debug mode with tile copy-out 2024-09-02 21:55:55 -07:00
Hansung Kim
9d71fa44a7 sgemm_tcore: Fix invocation with compile time threadblock size 2024-09-02 17:03:46 -07:00
Hansung Kim
64b9717064 sgemm_tcore: Remove duplicate float_type decl 2024-08-16 16:26:18 -07:00
Hansung Kim
014f7cd06f sgemm_tcore: Unpack arg params, remove threadblock_dim_y 
thread_block_gemm is meant to be reusable, so it shouldn't assume what
the kernel arg struct looks like.

threadblock_dim_y was ambiguous and didn't match the literal name either
(it was used as # of warps that participate in a barrier).
 2024-08-14 20:34:49 -07:00
Hansung Kim
ee6339a35f sgemm_tcore: Split all impl code into sgemm_impl.hpp 
This is to make thread_block_gemm a re-usable library function for GEMM
operations for use in other kernels.
 2024-08-14 16:24:48 -07:00
Hansung Kim
0534e5d1f6 sgemm_tcore: Fix addr gen for GMEM->SMEM for M-major A 
This fixes correctness for TRANSPOSE_AT_PRODUCE/COLUMN=0/0, provided the
matrices are already stored in the correct layout in GMEM.
 2024-08-14 15:35:35 -07:00
Hansung Kim
409424b032 sgemm_tcore: Fix fp16 addr gen in vx_wmma_load 2024-08-14 13:48:03 -07:00
Hansung Kim
e69fbea83a sgemm_tcore: Fix casting error 2024-08-12 17:57:50 -07:00
Hansung Kim
95e3e96c6c tensor: Change B in-memory layout to column-major 2024-08-12 15:22:07 -07:00
Hansung Kim
88cddc2b66 sgemm_tcore: Support data move for fp16-packed elements 
Since core does not support memory accesses to non-word-aligned
addresses, pack fp16 elements in pairs into fp32 values, and do regular
tile movement with conditionally compressed column dimensions.
Perf seems to stay the same for fp32 256x256.
 2024-07-30 21:43:10 -07:00
Hansung Kim
7f26548724 sgemm_tcore: Fix mem addr stride to 4 
Otherwise incurs misaligned accesses not supported in lsu.
 2024-07-30 14:06:46 -07:00
Hansung Kim
5d5a6fbad2 sgemm_tcore: Template-ize kernel code 2024-07-29 20:11:51 -07:00
Hansung Kim
bca53a9c76 sgemm_tcore: Skip load at last k-iter; do DMA by default 2024-07-19 16:37:51 -07:00
Hansung Kim
85cace9524 sgemm_tcore: Fix smem allocation for non-dma 2024-06-15 01:28:27 -07:00
Hansung Kim
cfb6ae4a91 sgemm_tcore: Fix wrong double-buf addr for wmma_load 2024-06-15 00:51:35 -07:00
Hansung Kim
9d6ff196b3 sgemm_tcore: Use old opcodes to match frozen rtl 2024-06-15 00:26:57 -07:00
Hansung Kim
ce4f3a24e3 sgemm_tcore: Replace hardcoded NUM_LANES with NUM_THREADS 2024-06-12 21:01:37 -07:00
Hansung Kim
21452661f2 sgemm_tcore: Fix double-buffered addr for GEMMINI_DMA 2024-06-12 13:36:29 -07:00
Hansung Kim
32e31c51a4 sgemm_tcore: Blocksize 64; Fix kernel launch on larger dim 
& fix addrgen assembly too large offset error
 2024-06-11 22:27:12 -07:00
Hansung Kim
03d1df8f53 sgemm_tcore: Separate transpose control on AS read/write 
Make separate control flags on transposed AS read/write to make it easy
to model bank-conflict-free GMEM _and_ SMEM access.
 2024-06-11 21:16:23 -07:00
Hansung Kim
9febfb9bdc sgemm_tcore: Move global_dmem_load back to kernel.cpp 2024-06-11 20:12:30 -07:00
Hansung Kim
ca7fd84a83 sgemm_tcore: Split util functions to a header file 2024-06-11 19:06:22 -07:00
Hansung Kim
dab9d7c6fc sgemm_tcore: Fix kernel launch for smaller TBs than cluster threads 
E.g. bm32bn32bk32wm16wn8
 2024-06-11 14:09:31 -07:00
Hansung Kim
e3c4a4d2f5 sgemm_tcore: Improve agen for !transpose_as smem load 2024-06-10 22:08:37 -07:00
Hansung Kim
dc7bd6b248 sgemm_tcore: Fix warp_row/col calculation bug 2024-06-10 19:52:37 -07:00
Hansung Kim
3b2f5a31de sgemm_tcore: Improve write_result addr gen 2024-06-10 19:34:00 -07:00
Hansung Kim
a22762db94 sgemm_tcore: Add GEMMINI_DMA to non-warp-specialized mode 
~63% util for 128x128; ~83% for the k-loop.

FIXME: result is not correct currently. Need to fix the transpose
 2024-06-10 19:32:39 -07:00
Hansung Kim
90662484b1 sgemm_tcore: inline left out 2024-06-07 20:34:29 -07:00
Hansung Kim
d61dd85872 sgemm_tcore: Remove unused SIMT core code 2024-06-07 20:32:02 -07:00
Hansung Kim
3a6427a491 sgemm_tcore: Hardcode threadblock id 0 
this is fine since we're statically dispatching only one threadblock to
the whole cluster.
 2024-06-07 16:10:02 -07:00
Hansung Kim
985c5fc0dc sgemm_tcore: Remove uneffective register asm 2024-06-06 18:50:31 -07:00
Hansung Kim
7c4d850074 sgemm_tcore: Experiment with high K; 48% util 2024-06-06 18:42:14 -07:00
Hansung Kim
062403066e sgemm_tcore: Bring M/N-loop inside the kernel 
Instead of spawning multiple threadblocks which comes with stack access
overhead, have 1 threadblock work on the entire M/N-space thru a loop.
Grid size is fixed to the hardware parallelism.

TODO currently only works with 1 cluster in the system.
 2024-06-06 15:22:01 -07:00
Hansung Kim
2c50b0cdce sgemm_tcore: Remove BM_d/BN_d 2024-06-06 14:27:57 -07:00
Hansung Kim
7f6f096191 sgemm_tcore: Use if constexpr 2024-06-06 13:43:57 -07:00
Hansung Kim
deb6e5eba2 sgemm_tcore: Move bank-conflicts to SMEM stores from GMEM loads 2024-06-06 12:43:36 -07:00
Hansung Kim
ab4d525970 sgemm_tcore: More asserts on manual unrolling 2024-06-06 12:26:07 -07:00
Hansung Kim
a42fa6a113 sgemm_tcore: Swap out mul with bitwise ops for addr ping-pong 2024-06-05 19:01:59 -07:00
Hansung Kim
65c653afde sgemm_tcore: Use arithmetic instead of branch for double-buffered addr 2024-06-05 18:03:08 -07:00
Hansung Kim
c7a6ed03de sgemm_tcore: Use constant offset to reduce SMEM addr calc 2024-06-05 18:02:34 -07:00
Hansung Kim
95b5719847 sgemm_tcore: Split K-dim loop between consumer/producer 
... so that you don't have to run (warpgroup_id == 0) condition at every
loop iteration which is expensive due to vx_split/join.
 2024-06-05 17:15:30 -07:00
Hansung Kim
150f14af25 sgemm_tcore: Use multiple fp regs for GMEM->SMEM 2024-06-05 16:53:39 -07:00
Hansung Kim
e44173c65e sgemm_tcore: Deconstruct addr calc for GMEM->SMEM 2024-06-05 15:11:01 -07:00
Hansung Kim
ff6e5bf6dc sgemm_tcore: Deconstruct smem addr calc to reduce reg alloc 2024-06-05 14:50:36 -07:00
Hansung Kim
d8944db369 sgemm_tcore: Double-buffer over K-dimension 
TODO: Not completely parameterized with DOUBLE_BUFFER yet.
 2024-06-04 18:23:27 -07:00
Hansung Kim
18e3653d31 sgemm_tcore: Increase RF data reuse for WMITER/WNITER 
... by splitting vx_wmma_load to vx_wmma_load_{a,b} and pulling it
out of the innermost loop.

TODO: there's some duplicate address compute being done in the both
functions.
 2024-06-03 21:10:42 -07:00
Hansung Kim
4e723c4655 sgemm_tcore: Support two accumulation reg tiles 2024-06-01 01:12:38 -07:00