Hansung Kim
e0daf226ef
flash: Change kernel arg to contain qkv; strip stimulus gen from host code
...
test data is now generated by the python script instead of the host
binary.
2024-08-15 21:03:02 -07:00
Hansung Kim
a1858e0c80
sgemm_impl: Parameterize BK/TCK by FP_SIZE
2024-08-15 20:33:33 -07:00
Hansung Kim
fd2ff6208d
Generate golden data for flash in generate_matrix.py
2024-08-15 17:41:57 -07:00
Hansung Kim
ac44633b39
flash: Compile time flag for skipping GEMM
2024-08-15 17:40:32 -07:00
Hansung Kim
f844d96eea
flash: Initialize rowmax/rowsum cache in sharedmem
2024-08-15 17:28:36 -07:00
Hansung Kim
745aa098ed
flash: Optimize spad use, fix rowsum
2024-08-15 16:54:56 -07:00
Hansung Kim
e809d25305
flash: Fix rowsum and write fake exp
...
GEMM part is disabled for faster debugging, the kernel reads the result
of A*B directly from input binary.
2024-08-15 16:32:21 -07:00
Hansung Kim
53dfc690b9
flash: Allocate smem properly for rowsum and scratch
2024-08-14 21:50:20 -07:00
Hansung Kim
9cabe3413b
Fix overlapping smem in rowmax
2024-08-14 21:09:53 -07:00
Hansung Kim
692d028afd
Add flash attention kernel skeleton
2024-08-14 20:46:09 -07:00
Hansung Kim
014f7cd06f
sgemm_tcore: Unpack arg params, remove threadblock_dim_y
...
thread_block_gemm is meant to be reusable, so it shouldn't assume what
the kernel arg struct looks like.
threadblock_dim_y was ambiguous and didn't match the literal name either
(it was used as # of warps that participate in a barrier).
2024-08-14 20:34:49 -07:00
Hansung Kim
70919c39c9
Encode dependency to sgemm header in makefile
2024-08-14 20:03:07 -07:00
Hansung Kim
1b1264207b
sgemm_tcore: Add compile-time write_to_gmem param to thread_block_gemm
2024-08-14 17:48:31 -07:00
Hansung Kim
ee6339a35f
sgemm_tcore: Split all impl code into sgemm_impl.hpp
...
This is to make thread_block_gemm a re-usable library function for GEMM
operations for use in other kernels.
2024-08-14 16:24:48 -07:00
Hansung Kim
0534e5d1f6
sgemm_tcore: Fix addr gen for GMEM->SMEM for M-major A
...
This fixes correctness for TRANSPOSE_AT_PRODUCE/COLUMN=0/0, provided the
matrices are already stored in the correct layout in GMEM.
2024-08-14 15:35:35 -07:00
Hansung Kim
409424b032
sgemm_tcore: Fix fp16 addr gen in vx_wmma_load
2024-08-14 13:48:03 -07:00
Hansung Kim
e69fbea83a
sgemm_tcore: Fix casting error
2024-08-12 17:57:50 -07:00
Hansung Kim
95e3e96c6c
tensor: Change B in-memory layout to column-major
2024-08-12 15:22:07 -07:00
Hansung Kim
07dd9e35a0
tensor: Fix dimensions for fp16 in script
2024-08-12 15:22:07 -07:00
Hansung Kim
c1906ebb4f
tensor: Embed binary instead of hardcoding literals
...
the C compiler doesn't support fp16
2024-08-12 15:22:07 -07:00
Hansung Kim
1b5daccac9
tensor: Generate fp16-packed matrix in script
2024-08-12 15:22:07 -07:00
Richard Yan
4fddca3d1a
fp16 kernel
2024-08-06 02:43:44 -07:00
Richard Yan
ea4819702e
oopsie doopsie
2024-08-06 02:43:27 -07:00
Hansung Kim
a12f2c296c
tensor: Update readme
2024-07-31 11:55:28 -07:00
Hansung Kim
446b1a4c2e
tensor: Add readme
2024-07-31 11:53:31 -07:00
Hansung Kim
285776404f
tensor: Fix tensor unittest kernel
2024-07-31 11:49:41 -07:00
Hansung Kim
29f7290948
tensor: Fix correctness script
2024-07-31 11:39:50 -07:00
Hansung Kim
88cddc2b66
sgemm_tcore: Support data move for fp16-packed elements
...
Since core does not support memory accesses to non-word-aligned
addresses, pack fp16 elements in pairs into fp32 values, and do regular
tile movement with conditionally compressed column dimensions.
Perf seems to stay the same for fp32 256x256.
2024-07-30 21:43:10 -07:00
Hansung Kim
7f26548724
sgemm_tcore: Fix mem addr stride to 4
...
Otherwise incurs misaligned accesses not supported in lsu.
2024-07-30 14:06:46 -07:00
Hansung Kim
5d5a6fbad2
sgemm_tcore: Template-ize kernel code
2024-07-29 20:11:51 -07:00
Hansung Kim
5f342914bd
sgemm_tcore: Support fp16 input generation in host code
2024-07-29 17:18:35 -07:00
Hansung Kim
bca53a9c76
sgemm_tcore: Skip load at last k-iter; do DMA by default
2024-07-19 16:37:51 -07:00
Hansung Kim
1f844fa9e9
Set BM==BN==64, update doc
2024-07-19 16:37:15 -07:00
Hansung Kim
02feb36b12
idle: Use barriers instead to hang the core
2024-06-22 01:37:00 -07:00
Hansung Kim
11e6d34e1c
Add idle kernel
...
Only spawns 1 thread that does a busy wait up to a counter. Other cores
do not issue any instructions after the scheduling prologue.
2024-06-20 14:00:32 -07:00
Hansung Kim
63418a7496
sgemm_gemmini_dma: Skip mvout to scratchpad
...
Not necessary either for activation on gmem
2024-06-19 20:49:44 -07:00
Richard Yan
12a96d9c16
Merge branch 'kernels' of https://github.com/hansungk/vortex-private into kernels
2024-06-19 17:46:24 -07:00
Richard Yan
a1e165724f
skip move to spad
2024-06-19 17:45:58 -07:00
Richard Yan
c06cc40e59
make non dma gemmini use 64x64 tile size
2024-06-19 17:45:01 -07:00
Hansung Kim
bebdd3353e
Use SWISH in activate_block for tcore and gemmini
2024-06-19 15:41:50 -07:00
Hansung Kim
ae9e707280
sgemm_{gemmini_dma,tcore}: Separate activate_block
2024-06-19 14:50:22 -07:00
Hansung Kim
b586e0f881
sgemm_gemmini_dma: Update activation to match tcore
2024-06-18 15:30:12 -07:00
Hansung Kim
50b843d8c4
sgemm_tcore: Fix address overlap for DMA
...
Enforce square shapes of tiles in smem. TODO need to configure loop
bounds correctly.
2024-06-18 15:06:07 -07:00
Hansung Kim
36b02ad595
sgemm_tcore: Add warp-specialized kernel with activations
...
FIXME; only tested with WARP_SPECIALIZED == 0.
2024-06-17 19:14:33 -07:00
Hansung Kim
1a44063c5d
sgemm_gemmini_dma: Initial activation kernel with gemmini+DMA
...
Currently does spurrious fmul's in repetition.
2024-06-17 16:56:29 -07:00
Hansung Kim
85cace9524
sgemm_tcore: Fix smem allocation for non-dma
2024-06-15 01:28:27 -07:00
Hansung Kim
cfb6ae4a91
sgemm_tcore: Fix wrong double-buf addr for wmma_load
2024-06-15 00:51:35 -07:00
Hansung Kim
9d6ff196b3
sgemm_tcore: Use old opcodes to match frozen rtl
2024-06-15 00:26:57 -07:00
Hansung Kim
095ccfd79a
sgemm_gemmini_duo: Check in serialized kernel as separate file
2024-06-12 22:44:14 -07:00
Hansung Kim
1f26b4ef10
Remove checked in binary
2024-06-12 22:03:29 -07:00
