wu-arch/kernels
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,684 Commits 6 Branches 0 Tags
ae
Commit Graph

2684 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Hansung Kim
b652e25945 flash: Warp-specialize between warp 0 and 1-7 
Finishes without stalls; No dependency check between O rescale and
GEMM-II.
 2024-09-09 16:42:30 -07:00
Hansung Kim
d31c8ffd7d flash: Fix grid size to hw cluster size 
Verified fast config, minus the barrier stall at the end.
 2024-09-09 15:44:03 -07:00
Hansung Kim
829af5d429 flash: Comment out mvout to smem 
Verified up to O_before_PV; still stalls without DEBUG
 2024-09-09 15:21:49 -07:00
Hansung Kim
ecc800964a flash: Change smem alloc for less bank conflicts; noskip stc 2024-09-09 13:47:18 -07:00
Hansung Kim
1f51f7f9d4 sgemm_impl: Mark threadblock_barrier convergent 
Thank you Chris Lattner
 2024-09-08 22:49:38 -07:00
Hansung Kim
714b9f501e flash: Restructure to do delayed fence for better concurrency 
Verified up to O_before_PV of 2nd iteration; O_after_PV needs preload
fix.
FIXME: Stalls at barrier without DEBUG set.
 2024-09-08 22:16:17 -07:00
Hansung Kim
6911843a82 flash: Remove unnecessary dmem preload, fix rowmax/rowsum dependency 2024-09-08 21:11:59 -07:00
Hansung Kim
a4dd45bc1b flash: Replace CISC with RISC 
spadQuartile in hw does not match spad addresses in kernel; match them
later for optimization.
 2024-09-08 20:52:28 -07:00
Hansung Kim
6547e92757 flash: Load Q to both quartiles; preload O for acc 2024-09-08 19:47:55 -07:00
Hansung Kim
8efa6868ea flash: Restructure for full software pipelining 
Verified up to P and O before PV; need to fix iteration for V load.
 2024-09-08 18:45:32 -07:00
Hansung Kim
cdb8377b62 flash: Do GEMM II in Gemmini; verify 1st iteration 2024-09-08 16:09:06 -07:00
Hansung Kim
3f50ac57ee flash: use 12bit dma interface 2024-09-08 15:29:56 -07:00
Hansung Kim
30a4d15cde sgemm_impl: Parameterize BM on NUM_CORES 2024-09-08 15:29:15 -07:00
Hansung Kim
443a37be6c sgemm_impl: Add DMA_FAST option; fix dbuf offset for dma 2024-09-08 14:58:41 -07:00
Hansung Kim
42913c00c4 sgemm_impl: Use 12-bit cmd interface, allow DIM=16 2024-09-08 14:28:27 -07:00
Hansung Kim
adcd0a9d49 sgemm_impl: Fix wrong smem address for fp16 
Verified results for fp16 256x256.
 2024-09-08 02:23:51 -07:00
Hansung Kim
c51dc4902d flash: Fix online softmax for DMA layout 2024-09-07 23:21:28 -07:00
Hansung Kim
2e1485877d flash: Add Gemmini-accelerated kernel 2024-09-07 22:40:58 -07:00
Hansung Kim
b3be271b88 flash: Split impl to header file 2024-09-07 21:16:35 -07:00
Hansung Kim
03308f8033 flash: Write fast config for DMA 
MAC utilization is 20-25% for the loop.
 2024-09-07 20:47:24 -07:00
Hansung Kim
8d32a03d09 flash: Write DMA code for warp-specialized 
TODO: result unverified
 2024-09-07 20:32:08 -07:00
Hansung Kim
33bc084c37 flash: Fix DMA layout for GEMM II 2024-09-07 19:51:05 -07:00
Hansung Kim
e02892ab7d flash: Fix DMA for up to GEMM II 
yeah
 2024-09-07 19:31:41 -07:00
Hansung Kim
4d6cdeb00b Fallback to 4 cores for flash 2024-09-07 17:40:49 -07:00
Hansung Kim
863e92a85e generate_matrix.py: Default to range, fp32 2024-09-07 17:40:21 -07:00
Hansung Kim
a967c262b1 sgemm_impl: Add new block-row-major layout for DMA 2024-09-07 16:38:22 -07:00
Hansung Kim
ed9bf6f73e common.mk: Switch to -Os to prevent branch code duplication 
Prevents erroneous stalls at vx_bar.  See comment in kernel.cpp
 2024-09-07 15:49:19 -07:00
Hansung Kim
d2f086344d flash: Fix DMA addr stride, stop at S=Q*K 2024-09-07 15:48:37 -07:00
Hansung Kim
9f067acdb9 sgemm_impl: Remove #if 0, FP_SIZE 16 2024-09-05 19:55:36 -07:00
Hansung Kim
a832fa7b84 sgemm_impl: 128x64 tile; fix unrolled asm, comment out actual gemm 2024-09-05 16:23:32 -07:00
Hansung Kim
137df9bee2 WIP: flash: Use Gemmini DMA 
Paused by the barrier bug in warp-divergent branches (tmask not being
considered)
 2024-09-05 16:23:32 -07:00
Hansung Kim
87a1c2bbfc Cores per cluster 4 to 8 2024-09-05 16:23:32 -07:00
Hansung Kim
bde6f0ea2e py: Write P_expected, don't rewrite vars 2024-09-05 16:23:32 -07:00
Hansung Kim
dcd69ea304 Increase SMEM size to 256KB 2024-09-05 16:23:32 -07:00
Hansung Kim
81924b601a sgemm_impl: Rewrite tile param constraint 2024-09-05 16:23:32 -07:00
Hansung Kim
bfb414c4eb flash: Add DMA config logic 2024-09-05 16:23:32 -07:00
Richard Yan
741bb80fe8 Merge branch 'kernels' of https://github.com/hansungk/vortex-private into kernels 2024-09-05 16:22:43 -07:00
Richard Yan
dd3244fba0 large fp16 kernel 2024-09-05 16:22:38 -07:00
Hansung Kim
ced98a6ff4 sgemm_impl: Refactor DMA layout remap logic into constexpr func 2024-09-03 16:20:31 -07:00
Hansung Kim
58fa2a3e91 sgemm_impl: Switch for allowing MN-major with DMA 2024-09-03 15:12:58 -07:00
Hansung Kim
f028a97f75 sgemm_tcore: Verify wo DMA; warn untested against K-major A + DMA 2024-09-03 14:42:19 -07:00
Hansung Kim
7aa0e6cbe4 sgemm_tcore: Fix correctness for GEMMINI_DMA 
Remap the logical SMEM row/col coordinates to the DMA's two-level
block-row-major layout.
 2024-09-02 23:46:50 -07:00
Hansung Kim
dd1b408f56 sgemm_tcore: Add debug mode with tile copy-out 2024-09-02 21:55:55 -07:00
Hansung Kim
9d71fa44a7 sgemm_tcore: Fix invocation with compile time threadblock size 2024-09-02 17:03:46 -07:00
Hansung Kim
70273fd00d flash: Cleanup debug code 2024-09-02 00:40:05 -07:00
Hansung Kim
8125192846 flash: Specify leading_dim for split QK GEMM; fix uninit'd RF before GEMM 2024-09-02 00:15:57 -07:00
Hansung Kim
bdd955836d sgemm_impl: Specify leading dimension to wmma load 
This is necessary for when loading a subtile from a full tile in SMEM
into RF, but that subtile is split by non-major dimension.
 2024-09-02 00:14:35 -07:00
Hansung Kim
602fe4a400 flash: Change timing for QKV move 
Verified with warp_specialized false; true remains to be fixed.
 2024-09-01 22:06:46 -07:00
Hansung Kim
aea257349a flash: Correct schedule with inter-warpgroup barriers 2024-09-01 20:40:51 -07:00
Hansung Kim
e5e65312d2 flash: Restructure to inter-warpgroup parallelism 
This is similar to
https://tridao.me/blog/2024/flash3/#inter-warpgroup-overlapping-with-pingpong-scheduling
 2024-09-01 19:58:33 -07:00