Fitzy
d184c6c622
Fix offload buffer stream lifetime
2026-08-08 19:59:23 -04:00
Jaret Burkett
f4e9130547
Fix race condition that can corrupt grads under certain conditions.
2026-08-07 14:52:47 -06:00
Jaret Burkett
4625406093
WIP working on convrot offloading
2026-07-11 15:37:28 -06:00
Jaret Burkett
1d1e21177a
Add ostris linear layer to linear layer searches.
2026-07-11 14:21:16 -06:00
Jaret Burkett
c78b1404e3
Deepen offload prefetch pipeline with per-slot events
...
Replace the 2-slot ping-pong + single global "compute-started" event
with a depth-N ring buffer where each transfer waits only on the slot
it's reusing (D layers back) instead of the most-recent compute. Applies
to forward and backward, Linear and Conv. Depth is tunable via
AI_TOOLKIT_OFFLOAD_DEPTH (default 4).
Bit-exact vs non-offload (output, grad_input, weight grads). No speedup
on a bandwidth-bound PCIe link (already saturated at depth 2), but the
cleaner per-slot design removes the fragile shared-event serialization
and lets deeper prefetch help on faster buses.
2026-06-07 16:07:13 -06:00
Jaret Burkett
cdff6e36aa
Pin inner stores of torachao to speed up layer offloading for quantized models around 25%
2026-06-07 15:52:01 -06:00
Jaret Burkett
75781fb5a5
Fix float8 weights not offloading to CPU in layer offloading
2026-06-07 15:36:17 -06:00
PlagueKind
7c1a76f336
Fix text encoder offload bug when caching embeddings ( #868 )
2026-06-07 14:09:32 -06:00
Jaret Burkett
323b4aaf5a
Do not copy pin memory if it fails, just move
2025-11-17 18:04:00 +00:00
Jaret Burkett
0d8a33dc16
Offload ARA with the layer if doing layer offloading. Add support to offload the LoRA. Still needs optimizer support
2025-10-21 06:03:27 -06:00
Jaret Burkett
76ce757e0c
Added initial support for layer offloading wit Wan 2.2 14B models.
2025-10-20 14:54:30 -06:00
Jaret Burkett
7abf5e20be
Add conv3d to memory management excluded modules
2025-10-15 10:12:06 -06:00
Jaret Burkett
1bc6dee127
Change auto_memory to be layer_offloading and allow you to set the amount to unload
2025-10-10 13:12:32 -06:00
Jaret Burkett
55b8b0e23e
Fix issue where ARA was not working when using memory manager
2025-10-07 13:39:44 -06:00
Jaret Burkett
c9f982af83
Add support for using quantized models with ramtorch
2025-10-06 13:46:57 -06:00
Jaret Burkett
4e5707854f
Initial support for RamTorch. Still a WIP
2025-10-05 13:03:26 -06:00
Jaret Burkett
3086a58e5b
git status
2025-10-01 14:12:17 -06:00
