Commit Graph

655 Commits

Author SHA1 Message Date
Jaret Burkett
01b6a9806b Another complete rework of automagic3. Added a decay to the LR spread to the mean to prevent LRs fighting with eachother 2026-06-09 12:08:45 -06:00
Jaret Burkett
acb1548722 Updated the comments and doc for Automagic v3 2026-06-09 07:05:48 -06:00
Jaret Burkett
a1ac6e8b01 Reworked automagic v3 again. Seems more stable. Still testing. 2026-06-08 22:03:54 -06:00
Jaret Burkett
5d6887fd98 Major updates to automagic3 optimizer. Seems to be functioning more ideally and naturally decays, as it should. 2026-06-08 14:09:02 -06:00
Jaret Burkett
e281df70dd Allow automagic3 to run unfused. Add some clipping. 2026-06-08 08:59:06 -06:00
Jaret Burkett
c78b1404e3 Deepen offload prefetch pipeline with per-slot events
Replace the 2-slot ping-pong + single global "compute-started" event
with a depth-N ring buffer where each transfer waits only on the slot
it's reusing (D layers back) instead of the most-recent compute. Applies
to forward and backward, Linear and Conv. Depth is tunable via
AI_TOOLKIT_OFFLOAD_DEPTH (default 4).

Bit-exact vs non-offload (output, grad_input, weight grads). No speedup
on a bandwidth-bound PCIe link (already saturated at depth 2), but the
cleaner per-slot design removes the fragile shared-event serialization
and lets deeper prefetch help on faster buses.
2026-06-07 16:07:13 -06:00
Jaret Burkett
cdff6e36aa Pin inner stores of torachao to speed up layer offloading for quantized models around 25% 2026-06-07 15:52:01 -06:00
Jaret Burkett
75781fb5a5 Fix float8 weights not offloading to CPU in layer offloading 2026-06-07 15:36:17 -06:00
PlagueKind
7c1a76f336 Fix text encoder offload bug when caching embeddings (#868) 2026-06-07 14:09:32 -06:00
Jaret Burkett
1dc9a797cf Added Automagic v3 2026-06-07 12:06:42 -06:00
Jaret Burkett
41157b460c Added ability to set the caption extention in dataset viewer, captioner, and trainer so one dataset can have multiple caption styles in different files with different extensions. Added dataset caption template for a blank ideogram 4 formatted template. 2026-06-06 08:32:24 -06:00
Jaret Burkett
6a8d9333b6 Improved the prompt handeling of ideogram4 model. Now used advanced prompts class to store them smaller and allow longer prompts 2026-06-04 13:00:23 -06:00
Jaret Burkett
d9f26c2f87 Add gradient checkpointing to tipsv2 heads 2026-06-01 05:00:15 -06:00
Jaret Burkett
e5439509b5 Added pure lpips dfe 2026-05-31 11:52:03 -06:00
Jaret Burkett
87bac27513 Fixed issue with new bucket scaler 2026-05-28 11:34:16 -06:00
Jaret Burkett
3e86d81fc6 Adjust bucket sizes to achieve maximum pixels without going over. 2026-05-28 09:36:17 -06:00
Jaret Burkett
15082cfb8a Round buckets for divisibility instead of always rounding down. 2026-05-28 09:08:21 -06:00
Jaret Burkett
c9264bdd0b Reworked the bucketing system to precisly match model specific divisibility. The old SDXL bucket sizes needed to go. 2026-05-28 08:35:58 -06:00
Jaret Burkett
2aa60e4ca5 Update default agreement threshold for automagic v2 to be 0.5 2026-05-26 07:57:54 -06:00
Jaret Burkett
12304e170f Added some experimental loss targets 2026-05-24 14:13:23 -06:00
Jaret Burkett
aecd554128 Add sapiens2 matting as a mask generator. Begin transition to model paths and model folders. 2026-05-20 08:56:16 -06:00
Jaret Burkett
b42acb988f Remove future steps from loss log if resuming from an earlier step 2026-05-18 09:23:55 -06:00
Jaret Burkett
4bfe944792 Scale dfe 7 with velocity_equiv_weight 2026-05-13 09:06:59 -06:00
Jaret Burkett (Ostris)
fc4d6ebf39 Add support for fine-tuning Hidream O1 (#831)
* Initial support for hidream. Lora keys likely need work

* Fix saving for hidream-o1

* Remove dependence on flash attention for hidream o1

* Fix gradient checkpointing for hidream o1

* A lot of fixes for hidream. Handle loading and saving as comfy model.

* Omit layers not used in comfy. Fix issue with lora loading keys in comfy

* Version bumpo
2026-05-12 11:15:16 -06:00
Jaret Burkett
f38de2a2fe Add tipsv2 locally and fix gradient checkpointing for it 2026-05-10 14:47:44 -06:00
Jaret Burkett
a12ddd72a1 Change the velocity weight cap on dfe 9 2026-05-07 07:37:05 -06:00
Jaret Burkett
6bb8acbffc Add agreement_threshold default of 0.6 to automagic 2 2026-05-05 19:13:00 -06:00
Jaret Burkett
963a9f42b2 Add decode latent to wan 2.1 models. Add gradinet checkpointing to wan vae. 2026-05-05 11:30:16 -06:00
Jaret Burkett
4260a3c5b6 Add optimizer test suite and make minor speed adjustments to Automagicv2 2026-05-05 10:02:30 -06:00
Jaret Burkett
aeca7fe404 Add Automagic v2 optimizer. It uses significantly less vram and is much more efficient. 2026-05-05 09:09:07 -06:00
Jaret Burkett
af6458d1b5 Enable caching of ACE step latents. 2026-04-28 13:39:20 -06:00
Jaret Burkett
acc6a36214 Scale DFE 9 to a velocity equiv weight to match flow matching gradient strength. Probably need to rework all DFEs to do this as the math checks out. 2026-04-28 09:10:02 -06:00
Jaret Burkett
1fc4ad3979 Add sapiens2 as a diffusion feature extractor 2026-04-27 15:59:03 -06:00
Jaret Burkett
f4445cd78c Added psuedo_huber loss 2026-04-19 09:51:46 -06:00
Jaret Burkett
488878f354 Use hidden layers in the loss for DFE 7 and 8 2026-04-18 13:07:38 -06:00
Jaret Burkett
beb40ae29b Add DFE8 with partial step 2026-04-17 17:40:16 -06:00
Jaret Burkett
22ea3dd620 Fixed issue on some systems where Logger didnt have atty 2026-04-16 21:09:52 +00:00
Jaret Burkett
ab1ee4df34 Hotfix some issues with Wan models caused by diffusers and transformers updates 2026-04-16 20:53:50 +00:00
Jaret Burkett
2faba22b46 Fix issue when saving advanced prompt embeds. No such file or directory error 2026-04-16 12:22:56 -06:00
Jaret Burkett
e74bc9ac7b Fix issue with concatinating advanced prompt embeds. 2026-04-14 16:04:34 -06:00
Jaret Burkett
7eb1226a6d Fix issue with loading advanced prompt configs metadata 2026-04-14 15:42:09 -06:00
Asaf Agami
e868fca562 fix custom_flowmatch_sampler (#783) 2026-04-13 09:42:05 -06:00
Jaret Burkett
233e292256 Added some experimental low step things for zeta 2026-04-13 09:37:34 -06:00
Jaret Burkett
1058ef3513 Made AdvancedPromptEmbeds that is compatable with previous PromptEmbeds functionality, but is more streamlined and can accomidate more model embedding paradigms. 2026-04-11 10:45:47 -06:00
Jaret Burkett
9b4e2d1b0b More flac support 2026-04-10 12:27:09 -06:00
Jaret Burkett
bfb373c8fa Prep for future breaking changes in newer versions of transformers library 2026-04-10 12:04:32 -06:00
Jaret Burkett
78cf049c29 Add support for ACE-Step 1.5 and ACE-Step 1.5 XL. Also added dataset captioning through the UI. (#785)
* Base ace step 1.5 xl added. Generating, still wip on training and ui

* Base training code done

* Fix some issues with caching text embeddings. Update sample cards to show audio

* Fix issue with quantizing ace step

* Add album artwork to samples with waveform.

* Cleanup logs

* Add album art endpoint to speed up album art loading

* Made an make video with artwork script

* Make ui handle basic audio models. Make multi line adjustments to the editor and better syntax hilighting.

* Add prompt tagging system for special tagged models.

* prompt tagging processing for ui working.

* Moved default samples to a special file so we can add more when needed and they can be adjusted for a specific model

* Add a captioner job with music captioner that is prepped for use with the ui

* Add basit ui setup for captioning modal and handeling captioning jobs

* Starting captioning job from ui working. Still better management for it.

* Better filtering of job options in the job view for captioning jobs

* Added qwen3 vl as a captioner for images

* Have an indicator when a dataset is being captioned.

* Adjust the way caption jobs look in the queue

* Fix a few issues. Adjust defaults.

* Version bump

* Added ace step to the readme.
2026-04-09 15:02:03 -06:00
Jaret Burkett
da2a79590f Add a merge network on save strength 2026-04-01 09:21:08 -06:00
Jaret Burkett
4a3251640a More work on compiling models 2026-03-31 12:11:56 -06:00
Jaret Burkett
171535833a Add Mac OS support for Apple Silicon (#770)
* Made an install script and auto updates env for mac

* GPU sensors and initial training working for MAC. Still WIP.

* Switch dataloader to single threaded until I can work around some mac pickeling issues.

* Get quantization working on mac

* Fix mac exclusive imports so they don't break other builds.

* Add mac instructions to the UI
2026-03-30 09:37:47 -06:00