Jaret Burkett
55ce6570f2
Automagic 3 rework. Stable in my testing.
2026-06-12 07:52:44 -06:00
fatalis
53ebb93edb
fix learning rate metric truncating to zero on graph ( #877 )
2026-06-12 07:30:21 -06:00
Jaret Burkett
88127557f5
Comment out logging supression that was keeping weight loadings from being shown with transformers library
2026-06-11 07:53:41 -06:00
Jaret Burkett
01b6a9806b
Another complete rework of automagic3. Added a decay to the LR spread to the mean to prevent LRs fighting with eachother
2026-06-09 12:08:45 -06:00
Jaret Burkett
9e99d3ce5d
Save loss graph view settings per page to local storage.
2026-06-09 07:29:51 -06:00
Jaret Burkett
acb1548722
Updated the comments and doc for Automagic v3
2026-06-09 07:05:48 -06:00
Jaret Burkett
a1ac6e8b01
Reworked automagic v3 again. Seems more stable. Still testing.
2026-06-08 22:03:54 -06:00
Jaret Burkett
5d6887fd98
Major updates to automagic3 optimizer. Seems to be functioning more ideally and naturally decays, as it should.
2026-06-08 14:09:02 -06:00
Jaret Burkett
0d018db689
Rework the toggles so we can hide the trend line from the loss graph. No need for a raw toggle either.
2026-06-08 13:56:37 -06:00
Jaret Burkett
cac3815b2c
Make DOP run in a single backward pass, should be faster and more stable. Show dop loss on ui
2026-06-08 13:55:47 -06:00
Jaret Burkett
687def6f7a
Add better smoothing with ema rounding of the ends of the loss graph for first and last instance do not dominate the smoothing factors.
2026-06-08 10:43:46 -06:00
Jaret Burkett
d7f8887bbf
Show all logged metrics on the loss graph
2026-06-08 09:43:03 -06:00
Jaret Burkett
e281df70dd
Allow automagic3 to run unfused. Add some clipping.
2026-06-08 08:59:06 -06:00
Jaret Burkett
c9cdbb5bb7
Version bump
2026-06-07 16:08:11 -06:00
Jaret Burkett
c78b1404e3
Deepen offload prefetch pipeline with per-slot events
...
Replace the 2-slot ping-pong + single global "compute-started" event
with a depth-N ring buffer where each transfer waits only on the slot
it's reusing (D layers back) instead of the most-recent compute. Applies
to forward and backward, Linear and Conv. Depth is tunable via
AI_TOOLKIT_OFFLOAD_DEPTH (default 4).
Bit-exact vs non-offload (output, grad_input, weight grads). No speedup
on a bandwidth-bound PCIe link (already saturated at depth 2), but the
cleaner per-slot design removes the fragile shared-event serialization
and lets deeper prefetch help on faster buses.
2026-06-07 16:07:13 -06:00
Jaret Burkett
cdff6e36aa
Pin inner stores of torachao to speed up layer offloading for quantized models around 25%
2026-06-07 15:52:01 -06:00
Jaret Burkett
75781fb5a5
Fix float8 weights not offloading to CPU in layer offloading
2026-06-07 15:36:17 -06:00
PlagueKind
7c1a76f336
Fix text encoder offload bug when caching embeddings ( #868 )
2026-06-07 14:09:32 -06:00
Jaret Burkett
35588726de
Fixed issue where a buffer was stuck on cpu when offloading ideogram4
2026-06-07 13:31:03 -06:00
Jaret Burkett
1dc9a797cf
Added Automagic v3
2026-06-07 12:06:42 -06:00
Jaret Burkett
82190b41e6
Fix bug where EMA was not being initialized even when setup in the config. EMA will now properly be setup and used.
2026-06-07 11:11:49 -06:00
Jaret Burkett
8968e41234
Set ideogram to use new loke saving format
2026-06-07 09:49:57 -06:00
Jaret Burkett
fa0dca288d
Version bump
2026-06-06 10:24:31 -06:00
Jaret Burkett
41157b460c
Added ability to set the caption extention in dataset viewer, captioner, and trainer so one dataset can have multiple caption styles in different files with different extensions. Added dataset caption template for a blank ideogram 4 formatted template.
2026-06-06 08:32:24 -06:00
Jaret Burkett
10cdeb394e
Version bump
2026-06-05 14:09:46 -06:00
Jaret Burkett
21a6beb194
Allow editing the boxes visually on the sample section for ideogram
2026-06-05 14:09:18 -06:00
Jaret Burkett
4441080c05
Rework the ui of ideogram dataset image caption editor
2026-06-05 13:20:58 -06:00
Jaret Burkett
b70083a74f
Fix issue with default sampels.
2026-06-05 12:33:42 -06:00
Jaret Burkett
c994398850
Rework prompts and captioning systems for ideogram to more strictly match the format provided by ideogram.
2026-06-05 11:14:36 -06:00
Jaret Burkett
6fd2253932
Imporved editing boxed on the dataset viewer
2026-06-05 10:30:06 -06:00
Jaret Burkett
ef12260b80
Add a prompt upsample ui for upsampling prompts to ideogram format prompts.
2026-06-05 09:56:02 -06:00
Jaret Burkett
90a2084f70
Allow adjusting, adding, and deleting bounding boxes.
2026-06-04 20:29:31 -06:00
Jaret Burkett
bb60f6d1d1
Show bounding boxes on the sample image card if we have them in the prompt
2026-06-04 15:59:02 -06:00
Jaret Burkett
edcc7415d1
Added an autocaptioner for Ideogram 4 captions
2026-06-04 13:04:23 -06:00
Jaret Burkett
6a8d9333b6
Improved the prompt handeling of ideogram4 model. Now used advanced prompts class to store them smaller and allow longer prompts
2026-06-04 13:00:23 -06:00
Jaret Burkett
2ddc2e1318
Update the default Ideogram 4 prompts to work significantly better.
2026-06-04 10:32:05 -06:00
Jaret Burkett
63b3181262
Added experimental support for Ideogram 4
2026-06-04 09:03:00 -06:00
Jaret Burkett
b5f21ae695
Fixed issue where the training type dropdown in the ui was not showing fully
2026-06-03 10:14:15 -06:00
Jaret Burkett
d9f26c2f87
Add gradient checkpointing to tipsv2 heads
2026-06-01 05:00:15 -06:00
Jaret Burkett
bd468727a6
Fixed issue where some image cards in samples and datasets would show gray instead of the image until you scroll.
2026-06-01 03:36:32 -06:00
Jaret Burkett
f5446c0d5f
Version bump
2026-06-01 02:16:47 -06:00
Jaret Burkett
e5439509b5
Added pure lpips dfe
2026-05-31 11:52:03 -06:00
Jaret Burkett
212cfe998a
Add ability to download or delete the optimizer state from the ui
2026-05-29 08:56:22 -06:00
Jaret Burkett
5e84bf0d0b
Fixed issue with hidream-01 that could cause a weird nan state. Took forever to track down as it was 1 in 10 starts.
2026-05-28 12:48:24 -06:00
Jaret Burkett
87bac27513
Fixed issue with new bucket scaler
2026-05-28 11:34:16 -06:00
Jaret Burkett
30886b8f92
Added a wallet indicator for Ostris Cloud
2026-05-28 11:09:51 -06:00
Jaret Burkett
3e86d81fc6
Adjust bucket sizes to achieve maximum pixels without going over.
2026-05-28 09:36:17 -06:00
Jaret Burkett
ef57c1077c
Change z image divisibility
2026-05-28 09:09:36 -06:00
Jaret Burkett
15082cfb8a
Round buckets for divisibility instead of always rounding down.
2026-05-28 09:08:21 -06:00
Jaret Burkett
c9264bdd0b
Reworked the bucketing system to precisly match model specific divisibility. The old SDXL bucket sizes needed to go.
2026-05-28 08:35:58 -06:00
