github/ai-toolkit
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Switched ip adapter dataloader to clip_image paths so the control paths can be used for training assistant adapters while training ip adapters

Browse Source
This commit is contained in:
Jaret Burkett
2023-12-20 10:32:24 -07:00
parent dfb64b5957
commit 0f597f453e
5 changed files with 94 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
toolkit/ip_adapter.py
View File

@@ -153,7 +153,10 @@ class IPAdapter(torch.nn.Module):
super().__init__()
self.config = adapter_config
self.sd_ref: weakref.ref = weakref.ref(sd)
self.clip_image_processor = CLIPImageProcessor.from_pretrained(adapter_config.image_encoder_path)
try:
self.clip_image_processor = CLIPImageProcessor.from_pretrained(adapter_config.image_encoder_path)
except EnvironmentError:
self.clip_image_processor = CLIPImageProcessor()
self.device = self.sd_ref().unet.device
self.image_encoder = CLIPVisionModelWithProjection.from_pretrained(adapter_config.image_encoder_path,
ignore_mismatched_sizes=True)