DALL·E 2 文生图模型实践指南
前言：本篇博客记录使用dalle2模型进行推断时借鉴的相关资料和DEBUG流程。
相关博客：超详细！DALL · E 文生图模型实践指南
目录
- 1. 环境搭建和预训练模型准备
- 环境搭建
- 预训练模型下载
- 2. 代码
- 3. BUG&DEBUG
- URLError
- CUDA error
- RuntimeError
- PydanticUserError
1. 环境搭建和预训练模型准备
本文使用的代码仓库为：https://github.com/lucidrains/DALLE2-pytorch
环境搭建
pip install dalle2-pytorch
预训练模型下载
地址：https://huggingface.co/laion/DALLE2-PyTorch
2. 代码
DALLE2 for inference 完整推断流程如下（from @cest_andre in Issues#282）：
import torch from torchvision.transforms import ToPILImage from dalle2_pytorch import DiffusionPrior, DiffusionPriorNetwork, OpenAIClipAdapter, Decoder, DALLE2 from dalle2_pytorch.train_configs import TrainDiffusionPriorConfig, TrainDecoderConfig prior_config = TrainDiffusionPriorConfig.from_json_path("weights/prior_config.json").prior prior = prior_config.create().cuda() prior_model_state = torch.load("weights/prior_latest.pth") prior.load_state_dict(prior_model_state, strict=True) decoder_config = TrainDecoderConfig.from_json_path("weights/decoder_config.json").decoder decoder = decoder_config.create().cuda() decoder_model_state = torch.load("weights/decoder_latest.pth")["model"] for k in decoder.clip.state_dict().keys(): decoder_model_state["clip." + k] = decoder.clip.state_dict()[k] decoder.load_state_dict(decoder_model_state, strict=True) dalle2 = DALLE2(prior=prior, decoder=decoder).cuda() images = dalle2( ['your prompt here'], cond_scale = 2. ).cpu() print(images.shape) for img in images: img = ToPILImage()(img) img.show()3. BUG&DEBUG
URLError
报错信息如下：
Traceback (most recent call last): File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 1350, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "/root/anaconda3/envs/ldm/lib/python3.8/http/client.py", line 1255, in request self._send_request(method, url, body, headers, encode_chunked) File "/root/anaconda3/envs/ldm/lib/python3.8/http/client.py", line 1301, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/root/anaconda3/envs/ldm/lib/python3.8/http/client.py", line 1250, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/root/anaconda3/envs/ldm/lib/python3.8/http/client.py", line 1010, in _send_output self.send(msg) File "/root/anaconda3/envs/ldm/lib/python3.8/http/client.py", line 950, in send self.connect() File "/root/anaconda3/envs/ldm/lib/python3.8/http/client.py", line 1424, in connect self.sock = self._context.wrap_socket(self.sock, File "/root/anaconda3/envs/ldm/lib/python3.8/ssl.py", line 500, in wrap_socket return self.sslsocket_class._create( File "/root/anaconda3/envs/ldm/lib/python3.8/ssl.py", line 1040, in _create self.do_handshake() File "/root/anaconda3/envs/ldm/lib/python3.8/ssl.py", line 1309, in do_handshake self._sslobj.do_handshake() ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/newdata/SD/extra/dalle2_cest.py", line 11, in prior = prior_config.create().cuda() File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/dalle2_pytorch/train_configs.py", line 185, in create clip = self.clip.create() File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/dalle2_pytorch/train_configs.py", line 122, in create return OpenAIClipAdapter(self.model) File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 313, in __init__ openai_clip, preprocess = clip.load(name) File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/clip/clip.py", line 122, in load model_path = _download(_MODELS[name], download_root or os.path.expanduser("~/.cache/clip")) File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/clip/clip.py", line 59, in _download with urllib.request.urlopen(url) as source, open(download_target, "wb") as output: File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 525, in open response = self._open(req, data) File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 542, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 502, in _call_chain result = func(*args) File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 1393, in https_open return self.do_open(http.client.HTTPSConnection, req, File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 1353, in do_open raise URLError(err) urllib.error.URLError:我使用的是https://github.com/lucidrains/DALLE2-pytorch这个网址。
找到 /root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py 中对应的位置，我这里是第1349行，修改方式也在下面代码中一并给出。
try: h.request(req.get_method(), req.selector, req.data, headers, encode_chunked=req.has_header('Transfer-encoding')) time.sleep(0.5) # 添加的一行 except OSError as err: # timeout error raise URLError(err)CUDA error
RuntimeError: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
解决方案：版本不匹配，更换与系统cuda相匹配的pytorch版本。比如我的cuda版本是12.0，可以使用如下命令安装pytorch：
pip install torch==2.0.0+cu118 torchvision==0.15.1+cu118 torchaudio==2.0.1+cu118 -f https://download.pytorch.org/whl/torch_stable.html
RuntimeError
Traceback (most recent call last): File "/newdata/SD/extra/dalle2_cest.py", line 14, in prior.load_state_dict(prior_model_state, strict=True) File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1604, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for DiffusionPrior: Missing key(s) in state_dict: "net.null_text_encodings", "net.null_text_embeds", "net.null_image_embed". Unexpected key(s) in state_dict: "net.null_text_embed".解决方案1️⃣：load_state_dict()函数中的 strict=True 改为 strict=False，如下：
... prior.load_state_dict(prior_model_state, strict=False) decoder.load_state_dict(decoder_model_state, strict=False) ...
但这种方法可能会导致模型的性能下降，而且会生成马赛克图像，这显然不是我们想要的结果。
解决方案2️⃣：参考Issues中cest-andre的答案。
步骤（1）降低dalle2_pytorch版本至1.1.0：
pip install dalle2-pytorch==1.1.0
步骤（2）：版本降低后，要修复dalle2_pytorch.py文件中一个小bug：将第2940行改为如下代码：
images = self.decoder.sample(image_embed = image_embed, text = text_cond, cond_scale = cond_scale)
PydanticUserError
降低dalle2_pytorch版本后，运行程序报错如下：
Traceback (most recent call last): File "/newdata/SD/extra/dalle2_cest.py", line 8, in from dalle2_pytorch.train_configs import TrainDiffusionPriorConfig, TrainDecoderConfig File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/dalle2_pytorch/train_configs.py", line 34, in class TrainSplitConfig(BaseModel): File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/dalle2_pytorch/train_configs.py", line 40, in TrainSplitConfig def validate_all(cls, fields): File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/pydantic/deprecated/class_validators.py", line 222, in root_validator return root_validator()(*__args) # type: ignore File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/pydantic/deprecated/class_validators.py", line 228, in root_validator raise PydanticUserError( pydantic.errors.PydanticUserError: If you use `@root_validator` with pre=False (the default) you MUST specify `skip_on_failure=True`. Note that `@root_validator` is deprecated and should be replaced with `@model_validator`.解决方案：参考Issues中JasbirCodeSpace的答案，降低Pydantic的版本：
pip install pydantic==1.10.6
到这里，模型就可以完成推断过程啦～嘻嘻！以下是prompt为a red car时生成的图像：
后记：感谢前人铺路！🌹
参考链接
- https://github.com/lucidrains/DALLE2-pytorch/issues/282
- python requests请求报错ConnectionError: (‘Connection aborted.‘, error(104, ‘Connection reset by peer‘))_铁朵斯提的博客-CSDN博客
- GPU版本pytorch（Cuda12.1）清华源快速安装一步一步教！小白教学~_清华源安装torch-CSDN博客
免责声明：我们致力于保护作者版权，注重分享，被刊用文章因无法核实真实出处，未能及时与作者取得联系，或有版权异议的，请联系管理员，我们会立即处理! 部分文章是来自自研大数据AI进行生成,内容摘自(百度百科,百度知道,头条百科,中国民法典,刑法,牛津词典,新华词典,汉语词典,国家院校,科普平台)等数据,内容仅供学习参考,不准确地方联系删除处理! 图片声明：本站部分配图来自人工智能系统AI生成,觅知网授权图片,PxHere摄影无版权图库和百度，360，搜狗等多加搜索引擎自动关键词搜索配图，如有侵权的图片，请第一时间联系我们，邮箱：ciyunidc@ciyunshuju.com。本站只作为美观性配图使用,无任何非法侵犯第三方意图,一切解释权归图片著作权方,本站不承担任何责任。如有恶意碰瓷者,必当奉陪到底严惩不贷!
