无法从检查点恢复模型

我无法从检查点恢复训练模型。错误日志如下：/data/zmining/jupyter-notebook/antnh/embeddings/tml/common/checkpointing/snapshot.py，第67行，在restore中：File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torchsnapshot/snapshot.py"，第406行，在restore中：self._load_stateful( self.state) #check File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torchsnapshot/snapshot.py"，第671行，在_load_stateful中：stateful.load_state_dict(state_dict) File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py"，第2027行，在load_state_dict中：load(self, state_dict) File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py"，第2015行，在load中：load(child, child_state_dict, child_prefix) File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py"，第2015行，在load中：load(child, child_state_dict, child_prefix) File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py"，第2015行，在load中：load(child, child_state_dict, child_prefix) [上一行重复了2次] File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py"，第2009行，在load_from_state_dict中：module._load_from_state_dict( File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py"，第1909行，在_load_from_state_dict中：hook(state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py"，第69行，在__call__中：return self.hook(module, *args, **kwargs) File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torchrec/distributed/embeddingbag.py"，第439行，在_pre_load_state_dict_hook中：local_shards = state_dict[key].local_shards() KeyError: 'model._dmp_wrapped_module.module.large_embeddings.ebc.embedding_bags.user.weight'