Llama2 transfer to Llama3
Author: SummoninggCreated Jun 21, 2024Updated Aug 6, 2025
Labelsquestion
Can I simply transfer a llama2 task to llama3 by just loading a llama3 with transformers? Or do i need to rewrite some codes?
I loaded the llama3 and it came like
raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
size mismatch for model.layers.0.self_attn.k_proj.weight: copying a param with shape torch.Size([1024, 4096]) from checkpoint, the shape in current model is torch.Size([4096, 4096]).
You may consider adding `ignore_mismatched_sizes=True` in the model `from_pretrained` method.and when I added the ignore_mismatched_sizes=True, it was like
Traceback (most recent call last):
File "train.py", line 53, in <module>
main()
File "train.py", line 49, in main
train(args)
File "train.py", line 35, in train
model = llama(args)
File ".py", line 96, in __init__
self.llama_model = AutoModelForCausalLM.from_pretrained(
File "/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 484, in from_pretrained
return model_class.from_pretrained(
File "/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2881, in from_pretrained
) = cls._load_pretrained_model(
File "/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3218, in _load_pretrained_model
mismatched_keys += _find_mismatched_keys(
File "/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3141, in _find_mismatched_keys
and state_dict[checkpoint_key].shape != model_state_dict[model_key].shape
KeyError: 'lm_head.weight'
How to fix this? or rewrite the code?
Source: meta-llama/llama3