I looked at the llama source code and there is an intermedie layer

Author: wac81Created Jun 10, 2023Updated Jun 10, 2023

I looked at the llama source code and there is an intermedie layer that palm doesn't have or I didn't find it? Can you point it out clearly? Or how to add it?

Source: lucidrains/PaLM-rlhf-pytorch