Acc vs acc_norm

Hi there. I'll preface this by saying I'm likely failing to understand something about how these logprobs calculations work.

So -- why is acc_norm consistently higher than acc? They're just ways of determining which answer the model thinks is most likely. So, acc underperforming means it's doing a worse job of determining the model's intended answer.

It seems that there ought to be a right and wrong way to do this. If the logprobs are a sum of the logprobs of the completion tokens, then it makes sense to normalise by completion length. If the logprobs value we are working with are an average of the logprobs of the completion tokens, it doesn't.

I haven't dug enough to figure out if it's a sum or an avg, but it seems reasonable to conclude that it's a sum, given that acc_norm produces higher scores. In which case, shouldn't we always be using acc_norm when calculating multiple choice scores with logprobs?