#3411·ParlAI

New Metric.from_mask helper method

Author: stephenrollerCreated Jan 26, 2021Updated Nov 23, 2022
LabelsEnhancementHelp WantedMediumnever-stale

We have quite a few instances where we have some per-token losses/metrics along with a corresponding mask

python
metric_per_token # torch.Tensor of shape (batchsize, num_tokens)
mask # torch.BoolTensor of shape (batchsize, num_tokens)

And we want a per-batch-example average:

python
tokens_per_ex = mask.long().sum(dim=-1)
metric_per_ex = (metric_per_token * mask).sum(dim=-1)
metrics: List[MyMetric] = MyMetric.many(metric_per_ex, tokens_per_ex)
self.record_local_metric('metric_name', metrics)

I'd like us to have a helper classmethod in Metric called from_mask:

python
class Metric:
    @classmethod
    def from_mask(cls, metric_per_token, token_mask):
        # returns the equivalent of the "metrics" object above

Once this is done, add unit tests for this (test AverageMetric and PPLMetric directly). Checkpoint there.

After you've implemented this, upgrade TorchGeneratorAgent to use your new helper, upgrading the code for loss, ppl, and token_acc.

Example: https://github.com/facebookresearch/ParlAI/blob/67433e376fc361dee5aa045cb6bb2b68d3faa478/parlai/core/torch_generator_agent.py#L711-L727

See if you can find at least one other place who can benefit from upgrading this pattern.