Why not add transformer tokens to vocabulary in init phrase
I met a problem yesterday when I want to get the vocab_size of the vocabulary namespace transformer_tags which I specify in PretrainedTransformerIndexer. I found this namespace doesn't defined when I want to use it in the Model init phrase. To figure out this problem, I read this part of source codes. I find the operation _add_encoding_to_vocabulary_if_needed is called in tokens_to_indices which is called util the _train_epoch starts.
...
class PretrainedTransformerIndexer
...
def _add_encoding_to_vocabulary_if_needed(self, vocab: Vocabulary) -> None:
"""
Copies tokens from ```transformers``` model's vocab to the specified namespace.
"""
if self._added_to_vocabulary:
return
vocab.add_transformer_vocab(self._tokenizer, self._namespace)
self._added_to_vocabulary = True
@overrides
def count_vocab_items(self, token: Token, counter: Dict[str, Dict[str, int]]):
# If we only use pretrained models, we don't need to do anything here.
pass
@overrides
def tokens_to_indices(self, tokens: List[Token], vocabulary: Vocabulary) -> IndexedTokenList:
self._add_encoding_to_vocabulary_if_needed(vocabulary)
... It means the transformer_tags namespace I defined in the vocabulary only can be used in the forward process and after it. This causes a probelm that I can't define a linear layer that transforms the model output to vocab_size logits. Of course, I can use from_pretrained_transformer constructor to get a same namespace that I can use in the init phrase. If this is planned, I wonder what's the usage of _add_encoding_to_vocabulary_if_needed in the PretrainedTransformerIndexer. Why not call _add_encoding_to_vocabulary_if_needed in the init method of PretrainedTransformerIndexer so that we can use the specified namespace from the Model init phrase?
Looking forward to your kind reply ~~ Thanks~~
Source: allenai/allennlp