Using RoBERTa with more than 512 tokens

❓ Questions and Help

What is your question?

Is there a way to use RoBERTa with sequences longer than 512 tokens (e.g., 2,000+)? I know RoBERTa was pretrained with a 512-token limit. I’m wondering if there’s an officially supported way or a recommended workaround — such as modifying the positional embeddings or using a sliding window approach — that works well with Fairseq’s RoBERTa implementation.

Would appreciate any advice or examples!

Code

No specific code yet — just exploring the feasibility before implementation.

What have you tried?

I’ve read some discussions suggesting modifying the max_positions and expanding the positional embedding matrix, but I’m unsure how stable or supported this is in Fairseq. I’ve also considered using chunking or windowed inference, but I’d prefer to keep as much global context as possible.