inmem: Truncate method doesn't take AST store into account
When the server is run with --optimize-store-for-read-speed passed, or an in-mem store is created programmatically with OptReturnASTValuesOnRead(true), snapshot bundle updates currently has the store's Truncate method:
- Read entire data tree from bundle as
map[string]any - Save it to store, converting it to AST in the process.
This means you'll pay twice for any data update when AST storage is enabled (and it should be!) as memory is spent only to account for an intermediate format that is immediately discarded.
I have only looked into it briefly, but I think the problem may go a little deeper than just the Truncate method, as the Data field on the Bundle struct is currently of type map[string]any. What we really want for the AST store is of course AST all the way. Meaning that data in a bundle is unmarshaled to AST with no steps in between, and then we use that to truncate what's currently stored.
Source: open-policy-agent/opa