[NEW] Allow Module extension of metadata for dump/restore of keys.

The problem/use-case that the feature addresses

In the search module, when indexes are created with the SKIPINITIALSCAN option, keys which match the prefix-list definition for that index are only conditionally present. Specifically, if a key is present at the time of the index creation AND has not been modified since that time, then it will not be in the index and queries against the index will not return that key. Conversely, if the key is either created or modified after the creation of that index, then the key will be present in the index and potentially be part of a query result.

Logically, this is metadata for a key. The current migrate/DUMP/RESTORE/RESTORE-ASKING machinery (hereinafter, referred to as machinery) recognizes that keys have metadata, but the machinery has a fixed list of metadata items which cannot be extended.

Today, search has worked around this limitation for the RDB, but has no solution for slot migration. Without per-key metadata

Description of the feature

The feature extends the machinery with per-module metadata associated with the key. Modules register their desire to append per-key metadata which is orchestrated by a pair of callbacks that are invoked for each dump/restored key. The machinery is enhanced to transfer the additional metadata in a backward compatible fashion.

The module interface

The module interface consists of two callback functions: one for dump and one for restore. Naturally there is a registration function too.

// (VM is short-hand for ValkeyModule, used here to simplify the full names.)
/*
The Dump metadata function is called as part of the dump operation.

It returns either a NULL indicating that no metadata is provided OR
It allocates and returns ownership of a VMString containing the metdata from this module for this key.
*/
typedef VMString *(*VM_KeyMetaDumpFunc)(VMCtx *ctx, VMString *key);

/*
The Restore Metadata function is invoked by any RESTORE command that has been decorated with
metadata from this module. It will only be called if metadata from this module was provided.

The "key" input parameter is read-only and ownership of the string remains with the caller.

The return parameter indicates validation of the input: SUCCESS or FAILURE.
If FAILURE is returned the caller is expected to fail the overall operation (RDB Load or slot migration).

Regardless of SUCCESS or FAILURE, ownership of the "metadata" parameter is transferred to the module.

*/
typedef int (*VM_KeyMetaRestoreFunc)(VMCtx *ctx, VMString *key, VMString *metadata);

/*
Register the callbacks.
Registering a NULL disables the callback.
Callbacks are automatically unregistered at module unload.
*/
int VM_RegisterKeyMetadata(VMCtx *ctx, VM_KeyMetaDumpFunc dump_func, VM_KeyMetaRestoreFunc restore_func);

The transport machinery

Restoring Commands

The RESTORE and RESTORE-ASKING commands are extended to provide per-module metadata. The simplest scheme is to simply allow additional name/value pairs on the command line, where the name is the module name. In order to preserve maximum flexibility, the additional name/value pairs are prefixed by a keyword: "METADATA" which indicates that the remainder of the command arguments are to be interpretted as module metadata name/value pairs. This enables additional parameters to be added to these commands in the future.

The syntax of the RESTORE/RESTORE-ASKING command becomes:

RESTORE[-ASKING] key ttl serialized-value [REPLACE] [ABSTTL] [IDLETIME seconds] [FREQ frequency] [METADATA [name value]*]

If a name doesn't correspond to a loaded module, then the associated value is simply dropped. A counter of dropped metadata fields is retained. The first dropped metadata field is logged (just the unknown name) but subsequent drops are only counted in order to avoid spamming the log.

Generating machinery

In normal operations, the only the slot migration and RDB operations need to generate RESTORE and RESTORE-ASKING commands. All of these existing mechanisms would be enhanced to include the new syntax as needed, i.e., only add the METADATA keyword when at least one registered module has actual metadata to convey.

For development, debugging and operational purposes, it is useful to be able to generate module metadata that is directly returned to a client application. One mechanism would be to extend the DEBUG command with a suboption to return a list of the metadata name/value pairs.

A potentially more useful mechanism would be a command that would generate a fully populated RESTORE/RESTORE-ASKING command. Here fully populated means that the result would be an array of strings which is a syntactically correct RESTORE/RESTORE-ASKING command, containing all of the core metadata (TTL, etc.) as well as module metadata as needed. This could be a new command or some new keyword option on an existing command.

It is proposed that new keywords "RESTORE" and "RESTORE-ASKING" be added to the DUMP command. If one of these keywords is present, then instead of the output being a single serialized string. The output becomes an array of strings which are a fully populated RESTORE/RESTORE-ASKING command.