[Feature]: Add a user-scoped write-to-retrieval verification command

Issue Origin

Observed or reproduced in a real environment

Problem Statement

A successful resource write does not prove that the resource is retrievable through the same identity and query path. This failure class is hard to detect because storage can accept a document while the search index remains stale or incomplete.

External reports show this operational problem:

• A RAG operator reported that an insert succeeded but the document was not searchable. The recommended check was a known sentinel through the same namespace, tenant, filter, and alias path as production queries: https://www.reddit.com/r/Rag/comments/1upo4j0/my_vector_db_said_the_insert_succeeded_but_the/
• A Mem0 operator found stored Valkey hashes while the index reported zero documents and search returned no results. The confirmed cause was a silent indexing defect: https://github.com/mem0ai/mem0/issues/3948
• Pinecone documents that acknowledged writes can become visible to queries later and provides a specific write-to-query freshness check: https://docs.pinecone.io/guides/index-data/check-data-freshness

OpenViking has server health, status, and configuration diagnostics. It does not have a user-scoped command that verifies one complete write-to-retrieval path. Operators must manually create a resource, wait for processing, inspect it, query it, and clean it up.

This request does not claim that current OpenViking loses accepted writes. An isolated check on current main completed the proposed path successfully. The gap is a repeatable operational diagnostic.

Proposed Solution

Add a top-level ov verify-retrieval command. It must use the configured non-root identity and perform one bounded check:

1. Create a unique Markdown sentinel at viking://~/resources/openviking-retrieval-check-<uuid>.md.
2. Use a generated marker as the content and query.
3. Write with mode=create, wait=true, and normal semantic and vector processing.
4. Verify the created file with stat.
5. Call find once with the exact sentinel URI as target_uri.
6. Require a resource result whose canonical URI equals the sentinel URI.
7. Remove the sentinel with wait=true.
8. Verify filesystem absence with stat.

The command must report these stages separately: write, stat, find, cleanup, and absence. Table and JSON output must include the overall result, stage results, elapsed time, and the retained URI when cleanup does not complete.

The command succeeds only when all required stages pass. An operational probe failure returns exit code 1. Existing parse and pre-execution validation keeps exit code 2. --sudo must be rejected before network access because root API keys cannot use tenant data APIs in API-key mode.

Cleanup must run after every successful creation attempt, including when stat or find fails. The command must use the configured client timeout. It must perform one retrieval attempt after the waited write and must not poll by default. Polling could hide an incorrect completion signal.

The command creates and queries only generated test content under the current user. It does not read existing user content. The generated content can reach the configured embedding or model provider through the normal resource-processing path.

This command verifies one operational visibility path. It does not measure general retrieval quality, recall, ranking, or immediate vector deletion. It does not change /health, /ready, server health semantics, retrieval semantics, or deletion guarantees.

Alternatives Considered

• Extend ov health or ov status: rejected because server component health does not prove tenant-scoped retrieval.
• Extend openviking-server doctor: rejected because it checks server configuration and providers, not the configured client identity and end-to-end data path.
• Document a manual command sequence: possible, but it cannot provide one structured result or reliable cleanup.
• Poll until the sentinel appears: rejected as the default because it can hide a false wait=true completion signal.
• Add a server-side freshness token: more invasive than required for this focused diagnostic.

Feature Area

CLI Tools

Use Case

Operators can run the command after deployment, identity, model, storage, or index configuration changes. Integration tests can also use the JSON result as a bounded smoke check before enabling an agent workload.

Example API (Optional)

ov verify-retrieval
ov --output json verify-retrieval

Additional Context

The current CLI already exposes the required client operations: write, stat, find, and rm. A credential-free isolated experiment on current main completed create, stat, exact-target find, cleanup, and absence verification in 5.37 seconds.

Searches of open and closed issues, discussions, pull requests, remote branches, and current source found no equivalent command or active implementation.

Contribution

• I am willing to contribute to implementing this feature