Feature: Built-in migration of legacy tables to True Data Reflection
Please confirm that the feature request does not already exist
- I confirm there is no existing issue for this feature request.
Use case
We have a PostgreSQL backed NocoDB instance containing tables that were originally created through the NocoDB UI before v0.257.
After upgrading NocoDB, True Data Reflection is enabled for newly created tables. As a result, the same NocoDB instance/base can now contain a mixture of:
- Legacy tables created under the pre-v0.257 storage/schema model.
- New tables created using the True Data Reflection model.
We need a way to migrate existing legacy NocoDB managed tables into the True Data Reflection model so that the entire PostgreSQL database is consistent. This is essential for any data flows out of nocodb. Currently, there does not appear to be a supported migration path for converting these existing tables or any documentation support.
Exporting the legacy tables and recreating/importing them through the NocoDB UI is not practical for us because some of the tables contain too much data to reasonably migrate through the UI, and it would also require us to reconfigure user access permissions for each base, re-link relational rows across tables, and potentially re-upload media stored in attachment columns, making the process error-prone and operationally expensive.
This leaves upgraded installations in an awkward state where legacy and True Data Reflection tables coexist indefinitely, despite both having originally been created and managed by NocoDB.
The desired end state is: Existing NocoDB-managed tables created before True Data Reflection should be migratable to the current True Data Reflection architecture without requiring a UI-based export/import of all of their data.
Suggested solution
Provide a built-in migration mechanism for converting legacy NocoDB managed tables to the True Data Reflection model.
This could be implemented as:
- A migration command/CLI intended for administrators.
- An administrative migration operation within NocoDB perhaps enforced through an environment variable Eg. NC_ENFORCE_PG_DATA_REFLECTION = True
- A migration script officially provided and supported by NocoDB.
- An automated migration that can be explicitly triggered for a legacy base, perhaps via the admin page.
Ideally, the migration would:
- Detect tables using the legacy storage model.
- Create the appropriate PostgreSQL schema/tables required by True Data Reflection.
- Migrate or restructure the underlying data directly at the database level rather than passing every record through the NocoDB UI/API.
- Preserve NocoDB metadata and configuration associated with the table including column types, relations, views, access control, formulas, etc.
- Validate that the migrated table and its metadata are consistent before switching NocoDB to the new representation.
- Ideally, provide clear errors/rollback guidance if a table cannot be migrated automatically.
For large installations, it would also be useful if this could operate on one base at a time, allowing administrators to migrate and validate incrementally.
The important requirement is that the migration should be performed efficiently at the PostgreSQL/database level and should not require exporting and re-importing potentially very large datasets through the NocoDB UI.
Additional context
This issue arises specifically from upgrading an existing NocoDB installation across the introduction of True Data Reflection.
Our tables were not externally created PostgreSQL tables. They were originally created and managed through the NocoDB UI using an older version of NocoDB (as early as 0.204.5).
After upgrading, newly created tables use True Data Reflection while the existing tables retain the legacy structure. We therefore have two generations of NocoDB managed tables in the same PostgreSQL instance.
I looked for documentation covering migration of these legacy tables but could not find a supported procedure. The NocoDB AI documentation assistant also indicated that there does not appear to be an explicit migration tool/path and suggested exporting and recreating the tables.
For small tables, export/import may be an acceptable workaround. For large tables or more complicated base setups, however, moving all records through the UI is impractical and potentially introduces unnecessary downtime and migration risk.
A supported migration path would be particularly valuable for long-running NocoDB installations because these are the installations, like ours, are most likely to have significant amounts of data created under the legacy architecture.
This would allow upgraded installations to eventually converge completely on True Data Reflection instead of retaining legacy tables indefinitely.
As you can tell, we are a big fan of True Data Reflection, it has made things quite a bit easier for us since the upgrade. Thanks for your continued work on the project!
Source: nocodb/nocodb