@orama/plugin-data-persistence seems to require Node.js transform streams
Describe the bug
I am using the restore function to load a previously persisted index and do client-side full-text search on the browser. When I am bundling the frontend using Vite I am presented with the following error:
Uncaught TypeError: Class extends value undefined is not a constructor or null
I haven't done a super deep investigation but my current understanding on why this is happening is the following:
- The
@orama/plugin-data-persistencemodule depends on thedpackmodule to supportdpackserialization/deserialization. (this happens regardless of whetherdpackis being used... I am only using JSON in my code) dpackrequires Transform streams from Node.js.- Vite, by default, does not polyfill Node.js modules and therefore the Transform stream is missing at runtime
I found a couple of workarounds for this problem so far:
Workaround 1: polyfill Node.js streams
Install vite-plugin-node-polyfills and then, In your Vite config:
// vite.config.ts
import { defineConfig } from 'vite'
import { nodePolyfills } from 'vite-plugin-node-polyfills'
// https://vite.dev/config/
export default defineConfig({
plugins: [nodePolyfills({ include: ['stream', 'util'] })]
})This will make the Node.js stream available at runtime. The drawback of this approach is that your bundle size will grow significantly (~500kb!).
Workaround 2: build your own JSON restore
If don't intend to use dpack and your persisted index is in JSON, you can "manually" restore it as follows (loosely extrapolated from this file):
import {
type Orama,
type RawData,
create,
load,
search,
} from '@orama/orama'
import type { Schema } from './types' // your index schema type
async function loadDbFromJson(data: RawData): Promise<Orama<Schema>> {
const db = await create({
schema: {
__placeholder: 'string',
},
})
await load(db, data)
return db as unknown as Orama<Schema>
}
// load your index data somehow (e.g. using fetch)
const resp = await fetch('https://.../_search.json.gz', { // your URL here
headers: {
'accept-encoding': 'gzip',
accept: 'application/json; charset=utf-8',
},
})
const data = await resp.json()
const db = await loadDbFromJson(data)
// use db for search
const results = await search(db, ...)This approach keeps your bundle even smaller because you don't need to use the @orama/plugin-data-persistence client-side! But it's more code and complexity to maintain for the developer...
To Reproduce
- Create an Orama index
- Populate it with some data
- Persist it using the
@orama/plugin-data-persistencepersistence plugin - restore the index in a client application bundled with Vite (default config)
This sample repo at this commit can be used as a starting point. Most of the restore business logic can be added in /src/TypeAhead.tsx
Expected behavior
- It would be nice if the internal implementation would not require the Node.js transform stream, so the plugin would work out of the box with a default Vite configuration
- It would be even better if
dpackwas optional...
On the second point, this is possibly a breaking change but I am thinking that it is a bit of a waste to require dpack even when is not going to be used.
An alternative approach would be to make that an optional dependency and to force the user to pass it explicitly to the plugin when needed. Something like:
import { restore } from "@orama/plugin-data-persistence";
import { dpackDeserializer } from "@orama/plugin-data-persistence/dpack"; // this could be a sub-module or another independent module that needs to be explicitly installed
const newInstance = await restore(JSONIndex, {
deserializer: new dpackDeserializer() // could be 'json' by default
});Environment Info
OS: Mac Os
Node: v20.18.1
Orama: 3.0.5
Orama persistence plugin: 3.0.5Affected areas
Initialization
Additional context
No response
Source: oramasearch/orama