linvodb3

LinvoDB

LinvoDB is a Node.js/NW.js/Electron persistent DB with MongoDB / Mongoose-like features and interface.

Features:

• MongoDB-like query language
• Persistence built on LevelUP - you can pick back-end
• NW.js/Electron friendly - JS-only backend is level-js or Medea
• Performant - steady performance unaffected by DB size - queries are always indexed
• Auto-indexing
• Live queries - make the query, get constantly up-to-date results
• Schemas - built-in schema support
• Efficient Map / Reduce / Limit

Coming soon:

• Streaming cursors
• Distributed dataset

Relationship to NeDB

LinvoDB is based on NeDB, the most significant core change is that it uses LevelUP as a back-end, meaning it doesn't have to keep the whole dataset in memory. LinvoDB also can do a query entirely by indexes, meaning it doesn't have to scan the full database on a query.

In general:

• LinvoDB is better for large datasets (many objects, or large objects) because it doesn't keep the whole DB in memory and doesn't need to always scan it
• LinvoDB does the entire query through the indexes, NeDB scans the DB
• Both LinvoDB and NeDB play well with NW.js (node-webkit). LinvoDB can be initialized with the JS-only level-js back-end.
• NeDB is ultra-fast because the DB is in memory, LinvoDB's performance is comparible to MongoDB. LinvoDB is faster for large datasets.
• LinvoDB has live queries, map/reduce and schema support.
• Both LinvoDB and NeDB are unsuitable for huge datasets (big data)
• Combining NeDB's in-memory data and LinvoDB's full-indexed queries would yield even better performance. If you want to sacrifice memory for query performance, you can use LinvoDB with a backend that works like that or with LevelDB + increased LRU cache

Install, Initialize, pick backend

Install:

npm install linvodb3 level-js # For NW.js, using level-js
npm install linvodb3 leveldown # For pure node.js, using LevelDB

Initialize:

var LinvoDB = require("linvodb3");

// The following two lines are very important
// Initialize the default store to level-js - which is a JS-only store which will work without recompiling in NW.js / Electron
LinvoDB.defaults.store = { db: require("level-js") }; // Comment out to use LevelDB instead of level-js
// Set dbPath - this should be done explicitly and will be the dir where each model's store is saved
LinvoDB.dbPath = process.cwd(); 

var Doc = new LinvoDB("doc", { /* schema, can be empty */ })

Initialization, detailed:

var LinvoDB = require("linvodb3");
var modelName = "doc";
var schema = { }; // Non-strict always, can be left empty
var options = { };
// options.filename = "./test.db"; // Path to database - not necessary 
// options.store = { db: require("level-js") }; // Options passed to LevelUP constructor 
var Doc = new LinvoDB(modelName, schema, options); // New model; Doc is the constructor

LinvoDB.dbPath // default path where data files are stored for each model
LinvoDB.defaults // default options for every model

Insert / Save

The native types are String, Number, Boolean, Date and null. You can also use arrays and subdocuments (objects). If a field is undefined, it will not be saved.

If the document does not contain an _id field, one will be automatically generated (a 16-characters alphanumerical string). The _id of a document, once set, cannot be modified.

…

Querying

Use find to look for multiple documents matching you query, or findOne to look for one specific document. You can select documents based on field equality or use comparison operators ($lt, $lte, $gt, $gte, $in, $nin, $ne, $regex, $exists). You can also use logical operators $or, $and and $not. See below for the syntax.

…

Operators ($lt, $lte, $gt, $gte, $in, $nin, $ne, $exists, $regex)

The syntax is { field: { $op: value } } where $op is any comparison operator:

• $lt, $lte: less than, less than or equal
• $gt, $gte: greater than, greater than or equal
• $in: member of. value must be an array of values
• $ne, $nin: not equal, not a member of
• $exists: checks whether the document posses the property field. value should be true or false
• $regex: checks whether a string is matched by the regular expression. Contrary to MongoDB, the use of $options with $regex is not supported, because it doesn't give you more power than regex flags. Basic queries are more readable so only use the $regex operator when you need to use another operator with it (see example below)

…

Array fields

When a field in a document is an array the query is treated as a query on every element and there is a match if at least one element matches.

…

Logical operators $or, $and, $not

You can combine queries using logical operators:

• For $or and $and, the syntax is { $op: [query1, query2, ...] }.
• For $not, the syntax is { $not: query }

Planet.find({ $or: [{ planet: 'Earth' }, { planet: 'Mars' }] }, function (err, docs) {
  // docs contains Earth and Mars
});

Planet.find({ $not: { planet: 'Earth' } }, function (err, docs) {
  // docs contains Mars, Jupiter, Omicron Persei 8
});

// You can mix normal queries, comparison queries and logical operators
Planet.find({ $or: [{ planet: 'Earth' }, { planet: 'Mars' }], inhabited: true }, function (err, docs) {
  // docs contains Earth
});

Sorting and paginating

If you don't specify a callback to find, findOne or count, a Cursor object is returned. You can modify the cursor with sort, skip and limit and then execute it with exec(callback).

…

Counting documents

You can use count to count documents. It has the same syntax as find. For example:

// Count all planets in the solar system
Planet.count({ system: 'solar' }, function (err, count) {
  // count equals to 3
});

// Count all documents via cursor
Planet.find({}).count(function (err, count) {
  // count equals to 4
});

Map / Reduce / Filter / Aggregate

Besides the standard pagination and sorting Cursor methods, we have the filter, map and reduce modifiers. Before seeing the examples, you should know that you can combine any of these modifiers in any order/way and all will be executed. For example, you can run a regular query with .find and then run a reduce on it. No matter how you combine those modifiers, the order of execution is: query, filter, sort, limit/skip, map, reduce, aggregate.

The basic syntax is:

Cursor.map(function(val){ return val })

Cursor.reduce(function reducer(a,b), initial);

Cursor.filter(function(val) { return true /* or false*/ }); // truthy / falsy values accepted

Cursor.aggregate(function(res) { /* do something to the result of the query right before serving */ return res })

…

Live Queries

Once you have a Cursor object, returned by calling find without a callback, you can turn it into a live query, meaning the .res property will always be up-to-date results from the query. Of course, all modifiers, such as limit, skip, sort, map, reduce, filter and aggregate will still apply.

An event will be emitted when the result is updated - liveQueryUpdate on the model itself.

Seriously consider if live queries can be utilized in your application - if you need particular results continuously, using live queries is extremely efficient, since you don't have to re-read the database but results are kept up-to-date as you update the documents.

…

Angular Disclaimer

If you plan to use Live Queries with AngularJS and update scope on the liveQueryUpdated event please be careful. First, I recommend using $digest when possible instead of $apply (dirty-check only the current scope). Second, I recommend debouncing the event before running the $scope.$apply() event to avoid $apply being called many times because of heavy DB use at a moment.

Updating

Re-saving a document

doc.save() - you can use save on a document instance to re-save it, therefore updating it.

// Let's use the same example collection as in the "finding document" part
// { _id: 'id1', planet: 'Mars', system: 'solar', inhabited: false }
// { _id: 'id2', planet: 'Earth', system: 'solar', inhabited: true }
// { _id: 'id3', planet: 'Jupiter', system: 'solar', inhabited: false }
// { _id: 'id4', planet: 'Omicron Persia 8', system: 'futurama', inhabited: true }

Planet.findOne({ planet: 'Earth' }, function(err, doc) {
	doc.inhabited = false;
	doc.save(function(err) { /* we have updated the Earth doc */ }); 
});

Atomic updating

Doc.update(query, update, options, callback) will update all documents matching query according to the update rules:

• query is the same kind of finding query you use with find and findOne
• update specifies how the documents should be modified. It is either a new document or a set of modifiers (you cannot use both together, it doesn't make sense!)
  • A new document will replace the matched docs
  • The modifiers create the fields they need to modify if they don't exist, and you can apply them to subdocs. Available field modifiers are $set to change a field's value, $unset to delete a field and $inc to increment a field's value. To work on arrays, you have $push, $pop, $addToSet, $pull, and the special $each. See examples below for the syntax.
• options is an object with two possible parameters
  • multi (defaults to false) which allows the modification of several documents if set to true
  • upsert (defaults to false) if you want to insert a new document corresponding to the update rules if your query doesn't match anything. If your update is a simple object with no modifiers, it is the inserted document. In the other case, the query is stripped from all operator recursively, and the update is applied to it.
• callback (optional) signature: err, numReplaced, newDoc
  • numReplaced is the number of documents replaced
  • newDoc is the created document if the upsert mode was chosen and a document was inserted

Note: you can't change a document's _id.

…

Removing

Removing a document instance

// if you have the document instance at hand, you can just
Doc.findOne({ planet: 'Mars' }, function(err, doc) {
	doc.remove(function() {
		// done
	});	
});

Removing from the collection

Doc.remove(query, options, callback) will remove all documents matching query according to options

• query is the same as the ones used for finding and updating
• options only one option for now: multi which allows the removal of multiple documents if set to true. Default is false
• callback is optional, signature: err, numRemoved

…

Events

…

Schemas

You can define a schema for a model, allowing you to enforce certain properties to types (String, Number, Date), set defaults and also define properties with getter/setter. Since schema support is implemented deep in LinvoDB, you can query on fields which are getter/setter-based and rely that types/defaults are always going to be enforced.

NOTE: when constructing a model with a schema, please specify options object after the schema, otherwise schema will be treated as options: new LinvoDB(name, schema, options)

Schemas are defined as an object of specs for each property. The spec can have properties:

• type - the type to be enforced, can be String, Number, Date along with "string", "number", "date" alternative syntax. Can also be a RegExp instance in case you want to validate against that expressio