swarm

This code is historical. See RDX C or RDX Go for the new generation of the technology. No JavaScript impl yet. Maybe you can help, please use Issues.

Swarm 2.0 real-time sync

Swarm.js is a JavaScript client for the Swarm database. Swarm is like "git for data" except it's real-time and never has a merge conflict. Swarm is based on Replicated Object Notation (RON), a distributed live data format. In turn, RON is based on Conflict-free Replicated Data Types (CRDTs), a new math apparatus for distributed data.

Like git, Swarm gives you a replica of your data which is as good as any other replica. You can read, you can write, you can see it updated by others. Concurrent changes merge automatically. Offline replicas re-sync on re-connection. All data is cached.

API-wise, Swarm is an object graph, although the API may depend on a particular mapper. Still, any API is a "view" of the system. The ground truth is RON.

Swarm picks its trade-offs in a rather opinionated way. It fits the world of abundant storage and unreliable wireless connections.

Swarm.js is isomorphic and is perfect for implementing synchronization, collaboration or continuity features in Web and mobile apps. Swarm works for:

• vanilla client-server sync,
• real-time collaboration,
• continuity,
• offline work,
• and other sync-related scenarios.

Unlike Firebase and others, you may run your own swarm server. If you understand how CRDT works, then you may implement your own data types. Free-as-in-freedom (MIT).

Table of contents:

• Intro
• Demos
• Setup
• CRDT implementations
• API
• GraphQL
• Using with React
• FAQ
• TODO
• Contributing
• Contacts

Demos

See our demos:

• "Mice" application - demo, source code.

  Preview

• Chat application - demos for webkit-backed browsers, iOS, Android and the source code

  Preview

• Todo application, based on the world-famous TodoMVC - demo and source

  Preview

Every app perfectly works offline.

Setup

A basic Swarm server implementation is available as a docker image. First, please make sure that you have the docker installed. Then, run the container:

$ docker run -d --name swarmdb -p 31415:31415 -v `pwd`:/var/lib/swarm olebedev/swarmdb

Once a Swarm server is listening incoming connections on ws://0.0.0.0:31415, we can initiate connections. But let's suppose that we can't talk RON over raw WebSocket.

Then, let's setup a JavaScript client project:

$ mkdir ./myapp
$ cd ./myapp
$ yarn add @swarm/db graphql-tag

Now we can initialize a client instance and connect it to the running server.

…

CRDT implementations

• LWW, a last-write-wins replicated data type that may host a variety of user-land data types, like: a dictionary, a simple 1D array (no splice, no index shifts). This LWW employs client-side logical timestamps to decide which write wins, on a field-by-field basis. That is similar to e.g. Cassandra LWW.

• Set, fully commutative, ordered, with tombstones set. You can either add or remove an atom.

• RGA, a replicated growable array.

• Counter, a positive-negative counter.

API

constructor(options: Options): SwarmDB

Creates a new Swarm instance. Fetches the metadata from the server, saves it locally, sets the replica up (async, see ensure() method).

Parameters:

• options - an object with properties:

  • storage - a storage interface implementation, required. There are tree implementations: in-memory, localStorage(web) and AsyncStorage(ReactNative).
  • upstream - URL string, e.g. 'wss://the.host.com', required
  • db - an object, required
    • name - A database name, required
    • auth - a JWT, makes sense if auth mode enabled at the server, ignored otherwise

Example:

const swarm = new SwarmDB({
  storage: new InMemory(),
  upstream: 'wss://example.com',
  db: { name: 'default' }
});

execute(request: Request, cbk<T>: (Response<T>) => void ): Promise<{ ok: boolean, off: () => void }>

Fetch data by a GraphQL query(see below).

Parameters:

• request - an object with properties:
  • query - GraphQL AST
  • variables - arguments for GraphQL statement
• cbk - a callback function which accepts Response<T> - object with properties:
  • data - T
  • off - a function which cancels a subscription, may be undefined for mutations and static queries
  • error - an error if any

Example:

const query = gql`
subscription GetChats($from: Int!, $to: Int!) { 
  chats @node(id: "chats") { 
    length
    list: id @slice(begin: $from, end: $to) {
      title
    }
  } 
}`

const variables = { from: 0, to: 20 }
swarm.execute({ query, variables }, ({ data, error, off }) => {
  // handle updates 
  // or stop receiving updates via call `off()`
})

close(): Promise<void>

Returns a Promise which will be resolved when the replica is closed(websocket closed and the state is written to the storage).

ensure(): Promise<void>

Returns a Promise which will be resolved when the replica is initialized and ready to use.

open(): void

(Re)Opens websocket connection to a database server.

uuid(): UUID

Returns a new timestamp-based UUID.

uuid.local()

Use local UUIDs for nodes you don't want to send to the server. Example:

const dashboard = swarm.uuid().local() // a local uuid works only for the current instance. 

Note that a local object can reference a shared object, but not the other way round. Please take a look at todo to see at fully working example of using local UUIDs for local state management.

GraphQL

Swarm has a GraphQL API on the client side. Server-client communication employs RON (state, ops, patches).

Swarm offers GraphQL as a schema-less interface to fetch and modify your data, with basic mutations operations for specific CRDTs, and with several additional directives(see below). GraphQL is used to describe(declare) the shape of the requested data tree.

Key features:

• data reactivity (full or partial)
• (sub)documents/(sub)collections as first-class citizens
• every object has a globally-unique Swarm UUID (see teh @node directive)

Basic primitives and conventions

UUID - is a Swarm UUID or a string representation of an UUID.

scalar UUID

Atom - a scalar (a union of scalar types).

union Atom = String | Int | Float | Boolean | UUID

Node - is an interface of CRDT which presented as a key/value object where a value is a union of Atom | Node | [Node]. Internally, each objects JS prototype regardless its type has id, type and version fields. Additionally, the set type has length field, and lww type can also has length in case if all the user-land field keys are numbers(say, index of array). There are no native JavaScript arrays by default, they can be produced by directives(see below).

interface Node {
  id: String!
  type: String!
  version: String!
  length: Int # only for array-like objects
  # ... various of keys and values
}

Payload - is a flat key/value object. Where keys are strings and values are Atoms.

interface Payload {
  # ... various of keys and values
}

Queries & Subscriptions

Both of these two root fields serve to declare the tree you intend to get as a result of the execution. There is nothing special except that you don't need to define schema.

Example. Let's subscribe to the last 100 users from the users collection.

const query = gql`
subscription Users($from: Int = 0, $to: Int = 100) { 
  users @node(id: "users") @slice(begin: $from, end: $to) {
    username
    picture
  } 
}`

swarm.execute({ query }, ({ data, error, off }) => {
  // handle updates 
  console.log(data.users.list) // prints array of users
})

The difference between subscription and query is a data reactivity, see more details here.

Mutations

Mutations are strictly defined and depend on CRDTs which are implemented. All of them listed below.

type Mutation {
  # for LWW
  set(id: UUID!, payload: Payload!): Bool
  
  # for Set
  add(id: UUID!, value: Atom): Bool
  remove(id: UUID!, value: Atom): Bool
}

Note that an error will be raised in case of type mismatch.

Example. Suppose we have to add subdocument into user profile. For that we need to have an identifier(UUID) for new node, then we can put it into the parent document.

…

Directives

Here are the directives defined in Swarm GraphQL runtime, in addition to default ones.

@node

directive @node(id: UUID!) on FIELD

Used to define which node of Swarm graph should be unwrapped. Can be missed if the field already contains a UUID itself(for nested objects).

directive @node(id: UUID!) on FIELD
# example
subscription {
  user @node(id: "X8Kq%~github") {
    version
    username
  }
  dashboard @node(id: "dash%~local") { # fetch locally presented node
    screen
    tasks @node(id: tasks) {
      length
      list: id @slice(begin: 0) {
        id
        title
        progress
      }
    }
  }
}

@live & @static

directive @static on FIELD
directive @live on FIELD

Both add an ability to define a partial reactivity. All the nodes in the subscription root field are reactive along with all the nodes in the query root field are static. Subscription and query root field are interchangeable. So, it's up to developer which root field will be used, depends on how many nodes in a tree should be static or reactive.

Once a static node was fetched from the server and cached, it will be returned from the cache without network activity.

# fetch user once and cache for further queries
# and install subscription to notificatoins
subscription {
  user @node(id: "X8Kq%~github") @static {
    version
    username
    notifications @slice(begin: 0) {
      title
      read
    }
  }
}

# and the same example with query root field
query {
  user @node(id: "X8Kq%~github") {
    version
    username
    notifications @slice(begin: 0) @live {
      title
      read
    }
  }
}

@weak

directive @weak on FIELD

Adds more control to data flow manage