synapse

Open source

A transparent service discovery framework for connecting an SOA

2.1K stars0 likes1 views

About

A transparent service discovery framework for connecting an SOA

Synapse

Synapse is Airbnb's new system for service discovery. Synapse solves the problem of automated fail-over in the cloud, where failover via network re-configuration is impossible. The end result is the ability to connect internal services together in a scalable, fault-tolerant way.

Motivation

Synapse emerged from the need to maintain high-availability applications in the cloud. Traditional high-availability techniques, which involve using a CRM like pacemaker, do not work in environments where the end-user has no control over the networking. In an environment like Amazon's EC2, all of the available workarounds are suboptimal:

Round-robin DNS: Slow to converge, and doesn't work when applications cache DNS lookups (which is frequent)
Elastic IPs: slow to converge, limited in number, public-facing-only, which makes them less useful for internal services
ELB: ultimately uses DNS (see above), can't tune load balancing, have to launch a new one for every service * region, autoscaling doesn't happen fast enough

One solution to this problem is a discovery service, like Apache Zookeeper. However, Zookeeper and similar services have their own problems:

Service discovery is embedded in all of your apps; often, integration is not simple
The discovery layer itself is subject to failure
Requires additional servers/instances

Synapse solves these difficulties in a simple and fault-tolerant way.

How Synapse Works

Synapse typically runs on your application servers, often every machine. At the heart of Synapse are proven routing components like HAProxy or NGINX.

For every external service that your application talks to, we assign a synapse local port on localhost. Synapse creates a proxy from the local port to the service, and you reconfigure your application to talk to the proxy.

Under the hood, Synapse supports service_watchers for service discovery and config_generators for configuring local state (e.g. load balancer configs) based on that service discovery state.

Synapse supports service discovery with pluggable service_watchers which take care of signaling to the config_generators so that they can react and reconfigure to point at available servers on the fly.

We've included a number of default watchers, including ones that query zookeeper and ones using the AWS API. It is easy to write your own watchers for your use case, and install them as gems that extend Synapse's functionality. Check out the docs on creating a watcher if you're interested, and if you think that the service watcher would be generally useful feel free to pull request with a link to your watcher.

Synapse also has pluggable config_generators, which are responsible for reacting to service discovery changes and writing out appropriate config. Right now HAProxy, and local files are built in, but you can plug your own in easily.

Example Migration

Let's suppose your rails application depends on a Postgres database instance. The database.yaml file has the DB host and port hardcoded:

yaml

production:
  database: mydb
  host: mydb.example.com
  port: 5432

You would like to be able to fail over to a different database in case the original dies. Let's suppose your instance is running in AWS and you're using the tag 'proddb' set to 'true' to indicate the prod DB. You set up synapse to proxy the DB connection on localhost:3219 in the synapse.conf.yaml file. Add a hash under services that looks like this:

yaml

---
 services:
  proddb:
   default_servers:
    -
     name: "default-db"
     host: "mydb.example.com"
     port: 5432
   discovery:
    method: "awstag"
    tag_name: "proddb"
    tag_value: "true"
   haproxy:
    port: 3219
    server_options: "check inter 2000 rise 3 fall 2"
    frontend: mode tcp
    backend: mode tcp

And then change your database.yaml file to look like this:

yaml

production:
  database: mydb
  host: localhost
  port: 3219

Start up synapse. It will configure HAProxy with a proxy from localhost:3219 to your DB. It will attempt to find the DB using the AWS API; if that does not work, it will default to the DB given in default_servers. In the worst case, if AWS API is down and you need to change which DB your application talks to, simply edit the synapse.conf.json file, update the default_servers and restart synapse. HAProxy will be transparently reloaded, and your application will keep running without a hiccup.

Installation

To download and run the synapse binary, first install a version of ruby. Then, install synapse with:

…

This will download synapse and its dependencies into /opt/smartstack/synapse. You might wish to omit the --install-dir flag to use your system's default gem path, however this will require you to run gem install synapse with root permissions.

You can now run the synapse binary like:

bash

export GEM_PATH=/opt/smartstack/synapse
/opt/smartstack/synapse/bin/synapse --help

Don't forget to install HAProxy or NGINX or whatever proxy your config_generator is configuring.

Configuration

Synapse depends on a single config file in JSON format; it's usually called synapse.conf.json. The file has a services section that describes how services are discovered and configured, and then top level sections for every supported proxy or configuration section. For example, the default Synapse supports three sections:

services: lists the services you'd like to connect.
haproxy: specifies how to configure and interact with HAProxy.
file_output (optional): specifies where to write service state to on the filesystem.
[<your config generator here>] (optional): configuration for your custom configuration generators (e.g. nginx, vulcand, envoy, etc ..., w.e. you want).

If you have synapse config_generator plugins installed, you'll want a top level as well, e.g.:

nginx (optional): configuration for how to configure and interact with NGINX.

Configuring a Service

The services section is a hash, where the keys are the name of the service to be configured. The name is just a human-readable string; it will be used in logs and notifications. Each value in the services hash is also a hash, and must contain the following keys:

discovery: how synapse will discover hosts providing this service (see below)

The services hash should contain a section on how to configure each routing component you wish to use for this particular service. The current choices are haproxy but you can access others e.g. nginx through plugins. Note that if you give a routing component at the top level but not at the service level the default is typically to make that service available via that routing component, sans listening ports. If you wish to only configure a single component explicitly pass the disabled option to the relevant routing component. For example if you want to only configure HAProxy and not NGINX for a particular service, you would pass disabled to the nginx section of that service's watcher config.

haproxy: how will the haproxy section for this service be configured. If the corresponding watcher is defined to use zookeeper and the service publishes its haproxy configure on ZK, the haproxy hash can be filled/updated via data from the ZK node.
nginx: how will the nginx section for this service be configured. NOTE to use this you must have the synapse-nginx plugin installed.

The services hash may contain the following additional keys:

default_servers (default: []): the list of default servers providing this service; synapse uses these if no others can be discovered. See Listing Default Servers.
keep_default_servers (default: false): whether default servers should be added to discovered services
use_previous_backends (default: true): if at any time the registry drops all backends, use previous backends we already know about.

* `backend_port_override`: the port that discovered servers listen on; you should specify this if your discovery mechanism only discovers names or addresses (like the DNS watcher or the Ec2TagWatcher). If the discovery method discovers a port along with hostnames (like the zookeeper watcher) this option may be left out, but will be used in preference if given.

Service Discovery

We've included a number of watchers which provide service discovery. Put these into the discovery section of the service hash, with these options:

Base

The base watcher is useful in situations where you only want to use the servers in the default_servers list. It has the following options:

method: base
label_filters: optional list of filters to be applied to discovered service nodes

Filtering service nodes

Synapse can be configured to only return service nodes that match a label_filters predicate. If provided, label_filters should be an array of hashes which contain the following:

label: The name of the label for which the filter is applied
value: The comparison value
condition (one of ['equals', 'not-equals']): The type of filter condition to be applied.

Given a label_filters: [{ "label": "cluster", "value": "dev", "condition": "equals" }], this will return only service nodes that contain the label value { "cluster": "dev" }.

Zookeeper

This watcher retrieves a list of servers and also service config data from zookeeper. It takes the following mandatory arguments:

method: zookeeper
path: the zookeeper path where ephemeral nodes will be created for each available service server
hosts: the list of zookeeper servers to query
retry_policy: the retry policy (exponential back-off) when connecting to zookeeper servers and making network calls: max_attempts maximum number of all attempts (including original attempt), max_delay maxmimum delay in seconds for all attempts (including original attempt), base_interval retry interval in seconds for first retry attempt, max_interval maximum interval in seconds for each retry attempt. By default the retry policy is disabled.

The watcher assumes that each node under path represents a service server.

The watcher assumes that the data (if any) retrieved at znode path is a hash, where each key is named by a valid config_generator (e.g. haproxy) and the value is a hash that configs the generator. Alternatively, if a generator_config_path argument is specified, the watcher will attempt to read generator config from that znode instead. If generator_config_path has the value disabled, then generator config will not be read from zookeeper at all.

The following arguments are optional:

decode: A hash containing configuration for how to decode the data found in zookeeper.

Decoding service nodes

Synapse attempts to decode the data in each of these nodes using JSON and you can control how it is decoded with the decode argument. If provided, the decode hash should contain the following:

method (one of ['nerve', 'serverset'], default: 'nerve'): The kind of data to expect to find in zookeeper nodes
endpoint_name (default: nil): If using the serverset method, this controls which of the additionalEndpoints is chosen instead of the serviceEndpoint data. If not supplied the serverset method will use the host/port from the serviceEndpoint data.