Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
D

dsync

> 编程语言
Open source

A distributed sync package.

392 stars0 likes3 views
WebsiteGitHub

About

A distributed sync package.

This project has moved to https://github.com/minio/minio/tree/master/pkg/dsync

dsync

A distributed locking and syncing package for Go.

Introduction

dsync is a package for doing distributed locks over a network of n nodes. It is designed with simplicity in mind and hence offers limited scalability (n NOTE: Previously if you were using dsync.Init([]NetLocker, nodeIndex)to initialize dsync has been changed todsync.New([]NetLocker, nodeIndex)which returns a*Dsyncobject to be used in every instance ofNewDRWMutex("test", *Dsync)`

Exclusive lock

Here is a simple example showing how to protect a single resource (drop-in replacement for sync.Mutex):

…

which gives the following output:

2016/09/02 14:50:00 first lock granted
2016/09/02 14:50:00 about to lock same resource again...
2016/09/02 14:50:05 first lock unlocked
2016/09/02 14:50:05 second lock granted

Read locks

DRWMutex also supports multiple simultaneous read locks as shown below (analogous to sync.RWMutex)

…

which gives the following output:

2016/09/02 15:05:20 1st read lock acquired, waiting...
2016/09/02 15:05:20 2nd read lock acquired, waiting...
2016/09/02 15:05:20 Trying to acquire write lock, waiting...
2016/09/02 15:05:22 1st read lock released, waiting...
2016/09/02 15:05:24 2nd read lock released, waiting...
2016/09/02 15:05:24 Write lock acquired, waiting...

Basic architecture

Lock process

The basic steps in the lock process are as follows:

  • broadcast lock message to all n nodes
  • collect all responses within certain time-out window
    • if quorum met (minimally n/2 + 1 responded positively) then grant lock
    • otherwise release all underlying locks and try again after a (semi-)random delay
  • release any locks that (still) came in after time time-out window

Unlock process

The unlock process is really simple:

  • broadcast unlock message to all nodes that granted lock
  • if a destination is not available, retry with gradually longer back-off window to still deliver
  • ignore the 'result' (cover for cases where destination node has gone down and came back up)

Dealing with Stale Locks

A 'stale' lock is a lock that is left at a node while the client that originally acquired the client either:

  • never released the lock (due to eg a crash) or
  • is disconnected from the network and henceforth not able to deliver the unlock message.

Too many stale locks can prevent a new lock on a resource from being acquired, that is, if the sum of the stale locks and the number of down nodes is greater than n/2 - 1. In dsync a recovery mechanism is implemented to remove stale locks (see here for the details).

Known deficiencies

Known deficiencies can be divided into two categories, namely a) more than one write lock granted and b) lock not becoming available anymore.

More than one write lock

So far we have identified one case during which this can happen (example for 8 node system):

  • 3 nodes are down (say 6, 7, and 8)
  • node 1 acquires a lock on "test" (nodes 1 through to 5 giving quorum)
  • node 4 and 5 crash (dropping the lock)
  • nodes 4 through to 8 restart
  • node 4 acquires a lock on "test" (nodes 4 through to 8 giving quorum)

Now we have two concurrent locks on the same resource name which violates the core requirement. Note that if just a single server out of 4 or 5 crashes that we are still fine because the second lock cannot acquire quorum.

This table summarizes the conditions for different configurations during which this can happen:

Nodes Down nodes Crashed nodes Total nodes
4 1 2 3
8 3 2 5
12 5 2 7
16 7 2 9

(for more info see testMultipleServersOverQuorumDownDuringLockKnownError in chaos.go)

Lock not available anymore

This would be due to too many stale locks and/or too many servers down (total over n/2 - 1). The following table shows the maximum toterable number for different node sizes:

Nodes Max tolerable
4 1
8 3
12 5
16 7

If you see any other short comings, we would be interested in hearing about them.

Tackled issues

  • When two nodes want to acquire the same lock at precisely the same time, it is possible for both to just acquire n/2 locks and there is no majority winner. Both will fail back to their clients and will retry later after a semi-randomized delay.

Server side logic

On the server side just the following logic needs to be added (barring some extra error checking):

…

If you also want RLock()/RUnlock() functionality, then add this as well:

…

See dsync-server_test.go for a full implementation.

Sub projects

  • See performance directory for performance measurements
  • See chaos directory for some edge cases

Testing

The full test code (including benchmarks) from sync/rwmutex_test.go is used for testing purposes.

Extensions / Other use cases

Robustness vs Performance

It is possible to trade some level of robustness with overall performance by not contacting each node for every Lock()/Unlock() cycle. In the normal case (example for n = 16 nodes) a total of 32 RPC messages is sent and the lock is granted if at least a quorum of n/2 + 1 nodes respond positively. When all nodes are functioning normally this would mean n = 16 positive responses and, in fact, n/2 - 1 = 7 responses over the (minimum) quorum of n/2 + 1 = 9. So you could say that this is some overkill, meaning that even if 6 nodes are down you still have an extra node over the quorum.

For this case it is possible to reduce the number of nodes to be contacted to for example 12. Instead of 32 RPC messages now 24 message will be sent which is 25% less. As the performance is mostly depending on the number of RPC messages sent, the total locks/second handled by all nodes would increase by 33% (given the same CPU load).

You do however want to make sure that you have some sort of 'random' selection of which 12 out of the 16 nodes will participate in every lock. See here for some sample code that could help with this.

Scale beyond 32 nodes?

Building on the previous example and depending on how resilient you want to be for outages of nodes, you can also go the other way, namely to increase the total number of nodes while keeping the number of nodes contacted per lock the same.

For instance you could imagine a system of 64 nodes where only a quorum majority of 17 would be needed out of 28 nodes. Again this requires some sort of pseudo-random 'deterministic' selection of 28 nodes out of the total of 64 servers (same example as above).

Other techniques

We are well aware that there are more sophisticated systems such as zookeeper, raft, etc. However we found that for our limited use case this was adding too much complexity. So if dsync does not meet your requirements than you are probably better off using one of those systems.

Other links that you may find interesting:

  • Distributed locks with Redis
  • Based on the above: Redis-based distributed mutual exclusion lock implementation for Go

Performance of net/rpc vs grpc

We did an analysis of the performance of net/rpc vs grpc, see here, so we'll stick with net/rpc for now.

License

Released under the Apache License v2.0. You can find the complete text in the file LICENSE.

Contributing

Contributions are welcome, please send PRs for any enhancements.

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Gocloud-storagedistributed-locksdistributed-systemslocking

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言