Fast, recursive, optimized, URL-Safe serialization for any Ruby object
Fast, recursive, optimized, URL-Safe serialization for any Ruby object
Universal ID leverages both MessagePack and Brotli (a combo built for speed and best-in-class data compression). When combined, these libraries are up to 30% faster and within 2-5% compression rates compared to Protobuf. ↗
Universal ID introduces a paradigm shift that enables straightforward simple solutions ↗ for a variety of complex problem domains.
[!TIP] All the code examples below can be tested on your local machine. Just clone the repo (↑or use Gitpod above↑) and run
bin/consoleto begin exploring. Don't forget to executebundlefirst to ensure all dependencies are up to date. Happy coding!
Universal ID introduces a new URI defintion that can recursively serialize any Ruby object into an URL-safe string which can be safely transported via HTTP.
[!NOTE] The payload is optimized to be as small as possible... especially notable with large objects.
The best part: The API is simple.
data = :ANY_OBJECT_YOU_CAN_IMAGINE
uid = URI::UID.build(data)
#
uid.payload
"Cw6AxxoAQU5ZX09CSkVDVF9ZT1VfQ0FOX0lNQUdJTkUD"
uid.fingerprint
"CwWAkccHf6ZTeW1ib2wD"
uri = uid.to_s
"uid://universalid/Cw6AxxoAQU5ZX09CSkVDVF9ZT1VfQ0FOX0lNQUdJTkUD#CwWAkccHf6ZTeW1ib2wD"
parsed = URI::UID.parse(uri)
#
parsed.decode
:ANY_OBJECT_YOU_CAN_IMAGINE
# it's also possible to parse the payload by itself
parsed = URI::UID.from_payload(uid.payload)
#
parsed.decode
:ANY_OBJECT_YOU_CAN_IMAGINEUniversal ID supports most native Ruby primitives:
NilClassBigDecimalComplexDateDateTimeFalseClassFloatIntegerRangeRationalRegexpStringSymbolTimeTrueClassYou can use Universal ID to serialize individual primitives, but this actually serves as the foundation for more advanced use-cases.
uri = URI::UID.build(:demo).to_s
#=> "uid://universalid/iwKA1gBkZW1vAw#CwWAkccHf6ZTeW1ib2wD"
uid = URI::UID.parse(uri)
#=> #
uid.decode
#=> :demoComposite (or complex, compound, etc.) datatype support is where things start to get interesting. Universal ID supports the following native Ruby composite datatypes:
ArrayHashOpenStructSetStructarray = [1, 2, 3, [:a, :b, :c, [true]]]
uri = URI::UID.build(array).to_s
#=> "uid://universalid/iweAlAECA5TUAGHUAGLUAGORwwM#iwSAkccGf6VBcnJheQM"
uid = URI::UID.parse(uri)
#=> #
uid.decode
#=> [1, 2, 3, [:a, :b, :c, [true]]]
uid.decode == array
#=> truehash = {a: 1, b: 2, c: 3, array: [1, 2, 3, [:a, :b, :c, [true]]]}
uri = URI::UID.build(hash).to_s
#=> "uid://universalid/CxKAhNQAYQHUAGIC1ABjA8cFAGFycmF5lAECA5TUAGHUAGLUAGORwwM#CwS..."
uid = URI::UID.parse(uri)
#=> #
uid.decode
#=> {:a=>1, :b=>2, :c=>3, :array=>[1, 2, 3, [:a, :b, :c, [true]]]}
uid.decode == hash
#=> trueBook = Struct.new(:title, :author, :isbn, :published_year)
book = Book.new("The Great Gatsby", "F. Scott Fitzgerald", "9780743273565", 1925)
uri = URI::UID.build(book).to_s
#=> "uid://universalid/G2YAoGTomv9tT1ilLRgVC9vIpmuBo-k84FZ0G8-siFMBNsbW0dpBE0Tnm96..."
uid = URI::UID.parse(uri)
#=> #
uid.decode
#=> #
uid.decode == book
#=> trueThe following extension datatypes ship with Universal ID:
ActiveRecord::BaseActiveRecord::RelationActiveSupport::Cache::EntryActiveSupport::Cache::StoreActiveSupport::TimeWithZoneGlobalIDSignedGlobalID[!NOTE] Extensions are autoloaded whenever the related datatype is detected.
[!IMPORTANT] Why Universal ID with ActiveRecord? ActiveRecord already has GlobalID, a robust library for serializing individual models. Universal ID covers a much wider range of use cases.
Here are a few reasons you may want to consider Universal ID with ActiveRecord.
New Records: Universal ID can serialize models that haven't been saved to the database yet.
Changesets: Universal ID can serialize ActiveRecord models with unsaved changes, ensuring that even transient states are captured.
Associations: Universal ID goes beyond single models. It can include associated records, even those with unsaved changes, creating a comprehensive snapshot of complex record states.
Copying/Cloning: Universal ID supports making copies of records (including associations), making it ideal for duplicating complex datasets.
More Control: Universal ID gives you control over the serialization process. You can choose which columns to include/exclude, allowing for tailored, optimized payloads to fit your needs.
Queries/Relations: Universal ID also supports ActiveRecord::Relations, enabling the serialization of complex database queries and scopes.
In summary, while GlobalID excels in its specific use case, Universal ID offers more power for use-cases that involve unsaved records, complex associations, data cloning, and database queries.
…Universal ID is extensible, enabling you to register your own datatypes with custom serialization rules. Simply convert the required data to a Ruby primitive or composite value.
…Universal ID supports a small, but powerful, set of options used to "prepack" the object before it's packed with MessagePack. These options instruct Universal ID on how to prepare the object for serialization.
…Options can be applied whenever creating a UID.
hash = { a: 1, b: 2, c: 3 }
uri = URI::UID.build(hash, exclude: [:b]).to_s
#=> "uid://universalid/CwSAgtQAYQHUAGMDAw#CwSAkccFf6RIYXNoAw"
uid = URI::UID.parse(uri)
#=> #
uid.decode
#=> {:a=>1, :c=>3}[!NOTE] Options can be passed in structured or flat format.
It's also possible to register frequently used options.
# app/config/changed.yml
prepack:
include_blank: false
database:
include_changes: true
include_descendants: true
descendant_depth: 2UniversalID::Settings.register :changed, File.expand_path("app/config/changed.yml", __dir__)
uid = URI::UID.build(record, UniversalID::Settings[:changed])Each UID is fingerprinted as part of the serialization process.
Fingerprints are comprised of the following components:
Class (Class) - The encoded object's classTimestamp (Time) - The mtime (UTC) of the file that defined the object's classFingerprints provide a simple mechanism to help manage data format versions... minimizing the need for custom versioning solutions.
Whenever the class definition changes, the mtime updates, resulting in a different fingerprint.
This is especially useful in scenarios where the data format evolves over time, such as in long-lived applications.
uid = URI::UID.build(campaign)
uid.fingerprint
#=> "CwuAkscJf6hDYW1wYWlnbtf_ReuZnGWeG5MD"
uid.fingerprint(decode: true)
#=> [Campaign(id: integer, ...), ][!NOTE] The timestamp or
mtimeis determined the moment a UID is created.
[!TIP] Fingerprints can help you maintain consistency and reliability when working with serialized data over time. While fingerpint creation is automatic and implicit, usage is optional... ready whenever you need it.
Make a copy of an ActiveRecord model (with loaded associations).
…[!TIP] If you don't need a URL-Safe UID, you can use
UniversalID::Packerto speed things up a bit.
packed = UniversalID::Packer.pack(campaign, options)
copy = UniversalID::Packer.unpack(packed)
copy.saveUniversal ID also supports ActiveRecord relations/scopes. You can easily serialize complex queries into a portable and sharable format.
relation = Campaign.joins(:emails).where("emails.subject LIKE ?", "Flash Sale%")
uri = URI::UID.build(relation).to_s
#=> "uid://universalid/G90EQCwLeEP1oQtHFksrdN5YS4ju5TryFZwBJgh2toqS3SKEVSl1FoNtZjI..."
uid = URI::UID.parse(encoded)
#=> #
decoded = uid.decode
# introspection
decoded == relation #=> true
decoded.is_a? ActiveRecord::Relation #=> true
decoded.loaded? #=> false
# run the query
campaigns = decoded.load[!NOTE] Universal ID clears cached data within the relation before encoding. This minimizes payload size while preserving the integrity of the underlying query.
Features like signing (to prevent tampering), purpose, and expiration are provided by SignedGlobalIDs.
These features (and more) will eventually be added to Universal ID, but until then...
simply convert your UID to a SignedGlobalID to add these features to any Universal ID.
data = OpenStruct.new(name: "Demo", value: "Example")
sgid = URI::UID.build(data).to_sgid_param(for: "purpose", expires_in: 1.hour)
#=> "eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaEpJZ0plQTJkcFpEb3ZMM1Z1YVhabGNuTmhiQzFwWkM5V..."
uid = URI::UID.from_sgid(sgid, for: "purpose")
#=> #
decoded = uid.decode
#=> #
# a mismatched purpose returns nil... as expected
URI::UID.from_sgid(sgid, for: "mismatch")
#=> nilProudly sponsored by
The gem is available as open source under the terms of the MIT License.
No open issues yet, or sync has not completed.