[Tracking] Monitoring

• #2208
• #2209
• #2210
• #2217
• #2021

shelved:

• #2207
• #2027

Old list

---

Monitoring Individual Services

We should be tracking the following services with the following conditions. Services are ranked in order of priority.

• Hubble
  • Should be indexing all chains.
  • All chains should be up to date, meaning we've indexed one of the top 4 blocks.
• Postgres
  • Should be operational
• Hasura
  • Should be operational
• Website app.union.build Should be online
• Website union.build Should be online
• Voyager
  • Should be relaying all packets
• Galois
  • Should be generating proofs within a reasonable time

Low priority/ skip for now:

• Validators/RPCs of the core team
  • Should be proposing blocks
  • Should respond to RPC queries

For all of these services, we should have a datadog agent running on them. We should also be testing all important vitals (CPU/RAM/Disk/Network) We should set up pagerduty uch that if any of these go down, we get a call

We should aggregate all of this on our datadog dashboard

Sentinel

We should also create a service that every half hour, sends a packet between all pairs of chains we have, and see if they arrive. Ideally this is nicely interspaced.

So if we have connected chains A, B, C, D

We need to send

A --> B
B --> A
A --> C
C --> A
B --> C
C --> B
...etc

Do this such that every transfer A --> B occurs every half-hour, and then evenly space out every X --> Y pair (rather than doing all of them at the same time).

This service should be written in Rust with a NixOS Module and be deployed to a dedicated machine.

The results of this should also be included in the before mentioned datadog dashboard.