Packet Loss Monitoring?

[clone1018]

Should we be monitoring packet loss between our Ingest / Edge servers? Should we be tracking NACK stats in prometheus as well?

[LukeHandle]

Interesting idea of whether use of the blackbox_exporter on every node ICMP pinging (and/or FTL or HTTP check?) every other node. The issue is really how would be helpfully use this data. We would have end to end/both ways tracking of everything (great!). But how to helpfully visualise..!

If a node or region had an issue, we would see all/most pings to it see latency/loss, and potentially pings out would never be recorded/captured. Maybe some sort of aggregation based on a target label and tracking deviations from averages?

[LukeHandle]

https://github.com/benjojo/sping might be another interesting tool to further investigate. Handily exposes /metrics as well.

[LukeHandle]

I had seen https://grafana.com/grafana/plugins/grafana-synthetic-monitoring-app/ before, but had assumed it was Grafana Cloud only.

In fact, it's all open source (plugin is AGPLv3, client is Apache 2), though I the reality is that it's just blackbox_exporter + logs, and pretty graphs. We could just use blackbox_exporter and the graphs as inspiration:

And it's the graphs that give at least some idea how we could use the data. We would have a dashboard like that ^^ with var selectors for each instance.

Nice world map would highlight issues between regions.

Would we opt for every edge tracking every ingress + every ingress tracking every edge? That's the internal flow of data, so most relevant.

I expect it'd be cheap to add more locations on diff hosting providers if we wanted an external perspective.

==

Thoughts @clone1018

[clone1018]

Would we opt for every edge tracking every ingress + every ingress tracking every edge? That's the internal flow of data, so most relevant.

I think this sounds like a good approach!