Propagation of NodeType via Prometheus metrics
Context: we rely on Prometheus metrics for various systems in the company. The setup of clusters until now was pretty standard, so basically we had replica sets and sharded clusters with electable nodes only – clusters with three nodes per RS or shard which numbers correspond to <clustername>-<shardnumber>-<(00|01|02)>>. Now we have a fourth analytics node in the clusters, which out of convention seems to get the next incremented number 03.
The thing is we cannot rely on convention and we need to have a way to distinguish the node types from Prometheus metrics (or at least from the Atlas API) to be able to separate how we react to metrics from different nodes. E.g. we have to react differently for production nodes (electable) than we have to in case of an analytics node.
I am aware of the tags for node types but I was neither able to see those tags exposed as a label in Prometheus metrics for a specific node nor was it possible to get this from the Atlas API.
Basically we need to be able to filter Prometheus metrics simplified like e.g. this: mongodbwhatevermetric{clname="<name>", nodetype="electable"}.
Mathias
shared this idea
-
Gregory
commented
you can use the `mongodb_repl_passive{}` metric to get all read-only and analytics nodes