Monitor Consul using Prometheus and Grafana

Consul as DNS to reduce the latency from client to server. reference: https://medium.com/containers-on-aws/how-to-setup-service-discovery-in-elastic-container-service-3d18479959e6

Start Consul

{
"telemetry": {
"prometheus_retention_time": "480h",
"disable_hostname": true
}
}
curl http://127.0.0.1:8500/v1/agent/metrics\?format\=prometheus

Configuration in prometheus

- job_name: consul
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: '/v1/agent/metrics'
scheme: http
param:
format: ["prometheus"]
static_configs:
- targets:
- <consulserver1>:8500
- <consulserver2>:8500
groups:- name: consul.rulesrules:- alert: ConsulServiceHealthcheckFailedexpr: consul_catalog_service_node_healthy == 0for: 1mlabels:severity: criticalannotations:summary: Consul service healthcheck failed (instance {{ $labels.instance }})description: "Service: `{{ $labels.service_name }}` Healthcheck: `{{ $labels.service_id }}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ConsulMissingMasterNodeexpr: consul_raft_peers < 3for: 0mlabels:severity: criticalannotations:summary: Consul missing master node (instance {{ $labels.instance }})description: "Numbers of consul raft peers should be 3, in order to preserve quorum.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"- alert: ConsulAgentUnhealthyexpr: consul_health_node_status{status="critical"} == 1for: 0mlabels:severity: criticalannotations:summary: Consul agent unhealthy (instance {{ $labels.instance }})description: "A Consul agent is down\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

--

--

--

SRE | DevOps | Software Architect

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Laravel Route | Grouping routes by controllers in Laravel 8 & 9

Databases in Coding Schools, Yay or Nay?

Algorithms

60 days to become a game developer. Day 27 “The Great Fleece”

Processing tweets with Spark : Spark streaming vs Streaming structured

Jez Timms on Unsplash.

Feature Highlight: Cooldown Systems & Coroutines

Using Sinatra and Ruby… to make a Custom API

Enable free MFA on AWS Account

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Irene W

Irene W

SRE | DevOps | Software Architect

More from Medium

Azure Self-managed Kubernetes High Availability for Open5gs [part 2]

Why We Should Use ‘latest’ Tag on Container Images

Gitlab runners on Kubernetes

Installing Red Hat Advanced Cluster Manager on OpenShift Cluster