Grafana Dashboards — Prometheus Integration and Alerting Setup

What is Grafana?

Grafana is an open-source observability platform for visualizing time-series, log, and trace data and sending alerts. Connect Prometheus as a data source and you can plot PromQL query results as dashboard panels, then fire alerts to Slack, email, and more when thresholds are exceeded.

• Multiple data sources: Prometheus, Loki, Elasticsearch, PostgreSQL, and more
• Dashboards: composed of panels, rows, and variables
• Unified Alerting: Grafana 8+ unified alerting (rules + contact points + routing)

1. Installation

sudo apt-get install -y apt-transport-https software-properties-common
wget -q -O - https://apt.grafana.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/grafana.gpg
echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://apt.grafana.com stable main" \
  | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update && sudo apt-get install -y grafana

sudo systemctl enable --now grafana-server

Open http://<server>:3000 (default credentials: admin/admin). Change the password on first login.

2. Connecting the Prometheus Data Source

You can do this in the UI (Connections → Data sources), but in production the standard practice is to codify it with provisioning. /etc/grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    jsonData:
      httpMethod: POST
 timeInterval: 15s # Prometheus scrape_interval

sudo systemctl restart grafana-server

3. Building Dashboards

Quick start — import a community dashboard

Import the Node Exporter Full dashboard (ID 1860) to get a rich set of panels immediately. Dashboards → New → Import → enter 1860 → select the Prometheus data source.

Creating panels yourself

Panel queries use PromQL as-is.

# CPU
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Selecting hosts with template variables

Create an instance variable under Dashboard Settings → Variables.

Type:  Query
Query: label_values(node_uname_info, instance)

Use the variable in panel queries.

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle",instance="$instance"}[5m])) * 100)

You can now reuse the same dashboard across hosts by switching the dropdown.

4. Dashboard provisioning (automated deployment)

Manage exported dashboard JSON as files. /etc/grafana/provisioning/dashboards/main.yml:

apiVersion: 1

providers:
  - name: 'default'
    orgId: 1
    folder: 'Infra'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards

Place dashboard JSON files in /var/lib/grafana/dashboards/ and Grafana will load them automatically. This works well with Git-based configuration management.

5. Unified Alerting — alert rules

Grafana unified alerting consists of three components.

Configuring a contact point (Slack example)

Alerting → Contact points → Add. Enter the Slack Incoming Webhook URL.

Name:        slack-infra
Integration: Slack
Webhook URL: https://hooks.slack.com/services/XXX/YYY/ZZZ

Alert rule — CPU above 90% for 5 minutes

Alerting → Alert rules → New alert rule.

Query A (Prometheus):
  100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Expression B (Reduce):  Last value of A
Condition  C (Threshold): B IS ABOVE 90

Evaluation: every 1m, for 5m
Labels:     severity = critical

for 5m fires the alert only when the condition stays true for 5 consecutive minutes, reducing alert fatigue from transient spikes.

Routing (notification policy)

Add a label matcher so that severity=critical alerts go to the slack-infra contact point. Tune grouping and re-notification with group_wait, group_interval, and repeat_interval.

6. Managing alerts as code

Alert rules can also be provisioned. /etc/grafana/provisioning/alerting/rules.yml:

apiVersion: 1

groups:
  - orgId: 1
    name: node-alerts
    folder: Infra
    interval: 1m
    rules:
      - uid: high-mem
        title: HighMemoryUsage
        condition: C
        data:
          - refId: A
            datasourceUid: prometheus
            model:
              expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
        for: 5m
        labels:
          severity: warning

7. Operational tips

• Prevent alert fatigue: Keep for duration and repeat_interval long enough.
• Handle No Data: Missing data is also an incident. Treat the rule's No Data state as Alerting.
• Dashboard variable $__rate_interval: Automatically sizes the rate window to the scrape interval.
• Read-only access control: Give operators the Viewer role only to prevent dashboard tampering.

Summary

Grafana is the last piece that makes Prometheus-collected data human-readable and notifies you the moment a threshold is crossed. Codify data sources, dashboards, and alerts with provisioning and you can operate the entire monitoring stack in a reproducible way.