Server Monitoring with Prometheus and Grafana: From Zero to Dashboard

You Can't Fix What You Can't See

It's 2 AM. Your site is down. Users are tweeting. You're SSH'd into the server, running top, df -h, netstat, piecing together what happened from scattered logs and memory snapshots. Twenty minutes later you find it: a runaway PHP-FPM process started leaking memory at 11:47 PM, slowly consuming RAM until the OOM killer took down MySQL.

If you'd had a monitoring stack, you would have seen that memory climb at 11:50 PM. An alert would have fired at 11:55 PM. You'd have fixed it before 70% of users even noticed.

That's the difference between reactive and proactive monitoring. Reactive means you learn about problems from users. Proactive means your infrastructure tells you first.

This guide walks through building a complete monitoring stack — Prometheus, Node Exporter, application-level exporters, Grafana dashboards, and Alertmanager — from scratch on a Linux server. By the end, you'll have real-time visibility into every layer of your system: CPU, memory, disk, network, databases, web servers, and more. If you just need simple uptime alerts without the full stack, see our guide to self-hosting Uptime Kuma instead.

No hand-waving. No "just follow the documentation." Full config files, tested PromQL queries, and a complete setup you can replicate in under two hours.

Architecture Overview

Before touching a terminal, it helps to understand how the pieces fit together. The monitoring stack has four layers:

Targets (node_exporter, nginx, mysql, postgres, redis, php-fpm)
  |
  | scrape (pull model)
  v
PROMETHEUS
  Scrapes metrics / Stores time series / Evaluates alert rules
  Web UI: :9090  |  PromQL query engine  |  TSDB storage
  |                            |
  | query                      | alerts
  v                            v
GRAFANA                  ALERTMANAGER
Dashboards + panels      Route: Email / Slack / Telegram / PagerDuty
Variables + templates    Silence + inhibition
Web UI: :3000

The key design principle is the pull model: Prometheus scrapes metrics from exporters on a schedule. Exporters expose an HTTP endpoint (usually /metrics) that returns metrics in a standardized text format. Prometheus handles storage, querying, and alerting. Grafana handles visualization.

This separation of concerns is what makes the stack so flexible. You can add new exporters without touching Prometheus. You can swap Grafana for another visualization layer. Each component has a single job and does it well.

Installing Prometheus

Download and Install

# Create a dedicated user (no login shell, no home dir)
useradd --no-create-home --shell /bin/false prometheus

# Create directories
mkdir -p /etc/prometheus /var/lib/prometheus

# Download latest release (check https://prometheus.io/download/ for current version)
PROM_VERSION="2.51.2"
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
tar xzf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
cd prometheus-${PROM_VERSION}.linux-amd64

# Copy binaries
cp prometheus /usr/local/bin/
cp promtool /usr/local/bin/
chown prometheus:prometheus /usr/local/bin/prometheus
chown prometheus:prometheus /usr/local/bin/promtool

# Copy console libraries
cp -r consoles /etc/prometheus/
cp -r console_libraries /etc/prometheus/
chown -R prometheus:prometheus /etc/prometheus/
chown -R prometheus:prometheus /var/lib/prometheus/

Main Configuration: prometheus.yml

Prometheus configuration lives in /etc/prometheus/prometheus.yml. This single file controls everything: how often to scrape, which targets to scrape, and where to send alerts.

global:
  scrape_interval: 15s          # Default scrape interval
  evaluation_interval: 15s      # How often to evaluate alert rules
  scrape_timeout: 10s           # Timeout per scrape

# Alertmanager connection
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

# Alert rule files
rule_files:
  - /etc/prometheus/rules/*.yml

# Scrape configs
scrape_configs:
  # Prometheus scrapes itself
  - job_name: prometheus
    static_configs:
      - targets:
          - localhost:9090

  # Node Exporter (system metrics)
  - job_name: node
    static_configs:
      - targets:
          - localhost:9100

Systemd Service

cat > /etc/systemd/system/prometheus.service << EOF
[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=10GB \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-lifecycle

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable prometheus
systemctl start prometheus

The --web.enable-lifecycle flag enables the /-/reload endpoint so you can reload configuration without restarting: curl -X POST http://localhost:9090/-/reload.

The --storage.tsdb.retention.time=30d keeps 30 days of data. Adjust based on your disk budget.

Verify Prometheus is running by visiting http://YOUR_SERVER:9090. The Status → Targets page shows all configured scrape targets and their health.

Node Exporter: System-Level Metrics

Node Exporter is the standard exporter for Linux system metrics. It exposes over 1,000 metrics covering CPU, memory, disk, filesystem, network, load average, open file descriptors, and more.

Installation

useradd --no-create-home --shell /bin/false node_exporter

NODE_EXP_VERSION="1.8.0"
cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXP_VERSION}/node_exporter-${NODE_EXP_VERSION}.linux-amd64.tar.gz
tar xzf node_exporter-${NODE_EXP_VERSION}.linux-amd64.tar.gz
cp node_exporter-${NODE_EXP_VERSION}.linux-amd64/node_exporter /usr/local/bin/
chown node_exporter:node_exporter /usr/local/bin/node_exporter

cat > /etc/systemd/system/node_exporter.service << EOF
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter \
  --collector.systemd \
  --collector.processes

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable node_exporter
systemctl start node_exporter

Verify: curl -s http://localhost:9100/metrics | head -20

Essential Node Exporter Metrics

Application Exporters

Node Exporter covers the OS layer. For your applications, you need specialized exporters. Each one speaks the language of that specific service and translates it into Prometheus metrics format.

Nginx: nginx-prometheus-exporter

First, enable stub_status in nginx:

# In your nginx config:
server {
    listen 127.0.0.1:8080;
    location /stub_status {
        stub_status;
        allow 127.0.0.1;
        deny all;
    }
}

NGINX_EXP_VERSION="1.1.0"
wget https://github.com/nginx/nginx-prometheus-exporter/releases/download/v${NGINX_EXP_VERSION}/nginx-prometheus-exporter_${NGINX_EXP_VERSION}_linux_amd64.tar.gz
tar xzf nginx-prometheus-exporter_${NGINX_EXP_VERSION}_linux_amd64.tar.gz
cp nginx-prometheus-exporter /usr/local/bin/

cat > /etc/systemd/system/nginx_exporter.service << EOF
[Unit]
Description=Nginx Prometheus Exporter
After=network.target

[Service]
User=nobody
ExecStart=/usr/local/bin/nginx-prometheus-exporter \
  -nginx.scrape-uri=http://127.0.0.1:8080/stub_status \
  -web.listen-address=:9113
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl enable --now nginx_exporter

Key metrics: nginx_connections_active, nginx_connections_waiting, nginx_http_requests_total.

MySQL: mysqld_exporter

# Create MySQL user for the exporter
mysql -e "CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'StrongPass123' WITH MAX_USER_CONNECTIONS 3;"
mysql -e "GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';"
mysql -e "FLUSH PRIVILEGES;"

# Credentials file
cat > /etc/.mysqld_exporter.cnf << EOF
[client]
user=exporter
password=StrongPass123
EOF
chmod 600 /etc/.mysqld_exporter.cnf

MYSQL_EXP_VERSION="0.15.1"
wget https://github.com/prometheus/mysqld_exporter/releases/download/v${MYSQL_EXP_VERSION}/mysqld_exporter-${MYSQL_EXP_VERSION}.linux-amd64.tar.gz
tar xzf mysqld_exporter-${MYSQL_EXP_VERSION}.linux-amd64.tar.gz
cp mysqld_exporter-${MYSQL_EXP_VERSION}.linux-amd64/mysqld_exporter /usr/local/bin/

cat > /etc/systemd/system/mysqld_exporter.service << EOF
[Unit]
Description=MySQL Exporter
After=mysql.service

[Service]
User=nobody
ExecStart=/usr/local/bin/mysqld_exporter \
  --config.my-cnf=/etc/.mysqld_exporter.cnf \
  --web.listen-address=:9104
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl enable --now mysqld_exporter

Key metrics: mysql_global_status_threads_connected, mysql_global_status_queries, mysql_global_status_innodb_buffer_pool_reads, mysql_up.

PostgreSQL: postgres_exporter

psql -U postgres -c "CREATE USER postgres_exporter WITH PASSWORD 'StrongPass123';"
psql -U postgres -c "GRANT pg_monitor TO postgres_exporter;"

PG_EXP_VERSION="0.15.0"
wget https://github.com/prometheus-community/postgres_exporter/releases/download/v${PG_EXP_VERSION}/postgres_exporter-${PG_EXP_VERSION}.linux-amd64.tar.gz
tar xzf postgres_exporter-${PG_EXP_VERSION}.linux-amd64.tar.gz
cp postgres_exporter-${PG_EXP_VERSION}.linux-amd64/postgres_exporter /usr/local/bin/

cat > /etc/systemd/system/postgres_exporter.service << EOF
[Unit]
Description=PostgreSQL Exporter
After=postgresql.service

[Service]
User=nobody
Environment=DATA_SOURCE_NAME="postgresql://postgres_exporter:StrongPass123@localhost:5432/postgres?sslmode=disable"
ExecStart=/usr/local/bin/postgres_exporter --web.listen-address=:9187
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl enable --now postgres_exporter

Redis: redis_exporter

REDIS_EXP_VERSION="1.62.0"
wget https://github.com/oliver006/redis_exporter/releases/download/v${REDIS_EXP_VERSION}/redis_exporter-v${REDIS_EXP_VERSION}.linux-amd64.tar.gz
tar xzf redis_exporter-v${REDIS_EXP_VERSION}.linux-amd64.tar.gz
cp redis_exporter-v${REDIS_EXP_VERSION}.linux-amd64/redis_exporter /usr/local/bin/

cat > /etc/systemd/system/redis_exporter.service << EOF
[Unit]
Description=Redis Exporter
After=redis.service

[Service]
User=nobody
ExecStart=/usr/local/bin/redis_exporter \
  --redis.addr=redis://localhost:6379 \
  --web.listen-address=:9121
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl enable --now redis_exporter

PHP-FPM Exporter

Enable the PHP-FPM status page in your pool config (pm.status_path = /status), then:

PHP_FPM_EXP_VERSION="2.2.0"
wget https://github.com/hipages/php-fpm_exporter/releases/download/v${PHP_FPM_EXP_VERSION}/php-fpm_exporter_${PHP_FPM_EXP_VERSION}_linux_amd64
chmod +x php-fpm_exporter_${PHP_FPM_EXP_VERSION}_linux_amd64
cp php-fpm_exporter_${PHP_FPM_EXP_VERSION}_linux_amd64 /usr/local/bin/php-fpm_exporter

cat > /etc/systemd/system/phpfpm_exporter.service << EOF
[Unit]
Description=PHP-FPM Exporter
After=php-fpm.service

[Service]
User=nobody
ExecStart=/usr/local/bin/php-fpm_exporter \
  --phpfpm.scrape-uri=tcp://127.0.0.1:9000/status \
  --web.listen-address=:9253
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl enable --now phpfpm_exporter

Blackbox Exporter: Probing Endpoints

The blackbox exporter probes external endpoints over HTTP, HTTPS, TCP, DNS, and ICMP. This lets you monitor from the outside: is your site responding? Is the SSL certificate valid? How long does a request take?

BB_EXP_VERSION="0.25.0"
wget https://github.com/prometheus/blackbox_exporter/releases/download/v${BB_EXP_VERSION}/blackbox_exporter-${BB_EXP_VERSION}.linux-amd64.tar.gz
tar xzf blackbox_exporter-${BB_EXP_VERSION}.linux-amd64.tar.gz
cp blackbox_exporter-${BB_EXP_VERSION}.linux-amd64/blackbox_exporter /usr/local/bin/

cat > /etc/prometheus/blackbox.yml << EOF
modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: [200, 301, 302]
      follow_redirects: true
      preferred_ip_protocol: ip4
  tcp_connect:
    prober: tcp
    timeout: 5s
EOF

Add to your prometheus.yml scrape_configs:

- job_name: blackbox_http
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
        - https://yoursite.com
        - https://yoursite.com/api/health
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: localhost:9115

Key blackbox metrics: probe_success (1=up, 0=down), probe_duration_seconds, probe_ssl_earliest_cert_expiry (Unix timestamp of cert expiry), probe_http_status_code. If probe_duration_seconds trends upward, see our guide on reducing server response time (TTFB) under 200ms.

PromQL: The Query Language

PromQL is the language you use to query Prometheus data — in Grafana panels and in alert rules. It has a distinctive data model that can feel unfamiliar at first, but it follows a clear logic.

The Data Model

Every metric is a time series identified by a name plus a set of key-value labels. For example:

node_cpu_seconds_total{cpu="0", instance="server1", mode="idle"}
node_cpu_seconds_total{cpu="0", instance="server1", mode="system"}
node_cpu_seconds_total{cpu="0", instance="server1", mode="user"}

These are three separate time series. PromQL lets you select, filter, aggregate, and transform them.

Instant vs Range Vectors

# Instant vector: current value of all CPU series
node_cpu_seconds_total

# Filter by label
node_cpu_seconds_total{mode="idle"}

# Range vector: last 5 minutes of data
node_cpu_seconds_total[5m]

Essential Functions

• rate(counter[interval]) — per-second rate of change over an interval (for counters)
• increase(counter[interval]) — total increase over an interval
• avg(), sum(), max(), min() — aggregation operators
• by (label) — group aggregation results by a label
• without (label) — aggregate while removing a specific label
• histogram_quantile(p, histogram) — compute a percentile from a histogram metric
• absent() — returns 1 if a metric has no current data (useful for "service down" alerts)

Practical PromQL Queries

CPU Usage Percentage (all cores, averaged):

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Memory Usage Percentage:

100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))

Disk Usage Percentage (root filesystem):

100 * (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}))

Network Traffic In (bytes/sec):

rate(node_network_receive_bytes_total{device!="lo"}[5m])

Disk I/O Utilization (%):

rate(node_disk_io_time_seconds_total[5m]) * 100

HTTP Requests Per Second (nginx):

rate(nginx_http_requests_total[5m])

MySQL Queries Per Second:

rate(mysql_global_status_queries[5m])

MySQL Slow Queries Per Second:

rate(mysql_global_status_slow_queries[5m])

Redis Ops Per Second:

rate(redis_commands_processed_total[5m])

Redis Memory Usage (%):

redis_memory_used_bytes / redis_memory_max_bytes * 100

SSL Certificate Expiry (days remaining):

(probe_ssl_earliest_cert_expiry - time()) / 86400

Site Up/Down:

probe_success{job="blackbox_http"}

HTTP Response Time 95th Percentile:

histogram_quantile(0.95, rate(probe_duration_seconds_bucket[5m]))

OOM Kills in the last hour:

increase(node_vmstat_oom_kill[1h])

Server uptime (seconds):

time() - node_boot_time_seconds

Installing Grafana

Installation (Ubuntu/Debian)

apt-get install -y apt-transport-https software-properties-common
wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | \
  tee /etc/apt/sources.list.d/grafana.list
apt-get update
apt-get install grafana

systemctl daemon-reload
systemctl enable grafana-server
systemctl start grafana-server

Grafana starts on port 3000. Default credentials: admin / admin. You'll be prompted to change the password on first login.

Adding Prometheus as a Data Source

1. Go to Connections → Data Sources → Add data source
2. Select Prometheus
3. Set URL to http://localhost:9090
4. Click Save & Test — you should see "Successfully queried the Prometheus API"

Building Dashboards in Grafana

Panel Types

Dashboard Variables

Variables let you build one dashboard that works for many servers. Instead of hardcoding an instance, create a $instance variable that populates from Prometheus labels.

In Dashboard Settings → Variables → Add variable:

• Name: instance
• Type: Query
• Data source: Prometheus
• Query: label_values(node_cpu_seconds_total, instance)
• Multi-value: Yes | Include All option: Yes

Then use $instance in your panel queries: node_memory_MemAvailable_bytes{instance="$instance"}. The dropdown at the top of the dashboard filters all panels simultaneously.

Recommended Community Dashboards

Don't build from scratch. Import proven dashboards from grafana.com/grafana/dashboards. Dashboards → New → Import → enter the ID → Load → select Prometheus data source → Import.

Blackbox probes tell you a page loaded; they do not tell you how it felt to a real visitor. For that layer, see Core Web Vitals: Fix LCP, CLS, and INP from the Server Side.

Alerting with Alertmanager

Alertmanager handles routing, deduplication, grouping, and silencing of alerts. Prometheus evaluates alert rules and sends firing alerts to Alertmanager. Alertmanager decides who gets notified, how, and when.

Installing Alertmanager

AM_VERSION="0.27.0"
cd /tmp
wget https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}/alertmanager-${AM_VERSION}.linux-amd64.tar.gz
tar xzf alertmanager-${AM_VERSION}.linux-amd64.tar.gz
cp alertmanager-${AM_VERSION}.linux-amd64/alertmanager /usr/local/bin/
cp alertmanager-${AM_VERSION}.linux-amd64/amtool /usr/local/bin/

useradd --no-create-home --shell /bin/false alertmanager
mkdir -p /etc/alertmanager /var/lib/alertmanager
chown alertmanager:alertmanager /etc/alertmanager /var/lib/alertmanager

Alertmanager Configuration

cat > /etc/alertmanager/alertmanager.yml << EOF
global:
  smtp_from: [email protected]
  smtp_smarthost: smtp.gmail.com:587
  smtp_auth_username: [email protected]
  smtp_auth_password: your-app-password
  slack_api_url: https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK

route:
  group_by: [alertname, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: email-and-slack
  routes:
    - match:
        severity: critical
      receiver: critical-receiver
      repeat_interval: 1h
    - match:
        severity: warning
      receiver: email-only

receivers:
  - name: email-and-slack
    email_configs:
      - to: [email protected]
    slack_configs:
      - channel: '#alerts'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
  - name: critical-receiver
    email_configs:
      - to: [email protected]
    slack_configs:
      - channel: '#critical-alerts'
  - name: email-only
    email_configs:
      - to: [email protected]

inhibit_rules:
  - source_match:
      alertname: InstanceDown
    target_match:
      severity: warning
    equal: [instance]
EOF

Alertmanager Systemd Service

cat > /etc/systemd/system/alertmanager.service << EOF
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.listen-address=0.0.0.0:9093

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl enable --now alertmanager

Alert Rules

Alert rules live in /etc/prometheus/rules/. Here is a production-ready ruleset covering the scenarios that cause the most incidents:

cat > /etc/prometheus/rules/system.yml << EOF
groups:
  - name: system
    rules:

      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ \$labels.instance }} is down"
          description: "{{ \$labels.instance }} has been unreachable for 2+ minutes."

      - alert: HighCPU
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ \$labels.instance }}"
          description: "CPU above 90% for 5+ minutes. Current: {{ \$value | printf "%.1f" }}%"

      - alert: HighMemory
        expr: 100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory on {{ \$labels.instance }}"
          description: "Memory above 90% for 5+ minutes. Current: {{ \$value | printf "%.1f" }}%"

      - alert: DiskAlmostFull
        expr: 100 * (1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|devtmpfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|devtmpfs"})) > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Disk almost full on {{ \$labels.instance }}"
          description: "Disk {{ \$labels.mountpoint }} is {{ \$value | printf "%.1f" }}% full."

      - alert: DiskCritical
        expr: 100 * (1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|devtmpfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|devtmpfs"})) > 95
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Disk critically full on {{ \$labels.instance }}"
          description: "{{ \$labels.mountpoint }} is {{ \$value | printf "%.1f" }}% full. Immediate action required."

      - alert: HighLoadAverage
        expr: node_load1 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High load on {{ \$labels.instance }}"
          description: "Load average is {{ \$value | printf "%.2f" }}x CPU count."

      - alert: SSLCertExpiryWarning
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 30
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "SSL cert expiring soon for {{ \$labels.instance }}"
          description: "SSL cert expires in {{ \$value | printf "%.0f" }} days."

      - alert: SSLCertExpiryCritical
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 7
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "SSL cert expiring in {{ \$value | printf "%.0f" }} days"
          description: "Renew immediately: {{ \$labels.instance }}"

      - alert: SiteDown
        expr: probe_success{job="blackbox_http"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "{{ \$labels.instance }} is unreachable"
          description: "HTTP probe has been failing for 2+ minutes."

  - name: mysql
    rules:
      - alert: MySQLDown
        expr: mysql_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "MySQL is down on {{ \$labels.instance }}"

      - alert: MySQLTooManyConnections
        expr: mysql_global_status_threads_connected / mysql_global_variables_max_connections * 100 > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "MySQL connections at {{ \$value | printf "%.1f" }}% of max"

      - alert: MySQLHighSlowQueries
        expr: rate(mysql_global_status_slow_queries[5m]) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "MySQL slow queries spiking on {{ \$labels.instance }}"
          description: "{{ \$value | printf "%.1f" }} slow queries/sec for 5+ minutes."
EOF

After adding rules, reload Prometheus: curl -X POST http://localhost:9090/-/reload. Check Status → Rules in the Prometheus UI to confirm they loaded.

Best Practices

Storage and Retention Planning

Prometheus uses approximately 1-2 bytes per sample. With 15s scrape intervals and 1,000 time series, that's about 5.76 million samples/day — roughly 7-14 MB/day. A 30-day retention on a typical server setup uses 200-500 MB.

If your storage grows faster, check the TSDB stats at http://localhost:9090/api/v1/status/tsdb. High-cardinality exporters with per-user or per-request labels can generate millions of series.

• 15 days — minimum useful operational history
• 30 days — good default, covers month-over-month comparisons
• 90 days — quarterly trends, roughly 1-3 GB for typical setups
• Long-term — use Thanos or Cortex to offload to object storage (S3, GCS)

Recording Rules for Expensive Queries

If a PromQL query runs across many dashboard panels, it gets re-evaluated on every refresh. Recording rules pre-compute these into new metrics, improving query performance significantly:

groups:
  - name: recording_rules
    interval: 1m
    rules:
      - record: instance:cpu_utilization:ratio
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

      - record: instance:memory_utilization:ratio
        expr: 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

Security: Keep Prometheus Internal

By default, Prometheus, Grafana, and exporters have no authentication. Never expose ports 9090, 9100, 9093, or exporter ports directly to the internet:

• Firewall rules — restrict access to trusted IPs: ufw allow from YOUR_IP to any port 9090
• Nginx reverse proxy with basic auth — put Prometheus behind nginx with HTTPS and a password
• Prometheus built-in basic auth — available since v2.24 via web.yml config
• Grafana as the only public endpoint — expose only Grafana with proper auth, keep everything else on localhost

Monitoring Multiple Servers

For a fleet of servers, you have two common patterns:

Centralized scraping — a single Prometheus scrapes all servers remotely. Scales to roughly 50-100 servers. Each server runs its exporters, and you open firewall rules to allow the central Prometheus to reach them.

Federation — each server runs its own Prometheus, and a central "meta-Prometheus" scrapes only aggregated metrics from each. Scales to hundreds of servers, keeps raw data local:

- job_name: federate
  honor_labels: true
  metrics_path: /federate
  params:
    match[]:
      - '{job="node"}'
      - '{__name__=~"instance:.*"}'
  static_configs:
    - targets:
        - server1.example.com:9090
        - server2.example.com:9090

Complete Setup: Quick Summary

To replicate this stack on a fresh server:

1. Create system users: prometheus, alertmanager, node_exporter
2. Install Prometheus, Node Exporter, application exporters, Grafana, Alertmanager (instructions above, in order)
3. Configure /etc/prometheus/prometheus.yml with all scrape targets
4. Create /etc/prometheus/rules/system.yml with alert rules
5. Start all services: systemctl enable --now prometheus alertmanager node_exporter grafana-server
6. Open http://SERVER:9090/targets — all targets should show UP
7. Open http://SERVER:3000 — add Prometheus as data source, import dashboard IDs: 1860, 12708, 7362, 9628, 11835, 13659
8. Update /etc/alertmanager/alertmanager.yml with your SMTP/Slack/Telegram credentials
9. Test alert config: amtool check-config /etc/alertmanager/alertmanager.yml

Quick Reference

Essential Metrics and Thresholds

PromQL Cheat Sheet

If this entire stack feels like a lot of moving parts to keep patched and running, that is a fair reaction — it is. Panelica ships Prometheus and Grafana as managed services out of the box, with node-level and per-user metrics pre-wired into the panel's dashboard, so you get the monitoring without owning the exporter fleet. See how this compares across panels in Server Monitoring: cPanel vs Plesk vs Panelica.

Frequently Asked Questions

Do I need Grafana if I already have Prometheus?

Prometheus has a basic built-in expression browser, but it is not meant for dashboards. Grafana is where you build the visual dashboards, set up variables for multi-server views, and get the panel types (gauges, heatmaps, tables) that make metrics actually readable at a glance.

How much disk space does Prometheus actually use?

Roughly 1-2 bytes per sample. With a 15-second scrape interval and around 1,000 time series, expect 7-14 MB per day, or 200-500 MB for 30 days of retention on a typical single-server setup. High-cardinality metrics (per-user or per-request labels) can push this much higher.

Is it safe to expose Prometheus or Grafana directly to the internet?

Prometheus and its exporters have no authentication by default and should never be exposed directly. Keep them on localhost or a private network, and put only Grafana behind a reverse proxy with proper authentication if you need external access.

What is the difference between Prometheus and Alertmanager?

Prometheus evaluates alert rules against your metrics and decides when an alert should fire. Alertmanager receives those firing alerts and handles what happens next: grouping, deduplication, silencing, and routing to the correct notification channel (email, Slack, PagerDuty).

What You Have Now

At this point you've built a monitoring stack covering the full server layer — from OS through application — with alerting configured to catch problems before users do. Prometheus collects and stores data. Grafana visualizes it. Alertmanager routes notifications to the right people at the right time.

The community dashboards cover 90% of what most teams need out of the box. The alert rules will catch the scenarios that historically cause the most incidents: disk full, memory exhaustion, service down, SSL expiry, slow queries.

The natural next step from here is adding Loki for log aggregation — the Grafana-native logging backend that lets you correlate metric anomalies with the log lines that explain them. But that's a separate guide.

What you have today will tell you, clearly and before your users do, when something is going wrong.