Install Prometheus with Docker: Scrape Metrics and Write PromQL Queries

What is Prometheus #

Prometheus is a monitoring system that pulls metrics: every few seconds it requests /metrics from each target over HTTP and stores the result as time series. You query the data with PromQL and draw it with Grafana. In this tutorial you run it with Docker, monitor a host with Node Exporter and write an alert.

Configuration file #

global:
  scrape_interval: 15s
rule_files:
  - /etc/prometheus/alerts.yml
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]

Each job_name groups targets. Prometheus adds the labels job and instance to every metric.

Alert rule #

groups:
  - name: basics
    rules:
      - alert: TargetDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is down"

The alert fires when a target has been unreachable for one minute.

Start Prometheus and Node Exporter #

services:
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alerts.yml:/etc/prometheus/alerts.yml:ro
      - prom-data:/prometheus
  node-exporter:
    image: prom/node-exporter
    ports:
      - "9100:9100"
volumes:
  prom-data:

docker compose up -d

Open http://localhost:9090/targets: both targets must be UP. The Node Exporter in a container sees the container, not the host, so for real host metrics install it directly on the machine.

PromQL in five queries #

Type these in the Graph tab:

up                                        # 1 if the target is reachable, 0 if not
rate(prometheus_http_requests_total[5m])  # requests per second over 5 minutes
sum by (code) (rate(prometheus_http_requests_total[5m]))   # grouped by status code
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes  # fraction of free memory
histogram_quantile(0.95, sum by (le) (rate(prometheus_http_request_duration_seconds_bucket[5m])))  # p95 latency

Use rate() on counters (values that only grow) and the raw value on gauges.

Test the alert #

docker compose stop node-exporter

After a minute the alert is Firing under http://localhost:9090/alerts. Start the exporter again and it resolves. To send notifications to email or Slack you add Alertmanager and point Prometheus at it with an alerting section.

Monitor your own application #

Any application can expose /metrics with a Prometheus client library. Add a new job with its address and Prometheus starts scraping it. In Kubernetes, the Prometheus Operator discovers targets automatically with ServiceMonitor resources; the Terraform Helm provider tutorial installs it with Helm.