Section 2.1: Setting Up Prometheus Configuration

To configure Prometheus, we need to create three key files:

1. prometheus/prometheus.yml — This file contains the Prometheus configuration.
2. prometheus/alert.yml — This file defines the alerts that Prometheus will monitor.
3. docker-compose.yml — This file orchestrates the deployment of our services.

We will begin by adding the following content to prometheus/prometheus.yml:

global:

scrape_interval: 30s

scrape_timeout: 10s

rule_files:

• alert.yml

scrape_configs:

• job_name: services

  metrics_path: /metrics

  static_configs:

  • targets:

    • 'prometheus:9090'
    • 'idonotexists:564'

In this configuration, scrape_configs specifies the locations of our applications. The first entry points to the Prometheus service itself, while the second serves as a placeholder for demonstration purposes.

The rule_files section indicates where to locate our alert rules, which we will cover shortly. The scrape_interval determines how frequently Prometheus checks for new metric values, and if a scrape exceeds the specified scrape_timeout, it is canceled.

Next, we will configure the alert.yml file with the following content:

groups:

• name: DemoAlerts

  rules:

  • alert: InstanceDown

    expr: up{job="services"} < 1

    for: 5m

Here, up is a built-in metric from Prometheus that returns zero if the services were unreachable during the last scrape. The {job="services"} filter allows us to focus solely on metrics tagged with the service name defined in prometheus.yml.

To launch Prometheus, we will add the following to docker-compose.yml:

version: '3'

services:

prometheus:

image: prom/prometheus:v2.46.0

ports:

• 9000:9090

volumes:

• ./prometheus:/etc/prometheus
• prometheus-data:/prometheus

command: --web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml

volumes:

prometheus-data:

This configuration mounts our Prometheus folder in the correct location for the image to utilize our configuration. The volume prometheus-data is designated to store the scraped data, preserving it after restarts.

To reload configuration files without restarting Prometheus, you can use the following command:

curl -X POST http://localhost:9000/-/reload

To start Prometheus, run:

docker-compose up -d

And access the UI at http://localhost:9000 in your browser, where you can run ad-hoc queries on your metrics.

Section 3.1: Configuring Grafana

Grafana can function without configuration files; however, to streamline the setup, we will configure Prometheus as a data source. Create the file grafana/provisioning/datasources/prometheus_ds.yml and insert the following:

datasources:

• name: Prometheus

  access: proxy

  type: prometheus

  url: http://prometheus:9090

  isDefault: true

This configuration informs Grafana about our Prometheus data source, with the URL pointing to the Prometheus server accessible within the Docker network.

Next, update the docker-compose.yml file to include the Grafana service:

grafana:

image: grafana/grafana:10.0.0

ports:

• 3000:3000

restart: unless-stopped

volumes:

• grafana-data:/var/lib/grafana
• ./grafana/provisioning:/etc/grafana/provisioning

volumes:

grafana-data:

The first volume mounts our data source configuration, while the second persists Grafana's data, including dashboards and user settings. To start both Prometheus and Grafana, run:

docker-compose up -d

Access Grafana at http://localhost:3000 in your browser, using admin as both the username and password.

After logging in, you can create your first dashboard by clicking the "+" icon and selecting "New Dashboard." Click "Add visualization" to begin creating visualizations using the Prometheus data source.

To visualize metrics, you can enter PromQL queries. For instance, to display the total number of HTTP requests over time, enter:

increase(prometheus_http_requests_total[1m])

To learn more about PromQL, refer to the official documentation. You can also import pre-existing dashboards from the Grafana Dashboards page.

Section 4.1: Configuring Alertmanager

To integrate Alertmanager, add the following configuration to docker-compose.yml:

alertmanager:

image: prom/alertmanager:v0.25.0

ports:

• 9093:9093

volumes:

• ./alertmanager:/config
• alertmanager-data:/data

command: --config.file=/config/alertmanager/alertmanager.yml

volumes:

alertmanager-data:

Alertmanager will save silence configurations to the specified volume. Next, create an alertmanager.yml file in the alertmanager folder with the following content:

receivers:

• name: 'mail'

  email_configs:

  • smarthost: 'smtp.gmail.com:587'

    from: '[email protected]'

    to: '[email protected]'

    auth_username: '...'

    auth_password: '...'

This configuration details the channels for sending alerts, with email as the primary method. The route section specifies which alerts to send, and you can filter them based on tags.

To inform Prometheus about Alertmanager, add the following to prometheus/prometheus.yml:

alerting:

alertmanagers:

• static_configs:

  • targets:

    • 'alertmanager:9093'

Section 4.2: Testing and Using Alertmanager

Run docker-compose up to start all services. Access the Alertmanager UI at http://localhost:9093 to monitor alerts. After a few minutes, test alerts will trigger, and you can verify them in both the Prometheus and Alertmanager interfaces.

The Alertmanager UI allows you to view active and silenced alerts, manage notification receivers, and configure alert routing. To silence an alert, navigate to the "Silences" tab, click "New Silence," and specify the alert and duration for the silence.