prometheus-configuration

仓库: wshobson/agents
安装量: 3.2K
排名: #689

安装

npx skills add https://github.com/wshobson/agents --skill prometheus-configuration

Prometheus Configuration

Complete guide to Prometheus setup, metric collection, scrape configuration, and recording rules.

Purpose

Configure Prometheus for comprehensive metric collection, alerting, and monitoring of infrastructure and applications.

When to Use Set up Prometheus monitoring Configure metric scraping Create recording rules Design alert rules Implement service discovery Prometheus Architecture ┌──────────────┐ │ Applications │ ← Instrumented with client libraries └──────┬───────┘ │ /metrics endpoint ↓ ┌──────────────┐ │ Prometheus │ ← Scrapes metrics periodically │ Server │ └──────┬───────┘ │ ├─→ AlertManager (alerts) ├─→ Grafana (visualization) └─→ Long-term storage (Thanos/Cortex)

Installation Kubernetes with Helm helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace \ --set prometheus.prometheusSpec.retention=30d \ --set prometheus.prometheusSpec.storageVolumeSize=50Gi

Docker Compose version: "3.8" services: prometheus: image: prom/prometheus:latest ports: - "9090:9090" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml - prometheus-data:/prometheus command: - "--config.file=/etc/prometheus/prometheus.yml" - "--storage.tsdb.path=/prometheus" - "--storage.tsdb.retention.time=30d"

volumes: prometheus-data:

Configuration File

prometheus.yml:

global: scrape_interval: 15s evaluation_interval: 15s external_labels: cluster: "production" region: "us-west-2"

Alertmanager configuration

alerting: alertmanagers: - static_configs: - targets: - alertmanager:9093

Load rules files

rule_files: - /etc/prometheus/rules/*.yml

Scrape configurations

scrape_configs: # Prometheus itself - job_name: "prometheus" static_configs: - targets: ["localhost:9090"]

# Node exporters - job_name: "node-exporter" static_configs: - targets: - "node1:9100" - "node2:9100" - "node3:9100" relabel_configs: - source_labels: [address] target_label: instance regex: "([^:]+)(:[0-9]+)?" replacement: "${1}"

# Kubernetes pods with annotations - job_name: "kubernetes-pods" kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path regex: (.+) - source_labels: [address, meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address - source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: pod

# Application metrics - job_name: "my-app" static_configs: - targets: - "app1.example.com:9090" - "app2.example.com:9090" metrics_path: "/metrics" scheme: "https" tls_config: ca_file: /etc/prometheus/ca.crt cert_file: /etc/prometheus/client.crt key_file: /etc/prometheus/client.key

Reference: See assets/prometheus.yml.template

Scrape Configurations Static Targets scrape_configs: - job_name: "static-targets" static_configs: - targets: ["host1:9100", "host2:9100"] labels: env: "production" region: "us-west-2"

File-based Service Discovery scrape_configs: - job_name: "file-sd" file_sd_configs: - files: - /etc/prometheus/targets/.json - /etc/prometheus/targets/.yml refresh_interval: 5m

targets/production.json:

[ { "targets": ["app1:9090", "app2:9090"], "labels": { "env": "production", "service": "api" } } ]

Kubernetes Service Discovery scrape_configs: - job_name: "kubernetes-services" kubernetes_sd_configs: - role: service relabel_configs: - source_labels: [meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: __scheme regex: (https?) - source_labels: [meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: __metrics_path regex: (.+)

Reference: See references/scrape-configs.md

Recording Rules

Create pre-computed metrics for frequently queried expressions:

/etc/prometheus/rules/recording_rules.yml

groups: - name: api_metrics interval: 15s rules: # HTTP request rate per service - record: job:http_requests:rate5m expr: sum by (job) (rate(http_requests_total[5m]))

  # Error rate percentage
  - record: job:http_requests_errors:rate5m
    expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))

  - record: job:http_requests_error_rate:percentage
    expr: |
      (job:http_requests_errors:rate5m / job:http_requests:rate5m) * 100

  # P95 latency
  - record: job:http_request_duration:p95
    expr: |
      histogram_quantile(0.95,
        sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
      )

  • name: resource_metrics interval: 30s rules: # CPU utilization percentage

    • record: instance:node_cpu:utilization expr: | 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

    # Memory utilization percentage - record: instance:node_memory:utilization expr: | 100 - ((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100)

    # Disk usage percentage - record: instance:node_disk:utilization expr: | 100 - ((node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100)

Reference: See references/recording-rules.md

Alert Rules

/etc/prometheus/rules/alert_rules.yml

groups: - name: availability interval: 30s rules: - alert: ServiceDown expr: up{job="my-app"} == 0 for: 1m labels: severity: critical annotations: summary: "Service {{ $labels.instance }} is down" description: "{{ $labels.job }} has been down for more than 1 minute"

  - alert: HighErrorRate
    expr: job:http_requests_error_rate:percentage > 5
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High error rate for {{ $labels.job }}"
      description: "Error rate is {{ $value }}% (threshold: 5%)"

  - alert: HighLatency
    expr: job:http_request_duration:p95 > 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High latency for {{ $labels.job }}"
      description: "P95 latency is {{ $value }}s (threshold: 1s)"

  • name: resources interval: 1m rules:

    • alert: HighCPUUsage expr: instance:node_cpu:utilization > 80 for: 5m labels: severity: warning annotations: summary: "High CPU usage on {{ $labels.instance }}" description: "CPU usage is {{ $value }}%"

    • alert: HighMemoryUsage expr: instance:node_memory:utilization > 85 for: 5m labels: severity: warning annotations: summary: "High memory usage on {{ $labels.instance }}" description: "Memory usage is {{ $value }}%"

    • alert: DiskSpaceLow expr: instance:node_disk:utilization > 90 for: 5m labels: severity: critical annotations: summary: "Low disk space on {{ $labels.instance }}" description: "Disk usage is {{ $value }}%"

Validation

Validate configuration

promtool check config prometheus.yml

Validate rules

promtool check rules /etc/prometheus/rules/*.yml

Test query

promtool query instant http://localhost:9090 'up'

Reference: See scripts/validate-prometheus.sh

Best Practices Use consistent naming for metrics (prefix_name_unit) Set appropriate scrape intervals (15-60s typical) Use recording rules for expensive queries Implement high availability (multiple Prometheus instances) Configure retention based on storage capacity Use relabeling for metric cleanup Monitor Prometheus itself Implement federation for large deployments Use Thanos/Cortex for long-term storage Document custom metrics Troubleshooting

Check scrape targets:

curl http://localhost:9090/api/v1/targets

Check configuration:

curl http://localhost:9090/api/v1/status/config

Test query:

curl 'http://localhost:9090/api/v1/query?query=up'

Reference Files assets/prometheus.yml.template - Complete configuration template references/scrape-configs.md - Scrape configuration patterns references/recording-rules.md - Recording rule examples scripts/validate-prometheus.sh - Validation script Related Skills grafana-dashboards - For visualization slo-implementation - For SLO monitoring distributed-tracing - For request tracing

← 返回排行榜