Monitoring Process Exporter

验证：未记录

适用版本: 未记录
验证范围: 尚未登记验证记录。

本文缩写（英文全称与中文含义）

缩写	英文全称／名称	中文含义
CPU	Central Processing Unit	中央处理器
EC2	Elastic Compute Cloud	AWS 云服务器服务
VM	Virtual Machine	虚拟机

1. Important Points#

process-exporter 用来把 Linux process 信息按“进程组名”导出成 Prometheus metrics，适合监控 VM 上的关键守护进程是否存在、CPU、内存、线程、文件句柄等。

use for:
    monitor named process on EC2 / VM / bare metal
    verify critical daemon is running
    track process CPU / memory / thread / fd usage
    monitor legacy app not exposing its own /metrics

do not use for:
    every short-lived process
    per-user process inventory
    high-cardinality command line labels
    replacing application metrics

核心原则：

node_exporter:
    host level metrics: CPU / memory / disk / network

process-exporter:
    selected process level metrics: process count / CPU / memory / fd / thread

app metrics:
    business and internal application metrics

2. Metrics Model#

process-exporter 根据配置把匹配到的 process 归到 groupname，metrics label 里最重要的是 groupname。

process_names config:
    defines which processes to match
    defines group name

groupname:
    logical process name in metrics
    should be stable and low-cardinality

example:
    groupname="victoriametrics"
    groupname="alertmanager"
    groupname="grafana"

常用 metrics：

Metric	Meaning
`namedprocess_namegroup_num_procs`	matching process count
`namedprocess_namegroup_num_threads`	thread count
`namedprocess_namegroup_cpu_seconds_total`	CPU seconds counter
`namedprocess_namegroup_memory_bytes`	memory by type, such as resident / virtual
`namedprocess_namegroup_open_filedesc`	open file descriptors
`namedprocess_namegroup_worst_fd_ratio`	highest fd usage ratio in group
`namedprocess_namegroup_read_bytes_total`	process read bytes counter
`namedprocess_namegroup_write_bytes_total`	process write bytes counter

3. Config#

# /etc/process-exporter/config.yml
process_names:
  - name: "victoriametrics"
    exe:
      - /usr/local/bin/victoria-metrics

  - name: "alertmanager"
    exe:
      - /usr/local/bin/alertmanager

  - name: "grafana"
    cmdline:
      - "grafana-server"

  - name: "spug-gunicorn"
    exe:
      - /usr/bin/python3
    cmdline:
      - "gunicorn"
      - "spug.wsgi"

匹配方式：

comm:
    from /proc/<pid>/comm
    short process name
    useful when process name is stable

exe:
    executable path
    stable when binary path is fixed

cmdline:
    command line args
    useful for Python / Java / Node.js / gunicorn apps
    be careful not to create too many different groups

先用 ps 确认该匹配什么：

ps -eo pid,comm,args | grep -E "victoria|alertmanager|grafana|gunicorn"

4. Systemd#

# /etc/systemd/system/process-exporter.service
[Unit]
Description=Prometheus Process Exporter
After=network-online.target

[Service]
User=process_exporter
Group=process_exporter
ExecStart=/usr/local/bin/process-exporter --web.listen-address=:9256 --config.path=/etc/process-exporter/config.yml
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo useradd --system --no-create-home --shell /usr/sbin/nologin process_exporter
sudo install -d -m 0755 /etc/process-exporter
sudo systemctl daemon-reload
sudo systemctl enable --now process-exporter
curl -s http://127.0.0.1:9256/metrics | grep namedprocess_namegroup_num_procs

5. Scrape Config#

scrape_configs:
  - job_name: process-exporter
    scrape_interval: 30s
    static_configs:
      - targets:
          - app-ec2-1:9256
          - monitoring-ec2:9256

如果使用 vmagent，写法一样，目标是 scrape :9256/metrics。

6. Useful PromQL#

# process missing
namedprocess_namegroup_num_procs{groupname="victoriametrics"} == 0

# process CPU usage by group
sum by (instance, groupname) (rate(namedprocess_namegroup_cpu_seconds_total[5m]))

# resident memory
namedprocess_namegroup_memory_bytes{memtype="resident"}

# open file descriptors
namedprocess_namegroup_open_filedesc

# fd usage ratio
namedprocess_namegroup_worst_fd_ratio

7. Alert Examples#

groups:
  - name: process-exporter
    rules:
      - alert: ProcessMissing
        expr: namedprocess_namegroup_num_procs{groupname="victoriametrics"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "process missing"
          description: "{{ $labels.instance }} {{ $labels.groupname }} has no running process"

      - alert: ProcessHighMemory
        expr: namedprocess_namegroup_memory_bytes{memtype="resident"} > 2 * 1024 * 1024 * 1024
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "process memory high"
          description: "{{ $labels.instance }} {{ $labels.groupname }} resident memory is high"

8. Common Pitfalls#

进程存在不代表应用健康；process-exporter 结合应用或 blackbox 探测，groupname 不放 PID、request_id 等动态高基数值。

matching too broad:
    cmdline pattern accidentally matches helper scripts or grep process

too many dynamic group names:
    do not include pid, user id, request id, port, or timestamp in groupname

only checking process exists:
    process exists does not mean app is healthy; combine with blackbox_exporter or app /metrics

public exporter port:
    do not expose :9256 to internet

wrong binary path:
    verify with ps -eo pid,comm,args and readlink -f /proc/<pid>/exe