Node Exporter Metrics


1. Scope#

这个页面只对应下面这个 dashboard JSON:

static/Grafana/aws-ec2-node-exporter-overview-dashboard.json

它只使用 node_exporter 指标，不使用 CloudWatch / YACE:

包含:
    CPU
    memory
    filesystem
    network
    disk
    load
    inode
    exporter up

不包含:
    AWS EC2 StatusCheckFailed
    AWS CPUCreditBalance
    AWS EBS BurstBalance
    AWS EBS VolumeQueueLength

如果需要 AWS 原生状态检查、CPU credit、EBS volume 维度，请使用:

static/Grafana/aws-ec2-overview-dashboard.json

2. Variables#

Variable Source Meaning
node_job custom node_exporter scrape job，默认 node / node-exporter
node_instance label_values(up{job=~"$node_job"}, instance) node_exporter target，通常是 ip:9100 或 hostname
mountpoint `label_values(node_filesystem_size_bytes{job="$node_job",instance="$node_instance",mountpoint!~"/boot($ /.*)
net_device textbox regex network device regex，例如 `eth0
disk_device textbox regex block device regex，例如 `nvme.*

filesystem filter note:

默认排除:
    /boot
    /boot/efi
    /run
    tmpfs / devtmpfs / overlay 等伪文件系统

原因:
    这些挂载点通常不是日常容量运营真正关心的对象。
    EC2 / VM 主视图更应该优先展示 /、/data、/var/lib/... 这类真实业务挂载点。

swap note:

当前 dashboard 默认不展示 swap。

原因:
    现在很多 EC2 / container host 默认不启用 swap，
    或者团队明确不允许使用 swap。

如果某些老机器或特殊节点明确启用了 swap，
再按需把 swap panel 加回去更合适。

3. Panel Metrics#

Row Panel PromQL Base Metrics
Overview node_exporter Up min(up{job=~"$node_job",instance=~"$node_instance"}) up
Overview CPU Used Percent 100 - (avg by (instance) (rate(node_cpu_seconds_total{job=~"$node_job",instance=~"$node_instance",mode="idle"}[5m])) * 100) node_cpu_seconds_total
Overview Memory Used Percent 100 * (1 - (node_memory_MemAvailable_bytes{job=~"$node_job",instance=~"$node_instance"} / node_memory_MemTotal_bytes{job=~"$node_job",instance=~"$node_instance"})) node_memory_MemAvailable_bytes, node_memory_MemTotal_bytes
Overview Filesystem Used Percent `100 * (1 - (node_filesystem_avail_bytes{job="$node_job",instance="$node_instance",mountpoint="$mountpoint",mountpoint!"/boot($ /.*)
Overview Network Receive / Transmit sum by (instance) (rate(node_network_receive_bytes_total{job=~"$node_job",instance=~"$node_instance",device=~"$net_device",device!~"lo|veth.*|docker.*|br-.*"}[5m])) node_network_receive_bytes_total
Overview Network Receive / Transmit sum by (instance) (rate(node_network_transmit_bytes_total{job=~"$node_job",instance=~"$node_instance",device=~"$net_device",device!~"lo|veth.*|docker.*|br-.*"}[5m])) node_network_transmit_bytes_total
Drilldown CPU By Mode avg by (instance, mode) (rate(node_cpu_seconds_total{job=~"$node_job",instance=~"$node_instance",mode!="idle"}[5m])) node_cpu_seconds_total
Drilldown Load Average node_load1{job=~"$node_job",instance=~"$node_instance"} node_load1
Drilldown Load Average node_load5{job=~"$node_job",instance=~"$node_instance"} node_load5
Drilldown Disk IOPS sum by (instance, device) (rate(node_disk_reads_completed_total{job=~"$node_job",instance=~"$node_instance",device=~"$disk_device"}[5m])) node_disk_reads_completed_total
Drilldown Disk IOPS sum by (instance, device) (rate(node_disk_writes_completed_total{job=~"$node_job",instance=~"$node_instance",device=~"$disk_device"}[5m])) node_disk_writes_completed_total
Drilldown Disk Throughput sum by (instance, device) (rate(node_disk_read_bytes_total{job=~"$node_job",instance=~"$node_instance",device=~"$disk_device"}[5m])) node_disk_read_bytes_total
Drilldown Disk Throughput sum by (instance, device) (rate(node_disk_written_bytes_total{job=~"$node_job",instance=~"$node_instance",device=~"$disk_device"}[5m])) node_disk_written_bytes_total
Capacity Filesystem Available Bytes `node_filesystem_avail_bytes{job="$node_job",instance="$node_instance",mountpoint="$mountpoint",mountpoint!"/boot($ /.*)
Capacity Memory Available Bytes node_memory_MemAvailable_bytes{job=~"$node_job",instance=~"$node_instance"} node_memory_MemAvailable_bytes
Debug Filesystem Inodes Used `100 * (1 - (node_filesystem_files_free{job="$node_job",instance="$node_instance",mountpoint="$mountpoint",mountpoint!"/boot($ /.*)
Debug Disk IO Time Ratio rate(node_disk_io_time_seconds_total{job=~"$node_job",instance=~"$node_instance",device=~"$disk_device"}[5m]) node_disk_io_time_seconds_total

4. Metric List#

Dashboard 需要 node_exporter 暴露这些 metrics:

up

node_cpu_seconds_total

node_memory_MemAvailable_bytes
node_memory_MemTotal_bytes

node_filesystem_avail_bytes
node_filesystem_size_bytes
node_filesystem_files_free
node_filesystem_files

node_network_receive_bytes_total
node_network_transmit_bytes_total

node_load1
node_load5

node_disk_reads_completed_total
node_disk_writes_completed_total
node_disk_read_bytes_total
node_disk_written_bytes_total
node_disk_io_time_seconds_total

5. Label Requirements#

这些 labels 必须存在，否则 dashboard filter 或 panel 会空:

up:
    job
    instance

node_cpu_seconds_total:
    job
    instance
    mode

node_filesystem_*:
    job
    instance
    mountpoint
    fstype

node_network_*:
    job
    instance
    device

node_disk_*:
    job
    instance
    device

6. Verify#

导入 dashboard 前先在 VMUI / Grafana Explore 里确认:

up{job=~"node|node-exporter"}
node_cpu_seconds_total
node_memory_MemAvailable_bytes
node_filesystem_size_bytes
node_network_receive_bytes_total
node_disk_reads_completed_total

如果 node_instance 为空，先确认 vmagent / Prometheus scrape config 的 job name:

scrape_configs:
  - job_name: node
    static_configs:
      - targets:
          - 10.0.1.10:9100

7. Known Gaps#

node_exporter 看的是 OS 视角，不等于 AWS 视角:

node_exporter 能看:
    OS CPU / memory / disk / network
    filesystem / inode
    block device IO

node_exporter 不能看:
    EC2 StatusCheckFailed
    instance/system status check split
    T family CPU credit
    EBS BurstBalance
    EBS VolumeQueueLength
    EBS volume id 维度
YACE Dashboard Forward