Kubernetes 企业监控与告警闭环

验证：未记录

适用版本: 未记录
验证范围: 尚未登记验证记录。

本文缩写（英文全称与中文含义）

缩写	英文全称／名称	中文含义
API	Application Programming Interface	应用程序编程接口
AZ	Availability Zone	可用区
CRD	Custom Resource Definition	Kubernetes 自定义资源定义
CSI	Container Storage Interface	容器存储接口
ESO	External Secrets Operator	将外部秘密系统中的值同步到 Kubernetes 的控制器
HA	High Availability	高可用
ID	Identifier	标识符
PDB	Pod Disruption Budget	Pod 中断预算，约束自愿驱逐
PVC	Persistent Volume Claim	持久卷申领
RBAC	Role-Based Access Control	基于角色的访问控制
SLI	Service Level Indicator	服务等级指标
SLO	Service Level Objective	服务等级目标
SSO	Single Sign-On	单点登录
TSDB	Time Series Database	时间序列数据库
URL	Uniform Resource Locator	统一资源定位符，通常指访问地址
WAL	Write-Ahead Log	预写日志

场景：复用/交付 kube-prometheus-stack，为订单服务提供集群、应用 SLI 和可验证的告警链路。前置：基线、CSI、企业通知接收器与已有 Grafana。已有监控平台先评估复用，不能再装一套重复采集。

架构要点#

Prometheus 两副本各自采集/存储，并不互相复制 TSDB；Alertmanager 三副本协调通知，不保证绝对零重复。长期存储、跨集群查询与 HA 去重另接 Thanos/Mimir/VictoriaMetrics/托管服务，设 replica 标签和去重规则。Operator HA

Grafana 使用企业 SSO/RBAC、外部数据库或已有托管实例，不能给多个 SQLite Grafana 副本共享一个 PVC 冒充 HA。Prometheus/Alertmanager 管理入口仅私网并加身份认证；API 本身不要裸露公网。

Hands-on：安装持久化监控#

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm search repo prometheus-community/kube-prometheus-stack --versions
export MONITORING_VERSION='REPLACE_CHART_VERSION'
helm show values prometheus-community/kube-prometheus-stack --version "$MONITORING_VERSION" > monitoring-defaults.yaml

先通过 ESO 交付 monitoring/alertmanager-platform Secret，key 为 alertmanager.yaml，内容来自企业通知配置，例如以下结构。真实 webhook token/URL 在秘密系统维护：

global:
  resolve_timeout: 5m
route:
  receiver: commerce-oncall
  group_by: [cluster, namespace, alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
receivers:
  - name: commerce-oncall
    webhook_configs:
      - url: https://REPLACE_ENTERPRISE_ALERT_RECEIVER
        send_resolved: true

保存 monitoring-values.yaml；合并组织资源限制、跨 AZ 放置/PDB、NetworkPolicy 与远程存储配置后再上线：

commonLabels:
  platform-monitoring: "true"
kube-state-metrics:
  prometheus:
    monitor:
      additionalLabels: {platform-monitoring: "true"}
prometheus-node-exporter:
  prometheus:
    monitor:
      additionalLabels: {platform-monitoring: "true"}
grafana:
  enabled: false
prometheus:
  prometheusSpec:
    replicas: 2
    retention: 15d
    retentionSize: 80GB
    externalLabels: {cluster: commerce-staging}
    serviceMonitorSelector:
      matchLabels: {platform-monitoring: "true"}
    serviceMonitorNamespaceSelector:
      matchLabels: {monitoring-enabled: "true"}
    ruleSelector:
      matchLabels: {platform-monitoring: "true"}
    ruleNamespaceSelector:
      matchLabels: {monitoring-enabled: "true"}
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: REPLACE_STORAGE_CLASS
          accessModes: [ReadWriteOnce]
          resources: {requests: {storage: 100Gi}}
alertmanager:
  alertmanagerSpec:
    replicas: 3
    configSecret: alertmanager-platform
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: REPLACE_STORAGE_CLASS
          accessModes: [ReadWriteOnce]
          resources: {requests: {storage: 10Gi}}

selector 收紧后，Chart 自带 ServiceMonitor/PrometheusRule 也须带 platform-monitoring: "true" 标签；通过固定 Chart 对应 commonLabels/自定义标签配置实现并检查渲染结果，否则默认集群监控会被意外排除。为相应 namespace 设置 monitoring-enabled 标签，只有平台能修改该标签。

kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace orders monitoring monitoring-enabled=true --overwrite
helm upgrade --install monitoring prometheus-community/kube-prometheus-stack --version "$MONITORING_VERSION" -n monitoring -f monitoring-values.yaml --wait --timeout 15m
kubectl -n monitoring get prometheus,alertmanager,pods,pvc

采集与业务告警#

保存 orders-monitor.yaml；up 规则只覆盖被发现的 targets，目标完全消失还需 absent/期望副本告警：

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: orders-api
  namespace: orders
  labels: {platform-monitoring: "true"}
spec:
  selector:
    matchLabels: {app.kubernetes.io/name: orders-api}
  endpoints:
    - {port: http, path: /metrics, interval: 30s, scrapeTimeout: 10s}
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: orders-platform
  namespace: orders
  labels: {platform-monitoring: "true"}
spec:
  groups:
    - name: orders-availability
      rules:
        - alert: OrdersScrapeFailed
          expr: up{namespace="orders",service="orders-api"} == 0
          for: 5m
          labels: {severity: warning, team: commerce}
          annotations:
            summary: Orders metrics target cannot be scraped
            runbook_url: https://xiezhao.me/SRE/Troubleshooting/Website/
        - alert: OrdersTargetsMissing
          expr: absent(up{namespace="orders",service="orders-api"})
          for: 5m
          labels: {severity: critical, team: commerce}
          annotations:
            summary: Orders metrics discovery has no targets
            runbook_url: https://xiezhao.me/SRE/Troubleshooting/PodStartup/

kubectl apply -f orders-monitor.yaml
kubectl -n orders get servicemonitor,prometheusrule

在 Targets/Rules 页面确认选中、无重复、数据新鲜且规则已加载。应用进一步提供成功率、延迟 histogram、队列年龄、业务失败指标并建立 SLO burn-rate；不要把订单 ID、用户 ID、原始 URL 当指标标签。参考已有 Prometheus 指标设计与成本分摊。

维护到期和监控自身失联的逐步验证见通知可靠性演练。

企业验收与恢复#

预生产短暂阻断 metrics 端点→告警 Firing→企业接收器通知→值班确认→恢复→Resolved，保存整条时间线。分别重建一个 Prometheus/Alertmanager Pod，验证查询、通知和 PVC 数据仍可用；检查 HA 重复通知与静默持久化。

容量验收测活跃 series、抓取速率、WAL、磁盘增长和 query p95；retentionSize 为数据留余量，不等于全文件系统上限。告警平台自身要有外部存活探测。升级先审 CRD/规则/存储兼容；恢复 Git 和通知 Secret、按存储方案恢复 TSDB，不能依赖 Helm rollback 自动还原时间序列数据。