跳到主要内容

在 Kubernetes 上部署 Prometheus

前面我们已经了解了 Prometheus 的基本使用方式,主要是使用的二进制方式进行部署的,在实际生产环境来说,Prometheus 更适合用来部署在 Kubernetes 集群中,本节我们将介绍如何用手动方式在 Kubernetes 集群上部署 Prometheus,关于 Kubernetes 本身的使用可以参考我们的另外课程《Kubernetes 进阶训练营》

安装

由于我们这里是要运行在 Kubernetes 系统中,所以我们直接用 Docker 镜像的方式运行。这里我们使用的实验环境是基于 Kubernetes v1.22 版本,一共 3 个节点:

☸ ➜ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready control-plane,master 49d v1.22.2
node1 Ready <none> 49d v1.22.2
node2 Ready <none> 49d v1.22.2

为了方便管理,我们将监控相关的所有资源对象都安装在 kube-mon 这个 namespace 下面,没有的话可以提前创建:

☸ ➜ kubectl create ns kube-mon

为了能够方便的管理配置文件,我们这里将 prometheus.yml 配置文件用 ConfigMap 的形式进行管理:

# config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: kube-mon
data:
prometheus.yml: |
global:
scrape_interval: 15s
scrape_timeout: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']

我们这里暂时只配置了对 prometheus 本身的监控,直接创建该资源对象:

☸ ➜ kubectl apply -f https://p8s.io/docs/k8s/manifests/prometheus/config-1.yaml
configmap "prometheus-config" created

配置文件创建完成了,以后如果我们有新的资源需要被监控,我们只需要将上面的 ConfigMap 对象更新即可。现在我们来创建 prometheus 的 Pod 资源:

# prometheus-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
namespace: kube-mon
labels:
app: prometheus
spec:
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
serviceAccountName: prometheus
containers:
- image: prom/prometheus:v2.31.1
name: prometheus
args:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus' # 指定tsdb数据路径
- '--storage.tsdb.retention.time=24h'
- '--web.enable-admin-api' # 控制对admin HTTP API的访问,其中包括删除时间序列等功能
- '--web.enable-lifecycle' # 支持热更新,直接执行localhost:9090/-/reload立即生效
ports:
- containerPort: 9090
name: http
volumeMounts:
- mountPath: '/etc/prometheus'
name: config-volume
- mountPath: '/prometheus'
name: data
resources:
requests:
cpu: 200m
memory: 1024Mi
limits:
cpu: 200m
memory: 1024Mi
volumes:
- name: data
persistentVolumeClaim:
claimName: prometheus-data
- configMap:
name: prometheus-config
name: config-volume

持久化

另外为了 prometheus 的性能和数据持久化我们这里是直接将通过一个 LocalPV 来进行数据持久化的,注意一定不能使用 nfs 来持久化数据,通过 --storage.tsdb.path=/prometheus 指定数据目录,创建如下所示的一个 PVC 资源对象,注意是一个 LocalPV,和 node2 节点具有亲和性:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: prometheus-local
labels:
app: prometheus
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 20Gi
storageClassName: local-storage
local:
path: /data/k8s/prometheus
persistentVolumeReclaimPolicy: Retain
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-data
namespace: kube-mon
spec:
selector:
matchLabels:
app: prometheus
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: local-storage

由于 prometheus 可以访问 Kubernetes 的一些资源对象,所以需要配置 rbac 相关认证,这里我们使用了一个名为 prometheus 的 serviceAccount 对象:

# rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: kube-mon
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups:
- ''
resources:
- nodes
- services
- endpoints
- pods
- nodes/proxy
verbs:
- get
- list
- watch
- apiGroups:
- 'extensions'
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- configmaps
- nodes/metrics
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: kube-mon

由于我们要获取的资源信息,在每一个 namespace 下面都有可能存在,所以我们这里使用的是 ClusterRole 的资源对象,值得一提的是我们这里的权限规则声明中有一个 nonResourceURLs 的属性,是用来对非资源型 metrics 进行操作的权限声明,这个在以前我们很少遇到过,然后直接创建上面的资源对象即可:

☸ ➜ kubectl apply -f https://p8s.io/docs/k8s/manifests/prometheus/rbac.yaml
serviceaccount "prometheus" created
clusterrole.rbac.authorization.k8s.io "prometheus" created
clusterrolebinding.rbac.authorization.k8s.io "prometheus" created

现在我们就可以添加 promethues 的资源对象了:

☸ ➜ kubectl apply -f https://p8s.io/docs/k8s/manifests/prometheus/deploy.yaml
deployment.apps/prometheus created
☸ ➜ kubectl get pods -n kube-mon
NAME READY STATUS RESTARTS AGE
prometheus-df4f47d95-vksmc 0/1 CrashLoopBackOff 3 98s
☸ ➜ kubectl logs -f prometheus-df4f47d95-vksmc -n kube-mon
level=info ts=2019-12-12T03:08:49.424Z caller=main.go:332 msg="Starting Prometheus" version="(version=2.14.0, branch=HEAD, revision=edeb7a44cbf745f1d8be4ea6f215e79e651bfe19)"
level=info ts=2019-12-12T03:08:49.424Z caller=main.go:333 build_context="(go=go1.13.4, user=root@df2327081015, date=20191111-14:27:12)"
level=info ts=2019-12-12T03:08:49.425Z caller=main.go:334 host_details="(Linux 3.10.0-1062.4.1.el7.x86_64 #1 SMP Fri Oct 18 17:15:30 UTC 2019 x86_64 prometheus-df4f47d95-vksmc (none))"
level=info ts=2019-12-12T03:08:49.425Z caller=main.go:335 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-12-12T03:08:49.425Z caller=main.go:336 vm_limits="(soft=unlimited, hard=unlimited)"
level=error ts=2019-12-12T03:08:49.425Z caller=query_logger.go:85 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log

goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffd8cf6ec5d, 0xb, 0x14, 0x2b4f400, 0xc0006f33b0, 0x2b4f400)
/app/promql/query_logger.go:115 +0x48c
main.main()
/app/cmd/prometheus/main.go:364 +0x5229

权限

创建 Pod 后,我们可以看到并没有成功运行,出现了 open /prometheus/queries.active: permission denied 这样的错误信息,这是因为我们的 prometheus 的镜像中是使用的 nobody 这个用户,然后现在我们通过 LocalPV 挂载到宿主机上面的目录的 ownership 却是 root

☸ ➜ ls -la /data/k8s
total 36
drwxr-xr-x 6 root root 4096 Dec 12 11:07 .
dr-xr-xr-x. 19 root root 4096 Nov 9 23:19 ..
drwxr-xr-x 2 root root 4096 Dec 12 11:07 prometheus

所以当然会出现操作权限问题了,这个时候我们就可以通过 securityContext 来为 Pod 设置下 volumes 的权限,通过设置 runAsUser=0 指定运行的用户为 root,也可以通过设置一个 initContainer 来修改数据目录权限:

......
initContainers:
- name: fix-permissions
image: busybox
command: [chown, -R, "nobody:nobody", /prometheus]
volumeMounts:
- name: data
mountPath: /prometheus

这个时候我们重新更新下 prometheus:

☸ ➜ kubectl apply -f https://p8s.io/docs/k8s/manifests/prometheus/deploy-fixed.yaml
deployment.apps/prometheus configured
☸ ➜ kubectl get pods -n kube-mon
NAME READY STATUS RESTARTS AGE
prometheus-649968556c-t4prd 1/1 Running 1 (14m ago) 27h
☸ ➜ kubectl logs -f prometheus-649968556c-t4prd -n kube-mon
ts=2021-12-15T06:55:25.752Z caller=main.go:444 level=info msg="Starting Prometheus" version="(version=2.31.1, branch=HEAD, revision=411021ada9ab41095923b8d2df9365b632fd40c3)"
ts=2021-12-15T06:55:25.752Z caller=main.go:449 level=info build_context="(go=go1.17.3, user=root@9419c9c2d4e0, date=20211105-20:35:02)"
ts=2021-12-15T06:55:25.752Z caller=main.go:450 level=info host_details="(Linux 3.10.0-1160.25.1.el7.x86_64 #1 SMP Wed Apr 28 21:49:45 UTC 2021 x86_64 prometheus-649968556c-t4prd (none))"
ts=2021-12-15T06:55:25.752Z caller=main.go:451 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2021-12-15T06:55:25.752Z caller=main.go:452 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2021-12-15T06:55:25.756Z caller=web.go:542 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2021-12-15T06:55:26.150Z caller=main.go:839 level=info msg="Starting TSDB ..."
......
ts=2021-12-15T06:55:27.048Z caller=main.go:869 level=info msg="TSDB started"
ts=2021-12-15T06:55:27.048Z caller=main.go:996 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2021-12-15T06:55:27.050Z caller=main.go:1033 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.555486ms db_storage=754ns remote_storage=38.847µs web_handler=433ns query_engine=852ns scrape=1.030952ms scrape_sd=73.933µs notify=894ns notify_sd=2.504µs rules=19.359µs
ts=2021-12-15T06:55:27.050Z caller=main.go:811 level=info msg="Server is ready to receive web requests."

Pod 创建成功后,为了能够在外部访问到 prometheus 的 webui 服务,我们还需要创建一个 Service 对象:

# prometheus-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: prometheus
namespace: kube-mon
labels:
app: prometheus
spec:
selector:
app: prometheus
type: NodePort
ports:
- name: web
port: 9090
targetPort: http

为了方便测试,我们这里创建一个 NodePort 类型的服务,当然我们可以创建一个 Ingress对象,通过域名来进行访问:

☸ ➜ kubectl apply -f https://p8s.io/docs/k8s/manifests/prometheus/svc.yaml
service "prometheus" created
☸ ➜ kubectl get svc -n kube-mon
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
prometheus NodePort 10.111.160.152 <none> 9090:30407/TCP 4m33s

现在我们就可以通过 http://任意节点IP:30407 访问 prometheus 的 webui 服务了:

prometheus webui

现在我们可以查看当前监控系统中的一些监控目标(Status -> Targets):

prometheus webui targets

由于我们现在还没有配置任何的报警信息,所以 Alerts 菜单下面现在没有任何数据,隔一会儿,我们可以去 Graph 菜单下面查看我们抓取的 prometheus 本身的一些监控数据了:

prometheus webui metrics

比如我们这里就选择 scrape_duration_seconds 这个指标,然后点击 Execute,就可以看到类似于下面的图表数据了:

prometheus webui query