Introduction for Prometheus with Grafana
Prometheus with Grafana is an Open Source metric and alerting system. To monitor the Linux servers additionally, we need to install node exporter to get the way to look into each resource usage of Linux servers, It will show a brief in-depth by creating a cool different type of graphs.
The packages can be acquired from the default CentOS AppStream. However, to get the latest packages we need to create the repo for both.
Instead of installing from source, Let’s follow the safest way of installation from a yum repository. We get Prometheus package from below RPM repository.
Setup Prometheus Repository
Create the repository file.
# vim /etc/yum.repos.d/prometheus.repo
And append with the repo information.
[prometheus] name=prometheus baseurl=https://packagecloud.io/prometheus-rpm/release/el/$releasever/$basearch repo_gpgcheck=1 enabled=1 gpgkey=https://packagecloud.io/prometheus-rpm/release/gpgkey https://raw.githubusercontent.com/lest/prometheus-rpm/master/RPM-GPG-KEY-prometheus-rpm gpgcheck=1 metadata_expire=300
If we have CentOS 6, 7 or 8 this repository will get the required packages for up and run the Prometheus with Grafana.
# yum makecache # yum search prometheus -y # yum install prometheus2 -y
Locally cache the repo and install the package.
Running transaction Preparing : 1/1 Running scriptlet: prometheus2-2.22.1-1.el8.x86_64 1/1 Installing : prometheus2-2.22.1-1.el8.x86_64 1/1 Running scriptlet: prometheus2-2.22.1-1.el8.x86_64 1/1 Verifying : prometheus2-2.22.1-1.el8.x86_64 1/1 Installed: prometheus2-2.22.1-1.el8.x86_64 Complete! [root@mon ~]#
Enable the prometheus persistently and start the service.
# systemctl enable prometheus.service # systemctl start prometheus.service
Allow the firewall for 9090/tcp port number and reload the firewall service.
# firewall-cmd --permanent --add-port=9090/tcp # firewall-cmd --reload
Accessing Prometheus Dashboard
Right after allowing the port, navigate to anyone of web browser and access the prometheus IP or valid name.
Once access the prometheus dashboard, Navigate to status –> target
Here we should get the localhost prometheus as a target up and running. Let’s continue the setup with Grafana before moving to client side setup.
Setup Grafana Repository
Create the repo file for grafana.
# vim /etc/yum.repos.d/grafana.repo
Append with the grafana repo information.
[grafana] name=grafana baseurl=https://packages.grafana.com/oss/rpm repo_gpgcheck=1 enabled=1 gpgcheck=1 gpgkey=https://packages.grafana.com/gpg.key sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt
Build the yum cache and install the package.
# yum makecache # yum install grafana -y
Once it installed, enable the service persistently and start the grafana service.
# systemctl enable grafana-server # systemctl start grafana-server # systemctl status grafana-server
As we done before same rule with different port (3000) number of grafana.
# firewall-cmd --add-port=3000/tcp --permanent # firewall-cmd --reload # firewall-cmd --list-all
Here is the output for reference.
[root@monitor ~]# firewall-cmd --list-all public (active) target: default icmp-block-inversion: no interfaces: ens18 sources: services: cockpit dhcpv6-client ssh ports: 9090/tcp 3000/tcp protocols: masquerade: no forward-ports: source-ports: icmp-blocks: rich rules: [root@monitor ~]#
Accessing Grafana Dashboard
Fire-up anyone of web browser and navigate to the same URL with port number 3000. The default username and password are “admin“.
Once we login it will prompt to create a new password, use any complex password than the default one.
Adding Data Source
This is how the dashboard will look after a successful login, To show the graph info we need data, so let’s start to import from Prometheus data source.
Click on the gear icon at the left side bottom and select “Data Sources“
Click Add data source.
Here we will get different type of data source, at the first we should get the prometheus, Click select.
We have installed Prometheus and Grafana on the same server, Let’s change the URL to localhost.
Click save and test, it will save the source as shown below.
We don’t want to put a lot of effort in creating a good looking dashboard with the required information. Instead, let’s import one.
Navigate to + symbol in left side pane and select “Import“
We should get the below window, type the dashboard ID and click Load. This will import a json file from grafana.com
To get the right dashboard look into below URL. The dashboard ID can be copied from right side of each dashboard page.
+ --> Import --> 1860 --> load
The dashboard ID i’m using one is for node exporter full.
Select Prometheus from drop down list and click Import.
Now, i need to monitor one of Linux server, let’s add it.
Installing Client Packages
First, create the Prometheus repository as we did on the server-side. Then, we need to install Prometheus and node exporter on client-side.
# yum install prometheus2 node_exporter -y
Output for reference
Running transaction Preparing : 1/1 Running scriptlet: prometheus2-2.22.1-1.el8.x86_64 1/2 Installing : prometheus2-2.22.1-1.el8.x86_64 1/2 Running scriptlet: prometheus2-2.22.1-1.el8.x86_64 1/2 Running scriptlet: node_exporter-1.0.1-2.el8.x86_64 2/2 Installing : node_exporter-1.0.1-2.el8.x86_64 2/2 Running scriptlet: node_exporter-1.0.1-2.el8.x86_64 2/2 Verifying : node_exporter-1.0.1-2.el8.x86_64 1/2 Verifying : prometheus2-2.22.1-1.el8.x86_64 2/2 Installed: node_exporter-1.0.1-2.el8.x86_64 prometheus2-2.22.1-1.el8.x86_64 Complete! [root@gateway ~]#
Edit the yaml file and add the name of client.
# vi /etc/prometheus/prometheus.yml
Only we need to give a name and point the prometheus server (192.168.0.50) with port number 9100.
scrape_configs: - job_name: 'node_exporter' static_configs: - targets: ['localhost:9100'] to scrape_configs: - job_name: 'gateway_server' static_configs: - targets: ['192.168.0.50:9100']
Enable both Prometheus and Node exporter services persistently, Right after that start the service as well.
# systemctl enable prometheus.service # systemctl enable node_exporter.service # systemctl start prometheus.service # systemctl start node_exporter.service
Firewall for Client Ports
Allow the outbound traffic for 9100/tcp.
# firewall-cmd --add-port=9100/tcp --permanent # firewall-cmd --reload # firewall-cmd --list-all
Output after allowing require port.
[root@gateway ~]# firewall-cmd --list-all public (active) target: default icmp-block-inversion: no interfaces: ens18 sources: services: cockpit dhcpv6-client ftp http mountd nfs nfs3 rpc-bind ssh ports: 9100/tcp protocols: masquerade: no forward-ports: source-ports: icmp-blocks: rich rules: [root@gateway ~]#
Next steps we need to configure on Prometheus server, Switch back to Prometheus Server.
Adding Target (Clients)
Add a target by editing the yaml configuration.
# vim /etc/prometheus/prometheus.yml
Just give a name and from which client or target you are getting the datas.
- job_name: 'gateway_server' static_configs: - targets: ['192.168.0.16:9100']
To make the changes effective, restart the Prometheus service.
# systemctl restart prometheus.service
Now we should get two targets.
Accessing the Dashboard
To access the dashboard, Click on “Dashboard” –> again click on “Node Exporter Full“
You should get the below dashboard now.
There are hundreds of metric information will be available for a single server, for reference few more attached below.
If you need to keep the dashboard for future use, we can save it by clicking on the Save button in the top right side of the dashboard, Give a name and save it.
Monitoring Hardware Temperatures
I have 4 numbers of Intel NUC in my home lab, they are screaming because of getting hotter. I need to know what is the current temperature of those NUC. Those tiny machines are running on ProxMox virtualization hosting minimum 8 Virtual machines each.
If we could have used InfluxDB then this could help to monitor those temperature.
Proxmox VE Sensors
Let’s stick with our Prometheus with node exporter.
Download and install the script to enable the required Repository for Prometheus.
# curl -s https://packagecloud.io/install/repositories/prometheus-rpm/release/script.deb.sh | sudo bash
Install the package.
# apt-get install prometheus prometheus-node-exporter
Make the required changes for in
- job_name: pve1 # If prometheus-node-exporter is installed, grab stats about the local # machine by default. static_configs: - targets: ['192.168.0.50:9100']
Restart the services.
# systemctl restart prometheus.service # systemctl restart prometheus-node-exporter.service # systemctl status prometheus.service # systemctl status prometheus-node-exporter.service
Once again, Back to Server side and add the client config under
- job_name: 'pve1' static_configs: - targets: ['192.168.0.11:9100']
Restart the prometheus service.
# systemctl start prometheus.service
Check from Dashboard to left corner and select the PVE1 from drop down list.
Kept a USB fan Next to NUC in full speed for one and half hour and this is the result. To conclude, Prometheus with Grafana show as something related to temperature as well.
That’s it, we have completed with monitoring a Linux server using Prometheus with Grafana.
Prometheus with Grafana monitoring will fire-up very cool dashboards by looking into each resource usage of Linux servers. Will come up with new guide related to Prometheus with Grafana, subscribe to newsletter and stay tuned.