Prometheus Setup

The following is a guide outlining the steps to setup Prometheus to monitor a Tangle node. If you do not have Tangle node setup yet, please review the Tangle Node Quickstart setup guide here. It is important to note that this guide's purpose is to help you get started with monitoring your Tangle node, not to advise on how to setup a node securely. Please take additional security and privacy measures into consideration.

In this guide we will configure the following modules to scrape metrics from the running Tangle node.

• Prometheus is the central module; it pulls metrics from different sources to provide them to the Grafana dashboard and Alert Manager.
• Node exporter provides hardware metrics of the dashboard.
• Process exporter provides processes metrics for the dashboard (optional).

What is Prometheus?

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company. To learn more about Prometheus, please visit the official docs site here (opens in a new tab).

Getting Started

Let's first start by downloading the latest releases of the above mentioned modules (Prometheus, Process exporter, and Node exporter).

1. Download Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.40.3/prometheus-2.40.3.darwin-amd64.tar.gz

wget https://github.com/prometheus/prometheus/releases/download/v2.40.3/prometheus-2.40.3.darwin-arm64.tar.gz

2. Download Node Exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.40.0/node_exporter-1.4.0.darwin-amd64.tar.gz

wget https://github.com/prometheus/node_exporter/releases/download/v1.40.0/node_exporter-1.4.0.darwin-arm64.tar.gz

3. Download Process Exporter

wget https://github.com/ncabatoff/process-exporter/releases/download/v0.7.10/process-exporter-0.7.10.linux-amd64.tar.gz

wget https://github.com/ncabatoff/process-exporter/releases/download/v0.7.10/process-exporter-0.7.10.linux-arm64.tar.gz

4. Extract the Downloaded Files:

Run the following command:

tar xvf prometheus-*.tar.gz &&
tar xvf node_exporter-*.tar.gz &&
tar xvf process-exporter-*.tar.gz

5. Copy the Extracted Files into /usr/local/bin:

We are first going to copy the prometheus binary:

sudo cp ./prometheus-*.linux-amd64/prometheus /usr/local/bin/

Next, we are going to copy over the prometheus console libraries:

sudo cp -r ./prometheus-*.linux-amd64/consoles /etc/prometheus &&
sudo cp -r ./prometheus-*.linux-amd64/console_libraries /etc/prometheus

We are going to do the same with node-exporter and process-exporter:

sudo cp ./node_exporter-*.linux-amd64/node_exporter /usr/local/bin/ &&
sudo cp ./process-exporter-*.linux-amd64/process-exporter /usr/local/bin/

6. Create Dedicated Users:

Now we want to create dedicated users for each of the modules we have installed:

sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus &&
sudo useradd --no-create-home --shell /usr/sbin/nologin node_exporter &&
sudo useradd --no-create-home --shell /usr/sbin/nologin process-exporter

7. Create Directories for Prometheus, and Process exporter:

sudo mkdir /var/lib/prometheus &&
sudo mkdir /etc/process-exporter

8. Change the Ownership for all Directories:

We need to give our user permissions to access these directories:

prometheus:

sudo chown prometheus:prometheus /etc/prometheus/ -R &&
sudo chown prometheus:prometheus /var/lib/prometheus/ -R &&
sudo chown prometheus:prometheus /usr/local/bin/prometheus

node_exporter:

sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

process-exporter:

sudo chown process-exporter:process-exporter /etc/process-exporter -R &&
sudo chown process-exporter:process-exporter /usr/local/bin/process-exporter

9. Finally, let's clean up these directories:

rm -rf ./prometheus* &&
rm -rf ./node_exporter* &&
rm -rf ./process-exporter*

Great! You have now installed and setup your environment. The next series of steps will be configuring each service.

Configuration

If you are interested to see how we configure the Tangle Network nodes for monitoring check out https://github.com/webb-tools/tangle/tree/main/monitoring (opens in a new tab).

Prometheus

Let’s edit the Prometheus config file and add all the modules in it:

sudo nano /etc/prometheus/prometheus.yml

Add the following code to the file and save:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - 'rules.yml'

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093

scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "substrate_node"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9615"]
  - job_name: "node_exporter"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9100"]
  - job_name: "process-exporter"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9256"]

• scrape_interval defines how often Prometheus scrapes targets, while evaluation_interval controls how often the software will evaluate rules.
• rule_files set the location of Alert manager rules we will add next.
• alerting contains the alert manager target.
• scrape_configs contain the services Prometheus will monitor.

You can notice the first scrap where Prometheus monitors itself.

Process exporter

Process exporter needs a config file to be told which processes they should take into account:

sudo touch /etc/process-exporter/config.yml
sudo nano /etc/process-exporter/config.yml

Add the following code to the file and save:

process_names:
  - name: "{{.Comm}}"
    cmdline:
    - '.+'

Service Setup

Prometheus

Create and open the Prometheus service file:

sudo touch /etc/systemd/system/prometheus.service &&
sudo nano /etc/systemd/system/prometheus.service

Add the following lines:

[Unit]
  Description=Prometheus Monitoring
  Wants=network-online.target
  After=network-online.target

[Service]
  User=prometheus
  Group=prometheus
  Type=simple
  ExecStart=/usr/local/bin/prometheus \
   --config.file /etc/prometheus/prometheus.yml \
   --storage.tsdb.path /var/lib/prometheus/ \
   --web.console.templates=/etc/prometheus/consoles \
   --web.console.libraries=/etc/prometheus/console_libraries
  ExecReload=/bin/kill -HUP $MAINPID

[Install]
  WantedBy=multi-user.target

Node exporter

Create and open the Node exporter service file:

sudo touch /etc/systemd/system/node_exporter.service &&
sudo nano /etc/systemd/system/node_exporter.service

Add the following lines:

[Unit]
  Description=Node Exporter
  Wants=network-online.target
  After=network-online.target

[Service]
  User=node_exporter
  Group=node_exporter
  Type=simple
  ExecStart=/usr/local/bin/node_exporter

[Install]
  WantedBy=multi-user.target

Process exporter

Create and open the Process exporter service file:

sudo touch /etc/systemd/system/process-exporter.service &&
sudo nano /etc/systemd/system/process-exporter.service

Add the following lines:

[Unit]
  Description=Process Exporter
  Wants=network-online.target
  After=network-online.target

[Service]
  User=process-exporter
  Group=process-exporter
  Type=simple
  ExecStart=/usr/local/bin/process-exporter \
   --config.path /etc/process-exporter/config.yml

[Install]
WantedBy=multi-user.target

Starting the Services

Launch a daemon reload to take the services into account in systemd:

sudo systemctl daemon-reload

Next, we will want to start each service:

prometheus:

sudo systemctl start prometheus.service

node_exporter:

sudo systemctl start node_exporter.service

process-exporter:

sudo systemctl start process-exporter.service

And check that they are working fine:

prometheus:

systemctl status prometheus.service

node_exporter:

systemctl status node_exporter.service

process-exporter:

systemctl status process-exporter.service

If everything is working adequately, activate the services!

prometheus:

sudo systemctl enable prometheus.service

node_exporter:

sudo systemctl enable node_exporter.service

process-exporter:

sudo systemctl enable process-exporter.service

Amazing! We have now completely setup our Prometheus monitoring and are scraping metrics from our running Tangle node.

You can view those metrics on the Prometheus dashboard by going to http://localhost:9090/metrics !