The Bifrost node and relayer also supports system metric monitoring. This guide will walk you through how to setup Prometheus and Grafana to monitor your node and relayer.
Enable Prometheus
In order to let your Bifrost node and relayer to collect Prometheus metrics, it must be manually enabled. To enable the Prometheus server of your node, the following CLI flags has to be provided and then restarted.
--prometheus-external : This exposes the Prometheus exporter on all interfaces.
--prometheus-port <PORT> : The default port will be set to 9615. However, if port changes are required, then this flag must be provided.
In case that you're operating a full-node, to enable the Prometheus server of your relayer, the following parameters of your configuration YAML file has to be updated as below and restarted.
Create the rules.yml file that will give the rules for the AlertManager.
sudo vi /etc/prometheus/rules.yml
We are going to create 2 basic rules that will trigger an alert in case the instance is down or the CPU usage crosses 80%. Add the following lines and save the file.
groups:
- name: alert_rules
rules:
- alert: InstanceDown
expr: up == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Instance $labels.instance down"
description: "[{{ $labels.instance }}] of job [{{ $labels.job }}] has been down for more than 1 minute."
- alert: HostHighCpuLoad
expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 80
for: 0m
labels:
severity: warning
annotations:
summary: Host high CPU load (instance bLd Kusama)
description: "CPU load is > 80%\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
The alertmanager.yml file is used to set the external service that will be called when an alert is triggered. Here, we are going to use the Gmail notification.
For Gmail notification, you will need to generate an app password. We recommend you to use a dedicated email address for your alerts. In order to set-up follow this link.
Create the file in the following path.
sudo vi /etc/alertmanager/alertmanager.yml
And add the Gmail configuration to it and save the file as below.
To visualize your Prometheus metrics, you should install Grafana, which queries the Prometheus server. The latest releases can be checked on their download page. Execute the following commands to install the necessary dependencies.
You can now access it by going to YOUR_SERVER_IP_ADDRESS:3000/login. The default user and password is admin/admin.
Using Docker
This section contains how to install and setup Prometheus and Grafana by using Docker.
Requirements
First, Docker and Docker Compose should be installed in your server. Then you can download the docker-compose.yml file that is provided in Bifrost node's Github repository. Download the file by using the command below. The file will be located in the maintenance directory.
git clone https://github.com/bifrost-platform/bifrost-node.git
cd bifrost-node/maintenance
Configure AlertManager
The alert rules are pre-defined in the maintenance/prometheus/rules.yml file. It contains 2 basic rules that will trigger an alert in case the instance is down or the CPU usage crosses 80%. The file will be provided as below.
groups:
- name: alert_rules
rules:
- alert: InstanceDown
expr: up == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Instance $labels.instance down"
description: "[{{ $labels.instance }}] of job [{{ $labels.job }}] has been down for more than 1 minute."
- alert: HostHighCpuLoad
expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 80
for: 0m
labels:
severity: warning
annotations:
summary: Host high CPU load (instance bLd Kusama)
description: "CPU load is > 80%\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
The alertmanager.yml file is used to set the external service that will be called when an alert is triggered. Here, we are going to use the Gmail notification.
For Gmail notification, you will need to generate an app password. We recommend you to use a dedicated email address for your alerts. In order to set-up follow this link.
The file locates in the maintenance/alertmanager/alertmanager.yml directory. Then, add the Gmail configuration to it and save the file as below.
In order to start Prometheus, it needs some configuration. The configuration file locates at maintenance/prometheus/prometheus.yml. For Full-Node operators who runs the node and relayer both, should manually uncomment the below "relayer" job.
If you have followed every processes above, return to the maintenance directory and execute the following command.
docker compose up -d
You can now access it by going to YOUR_SERVER_IP_ADDRESS:3000/login. The default user and password is admin/admin.
Datasource Configuration
If it is all set, create a new Prometheus datasource and input the URL as http://localhost:9090 and then click “Save & Test” as below.
For Docker users, the URL should be set to http://prometheus:9090.
Then, create a new Prometheus AlertManager datasource and input the URL as http://localhost:9093 and then click "Save & Test" as below.
For Docker users, the URL should be set to http://alertmanager:9093.
Next, the dashboard has to be imported. Access the "Dashboards" tab and click on "New" to import the dashboard as below.
Now, in the "Import via grafana.com" section, input the dashboard ID as 19207 and then click "Load" to continue.
If it has been successfully loaded, set the correct datasources that you have just created before. The Prometheus and the AlertManager has to be set correctly. Then click "Import" to continue.
In the meantime, if your node and relayer is running in the background, the collected metrics will be visualized as below.
