Prometheus and Grafana
Last updated
Last updated
The Bifrost node and relayer also supports system metric monitoring. This guide will walk you through how to setup and to monitor your node and relayer.
In order to let your Bifrost node and relayer to collect Prometheus metrics, it must be manually enabled. To enable the Prometheus server of your node, the following CLI flags has to be provided and then restarted.
--prometheus-external
: This exposes the Prometheus exporter on all interfaces.
--prometheus-port <PORT>
: The default port will be set to 9615
. However, if port changes are required, then this flag must be provided.
In case that you're operating a full-node, to enable the Prometheus server of your relayer, the following parameters of your configuration YAML file has to be updated as below and restarted.
If it has been successfully restarted, in both of your services the following log will be printed at the initial launch.
This section contains how to install and setup Prometheus and Grafana by using Systemd.
First, create the directories required to store the configuration and executable files.
Copy the executable files to the /usr/local/bin/
directory.
Copy the console files to the /etc/prometheus
directory.
Once everything is done, remove the prometheus
directory.
First, create the directories required to store the configuration and executable files.
Install the AlertManager plugins required for Grafana.
Create the rules.yml
file that will give the rules for the AlertManager.
We are going to create 2 basic rules that will trigger an alert in case the instance is down or the CPU usage crosses 80%. Add the following lines and save the file.
The alertmanager.yml
file is used to set the external service that will be called when an alert is triggered. Here, we are going to use the Gmail notification.
Create the file in the following path.
And add the Gmail configuration to it and save the file as below.
In order to start Prometheus, it needs some configuration. Create a configuration yaml file in the following directory.
The configuration file should look as below.
Next, the Systemd configuration should be set for Prometheus. Create a configuration file in the following directory.
The configuration file should look as below.
Now, enable and start the service.
To test out if it all successfully worked, access YOUR_SERVER_IP_ADDRESS:9090
. If the Prometheus dashboard appears, it is good to go.
The Systemd configuration should be set for the NodeExporter. Crate a configuration file in the following directory.
The configuration file should look as below.
Now, enable and start the service.
The Systemd configuration should be set for the AlertManager. Crate a configuration file in the following directory.
The configuration file should look as below.
Now, enable and start the service.
Then enable and start the service with default configurations.
You can now access it by going to YOUR_SERVER_IP_ADDRESS:3000/login
. The default user and password is admin/admin
.
This section contains how to install and setup Prometheus and Grafana by using Docker.
The alert rules are pre-defined in the maintenance/prometheus/rules.yml
file. It contains 2 basic rules that will trigger an alert in case the instance is down or the CPU usage crosses 80%. The file will be provided as below.
The alertmanager.yml
file is used to set the external service that will be called when an alert is triggered. Here, we are going to use the Gmail notification.
The file locates in the maintenance/alertmanager/alertmanager.yml
directory. Then, add the Gmail configuration to it and save the file as below.
In order to start Prometheus, it needs some configuration. The configuration file locates at maintenance/prometheus/prometheus.yml
. For Full-Node operators who runs the node and relayer both, should manually uncomment the below "relayer" job.
If you have followed every processes above, return to the maintenance
directory and execute the following command.
You can now access it by going to YOUR_SERVER_IP_ADDRESS:3000/login
. The default user and password is admin/admin
.
If it is all set, create a new Prometheus datasource and input the URL
as http://localhost:9090
and then click “Save & Test” as below.
Then, create a new Prometheus AlertManager datasource and input the URL as http://localhost:9093
and then click "Save & Test" as below.
Next, the dashboard has to be imported. Access the "Dashboards" tab and click on "New" to import the dashboard as below.
Now, in the "Import via grafana.com" section, input the dashboard ID as 19207
and then click "Load" to continue.
If it has been successfully loaded, set the correct datasources that you have just created before. The Prometheus and the AlertManager has to be set correctly. Then click "Import" to continue.
In the meantime, if your node and relayer is running in the background, the collected metrics will be visualized as below.
Then, update your OS and install the latest Prometheus. You can check the latest releases by going to their GitHub repository under the page.
Now, install the NodeExporter. You can check the latest releases by going to their Github repository under the page.
Next, install the AlertManager. You can check the latest releases by going to their Github repository under the page.
For Gmail notification, you will need to generate an app password. We recommend you to use a dedicated email address for your alerts. In order to set-up follow this .
To visualize your Prometheus metrics, you should install Grafana, which queries the Prometheus server. The latest releases can be checked on their . Execute the following commands to install the necessary dependencies.
First, Docker and Docker Compose should be installed in your server. Then you can download the docker-compose.yml
file that is provided in Bifrost node's repository. Download the file by using the command below. The file will be located in the maintenance
directory.
For Gmail notification, you will need to generate an app password. We recommend you to use a dedicated email address for your alerts. In order to set-up follow this .