Monitoring node metrics

In this chapter, we will learn how to monitor our entire network with the help of tools like Prometheus and Grafana.

Install Prometheus and Grafana -

Since this chapter is about using Prometheus and Grafana for substrate, we need these installed in our system. Depending on the operating system you have, you will need to follow different set of instructions for both Prometheus and Grafana.

1. For prometheus on Ubuntu, please follow this guide - https://serverspace.io/support/help/install-prometheus-ubuntu-20-04/

https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04 

1. For prometheus on windows, please follow the instructions - 

http://www.liferaysavvy.com/2021/07/prometheus-installation-on-windows.html

1. For prometheus on mac, please follow these - https://formulae.brew.sh/formula/prometheus 
2. If you want to be operating system agnostic, the best way to get prometheus is using Docker - https://prometheus.io/docs/prometheus/latest/installation/
3. For Grafana on Ubuntu, follow these steps - https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/ 
4. For Grafana on Windows, follow these steps - https://grafana.com/docs/grafana/latest/setup-grafana/installation/windows/ 
5. For Grafana on mac, follow these steps - https://grafana.com/docs/grafana/latest/setup-grafana/installation/mac/ 
6. You can also run a docker image of Grafana if you like to stay OS-agnostic - https://grafana.com/docs/grafana/latest/setup-grafana/installation/docker/ 

Configure Prometheus endpoint

Let’s first start a substrate node, start a terminal and cd into the project and then run the below command - 

./target/release/node-template --dev

• Open a new terminal shell on your computer.
• Change to the root of the Prometheus working directory.
• Open the prometheus.yml configuration file in a text editor.
• Add substrate_node as a scrape_config endpoint.

For example, add a section similar to the following:

# A scrape configuration containing exactly one endpoint to scrape:
scrape_configs:
  - job_name: "substrate_node"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9615"]

Start a Prometheus instance with the modified prometheus.yml configuration file.

For example, if you are currently in the Prometheus working directory and using the default configuration file name, start Prometheus by running the following command:

./prometheus --config.file prometheus.yml

OR USE THE BELOW COMMAND - 

sudo -u prometheus /usr/local/bin/prometheus
    --config.file /etc/prometheus/prometheus.yml
    --storage.tsdb.path /var/lib/prometheus/
    --web.console.templates=/etc/prometheus/consoles
    --web.console.libraries=/etc/prometheus/console_libraries

Open a new terminal shell and check the metrics retrieved for the Substrate node by running the following command:

curl localhost:9615/metrics

Configure Grafana data source

After you start Grafana, you can navigate to it in a browser.

• Open a browser and navigate to the port Grafana uses.
• By default, Grafana uses http://localhost:3000 unless you have configured a different host or port.
• Log in using the default admin user name and password admin, then click Log in.
• On the Welcome page, under the Configuration menu, click Data Sources.
• Click Prometheus to configure the Prometheus endpoint as the data source for Substrate node metrics.
• With both the Substrate node and Prometheus instance running, configure Grafana to look for Prometheus on its default port http://localhost:9090 or the port you configured Grafana to use if you customized the port information.
• You shouldn't specify the Prometheus port you set in the prometheus.yml file. That port is where your node is publishing its data.
• Click Save & Test to ensure that you have the data source set correctly.
• If the data source is working, you are ready to configure a dashboard to display node metrics.

Import template dashboard

If you want a basic dashboard to start, you can import a Substrate dashboard template to get basic information about your node.

To import the dashboard template:

• On the Grafana Welcome page, click Dashboards.
• In the left navigation, click Dashboards and select Browse.
• For the Search options, click New and select Import.
• Copy the Substrate dashboard template and paste it into the Import via panel json text box.
• Click Load.
• Review and modify, if necessary, the name, folder, and unique identifier for the dashboard.
• Select Prometheus (default), then click Import.