KiChain: How to use the Ki Monitor

Welcome to the Ki-steps series, a sequence of how-to posts where we explain the detailed steps to perform more or less advanced tasks on the KiChain. This post consists of a tutorial detailing the usage of the Ki Monitor.

In a previous post, we discussed the jailing process in the KiChain and argued that it is important for validators to have an active monitoring that alerts them when the number of missed blocks hits a worrying level. The Ki Monitor is a tool that does just this. Hereafter, we share a tutorial on how to deploy your own instance of the Ki Monitor and on configuring it to alert you on your private slack and telegram channels.

First, let's have a quick look at the architecture and the functionalities of this tool.

Architecture

The Ki Monitor consists of two main scripts: the validator monitor and the block monitor. The former monitors the blocks missed by a validator and the latter checks whether a node is always at the latest blockchain height by checking the latest local block height.

The two components of the Ki Monitor: the validator monitor (left) and the block monitor (right)

Functionalities

The validator monitor allows to:

• Track the number of missed block over set time windows.
• Send Slack messages to alert the operators.
• Send Telegram messages to alert the operators.
• Configure the level and thresholds of alerts.

The block monitor allows to:

• Ensure that the node is always synched to the highest block number.
• Send Slack messages to alert the operators.
• Monitor multiple APIs (nodes).

Configurations

The validator monitor is configured from the valmonitor section of config.json file. Following are the details of its parameters:

• api : the URL of the REST server to query.
• watcher: the ping URL of a Cron Job Monitoring service (such as healthchecks.io). To receive an alert if this script isn't running.
• test_interval_min: The time interval (in minutes) between the monitoring cycle.
• state: the path to a file where the state of each validator is stored at each monitor cycle.
• validators : an array with the operator addresses (tkivaloper1...) of the nodes to monitor. An empty array [] means that all the active validators are monitored.
• slack_users : the slack users to notify with their personal configuration. This is an object (see below).
• alert_thresholds : an object with key/values pair, where the keys are integers and values are the number of missed blocks that defines the severity of the alert.

The block monitor is configured from the blcmonitor section of the config.json file. Following are the details of its parameters:

• apis : an object with key/values pairs with the names of the REST servers to watch and their URLs.
• slack_hook : the Slack Webhook to send Slack alerts.
• watcher: the ping URL of a Cron Job Monitoring service (such as healthchecks.io). To receive an alert if this script isn't running.
• test_interval_min: The time interval (in minutes) between the monitoring cycle.

Both of the monitor can report their detected anomalies in telegram and slack. Thus They share some common configurations which are described in the following:

• tg_group: the ID of the telegram group where the monitor will post alerts.
• tg_bot_token: the Telegram bot token as generated through BotFather.
• slack_hook : the Slack Webhook to send Slack alerts.
• emoji : an object with key/values pair, where the keys are integers and values are emojis to be included in the Slack alert to reflect the severity of the alert.

How to run The Ki Monitor

In order to allow XKI holders to follow the validator health, we have deployed a public instance of the Ki Monitor. This instance posts alerts on the dedicated Slack and Telegramᵇᵉᵗᵃ channels. However, as a validator operator, you might be interested in running your own instance of the monitor with your own custom configurations. Here is how to do that.

Start by cloning the monitor repository :

git clone https://github.com/KiFoundation/ki-monitor.git

Then navigate to the monitor directory and install the project:

cd ki-monitor & npm install

Before you launch the monitor you need to fill in the missing configurations. The first step consists of creating the hooks that will be used by the monitor to post in the Slack and Telegram channels of your choice.

Creating a Slack app and hook is fairly easy and the whole process is well described here (no one can explain it better than Slack's people themselves! right?). If you follow the aforementioned tutorial, you will end up with the URL of the hook that you can use to post in the channel of you choice. Use this URL as your slack_hook in the config.json file.

Creating a Telegram hook is a little bit more tricky. First you will need to create a telegram bot using Telegram's BotFather. The process is explained in this section of telegram's bot documentation. In a nutshell, you need to send /newbot to @botfather on telegram. The BotFather will then ask you for a name and username, then generate an authorisation token for your new bot. The token looks something like 123456789:AAHdqTcvC.... Keep it safe! Fill in this token in the field tg_bot_token in the config.json file.

After creating the bot, search for its handle in the Telegram app to start a conversation with it. This conversation will be the place where you will be receiving the the monitor alerts. To link this conversation to your monitor script, we need to determine the chat ID to set the tg_group variable in the configuration file. To do so, send whatever message in your bot conversation, then navigate to :

https://api.telegram.org/bot<bot_token>/getUpdates

Where <bot_token> is your secret bot token generated earlier with BotFather. This will return a JSON object that contains the latest messages (in this case the whatever message you sent to your bot) and some other metadata. Identify the message.chat.id key and copy its value to the tg_group field in the config.json file.

Note : You can also create a group or a channel and add the bot to it. This will allow you to receive notifications in a conversation shared with other team members for instance. The tricky part here resides in determining the ID of the conversation. I will not enter into the details of this subject as it highly depends on your setup.

Now that you've created and configured the hooks, let's set the remaining configurations. We will start with the validator monitor configurations:

• Set the api variable to your node REST server URL.
• Set the watcher variable to your watcher hook or leave it empty if you don't want to use a watcher.
• Set the test_interval_min to what is convenient to your set up (e.g., your watcher timeout settings…).
• The default value of the state_file parameter is ./state.json you can change to redirect the state output or leave it as is otherwise.
• Add the validators you want to monitor to the validators array. If you leave it empty, your monitor will be watching all of the chain validators.
• Subscribe to the tagged Slack alerts by creating an entry in the slack_users object as follows:

"Your_operator_address":{
    slack_username : "<@your_slack_user_ID>"
    alert_threshold : the_alert_threshold
   }

As mentioned earlier, the monitor will send and alert once the first anomaly threshold i.e. notice is reached. However, it will not tag you in the alert unless the configured the_alert_threshold is reached. This allows you a higher flexibility in setting a "danger threshold" that depends on your expected intervention time and duration.

The block monitor has fewer configurations. Set the values of the watcher and test_interval_min as explained earlier for the validator monitor. Then fill in the apis object with the your API URLs or IPs as follow :

"apis": {
 "api-1": "https://api-1.your.domain",
 "api-2": "https://api-2.your.domain"
 },

Now that the required configuration are made, we can go ahead and launch the monitor. Here we will use a screen to run the monitor in the background. You can also use tmux or run the monitor as a service using systemd to ensure it will be automatically launched in the case of a system reboot.

To create and rename a screen session follow these steps:

• Run the screen command in your terminal
• Enter the screen Control sequence i.e. CTRL+a
• Type :sessionname <name_of_your_screen>

Now run the monitor with the following command line:

node monitor.js ./config.json <param2> &> monitor.log

This will run the script in deamon mode where ./config.json is the configuration file, and <param2> is a configuration to indicate the monitoring script to run. It can take 3 values: val to run the validator monitor, blc to run the block monitor and val,blc to run both. The output of the validator monitor should look something like this:

2020–10–19T08:55:38.590Z: Satrting a monitoring cycle
No validator needs to be alerted
2020–10–19T08:55:39.778Z: Monitoring cycle ended
Waiting for the next cycle …

The output of the block monitor should look something like this:

api-1
current_height_date 2020-10-26T15:03:50.117921367Z
current_height_delay 6.699
api-2
current_height_date 2020-10-26T15:03:46.486929922Z
current_height_delay 10.33

API heights are up to the date

you can now safely detach from your screen with CTRL+a followed by d. Note that to reattach to the screen you can use the following command:

screen -r <name_of_your_screen>

The Ki Monitor is meant to be an easy and fast-to-deploy — yet useful and practical — anomaly monitoring script. If you require more features that goes beyond alerting for anomalies or simply prefer Python over JavaScript, you might need to have a look to the Cosmos PANIC monitor by the KiChain validator Simply VC.

• Buy Ki Tokens
• Join our community on Slack
• Join our telegram account
• Add Us On Facebook
• Follow Us On Twitter
• Check Us Out On Github
• Connect With Us On LinkedIn