Using dnl_cloud_tool

Introduction

Package provides a set of applications:

  1. Database manager (dnl_cloud_dbman) - program, which performs data manipulations (backup, outdated files compression, storage cleanup), maintains the internal database and coordinates other applications. It is necessary to run database manager at least once before starting other programs.

  2. Files downloader (dnl_cloud_downloader) - program which provides access to the actual data (CDR and PCAP files).

  3. Database search (dnl_cloud_search) - program to filter CDR and PCAP by a set of parameters.

Configuration

Every program is fully configurable from PostgreSQL database, yet the global server parameters must be put in configuration file.

For complete list of parameters with their description and constrains see doc/class_cfg_tables.sql.

Server configuration

Server global configuration is placed in configuration file. Location of this file must be passed to each daemon in command-line arguments.

General

  • server_name - unique human-readable name of the server. It is used to identify server across the network.

  • sqlite_db - path to SQLite DB. On first run Database Manager will create database at this location.

SSL configuration

In order to run HTTP server with SSL support, you need to provide server certificate and key file.

  • key_pem_path - path to server key file

  • cert_pem_path - Path to server certificate

PostgreSQL configuration

  • pgsql_hostaddr - binding address. Default: connect to Unix-domain socket

  • pgsql_port - binding port. Default: 5432

  • pgsql_user - username. Default: postgres

  • pgsql_pass - account password. Default: none

  • pgsql_dbname - name of the database (mandatory)

See conf/dnl_cloud_solution.conf

Connecting remote storage

The following tables contain remote storage configuration:

  • dnl_cloud_storages - list of all storage connections

  • dnl_cloud_ftp_cfg - FTP connection settings

  • dnl_cloud_sftp_cfg - SFTP connection settings

  • dnl_cloud_gcloud_cfg - Google Cloud connection settings

In order to add new remote storage:

  1. Generate UUID and put UUID-type pair into the dnl_cloud_storages table.

  2. With generated UUID create new configuration in one of the tables dnl_cloud_ftp_cfg, dnl_cloud_sftp_cfg, and dnl_cloud_gcloud_cfg for FTP, SFTP or Google Cloud connections respectively.

  3. Put UUID into cdr_storage_cfg_uuid or pcap_storage_cfg_uuid field of dnl_cloud_dbman_cfg table for CDR and PCAP storage respectively.

The same configuration may be used for CDR and PCAP storage at the same time. However, it is recommended to use configurations with different workdir to prevent possible data overlap and confusion.

Remote files will be stored at:

  • CDR: /workdir/switch_name/YYYY/MM/DD.tar.gz

  • PCAP: /workdir/switch_name/YYYY/MM/YYYYMMDD.tar.gz

NOTE: Instead of UUID any unique name can be used.

HTTP server

Every program runs separate HTTP server. Configuration parameters have the same names and meaning between programs.

  • server_ip - sever bind address. Default: 127.0.0.1

  • server_port - server port. Default: 33500 - DB manager; 33501 - Downloader; 33502 - Search

  • server_use_ssl - enable SSL

Firewall

The following parameters accept comma-separated list of wildcards to allow or to block. Example: 127.0.0.1,192.168.*.*. By default all connections are allowed.

  • server_ip_whitelist

  • server_ip_blacklist

NOTE: Naturally you cannot have both lists enabled.

DB manager

  • keep_cdr, keep_pcap - compress local files older than specified amount of days for CDR and PCAP respectively. Default: 0 - do not compress local files.

  • keep_cdr_archives, keep_pcap_archives - remove local archives older than specified amount of days for CDR and PCAP files respectively. Default: 0 - do not remove local archives.

  • keep_cdr_remote, keep_pcap_remote - remove remote archives older than specified amount of days for CDR and PCAP files respectively. Default: 0 - do not remove remote archives.

  • hourly_cdr_backup- upload temporary CDR backup every hour

  • startup_delay - amount of seconds to wait after midnight before starting the backup. Default: 10800 - 03:00 AM. NOTE: 0 represents 00:00:59 (this minimum delay is required to let switch finish updating latest data).

  • run_on_start - run backup thread immediately on start. Otherwise run as scheduled in startup_delay.

Search

  • pcap_time_margin - time-window margin in seconds. This value is used to expand PCAP search time-window, which is necessary to ensure, that call has ended. Default: 60 seconds (recommended value).

  • req_exp_time - request expiration time after completion in seconds. Expired requests will be removed from the database automatically with all data. If set to 0, daemon will not cleanup requests. Default: 604800 (7 days).

  • threads_max - maximum amount of simultaneous processing threads. Default: 0 - unlimited

Switch detection

On start DB manager will try to automatically find all the running switches in system, which meet the following criteria:

  • Switch must be running.

  • Switch binary must be called dnl_softswitch.

  • Switch configuration file must be located near the binary (sub-directories are allowed).

You can add switches manually, using REST API (see bellow).

REST API

Content type

All POST methods allow both url-encoded and JSON parameters. However, when using JSON one must specify Content-Type header.

Example:

% curl -L -k "https://127.0.0.1:33502/cdr" -X POST -H "Content-Type: application/json" --data '{ "switch_name" : "class", "start_time" : 1582329600 , "end_time" : 1582329600 }'

% curl -L -k "https://127.0.0.1/cdr" -X POST --data 'switch_name=class&start_time=1582329600&end_time=1582329600'

App control functions

Every program supports the same list of control functions.

Get list of running applications with their status

Request: GET /status

Arguments: none

Return: JSON array with full list of servers

  • server_name - identification string of the server

  • ip - HTTP server binding IP address

  • port - HTTP server binding port

  • use_ssl - whether HTTP server uses SSL or not

  • is_online - whether HTTP server is online

  • operation - current server operation

Example:

% curl -L -k "https://127.0.0.1:33500/status"
{
  "code" : 200,
  "error" : null,
  "servers" :
  [
    {
      "server_name" : "class4-main",
      "ip" : "127.0.0.1",
      "port" : 33500,
      "use_ssl" : true,
      "is_online" : true,
      "operation" : "Database re-indexing"
    },
    {
      "server_name" : "class4-secondary",
      "ip" : "192.168.0.1",
      "port" : 11500,
      "use_ssl" : true,
      "is_online" : true,
      "operation" : null
    }
  ]
}

Common control functions

  • POST /server_name/reload - reload configuration. This must be called to notify daemon when configuration has changed.

  • POST /server_name/restart - restart the application.

  • POST /server_name/shutdown - terminate the application.

DB Manager

DB control operations

  • POST /server_name/reindex - manually trigger DB re-indexing

  • POST /server_name/run-backup - manually trigger backup operation

Get list of connected switches

Request: GET /server_name/switch/list

Arguments: none

Return: JSON array with list of switches

  • name - switch name

  • cdr_dir - path to CDR directory

  • pcap_dir - path to PCAP directory

  • autodetected - whether switch was automatically detected, or added manually

Example:

% curl -L -k "https://127.0.0.1:33500/class4-main/switch/list"
{
  "code" : 200,
  "error" : null,
  "data" : [
  {
    "name" : "class4",
    "cdr_dir" : "/opt/class4/dnl_softswitch/cdr",
    "pcap_dir" : "/opt/class4/dnl_softswitch/pcap",
    "autodetected" : true
  },
  {
    "name" : "class4-1",
    "cdr_dir" : "/opt/class4_1/dnl_softswitch/cdr",
    "pcap_dir" : "/opt/class4_1/dnl_softswitch/pcap",
    "autodetected" : false
  }
  ]
}

Add new switch to the database

Request: POST /switch/add

Arguments:

  • name - unique name (mandatory)

  • cdr_dir - path to CDR directory (mandatory)

  • pcap_dir - path to PCAP directory (mandatory)

Return: status of the operation

NOTE: provided CDR and PCAP directories must exist.

Remove switch from the database

Request: POST /switch/delete

Arguments:

  • name - name of the switch to remove (mandatory)

Return: status of the operation

Downloader

List all files on server

Request: GET /server_name/list/cdr - list CDR files; GET /server_name/list/pcap - list PCAP files

Arguments:

  • switch_name - name of the switch to filter

  • start_time, end_time - time filter (Unix timestamp)

Return: list of files available for downloading

  • switch_name - name of the switch

  • timestamp - Unix timestamp

  • has_backup - whether file has backup in remote storage or not

Example:

% curl -L -k "https://127.0.0.1:33501/class4-main/list/cdr?start_time=1582329600&end_time=1582416000"
{
  "code" : 200,
  "error" : null,
  "data" : [
  {
    "switch_name" : "class4",
    "timestamp" : 1582416000,
    "has_backup" : false
  },
  {
    "switch_name" : "class4",
    "timestamp" : 1582329600,
    "has_backup" : true
  },
  {
    "switch_name" : "class4-1",
    "timestamp" : 1582416000,
    "has_backup" : true
  },
  {
    "switch_name" : "class4-1",
    "timestamp" : 1582329600,
    "has_backup" : true
  }
  ]
}

List all switch files

Request: GET /list/cdr - list CDR files; GET /list/pcap - list PCAP files

Arguments:

  • switch_name - name of the switch to filter (mandatory)

  • start_time, end_time - time filter (Unix timestamp)

Return: list of files available for downloading

Download data

Request: GET /get/cdr - download CDR file; GET /get/pcap - download PCAP file

Arguments:

  • switch_name - name of the switch (mandatory)

  • timestamp - timestamp of data file (mandatory)

Return: compressed binary data (tar.gz archive for CDR; tar archive for PCAP)

Example:

% curl -L -k "https://127.0.0.1:33501/get/cdr?switch_name=class4&timestamp=1582329600" -O -J
curl: Saved to filename 'class4_20200222.tar.gz'

Search

Create CDR search request

Request: POST /cdr

Arguments:

  • switch_name - switch to filter (mandatory)

  • start_time - call start time since Unix Epoch (mandatory)

  • end_time - call end time since Unix Epoch (mandatory)

  • fields - CDR fields to printout (comma-separated list of digits [0;107])

  • ani, dnis - comma-separated list of phone numbers

  • ingress_trunk_id, egress_trunk_id - comma-separated list of trunk IDs

  • ingress_call_id, egress_call_id - comma-separated list of call IDs

Return: unique request ID

Example:

% curl -L -k "https://127.0.0.1:33502/cdr" --data "switch_name=class4&start_time=1582416000&end_time=1582417200"
{
  "code" : 200,
  "error" : null,
  "uuid" : "cb875266-bfa9-4e5e-aba9-0875508c8dae"
}

Create PCAP search request

Request: POST /pcap

Arguments:

  • switch_name - switch to filter (mandatory)

  • start_time - call start time since Unix Epoch (mandatory)

  • end_time - call end time since Unix Epoch (mandatory)

  • call_id - comma-separated list of Call IDs (mandatory)

  • ani, dnis - comma-separated list of phone numbers

Return: unique request ID

Get status of search request

Request: GET /<uuid>

Arguments: none

Return: request status

  • uuid - request UUID

  • type - type of request ("cdr" or "pcap")

  • created - request creation time

  • started - request start time

  • finished - request completion time

  • exp_time - request expiration time

  • status - request status ("invalid", "created", "processing", "complete", "failed")

  • error - error description

  • empty - whether output file is empty

  • downloaded - whether file was downloaded

  • list of requested arguments

Example:

% curl -L -k "https://127.0.0.1:33502/cb875266-bfa9-4e5e-aba9-0875508c8dae"
{
  "code" : 200,
  "error" : null,
  "data" : {
    "uuid" : "cb875266-bfa9-4e5e-aba9-0875508c8dae",
    "type" : "cdr",
    "created" : 1582481551,
    "started" : 1582481551,
    "finished" : 1582481597,
    "exp_time" : 1583086397,
    "status" : "complete",
    "error" : null,
    "empty" : false,
    "downloaded" : false,
    "switch_name" : "class4",
    "start_time" : 1582416000,
    "end_time" : 1582417200,
    "ani" : null,
    "dnis" : null,
    "fields" : null,
    "ingress_trunk_id" : null,
    "egress_trunk_id" : null,
    "ingress_call_id" : null,
    "egress_call_id" : null,
  }
}

List all search requests

Request: GET /list - list all search requests; GET /list/cdr - only CDR; GET /list/pcap - only PCAP.

Arguments:

  • start_time, end_time - time filter (Unix timestamp)

Return: list of non-expired search requests with full status (see "Get status of search request" section)

Download file

Request: GET /<uuid>/download

Arguments: none

Return: binary file (csv file for CDR; zip archive for PCAP)

Example:

% curl -L -k "https://127.0.0.1:33502/cb875266-bfa9-4e5e-aba9-0875508c8dae/download" -O -J
curl: Saved to filename '2020-02-23_1582416000-1582417200.csv'

Delete request

Request: DELETE /<uuid>

Arguments: none

Return: status of the operation

Example:

% curl -L -k "https://127.0.0.1:33502/cb875266-bfa9-4e5e-aba9-0875508c8dae" -X DELETE
{
  "code" : 200,
  "error" : null
}

Multi server setup

It is possible to synchronize many running daemons with each other, allowing unified access to all servers from a single IP address.

In such setup one of the servers will act as a main host, redirecting requests from UI to server with requested data.

Security

Neither of these programs support authorization, so be sure to enable SSL and setup firewall for every application. The firewall must allow connections only from server where UI is running, plus server which will act as the main access point for UI.

Deployment steps

  1. Install package on every server with running switch(-es).

  2. On each server set unique server_name in configuration file.

  3. Set paths to SSL key/cert files and PostgreSQL connection in configuration file on each server.

  4. Run dnl_cloud_dbman service on every server.

  5. Run dnl_cloud_downloader and dnl_cloud_search on every server.

Daemons will automatically create default configurations in PostgreSQL database.

Configuration steps

  1. For each entry in dnl_cloud_dbman_cfg, dnl_cloud_downloader_cfg and dnl_cloud_search_cfg tables set server_ip, server_port and server_use_ssl accordingly. Make sure that IP addresses are visible to the main access server, and that configured ports are not blocked by firewall. The default ports are: 33500 for DB manager; 33501 - downloader; 33502 - search.

  2. For each entry in these tables set server_ip_whitelist value to ip_addr_of_main_host,ip_address_of_UI.

  3. Reload or restart every application.

Test

Login to the server, which serves as an access point, and check status of each application:

% curl -L -k "https://<bind-address>:<port>/status

This command must return a list of recognized servers with their IP addresses.

PreviousUsing dnl_backup_toolNextGetting "Missing table" error during invoice generation

Last updated