# Using dnl\_cloud\_tool

### Introduction

Package provides a set of applications:

1. Database manager (`dnl_cloud_dbman`) - program, which performs data manipulations (backup, outdated files compression, storage cleanup), maintains the internal database and coordinates other applications. It is necessary to run database manager at least once before starting other programs.
2. Files downloader (`dnl_cloud_downloader`) - program which provides access to the actual data (CDR and PCAP files).
3. Database search (`dnl_cloud_search`) - program to filter CDR and PCAP by a set of parameters.

### Configuration

Every program is fully configurable from PostgreSQL database, yet the global server parameters must be put in configuration file.

For complete list of parameters with their description and constrains see `doc/class_cfg_tables.sql`.

#### Server configuration

Server global configuration is placed in configuration file. Location of this file must be passed to each daemon in command-line arguments.

**General**

* `server_name` - unique human-readable name of the server. It is used to identify server across the network.
* `sqlite_db` - path to SQLite DB. On first run Database Manager will create database at this location.

**SSL configuration**

In order to run HTTP server with SSL support, you need to provide server certificate and key file.

* `key_pem_path` - path to server key file
* `cert_pem_path` - Path to server certificate

**PostgreSQL configuration**

* `pgsql_hostaddr` - binding address. Default: connect to Unix-domain socket
* `pgsql_port` - binding port. Default: 5432
* `pgsql_user` - username. Default: postgres
* `pgsql_pass` - account password. Default: none
* `pgsql_dbname` - name of the database (mandatory)

See `conf/dnl_cloud_solution.conf`

#### Connecting remote storage

The following tables contain remote storage configuration:

* `dnl_cloud_storages` - list of all storage connections
* `dnl_cloud_ftp_cfg` - FTP connection settings
* `dnl_cloud_sftp_cfg` - SFTP connection settings
* `dnl_cloud_gcloud_cfg` - Google Cloud connection settings

**In order to add new remote storage:**

1. Generate UUID and put UUID-type pair into the `dnl_cloud_storages` table.
2. With generated UUID create new configuration in one of the tables `dnl_cloud_ftp_cfg`, `dnl_cloud_sftp_cfg`, and `dnl_cloud_gcloud_cfg` for FTP, SFTP or Google Cloud connections respectively.
3. Put UUID into `cdr_storage_cfg_uuid` or `pcap_storage_cfg_uuid` field of `dnl_cloud_dbman_cfg` table for CDR and PCAP storage respectively.

The same configuration may be used for CDR and PCAP storage at the same time. However, it is recommended to use configurations with different `workdir` to prevent possible data overlap and confusion.

Remote files will be stored at:

* CDR: `/workdir/switch_name/YYYY/MM/DD.tar.gz`
* PCAP: `/workdir/switch_name/YYYY/MM/YYYYMMDD.tar.gz`

**NOTE:** Instead of UUID any unique name can be used.

#### HTTP server

Every program runs separate HTTP server. Configuration parameters have the same names and meaning between programs.

* `server_ip` - sever bind address. Default: 127.0.0.1
* `server_port` - server port. Default: 33500 - DB manager; 33501 - Downloader; 33502 - Search
* `server_use_ssl` - enable SSL

Firewall

The following parameters accept comma-separated list of wildcards to allow or to block. Example: `127.0.0.1,192.168.*.*`. By default all connections are allowed.

* `server_ip_whitelist`
* `server_ip_blacklist`

**NOTE**: Naturally you cannot have both lists enabled.

#### DB manager

* `keep_cdr`, `keep_pcap` - compress local files older than specified amount of days for CDR and PCAP respectively. Default: 0 - do not compress local files.
* `keep_cdr_archives`, `keep_pcap_archives` - remove local archives older than specified amount of days for CDR and PCAP files respectively. Default: 0 - do not remove local archives.
* `keep_cdr_remote`, `keep_pcap_remote` - remove remote archives older than specified amount of days for CDR and PCAP files respectively. Default: 0 - do not remove remote archives.
* `hourly_cdr_backup`- upload temporary CDR backup every hour
* `startup_delay` - amount of seconds to wait after midnight before starting the backup. Default: 10800 - 03:00 AM. **NOTE:** 0 represents 00:00:59 (this minimum delay is required to let switch finish updating latest data).
* `run_on_start` - run backup thread immediately on start. Otherwise run as scheduled in startup\_delay.

#### Search

* `pcap_time_margin` - time-window margin in seconds. This value is used to expand PCAP search time-window, which is necessary to ensure, that call has ended. Default: 60 seconds (recommended value).
* `req_exp_time` - request expiration time *after completion* in seconds. Expired requests will be removed from the database automatically with all data. If set to 0, daemon will not cleanup requests. Default: 604800 (7 days).
* `threads_max` - maximum amount of simultaneous processing threads. Default: 0 - unlimited

### Switch detection

On start DB manager will try to automatically find all the running switches in system, which meet the following criteria:

* Switch must be running.
* Switch binary must be called `dnl_softswitch`.
* Switch configuration file must be located near the binary (sub-directories are allowed).

You can add switches manually, using REST API (see bellow).

### REST API

#### Content type

All POST methods allow both url-encoded and JSON parameters. However, when using JSON one must specify Content-Type header.

Example:

```
% curl -L -k "https://127.0.0.1:33502/cdr" -X POST -H "Content-Type: application/json" --data '{ "switch_name" : "class", "start_time" : 1582329600 , "end_time" : 1582329600 }'

% curl -L -k "https://127.0.0.1/cdr" -X POST --data 'switch_name=class&start_time=1582329600&end_time=1582329600'
```

#### App control functions

Every program supports the same list of control functions.

**Get list of running applications with their status**

Request: `GET /status`

Arguments: none

Return: JSON array with full list of servers

* `server_name` - identification string of the server
* `ip` - HTTP server binding IP address
* `port` - HTTP server binding port
* `use_ssl` - whether HTTP server uses SSL or not
* `is_online` - whether HTTP server is online
* `operation` - current server operation

Example:

```
% curl -L -k "https://127.0.0.1:33500/status"
{
  "code" : 200,
  "error" : null,
  "servers" :
  [
    {
      "server_name" : "class4-main",
      "ip" : "127.0.0.1",
      "port" : 33500,
      "use_ssl" : true,
      "is_online" : true,
      "operation" : "Database re-indexing"
    },
    {
      "server_name" : "class4-secondary",
      "ip" : "192.168.0.1",
      "port" : 11500,
      "use_ssl" : true,
      "is_online" : true,
      "operation" : null
    }
  ]
}
```

**Common control functions**

* POST `/server_name/reload` - reload configuration. This must be called to notify daemon when configuration has changed.
* POST `/server_name/restart` - restart the application.
* POST `/server_name/shutdown` - terminate the application.

#### DB Manager

**DB control operations**

* POST `/server_name/reindex` - manually trigger DB re-indexing
* POST `/server_name/run-backup` - manually trigger backup operation

**Get list of connected switches**

Request: `GET /server_name/switch/list`

Arguments: none

Return: JSON array with list of switches

* `name` - switch name
* `cdr_dir` - path to CDR directory
* `pcap_dir` - path to PCAP directory
* `autodetected` - whether switch was automatically detected, or added manually

Example:

```
% curl -L -k "https://127.0.0.1:33500/class4-main/switch/list"
{
  "code" : 200,
  "error" : null,
  "data" : [
  {
    "name" : "class4",
    "cdr_dir" : "/opt/class4/dnl_softswitch/cdr",
    "pcap_dir" : "/opt/class4/dnl_softswitch/pcap",
    "autodetected" : true
  },
  {
    "name" : "class4-1",
    "cdr_dir" : "/opt/class4_1/dnl_softswitch/cdr",
    "pcap_dir" : "/opt/class4_1/dnl_softswitch/pcap",
    "autodetected" : false
  }
  ]
}
```

**Add new switch to the database**

Request: `POST /switch/add`

Arguments:

* `name` - unique name (mandatory)
* `cdr_dir` - path to CDR directory (mandatory)
* `pcap_dir` - path to PCAP directory (mandatory)

Return: status of the operation

NOTE: provided CDR and PCAP directories must exist.

**Remove switch from the database**

Request: `POST /switch/delete`

Arguments:

* `name` - name of the switch to remove (mandatory)

Return: status of the operation

#### Downloader

**List all files on server**

Request: `GET /server_name/list/cdr` - list CDR files; `GET /server_name/list/pcap` - list PCAP files

Arguments:

* `switch_name` - name of the switch to filter
* `start_time`, `end_time` - time filter (Unix timestamp)

Return: list of files available for downloading

* `switch_name` - name of the switch
* `timestamp` - Unix timestamp
* `has_backup` - whether file has backup in remote storage or not

Example:

```
% curl -L -k "https://127.0.0.1:33501/class4-main/list/cdr?start_time=1582329600&end_time=1582416000"
{
  "code" : 200,
  "error" : null,
  "data" : [
  {
    "switch_name" : "class4",
    "timestamp" : 1582416000,
    "has_backup" : false
  },
  {
    "switch_name" : "class4",
    "timestamp" : 1582329600,
    "has_backup" : true
  },
  {
    "switch_name" : "class4-1",
    "timestamp" : 1582416000,
    "has_backup" : true
  },
  {
    "switch_name" : "class4-1",
    "timestamp" : 1582329600,
    "has_backup" : true
  }
  ]
}
```

**List all switch files**

Request: `GET /list/cdr` - list CDR files; `GET /list/pcap` - list PCAP files

Arguments:

* `switch_name` - name of the switch to filter (mandatory)
* `start_time`, `end_time` - time filter (Unix timestamp)

Return: list of files available for downloading

**Download data**

Request: `GET /get/cdr` - download CDR file; `GET /get/pcap` - download PCAP file

Arguments:

* `switch_name` - name of the switch (mandatory)
* `timestamp` - timestamp of data file (mandatory)

Return: compressed binary data (tar.gz archive for CDR; tar archive for PCAP)

Example:

```
% curl -L -k "https://127.0.0.1:33501/get/cdr?switch_name=class4&timestamp=1582329600" -O -J
curl: Saved to filename 'class4_20200222.tar.gz'
```

#### Search

**Create CDR search request**

Request: `POST /cdr`

Arguments:

* `switch_name` - switch to filter (mandatory)
* `start_time` - call start time since Unix Epoch (mandatory)
* `end_time` - call end time since Unix Epoch (mandatory)
* `fields` - CDR fields to printout (comma-separated list of digits \[0;107])
* `ani`, `dnis` - comma-separated list of phone numbers
* `ingress_trunk_id`, `egress_trunk_id` - comma-separated list of trunk IDs
* `ingress_call_id`, `egress_call_id` - comma-separated list of call IDs

Return: unique request ID

Example:

```
% curl -L -k "https://127.0.0.1:33502/cdr" --data "switch_name=class4&start_time=1582416000&end_time=1582417200"
{
  "code" : 200,
  "error" : null,
  "uuid" : "cb875266-bfa9-4e5e-aba9-0875508c8dae"
}
```

**Create PCAP search request**

Request: `POST /pcap`

Arguments:

* `switch_name` - switch to filter (mandatory)
* `start_time` - call start time since Unix Epoch (mandatory)
* `end_time` - call end time since Unix Epoch (mandatory)
* `call_id` - comma-separated list of Call IDs (mandatory)
* `ani`, `dnis` - comma-separated list of phone numbers

Return: unique request ID

**Get status of search request**

Request: `GET /<uuid>`

Arguments: none

Return: request status

* `uuid` - request UUID
* `type` - type of request ("cdr" or "pcap")
* `created` - request creation time
* `started` - request start time
* `finished` - request completion time
* `exp_time` - request expiration time
* `status` - request status ("invalid", "created", "processing", "complete", "failed")
* `error` - error description
* `empty` - whether output file is empty
* `downloaded` - whether file was downloaded
* list of requested arguments

Example:

```
% curl -L -k "https://127.0.0.1:33502/cb875266-bfa9-4e5e-aba9-0875508c8dae"
{
  "code" : 200,
  "error" : null,
  "data" : {
    "uuid" : "cb875266-bfa9-4e5e-aba9-0875508c8dae",
    "type" : "cdr",
    "created" : 1582481551,
    "started" : 1582481551,
    "finished" : 1582481597,
    "exp_time" : 1583086397,
    "status" : "complete",
    "error" : null,
    "empty" : false,
    "downloaded" : false,
    "switch_name" : "class4",
    "start_time" : 1582416000,
    "end_time" : 1582417200,
    "ani" : null,
    "dnis" : null,
    "fields" : null,
    "ingress_trunk_id" : null,
    "egress_trunk_id" : null,
    "ingress_call_id" : null,
    "egress_call_id" : null,
  }
}
```

**List all search requests**

Request: `GET /list` - list all search requests; `GET /list/cdr` - only CDR; `GET /list/pcap` - only PCAP.

Arguments:

* `start_time`, `end_time` - time filter (Unix timestamp)

Return: list of non-expired search requests with full status (see "Get status of search request" section)

**Download file**

Request: `GET /<uuid>/download`

Arguments: none

Return: binary file (csv file for CDR; zip archive for PCAP)

Example:

```
% curl -L -k "https://127.0.0.1:33502/cb875266-bfa9-4e5e-aba9-0875508c8dae/download" -O -J
curl: Saved to filename '2020-02-23_1582416000-1582417200.csv'
```

**Delete request**

Request: `DELETE /<uuid>`

Arguments: none

Return: status of the operation

Example:

```
% curl -L -k "https://127.0.0.1:33502/cb875266-bfa9-4e5e-aba9-0875508c8dae" -X DELETE
{
  "code" : 200,
  "error" : null
}
```

### Multi server setup

It is possible to synchronize many running daemons with each other, allowing unified access to all servers from a single IP address.

In such setup one of the servers will act as a main host, redirecting requests from UI to server with requested data.

**Security**

Neither of these programs support authorization, so be sure to enable SSL and setup firewall for every application. The firewall must allow connections only from server where UI is running, plus server which will act as the main access point for UI.

**Deployment steps**

1. Install package on every server with running switch(-es).
2. On each server set unique server\_name in configuration file.
3. Set paths to SSL key/cert files and PostgreSQL connection in configuration file on each server.
4. Run `dnl_cloud_dbman` service on every server.
5. Run `dnl_cloud_downloader` and `dnl_cloud_search` on every server.

Daemons will automatically create default configurations in PostgreSQL database.

**Configuration steps**

1. For each entry in `dnl_cloud_dbman_cfg`, `dnl_cloud_downloader_cfg` and `dnl_cloud_search_cfg` tables set `server_ip`, `server_port` and `server_use_ssl` accordingly. Make sure that IP addresses are visible to the main access server, and that configured ports are not blocked by firewall. The default ports are: 33500 for DB manager; 33501 - downloader; 33502 - search.
2. For each entry in these tables set `server_ip_whitelist` value to `ip_addr_of_main_host,ip_address_of_UI`.
3. Reload or restart every application.

**Test**

Login to the server, which serves as an access point, and check status of each application:

```
% curl -L -k "https://<bind-address>:<port>/status
```

This command must return a list of recognized servers with their IP addresses.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://v5cookbook.denovolab.com/maintainence/how-to-setup-auto-cdr-cleaning/using-dnl_cloud_tool.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.