Configure Backups with Archivist  

Introduction

Configure a Backup Procedure Using the Default Settings

Configure a Backup Procedure Task by Task

Create a New Remote Destination

Remove Scheduled Backup Task

Run a Backup Task Manually

Check Backup Tasks in the Calendar View

Check Backup Logs, Stop Running Backup Tasks and Remove Backup Files in the Logs Viewer

Data Recovery

 

Introduction

The Archivist utility allows you to configure and create backups to make sure that the most critical data on your installation is preserved. You can back up the following kinds of data:

·      MySQL database (hot backups performed while the database is running will not interfere with normal database operations).

·       Cassandra database (storage for invoices, xDRs, reports, etc.).

·       Recorded phone conversations.

·       The general configuration of a server’s operating system (the “/etc” folder).

·       Applied custom modifications (added to the deposit files).

·       Custom RPM files.

·       Billing statistics.

·       Voice prompt files.

·       Radacc files.

·       Elasticsearch database (storage for log files and CDR collections)

How to access Archivist

You can access Archivist from the Configuration server web interface:

1.     Log in to the Configuration server web interface.

2.     Click the house Home button in the right-hand corner of the page.

3.     Click the Archivist button.

 

Access Archivist

 

Configure a Backup Procedure Using the Default Settings

timesaver You can use preset settings to configure a backup procedure. In this case you specify a time period during which the system should perform a backup. The system selects the kinds of data that it’s possible to back up within this period and will back up these kinds of data to the Configuration server from all the servers that contain them. However, backup parameters can be adjusted later as described in this section and in the Configure a Backup Procedure Task by Task section.

Note that when you apply default settings all currently configured backup tasks are removed.

1.     At the top of the Tasks panel, click the add Default settings button.

 

Input time period for backup tasks

 

 

2.     Specify the time period during which the backup procedure should be executed. Time period should be specified in the format “hr{hh:mm-hh:mm}.”

 

For example, if you specify hr{00:00-03:30}, it means that the backup procedure will start after 00:00 and finish before 03:30. If the backup procedure is not completed before 03:30, the system terminates the procedure.

 

3.     Select Remove all current tasks to confirm that you know that the default settings will override the current ones.

4.     Click Next.

 

Review and adjust the list of backup tasks to be executed.

 

5.     You can review and adjust the list of backup tasks to be executed.

 

Adjust the scheduled time and frequency of backup

 

·      To adjust the scheduled time and frequency of backup, click on the Schedule (in cron format) column and type the required value in the cron format, i.e. “Minute(0,..,59) Hour(0,..,23) Day(1,..,31) Month(1,..,12 or Jan,..,Dec) Weekday(0,1,..,7 or Sun,Mon,..,Sun).” Use a hyphen to indicate a range of values, a comma to indicate a list of values and an asterisk * to indicate every possible value.

 

For example, if you specify Schedule (in cron format): 30 2 5,25 * 5-7, then your backup procedure will be scheduled to be performed at 02:30 on the 5th and 25th of every month as well as on every Friday, Saturday and Sunday.

 

 

Adjust the timeout

 

·      To adjust the timeout, click on the Timeout (min.) field and specify the required value. If the backup task is not completed within the timeout period, the system terminates the task.

 

For example, if you specify Timeout (min.): 180, then the backup process should finish within 3 hours from starting. 

 

_Hint_ExclamationMark Make sure that you give your system enough time to complete the backup task. Many kinds of data require no more than 10 minutes, but for example, a MySQL database backup procedure can take more than 2 hours.

 

Adjust the number of recent backups to be kept

 

·      To adjust how many of the most recent backups the system should keep, click on the Backup count field and specify the required value.

·      To remove the task click on the remove Remove button.

 

6.     Click Apply.

7.     If necessary, now you can adjust settings for every backup task as described in steps 1–7 of the Configure a Backup Procedure Task by Task section.

Configure a Backup Procedure Task by Task

A backup task is a combination of a type of data that you want to back up, a backup source and backup destinations. For example, one can define such a backup task as, “Backup statistics data from PortaBilling® Web server to the Configuration server and to the remote FTP server.” To configure a backup procedure task by task, please complete the following steps.

 

Configure a backup procedure task by task

 

1.     In the Tasks panel, click on the kind of data that you want to back up (e.g. “Backup statistics”).

 

Select the data to backup

 

2.     In the Backup Source panel, select the check boxes against the servers where this data is located (e.g. select your PortaSwitch® Web servers if your statistics are located on them).

 

Select the data location

 

·      _Hint_Glasses If you want to review which instances are configured on which server, click the expand-collapse Expand/Collapse button at the Backup Source panel top.

 

Review the instances configured

 

3.     To adjust backup parameters, in the Backup Source panel, click on the first of the backup sources selected in the previous step.

 

_Hint_ExclamationMark Be careful to click away from the check box, because when you click on the check box and clear it, you remove the backup source. 

 

Adjust backup parameters

 

4.     In the Schedule panel, specify when the backup should be performed, how long the backup process can be active for and how many of the most recent backups the system must keep.

·      Minute – Type minutes in diapason from 0 to 59 or *, where * corresponds to every minute.

·      Hour – Type hours in diapason from 0 to 23 or *, where * corresponds to every hour.

·      Day – Type days of the month in diapason from 1 to 31 or *, where * corresponds to every day.

·      Month – Type months in diapason from 1 to 12 or *, where * corresponds to every month. You can use Jan, …, Dec as well.

·      Weekday – Type weekday numbers in diapason from 0 to 6 or *, where 0 (and 7) correspond to Sunday, 6 to Saturday and so on and where * sets the weekday to undefined. You can use Mon, …, Sun as well.

·      Timeout – Type the maximum time in minutes that you want to allocate for the backup process. If the process is not completed within this period from its start, the system terminates it.

 

Make sure that you give your system enough time to complete the backup task. Many kinds of data require no more than 10 minutes, but for example, a MySQL database backup procedure can take more than 2 hours.

 

·      Backup count – Type the number of the most recent backups to keep.  

 

Use a hyphen to indicate a range and a comma to indicate a list of values.

 

For example, if you specify Minute: 30; Hour: 2; Day: 5,25; Month: *; Weekday: 5-7; Timeout: 180; Backup count: 5; then your backup procedure will be scheduled to be performed at 02:30 on the 5th and 25th of every month as well as on every Friday, Saturday and Sunday. The backup process should finish within 3 hours from its start, and the system will keep the most recent 5 backups of this data.

 

You can review the descriptions of configured schedule settings and see exactly when the next backup will be performed on the bottom of the Schedule panel.

 

_Hint_Glasses In addition, after several backup runs, you can see how long it takes to back up selected types of data. Check the following performance characteristics:

·      Avg execution time shows how long it takes to copy data to the final destination (if single destination is used) or to the Configuration server (used as an intermediary between a backup source and final destinations when you select more than one or an external destination). This value is of higher importance because the source server is actively engaged in the backup procedure during this time.

·      Avg upload time shows how long it takes to copy data from the Configuration server to final destinations.

 

Utilize this information to adjust the time frames for backup procedures if required.

 

Note that to provide the best performance and to avoid mistakes it is recommended that backup procedures be scheduled for off-peak periods. Make sure they don’t overlap with resource-intensive calculations such as creating statistics / invoices.

 

 

Schedule the backup procedure

 

5.     In the Backup Destination panel, select servers where files with backup data will be placed. If you want to add a new remote destination, please refer to the Create a New Remote Destination section of this handbook.

 

Note that when you select more than one destination or if you select an external destination, the configurator backup destination is automatically selected too. The reason for this is that sending backups to external or several destinations creates an additional load on the server, therefore it’s safer to perform this operation from the server that is not actively involved in performing critical real-time services – the Configuration server. So the system first copies backup files to the Configuration server and only then copies them from the Configuration server to specified destinations. (Note that the backup copy on the Configuration server won’t be deleted after the backup task finishes.)

 

_Hint_Glasses It’s recommended to choose more than one destination per source.

 

_Hint_ExclamationMark Please make sure that there is enough free space on the destination servers.

 

6.     If you have selected more than one backup source in step 2, repeat steps 4–5 for the rest of them. 

7.     Click the Save button at the bottom of the page.

8.     If you want to configure the backup procedure for any other kinds of data, repeat steps 1–7.

Create a New Remote Destination

1.     At the top of the Backup Destination panel, click the add Add button.

 

Create a new remote destination

 

2.     In the Add custom destination dialog box, specify the following options:

·      Server name – Type the name under which you want this destination to be shown in the list.

·      Protocol – Select the protocol to be used to send files to the remote destination. Available options are “ftp,” “ssh (scp),” and “rsync over ssh.”

·      Path – Specify the folder on your server where the system will save your backup files, use the following format: “/folder_name”. The system creates two additional nested subfolders:  “/IP_of_the_server_it_saves_data_from/A_kind_of_data_you_are_backing_up.”  

 

For example, if you specify “/backup” for this option and you back up statistics data from the server with IP “10.10.10.10,” the system will save your data in: 

/home/backup/10.10.10.10/statistics.

 

·      Hostname – Type the IP address of your destination server.

·      Port – Type the port for connection.  

·      Login – Type the login for authorization on the destination server.

·      Password – Type the password for authorization on the destination server.

·      Login Script – If required, for ssh (scp) and rsync over ssh protocols you may specify the path to the login script file. The login script includes a set of commands that are executed each time a user logs into the remote destination via an ssh connection. The script is mostly used to allow password-less user authentication. The Login parameter is still mandatory here to explicitly define the user who connects to the remote destination.

 

The login script can also be used to add additional ssh connection options such as connect request timeout, number of connection attempts, etc.

 

3.     Click Save.

Remove Scheduled Backup Task

1.     In the Task panel, click on the kinds of data that you no longer want to back up.

2.     In the Backup Source panel, clear the check boxes against the servers where this data is located.

3.     Click the Save button at the bottom of the page.

4.     If you don’t want to back up any other kinds of data, repeat steps 1–3.

 

_Hint_ExclamationMark You need to repeat all three steps for each kind of data that you no longer want to back up. If you clear the check boxes against the servers where this data is located for one kind of data and then move to another without clicking Save, then all the changes that you made will be lost.

Run a Backup Task Manually

The main purpose of this option is to give you a possibility to check whether a backup task configuration is operable: that the chosen backup source contains the required data and that the connection to the backup destination works properly. We recommend that you always double-check a backup task configuration when you select remote destinations.

 

Run a backup task manually

 

1.     In the Tasks panel, click on the kind of data that you configured the backup task for (e.g., “Backup statistics”).

2.     In the Backup Source panel, click on the required backup source.

 

_Hint_ExclamationMark Be careful to click away from the check box, because when you click on the check box and clear it, you remove the backup source. 

 

3.     At the top of the Schedule panel, click the runnow Run Now button.

 

Check Backup Tasks in the Calendar View

Use the Calendar view to get an overview of forthcoming and finished backup tasks by month, week or day.

 

Open the calendar

 

To open the Calendar view, click the calendar Show Calendar button at the top of the Schedule panel.

 

Check backup tasks in the calendar view

Check Backup Logs, Stop Running Backup Tasks and Remove Backup Files in the Logs Viewer

The system prompts you to open the Logs Viewer every time you run the backup procedure manually (for how to perform this, see the Run a Backup Task Manually  section). However, you can check your backup logs with the Logs Viewer anytime you need to.   

 

Open the log viewer

 

To open the Logs Viewer click the log Logs button at the top of the Schedule panel.

 

Check the stage of the backup procedure

 

The Logs Viewer shows the logs at the run time of the backup process. So you can always check what stage the backup procedure is working on.

·      To review a log, select the backup task from the list in the left-hand panel. You can review log details in the right-hand panel.

·      To terminate the currently running backup procedure, select it in the left-hand panel and click the killtask Kill task button.

·      To delete a backup and all the logged information about it, select the backup in the left-hand panel and click the delete Delete button.

·      To refresh a log immediately click the refresh Refresh button or set it to refresh automatically by clicking the arrow on the right-hand side of the Refresh button and selecting the required interval from the list.

·      To close the Log Viewer, click the closelogviewer Close button in the top right-hand corner of the page.

 

The backup tasks in the left-hand panel are grouped by backup sources and kinds of backup data. The icons near the backup task items indicate information about the status of the backup tasks. Backup tasks can have the following status:

·      startedtask Started – The backup task is waiting for processing.

·      running Running – The backup task is being executed.

·      donetask Done – The data was copied to the backup destination (when only one backup destination was selected) or to the Configuration server (when more than one destination was selected), but the synchronization script is still being executed.

·      successtask Synced – The backup task has successfully finished.

·      killedtask Killed – The backup task was manually terminated.

·      failedtask Fail – The backup task wasn’t completed because of errors.

·      umknowntask Unknown status – The system cannot identify the backup task status.

 

Data Recovery

The recovery procedure can be manually executed from the CLI (command-line interface). Please contact support@portaone.com for more information on recovery procedures.

How to use rsync for remote hosts

If your backup is located on another server you can perform rsync command remotely.

To copy something from the remote host

Executed on a restoration host.

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

  --rsh=/home/porta-one/scripts/rsh_porta.sh \

  <ip_address>:/remote/path/to/file /local/path/to/file


To copy something from the remote host

Executed on a host with backups.

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

  --rsh=/home/porta-one/scripts/rsh_porta.sh \

  /local/path/to/file <ip_address>:/remote/path/to/file

How to restore /etc

Executed on a restoration host.

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

  --rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/etc_dir/<date_time>/_etc/ /etc/

How to restore Cassandra

  _Hint_ExclamationMark If restoring a single node, you must first shutdown the node. If restoring an entire cluster, you must shut down all nodes, restore the snapshot data, and then start all nodes again.

Note that the process must be executed on all nodes in the cluster, otherwise nodes that did not get the restored data will “update” the restored nodes with the newer, bad data.

 

1.     Shut down the Cassandra node.

Executed on a restoration host.

 

sudo systemctl stop porta-mysqld

 

2.     Clear all files in /var/lib/cassandra/commitlog/ .

Executed on a restoration host.

 

sudo rm -rf /porta_var/<ip_address>/cassandra/commitlog/*.log

 

Delete all .db files in the cassandra directory /porta_var/<ip_address>/cassandra/data/ <keyspace_name>/<table_name>/
Executed on a restoration host.

 

sudo rm -rf /porta_var/<ip_address>/cassandra/data/<keyspace_name>/<table_name>/*

 

3.     Find the most recent snapshot folder in this directory


/porta_var/backup/<ip_address>/cassandra/<date_time>/porta_var_<cassandra_node>_cassandra_data_<keyspace_name>_<table_name>_snapshots_incremental

 

4.     Copy its contents into this directory /porta_var/<ip_address>/cassandra/data/<keyspace_name>/<table_name>/
Executed on a restoration host.

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

  --rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/cassandra/<date_time>/_porta_var_<cassandra_node>_cassandra_data_<keyspace_name>_<table_name>_snapshots_incremental/ \

/porta_var/<ip_address>/cassandra/data/<keyspace_name>/<table_name>/

 

5.     Restart the Cassandra node.

Executed on a restoration host.

 

sudo systemctl start cassandra

 

6.     Restarting causes a temporary burst of I/O activity and consumes a large amount of CPU resources. Run nodetool repair.

Executed on a restoration host.

 

sudo nodetool -h localhost -p 7199 repair

 

How to restore mysql

1.     Create a new directory on restoration host where you will copy the backup file with the mysql database.

Executed on a restoration host.


mkdir /porta_var/tmp/mysql_restore_$(date -I)

 

2.     Copy the backup file with the mysql database locally.

Executed on a restoration host.

 

cd /porta_var/tmp/mysql_restore_$(date -I)

rsync -v --rsh=/home/porta-one/scripts/rsh_porta.sh \

<ip_address>:/porta_var/backup/<ip_address>/mysql/<date_time>/mysql.tar.bz2 .

rsync -v --rsh=/home/porta-one/scripts/rsh_porta.sh \

<ip_address>:/porta_var/backup/<ip_address>/mysql/<date_time>/my.cnf .

 

3.     Untar the backup file.

Executed on a restoration host.

 

tar -jxvif mysql.tar.bz2

 

4.     Stop the mysql service.

Executed on a restoration host.

 

sudo systemctl stop porta-mysqld

 

5.     Delete or move the old mysql database to some place (be very careful!).

Executed on a restoration host.

 

sudo mv /porta_var/<ip_address>/mysql /porta_var/tmp/mysql_backup_$(date -I)

 

6.     Create a mysql directory.

Executed on a restoration host.

 

sudo mkdir /porta_var/<ip_address>/mysql

 

7.     Run the restoration command.

Executed on a restoration host.

 

sudo innobackupex --use-memory=4G --apply-log ./

sudo innobackupex --copy-back ./

 

8.     Copy the my.cnf file to the mysql directory.

Executed on a restoration host.

 

sudo cp ./my.cnf  /porta_var/<ip_address>/mysql/my.cnf

 

9.     Set correct user rights.

Executed on a restoration host.

 

sudo chown -R mysql:mysql /porta_var/<ip_address>/mysql/

 

10.   Start the mysql service.

Executed on a restoration host.

 

sudo systemctl start porta-mysqld

 

How to restore prompts

Please note that the commands should be executed depending on the availability of directories.

 

Executed on a restoration host.

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_home_porta-um_apache_prompts/ \

/home/porta-um/apache/prompts/

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_asterisk_sounds/ \

/var/lib/asterisk/sounds/

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_porta-sip_sounds/ \

/var/lib/porta-sip/sounds/

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_porta-um_prompts/ \

/var/lib/porta-um/prompts/

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_porta-um_video_brands/ \

/var/lib/porta-um/video/brands/

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_porta_var_psmsc_custom-files_prompts/ \

/porta_var/psmsc/custom-files/prompts/

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_psmsc_prompts/ \

/var/lib/psmsc/prompts/

 

How to restore Deposit files

Deposit files are placed into the directories which names are identical to the file patch + file name. So they also can be copied with rsync command in the following way:

·       file

Executed on a restoration host.

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/deposits/<date_time>/<directory_name>/<file_name> \

<local_path_to_file>

 

·       directory

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/deposits/<date_time>/<directory_name>/ \

<local_path_to_directory>/

 

How to restore CallRecording

Please note that paths on your system might be different. For example /porta_var/porta-call-recording-1/ can be
/porta_var/porta-call-recording-2/ or something like that.


Executed on a restoration host.

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/call_recording/<date_time>/_porta_var_porta-call-recording-1_call-recording_wav/

/porta_var/porta-call-recording-1/call-recording/wav/

How to restore Statistics

Executed on a restoration host.

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/statistics/<date_time>/_porta_var_porta-billing-web-1_statistics/ \

/porta_var/porta-billing-web-1/statistics/

How to restore Custom RPMs

Executed on a restoration host.

 

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_patches/<date_time>/_porta_var_custom_patches/ \

/porta_var/custom_patches/

How to restore Elasticsearch

The following steps should be performed to restore data from Elasticsearch snapshot.

 

Executed on a restoration host.

 

1.     Find the IP address which is used by Elasticsearch and its http port.

 

export elasticip=`grep 'network.host:' /etc/elasticsearch/elasticsearch.yml| cut -d ' ' -f2`

export elasticport=`grep 'http.port:' /etc/elasticsearch/elasticsearch.yml| cut -d ' ' -f2`

 

2.     Verify that snapshots repository “archivist” exists.

 

curl -XGET "http://${elasticip}:${elasticport}/_cat/repositories?v"

 

3.     Check for available snapshots in the “archivist” repository.

 

curl -XGET "http://${elasticip}:${elasticport}/_snapshot/archivist/_all?pretty"

 

4.     Check the state of all available indices.

 

curl -XGET "http://${elasticip}:${elasticport}/_cat/indices/_all?v&pretty"

 

5.     Close all indices before restoring the snapshot.

 

curl -XPOST "http://${elasticip}:${elasticport}/_all/_close/?v&pretty"

 

6.     Check that all indices are closed.

 

curl -XGET "http://${elasticip}:${elasticport}/_cat/indices/_all?v&pretty"

 

7.     Restore the snapshot (e.g. the “snapshot-1501664714” snapshot).

 

curl -XPOST "http://${elasticip}:${elasticport}/_snapshot/archivist/snapshot-1501664714/_restore?wait_for_completion=true&pretty"

 

8.     Check that all indices are open and their “health” is marked as “green”.

 

curl -XGET "http://${elasticip}:${elasticport}/_cat/indices/_all?v&pretty"