Disaster Recovery

Cloud Execution Environment

1	Introduction
1.1	Scope
2	Prerequisites
2.1	Prerequisites for Backup
2.2	Prerequisites for Restore
3	Command Parameters
4	Procedures
4.1	Backup Procedure
4.2	Restore Procedure

1 Introduction

The aim of this document is to provide instruction on creating a backup of the CEE infrastructure for disaster recovery, and provide a general flow on the recovery (redeployment) procedure using the backed up configuration data.

The purpose of the Cloud Execution Environment (CEE) disaster recovery procedure is to create a backup of the CEE infrastructure for recovery purposes following a man-made or natural disaster, such as hurricane, flood or similar. Disaster recovery is achieved by redeployment of the CEE Infrastructure based on the backup of the infrastructure configuration data from an earlier, healthy state.

The restore process of the CEE includes the redeployment of CEE using the backed up configuration files and tarball, on a hardware topology which is the same as that of the backed up CEE infrastructure regarding parameters described in the configuration files.

Refer to the document Backup and Restore Overview for more information on disaster recovery, including the following:

Backup contents
Backup sizes and locations
Retention policies
Procedure durations
External storage requirements
Recommended backup strategy

Note:: The procedure described in this document only includes the backup and recovery of the CEE infrastructure.

For an overview of all backup and restore options available in CEE, refer to the Backup and Restore Overview.

1.1 Scope

After redeployment, only infrastructure functions are operational. Databases and tenant data are not restored in Virtual Cloud Infrastructure Controllers (vCICs).

For example, Virtual Network Function (VNF) redeployment is out of the scope of this document. VNFs are redeployed or restored from their respective backups by, for example, the Ericsson Cloud Manager (ECM).

Note:: CIC Domain Data Restore cannot be performed after disaster recovery.

In the current release of CEE, the disaster recovery procedure is not available on HDS platform. Disaster recovery with HDS use cased is to be added in an upcoming revision.

2 Prerequisites

2.1 Prerequisites for Backup

Before creating a CEE infrastructure backup, the following information and preconditions must be collected and fulfilled:

It is recommended that the system is healthy at the time of backup. Before backing up the infrastructure, perform health check procedure as described in the document Health Check Procedure. Optionally, any results of the health check procedure can be included in the backup as reference for checking system health after restore.
If the system is not healthy at the time of backup, the same faults are present after infrastructure restore.

Note:
Measures to correct in case the system is not healthy are out of the scope of this document. Refer to the documents Health Check Procedure and Emergency Recovery Procedure for corrective measures, or collect necessary information according to Data Collection Guideline and contact next level of support.
Make sure that no configuration and deployment changes take place while creating backups.

2.1.1 Tools

The following tools are required for the backup of the CEE infrastructure:

Storage configured for backup, see Backup and Restore Overview.
Enabled vFuel. If vFuel is disabled, enable it again as described in vFuel On Demand Use.

2.1.2 User Access

The user performing the backup must log on as a dedicated infrastructure administration user , which is by default ceebackup. The credentials for this user must be available.

2.2 Prerequisites for Restore

To restore the CEE infrastructure from a backup, the following information/preconditions must be collected/fulfilled:

The hardware topology of the backed up CEE infrastructure must be the same as that of the CEE infrastructure to be restored regarding parameters described in the configuration files. Such parameters are, for example, blade, networking, and cabling information, depending on the configuration. For more information, refer to Configuration File Guide.

2.2.1 Tools

The following tools are required for the restore of the CEE infrastructure:

Backup file, created as described in Section 4.1.
Tarball including installation media, used for the latest update of the CEE. If the CEE has not been updated since deployment, the tarball used for deployment is required.
Configured kickstart server, with vFuel deployed using the tarball used for the latest update or deployment of CEE. Refer to the document Preparation of Kickstart Server and Section 4.2.1.

3 Command Parameters

To perform operations of the sudo /opt/ecs-fuel-utils/disaster-backup.sh shell script, do the following:

Log onto vFuel as dedicated infrastructure administrator user, as described in the document CEE Connectivity User Guide.

Execute the following command

sudo /opt/ecs-fuel-utils/disaster-backup.sh <parameter>

The available operations are the following:

Parameter	Description	Additional information	Example Printout
`create`	Creates a new disaster recovery backup	See Section 4.1	See Section 4.1
`list`	Prints a list of all backups	The output contains the following fields: ID of the backup Size of the backup Date and time when the backup was created Version of the CEE software release Status of the backup. Possible values are `SYNCHRONIZING` and `COMPLETED` Name of the backup Directory of the backup	Example 1xmpl_list_pdf
`showPolicy`	Prints the current retention policy	For more information on retention policy, see Backup and Restore Overview	Example 2
`changePolicy <n>` where `n` must be an integer number and larger than the current retention policy	Increases retention policy value	The default value of the retention policy is `5`, meaning that five backups are kept.By increasing the retention policy value, more backups are kept on the system.⁽¹⁾	Example 3
`clean`	Removes temporary files which were not deleted upon an aborted backup	No additional information	Example 4
`removeAll`	Removes all active backups	Upon prompt, all backups are permanently removed.	Example 5
`help` `-h` `--help`	Prints all available parameters of the `disaster-backup.sh` command	No additional information	Example 6

(1) By increasing this parameter, the storage space used by the backups increases. Make sure that there is enough free space under /var/disaster_backup/ to keep additional backups

An example printout of the command sudo /opt/ecs-fuel-utils/disaster-backup.sh list is the following:

Example 1 List Backups

+----------------------------------+--------+---------------------+----------+--------------+-----------------+----------------------+
| ID | Size | Date | Version | Status | Name | Directory |
+----------------------------------+--------+---------------------+----------+--------------+-----------------+----------------------+
| cb41cbfd089359c5dadad3092ef12c8c | 32K | 03/03/2017,14:53:12 | 9.0 | COMPLETED | Disaster-backup |Disaster-backup.0 |
+----------------------------------+--------+---------------------+----------+--------------+-----------------+----------------------+
| a8465a055e6168368e76c4ed16f4535e | 32K |03/03/2017,14:53:03 | 9.0 | COMPLETED | Disaster-backup | Disaster-backup.1 |
+----------------------------------+--------+---------------------+----------+--------------+-----------------+----------------------+
| 8fa5e5c173e259106f6e872d77381474 | 32K | 03/03/2017,14:36:15 | 9.0 | COMPLETED | Disaster-backup |Disaster-backup.2 |
+----------------------------------+--------+---------------------+----------+--------------+-----------------+----------------------+
| 10fd5dac8b59a7f2ce141fec2b89dba2 | 32K |03/03/2017,12:27:18 | 9.0 | COMPLETED | Disaster-backup | Disaster-backup.3 |
+----------------------------------+--------+---------------------+----------+--------------+-----------------+----------------------+
| b92fbadf58aeffd6641b1b9216513d7e | 32K | 03/03/2017,12:26:26 | 9.0 | COMPLETED | Disaster-backup |Disaster-backup.4 |
+----------------------------------+--------+---------------------+----------+--------------+-----------------+----------------------+

Example 1 List Backups

+----------------------------------+--------+--------------
-------+----------+--------------+-----------------+----------------------+
| ID | Size | Date | Version | Status | Name | Directory |
+----------------------------------+--------+--------------
-------+----------+--------------+-----------------+----------------------+
| cb41cbfd089359c5dadad3092ef12c8c | 32K | 03/03/2017,14:53:12 | 9.0 | COMPLETED | Disaster-backup |
Disaster-backup.0 |
+----------------------------------+--------+--------------
-------+----------+--------------+-----------------+----------------------+
| a8465a055e6168368e76c4ed16f4535e | 32K |
03/03/2017,14:53:03 | 9.0 | COMPLETED | Disaster-backup | Disaster-backup.1 |
+----------------------------------+--------+--------------
-------+----------+--------------+-----------------+----------------------+
| 8fa5e5c173e259106f6e872d77381474 | 32K | 03/03/2017,14:36:15 | 9.0 | COMPLETED | Disaster-backup |
Disaster-backup.2 |
+----------------------------------+--------+--------------
-------+----------+--------------+-----------------+----------------------+
| 10fd5dac8b59a7f2ce141fec2b89dba2 | 32K |
03/03/2017,12:27:18 | 9.0 | COMPLETED | Disaster-backup | Disaster-backup.3 |
+----------------------------------+--------+--------------
-------+----------+--------------+-----------------+----------------------+
| b92fbadf58aeffd6641b1b9216513d7e | 32K | 03/03/2017,12:26:26 | 9.0 | COMPLETED | Disaster-backup |
Disaster-backup.4 |
+----------------------------------+--------+--------------
-------+----------+--------------+-----------------+----------------------+

An example printout of the command sudo /opt/ecs-fuel-utils/disaster-backup.sh showPolicy is the following:

Example 2 Show the Retention Policy

Retention policy=5

An example printout of the command sudo /opt/ecs-fuel-utils/disaster-backup.sh changePolicy <n> is the following:

Example 3 Change the Retention Policy

INFO: Change retention policy=6 successful

An example printout of the command sudo /opt/ecs-fuel-utils/disaster-backup.sh clean is the following:

Example 4 Clean

INFO Cleaning temporary files and directories during backup
INFO Cleanup finished

An example printout of the command sudo /opt/ecs-fuel-utils/disaster-backup.sh removeAll with prompt yes is the following:

Example 5 Remove All Active Backups

Backup directories will be removed on fuel . Are you sure? [y/n]?y 
Removing /var/disaster-backup on host ...

An example printout of the command sudo /opt/ecs-fuel-utils/disaster-backup.sh help is the following:

Example 6 Show All Available Parameters

cic-data-backup
        create [name]:  create new backup named [name]
        list:           list the current active backups
        showPolicy:     show the current retention policy
        changePolicy N: change the retention policy to N(integer)
        clean:          cleanup temporary directories  
        removeAll:      remove the current active backups
        help:           show this help and exit

4 Procedures

This section describes the steps required for the following actions:

Creating a disaster recovery backup
Restoring CEE infrastructure using a previously created backup

4.1 Backup Procedure

Do the following:

Log onto vFuel as dedicated infrastructure administrator user, as described in the document CEE Connectivity User Guide.

To create a disaster recovery backup, execute the following command on Fuel:

sudo /opt/ecs-fuel-utils/disaster-backup.sh create

The following is an example of the printout:

INFO Create backup "Disaster-backup" with⇒
 ID=e79ddd74c775e921957537f3b65a21a0 ...
INFO Disaster Backup Started...
INFO Backup the Fuel Config files ...
INFO Disaster Backup Done... 
INFO Cleaning temporary files and directories during backup 
INFO Cleanup finished 
INFO Backup with ID=e79ddd74c775e921957537f3b65a21a0⇒
 performed successfully 
INFO Backup created with File name⇒
 << disaster_backup_22032017_132859.tgz >> in⇒
 /var/disaster-backup/Disaster-backup.0

The newly created backup is located in /var/disaster-backup/Disaster-backup.0/.

The name of the backup file has the following format: disaster_backup_<date>.tgz, where <date> has the following format: <ddmmyyyy>_<hhmmss>.

Move the backup to the storage configured according to the Backup and Restore Overview, outside of CEE.

4.2 Restore Procedure

The restore consists of the redeployment of CEE using the backed up infrastructure configuration files.

Note:: After redeployment, only infrastructure functions are operational. Databases and tenant data are not restored in vCICs. CIC domain data restore operation cannot be performed after disaster recovery.

4.2.1 Prepare Kickstart Server

Copy the backup .tgz file to the kickstart server.
Copy the backed up tarball to the kickstart server.
Extract config.yaml from the backup file by executing the following command:
```
sudo tar xvzf cee_backup_<date>.tgz mnt/cee_config/config.yaml --strip=2
```
The command extracts config.yaml from the backup to the current directory.
Install Fuel as described in Preparation of Kickstart Server, using the config.yaml extracted from the backup.

4.2.2 Restore CEE Infrastructure

The restore procedure consists of the installation of CEE using the kickstart server prepared with the backup files. Do the following:

Copy the backup data to Fuel.
Extract the contents of the backup to Fuel by executing the following command:
```
sudo tar xvzfC cee_config_backup_<date>.tgz /
```
The command puts the configuration files in place.
If necessary, adjust configuration changes with deployment-specific data as described in the Configuration File Guide.
Note:
Major changes—such as turning on features that were not in use at time of backup—are not allowed.
Install CEE as described in SW Installation in Multi-Server Deployment or SW Installation in Single Server Deployment, using the installation media described in Section 2.2.1.
After deployment, perform the health check procedure described in the document Health Check Procedure. If the results of the health check performed before backup were included in the backup file, compare post-restore results to pre-backup results to verify the successful restore.
Note:
Measures to correct in case the system is not healthy are out of the scope of this document. Refer to the documents Health Check Procedure and Emergency Recovery Procedure for corrective measures, or collect necessary information according to Data Collection Guideline and contact next level of support.