Contents

1   Introduction

This document describes the backup and restore options available in the Cloud Execution Environment (CEE).

For a quick reference to the procedures, see Table 1 for a list of the respective operating instruction (OPI) documents.

1.1   Scope

The contents of the individual backup solutions are described in the respective subsections of Section 3.

The current CEE backup and restore solutions do not cover backup or restore of any of the following:

• Glance Image Repository
• Tenant VMs and VM images
• Configurations of backends not managed by CEE

For more information on the scope of the individual procedures, refer to the relevant sections of the OPIs listed in Table 1.

1.1.1   Target Audience

This document is primarily intended to be used by staff handling CEE, including operational personnel performing installing, upgrading/updating, or maintaining activities.

1.2   Overview

The document has the following sections:

2   Backup and Restore Solutions

This section gives a brief description of each of the backup and restore options available in the current CEE release.

2.1   CIC Domain Data Backup and Restore

Description

CIC Domain Data Backup creates a copy of vital infrastructure data of the multi-node CIC domain. SDN configuration files are included in the backup if tightly integrated SDN is enabled on the system.

Restore is required in the following cases:

• Corrupted databases
• Misconfigurations

In some situations, a full restore of all backed up components is needed, while in others, a partial restore of one or more components is enough.

Note:  

CIC Domain Data Restore cannot be performed after Disaster Recovery.

The CIC domain data backup command implements a retention policy, which defines the number of backups kept. For more information on retention policies, see Section 5.

The procedures cannot be performed on single server deployment.

Procedures

CIC Domain Data Backup

CIC Domain Data Restore

2.2   Atlas Backup and Restore

Description

Atlas backup is used to restore the Atlas configuration to a previous state.

Restore is required in the following cases:

• Software rollback
• Data corruption
• If Atlas is deployed in a system without shared block storage and the compute server hosting it becomes unavailable

The Atlas backup command implements a retention policy, which defines the number of backups kept. For more information on retention policies, see Section 5.

The procedures are supported on all systems running Atlas.

Procedures

Atlas Backup

Atlas Restore

2.3   Fuel Synchronization

Description

The purpose of Fuel synchronization is to achieve redundancy by creating a replica of the Fuel VM image on another compute host.

In CEE with Fuel synchronization, two Fuel VMs are hosted on separate compute servers: an active Fuel VM is hosted on one compute server, and another passive (or cold-standby) Fuel VM is hosted on another compute server. During synchronization, the active Fuel VM is copied over to the passive Fuel VM. This is useful if either the active Fuel VM, or the compute server hosting it is not operational. In these situations, failover to the passive Fuel VM is possible. Earlier states of the Fuel VM are not preserved.

Synchronization and failover are manual procedures; automatic backup and restore are not available in this CEE release.

The procedure cannot be performed on single server deployment.

Procedures

Fuel Synchronization

2.4   Disaster Recovery

Description

The purpose of Disaster Recovery is to restore the CEE infrastructure following a man-made or natural disaster, such as a hurricane, flood, or similar. Disaster recovery is achieved by redeployment of the CEE Infrastructure based on the backup of the infrastructure configuration data from an earlier, healthy state.

After recovery, only infrastructure functions are operational.

Note:  

CIC Domain Data Restore cannot be performed after Disaster Recovery.

For redeployment to be successful, the hardware topology of the backed up CEE infrastructure and the CEE infrastructure to be restored must be the same regarding parameters described in the configuration files. These parameters include blade, networking, and cabling information. For more information, refer to Configuration File Guide.

The disaster recovery backup command implements a retention policy, which defines the number of backups kept. For more information on retention policies, see Section 5.

Procedures

Disaster Recovery

2.5   Exporting Fuel VM Image

In addition to Fuel synchronization (creating a local cold standby for Fuel), the Fuel VM image and configuration dump XML file can be exported and stored outside the CEE Region. This stored Fuel VM image and configuration file can be used for rollback purposes and for restoring the Fuel VM to a previous state.

Note:  

If the VM image is exported directly to the external storage, additional steps are necessary. For more information, see Section 9.

The overview of the procedure for creating an externally stored copy of the Fuel VM is the following:

1. Perform Fuel Synchronization as described in the document Fuel Synchronization.
2. Insert the forwarding rule for the routes to the external FTPS server on all vCICs as follows:
   1. Log on to one of the vCICs using SSH. For more information, refer to the CEE Connectivity User Guide.
   2. Execute the following command:

      ip netns e vrouter iptables -I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

      ip netns e vrouter iptables -I FORWARD -m state --state⇒
       RELATED,ESTABLISHED -j ACCEPT

   3. Log out of the vCIC.
3. Add a route between the vFuel compute host and the external storage using the ip netns command on all vCICs.
   1. Execute the command:

      ip netns e vrouter iptables -I FORWARD -s <host_mgmt_ip>/32 -d <ftps_server_ip>/32 -p tcp -j ACCEPT

      ip netns e vrouter iptables -I FORWARD -s ⇒
      <host_mgmt_ip>/32 -d <ftps_server_ip>/32⇒
       -p tcp -j ACCEPT

   2. Verify that the route is created by executing the following command:

      ip netns e vrouter iptables-save | grep <ftps_server_ip>

      ip netns e vrouter iptables-save ⇒
      | grep <ftps_server_ip>

   For the variable <host_mgmt_ip>, use the management IP address of the compute host hosting vFuel. This information can be collected by executing the following command on the compute host:

   ifconfig br-mgmt

   
   The management IP address of the host is listed as inet addr: in the printout.

   An example of the printout is the following:

   root@compute-0-1:~# ifconfig br-mgmt
   br-mgmt   Link encap:Ethernet  HWaddr ca:59:75:45:4c:45
   inet addr:192.168.2.23  Bcast:0.0.0.0  Mask:255.255.255.128

   root@compute-0-1:~# ifconfig br-mgmt
   br-mgmt   Link encap:Ethernet  HWaddr ca:59:75:45:4c:45
   inet addr:192.168.2.23  Bcast:0.0.0.0  ⇒
   Mask:255.255.255.128

4. Log on to vFuel using SSH. For more information, refer to the CEE Connectivity User Guide.
5. Identify the compute hosts hosting the active vFuel VM and the cold standby vFuel VM:

   [root@fuel ~]# for node in primary secondary 
   do
   ip=$(get_vfuel_info --ip --$node);
   name=$(ssh $ip hostname -s 2>&1 | grep compute);
   stat=$(ssh $ip sudo virsh list --all 2>&1 | grep fuel);
   stat=$(echo $stat | awk '{print $3 " " $4}');
   printf "%-10s | %s | %s\n" "$name" "$ip" "$stat";
   done

   An example of the printout is the following:

   compute-0-6 | 192.168.0.23 | running
   compute-0-1 | 192.168.0.20 | shut off
   

   
   In the printout, running identifies the compute host hosting the active vFuel VM, and shut off identifies the compute host hosting the cold standby vFuel VM. Record the IP addresses of the hosts hosting the vFuel VMs from the printouts as this data is required in the rollback procedure.

6. Log on to the host hosting the active vFuel VM using SSH. For more information, refer to the CEE Connectivity User Guide
7. On the vFuel host, retrieve the configuration dump of the vFuel VM using the virsh dumpxml command:

   virsh dumpxml fuel_master > <dump_file_name>.xml

   An example of the command is the following:

   root@compute-0-6:/var/lib/nova# virsh dumpxml fuel_master > fuel_master_compute6_running.xml

8. Shut down the vFuel VM using the virsh shutdown command:
   virsh shutdown fuel_master

   The expected printout is the following:
   Domain fuel_master is being shutdown.

9. Copy and transfer the dump XML file to the external storage using the curl command.

   curl -k --ftp-ssl --upload-file "<file_name>" ftp://<username>:<password>@<ftps_server_ip>//<target_path>

   curl -k --ftp-ssl --upload-file "<file_name>"⇒
    ftp://<username>:<password>⇒
   @<ftps_server_ip>//<target_path>

   The variables are the following:

   • <file_name> is the name of the dump XML file.
   • <username> and <password> are the credentials to the FTPS server.
   • <ftps_server_ip> is the IP address of the external FTPS server used for storing the CEE component backups.
   • <target_path> is the path on the FTPS server to the directory for storing the CEE component backup files.

   An example of the command is the following:

   root@compute-0-6:/var/lib/nova# curl -k --ftp-ssl ⇒
   --upload-file "fuel_master_compute6_running.xml" ftp://admin:admin@10.0.0.1//rollback/

10. Compress, copy and transport the vFuel VM .qcow2 image file to the external storage using the curl command:

    curl -k --ftp-ssl --upload-file <(pigz -1 --stdout --keep --blocksize 4096 --processes $(xmlstarlet sel -t -v /domain/vcpu < /etc/libvirt/qemu/<fuel_xml_name>) <vfuel-img_file_name>) ftp:// <username> : <password> @ <ftps_server_ip> // <target_path>/<compressed_file_name>
    

    curl -k --ftp-ssl --upload-file <(pigz -1 --stdout --keep⇒
     --blocksize 4096 --processes $(xmlstarlet sel -t -v⇒
     /domain/vcpu < /etc/libvirt/qemu/<fuel_xml_name>)⇒
     <vfuel-img_file_name>) ftp:// <username> : <password> @⇒
     <ftps_server_ip> // <target_path>/<compressed_file_name>
    

    The variables are the following:

    • <vfuel-img_file_name> is the vFuel VM image file name.
    • <fuel_xml_name> is the corresponding configuration XML file.
    • <username> and <password> are the credentials to the FTPS server.
    • <ftps_server_ip> is the IP address of the external FTPS server used for storing the CEE component backups.
    • <target_path> is the path on the FTPS server to the directory for storing the CEE component backup files.
    • <compressed_file_name> is the file name for the compressed vCIC image. It is recommended to use the value <vfuel-img_file_name>.gz.

    An example of the command is the following:

    root@compute-0-6:/var/lib/nova# curl -k --ftp-ssl --upload-file <(pigz -1 --stdout⇒
     --keep --blocksize 4096 --processes $(xmlstarlet sel -t -v /domain/vcpu <⇒
     /etc/libvirt/qemu/fuel_master_compute6_running.xml) fuel_br3035.qcow2)⇒
     ftp://admin:admin@10.0.0.1//rollback/fuel_br3035.qcow2.gz

11. Start the active vFuel VM using the virsh start command:
    virsh start fuel_master

    The expected printout is the following:
    Domain fuel_master started

12. Log out of vFuel.
13. Remove the route between the vFuel compute host and the external storage using the ip netns command on all vCICs:

    ip netns e vrouter iptables -D FORWARD -s  <host_mgmt_ip>/32 -d <ftps_server_ip>/32 -p tcp -j ACCEPT

    ip netns e vrouter iptables -D FORWARD -s  ⇒
    <host_mgmt_ip>/32 -d <ftps_server_ip>/32 ⇒
    -p tcp -j ACCEPT

    For the variable <host_mgmt_ip>, use the management IP address of the compute host hosting vFuel. For more information, see Step 3.

14. After all backup files required for exporting the Fuel VM image have been transferred out of the region, delete the forwarding rule for the routes to the external FTPS server on all vCICs:
    1. Log on to one of the vCICs using SSH. For more information, refer to the CEE Connectivity User Guide.
    2. Execute the following command:

       ip netns e vrouter iptables -D FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

       ip netns e vrouter iptables -D FORWARD -m state --state ⇒
       RELATED,ESTABLISHED -j ACCEPT

    3. Log out of the vCIC.

During the procedure, Fuel failed alarm is expected to be issued. This is intended behavior and the alarm ceases after the affected node becomes operational again.

2.6   Exporting CIC VM Images

In addition to CIC domain data backup, the CIC VM images, configuration files, and XML templates can be exported and stored outside the CEE Region. These stored CIC VM images can be used for rollback purposes and for restoring the entire vCIC quorum to an earlier state.

Note:  

The size of the exported CIC VMs can exceed 500 GB depending on the deployment. The transfer time using SFTP as described in Section 9 can take 8 hours for three vCICs with a size of 494 GB, as PIGZ tool is used for compression along with file transfer.

If the VM images are exported directly to the external storage, additional steps are necessary. For more information, see Section 9.

In this section, the three vCICs are referred to as vCIC1, vCIC2 and vCIC3. The procedure below describes exporting CIC VM images from vCIC1. The procedure must be repeated for all vCIC that need to be exported, with different values for the variables, respectively.

The overview of the procedure for creating an externally stored copy of one of the CIC VMs is the following:

1. Perform CIC domain data backup as described in the document CIC Domain Data Backup.
2. Insert the forwarding rule for the routes to the external FTPS server on all vCICs as described in Step 2 in Section 2.5.
3. On vCIC2 and vCIC3, add a route between the vCIC1 compute host and the external storage using the ip netns command.
   1. Execute the following command:

      ip netns e vrouter iptables -I FORWARD -s <host_mgmt_ip>/32 -d <ftps_server_ip>/32 -p tcp -j ACCEPT

      ip netns e vrouter iptables -I ⇒
      FORWARD -s <host_mgmt_ip>/32 -d ⇒
      <ftps_server_ip>/32 -p tcp -j ACCEPT

   2. Verify that the route is created by executing the following command:

      ip netns e vrouter iptables-save | grep <ftps_server_ip>

      ip netns e vrouter iptables-save ⇒
      | grep <ftps_server_ip>

   For the variable <host_mgmt_ip>, use the management IP address of the compute host hosting vCIC1. This information can be collected by executing the following command on the compute host:

   ifconfig br-mgmt

   
   The management IP address of the host is listed as inet addr: in the printout.

   An example of the printout is the following:

   root@compute-0-1:~# ifconfig br-mgmt
   br-mgmt   Link encap:Ethernet  HWaddr ca:59:75:45:4c:45
   inet addr:192.168.2.23  Bcast:0.0.0.0  Mask:255.255.255.128

   root@compute-0-1:~# ifconfig br-mgmt
   br-mgmt   Link encap:Ethernet  HWaddr ca:59:75:45:4c:45
   inet addr:192.168.2.23  Bcast:0.0.0.0  ⇒
   Mask:255.255.255.128

4. Log on to vFuel using SSH. For more information, refer to the CEE Connectivity User Guide.
5. Identify the hosts hosting the vCICs by executing the fuel node command.
6. On the compute host hosting vCIC1, shut down the CIC VM using the virsh shutdown command:
   virsh shutdown <cic_vm_name>

   The expected printout is the following:
   Domain <cic_vm_name> is being shutdown

7. Copy and transfer the dump XML file and the XML template one by one to the external storage using the curl command.

   curl -k --ftp-ssl --upload-file "<file_name>" ftp://<username>:<password>@<ftps_server_ip>//<target_path>

   curl -k --ftp-ssl --upload-file "<file_name>" ⇒
   ftp://<username>:<password>@<ftps_server_ip>//<target_path>

   The variables are the following:

   • <file_name> is the filename of one of the following:
     • The corresponding <cic_name>_vm.xml configuration XML file
     • The corresponding template_<cic_name>_vm.xml template file
   • <username> and <password> are the credentials to the FTPS server.
   • <ftps_server_ip> is the IP address of the external FTPS server used for storing the CEE component backups.
   • <target_path> is the path on the FTPS server to the directory for storing the CEE component backup files.

   An example of the commands is the following:

   root@compute-0-6:/var/lib/nova# curl -k --ftp-ssl ⇒
   --upload-file "cic-1_vm.xml" ftp://admin:admin@10.0.0.1//rollback/
   root@compute-0-6:/var/lib/nova# curl -k --ftp-ssl ⇒
   --upload-file "template_cic-1_vm.xml" ftp://admin:admin@10.0.0.1//rollback/

8. Compress, copy, and transport the CIC VM .img image file to the external storage using the curl command:

   curl -k --ftp-ssl --upload-file <(pigz -1 --stdout --keep --blocksize 4096 --processes $(xmlstarlet sel -t -v /domain/vcpu < /etc/libvirt/qemu/<<cic_name>_vm.xml>) <vcic-img_file_name>) ftp:// <username> : <password> @ <ftps_server_ip> // <target_path>/<compressed_file_name>
   

   curl -k --ftp-ssl --upload-file <(pigz -1 --stdout --keep⇒
    --blocksize 4096 --processes $(xmlstarlet sel -t -v⇒
    /domain/vcpu < /etc/libvirt/qemu/<<cic_name>_vm.xml>)⇒
    <vcic-img_file_name>) ftp:// <username> : <password> @⇒
    <ftps_server_ip> // <target_path>/<compressed_file_name>
   

   The variables are the following:

   • <vcic-img_file_name> is the vCIC VM image file name.
   • <cic_name>_vm.xml is the corresponding configuration XML file.
   • <username> and <password> are the credentials to the FTPS server.
   • <ftps_server_ip> is the IP address of the external FTPS server used for storing the CEE component backups.
   • <target_path> is the path on the FTPS server to the directory for storing the CEE component backup files.
   • <compressed_file_name> is the file name for the compressed vCIC image. It is recommended to use the value <vcic-img_file_name>.gz.

   An example of the command is the following:

   root@compute-0-6:/var/lib/nova# curl -k --ftp-ssl --upload-file <(pigz -1 --stdout⇒
    --keep --blocksize 4096 --processes $(xmlstarlet sel -t -v /domain/vcpu <⇒
    /etc/libvirt/qemu/cic-1_vm.xml) cic-1_vm.img)⇒
    ftp://admin:admin@10.0.0.1//rollback/cic-1_vm.img.gz

9. Start the CIC VM using the virsh start command:
   virsh start <cic_vm_name>

   The expected printout is the following:
   Domain <cic_vm_name> started

10. As the forwarding rule is deleted when the vCIC is restarted, insert the forwarding rule for the routes to the external storage again:

    ip netns e vrouter iptables -I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

    ip netns e vrouter iptables -I FORWARD -m state --state⇒
     RELATED,ESTABLISHED -j ACCEPT

11. Log out of the vCIC.
12. Perform a health check as described in the Health Check Procedure.
13. Remove the route between the vCIC1 compute host and the external storage from vCIC2 and vCIC3 using the ip netns command:

    ip netns e vrouter iptables -D FORWARD -s <host_mgmt_ip>/32 -d <ftps_server_ip>/32 -p tcp -j ACCEPT

    ip netns e vrouter iptables -D FORWARD -s ⇒
    <host_mgmt_ip>/32 -d <ftps_server_ip>/32⇒
     -p tcp -j ACCEPT

    For the variable <host_mgmt_ip>, use the management IP address of the compute host hosting vCIC1. For more information, see Step 3.

14. Repeat the procedure from Step 6 on the remaining vCICs.
    • For vCIC2, the route must be added and removed on vCIC1 and vCIC3.
    • For vCIC3, the route must be added and removed on vCIC1 and vCIC2.
15. After all backup files required for exporting the CIC VM images have been transferred out of the region, delete the forwarding rule for the routes to the external FTPS server on all vCICs:
    1. Log on to one of the vCICs using SSH. For more information, refer to the CEE Connectivity User Guide.
    2. Execute the following command:

       ip netns e vrouter iptables -D FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

       ip netns e vrouter iptables -D FORWARD -m state --state ⇒
       RELATED,ESTABLISHED -j ACCEPT

    3. Log out of the vCIC.

During the procedure, CIC failed alarms are expected to be issued. This is intended behavior and the alarms cease after the affected nodes become operational again.

3   Backup Contents

This section describes the data contained in the individual backup options.

3.1   CIC Domain Data

CIC Domain Data Backup contains the following:

• The dump file of the Identity and Access Management (IdAM) OpenLDAP component

  Restore it if the OpenLDAP database is corrupted or IdAM information is lost.

• The configuration files of the following OpenStack components:
  • nova
  • cinder
  • keystone
  • glance
  • swift
  • neutron

  Restore the OpenStack configuration files if they are deleted, altered by mistake, or a fallback to an older configuration is needed.

• The dumps of the following MySQL databases:
  • nova
  • nova-api
  • cinder
  • glance
  • keystone
  • neutron
  • mysql

  Restore the MySQL files if a corruption in the MySQL databases is suspected.

• The following SDN configuration files, if SDN is enabled on the system:
  • opendaylight
  • quagga

  Restore the SDN configuration files if SDN appears to be corrupted.

3.2   Atlas

The following information is stored as Atlas backup:

• A file containing the checksums of the other files, using sha256
• The content of all databases belonging to the running mysql instance
• The content of the passwords.yaml file in the /etc/puppet/hieradata/ directory
• The content of the root user directory
• The content of the atlasadm user directory

Note:  

The Atlas backup does not store time zone and Atlas GUI settings information.

3.3   Fuel

After synchronization, the cold standby Fuel VM is identical to the active Fuel VM.

3.4   Disaster Recovery

The following files from the Fuel VM are required for a successful restoration of CEE infrastructure and are preconfigured to be included in the backup:

• The complete contents of the folder /mnt/cee_config/
• The CEE version information file /ect/cee_version.txt
• The following user information files:
  • /ect/openstack_deploy/user_secrets.yml
  • /ect/openstack_deploy/user_variables.yml

Note:  

ScaleIO operations executed through the Meta Data Manager (MDM) CLI or the ScaleIO GUI are not propagated back to the config.yaml, therefore such changes are not included in the backup.

In addition, the following files can optionally be added to the backup:

• Health check procedure results performed before the backup

4   Backup Location

This section describes the root location of the created local backups, and the naming rules for the backup directories.

Note:  

It is strongly recommended to export the backups regularly to an external, permanent storage solution. The requirements for the external storage are described in Section 8. The recommended procedure for transferring the backups out of the CEE region is described in Section 9.

4.1   CIC Domain Data

The local backup root is at /var/lib/glance/backup on each vCIC. For every backup, a separate directory is created in the backup root. The directory is named cic_data_backup.<x>, where x=0 is the latest backup created, and the highest number is the oldest backup, according to the retention policy.

In Example 1, the retention policy is three.

cic_data_backup.0 (newest)
cic_data_backup.1
cic_data_backup.2 (oldest)

One copy of all backups is stored on each virtual Cloud Infrastructure Controller (vCIC). After the local backup is created, it is synchronized to the other vCICs (referred to as remote backup).

It is highly recommended to regularly export the entire backup root directory from one vCIC to a remote location, such as an external backup solution. Glance images are not included in the CIC domain backup and are expected to be kept as part of an external backup.

4.2   Atlas

The backup root is at /var/archives/ on Atlas. For every backup, a directory is created in the backup root. The directory is named atlas_backup.<date>, where <date> is in s format and corresponds to the number of seconds elapsed since 1970-01-01 00:00:00 UTC.

The user must ensure that the backup files are stored externally in a persistent storage location, in case Atlas becomes inaccessible or corrupted.

4.3   Fuel

Replicas are created upon copying the original Fuel VM image to the external storage. The active and the cold standby Fuel VMs are located on separate compute hosts. To identify the compute host hosting the cold standby Fuel VM, refer to the relevant section of the document Fuel Synchronization.

4.4   Disaster Recovery

The backup root is at /var/disaster_backup on Fuel. For every backup, a directory is created underneath the backup root. The directory is named Disaster_backup.<x>, where x=0 is the latest backup created, and the highest number is the oldest backup, according to the retention policy.

In Example 2, the retention policy is three.

Disaster_backup.0 (newest)
Disaster_backup.1
Disaster_backup.2 (oldest)

For more information on retention policies, see Section 5.

The .tgz backup file must be copied or moved to a storage outside the CEE region, at a separate location.

For the storage requirements of the recovery backup files, refer to Disaster Recovery.

5   Retention Policy

Some of the backup solutions implement a retention policy, which defines the maximum number of backups retained before they are automatically deleted.

5.1   CIC Domain Data

The maximum number of backups retained is defined in the retain cic_data_backup variable in /etc/rsnapshot-cee.conf on every vCIC. The default number of retained backups is three.

Note:  

The retention policy is always the same on all vCICs.

Upon creating a backup, the existing backups are modified as follows:

1. The oldest backup in cic_data_backup.2 is deleted.
2. cic_data_backup.1 is renamed to cic_data_backup.2.
3. cic_data_backup.0 is renamed to cic_data_backup.1.
4. The new backup is named cic_data_backup.0.

The retention policy can be changed as described in the relevant section of the document CIC Domain Data Backup.

5.2   Atlas

The maximum number of backups retained is defined in the BM_ARCHIVE_COUNT variable in /etc/backup-manager.conf. The default number of retained backups is 10.

5.3   Disaster Recovery

The maximum number of backups retained is defined in the retain Disaster-backup variable in /etc/disaster-backup.conf on vFuel. The default number of retained backups is three.

Upon creating a backup, the existing backups are modified as follows:

1. The oldest backup in Disaster_backup.2 is deleted.
2. Disaster_backup.1 is renamed to Disaster_backup.2.
3. Disaster_backup.0 is renamed to Disaster_backup.1.
4. The new backup is named Disaster_backup.0.

The retention policy can be changed as described in the relevant section of the document Disaster Recovery.

6   Backup Sizes

CEE backup files have the following storage volume requirements:

(1)  The estimated size is valid for a backup taken after CEE deployment.

7   Procedure Durations

The approximate duration of the individual procedures is the following:

7.1   Transfer Durations

Based on the backup sizes listed in Section 6, the approximate transfer time of the backup files to or from the FTPS server using the procedures described in Section 9 can be calculated based on the following table:

(1)  Calculated for an average vFuel VM size of 65 GB

(2)  Calculated for a vCIC size of 500 GB

8   Storage Requirements

This section describes requirements and recommendations regarding the external storage used for storing the CEE component backup files.

8.1   Storage Volume

The minimum required size of the external storage can be calculated based on the size of the backups to be stored on the storage. For more information on the size of backup files, see Section 6

8.2   Accessible Storage

Storage used for the backup of the CEE infrastructure must be accessible, as backup files need to be transferred to and from the CEE environment.

8.3   Bandwidth

The bandwidth of the network required for efficient data transfer and the required transfer time to and from the backup storage can be calculated based on the size of the files to be backed up. For more information, see Section 6.

8.4   Security Policy

By default, the created backup files are not encrypted, with the exception of Atlas backup files. It is the responsibility of the user to ensure the secure encryption, transfer and handling of backup files outside the CEE region. It is recommended to implement an encryption solution on the CEE data stored on the external storage.

The user must be aware of security policies. Refer to the document Security User Guide.

9   Transferring Backups Using FTPS

This section describes an example procedure for transferring the created backup files to an external persistent storage using FTPS.

Note:  

If this procedure is used, the following conditions must be considered:

• Due to the transfer of the backup files between the CEE region and the external server, bandwidth is affected, and transfer time must be taken into consideration.
• During the transfer of Fuel VM image, the Fuel VM is shut off.
• During the transfer of a CIC VM image, the exported vCIC must be shut off.

9.1   Remote FTPS Server for Storing Backups

The remote server must fulfill the following requirements:

• Storage according to the requirements described in Section 8
• External connectivity through at least 1 Gbps Ethernet
• Installed and operational Ubuntu Linux 14.04 OS
• vstfpd FTPS server, with the following configuration:
  • /etc/vsftpd.conf file updated with the following values:

    # Turn ON SSL
    ssl_enable=YES 
    allow_anon_ssl=NO 
    
    # Use encryption for data 
    force_local_data_ssl=YES 
    
    # Use encryption for authentication 
    force_local_logins_ssl=YES 
    
    ssl_tlsv1=YES 
    ssl_sslv2=NO :
    ssl_sslv3=NO 
    ssl_ciphers=HIGH 
    
    local_max_rate=41943040

9.2   Procedures

9.2.1   Entering and Exiting vCIC Maintenance Mode

During the transfer of a Fuel VM image, all vCICs must be in maintenance mode. During the transfer of a CIC VM image, the exported vCIC must be shut off, and the other vCICs must be in maintenance mode.

To enter maintenance mode on a vCIC, execute the following command on the vCIC:

umm on

To exit maintenance mode and enter operational mode after the transfer is complete, execute the following command on the vCIC:

umm off

9.2.2   Inserting Forwarding Rule on the vCICs

To insert a forwarding rule for the communication with the FTPS servers, execute the following command on all vCICs:

iptables -t nat -A POSTROUTING -j MASQUERADE

9.2.3   Adding Route between Compute and FTPS Server

To add a route between a compute host and the remote FTPS server for transferring backup files to and from the region, execute the following command on the compute host:

route add <ftps_server_ip> gw <vcic_ip>

The variables are the following:

• <ftps_server_ip> is the IP address of the external FTPS server used for storing the CEE component backups.
• <vcic_ip> is the management IP address of a vCIC that is operational or in maintenance mode.

9.2.4   Transferring Files between a Host and the FTPS Server

Files must be transferred to the FTPS server one by one.

To transfer a file from a compute host to the FTPS server, execute the following command on the compute host containing the file to be transferred:

curl -k --ftp-ssl --upload-file "<file_name>" ftp://<username>:<password>@<ftps_server_ip>//<target_path>

curl -k --ftp-ssl --upload-file "<file_name>"⇒
 ftp://<username>:<password>@<ftps_server_ip>//<target_path>

The variables are the following:

• <file_name> is the name of the file to be transferred.
• <username> and <password> are the credentials to the FTPS server.
• <ftps_server_ip> is the IP address of the external FTPS server used for storing the backup files.
• <target_path> is the path on the FTPS server to the directory for storing the backup files.

To compress and transfer an image file from a compute host to the FTPS server, execute the following command on the compute host containing the image file to be transferred:

curl -k --ftp-ssl --upload-file <(pigz -1 --stdout --keep --blocksize 4096 --processes $(xmlstarlet sel -t -v /domain/vcpu < /etc/libvirt/qemu/<xml_file_name>) <img_file_name>) ftp:// <username> : <password> @ <ftps_server_ip> // <target_path> / <compressed_file_name>

curl -k --ftp-ssl --upload-file <(pigz -1 --stdout --keep⇒
 --blocksize 4096 --processes $(xmlstarlet sel -t -v⇒
 /domain/vcpu < /etc/libvirt/qemu/<xml_file_name>)⇒
 <img_file_name>) ftp:// <username> : <password> @⇒
 <ftps_server_ip> // <target_path> / <compressed_file_name>

The variables are the following:

• <img_file_name> is the image file name.
• <xml_file_name> is the corresponding configuration XML file.
• <username> and <password> are the credentials to the FTPS server.
• <ftps_server_ip> is the IP address of the external FTPS server used for storing the CEE component backups.
• <target_path> is the path on the FTPS server to the directory for storing the CEE component backup files.
• <compressed_file_name> is the file name for the compressed image. It is recommended to use the value <img_file_name>.gz.

To transfer a file from the FTPS server to a compute host, execute the following command on the destination compute host:

curl -k --ftp-ssl ftp://<username>:<password>@<ftps_server_ip>/<source_path>/<file_name> > <target_path>/<file_name>

curl -k --ftp-ssl ftp://<username>:<password>@⇒
<ftps_server_ip>/<source_path>/<file_name> > ⇒
<target_path>/<file_name>

The variables are the following:

• <file_name> is the name of the file to be transferred.
• <username> and <password> are the credentials to the FTPS server.
• <ftps_server_ip> is the IP address of the external FTPS server used for storing the backup files.
• <source_path> is the path on the FTPS server to the directory for storing the backup files.
• <target_path> is the path to the destination folder on the compute host.

To transfer an image file from the FTPS server to a compute host, execute the following command on the destination compute host:

curl -k --ftp-ssl ftp:// <username> : <password> @ <ftps_server_ip> // <source_path> / <compressed_file_name> | pigz --stdout --decompress --processes $(xmlstarlet sel -t -v /domain/vcpu < ./<xml_file_name>) > /var/lib/nova/<img_file_name>

curl -k --ftp-ssl ftp:// <username> : <password> @⇒
 <ftps_server_ip> // <source_path> / <compressed_file_name> ⇒
 | pigz --stdout --decompress --processes⇒
 $(xmlstarlet sel -t -v /domain/vcpu < ./<xml_file_name>)⇒
 > /var/lib/nova/<img_file_name>

The variables are the following:

• <username> and <password> are the credentials to the FTPS server.
• <ftps_server_ip> is the IP address of the external FTPS server used for storing the backup files.
• <source_path> is the path on the FTPS server to the directory for storing the backup files.
• <compressed_file_name> is the file name for the compressed image.
• <img_file_name> is the name of the image file to be transferred.
• <xml_file_name> is the corresponding configuration XML file.

9.2.5   Removing Route between Compute and FTPS Server

To remove a route between a compute host and the remote FTPS server, execute the following command on the compute host:

route delete <ftps_server_ip>

where <ftps_server_ip> is the IP address of the external FTPS server used for storing the CEE component backups.

9.2.6   Removing Forwarding Rule from the vCICs

After all necessary backup files have been transferred to or from the external FTPS server, delete the forwarding rule by executing the following command on all vCICs:

iptables -t nat -D POSTROUTING -j MASQUERADE

10   Backup Strategy

Based on the frequency of changes to the infrastructure, it is recommended to perform backups at regular intervals, for example, on a weekly or monthly basis.

The following list describes the recommended backups to perform after each change to the CEE environment:

Note:  

If any of the listed operations occur in your deployment since the latest backups were taken, the restore process can cause inconsistencies in the node, and the restore procedures can fail.

(1)  The CEE Software package must also be backed up for disaster recovery purposes.

Based on the frequency of changes to the infrastructure, it is recommended to perform backups at regular intervals as well, for example, on a weekly or monthly bases.

10.1   Recommended Failover Strategy for Fuel

Failover to the passive standby Fuel VM must be considered in the following cases:

• The Fuel VM is not operational.
• The compute server that hosts the active Fuel VM is not operational or is under maintenance.

For the failover procedure refer to Fuel Synchronization.