Contents

1   Introduction

The aim of this document is to give operating instructions on creating and managing CIC domain data backups.

The purpose of the CIC domain data restore operation is to enable recovery from corrupted databases or misconfigurations.

For instructions on creating and managing a CIC domain data backup, refer to the document CIC Domain Data Backup.

Refer to the document Backup and Restore Overview for more information on CIC domain data restore, including the following:

• Backup contents
• Backup sizes and locations
• Retention policies
• Procedure durations
• External storage requirements
• Recommended backup strategy

For an overview of all backup and restore options available in CEE, refer to the Backup and Restore Overview.

1.1   Scope

This procedure restores the following data of the multi-node CIC domain from an existing backup:

• OpenLDAP database
• OpenStack configuration files
• MySQL databases
• SDN configuration files

In some situations, a partial restore of one or more components of the system is needed. In others, a full restore of all components has to be done.

SDN configuration file restore is only applicable if SDN is enabled on the system.

2   Overview

2.1   Restore Procedure Flow

To perform a CIC domain data restore, follow the below procedure:

1. Prepare the system for a restore. Log in and set the vCICs to maintenance mode (MM). Make sure all vCICs are in MM before starting the restore procedure. See Step 1 in Section 4, Step 2 in Section 4, and Step 3 in Section 4 in Section 4.
2. Restore one or more of the following components, as needed:
   • OpenLDAP database
   • MySQL databases
   • OpenStack configuration files
   • SDN configuration files

   Step 5 in Section 4 in Section 4 describes the procedure for restoring the different components.

   Note:  

   It is possible that all components need to be restored.

   
   
3. After the components are restored, finalize the restore by setting the vCICs to normal operation mode. See Step 6 in Section 4 in Section 4.

3   Prerequisites

Before starting the process, the following conditions must be met:

• Check that all vCICs are in operational mode.
• Check the below list of operations. If any of the listed operations occurred in your deployment since the chosen backup (see Section 3.3) was taken, the restore process can cause inconsistency in nodes affected by the OpenStack and SDN configuration file restore. In this case, do not start the restore procedure. Contact the next level of support.
  • Deployment changes of the CIC domain data, for example after adding new compute nodes.
  • Changes in the deployment of Virtual Network Functions (VNFs), for example after the creation of new VMs belonging to an already deployed VNF or the deployment of a new VNF.
  • Upgrades and updates of the CEE software version.
  • Any VM lifecycle event, including the following:
    • boot
    • start
    • stop
    • delete
    • migrate
    • evacuate
    • forcemove
  • Automated VM lifecycle events (evacuation or migration) executed by Continuous Monitoring High Availability (CM-HA) or the update framework.
  • Any modification in the OpenStack configuration database, for example creation or deletion of the following:
    • Glance images
    • Cinder volumes
    • Neutron networks

3.1   Node Names

Throughout the document, node names cic1, cic2, and cic3 are used to refer to the vCICs used in the restore procedure. Write down the real node names from the actual deployment and use them when issuing actual command lines while following the examples provided in this document.

Table 1    vCIC Actual Names

Name Throughout the Document	Real Name in Deployment
cic1
cic2
cic3

3.2   Login Credentials

The restore procedure must be performed by a dedicated CIC domain data administration user, which is by default ceebackup. The credentials for this user must be available.

3.3   Selecting Backup and Components

The system or the external storage can contain more than one backup. In most cases, the latest backup is the most appropriate and is used for examples throughout this document. However, it can be necessary to restore the CIC domain using an older backup. The backup to be used in the restore procedure must be identified.

The CIC domain restore procedure can be used to restore the entire CIC domain, multiple components, or a single component. For more information, see Section 2.1. The components to be restored must be identified.

4   Restore Procedure

Follow the below procedure to restore the backup contents:

1. Log on to one of the vCICs as a dedicated CIC domain data administration user (by default ceebackup). For more information, refer to CEE Connectivity User Guide.
2. To move all the vCICs to maintenance mode, run pre-cic-data-restore, located in /usr/bin/.

   An example of the output:

   Note:  

   The output contains warnings and error messages.

   The warning for unauthorized access is a default SSH message issued on the vCIC. Error messages are issued because MySQL service is down during restore.

   
   

   ceebackup@cic-1:~$ pre-cic-data-restore
   INFO    3 cics online.
   INFO    Moving CICs in Maintenance Mode
   
   Attention! Unauthorized remote access is strictly prohibited!
   INFO    Moves cic-2.domain.tld into Maintenance mode
   Attention! Unauthorized remote access is strictly prohibited!
   INFO    cic-2.domain.tld  yet to go into Maintenance mode..
   Attention! Unauthorized remote access is strictly prohibited!
   INFO    Moves cic-3.domain.tld into Maintenance mode
   Attention! Unauthorized remote access is strictly prohibited!
   INFO    cic-3.domain.tld  yet to go into Maintenance  mode..
   INFO    cic-1.domain.tld Moving to Maintenance mode
   Broadcast message from root@cic-1.domain.tld
    (/dev/pts/25) at 15:31 ...
   The system is going down for reboot NOW!
   
   ceebackup@cic-1:~$ Write failed: Broken pipe
   
   Note: Login session to vCIC will be disconnected, at the ⇒
   end of pre-cic-data-restore.

   Ensure that all the vCICs are in MM by logging in again.

   An example of the output:

   ssh ceebackup@<cic-1_ip_address>
   ceebackup@cic-1's password:
   ceebackup@cic-1:~$  
   
   #     #  #     #    #   ##      ##  #   #
   ##   ##  ##   ##       #       #  # ##  #
   # # # #  # # # #    #   ##     #  # # # #
   #  #  #  #  #  #    #     #    #  # #  ##
   #     #  #     #    #   ##      ##  #   #
   
   
   You are in Maintenance mode now
   
   ceebackup@cic-1:~$ umm status
   umm
   
   
   umm
   

3. On systems using ScaleIO, do the following:
   1. Restart the scini service on all vCICs by executing the following commands:

      ceebackup@cic1:~$ sudo /etc/init.d/scini restart
      ceebackup@cic2:~$ sudo /etc/init.d/scini restart
      ceebackup@cic3:~$ sudo /etc/init.d/scini restart

   2. Mount /var/lib/glance on all vCICs by executing the following commands:

      ceebackup@cic1:~$ sudo mount /var/lib/glance
      ceebackup@cic2:~$ sudo mount /var/lib/glance
      ceebackup@cic3:~$ sudo mount /var/lib/glance

4. On systems not using ScaleIO, restart the iSCSI service on all vCICs by executing the following commands:

   ceebackup@cic1:~$ sudo /etc/init.d/open-iscsi restart
   ceebackup@cic2:~$ sudo /etc/init.d/open-iscsi restart
   ceebackup@cic3:~$ sudo /etc/init.d/open-iscsi restart

5. Execute the cic-data-restore script on the vCIC where pre-cic-data-restore (see Step 2) was executed.
   1. Restore all contents from the backup file.

      Run the below command:

      cic-data-restore -f <file> -m <component_name>

      Where <file> is the path of the backup files located and <component_name> is the module the user wants to restore.

      <component_name> can have the following values:

      • mysql
      • ldap
      • sdn
      • config
      • all

      If more than one component needs to be restored, restore them one by one. If cic-data-restore -f <file> -m all is used, all modules will be restored.

      An example of the output:

      Note:  

      The output contains warnings and error messages.

      The warning for unauthorized access is a default SSH message issued on the vCIC. Error messages are issued because MySQL service is down during restore.

      
      

      ceebackup@cic-1:~$ cic-data-restore -f ⇒
      /var/lib/glance/backup/cic-data-backup.0/⇒
      cic-data-backup _06032017_071500.tgz –m All
      
      INFO    Checking whether cics are in Maintenance mode
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      INFO    Started OpenLDAP restore
      INFO    Removing the old LDAP configuration on cic-1.⇒
      domain.tld ...
      INFO    Removing the old LDAP configuration on cic-2.⇒
      domain.tld ...
      INFO    Removing the old LDAP configuration on cic-3.⇒
      domain.tld ...
      INFO    started LDAP database restore on cic-1.⇒
      domain.tld
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      
      Attention! Unauthorized remote access is strictly p⇒
      rohibited!
      
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      INFO    Completed OpenLDAP restore
      INFO    Started Restore OpenStack Configuration ⇒
      Files
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      Copying openstack config from backup file on ⇒
      cic-1.domain.tld
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      Copying openstack config from backup file on ⇒
      cic-2.domain.tld
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      Copying openstack config from backup file on ⇒
      cic-3.domain.tld
      INFO    Completed Restore OpenStack Configuration Files
      INFO    Seems SDN is not present in current setup
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      Cleanup MySQL on cic-2.domain.tld
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      Cleanup MySQL on cic-3.domain.tld
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      Exporting variables on cic-1.domain.tld
      Exporting Extra variable on cic-1.domain.tld
      Starting MySQL on cic-1.domain.tld
      Error signing on to the CIB service: Transport ⇒
      endpoint is not connected
      73fc7222-15ce-11e7-9e15-4f048f9684b4:380860
      /usr/lib/ocf/resource.d/fuel/mysql-wss: line 371: ⇒
      [: too many arguments
      Error signing on to the CIB service: Transport ⇒
      endpoint is not connected
      Warning: Using a password on the command line ⇒
      interface can be insecure.
      ERROR 1045 (28000): Access denied for user ⇒
      'root'@'localhost' (using password: NO)
      Warning: Using a password on the command line ⇒
      interface can be insecure.
      ERROR 1045 (28000): Access denied for user ⇒
      'root'@'localhost' (using password: NO)
      0
      Restoring Mysql Openstack databases from backup ⇒
      file on cic-1.domain.tld
      
      Attention! Unauthorized remote access is ⇒
      strictly prohibited!
      
      Exporting variables on cic-2.domain.tld
      Starting MySQL on cic-2.domain.tld
      Error signing on to the CIB service: Transport ⇒
      endpoint is not connected
      00000000-0000-0000-0000-000000000000:-1
      /usr/lib/ocf/resource.d/fuel/mysql-wss: line 371: [: ⇒
      too many arguments
      Error signing on to the CIB service: Transport ⇒
      endpoint is not connected
      
      Attention! Unauthorized remote access is ⇒
      strictly prohibited!
      
      Exporting variables on cic-3.domain.tld
      Starting MySQL on cic-3.domain.tld
      Error signing on to the CIB service: Transport ⇒
      endpoint is not connected
      00000000-0000-0000-0000-000000000000:-1
      /usr/lib/ocf/resource.d/fuel/mysql-wss: line 371: ⇒
      [: too many arguments
      Error signing on to the CIB service: Transport ⇒
      endpoint is not connected
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      Stopping MySQL on cic-1.domain.tld
      Error signing on to the CIB service: Transport ⇒
      endpoint is not connected
      Error signing on to the CIB service: Transport ⇒
      endpoint is not connected
      cat: /var/run/resource-agents/mysql-wss/mysql-wss.pid: ⇒
      No such file or directory
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      Stopping MySQL on cic-2.domain.tld
      Error signing on to the CIB service: Transport ⇒
      endpoint is not connected
      Error signing on to the CIB service: Transport ⇒
      endpoint is not connected
      cat: /var/run/resource-agents/mysql-wss/mysql-wss.pid: ⇒
      No such file or directory
      
      Attention! Unauthorized remote access is strictly ⇒
      prohibited!
      
      Stopping MySQL on cic-3.domain.tld
      Error signing on to the CIB service: Transport ⇒
      endpoint is not connected
      Error signing on to the CIB service: Transport ⇒
      endpoint is not connected
      cat: /var/run/resource-agents/mysql-wss/mysql-wss.pid:⇒
       No such file or directory
      INFO    Completed Restore of MySQL Databases
      ****************************************************⇒
      ***************
      *   Run sudo umm off command on all the vCICs as ⇒
      shown below order*
      cic-1.domain.tld
      cic-2.domain.tld
      cic-3.domain.tld
      ****************************************************⇒
      ***************

      ceebackup@cic-1:~$ cic-data-restore -f /var/lib/glance/backup/cic-data-backup.0/⇒
      cic-data-backup _06032017_071500.tgz –m All
      
      INFO    Checking whether cics are in Maintenance mode
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      INFO    Started OpenLDAP restore
      INFO    Removing the old LDAP configuration on cic-1.domain.tld ...
      INFO    Removing the old LDAP configuration on cic-2.domain.tld ...
      INFO    Removing the old LDAP configuration on cic-3.domain.tld ...
      INFO    started LDAP database restore on cic-1.domain.tld
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      INFO    Completed OpenLDAP restore
      INFO    Started Restore OpenStack Configuration Files
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      Copying openstack config from backup file on cic-1.domain.tld
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      Copying openstack config from backup file on cic-2.domain.tld
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      Copying openstack config from backup file on cic-3.domain.tld
      INFO    Completed Restore OpenStack Configuration Files
      INFO    Seems SDN is not present in current setup
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      Cleanup MySQL on cic-2.domain.tld
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      Cleanup MySQL on cic-3.domain.tld
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      Exporting variables on cic-1.domain.tld
      Exporting Extra variable on cic-1.domain.tld
      Starting MySQL on cic-1.domain.tld
      Error signing on to the CIB service: Transport endpoint is not connected
      73fc7222-15ce-11e7-9e15-4f048f9684b4:380860
      /usr/lib/ocf/resource.d/fuel/mysql-wss: line 371: [: too many arguments
      Error signing on to the CIB service: Transport endpoint is not connected
      Warning: Using a password on the command line interface can be insecure.
      ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
      Warning: Using a password on the command line interface can be insecure.
      ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
      0
      Restoring Mysql Openstack databases from backup file on cic-1.domain.tld
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      Exporting variables on cic-2.domain.tld
      Starting MySQL on cic-2.domain.tld
      Error signing on to the CIB service: Transport endpoint is not connected
      00000000-0000-0000-0000-000000000000:-1
      /usr/lib/ocf/resource.d/fuel/mysql-wss: line 371: [: too many arguments
      Error signing on to the CIB service: Transport endpoint is not connected
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      Exporting variables on cic-3.domain.tld
      Starting MySQL on cic-3.domain.tld
      Error signing on to the CIB service: Transport endpoint is not connected
      00000000-0000-0000-0000-000000000000:-1
      /usr/lib/ocf/resource.d/fuel/mysql-wss: line 371: [: too many arguments
      Error signing on to the CIB service: Transport endpoint is not connected
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      Stopping MySQL on cic-1.domain.tld
      Error signing on to the CIB service: Transport endpoint is not connected
      Error signing on to the CIB service: Transport endpoint is not connected
      cat: /var/run/resource-agents/mysql-wss/mysql-wss.pid: No such file or directory
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      Stopping MySQL on cic-2.domain.tld
      Error signing on to the CIB service: Transport endpoint is not connected
      Error signing on to the CIB service: Transport endpoint is not connected
      cat: /var/run/resource-agents/mysql-wss/mysql-wss.pid: No such file or directory
      
      Attention! Unauthorized remote access is strictly prohibited!
      
      Stopping MySQL on cic-3.domain.tld
      Error signing on to the CIB service: Transport endpoint is not connected
      Error signing on to the CIB service: Transport endpoint is not connected
      cat: /var/run/resource-agents/mysql-wss/mysql-wss.pid: No such file or directory
      INFO    Completed Restore of MySQL Databases
      *******************************************************************
      *   Run sudo umm off command on all the vCICs as shown below order*
      cic-1.domain.tld
      cic-2.domain.tld
      cic-3.domain.tld
      *******************************************************************

   2. Restore the OpenLDAP database.

      An example of the command:

      ceebackup@cic-1:~$ cic-data-restore -f /var/lib/galnce/backup/cic-data-backup.0/cic-data-backup _06032017_071500.tgz –m ldap

      ceebackup@cic-1:~$ cic-data-restore -f /var/lib/⇒
      galnce/backup/cic-data-backup.0/cic-data-backup⇒
       _06032017_071500.tgz –m ldap

   3. Restore OpenStack configuration files.

      An example of the command:

      ceebackup@cic-1:~$ cic-data-restore -f /var/lib/glance/backup/cic-data-backup.0/cic-data-backup _06032017_071500.tgz –m config

      ceebackup@cic-1:~$ cic-data-restore -f /var/lib/⇒
      glance/backup/cic-data-backup.0/cic-data-backup⇒
       _06032017_071500.tgz –m config

   4. Restore MySQL databases.

      An example of the output:

      Note:  

      The output contains warnings and error messages.

      The warning for unauthorized access is a default SSH message issued on the vCIC. Error messages are issued because MySQL service is down during restore.

      
      

      ceebackup@cic-1:~$ cic-data-restore -f /var/lib/glance/backup/cic-data-backup.0/⇒ cic-data-backup _06032017_071500.tgz –m mysql Attention! Unauthorized remote access is strictly prohibited! Cleanup MySQL on cic-2.domain.tld Attention! Unauthorized remote access is strictly prohibited! Cleanup MySQL on cic-3.domain.tld Attention! Unauthorized remote access is strictly prohibited! Exporting variables on cic-1.domain.tld Exporting Extra variable on cic-1.domain.tld Starting MySQL on cic-1.domain.tld Error signing on to the CIB service: Transport endpoint is not connected 73fc7222-15ce-11e7-9e15-4f048f9684b4:380860 /usr/lib/ocf/resource.d/fuel/mysql-wss: line 371: [: too many arguments Error signing on to the CIB service: Transport endpoint is not connected Warning: Using a password on the command line interface can be insecure. ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) Warning: Using a password on the command line interface can be insecure. ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) 0 Restoring Mysql Openstack databases from backup file on cic-1.domain.tld Attention! Unauthorized remote access is strictly prohibited! Exporting variables on cic-2.domain.tld Starting MySQL on cic-2.domain.tld Error signing on to the CIB service: Transport endpoint is not connected 00000000-0000-0000-0000-000000000000:-1 /usr/lib/ocf/resource.d/fuel/mysql-wss: line 371: [: too many arguments Error signing on to the CIB service: Transport endpoint is not connected Attention! Unauthorized remote access is strictly prohibited! Exporting variables on cic-3.domain.tld Starting MySQL on cic-3.domain.tld Error signing on to the CIB service: Transport endpoint is not connected 00000000-0000-0000-0000-000000000000:-1 /usr/lib/ocf/resource.d/fuel/mysql-wss: line 371: [: too many arguments Error signing on to the CIB service: Transport endpoint is not connected Attention! Unauthorized remote access is strictly prohibited! Stopping MySQL on cic-1.domain.tld Error signing on to the CIB service: Transport endpoint is not connected Error signing on to the CIB service: Transport endpoint is not connected cat: /var/run/resource-agents/mysql-wss/mysql-wss.pid: No such file or directory Attention! Unauthorized remote access is strictly prohibited! Stopping MySQL on cic-2.domain.tld Error signing on to the CIB service: Transport endpoint is not connected Error signing on to the CIB service: Transport endpoint is not connected cat: /var/run/resource-agents/mysql-wss/mysql-wss.pid: No such file or directory Attention! Unauthorized remote access is strictly prohibited! Stopping MySQL on cic-3.domain.tld Error signing on to the CIB service: Transport endpoint is not connected Error signing on to the CIB service: Transport endpoint is not connected cat: /var/run/resource-agents/mysql-wss/mysql-wss.pid: No such file or directory INFO    Completed Restore of MySQL Databases. Note: Above error messages can be observed while restoration of MySQL.

   5. Restore SDN configuration files.

      An example of the command:

      ceebackup@cic-1:~$ cic-data-restore -f /var/lib/glance/backup/cic-data-backup.0/cic-data-backup_06032017_071500.tgz –m sdn

      ceebackup@cic-1:~$ cic-data-restore -f⇒
       /var/lib/glance/backup/cic-data-backup.0⇒
      /cic-data-backup_06032017_071500.tgz –m sdn

      Note:  

      It is recommended to restore OpenStack configuration prior to SDN configuration. SDN data can be restored alone, if OpenStack data is aligned with SDN data which is yet to be restored.

      
      
6. After completing the restore process, turn off the maintenance mode on the vCICs manually.

   ceebackup@cic-1:~$ sudo umm off

   Note:  

   The vCICs must be set back to operational mode in the order that is presented in the cic-data-restore output in sl-restoreBackupContents in Step 5.

   The relevant part of the output:

   
   

   ******************************************************************* *   Run sudo umm off command on all the vCICs as shown below order* cic-1.domain.tld cic-2.domain.tld cic-3.domain.tld *******************************************************************

7. Perform a health check, as in Health Check Procedure see Section 5.

Note:  

Neutron commands in some cases return the following error after the restore procedure on BSP platforms:

message": "<html><body><h1>504 Gateway Time-out</h1> The server didn't respond in time

message": "<html><body><h1>504 Gateway Time-out</h1> 
The server didn't respond in time

In this case, log on to one of the vCICs and restart the Neutron server:

crm resource restart neutron-server

Verify that the neutron server successfully restarted:

crm resource status neutron-server

5   Perform Health Check

After the MM is off, wait for 10 minutes for CEE to be running again, then perform a health check for the OpenStack services.

1. To check vCIC MM, refer to section Check vCIC Maintenance Mode in Health Check Procedure.
2. To check Pacemaker (vCIC state and cluster resource state), refer to section Check Pacemaker - vCIC State and Cluster Resource State in Health Check Procedure.
3. To check Nova services, refer to section Check Nova Services in Health Check Procedure.
4. To check RabbitMQ cluster status, refer to section Check RabbitMQ Cluster Status in Health Check Procedure.
5. To check OpenStack components, refer to section Check OpenStack Components in Health Check Procedure.