CIC Domain Data Backup

Cloud Execution Environment

1	Introduction
1.1	Scope
2	Prerequisites
2.1	Node Names
2.2	Login Credentials
3	Operations
3.1	Create a Backup
3.2	List Backups
3.3	Show the Contents of Backups
3.4	Show the History of All Backups
3.5	Show the Log File
3.6	Show the Retention Policy
3.7	Change the Retention Policy
3.8	Check Consistency, Clean, and Synchronize Backups
3.9	Show All Available Parameters
4	Cleanup of Aborted or Failed Backups
5	Troubleshooting

1 Introduction

The aim of this document is to give operating instructions on creating and managing CIC domain data backups.

The purpose of the CIC domain data backup operation is to take a copy of CIC domain data that can be used to recover from corrupted databases or misconfigurations.

For instructions on restoring the CEE CIC domain using a CIC domain data backup, refer to the document CIC Domain Data Restore.

Refer to the document Backup and Restore Overview for more information on CIC domain data backup, including the following:

Backup contents
Backup sizes and locations
Retention policies
Procedure durations
External storage requirements
Recommended backup strategy

For an overview of all backup and restore options available in CEE, refer to the Backup and Restore Overview.

1.1 Scope

The procedure described in this document is applicable with the following limitations:

Glance images are not included in the CEE infrastructure backup and are expected to be kept as part of an external backup.

2 Prerequisites

Make sure that no configuration and deployment changes take place while creating backups.

The following conditions are automatically checked by the cic-data-backup command:

All three vCICs are in operational mode.
The mysql service is running in all three vCICs.
The slapd process of the OpenLDAP database is running on all three vCICs.
If SDN is enabled, the SDN service is running on all three vCICs.

If upon the execution of the cic-data-backup create command any of these conditions are not met, the script exits with an error message. For more information, see Section 5.

Perform a health check procedure as described in the document Health Check Procedure. If the system is not healthy at the time of backup, the same faults are present after the CIC domain data restore.

2.1 Node Names

Throughout the document, node names cic1, cic2 and cic3 are used to refer to the vCICs used in the backup procedure. Write down the real node names from the actual deployment and use them when issuing actual command lines while following the examples provided in this document.

Table 1 vCIC Actual Names
Name Throughout the Document	Real Name in Deployment
`cic1`
`cic2`
`cic3`

2.2 Login Credentials

The cic-data-backup command must be run manually by a dedicated CIC domain data infrastructure administration user, which is by default ceebackup. The credentials for this user must be available.

For more information on the user ceebackup, refer to the section on local administrators in the Security User Guide. For more information on initial credentials, refer to the corresponding section in the System Hardening Guideline.

3 Operations

This section describes the parameters of the cic-data-backup command.

3.1 Create a Backup

Syntax

cic-data-backup create [name]

[name] must be a string without spaces. If not specified, the default name is cic_data_backup.

Description

This parameter creates a backup. The new backup is placed under the backup root at /var/lib/glance/backup/cic_data_backup.0

The following items are included in the backup:

SQL dump files of the MySQL databases nova, nova-api, cinder, neutron, glance, keystone, and mysql
A dump file of the LDAP database
The OpenStack configuration files located in the directories /etc/{nova,glance,keystone,cinder,swift,neutron}
In SDN enabled system, SDN configuration files located in the directories /opt/sdnc/opendaylight and /opt/quagga

More detailed information on the contents of the backup is described in Backup and Restore Overview.

Note:: The backup process can be aborted, for example by the user, or because of a system failure. If this occurs, promptly assess the synchronization state of the local and remote backups by following the cleanup guide in Section 4. Cleanup is highly recommended.

Example

Example 1 Create a Backup

ceebackup@cic-1:~$ cic-data-backup create
INFO: Create backup "cic_data_backup" with ID=56e5be90b0e4288105b22906491f056a ...
INFO: Backup STATUS=INCOMPLETE
INFO: Backup the OpenStack Database ...
INFO: Backup the OpenLDAP Database ...
INFO: Backup the Openstack configuration files on host cic-1.domain.tld ...
INFO: Backup the Openstack configuration files on host cic-2.domain.tld ...
INFO: Backup the Openstack configuration files on host cic-3.domain.tld ...
INFO: Backup STATUS=SYNCRONIZING
INFO: Backup STATUS=COMPLETED
WARNING: cic_data_backup.3 on cic-2.domain.tld and cic-1.domain.tld not consistent
WARNING: cic_data_backup.3 on cic-3.domain.tld and cic-1.domain.tld not consistent
INFO: Backup with ID=56e5be90b0e4288105b22906491f056a performed successfully
INFO: Cee-Infra backup file << cee_backup_03032017_155126.tgz >> created on
/var/lib/glance/backup/cic_data_backup.0

The command automatically checks the status of services described in Section 2. If any of the services are not running, the command exits with an error message specifying the service not running. In this case, the backup is not executed. For more information, see Section 5

In some cases, the backup files are not consistent across vCICs. This results in a warning message. The warning can be considered harmless.

An example of the warning message is shown in Example 1, with the following text:

WARNING: cic_data_backup.3 on cic-2.domain.tld and cic-1.domain.tld not consistent
WARNING: cic_data_backup.3 on cic-3.domain.tld and cic-1.domain.tld not consistent

3.2 List Backups

Syntax

cic-data-backup list

Description

This parameter prints a list of all backups in a format compatible with OpenStack. The output contains the following fields:

ID of the backup
Size of the backup
Date and time when the backup was created
Version of the CEE software release
Status of the backup. Possible values are SYNCHRONIZING and COMPLETED
Name of the backup
Directory of the backup

Example

Example 2 List Backups

ceebackup@cic-1:~$ cic-data-backup list
+----------------------------------+--------+---------------------+--------------------------+-------------+-------------------------+--------------+
| ID                               | Size   | Date                | Version                  | Status      | Name                    | Directory    |
+----------------------------------+--------+---------------------+--------------------------+-------------+-------------------------+--------------+
| 56e5be90b0e4288105b22906491f056a | 2.3M   | 03/03/2017,15:51:30 | R6-R6A01-c852ff171e1-9.0 | COMPLETED   | cic_data_backup         | cic_data_backup.0 |
+----------------------------------+--------+---------------------+--------------------------+-------------+-------------------------+--------------+
| f22e9b23eea4a4e420daa332fc54a8e8 | 2.3M   | 03/03/2017,15:31:08 | R6-R6A01-c852ff171e1-9.0 | COMPLETED   | cic_data_backup         | cic_data_backup.1 |
+----------------------------------+--------+---------------------+--------------------------+-------------+-------------------------+--------------+
| aafa83f705aae2ac0860554f7032d4da | 2.3M   | 03/03/2017,15:22:56 | R6-R6A01-c852ff171e1-9.0 | COMPLETED   | cic_data_backup         | cic_data_backup.2 |
+----------------------------------+--------+---------------------+--------------------------+-------------+-------------------------+--------------+
| 2c085180ff7b0b906d3da56a8f07b896 | 2.3M   | 03/03/2017,15:18:22 | R6-R6A01-c852ff171e1-9.0 | SYNCRONIZING | cic_data_backup        | cic_data_backup.3 |
+----------------------------------+--------+---------------------+--------------------------+-------------+-------------------------+--------------+

Example 2 List Backups

ceebackup@cic-1:~$ cic-data-backup list
+----------------------------------+--------+---------------------+--------------------------+⇒
-------------+-------------------------+--------------+
| ID                               | Size   | Date                | Version                  |⇒
 Status      | Name                    | Directory    |
+----------------------------------+--------+---------------------+--------------------------+⇒
-------------+-------------------------+--------------+
| 56e5be90b0e4288105b22906491f056a | 2.3M   | 03/03/2017,15:51:30 | R6-R6A01-c852ff171e1-9.0 |⇒
 COMPLETED   | cic_data_backup         | cic_data_backup.0 |
+----------------------------------+--------+---------------------+--------------------------+⇒
-------------+-------------------------+--------------+
| f22e9b23eea4a4e420daa332fc54a8e8 | 2.3M   | 03/03/2017,15:31:08 | R6-R6A01-c852ff171e1-9.0 |⇒
 COMPLETED   | cic_data_backup         | cic_data_backup.1 |
+----------------------------------+--------+---------------------+--------------------------+⇒
-------------+-------------------------+--------------+
| aafa83f705aae2ac0860554f7032d4da | 2.3M   | 03/03/2017,15:22:56 | R6-R6A01-c852ff171e1-9.0 |⇒
 COMPLETED   | cic_data_backup         | cic_data_backup.2 |
+----------------------------------+--------+---------------------+--------------------------+⇒
-------------+-------------------------+--------------+
| 2c085180ff7b0b906d3da56a8f07b896 | 2.3M   | 03/03/2017,15:18:22 | R6-R6A01-c852ff171e1-9.0 |⇒
 SYNCRONIZING | cic_data_backup        | cic_data_backup.3 |
+----------------------------------+--------+---------------------+--------------------------+⇒
-------------+-------------------------+--------------+

3.3 Show the Contents of Backups

Syntax

cic-data-backup show <ID|name>

ID|name is a mandatory field and the parameter accepts either a backup ID or a name. Both can be obtained using the cic-data-backup list command.

Description

This parameter shows the contents of all backups matching the given ID or name, that is, all files in the corresponding cic_data_backup.<x> directory. If the given name is not unique, the contents of all matching backups are listed separately. The output contains the following fields:

Size of the file in the backup
Date and time when the file in the backup was modified
Path and filename of the file in the local file system

Example

The following is an example of a partial printout for the command cic-data-backup show:

Example 3 Show the Contents of a Backup

ceebackup@cic-1:~$ cic-data-backup show 56e5be90b0e4288105b22906491f056a
Contents of backup cic_data_backup (56e5be90b0e4288105b22906491f056a):
2.3M 2017-03-03 15:51 /var/lib/glance/backup/cic_data_backup.0
652K 2017-03-03 15:51 /var/lib/glance/backup/cic_data_backup.0/cee_backup_03032017_155126.tgz
4.0K 2017-03-03 15:51 /var/lib/glance/backup/cic_data_backup.0/.metadata
1.1M 2017-03-03 15:51 /var/lib/glance/backup/cic_data_backup.0/openstack_db
12K 2017-03-03 15:51 /var/lib/glance/backup/cic_data_backup.0/openstack_db/nova_api.sql
152K 2017-03-03 15:51 /var/lib/glance/backup/cic_data_backup.0/openstack_db/nova.sql
248K 2017-03-03 15:51 /var/lib/glance/backup/cic_data_backup.0/openstack_db/neutron.sql
528K 2017-03-03 15:51 /var/lib/glance/backup/cic_data_backup.0/openstack_db/mysql.sql
[...]

3.4 Show the History of All Backups

Syntax

cic-data-backup showHistory

Description

This parameter prints the list of all backups that have been initiated since the system was installed, independent of whether they have been completed or not.

The output shows the following fields:

ID of the backup
Size of the backup
Date and time when the backup was created
Version of the CEE software release
Status of the backup. Possible values are SYNCHRONIZING and COMPLETED
Name of the backup

Example

The following is an example of a partial printout for the command cic-data-backup showHistory:

Example 4 Show the History of All Backups

ceebackup@cic-1:~$ cic-data-backup showHistory
926da6612e88e4993eb74dd6572e0494 5.1M Mon 06 Mar 2017 07:14:34 R6-R6A02-8de25627fe1-9.0 COMPLETED ⇒
cic_data_backup
cd9d54a98b0aef59193476947da51f46 5.1M Mon 06 Mar 2017 07:15:03 R6-R6A02-8de25627fe1-9.0 COMPLETED ⇒
cic_data_backup
[...]
Example 5 Show the History of All Backups

3.5 Show the Log File

Syntax

cic-data-backup showLog <n>

n must be an integer number.

Description

This parameter prints the last n lines of the log file. If n is not specified, it shows the entire log file. The log file records all executions of the cic-data-backup command and the output of operations executed by the command. The log file can also be found under /var/lib/glance/backup/logs/backup.log

Note:: The log file is only available on the vCIC where the backup was performed.

The output shows the following fields:

Date and time when the log entry was written.
Log level of the log entry. Possible values are INFO, WARNING, and ERROR.
Log entry

Example

Example 5 Show the Log File

ceebackup@cic-1:~$ cic-data-backup showLog 5
Fri 03 Mar 2017 15:51:57 INFO -------------------------------------------------------------
Fri 03 Mar 2017 15:53:04 INFO >>backup list
Fri 03 Mar 2017 15:53:36 INFO >>backup show cic_data_backup.0
Fri 03 Mar 2017 15:53:48 INFO >>backup show 56e5be90b0e4288105b22906491f056a
Fri 03 Mar 2017 15:54:37 INFO >>backup showHistory

3.6 Show the Retention Policy

Syntax

cic-data-backup showPolicy

Description

This parameter prints the current retention policy configured on all vCICs. This parameter is taken from the corresponding rsnapshot configuration file which is located under /etc/rsnapshot-cee.conf on every vCIC.

Note:: The retention policy is always the same on all vCICs.

Example

Example 6 Show the Retention Policy

ceebackup@cic-1:~$ cic-data-backup showPolicy
cic-1.domain.tld Retention policy=3
cic-2.domain.tld Retention policy=3
cic-3.domain.tld Retention policy=3

3.7 Change the Retention Policy

Note:: The retention policy value can only be increased. Changing the value to a lower number is not allowed as it would delete existing backups.
As increasing the number of backups increases the required amount of storage for the backups, it is strongly recommended that the default value of 3 is used.

Syntax

cic-data-backup changePolicy <n>

n must be an integer number and larger than the current retention policy, which can be shown by issuing the command: cic-data-backup showPolicy described in Section 3.6.

Description

The default value of the retention policy is 3, meaning that three backups are kept. If it is required to keep more backups, the retention policy can be changed with the cic-data-backup command.

By increasing this parameter, the storage space used by the backups increases. Make sure that there is enough free space under /var/lib/glance/backup/ to keep additional backups.

Note:: If changing the retention policy fails, inconsistent configurations can occur on the vCICs. Immediately check the retention policy on all vCICs through the cic-data-backup showPolicy after failing to change the retention policy and try to change it again.

Example

Example 7 Change the Retention Policy

ceebackup@cic1:~$ cic-data-backup changePolicy 6
INFO: Change retention policy=6 successful

3.8 Check Consistency, Clean, and Synchronize Backups

Syntax

cic-data-backup clean [-q]

Description

When this parameter is provided, the cic-data-backup command removes temporary files which were not deleted upon an aborted backup. It also checks if the backups on all vCICs are consistent and synchronized. If it detects inconsistencies, it replaces remote backups with the corresponding local copy. For example, the backup in the local cic_data_backup directory replaces the corresponding remote backup in the cic_data_backup directory. Resolving the inconsistencies is not forced by default with the clean option; the user will be prompted to decide whether to resolve them or not. The optional -q parameter overrides the default behavior by resolving the inconsistencies without prompting the user. See the cleanup procedure described in Section 4 to determine when it is safe to replace remote backups.

Example

Example 8 Consistency Check, Cleanup, and Synchronization

ceebackup@cic-1:~$ cic-data-backup clean
WARNING: cic_data_backup.3 on cic-2.domain.tld and cic-1.domain.tld not consistent
Do you want to rsync cic_data_backup.3 from local host cic-1.domain.tld to cic-2.domain.tld?
[y|n]: y
INFO: rsync cic_data_backup.3 from local host cic-1.domain.tld to cic-2.domain.tld succeeded
WARNING: cic_data_backup.3 on cic-3.domain.tld and cic-1.domain.tld not consistent
Do you want to rsync cic_data_backup.3 from local host cic-1.domain.tld to cic-3.domain.tld?
[y|n]: y
INFO: rsync cic_data_backup.3 from local host cic-1.domain.tld to cic-3.domain.tld succeeded
Cleanup finished

3.9 Show All Available Parameters

Syntax

cic-data-backup help | -h | --help

Description

This parameter prints all available parameters of the cic-data-backup command and the corresponding syntax.

Example

Example 9 Show All Available Parameters

ceebackup@cic1:~$ cic-data-backup help

cic-data-backup
        create [name]:  create new backup named [name]
        list:           list the current active backups
        show ID/Name:   show the content of a specified backup ID/Name
        showHistory:    show history of all the backups created since CIC is up
        showLog [N]:    show N(integer) last lines of the log file. If N not specified, it will
                        show everything
        showPolicy:     show the current retention policy
        changePolicy N: change the retention policy to N(integer)
        clean [-q]:     check consistency and cleanup temporary directories  
                        -q quiet mode, does not prompt user when cleaning inconsistencies
        help:           show this help and exit

4 Cleanup of Aborted or Failed Backups

The backup process can be aborted by the user. The backup can also fail if a system failure occurs while the backup is created. If a backup is aborted or fails, the synchronization state of local and remote backups must be assessed immediately.

The following scenarios are possible:

The backup was not created. In this case, the backup is not listed among the backups, but temporary files can be present in the system. The temporary files must be cleared as described in Section 3.8 on the same vCIC where the aborted backup was run, and a new backup must be performed as described in Section 3.1.
The backup was created, but it was not synchronized among the vCICs. In this case, the local backup is listed with the status SYNCHRONIZING, and there are no remote backups. The backups must be synchronized as described in Section 3.8, on the same vCIC where the aborted backup was run.
Note:
The backup remains in state SYNCHRONIZING in the printout of the command cic-data-backup list, even after successful manual synchronization. This is a known fault.
The backup was created and synchronized before the process was aborted. In this case, the backup is listed with the status COMPLETED. No action is required.

To assess the status of the backup, list all backups on the same vCIC where the aborted backup was run, as described in Section 3.2. The backup can be identified based on the time of creation listed under Date in the printout.

An example printout with a not synchronized backup and cleanup is the following:

Example 10 Cleanup of an Aborted Backup

ceebackup@cic-1:~$ cic-data-backup create backup1
INFO: Create backup "backup1" with ID=23cf72fa1b1cee42b46365fa9d92e8a7 ...
INFO: Backup STATUS=INCOMPLETE
INFO: Backup the OpenStack Database ...
INFO: Backup the OpenLDAP Database ...
INFO: Backup the Openstack configuration files on host cic-1.domain.tld ...
^CWARNING: Cleanup after aborted "backup create"

ceebackup@cic-1:~$ cic-data-backup list
+----------------------------------+--------+---------------------+--------------------------+---------------+-------------------------+--------------+
| ID                               | Size   | Date                | Version                  | Status        | Name                    | Directory    |
+----------------------------------+--------+---------------------+--------------------------+---------------+-------------------------+--------------+
| 23cf72fa1b1cee42b46365fa9d92e8a7 | 5.4M   | 03/09/2017,14:07:24 | R6-R6A02-14ed247cd62-9.0 | SYNCHRONIZING | backup1                 | cic_data_backup.0|
+----------------------------------+--------+---------------------+--------------------------+---------------+-------------------------+--------------+
| 4ac6f37b8315765495ff2e67c8a6a6f0 | 5.4M   | 03/09/2017,14:06:52 | R6-R6A02-14ed247cd62-9.0 | COMPLETED     | cic_data_backup         | cic_data_backup.1|
+----------------------------------+--------+---------------------+--------------------------+---------------+-------------------------+--------------+
| 2b98579cc963aef98e26ec8f80723cf3 | 5.0M   | 03/09/2017,06:28:47 | R6-R6A02-14ed247cd62-9.0 | COMPLETED     | cic_data_backup         | cic_data_backup.2|
+----------------------------------+--------+---------------------+--------------------------+---------------+-------------------------+--------------+
| 006c0d5ffd6ca4426e00ad1dceb5442b | 5.0M   | 03/09/2017,06:15:31 | R6-R6A02-14ed247cd62-9.0 | COMPLETED     | cic_data_backup         | cic_data_backup.3|
+----------------------------------+--------+---------------------+--------------------------+---------------+-------------------------+--------------+

ceebackup@cic-1:~$ cic-data-backup clean
WARNING: cic_data_backup.0 on cic-2.domain.tld and cic-1.domain.tld not consistent
Do you want to rsync cic_data_backup.0 from local host cic-1.domain.tld to cic-2.domain.tld?
[y|n]: y
INFO: rsync cic_data_backup.0 from local host cic-1.domain.tld to cic-2.domain.tld succeeded
WARNING: cic_data_backup.0 on cic-3.domain.tld and cic-1.domain.tld not consistent
Do you want to rsync cic_data_backup.0 from local host cic-1.domain.tld to cic-3.domain.tld?
[y|n]: y
INFO: rsync cic_data_backup.0 from local host cic-1.domain.tld to cic-3.domain.tld succeeded
Cleanup finished

ceebackup@cic-1:~$ cic-data-backup list
+----------------------------------+--------+---------------------+--------------------------+---------------+-------------------------+--------------+
| ID                               | Size   | Date                | Version                  | Status        | Name                    | Directory    |
+----------------------------------+--------+---------------------+--------------------------+---------------+-------------------------+--------------+
| 23cf72fa1b1cee42b46365fa9d92e8a7 | 5.4M   | 03/09/2017,14:07:24 | R6-R6A02-14ed247cd62-9.0 | SYNCHRONIZING | backup1                 | cic_data_backup.0|
+----------------------------------+--------+---------------------+--------------------------+---------------+-------------------------+--------------+
| 4ac6f37b8315765495ff2e67c8a6a6f0 | 5.4M   | 03/09/2017,14:06:52 | R6-R6A02-14ed247cd62-9.0 | COMPLETED     | cic_data_backup         | cic_data_backup.1|
+----------------------------------+--------+---------------------+--------------------------+---------------+-------------------------+--------------+
| 2b98579cc963aef98e26ec8f80723cf3 | 5.0M   | 03/09/2017,06:28:47 | R6-R6A02-14ed247cd62-9.0 | COMPLETED     | cic_data_backup         | cic_data_backup.2|
+----------------------------------+--------+---------------------+--------------------------+---------------+-------------------------+--------------+
| 006c0d5ffd6ca4426e00ad1dceb5442b | 5.0M   | 03/09/2017,06:15:31 | R6-R6A02-14ed247cd62-9.0 | COMPLETED     | cic_data_backup         | cic_data_backup.3|
+----------------------------------+--------+---------------------+--------------------------+---------------+-------------------------+--------------+

Example 10 Cleanup of an Aborted Backup

ceebackup@cic-1:~$ cic-data-backup create backup1
INFO: Create backup "backup1" with ID=23cf72fa1b1cee42b46365fa9d92e8a7 ...
INFO: Backup STATUS=INCOMPLETE
INFO: Backup the OpenStack Database ...
INFO: Backup the OpenLDAP Database ...
INFO: Backup the Openstack configuration files on host cic-1.domain.tld ...
^CWARNING: Cleanup after aborted "backup create"
ceebackup@cic-1:~$ cic-data-backup list
+----------------------------------+--------+---------------------+--------------------------+⇒
---------------+-------------------------+--------------+
| ID                               | Size   | Date                | Version                  |⇒
 Status        | Name                    | Directory    |
+----------------------------------+--------+---------------------+--------------------------+⇒
---------------+-------------------------+--------------+
| 23cf72fa1b1cee42b46365fa9d92e8a7 | 5.4M   | 03/09/2017,14:07:24 | R6-R6A02-14ed247cd62-9.0 |⇒
 SYNCHRONIZING | backup1                 | cic_data_backup.0|
+----------------------------------+--------+---------------------+--------------------------+⇒
---------------+-------------------------+--------------+
| 4ac6f37b8315765495ff2e67c8a6a6f0 | 5.4M   | 03/09/2017,14:06:52 | R6-R6A02-14ed247cd62-9.0 |⇒
 COMPLETED     | cic_data_backup         | cic_data_backup.1|
+----------------------------------+--------+---------------------+--------------------------+⇒
---------------+-------------------------+--------------+
| 2b98579cc963aef98e26ec8f80723cf3 | 5.0M   | 03/09/2017,06:28:47 | R6-R6A02-14ed247cd62-9.0 |⇒
 COMPLETED     | cic_data_backup         | cic_data_backup.2|
+----------------------------------+--------+---------------------+--------------------------+⇒
---------------+-------------------------+--------------+

ceebackup@cic-1:~$ cic-data-backup clean
WARNING: cic_data_backup.0 on cic-2.domain.tld and cic-1.domain.tld not consistent
Do you want to rsync cic_data_backup.0 from local host cic-1.domain.tld to cic-2.domain.tld?
[y|n]: y
INFO: rsync cic_data_backup.0 from local host cic-1.domain.tld to cic-2.domain.tld succeeded
WARNING: cic_data_backup.0 on cic-3.domain.tld and cic-1.domain.tld not consistent
Do you want to rsync cic_data_backup.0 from local host cic-1.domain.tld to cic-3.domain.tld?
[y|n]: y
INFO: rsync cic_data_backup.0 from local host cic-1.domain.tld to cic-3.domain.tld succeeded
Cleanup finished

ceebackup@cic-1:~$ cic-data-backup list
+----------------------------------+--------+---------------------+--------------------------+⇒
---------------+-------------------------+--------------+
| ID                               | Size   | Date                | Version                  |⇒
 Status        | Name                    | Directory    |
+----------------------------------+--------+---------------------+--------------------------+⇒
---------------+-------------------------+--------------+
| 23cf72fa1b1cee42b46365fa9d92e8a7 | 5.4M   | 03/09/2017,14:07:24 | R6-R6A02-14ed247cd62-9.0 |⇒
 SYNCHRONIZING | backup1                 | cic_data_backup.0|
+----------------------------------+--------+---------------------+--------------------------+⇒
---------------+-------------------------+--------------+
| 4ac6f37b8315765495ff2e67c8a6a6f0 | 5.4M   | 03/09/2017,14:06:52 | R6-R6A02-14ed247cd62-9.0 |⇒
 COMPLETED     | cic_data_backup         | cic_data_backup.1|
+----------------------------------+--------+---------------------+--------------------------+⇒
---------------+-------------------------+--------------+
| 2b98579cc963aef98e26ec8f80723cf3 | 5.0M   | 03/09/2017,06:28:47 | R6-R6A02-14ed247cd62-9.0 |⇒
 COMPLETED     | cic_data_backup         | cic_data_backup.2|
+----------------------------------+--------+---------------------+--------------------------+⇒
---------------+-------------------------+--------------+

5 Troubleshooting

If any of the services described in Section 2 are not running when the command cic-data-backup create is executed, the script exits with the following error message:

ERROR: only <N> controllers with <service_name> started.

where <N> is the number of vCICs with the service running, and <service_name> identifies the service that is not running on all three vCICs. Do the following:

Depending on the service name in the error message, identify manually the node on which the service is not running, using the relevant command:

Service name	Command	Execute on	Example printout
vCIC	`sudo crm status \| grep Online`	any vCIC	`Online: [ cic1 cic2 cic3 ]`
`mysql`	`sudo crm resource status \| grep ⇒ p_mysql -A1 \| grep Startedsudo crm resource status \| grep p_mysql -A1 \| grep Started`	any vCIC	`Started: [ cic1 cic2 cic3 ]`
`slapd`	`sudo pgrep slapd`	each vCIC	`ceebackup@cic1:~$ sudo pgrep slapd` `1234` `ceebackup@cic2:~$ sudo pgrep slapd` `5678` `ceebackup@cic3:~$ sudo pgrep slapd` `4321`
SDN⁽¹⁾	`sudo crm resource status \| grep ⇒ -A 1 p_sdnc-service \|grep Startedsudo crm resource status \| grep -A 1 p_sdnc-service \|grep Started`	any vCIC	`Started: [ cic1 cic2 cic3 ]`⁽²⁾

(1) If SDN is enabled

(2) If SDN is not enabled, the command gives an empty output

Collect troubleshooting data as described in the Data Collection Guideline
Consult the next level of maintenance support. Further actions are outside the scope of this instruction.