User Guide 1/1553-CNL 121 695/10 Uen E

CUDB VNF Lifecycle Management

Contents


1 Introduction

This document describes system administration tasks performed in the Virtualized Network Function (VNF) Lifecycle Manager (VNF-LCM). The VNF-LCM provides a workflow execution environment and a web-based application for managing VNF lifecycle procedures.

The workflows are ordered sequences of steps for automating common use cases of the VNFs. A workflow provides a means to orchestrate simple and complex sequences of manual or automated tasks.

1.1 Purpose and Scope

This document covers the following workflow-based lifecycle management procedures:

  • Instantiate vCUDB system.

  • Terminate vCUDB system.

  • vCUDB Upgrade Preparation.

  • Upgrade vCUDB system.

    Note: Application Schema update is not included.

All manual preparation steps that must be executed by Ericsson personnel are out of the scope of this document. See Prerequisites for more information.

The Instantiate vCUDB system is not applicable for vCUDB system expansions.

Note: A Virtualized CUDB node is referred to as a vCUDB VNF instance throughout the document.
Attention!

If any step fails during workflow execution, refer to Troubleshooting section and contact CUDB Administrator as Ericsson support to follow the VNF-LCM Admin Guide or the corresponding manual procedures (for example installation, upgrade, and so on) for a deeper troubleshooting. The troubleshooting of those manual procedures is out of the scope of this document.

1.2 Revision Information

Rev. A

Rev. B

Rev. C

Rev. D

Rev. E

Other than editorial changes, this document has been updated as follows:

1.3 Target Groups

This document is intended for CUDB Operator.

For some of the actions described in the document, Ericsson personnel must be contacted as:

CUDB Administrator  

The CUDB Administrator prepares configuration files, execute needed preparation steps and can run certain troubleshooting tasks.
Cloud Administration  

The Cloud Administrator is the cloud service provider who executes required actions on the cloud infrastructure.

1.4 Typographic Conventions

Typographic Conventions can be found in the following document:

1.5 Prerequisites

This section describes the prerequisites that must be fulfilled before executing any of the workflows.

CUDB Administrator or Cloud Administrator, or both, must execute all initial steps included in the manual installation instruction for vCUDB to provide system preparation and configuration files needed to run the workflow-based lifecycle management procedures and, in case of upgrade, to be able to have all preparations ready from upgrade documentation.

CUDB Administrator must check if there is any limitation to execute graceful termination workflow according to the node decommissioning procedure.

1.5.1 Hardware and Software

The following virtual and physical hardware and software are required:

  • Software delivery package (CUDB Workflow pack).

  • VNF-LCM release is Media build version 4.4.18 or higher.

  • VNF-LCM up and running using Ericsson Network Management System (NMS), either Operations Support System for Radio and Core (OSS-RC) or Ericsson Network Manager (ENM).

    For example, in case of OSS-RC, to check the correct functioning of VNF-LCM, follow the steps defined in the Post Installation Verification section of VNF-LCM CEE/Openstack Installation Instructions, 1, in the OSS-RC documentation.

    Note: The Virtualized Infrastructure Manager (VIM) connection information in VNF-LCM framework has to be configured. This information is used by the workflows to connect to VIM and perform operations. Add as many VIMs as needed to VNF-LCM framework and add as many tenants as needed to the previously added VIM. Refer to the VNF-LCM CLI Admin section of VNF-Lifecycle Manager System Administration Guide, in the OSS-RC documentation.

  • Check the size of the workflow package and make sure the VNF-LCM disk available size is enough.

  • Any extra file not mentioned in that document must be stored in VNF-LCM under /vnflcm-ext/ directory to prevent the disk from getting full where workflows are executed.

  • Virtual infrastructure must be prepared for vCUDB deployment, according to the manual installation instruction.

1.5.2 Configuration Files

1.5.2.1 Instantiate Configuration Files

Contact CUDB Administrator to obtain the following files. CUDB Administrator will get the files after executing all initial steps included in the installation instruction for vCUDB.

  • vcudb_hot.yaml

  • input_files

  • vcudb_DS_scaling.yaml

  • vcudb_hot_parameters.json

1.5.2.2 Upgrade Configuration Files

Contact CUDB Administrator to obtain the following files. CUDB Administrator will get the files after all manual preparation tasks from upgrade documentation are finished.

  • License files named CUDB_LKF_<cudb_node_id> and fingerprint file named lm_fingerprint_<cudb_node_id> in case the vCUDB software version is previous to 1.10.

    Note: _<cudb_node_id> corresponds to the cudbLocalNodeId/cudbRemoteNodeId defined in the CUDB configuration model for each CUDB node of the system.

  • Upgrade package named CUDB_PACKAGE_x86_64-CXP.

  • preventive_Was.tar, if available.

1.5.3 Other Requirements

Contact Cloud Administrator regarding the following aspects:

  • If vCUDBs are already installed, ensure that property tags included in the stacks are "vCUDB". If not, update them as the following:

    Select the appropriate stack for updating and update the "vCUDB" tag, if needed.

    For example:
    source <openrc>
heat stack-list
heat stack-update -x --tags "vCUDB" <stack_name>

Contact CUDB Administrator regarding the following aspects:

  • In the case of instantiate:

    • To add multiple SITE_VIP IPs in a live vCUDB VNF (according to the network configuration of the vCUDB), if the vCUDB system consists of more than 10 vCUDB VNFs.

    • To change the image in System Controllers (according to the installation instruction for vCUDB), when instantiation is finished.

  • In the case of upgrade:

    • If upgrade is not performed again after executing an automatic fallback and the system needs to be fully restored to previous CUDB software version.

    • If vCUDB base software version is previous to 1.12, to enable VM evacuation support in Cloud Execution Environment (CEE).

2 Onboard

This section describes how to prepare for workflow-based VNF operations using VNF-LCM. Performing this procedure is a prerequisite for lifecycle operations.

For more information on installing workflows, refer to the Workflow Bundle Administration section of VNF-Lifecycle Manager System Administration Guide document in the OSS-RC documentation.

Execute the following commands on the VNF-LCM Services Virtual Machine (VM):

Steps

  1. Connect to VNF-LCM:
    ssh cloud-user@<VNFLAF-services_ip>
  2. Copy the CUDB Workflow pack CUDB_VNFLCM_WORKFLOWS-CXP9040847.tar file into /home/cloud-user directory.
  3. Decompress the CUDB Workflow pack CUDB_VNFLCM_WORKFLOWS-CXP9040847.tar
    [cloud-user@vnflaf-services ~]$ tar -xvf CUDB_VNFLCM_WORKFLOWS-CXP9040847.tar
  4. Install the CUDB Workflow pack:
    1. Switch to root user on vnflaf-services VM:
      [cloud-user@vnflaf-services ~]$ su – root
      [root@vnflaf-services ~]#
    2. Verify that the pack is not installed, by running the list command:
      rpm -qa |grep ERICvCUDB
    3. Uninstall the previous version, if there is one, and take the input data from the previous printout:
      # wfmgr bundle uninstall --package=<Name> --version=<Version>
    4. To install the Workflow pack, run the install command. The rpm file is located in the /home/cloud-user folder by default.
      # wfmgr bundle install --package=/home/cloud-user/<workflow_bundle_rpm_file>
      The expected output must be similar to the below example:

      Example

      -----------------------------------------------------------------------------------------------------------

package_name                         | pre_install   | install    | post_install    |             message |

------------------------------------------------------------------------------------------------------------

| ERICvCUDB_CXP9035445-1.9.20-1.noarch.rpm |   success   | success |   success    | package installation successful |

------------------------------------------------------------------------------------------------------------
      For more information on the output of the command, go to /var/log/wfmgr-cli-log/logfile.log.

3 Procedures

This section describes how to perform LCM operations. VNF-LCM procedures use workflow instances.

Launch VNF-LCM from GUI web browser:

http://<vnflaf-services_ip>/index.html#workflows

Figure 1 shows the example of VNF Lifecycle Management, where the workflow is shown.

Figure 1   Workflow Overview

3.1 Instantiate vCUDB System

This section describes how to instantiate a VNF using VNF-LCM.

This workflow is used to install a vCUDB node in a CUDB system.

Note: To have an installed a vCUDB system replicating among its vCUDB nodes, this workflow must be executed several times and it has to be launched consecutively without waiting for one VNF to finish before launching the next one (one workfllow per each VNF comprising the vCUDB system running in parallel).

If a vCUDB system consists of more than 10 vCUDB VNFs, once the instantiations are finished, add multiple SITE_VIP IPs in a live CUDB node.

3.1.1 Instantiate vCUDB System Prerequisites

Check that all general Prerequisites have been fulfilled at this point.

Besides, the following configuration files for one vCUDB system must be available:

  • Per vCUDB VNF:

    • Input files

    • vcudb_hot.yaml

    • vcudb_hot_parameters.json

    • vcudb_DS_scaling.yaml

The above mentioned files are generated by the CUDB Installation Files Generator Tool under the structure <CUDB System>/configurations/<node_id>, and it must go under /vnflcm-ext/backups/workflows/cudbvnfd directory.

Note: Remember that all files must have permission to be executed by jboss at least. If it is not the case, change it as follows:
 
cd /vnflcm-ext/backups/workflows/cudbvnfd
chown -R jboss_user:jboss *

The final structure directory is created manually as shown in Figure 2.

Figure 2   Final Structure Directory for Instantiate vCUDB System
Note: Different vCUDB systems can be defined. Select one during instantiation of a VNF. Moreover, one vCUDB system consists of one or several VNFs, that is, CUDB nodes.

All configuration files must be placed manually in the corresponding directories.

3.1.2 Instantiate vCUDB VNF

  1. In the VNF-LCM Workflows screen, select Instantiate vCUDB VNF and click Start a New Instance.
  2. Instance Name field is filled out, click Submit.

  3. Select the newly-created workflow from the Instance with User Tasks panel, and click on the man icon.
  4. On the Workflow Instance screen, add VNF Name, select VNF descriptor ID to instantiate, and click Submit.

    The Select VNF descriptor ID field displays VNF configurations available for instantiation in the /vnflcm-ext/backups/workflows/cudbvnfd/ directory.
  5. On the Get Instance Configuration screen, select a VNF configuration to instantiate, and click Submit.

    The Select Configuration for the VNF instance field displays VNF configurations available for instantiation in the /vnflcmext/backups/workflows/cudbvnfd/vCUDB_1/configurations directory.
    Refresh the web page.
  6. On the Select VIM screen, select a VIM, and click Submit.

  7. On the Select Tenant screen, select a Tenant, and click Submit.

Result: On the Workflow Instance screen, click on Workflow Diagram and Workflow Log to see the progress.

Note: Refresh the web page from time to time.
  1. When instantiation is finished, change the image in the System Controllers, contact CUDB Administrator.

3.1.2.1 Workflow Log in GUI

The workflow log shows the ongoing execution steps. The expected progress information output must be similar to the following example:

3.1.2.2 vCUDB System Completion Steps

The workflow log contains the system completions step results, that must be executed once per CUDB system.

All the vCUDB nodes but the last will report a WARNING message like the following:

The last vCUDB node handles the system completion steps, and it is reported as completed in the log:

Completed

Do!

When the system completion steps have been executed successfully, the input_files must be removed from the VNF-LCM server by the Cloud Administrator.

If the system completion is not executed, vCUDB installation is not finished and new instantiation of the VNFs or manual intervention is needed. See Troubleshooting for further details.

3.1.2.3 vCUDB System Integration

After the successful execution of the vCUDB instantiation, contact the CUDB Administrator to run manual installation instructions for vCUDB to complete the VNF initial configuration. CUDB Administrator has to complete the integration of the newly deployed vCUDB system with the surrounding nodes too.

3.2 Terminate vCUDB System

This section describes how to terminate a VNF using VNF-LCM.

This workflow can be used to decommission a vCUDB system and free the resources by executing it consecutively on each VNF comprising the vCUDB system.

Stop!

Decommisioning of a full CUDB Site is not allowed since the last node in the site cannot be decommisioned.

Terminate the VNF by executing either of the following options:

  • Forceful: delete the VNF instance where data loss can occur while executing.

  • Graceful: disabling and deleting VNF instance in proper order to minimize data loss.

3.2.1 Terminate vCUDB VNF

Steps

  1. In the VNF-LCM Workflows screen, select Terminate vCUDB VNF, and click Start a New Instance.
  2. Instance Name field is filled out, click Submit.
  3. Select the newly-created workflow from the Active Instances panel and click on the man icon.
    Result: Traffic stops after VNF is terminated. On the Workflow Instances screen, click on Workflow Diagram and Workflow Log to see the progress.
    Note: Refresh the web page.
  4. On the Select VIM screen, select a VIM, and click Submit.
  5. On the Select Tenant screen, select a Tenant, and click Submit.
  6. On the Workflow Instances screen, select the VNF to terminate and the termination option to be executed, either graceful or forceful. After that, click Submit.
    1

    Graceful termination: If VNF is gracefully terminated, there will be no data loss in the ongoing traffic.

    Forceful termination: If VNF is forcefully terminated, all ongoing traffic will be lost.
  7. The chosen option must be confirmed on the next screen.
    2

Results

The VNF instance is terminated. On the Workflow Instances screen, click on Workflow Diagram and Workflow Log to see the progress.

3.2.1.1 Workflow Log in GUI

The workflow log shows the ongoing execution steps. The expected progress information output must be similar to the following example:

3
Note: Refresh the web page.

3.3 vCUDB Upgrade Preparation

This section describes how to execute upgrade preparation for a vCUDB system using VNF-LCM. This workflow is used to prepare vCUDB system for upgrade execution and execute different health checks to verify that the system is working properly before executing upgrade. To execute upgrade preparation phase on a vCUDB system, this workflow must be executed only once for the system on chosen VNF. Operations executed in this workflow are system-level operations.

Before starting the upgrade preparation workflow execution, see Appendix: General Considerations when Upgrading for more information about the upgrade procedure.

Note: The troubleshooting of the upgrade workflow is out of the scope of this document. In case of failure, please contact CUDB Administrator.

3.3.1 vCUDB Upgrade Preparation Prerequisites

Check that all general Prerequisites have been fulfilled at this point.

Besides, the following configuration files for one vCUDB system must be available:

  • Per vCUDB system:

    • upgrade package named CUDB_PACKAGE_x86_64-CXP

    • preventive_WAs.tar , if available.

  • Per vCUDB node:

    • licenses file named CUDB_LKF_<cudb_node_id>.xml

      Example:

      CUDB_LKF_10.xml

    • fingerprint file named lm_fingerprint_<cudb_node_id>

      Example:

      lm_fingerprint_10

    Note: Licenses and fingerprint files are required if the CUDB system version is previous to 1.10.

All the previous files must go under /vnflcm-ext/backups/workflows/cudbvnfd/upgrade directory. The final structure directory is created manually as shown in Figure 3.

Figure 3   Final Structure Directory for vCUDB Upgrade Preparation

All configuration files must be placed manually in the corresponding directories.

3.3.2 vCUDB Upgrade Preparation Steps

Steps

  1. In the VNF-LCM Workflows screen, select vCUDB Upgrade Preparation, and click Start a New Instance.
  2. Instance Name field is filled out, click Submit.
  3. Select the newly-created workflow from the Instance with User Tasks panel, and click on the man icon.
  4. On the Select VIM screen, select a VIM, and click Submit.
  5. On the Select Tenant screen, select a Tenant, and click Submit.
  6. On the Collect User Data for Upgrade Preparation screen, select a folder with upgrade input files, VNF instance and root password for selected VNF instance, and click Submit.
    The Select folder containing input files for upgrade of the vCUDB system field displays VNF configurations available for upgrade preparation in the /vnflcm-ext/backups/workflows/cudbvnfd/upgrade directory.

    If the upgrade preparation workflow execution fails only due to wrongly prepared input files, choose the Restart Upgrade Preparation option to execute workflow execution from the start.

3.3.2.1 Workflow Log in GUI

The workflow log shows the executed procedures and the progress in percentage. The expected progress information output must be similar to the following example:

The number of executed procedures and the total number of procedures in this phase depend on the upgrade path. The duration of each procedure is different. It takes more time for some procedures to be executed on all the nodes of the system.

3.4 vCUDB Upgrade System

This section describes how to execute upgrade of CUDB SW for a vCUDB system using VNF-LCM. To execute upgrade on a vCUDB system, this workflow must be executed once for every VNF in the system. Upgrade workflow execution should be performed one by one. Some operations done in this workflow are system wide.

Note: The troubleshooting of the upgrade workflow is out of the scope of this document. In case of failure, please contact CUDB Administrator.

3.4.1 vCUDB Upgrade System Prerequisites

Check that all general Prerequisites have been fulfilled at this point.

Besides, vCUDB Upgrade preparation workflow must be executed previously as described in vCUDB Upgrade Preparation, without any error. See also Appendix: General Considerations when Upgrading.

3.4.2 vCUDB Upgrade VNF Steps

Steps

  1. In the VNF-LCM Workflows screen, select vCUDB Upgrade, and click Start a New Instance.
  2. Instance Name field is filled out, click Submit.
  3. Select the newly-created workflow from the Instance with User Tasks panel, and click on the man icon.
  4. On the Select VIM screen, select a VIM, and click Submit.
  5. On the Select Tenant screen, select a Tenant, and click Submit.
  6. On the Collect User data for Upgrade screen, select VNF instance, check Automatic Fallback to restore the previous CUDB software version in case the upgrade fails, and root password for the selected VNF instances, and click Submit.
    Note: When Automatic Fallback option is chosen, if the upgrade fails, the fallback is executed without stopping. If it is not selected and the upgrade fails, the execution is stopped and manual intervention is needed. Contact CUDB Administrator.
  7. On the Collect user data for Orchestrator Node screen, insert the password for the specified VNF instance that is used as an orchestrator from which the upgrade will be started for the selected VNF, and click Submit.

3.4.2.1 Workflow Log in GUI

The workflow log shows the executed procedures and the progress in percentage. The expected progress information output must be similar to the following example:

The number of executed procedures and the total number of procedures in this phase depends on the upgrade path. The duration of each procedure is different. It takes more time for some procedures to be executed, for example NI_PROC_IBU_WAIT_AIT and NI_PROC_RESTORE_DATA.

Do!

If vCUDB base software version is previous to 1.12, after the CUDB VNF, deployed on CEE is upgraded, all VMs must be configured with ha-offline High Availability (HA) policy, as stated in the Other Requirements section of CUDB Virtual Infrastructure Requirements. For more information, see Other Requirements.

4 Troubleshooting

If the workflow execution is unsuccessful, see the following options for more information on the cause of failure:

  • Workflow Log view. Check the progress information output on VNF-LCM GUI web browser.

  • Jboss Server log on VNF-LCM. During execution, log information is saved in this file:

    # tail -f /ericsson/3pp/jboss/standalone/log/server.log

    Contact Ericsson personnel if support is needed.

  • If Instantiate VNF workflow was failing during CUDB installation, connect through either sysmgmt or oam vip.

    ssh <cudb_user>@<vip>

    cd /home/coremw_appdata/incoming/cudb-install-temp/

    ./cudbAutomatedInstallProgressTracker.py -c

    Check the automatedInstall.log file in SC_2_1 for further details.

  • If vCUDB Upgrade Preparation workflow was failing in a procedure, more information about the error can be found in the /var/tmp/upgradeMonitor.log file on VNF-LCM GUI.

For further help, contact a CUDB Administrator as Ericsson support to follow the VNF-LCM Admin Guide or the corresponding manual procedures (for example installation, upgrade and so on) for a deeper troubleshooting.

5 Appendix: General Considerations when Upgrading

During the CUDB upgrade:

  • The software is upgraded to the same software used in a maiden installation. Therefore, the same behavior and performance is expected.

  • All previous configuration is preserved, except some hardening parameters.

5.1 Hardening

Contact the CUDB Administrator to follow the hardening instructions of the manual upgrade for vCUDB.

5.2 Deviations from Standard Configuration

Upgrade is aimed to run on standard CUDB environments. The existence of any deviation, meaning any customization present in the target system, could affect the Upgrade execution, so it is recommended to contact the next level of support in advance to analyze the case. Deviations can include:

  • Software Delivery Packages (SDPs) or Red Hat Package Manager (RPM) software files

  • System configuration: administrators, sudoers, and so on

  • Network configuration: for example, QoS, VLAN IDs and subnets

5.3 Prerequisites

Before running the upgrade workflows, ensure that the following requirements are fulfilled:

  • The upgrade must be performed during a low system traffic period (maintenance window) and scheduled according to the estimated times. Both the upgrade and fallback times must be considered during the planning activities.

  • In case the system memory usage is above 85% in any node, contact Ericsson personnel to evaluate a Data Store (DS) expansion procedure before the upgrade.

  • The system is up and running with no relevant alarms issued.

    Examples of alarms not relevant for upgrade:

    • Logchecker found minor error(s), Preventive Maintenance

    • Root Login Failed, Security

    • Fault retrieving subscriber statistics, Application Counters

    • Deleted data due to reconciliation, Storage Engine

    • Memory usage at Warning level

  • During upgrade execution, mastership movements are expected. To avoid mastership movements in some specific cases, move them manually.

  • For information on the estimated times, consult with Ericsson personnel.

  • Both the upgrade and fallback times must be considered during the planning activities.

  • Enough space is available in /cluster to store the data, the software, the configuration backups and the files related to the upgrade. If not, free up storage space by removing existing backups or not needed information, otherwise an external storage is needed:

    available space at /cluster > SW and Configuration backup(1 GB) + Upgrade-created stuff(1GB) + Data backup (250MB (GEP3), 750MB (GEP5), 25MB (small vCUDB)) * number of ndb blades

5.4 Network Impacts

The CUDB system upgrade can include changes in the interfaces at network level that may require further actions external to the CUDB nodes. Refer to the Interface section of CUDB 1.13 Network Impact Report.

5.5 System Downtime and Service Disturbances

When a node is being disabled in the system, provisioning traffic is stopped while the masters are moved away of the upgrading node. In case there are no masters, provisioning does not stop.

No system downtime is expected during the whole procedure.

Minor traffic loss and dropped connections can be experienced in the following cases:

  • while the CUDB nodes are disabled in the system.

  • during the LDAP servers restart performed in the Node upgrade and Fallback phases needed due to the modification of database tables structure.

5.6 Restrictions

Consider the following restrictions before performing a software update on the CUDB system:

  • No configuration changes are allowed until the system upgrade is completed.

  • Once the system is in mixed state (that is, some nodes are updated, but not all of them), do not use standard CUDB restore commands.

  • Current upgrade/fallback procedure does not support handling of faulty blades such as ignoring or skipping them. If any blade is faulty it must be immediately replaced before continuing with upgrade/fallback.

  • No system task scheduling changes are allowed. Therefore, avoid the execution of any command modifying the crontab, for example: cudbConsistencyMgr.

  • The upgrade phases must not be executed in different nodes at the same time.

  • The cudbCheckReplication command can dump an internal error in the response printout when checking replication in slave DS units that might lead to erroneous interpretation of the command or in the status of the replication:

    awk: fatal: can't open source file `/opt/ericsson/cudb/OAM/bin/cudbSystemStatus.awk' for reading (No such file or directory)
[info] Node20-DSG0 replication: OK

    So, the use of the cudbCheckReplication command is restricted while the system has a mixed status. This restriction only applies if the base release is CUDB 1.2 or a previous version.

  • Multiple LDAP request is not allowed until the system is fully upgraded.

Reference List

CUDB Documents

  1. CUDB Glossary of Terms and Acronyms 0033-HDA 104 03/10
  2. CUDB Virtual Infrastructure Requirements 15/1553-HDA 104 03/10

ENM CPI Library References

  1. VNF-LCM Installation Instructions 1/1531-CNA 403 3313
  2. ENM Configuration System Administration Guide 1/1543-AOM 901 151-1

OSS-RC CPI Library References

  1. VNF-LCM CEE/Openstack Installation Instructions 1/153 72-APR 901 0578
  2. VNF-Lifecycle Manager System Administration Guide 1543-APR 901 0578
Legal |© Ericsson AB 2016–2018