MTAS VNF Life Cycle Management Guide

MTAS

1	Introduction
2	Prerequisites
2.1	Hardware and Software
3	Onboarding
4	Procedures
4.1	Instantiate VNF
4.2	Scale-out VNF
4.3	Scale-In VNF
4.4	Heal VNF
4.5	Terminate VNF
4.6	Upgrade VNF
5	Logging

1 Introduction

This document describes system administration tasks performed in the VNF life cycle Manager (VNF-LCM). The VNF-LCM provides a workflow execution environment and a web-based application for managing VNF life cycle procedures.

In this document, the term "vMTAS" refers to the product and the term "MTAS" refers to the MTAS application, independent of being deployed in a native or virtual environment.

This document covers the following workflow-based life cycle management procedures:

Instantiate VNF
Scale-out VNF
Scale-in VNF
Heal VNF
Terminate VNF
Upgrade VNF
Note:
The Upgrade VNF Work Flow (WF) is not available in the current release.

2 Prerequisites

This section describes the prerequisites which must be fulfilled before the MTAS can be installed.

2.1 Hardware and Software

The following hardware (virtual and physical), and software is required:

The software delivery package (so called MTAS Workflow pack) including vIMS workflow bundle, and the MTAS LCM scripts is available.
VNF-LCM is available using either Operations Support System for Radio and Core (OSS-RC) or Ericsson Network Manager (ENM).
Openstack Mitaka (or newer) release-based cloud environment is used. For example: Ericsson Cloud Execution Environment (CEE) R6 or newer.
The Tenant(s) in which the VNF instance(s) resides and the connected Virtualized Infrastructure Manager(s) (VIM) are configured in VNF-LCM.
The VIM configuration in VNF-LCM can be checked with the vnflcm vim list command.

For more information on VIM/Tenant configuration, refer to ENM/OSS-RC documentation.
OSS-RC/ENM related parameters are configured properly on VNF-LCM.
For more information, refer to ENM/OSS-RC documentation.
The version of the used VNF-LCM is 18.03 (- 4.1.16), or higher.

3 Onboarding

This section describes how to prepare for workflow-based VNF operations using VNF-LCM. Performing this procedure is a prerequisite for life cycle operations.

Note:: The following commands are to be executed on the VNF-LCM's Services VM.

Decompress the vMTAS workflow pack CXP9034815_1-<R-state>.tar.gz where <R-state> defines the revision state of the package.
cd /home/cloud-user

tar -zxvf CXP9034815_1-<R-state>.tar.gz
Install the vIMS workflow bundle:
1. Log on to vnflaf-services VM as cloud-user and switch to root user.
2. Verify that the bundle is not already installed by running the List command.
  wfmgr bundle list
  
  If the version of vIMSWorkflows is older than the version included in the vMTAS Workflow Pack CXP9034815_1-<R-state>.tar.gz, install the newer version.
3. Install the workflow bundle package by running the install command.
  wfmgr bundle install --package=\ <workflow_bundle_rpm_file _path>
For more information on workflow bundle installation in VNF-LCM, refer to ENM/OSS-RC documentation.
Decompress vMTAS LCM scripts CXP9034788_1-<R-state>.tar.gz where <R-state> defines the revision state of the package:
mkdir /vnflcm-ext/backups/workflows/vnfd/\ <VNFType__VNFVersion>

cd /vnflcm-ext/backups/workflows/vnfd/\ <VNFType__VNFVersion>

tar -zxvf /home/cloud-user/CXP9034788_1-<R-state>.tar.gz

Note:
Follow the naming convention: VNF type and VNF version separated by __. (Where "__" is a double underscore.)
In /vnflcm-ext/backups/workflows/vnfd/<VNFType__VNFVersion>, create a configurations subdirectory with write permission for the jboss_user and a child directory for each VNF configuration (that is, VNF instance), and copy the VNF configuration to the child directory:
mkdir /vnflcm-ext/backups/workflows/vnfd/\ <VNFType__VNFVersion>/configurations

mkdir /vnflcm-ext/backups/workflows/vnfd/\ <VNFType__VNFVersion>/configurations/<VNF_1 configuration>

cd /vnflcm-ext/backups/workflows/vnfd/\ <VNFType__VNFVersion>/configurations/<VNF_1 configuration>

cp -r /home/cloud-user/<VNF_1 configuration> .

Note:
Each directory in configurations contains a VNF instance-specific env.yaml environment (HOT) file, configuration files evip_cli.txt for eVIP, ss7.conf for SS7, and pdb_bundle.zip for application configuration. Each directory also contains id_rsa.pub public key of the VNF-LCM's jboss_user. The files evip_cli.txt, ss7.conf, and pdb_bundle.zip are optional.
Upload the HOT template and the scaling template into the /vnflcm-ext/backups/workflows/vnfd/<VNFType__VNFVersion> directory.
cp -r /home/cloud-user/main.yaml /vnflcm-ext/backups/\ workflows/vnfd/<VNFType__VNFVersion>

cp -r /home/cloud-user/MTAS_HOT_scaling.yaml /vnflcm-ext/\ backups/workflows/vnfd/<VNFType__VNFVersion>
If the SSH key is not available yet for jboss, create it using the following commands:
su jboss_user

ssh-keygen -t rsa

exit

Note:
The use of encrypted private keys is not supported in the current release (that is, do not use passphrase).
Copy the public SSH key into the configuration directory of VNF instance:
cp /home/jboss_user/.ssh/id_rsa.pub /vnflcm-ext/backups/workflows/vnfd/<VNFType__VNFVersion>\ /configurations/<VNF_1 configuration>/

Note:
The public key must be added in the configuration directory for each instance.

Verify that the structure of the /vnflcm-ext/backups/workflows/vnfd/<VNFType__VNFVersion> directory is as follows:

`-- vMTAS__<x.y>
     |-- vnfLcmOperationsConfiguration.json
     |-- configurations
     |   |-- <VNF_1 configuration>
     |       |-- env.yaml
     |       |-- id_rsa.pub
     |       |-- evip_cli.txt*
     |       |-- ss7.conf*
     |       `-- instance1_pdb_bundle.zip*
     |    `-- <VNF_2 configuration>
     |       |-- env.yaml
     |       |-- id_rsa.pub
     |       |-- evip_cli.txt*
     |       |-- ss7.conf*
     |       `-- instance1_pdb_bundle.zip*
     |-- main.yaml
     |-- MTAS_HOT_scaling.yaml
     `-- lcmScripts

The files marked with * are optional.

In case automatic invocation of Heal VNF WF (so called auto-heal WF) shall be enabled:

Note:: For CEE deployments, this step is not necessarily needed since CEE's CM-HA feature can be primary used to Heal a VNF.

Verify the presence of file vIMS-autostart-rules.xml in services VM of VNF-LCM:
ll /vnflcm-ext/current/workflows/auto-start-rules/vIMS-autostart-rules.xml
If the file is not present, copy the template autostart-rule file to the above mentioned directory and remove all existing template workflowTriggerRule:
sed '24,45d' /etc/opt/ericsson/ERICvnflafautostartservice_CXP9032572/vIMS-autostart-rules.xml \

> /vnflcm-ext/current/workflows/auto-start-rules/vIMS-autostart-rules.xml

Note:
It can be needed to verify that the first template workflowTriggerRule starts in line 24 and the last workflowTriggerRule ends in line 45.

The file /vnflcm-ext/current/workflows/auto-start-rules/vIMS-autostart-rules.xml must contain the following workflowTriggerRule:

<wft:workflowTriggerRule name="vMTAS-AutoHealing" id="vMTAS-AutoHealing" type="FM_ALARM">
                <wft:triggerCondition>
                        <wft:attributeName>specificProblem</wft:attributeName>
                        <wft:triggerConditionOperator>EQUALS</wft:triggerConditionOperator>
                        <wft:attributeValue>COM SA, CLM Cluster Node Unavailable</wft:attributeValue>
                </wft:triggerCondition>
                <wft:triggerAction WorkflowDefinitionName="Heal VNF"
                        WorkflowBundleName="vIMSWorkflows">
                        <wft:param />
                </wft:triggerAction>
        </wft:workflowTriggerRule>

Verify that the content of /vnflcm-ext/current/workflows/auto-start-rules/vIMS-autostart-rules.xml is similar to the following example: (it can contain additional workflowTriggerRule):

Note:: If the user wants to enable auto-healing only for a particular VNF instance, the workflowTriggerRule can be refined by adding additional triggerCondition(s) to it.

Update the VNF-LCM's database with VNF instance-specific information for VNF instances instantiated with vIMSWorkflows version lower than 1.4.0. If it is acceptable, it is recommended to re-Instantiate those VNF instances instead of manually updating the database.
For more information on adding existing VNF instance to VNF-LCM's database, refer to ENM/OSS-RC documentation.

4 Procedures

The following sections describe how to perform LCM operations.

VNF-LCM procedures use workflow instances. Figure 1 shows the example of a workflow instance, where workflow progress can be tracked in the Workflow Diagram view. The Workflow Diagram only represents stages of the various procedures, operations are performed in the Task view.

Figure 1 Workflow Instance Overview

4.1 Instantiate VNF

This section describes how to instantiate a VNF using VNF-LCM.

Note:: In case the HEAT stack has to be updated manually, make sure that the --tags argument is used during the stack update, otherwise the VNF does not show up in the Workflow Instance screen.
Use the following command on a stack created using the VNF-LCM to find out what value to set the --tags argument to:
heat --os-tenant-name <tenant name> stack-show <MTAS stack name that is name of VNF> |grep -A 2 tags

To instantiate a VNF:

Navigate to the VNF-LCM Workflows > Instantiate VNF > Start a New Instance. See Figure 2.

Figure 2 Workflows

In Start a Workflow, fill out the Instance Name, and click Submit.
Select the newly created workflow from the Instance Activity panel.
On the Workflow Instance, add VNF Name ,VNF Instance Description, and Select VNF descriptor id*, and click Submit. See example in Figure 3.
Note:
The Select VNF descriptor id* displays VNF configurations available for instantiation in the /vnflcm-ext/backups/workflows/vnfd/ directory.

Figure 3 Instantiate VNF

In Select VIM, choose Select VIM to be used, and click Submit. See example in Figure 4.

Figure 4 Select VIM

In Select Tenant, choose Select Tenant to be used, and click Submit. See example in Figure 5.

Figure 5 Select Tenant

In Zone and Region Details, fill in cloud region id and cloud zone id, and click Submit. See Figure 6.
Note:
This step is optional (parameters can be left empty).

Figure 6 Zone and Region Details

In Get Instance Configuration, choose Select Configuration for the VNF instance*, and click Submit. See example in Figure 7.
Note:
The Select Configuration for the VNF instance* displays VNF configurations available for instantiation in the /vnflcmext/backups/workflows/vnfd/<VNFType__VNFVersion>/configurations directory.

Figure 7 Get Instance Configuration

If the ENM/OSS-RC network management application is used, provide VNF instance-specific parameters for ENM/OSS-RC topology update, and click Submit. See example in Figure 8.
Note:
To fill out the Network element version supported by OSS-RC/ENM *, check the supported VNF version on OSS-RC/ENM. For example: on ENM, the supported VNF version can be checked with the following command: cmedit describe --netype <VNF_type>

Figure 8 Enter the Parameters Required by OSS/ENM

Note:: In case of failed installation, remove the VNF instance by means of Terminate VNF WF (FORCEFUL). If the user cannot select the VNF instance in Step 4 in Section 4.5, the VNF instance-specific Heat stack shall be manually deleted from OpenStack.

4.2 Scale-out VNF

This section describes how to scale-out a VNF using VNF-LCM.

Continue with this procedure only if the VNF to be scaled-out is instantiated using the VNF-LCM.

To scale-out a VNF:

In the VNF-LCM, click Start a Workflow > Scale-Out VNF > Start a New Instance. See Figure 2.

In Start a Workflow, fill out Instance Name, and click Submit.
Select the newly created workflow from the Instance Activity.
In Workflow Instance, choose Select VNF instance*, specify the number of VMs to be added to the VNF, and click Submit. See example in Figure 9.

Figure 9 Collect User Data for Scale-Out

The VNF instance is scaled-out, new VMs are added to the cluster.

4.3 Scale-In VNF

This section describes how to scale-in a VNF using VNF-LCM.

Continue with this procedure only if the VNF to be scaled-in was instantiated using the VNF-LCM.

Prerequisites

To gracefully scale-in a node, the system status must meet the following conditions:

The system has sufficient memory. If the system has insufficient memory, the DBS, Memory Limit Reached alarm appears.
One of the following CM parameters is in TRUE state:

ManagedElement=1,MtasFunction=MtasFunction,MtasSupportFunctions=0,
CarSelApplication=CarrierSelect,CarSelDialedStringAnalysisTable=0,
carSelDialedStringAnalysisTableSynchronization 
ManagedElement=1,MtasFunction=MtasFunction,MtasSupportFunctions=0,
CarSelApplication=CarrierSelect,CarSelCarrierTable=0,
carSelCarrierTableSynchronization 
ManagedElement=1,MtasFunction=MtasFunction,MtasSupportFunctions=0,
NumAnaApplication=NumberAnalysis,NumAnaLocalCallTable=0,
numAnaLocalCallTableSynchronization 
ManagedElement=1,MtasFunction=MtasFunction,MtasSupportFunctions=0,
NumberNormalisation=NumberNormalisation,
numberNormalisationTableSync

For more information, refer to MTAS Troubleshooting Guideline.

To scale-in VNF:

In the VNF-LCM, click Start a Workflow > Scale-In VNF >Start a New Instance. See Figure 2.

In Start a Workflow, fill out Instance Name, and click Submit.
Select the newly created workflow from the Instance Activity panel.
In Workflow Instance, choose Select VNF Instance* to be scaled in, specify Number of VMs to Scale-In*, and click Submit. See example in Figure 10.

Figure 10 Collect User Data for Scale-In

The VNF instance is scaled-in, the specified number of VMs is removed from the cluster.

On the Collect extra parameters screen, select Scale-in type and optionally specify the list of VM UUIDs to be Scale-in with, and click Submit. See example in Figure 11.
Note:
FORCEFUL Scale-in is a disruptive action therefore it is only to be used for Scale-in unavailable/failed PL(s). (The virtual resources reserved for the PL(s) are freed up first and then the MTAS cluster is "cleaned".)

Figure 11 Collect Extra Parameters

The VNF instance is scaled-in, the specified number of PLs is removed from the cluster.

Note:: If any UUID was specified in Step 5, the VMs with the specified UUIDs are removed, otherwise VMs with the highest index in their name in OpenStack are removed.

4.4 Heal VNF

This section describes how to heal a VNF from compute resource or network resource (for instance Neutron port) failure using VNF-LCM.

Heal VNF WF can be started two ways:

Manually from VNF-LCM User Interface (UI)
Automatically, triggered on the reception of the CLM Cluster Node Unavailable alarm from the VNF instance

To enable the use of auto-heal VNF WF, set up an autostart-rule as it is specified in Step 9 of Onboarding.

The progress of auto-heal VNF WF can be tracked on the Instance Activity panel of VNF-LCM. The Heal VNF WF consists a FORCEFUL Scale-in and a Scale-out operation so that three WF instances are visible for the auto-heal VNF WF (Heal VNF, Scale-in (Heal VNF) and Scale-out (Heal VNF)) on the Instance Activity panel.

Note:

It is recommended to lower the node (VM) alarm timeout on the VNF instance from 15 minutes to 5 minutes in order to trigger the CLM Cluster Node Unavailable alarm, if a VM has lost contact with the remaining cluster members for more than 5 minutes. To lower the value of the node (VM) alarm timeout, use command: cmw-node-alarm-timeout 300
The Heal VNF WF can only heal the VNF if sufficient compute resource is available for OpenStack's Nova scheduler.
Healing of non-scalable VMs (SC-1, SC-2, PL-3, PL-4) is not possible.

Continue with this procedure only if the VNF to be healed was instantiated using the VNF-LCM and the Heal VNF WF shall be started manually.

To heal the VNF manually:

In the VNF-LCM Workflows, select Heal VNF, and click Start a New Instance. See Figure 2.

In Start a Workflow, fill out Instance Name, and click Submit.
Select the newly created workflow from the Instance Activity panel.
In Workflow Instance , choose Select VNF instance*, and click Submit. See example in Figure 12.

Figure 12 Collect User Data for Healing

In Input additional parameters for workflow, specify UUID of VM to be removed from the cluster, and click Submit. See example in Figure 13.

Figure 13 Collect Extra Parameters

Get UUID of unavailable/failed PL(VM) from ECLI using command:
```
show -r ManagedElement=1,Equipment=1
```

The VNF instance is scaled-in, the specified VM is forcefully removed from the cluster. After this, the VNF instance is scaled-out, and a PL is added to the cluster.

4.5 Terminate VNF

This section describes how to terminate a VNF using VNF-LCM.

Continue with this procedure only if the VNF to be terminated is instantiated using the VNF-LCM.

Steps

In the VNF-LCM Workflows, select Terminate VNF, and click Start a New Instance. See Figure 2.

In Start a Workflow, fill out the Instance Name field, and click Submit.
Select the newly created workflow from the Instance Activity panel.
On the Workflow Instance screen, choose Select VNF instance and click Submit. See example in Figure 14.

Figure 14 Collect User Data for Terminate

Graceful	The VNF instance is gracefully locked (by setting `mtasFunctionAdministrativeState` to `SHUTTING DOWN(2)`), it gradually stops processing traffic. The VNF is terminated after the expiration of the graceful termination period or even earlier when all ongoing sessions have stopped on the node.

Forceful	The VNF is terminated immediately, all ongoing traffic is lost. This option must be confirmed on the next screen, as it stops all traffic.

Graceful termination timeout (sec)	The graceful termination timeout value defines after how many seconds the VNF is terminated when graceful termination has been applied but there is still ongoing traffic. Default value: -1, meaning that there is no graceful termination period, that is, the VNF is terminated only after the VNF stopped processing traffic.

The VNF instance is terminated. In case of Graceful termination, the VNF instance stops processing traffic gracefully and then is terminated.

4.6 Upgrade VNF

The Upgrade VNF WF is not supported by the current release.

5 Logging

In case of an unsuccessful workflow execution, find more information on the cause of the failure:

In the Workflow Log view.
In the Jboss Server log:
/ericsson/3pp/jboss/standalone/log/server.log

It is recommended to increase the log level from INFO to DEBUG during troubleshooting of the unsuccessful Workflow execution. More information on the procedure can be found in VNF-LCM documentation.