Troubleshooting 1/154 51-AXM10103/1 Uen B

VNF Life Cycle Management Troubleshooting Guideline
Virtual Border Gateway Function

Contents


1 Introduction

This document describes the procedures to follow in case the VNF cannot be instantiated or terminated properly using VNF-LCM.

The workflows can be managed and monitored using the following interfaces:

  • Full Stack Scenario

    • The VNF-LCM GUI when using the ENM (EM-triggered mode)

    • The EO GUI when using the EO (NFVO-triggered mode)

  • Small Stack Scenario

The following applies for life cycle operations for EO through the Or-Vnfm interface:

  • EO is the only supported NFVO.

  • The only supported workflows are instantiation and termination.

  • The supported VIM is OpenStack.

1.1 Conditions

The following conditions apply:

  • VNF-LCM is available using either Operations Support System, Radio and Core (OSS-RC) or Ericsson Network Manager (ENM).

  • The following minimum version requirements are met:

    • Virtual Infrastructure Managers (VIMs):

      • CEE: R6 or newer

      • OpenStack: Mitaka or newer

    • vIMS workflows: 1.15.0

    • VNF-LCM: 18.18 or higher (minimum SW media version: 4.10.14)

    • EO: 18.0

  • This document is applicable to Ericsson Orchestrator (EO) version 18.1.

2 Troubleshooting Procedure

2.1 Solve Instantiation Trouble Cases

Follow these instructions if the VNF fails to instantiate in any phase of the instantiation workflow (including before executing the workflow).

  1. Identify, in which phase has the instantiation workflow failed.

    Workflow Phase

    Procedure to Follow

    Before executing the workflow

    No manual cleanup actions are required.

    Workflow initialization

    No manual cleanup actions are required.

    Instantiation granting during orchestration
    Clean Up VAPP in ENM(1)

    Resource allocation in VNF-LCM

    Terminate the VNF forcefully. If forceful termination fails, continue with the manual Cleanup Instructions.

    Remove every VAPP part from the system. If there are no VAPP parts left in a component, check and continue with the remaining.

    After instantiation, during VAPP acceptance in ENM

    After instantiation, when sending instantiation completed message
    (1) Ericsson Orchestrator uses the term "VAPP" to denote a logical group composed of one or more VMs. VMware based deployments use the term "vApp" to denote a preconfigured cluster of VMs packaging applications with their required operating systems.
     For more information, see Glossary of Terms and Acronyms.

2.2 Solve Termination Trouble Cases

Follow these instructions if the VNF fails to terminate in any phase of the termination workflow (including before executing the workflow).

  1. Identify, in which phase has the instantiation workflow failed.

    Workflow Phase

    Procedure to Follow

    Before executing the workflow

    No manual cleanup actions are required.

    Workflow initialization

    No manual cleanup actions are required.

    Termination granting during orchestration
    Clean Up VAPP in ENM(1)

    VNF instance locking

    In case the VNF-LCM GUI keeps repeating the message SSH Connection Failed (AuthenticationException), the workflow cannot log into the VNF. To solve the issue, continue with Solve jboss_user Authentication Issue.

    In all other cases, terminate the VNF forcefully. If forceful termination fails, continue with the manual Cleanup Instructions.

    Stack deletion

    Terminate the VNF forcefully. If forceful termination fails, continue with the manual Cleanup Instructions.

    Remove every VAPP part from the system. If there are no VAPP parts left in a component, check and continue with the remaining.

    After termination when removing VNF entry from ENM
    (1) Ericsson Orchestrator uses the term "VAPP" to denote a logical group composed of one or more VMs. VMware based deployments use the term "vApp" to denote a preconfigured cluster of VMs packaging applications with their required operating systems.
     For more information, see Glossary of Terms and Acronyms.

2.3 Solve SSH Connection related Trouble Cases

  1. Identify, in which phase has the instantiation workflow failed.

    Workflow Phase

    Procedure to Follow

    VNF instance locking

    In case the VNF-LCM GUI keeps repeating the message SSH Connection Failed (AuthenticationException), the workflow cannot log into the VNF. To solve the issue, continue with Solve jboss_user Authentication Issue.

    In all other cases, terminate the VNF forcefully. If forceful termination fails, continue with the manual Cleanup Instructions.

3 Cleanup Instructions

This section contains procedures for VAPP cleanup if it is in an inconsistent state between EO, ENM, VNF-LCM, and the VIM after running the forceful termination workflow.

3.1 Clean Up VAPP in EO

Prerequisites

  • You are logged in to EO.

Steps

  1. Get the VNF instance ID of the VAPP.

    • If using the EO GUI, select External Management System ID.

    • If using REST API of EO, use the following request:

      GET /vapps/<vapp_ID>

      For more information on the output, see Example 1.

  2. Get the list of resource names that are using the VNF instance ID.

    • vnfcResourceInfo

    • vnfVirtualLinkResourceInfo

    • virtualStorageResourceInfo

    Example

    GET /vnflcm/vs/vnf_instances/<vnf_instance_ID>

    For more information on the output, see Example 2.

  3. Get the list of resource IDs that are using the resource names collected in Step 2.
    1. Dump the resources from EO.
    2. Search for the resource names collected in Step 2 in the following resources:

      • compute

        Use the following request: GET <ecm_service_path>/vms

        For more information on the output, see Example 3.

      • networking

        Use the following request: GET <ecm_service_path>/v2/vns

        For more information on the output, see Example 4.

      • storage

        Use the following request: GET <ecm_service_path>/bsvs

        For more information on the output, see Example 5.

  4. Remove the resources identified by their IDs from the EO database. Repeat this step for each compute, networking, and storage resources.

    Example

    DELETE /cmdb/vms/<VM_ID>

    DELETE /cmdb/vns/<VM_ID>

    DELETE /cmdb/bsvs/<VM_ID>

    DELETE /cmdb/vapps/<vapp_ID>

3.2 Clean Up VAPP in VNF-LCM

  1. Disable NFVO.
  2. Run the Terminate VNF workflow with termination type forceful.

If disabling the NFVO or running the Terminate VNF workflow fails:

  1. Remove the heat stack and the possibly stuck resource from the VIM.
  2. When there are no resources left in the VIM, run the Terminate VNF workflow with termination type forceful.
  3. Enable NFVO.

3.3 Clean Up VAPP in ENM

  1. Remove the VAPP from ENM topology manually, using the following commands:

    Example

    cmedit set NetworkElement=<VAPP_name>,CmNodeHeartbeatSupervision=1 active=false
cmedit set NetworkElement=<VAPP_name>,InventorySupervision=1 active=false
cmedit set NetworkElement=<VAPP_name>,PmFunction=1 pmEnabled=false
alarm disable <vAPP_name>
cmedit action NetworkElement=<VAPP_name>,CmFunction=1 deleteNrmDataFromEnm
cmedit delete NetworkElement=<VAPP_name> -ALL --force
    Note: Save the commands in a text file, drag and drop it in ENM CLI, and run them using the following command: batch execute file:<Commands_in_text_file_name>.txt.

4 Remedies

4.1 Solve jboss_user Authentication Issue

Complete the following steps to add the public key of the jboss_user to the authorized keys.

Prerequisites

  • The public key of the jboss_user is available.

Steps

  1. Log in to the VM manually:

    ssh <user_ID>@<O&M_IP_address>

  2. Obtain the public key using the following command:

    sudo -u jboss_user cat /vnflcm-ext/backups/workflows/private_keys/default/.ssh/id_rsa.pub

  3. Append the public key to the list of authorized keys:

    cat .ssh/<jboss_user_public_key_file_name> >> .ssh/<authorized_keys_file_name>

  4. If the SSH Connection Failed (AuthenticationException) error message is still repeating, terminate the VNF and instantiate it using the correct public key.

5 EO REST API Examples

Example 1   Get the VNF Instance ID

Example 2   List the Resource Names Using the VNF Instance ID

Example 3   Get the Compute Resource ID

Example 4   Get the Network Resource ID

Example 5   Get the Storage Resource ID

Legal |© Ericsson AB 2019