Upgrade Information 153 72-AXB 901 33/7 Uen A1

SAPC Upgrade Instruction
Ericsson Service-Aware Policy Controller

Contents


1 SAPC Upgrade Introduction

The purpose of this document is to provide detailed information on how to upgrade the SAPC to SAPC 1.X delivery ( CXP 903 0138/7 <to_release_version>).

2 SAPC Upgrade Prerequisites

The following conditions must be fulfilled to upgrade the SAPC.

2.1 Upgrade General Requirements

2.1.1 HW Requirements

  • External Machine.

  • At least 5 GB of free space in mounted hard disk at/cluster.

2.1.2 SW Requirements

  • No alarms must be present.

  • No O&M operations must be executed during the entire duration of the upgrade.

  • The upgrade must be performed during low traffic hours.

  • Disable any scheduled backup.

  • Bash interpreter 4.2 or higher is required in External Machine.

  • Enable root user to log on to the SAPC through SSH.

2.2 Upgrade Deliverables

The required software can be downloaded from Ericsson Software Gateway. The deliverables required for the upgrade are <SAPC_UPGRADE_PACKAGE>.zip and <HOST_SLES_PATCHES>.tar.gz.

Note: The package <HOST_SLES_PATCHES>.tar.gz is required only in PNF deployment.

2.3 User Data Consolidation (UDC)

For impacts related to UDC solution, refer to Upgrade chapter of SAPC 1 Network Impact Report.

3 Software Upgrade for Stand Alone

3.1 Pre-Upgrade Installation

Follow these procedures before applying the upgrade:

3.1.1 System Data Backup Creation

Before any SW upgrade, create a System Data backup. For details explaining how to create a backup refer to Backup and Restore.

3.1.2 Obtain SAPC Upgrade Package

Follow these steps to obtain the upgrade package which is necessary to execute the upgrade:

Steps

  1. Download the SAPC upgrade package from the Software Gateway to a directory <SAPC_UPGRADE_PACKAGE_PATH> on a machine (External_Machine) which has access to the <VIP_OAM> of the SAPC.
  2. Extract <SAPC_UPGRADE_PACKAGE>.zip to obtain the file <SAPC_UPGRADE_PACKAGE>.tar.gz executing this command:
    <External_Machine>:$ unzip <SAPC_UPGRADE_PACKAGE_PATH>/<SAPC_UPGRADE_PACKAGE>.zip
  3. Create a directory on the <VIP_OAM> to store SAPC upgrade package with following command:
    <External_Machine>:$ ssh root@<VIP_OAM> mkdir /cluster/storage/no-backup/<SAPC_UPGRADE_PACKAGE>
  4. Copy the SAPC upgrade package downloaded from the software gateway to the <VIP_OAM>, with the following command:
    <External_Machine>:$ scp <SAPC_UPGRADE_PACKAGE>.tar.gz root@<VIP_OAM>:/cluster/storage/no-backup/<SAPC_UPGRADE_PACKAGE>
  5. Log in as "root" userId to the<VIP_OAM>, with the following command:
    <External_Machine>:$ ssh root@<VIP_OAM>
  6. Extract the <SAPC_UPGRADE_PACKAGE>.tar.gz obtained in the step before executing this command:
    <SC-x>:# tar xvfz /cluster/storage/no-backup/<SAPC_UPGRADE_PACKAGE>/<SAPC_UPGRADE_PACKAGE>.tar.gz
    -C /cluster/storage/no-backup/<SAPC_UPGRADE_PACKAGE>

3.1.3 Upgrade BIOS Version in NSP 6.1 Hardware

For the SAPC PNF upgrade, if the hardware used is NSP 6.1 with GEP3 blades, the validated and mandatory GEP3 firmware version to ensure a successful upgrade is R11A or later.

The GEP3 firmware upgrade to the validated R11A version or later is mandatory for the blades hosting SCs and PLs. This procedure is complex and needs special knowledge, therefore it is recommended to be performed by Ericsson personnel. The upgrade cannot be performed in a blade in service, as it implies powering off the blades.

To perform the upgrade, follow the steps in the BIOS Upgrade Instruction document. If there is a problem, contact GEP support.

3.1.4 Disable Session Inactivity Cleanup Mechanism

Disable the Session Inactivity Cleanup mechanism during the upgrade procedure. To accomplish this task, follow the next steps:

Steps

  1. Look up if the Session Inactivity Cleanup mechanism is enabled. This procedure is detailed in Showing the current configuration section from the Configure Session Inactivity Cleanup Mechanism document.
  2. If the Session Inactivity Cleanup mechanism is enabled, deactivate it. This procedure is detailed in Enabling/disabling the mechanism subchapter from the Configure Session Inactivity Cleanup Mechanism document.

3.1.5 Memory dimensioning in PLs

The minimum memory for the PLs in VNF deployments of SAPC is 10GB. If the PLs have less memory, it must be increased. The detailed procedure can be found on the Memory dimensioning in PLs in a Live SAPC document.

3.1.6 SAPC State Verification

Steps

  1. Log in as "root" userId to the<VIP_OAM>, with the following command:
    <External_Machine>:$ ssh root@<VIP_OAM>
  2. Execute sapcHealthCheck:
    <SC-x>:# sapcHealthCheck
    .
    .
    .
    *** SAPC HEALTH CHECK SUMMARY ***
    WARNINGS: 0
    CRITICAL ERRORS: 0
    **********************************
    SAPC Health Check finished: OK

3.2 Host SLES12 Upgrade to SP2

For SAPC VNF deployment, skip this section, and continue with Upgrade for Stand Alone.

For SAPC PNF deployment, the procedure consists of two steps:

Caution!

All these steps must be done during the maintenance window.During the upgrade process, the blades of the SAPC cluster are rebooted.

3.2.1 Upgrade Reinstalling Blades with SLES12 SP2

Steps

  1. Follow the steps to prepare the Installation Server.
    Prepare the Installation Server following the steps described in SAPC PNF Deployment Instruction:
  2. Connect to VIP_OAM and lock the SC that is going to be upgraded to SP2.
    External_Machine:$ ssh root@<VIP_OAM>
    SC-x:# cmw-node-lock SC-<x>
    SC-x:# cmw-status -v node | grep SC-<x> -A2
  3. Power off the VM of the blade hosting the previously locked SC.
    External_Machine:$ ssh root@<Host_x>
    Host_x:# /mnt/store/SAPC/host-config/scripts/management/sapc_vm-manager_cxp9030138.sh -c stop -s <x>
    Where <x> corresponds to the SC number, for example for SC-2, the <x> is 2.
  4. Create and export a tmp_sparsify directory.
    Host_x:# mkdir /mnt/images/tmp_sparsify
    Host_x:# export TMPDIR=/mnt/images/tmp_sparsify
  5. Create a backup of the VM image and needed files of the SC.
    External_Machine:$ mkdir -p /home/SAPCInstallation/Backup_<Host_x>
    External_Machine:$ ssh root@<Host_x>
    Host_x:# virt-sparsify --check-tmpdir continue /mnt/images/sapc_sc-<x>_cxp9030138.qcow2 /mnt/images/sapc_sc-<x>_cxp9030138.qcow2.SHRUNK
    Host_x:# scp sapc_sc-<x>_cxp9030138.qcow2.SHRUNK <External_Machine>:/home/SAPCInstallation/Backup_<Host_x>/sapc_sc-<x>_cxp9030138.qcow2
    Host_x:# scp -r /mnt/store/SAPC/host-config <External_Machine>:/home/SAPCInstallation/Backup_<Host_x>/
    Host_x:# scp /mnt/images/reboot.img /mnt/images/adapt_cluster.iso <External_Machine>:/home/SAPCInstallation/Backup_<Host_x>/
  6. Connect to the SC that is not going to be upgraded and stop dhcpd service.
    External_Machine:$ ssh root@<VIP_OAM>
    SC-x:# ssh SC-<y>
    SC-y:# systemctl stop dhcpd.service
  7. Install SLES12 SP2 and updates in Host_x.
    Follow the instructions described in SAPC PNF Deployment Instruction only for the Host_x that is going to be reinstalled:
  8. Copy the host-config directory, the reboot.img, adapt_cluster.iso files and the previously backed up VM image of the SC.
    External_Machine# scp -r /home/SAPCInstallation/Backup_<Host_x>/host-config <Host_x>:/mnt/store/SAPC/
    External_Machine# scp /home/SAPCInstallation/Backup_<Host_x>/reboot.img /home/SAPCInstallation/Backup_<Host_x>/adapt_cluster.iso <Host_x>:/mnt/images
    External_Machine# scp /home/SAPCInstallation/Backup_<Host_x>/sapc_sc-<x>_cxp9030138.qcow2 root@<Host_x>:/mnt/images
  9. (Only for NSP 6.1) Add this line to ssh root config.
    External_Machine:$ ssh root@<Host_x>
    Host_x:# vi /root/.ssh/config
    Add this line:
    KexAlgorithms +diffie-hellman-group1-sha1
  10. Connect to Host_x. Define and start the VM of the SC.
    External_Machine:$ ssh root@<Host_x>
    Host_x:# cd /mnt/images
    Host_x:# qemu-img resize sapc_sc-<x>_cxp9030138.qcow2 100G
    Host_x:# virsh define /mnt/store/SAPC/host-config/VM/vms/sc<x>.xml
    Host_x:# virsh start SC-<x>.<Host_x> --console
    Wait until the VM starts.
    Press Ctrl+5 to exit the console.
  11. Check disk synchronization status.
    External_Machine:$ ssh root@<VIP_OAM>
    SC-x:# drbdsetup status
  12. Start dhcpd service that was stopped previously in the other SC.
    SC-x:# ssh SC-<y>
    SC-y:# systemctl start dhcpd.service
  13. Unlock the SC in the cluster.
    SC-y:# cmw-node-unlock SC-<x>
    SC-y:# cmw-status -v node | grep SC-<x> -A2
  14. Perform a SAPC HealthCheck.
    SC-y:# sapcHealthCheck
  15. Repeat steps from Step 2 to Step 14 in the other blade containing the SC.

3.3 Upgrade for Stand Alone

Caution!

All these steps must be done during the maintenance window.During the upgrade process, the blades of the SAPC cluster are rebooted.

For further information about upgrade timing, refer to SAPC Upgrade Information.

Steps

  1. Extract the SAPC upgrade package on the External_Machine which has access to the <VIP_OAM> executing the command:
    <External_Machine>:$ tar xvfz <SAPC_UPGRADE_PACKAGE_PATH>/<SAPC_UPGRADE_PACKAGE>.tar.gz
    -C <SAPC_UPGRADE_PACKAGE_PATH>
  2. Execute following script to apply the upgrade:
    Warning!

    Upgrade procedure is executed in the SAPC with root user.

    <External_Machine>:$ <SAPC_UPGRADE_PACKAGE_PATH>/upgrade/run.sh root <VIP_OAM>
    Note: This command requests to type the root password in the SAPC
    Warning!

    During the upgrade procedure execution, Pm jobs are stopped. For that reason, during upgrade procedure, measurements are not increased and their related threshold alarms are not raised. Pm jobs are automatically started once the upgrade process is completed and measurements and their related alarms recover the normal behavior.Avoid the execution of cluster reboot of the system before post-upgrade installation steps are done and a backup of the SAPC is created. Otherwise, DBS will be inconsistent. Execute the rollback procedure as described in Rollback Procedure, to recover from this faulty situation.

    The next example shows the script output for a succeed state (message depends on the SDPs being upgraded and the SAPC version you are upgrading to):
    Generating and installing SSH keys in 136.225.73.33

Warning: Permanently added '136.225.73.33' (ECDSA) to 
the list of known hosts.
Adding EnvironmentAggregation-ssh-persistency to node 1
Starting RPM activation
Installing EnvironmentAggregation-ssh-persistency
Completed RPM activation
Adding EnvironmentAggregation-ssh-persistency to node 2
Starting RPM activation
Installing EnvironmentAggregation-ssh-persistency
Completed RPM activation
********************************************************
 Applying Software Package: SAPC-CXP9030138_7-7.2.0-16
********************************************************

WARNING: Ignoring unknown argument(s): 
Running preconditions ...
Current I/O write speed: 97.4

Checking alarms ...
(pass)

Available kbytes at /cluster: 13239152
Required  kbytes at /cluster: 3670016
(pass)

Patching software package '... eric-sapc_cxp9030138_
7r3a16/SAPC_software.tar.gz'...
Patching file '... eric-sapc_cxp9030138_7r3a16/unpack/
csm/csm.yml'...
Done: file '... eric-sapc_cxp9030138_7r3a16/unpack/
csm/csm.yml' patched with nodes 'safAmfNode=PL-4,
safAmfCluster=myAmfCluster:safAmfNode=PL-3,
safAmfCluster=myAmfCluster'.
Done: software package '... eric-sapc_cxp9030138_
7r3a16/SAPC_software.tar.gz' has been patched.

Creating UP package...
&lt;12>Mar 21 01:43:16 CMW: Current subType is &lt;Empty>. 
You can run cmw-swm-config-set --subType to set it.
Extracting '/storage/no-backup/sapc/deploy_20180321-014128/SAPC_software.tar.gz' to '/storage/no-backup/nbi_root/SoftwareManagement/
cmwcsp-SAPC_software.tar.gz-JXVI'...
DEPLOYMENT.ready
csm.metadata
ovf-env.xml
packageChsum_2.sha2
schema_ver
unpack/
unpack/plugin/
unpack/plugin/coremw_x86_64-4.5.1-09-deployment-sle-cxp9017564/
unpack/plugin/coremw_x86_64-4.5.1-09-deployment-sle-cxp9017564/support_tools
unpack/plugin/coremw_x86_64-4.5.1-09-deployment-sle-cxp9017564/upgrade-campaigns.metadata
unpack/plugin/coremw_x86_64-4.5.1-09-deployment-sle-cxp9017564/preinstall.sh
unpack/plugin/coremw_x86_64-4.5.1-09-deployment-sle-cxp9017564/ait-config
...
unpack/lm/sapc-soap-notification-service-7.2.0-16-runtime-cxp9035722_7r3a16/ERIC-SAPC_SoapNotificationService-CXP9032730_7-R3A16.sdp
unpack/csm/csm.yml
&lt;14>Mar 21 01:43:47 CMW: URI: file:///
SoftwareManagement/cmwcsp-SAPC_software.tar.gz-JXVI
4
Done!

Running actions before prepare phase...
PM job ResourcesCountersJob stopped.
PM job ResourcesCountersThresholdJob stopped.
PM job policyControlFunctionCapacityMeasuresJob 
stopped.
PM job policyControlFunctionMeasuresJob stopped.
PM job policyControlFunctionThresholdsJob stopped.
Disabling SAPC PDC...
2018-03-21 01:43:54,186 - INFO - Mode command
2018-03-21 01:43:54,186 - INFO - Disabling pdc collect 
execution in crontab
2018-03-21 01:43:54,215 - INFO - Remember to execute 
the same operation in the other SC in order to be 
coherent. 
Indeed, this is an internal procedure.
2018-03-21 01:43:54,474 - INFO - Mode command
2018-03-21 01:43:54,474 - INFO - Disabling pdc collect 
execution in crontab
2018-03-21 01:43:54,504 - INFO - Remember to execute 
the same operation in the other SC in order to be 
coherent. 
Indeed, this is an internal procedure.
Skipping. MIPS already defined
Emptying /var/log/messages ...

[ 01:43:58 ]  PREPARE_IN_PROGRESS
[ 01:44:09 ]  PREPARE_IN_PROGRESS
[ 01:44:19 ]  PREPARE_IN_PROGRESS
[ 01:44:29 ]  PREPARE_IN_PROGRESS
...
[ 01:47:40 ]  ACTIVATION_IN_PROGRESS
[ 01:47:50 ]  ACTIVATION_IN_PROGRESS
[ 01:48:00 ]  ACTIVATION_IN_PROGRESS
[ 01:48:11 ]  ACTIVATION_IN_PROGRESS
[ 01:48:21 ]  ACTIVATION_IN_PROGRESS
[ 01:48:32 ]  ACTIVATION_IN_PROGRESS
...
[ 01:57:53 ]  ACTIVATION_IN_PROGRESS
Error getting state of "SAPC-CXP9030138_7-7.2.0-16"
5 secs to retry [get_current_state 
SAPC-CXP9030138_7-7.2.0-16] (100 attempts still remain)
...
[ 02:01:18 ]  ACTIVATION_IN_PROGRESS
[ 02:01:29 ]  ACTIVATION_IN_PROGRESS
... 

SAPC-CXP9030138_7-7.2.0-16 committed successfuly!

Running actions on commit commpleted...
Inventory cleanup ...
Create swVersion ...
Create consist-of dependencies ...
Create swVersion administrative data ...
Activate swVersion ...
PM job ResourcesCountersJob started.
PM job ResourcesCountersThresholdJob started.
PM job policyControlFunctionCapacityMeasuresJob 
started.
PM job policyControlFunctionMeasuresJob started.
PM job policyControlFunctionThresholdsJob started.
2018-03-21 02:05:27,852 - INFO - Mode command
2018-03-21 02:05:27,852 - INFO - Enabling pdc collect execution in crontab no crontab for root
2018-03-21 02:05:27,871 - INFO - Remember to execute the same operation in the other SC in order to be coherent. Indeed, this is an internal procedure.
2018-03-21 02:05:28,150 - INFO - Mode command
2018-03-21 02:05:28,150 - INFO - Enabling pdc collect execution in crontab no crontab for root
2018-03-21 02:05:28,169 - INFO - Remember to execute the same operation in the other SC in order to be coherent. Indeed, this is an internal procedure.



CBA Repository list
--------------------
Not used:
   ERIC-CMWUpgrade NotUsed
   ERIC-ss7caf-eabss7053-cxp9029429-5.1.6-09 NotUsed

Used:
   3PP-lm-cli-cxp9060355-6.3.0-90.sle12 Used
   ERIC-AmfImmObjectsLib-CXC1731655_4-R5A69 Used
   ERIC-AmfImmObjectsLib-CXC1731655_4-R5B01 Used
   ERIC-Boost-CXC1739275_4-R5A69 Used
   ERIC-Boost-CXC1739275_4-R5B01 Used
   ERIC-Brf-CXP9018859_2-R6B09 Used
   ERIC-BrfCmwA-CXP9018859_2-R6B09 Used
   ERIC-BrfEia-CXP9024651_2-R6B09 Used
   ERIC-BrfImmObjectsLib-CXC1734455_4-R5A69 Used
   ERIC-BrfImmObjectsLib-CXC1734455_4-R5B01 Used
   ERIC-BrfLib-CXC1734454_4-R5A69 Used
   ERIC-BrfLib-CXC1734454_4-R5B01 Used
   ERIC-Brfp-CXP9018859_2-R6B09 Used
   ERIC-CDiameter-CXP9034135_2-R2A183 Used
   ERIC-COM-CXP9028493_2-R5A26 Used
   ERIC-COREMW_COMMON-CXP9017566_2-R6B09 Used
   ERIC-COREMW_OPENSAF-CXP9017656_2-R6B09 Used
   ERIC-COREMW_SC-CXP9017565_2-R6B09 Used
   ERIC-CdclsvAgent-CXC1732319_4-R5B01 Used
   ERIC-CdclsvComTools-CXC1738193_4-R5A69 Used
   ERIC-CdclsvComTools-CXC1738193_4-R5B01 Used
   ERIC-CdclsvDirector-CXC1732318_4-R5B01 Used
   ERIC-CdclsvImmObjectsLib-CXC1732321_4-R5A69 Used
   ERIC-CdclsvImmObjectsLib-CXC1732321_4-R5B01 Used
   ERIC-CdclsvLogRotator-CXC1732320_4-R5A69 Used
   ERIC-CdclsvLogRotator-CXC1732320_4-R5B01 Used
   ERIC-CdfCampaignPlugin-CXC1739791_3-R5B01 Used
   ERIC-CdsvControlAPI-CXC1731657_4-R5A69 Used
   ERIC-CdsvControlAPI-CXC1731657_4-R5B01 Used
   ERIC-CdsvDirector-CXC1731651_4-R5B01 Used
   ERIC-CdsvDistributionAPI-CXC1731656_4-R5A69 Used
   ERIC-CdsvDistributionAPI-CXC1731656_4-R5B01 Used
   ERIC-CdsvImmObjectsLib-CXC1731654_4-R5A69 Used
   ERIC-CdsvImmObjectsLib-CXC1731654_4-R5B01 Used
   ERIC-CluAPI-CXC1732367_4-R5A69 Used
   ERIC-CluAPI-CXC1732367_4-R5B01 Used
   ERIC-CluDirector-CXC1732315_4-R5B01 Used
   ERIC-CluImmObjectsLib-CXC1732317_4-R5A69 Used
   ERIC-CluImmObjectsLib-CXC1732317_4-R5B01 Used
   ERIC-CluRun-CXC1732316_4-R5A69 Used
   ERIC-CluRun-CXC1732316_4-R5B01 Used
   ERIC-Dbn-CXC1732304_5-R6C01 Used
   ERIC-DbnRPCPluginCmw-CXC1732368_5-R6C01 Used
   ERIC-DbsCsmPlugin-CXC1738069_5-R6C01 Used
   ERIC-DbsDSCPConfig-CXC1740640_5-R6C01 Used
   ERIC-DbsDbApi-CXC1732369_5-R6C01 Used
   ERIC-DbsvAgent-CXC1731659_5-R6C01 Used
   ERIC-DbsvDirector-CXC1731661_5-R6C01 Used
   ERIC-DbsvImmObjectsLib-CXC1732370_5-R6C01 Used
   ERIC-DscpLib-CXC1739936_4-R5A69 Used
   ERIC-DscpLib-CXC1739936_4-R5B01 Used
   ERIC-EVIP-CXP9017652_3-R6B01 Used
   ERIC-Eshsv-CXC1736038_4-R5B01 Used
   ERIC-Hbmsv-CXC1739497_4-R5B01 Used
   ERIC-HbmsvUserLib-CXC1739498_4-R5A69 Used
   ERIC-HbmsvUserLib-CXC1739498_4-R5B01 Used
   ERIC-LINUX_CONTROL-CXP9013151_4-R5A24 Used
   ERIC-LINUX_PAYLOAD-CXP9013152_4-R5A24 Used
   ERIC-LemControlTools-CXP9050134_4-R5A69 Used
   ERIC-LemControlTools-CXP9050134_4-R5B01 Used
   ERIC-LmClientLibrary-CXP9022092_6-R4A90 Used
   ERIC-LmSa-CXP9021377_2-R5A03 Used
   ERIC-LmServer-CXP9022159_6-R4A90 Used
   ERIC-ScsvCollector-CXC1734482_4-R5B01 Used
   ERIC-ScsvStreamer-CXC1734483_4-R5B01 Used
   ERIC-SeeLib-CXC1731650_4-R5A69 Used
   ERIC-SeeLib-CXC1731650_4-R5B01 Used
   ERIC-SeePort-CXC1731653_4-R5A69 Used
   ERIC-SeePort-CXC1731653_4-R5B01 Used
   ERIC-SmfImmObjectsLib-CXC1739038_4-R5A69 Used
   ERIC-SmfImmObjectsLib-CXC1739038_4-R5B01 Used
   ERIC-TRACE_EA_Common-CXP9040232_3-R6A13 Used
   ERIC-TRACE_EA_SC-CXP9040241_3-R6A13 Used
   ERIC-Thrift-CXC1739276_4-R5A69 Used
   ERIC-Thrift-CXC1739276_4-R5B01 Used
   ERIC-TraceC-CXP9019680_3-R6A04 Used
   ERIC-TraceP-CXP9019681_3-R6A04 Used
   ERIC-javaoam-core-cxp9030376-4.0.0-6 Used
   ERIC-lde-brf-script-cxp9021148-2.5.0-1.sle12 Used
   ERIC-sec-acs-cxp9026450-2.5.0-082.sle12 Used
   ERIC-sec-cert-agent-cxp9027891-2.5.0-082.sle12 Used
   ERIC-sec-cert-manager-cxp9027891-2.5.0-082.sle12 Used
   ERIC-sec-crypto-cxp9027895-2.5.0-082.sle12 Used
   ERIC-sec-la-ldap-cxp9026994-2.5.0-082.sle12 Used
   ERIC-sec-la-oi-cxp9026994-2.5.0-082.sle12 Used
   ERIC-sec-la-sm-cxp9026994-2.5.0-082.sle12 Used
   ERIC-sec-secm-cxp9028976-2.5.0-082.sle12 Used
   ERIC-sec-secm-ln-cxp9028976-2.4.0-078.sle12 Used
   ERIC-sec-secm-ln-cxp9028976-2.5.0-082.sle12 Used
   ERIC-ss7caf-eabss7038-cxp9029429-6.2.0-648 Used
   ERIC-ss7caf-eabss7039-cxp9029429-6.2.0-648 Used
   ERIC-ss7caf-eabss7041-cxp9029429-6.2.0-648 Used
   ERIC-ss7caf-eabss7044-cxp9029429-6.2.0-648 Used
   ERIC-ss7caf-eabss7049-cxp9029429-6.2.0-648 Used
   ERIC-ss7caf-eabss7050-cxp9029429-6.2.0-648 Used
   ERIC-ss7caf-eabss7052-cxp9029429-6.2.0-648 Used
   ERIC-ss7caf-eabss7053-cxp9029429-6.2.0-648 Used
   ERIC-ss7caf-eabss7069-cxp9029429-6.2.0-648 Used
   ERIC-ss7caf-eabss7072-cxp9029429-6.2.0-648 Used
   ERIC-ss7caf-eabss7077-cxp9029429-6.2.0-648 Used
   ERIC-ss7caf-eabss7078-cxp9029429-6.2.0-648 Used


SAPC Repository list
--------------------
Used:
   ERIC-SAPC_AppDetection-CXP9034938_7-R3A16 Used
   ERIC-SAPC_CDM-CXP9030054_7-R3A16 Used
   ERIC-SAPC_DbsProvisioning-CXP9033057_7-R3A16 Used
   ERIC-SAPC_DbsRestartMonitor-CXP9031668_7-R3A16 Used
   ERIC-SAPC_DbsSession-CXP9031460_7-R3A16 Used
   ERIC-SAPC_DeployTools-CXP9032330_7-R3A16 Used
   ERIC-SAPC_DeployToolsPayload-CXP9035387_7-R3A16 Used
   ERIC-SAPC_EndUserNotifier-CXP9030973_7-R3A16 Used
   ERIC-SAPC_EnvironmentAggregation-CXP9035385_7-R3A16 Used
   ERIC-SAPC_EnvironmentAll-CXP9035384_7-R3A16 Used
   ERIC-SAPC_EnvironmentPayload-CXP9035386_7-R3A16 Used
   ERIC-SAPC_ExternalDatabase-CXP9032415_7-R3A16 Used
   ERIC-SAPC_GeoRedAggregation-CXP9035797_7-R3A16 Used
   ERIC-SAPC_GeoRedPayload-CXP9032329_7-R3A16 Used
   ERIC-SAPC_JRE-CXP9032038_7-R3A16 Used
   ERIC-SAPC_LicenseMonitor-CXP9032519_7-R3A16 Used
   ERIC-SAPC_MobilityPolicy-CXP9034045_7-R3A16 Used
   ERIC-SAPC_ObsoleteSesNotif-CXP9032520_7-R3A16 Used
   ERIC-SAPC_PcrfProc-CXP9030974_7-R3A16 Used
   ERIC-SAPC_Pdc-CXP9034416_7-R3A16 Used
   ERIC-SAPC_PeerConfMonitor-CXP9034686_7-R3A16 Used
   ERIC-SAPC_PotUtility-CXP9030831_7-R3A16 Used
   ERIC-SAPC_RestServer-CXP9032712_7-R3A16 Used
   ERIC-SAPC_SoapNotificationService-CXP9032730_7-R3A16 Used
   ERIC-SAPC_SubsChargingProc-CXP9031741_7-R3A16 Used
   ERIC-SAPC_TTNotifier-CXP9030264_7-R3A16 Used
   ERIC-SAPC_UeTraceControl-CXP9034640_7-R3A16 Used
   ERIC-SAPC_UeTraceRelay-CXP9034792_7-R3A16 Used


Execution completed in 1302 seconds

Uninstall SSH public key in 136.225.73.33
  3. If the script execution was interrupted for any reason:
    Warning!

    Launch it again. The execution will be resumed in the step where the script was interrupted:

    <External_Machine>:$ <SAPC_UPGRADE_PACKAGE_PATH>/upgrade/run.sh root <VIP_OAM>
    Note: This command requests to type the root password in the SAPC.
  4. If the upgrade was successfully applied, continue with steps described in Post-Upgrade Installation.
    Output example of successful upgrade:
    ...
Execution completed in 1302 seconds

Uninstall SSH public key in 136.225.73.33
  5. In case the upgrade was not successfully applied, execute rollback procedure as described in Rollback Procedure.
    Output example of unsuccessful upgrade:
    ...
[ 17:46:50 ]  ACTIVATION_IN_PROGRESS
ERROR: Activation phase did not end successfully for 
"SAPC-CXP9030138_7-7.2.0-12"

3.3.1 Cleanup Procedure

Perform a cleanup procedure following the steps below.

Caution!

All these steps must be done during the maintenance window. During the upgrade process, the blades of the SAPC cluster are rebooted.

Steps

  1. Create a System Data Backup.
    To create a backup, refer to Backup and Restore.
  2. For the SAPC VNF deployment, to have the new version of the SAPC SW images, update them in OpenStack or VMware environments. For more information, refer to:
  3. Remove the upgrade package directory.
    SC-x:~ # rm -rf /cluster/storage/no-backup/<SAPC_UPGRADE_PACKAGE>

3.4 Post-Upgrade Installation

3.4.1 Enable Session Inactivity Cleanup Mechanism

If the Session Inactivity Cleanup mechanism was active before the upgrade and disabled as told in Disable Session Inactivity Cleanup Mechanism, enable it again. This procedure is detailed in Enabling/disabling the mechanism subchapter from the Configure Session Inactivity Cleanup Mechanism document.

3.4.2 Upgrade Verification

Once the upgrade has finished, do some verification checks to approve the status of the upgrade.

Steps

  1. The new SAPC Components have to be in "Used" state. If old ones are in "Used" state, follow the rollback procedure explained in Rollback Procedure and contact the next support level.
    $ssh -l root <OAM VIP>
    SC-X:# cmw-repository-list | grep SAPC
  2. Check the health of the SAPC as described in SAPC State Verification
  3. Check that the node is licensed accordingly to SAPC SW and, if necessary, update or install required ones by following SAPC licensing instructions.

3.4.3 Entities Definition Upgrade

Run the following commands to upgrade the entities definition:

Steps

  1. Log in with the "root" user ID to <VIP_OAM>, using the following command:
    External_Machine:$ ssh root@<VIP_OAM>
  2. Execute the entities definition upgrade:
    SC-x:# cd /cluster/storage/no-backup/<SAPC_UPGRADE_PACKAGE>/upgrade/
    SC-x:# python entitiesDefinitionUpgradeTool.py
    Attention!

    In external subscriber repository deployments, only when there are new attributes for existing entities, or new entities that are eligible to be configured in an external repository (see SAPC 1 Network Impact Report), update their fieldDef definition and URL as necessary. For detailed information on how to update the fieldDef block and the URL, see Integration in User Data Consolidation and Database Access.

4 Software Upgrade for Active-Standby Geographical Redundancy

4.1 Prepare to Upgrade Active-Standby Geographical Redundancy

This section describes the preparations needed before both SAPC peers are upgraded.

To upgrade both SAPC peers within an Active-Standby Geographical Redundancy, follow these steps:

Steps

  1. To Identify what is the SAPC acting as Active and acting as Standby, follow section Monitor Geographical Redundancy State in Active-Standby Geographical Redundancy in both SAPC peers. Also, check if the SAPC acting as Active is configured as Preferred following section Check Geographical Redundancy Configuration in Active-Standby Geographical Redundancy. If the SAPC acting as Active is configured as Non Preferred, follow section Configure Active SAPC as Preferred in Change Preferred Role in Active-Standby Geographical Redundancy.
  2. In Standby SAPC, follow section Disable Active-Standby Geographical Redundancy in Temporarily Disable Active-Standby Geographical Redundancy. This procedure stops replication and changes the state to Halted.
  3. In Active SAPC execute next command to disable replication:
    immcfg -a isEnabled=0 dbsNetsharedConfigId=1

4.2 Software Upgrade in Active SAPC

4.2.1 Pre-Upgrade Installation

Before applying the upgrade follow the same procedures for Stand Alone deployments, indicated in Pre-Upgrade Installation.

4.2.2 Host SLES Patches Update

For SAPC VNF deployment, skip this section, and continue with sectionUpgrade for Active SAPC

For SAPC PNF deployment, follow the same procedures for Stand Alone deployments, indicated in Host SLES12 Upgrade to SP2.

4.2.3 Upgrade for Active SAPC

Follow the same procedures for Stand Alone deployments, indicated in Upgrade for Stand Alone, except that in Step 2 in Upgrade for Stand Alone, run this command instead:

<External_Machine>:$

<SAPC_UPGRADE_PACKAGE_PATH>/upgrade/run.sh -s '7' root <VIP_OAM>

4.2.4 Post-Upgrade Installation

Follow the same procedures for Stand Alone deployments, indicated in Post-Upgrade Installation.

4.3 Clean up Procedure in Active SAPC

Follow the same procedures for Stand Alone deployments, indicated in Clean up Procedure.

4.4 Software Upgrade in Halted SAPC

4.4.1 Pre-Upgrade Installation

Before applying the upgrade follow the same procedures for Stand Alone deployments, indicated in Pre-Upgrade Installation.

4.4.2 Host SLES Patches Update

For SAPC VNF deployment, skip this section, and continue with sectionUpgrade for Halted SAPC

For SAPC PNF deployment, follow the same procedures for Stand Alone deployments, indicated in Host SLES12 Upgrade to SP2.

4.4.3 Upgrade for Halted SAPC

Follow the same procedures for Stand Alone deployments, indicated in Upgrade for Stand Alone, except that in Step 2 in Upgrade for Stand Alone, run this command instead:

<External_Machine>:$

<SAPC_UPGRADE_PACKAGE_PATH>/upgrade/run.sh -s '7' root <VIP_OAM>

4.4.4 Post-Upgrade Installation

Follow the same procedures for Stand Alone deployments, indicated in Post-Upgrade Installation.

4.5 Clean up Procedure in Halted SAPC

Follow the same procedures for Stand Alone deployments, indicated in Clean up Procedure.

4.6 Post-Upgrade Activities for Active-Standby Geographical Redundancy

This section describes the post upgrade activities needed once both SAPC peers are upgraded.

To enable replication in both SAPC peers within an Active-Standby Geographical Redundancy, follow these steps:

Steps

  1. In Active SAPC execute next command to enable replication:
    immcfg -a isEnabled=1 dbsNetsharedConfigId=1
  2. In Halted SAPC, follow section Enable Active-Standby Geographical Redundancy in Temporarily Disable Active-Standby Geographical Redundancy.

5 Software Upgrade for Active-Active Geographical Redundancy

5.1 Prepare to Upgrade Active-Active Geographical Redundancy

This section describes the preparations needed before both SAPC peers are upgraded.

To upgrade both SAPC peers within an Active-Active Geographical Redundancy, follow these steps:

Steps

  1. To Identify what is the SAPC configured as Preferred and Non Preferred, follow section Check Geographical Redundancy Configuration in Active-Active Geographical Redundancy.
  2. In the SAPC configured as Non Preferred, follow section Disable Active-Active Geographical Redundancy in Temporarily Disable Active-Active Geographical Redundancy. This procedure stops replication and changes the state to Halted.
  3. In the SAPC configured as Preferred, execute next command to disable replication:
    immcfg -a isEnabled=0 dbsNetsharedConfigId=1

5.2 Software Upgrade in SAPC Configured as Preferred

5.2.1 Pre-Upgrade Installation

Before applying the upgrade follow the same procedures for Stand Alone deployments, indicated in Pre-Upgrade Installation.

5.2.2 Host SLES Patches Update

For SAPC VNF deployment, skip this section, and continue with sectionUpgrade for SAPC Configured as Preferred

For SAPC PNF deployment, follow the same procedures for Stand Alone deployments, indicated in Host SLES12 Upgrade to SP2.

5.2.3 Upgrade for SAPC Configured as Preferred

Follow the same procedures for Stand Alone deployments, indicated in Upgrade for Stand Alone, except that in Step 2 in Upgrade for Stand Alone, run this command instead:

<External_Machine>:$

<SAPC_UPGRADE_PACKAGE_PATH>/upgrade/run.sh -s '7' root <VIP_OAM>

5.2.4 Post-Upgrade Installation

Follow the same procedures for Stand Alone deployments, indicated in Post-Upgrade Installation.

5.3 Clean up Procedure in SAPC Configured as Preferred

Follow the same procedures for Stand Alone deployments, indicated in Clean up Procedure.

5.4 Software Upgrade in SAPC Configured as Non Preferred

5.4.1 Pre-Upgrade Installation

Before applying the upgrade follow the same procedures for Stand Alone deployments, indicated in Pre-Upgrade Installation.

5.4.2 Host SLES Patches Update

For SAPC VNF deployment, skip this section, and continue with sectionUpgrade for SAPC Configured as Non Preferred

For SAPC PNF deployment, follow the same procedures for Stand Alone deployments, indicated in Host SLES12 Upgrade to SP2.

5.4.3 Upgrade for SAPC Configured as Non Preferred

Follow the same procedures for Stand Alone deployments, indicated in Upgrade for Stand Alone, except that in Step 2 in Upgrade for Stand Alone, run this command instead:

<External_Machine>:$

<SAPC_UPGRADE_PACKAGE_PATH>/upgrade/run.sh -s '7' root <VIP_OAM>

5.4.4 Post-Upgrade Installation

Follow the same procedures for Stand Alone deployments, indicated in Post-Upgrade Installation.

5.5 Clean up Procedure in SAPC Configured as Non Preferred

Follow the same procedures for Stand Alone deployments, indicated in Clean up Procedure.

5.6 Post-Upgrade Activities for Active-Active Geographical Redundancy

This section describes the post upgrade activities needed once both SAPC peers are upgraded.

To enable replication in both SAPC peers within an Active-Active Geographical Redundancy, follow these steps:

Steps

  1. In the SAPC configured as Preferred, execute next command to enable replication:
    immcfg -a isEnabled=1 dbsNetsharedConfigId=1
  2. In the SAPC configured as Non Preferred, follow section Enable Active-Active Geographical Redundancy in Temporarily Disable Active-Active Geographical Redundancy.

6 Clean up Procedure

Clean up procedure must be done whenever the upgrade has finished successfully. Clean up procedure has several steps.

Steps

  1. Create a System Data Backup.
    To create a backup, refer to Backup and Restore.
  2. For SAPC VNF deployment, to have the new version of the SAPC SW images, update them in OpenStack environments (refer to SAPC VNF Deployment Instruction for OpenStack) or VMware environments (refer to SAPC VNF Deployment Instruction for VMware).
  3. For SAPC VNF deployment, to perform lifecycle management operations through the VNF LifeCycle Manager (VNF-LCM), it is required to have the new version of the SAPC SW images and the SAPC workflows in the OpenStack or VMware environments. Refer to SAPC VNF Lifecycle Manager Workflow Instruction for OpenStack and SAPC VNF Lifecycle Manager Workflow Instruction for VMware.
  4. Remove the upgrade package directory.
    SC-x:~ # rm -rf /cluster/storage/no-backup/<SAPC_UPGRADE_PACKAGE>

7 Rollback Procedure

Owing to abnormal upgrade procedure termination, it could be necessary to recover from situations where the software has not been upgraded. Follow the next steps to recover from an unsuccessful upgrade to the previous CBA state:

8 Upgrade Logs

SAPC Component progress is reported in safLog.

SC-x:~ # tail -f /cluster/storage/no-backup/coremw/var/log/saflog/saLogSystem_<date>_<time>.log | grep safSmfCampaign

Legal |© Ericsson España, S.A. 2018