Contents

1   Introduction

This section is an introduction to this document. It contains information about the prerequisites, purpose, scope, and target group for the document. This section also contains explanations of typographic conventions used in this document.

1.1   Purpose and Scope

This document covers the system administration, operation, and maintenance routines available for Dynamic Activation 1.

1.2   Target Groups

The target groups for this document are as follows:

• System Administrator
• Network Administrator
• Network Supervision Administrator

1.3   Typographic Conventions

Typographic conventions are described in the document Library Overview, Reference [1].

For information about abbreviations and terms used throughout this document refer to Glossary of Terms and Acronyms, Reference [2].

1.4   Prerequisites

The following are the prerequisites to make full use of this document:

• Knowledge about the Ericsson Dynamic Activation (EDA) product
• Knowledge about Linux operating system
• Knowledge about the Linux Distribution Extensions (LDE) product.

2   System Information

This section holds information about the Dynamic Activation system and how to retrieve useful hardware and software information.

2.1   System Architecture

This section describes the Dynamic Activation system architecture, as shown in Figure 1.

The system consists of two types of nodes:

• System Controller Node

  The System Controller (SC) nodes host the High Availability (HA) file system that is used by the Payload (PL) nodes. The SC nodes also provide generic services to the rest of the system, including hosting of the GUI. In a Dynamic Activation system two SC nodes are always present.

  Note:  

  ZooKeeper is deployed on the SC nodes and on the first PL node (PL3) by default.

  
  
• Payload Node

  The PL nodes execute the provisioning operations and host the provisioning logic. In a Dynamic Activation system 2-10 PL nodes can be present.

ZooKeeper

ZooKeeper is a cluster storage engine that provides mechanism to get notifications when data is modified, internal data distributor for configuration (for example NEs, NE Groups, Routing), symmetric data modifications, user management, session replication, group membership, and license management.

ZooKeeper runs in a separate process. It runs (by default) on three nodes in the cluster, on both SC nodes and on the first PL node.

For ZooKeeper to work, at least two ZooKeeper instances need to be operational at the same time. Stopping of ZooKeeper instances must be handled with caution, since the Dynamic Activation application is depending on its function. If more than one ZooKeeper instance is brought down simultaneously, the Dynamic Activation application will not function properly.

Restarting one ZooKeeper instance during traffic could cause short traffic disturbance.

Cassandra

Cassandra is a distributed database providing data access through the Cassandra Query Language (CQL) interface. Cassandra is used for storage and presentation of provisioning events performed by the Business Logic, these are the Dynamic Activation processing logs.

Function deployment in Dynamic Activation

The Dynamic Activation functions are implemented as Module and 3PP software, used by the modules. For each module there is a corresponding process (JVM), deployed on one or several nodes, see Table 2.

The functions showed in Figure 1 are implemented by modules, see Table 1.

• The dve-inbound-interfaces-application module supports the inbound MML Interface, mediating incoming requests to different services.
• The nbia-module supports the CAI3G , the CAI and the EDIFACT inbound interfaces, mediating incoming requests to different services.
• The tomcat-server module hosts the GUI function. It is based on the Apache Tomcat 3PP.
• The dve-application module executes provisioning logic, converting service invocations into provisioning requests towards the NEs, for example HLR, CUDB, HSS. It also supports massive updates, and multi-destination provisioning activities through the CLI.
• The arh-module provides support for asynchronous requests. When an asynchronous service request is received, the arh-module module generates a notification to the service consumer. Once the service request is processed by the business logic, the module generates a provisioning result notification.
• The akka-cluster-service-lib module implements the internal Service Location Register (SLR) function. Service provider modules register their services in the SLR when the module is started. Service consumer modules request service Lookups from the SLR, when needed.
• The activation-orchestration-module provides support for stateful and resilient execution of model driven activation logic and queueing towards provisioning resource endpoints.
• The lockservice-module provides support for preventing the concurrent provisioning to the same subscriber. When more than one requests are received on the same subscriber, the first request acquires a lock and other requests are rejected with an error code.
• The ema-pq-module provides support for asynchronous queues to process failed commands. Commands for a subscriber in the same NE are executed in sequence.
• The admin-tool provides an import for storing and an export function of the stored processing logs generated by the different modules.
• The cudb-block-proxy mediates incoming blocking and unblocking requests into the dve-application module from the CUDB. Upon a blocking request, the dve-application module holds certain provisioning requests towards the CUDB, until a deblocking request is received, on which the provisioning requests are resumed.

  For information about which provisioning requests that are blocked, see Configuration Manual for Resource Activation, Reference [9].

  The backup notification from CUDB is received through the O&M VLAN in the O&M IP address. It is also necessary to configure the JMX ports in the right order, <VIP-OAM-IP>:8994:8099

• The ema-faultmgmt-module is responsible of receiving faults from all other modules. The faults are propagated from this module to ESA for sending SNMP alarms, and also sent out as SMTP if e-mail notifications are configured.
• The ema-authentication-module is responsible for authentication of users. Other modules in the system that need to authenticate user data, interacts with this module.
• The ema-batch-module provides support for bulk provisioning of CAI and synchronous CAI3G requests towards the NEs, it interacts with nbia-module to execute provisioning requests.

The deployment of modules and 3PPs is shown in Table 2 and Table 3.

(1)   By default, installed on the first PL node (PL3)

2.1.1   Dynamic Activation Configurations

This section describes the available Dynamic Activation configurations.

The minimum Dynamic Activation system consists of two SC nodes and two PL nodes.

If more provisioning capacity is needed, extra PL nodes can be added to the system.

The largest Dynamic Activation system consists of two SC nodes and ten PL nodes.

For information about the hardware configurations in Dynamic Activation 1, refer to BSP 8100 documentation. For details, see document ECP Guideline for Ordering BSP8100 in Dynamic Activation 1, in http://prodcat.internal.ericsson.com.

2.2   Directory Structure

The most important parts of Dynamic Activation directory structure are shown in Figure 2.

2.2.1   Description of /opt

The folder /opt contains add on application software packages. See Table 4 for more specific information on the contents.

Note:  

The folders in this directory are non-persistent, they are only kept in-memory.

(1)   By default installed on the first PL node (PL3).

2.2.2   Description of /var

The folder /var contains variable files, such as logs and spool files. See Table 5 for more specific information.

Note:  

The folders in this directory are persistent.

(1)  By default installed on the first PL node (PL3)

2.2.3   Description of /cluster

The folder /cluster contains Dynamic Activation configuration files and shared deployment directories. See Table 6 for more specific information.

Note:  

The folders in this directory are persistent.

2.2.4   Description of /usr

The folder /usr application software packages. See Table 7 for more specific information on the contents.

Note:  

The folders in this directory are non-persistent, they are only kept in-memory.

2.3   Log Files

This section contains information about different log files in the system.

Note:  

Personal data consists of information related to an identifiable person. An identifiable person is one that can be identified indirectly, for example by its MSISDN, IMSI, or passport number, or directly, for example name, date of birth and postal address. Dynamic Activation handles personal data. In some cases that personal data can be saved in certain log and result files. Those files are automatically maintained, to make sure that personal data is not retained too long. For more information, see Section 5.12.5.

Sensitive information such as passwords, is not recorded in log files.

• Information about how the log files are rotated in the nbia-module is available in the configuration files below on each PL node:
  • /usr/local/pgngn/<module>-<version>/etc/org.ops4j.pax.logging.cfg: The configuration of how module log files (including debug information, exceptions and errors) are rotated.
  • /usr/local/pgngn/<module>-<version>/config/proclog-log4j.properties: The configuration of Processing Log.
• Information about how the log files are rotated in all other modules is available in each corresponding log4j.xml file. These xml files are located in the /usr/local/pgngn/<module>-<version>/config directory, on each PL node.

  As a default setting, the logs have a maximum size of 15MB before they are rotated, and will store a maximum of 10 log files before the oldest logs are deleted.

2.3.1   Operating System

The following log files are available on both SC nodes:

• /var/log/<SC_hostname>/messages

  This is the Linux system log file. It is rotated when larger than 4096 kb.

• /var/log/<SC_hostname>/auth

  This is the log file for system login attempts.

Note:  

For a full list of messages, check both SC nodes.

2.3.2   Dynamic Activation Application Logs

Log File Sizes

All the log files introduced in this section have a maximum size of 15 MB before they are rotated, and a maximum of 10 log files are stored before the oldest logs are deleted.

When a log file gets rotated, it is backed up with the same name, but appended with a "1", "2", "3", and then stepping on.

For example, massiveoperations.log is updated to massiveoperations.log.1, and then massiveoperations.log.2.

Ten rotated files can exist at the same time, meaning when the 11^th log file gets rotated, it overwrites the previous log file appended with a "10".

Module Log Files

Modules in Dynamic Activation have their respective log files, which are located in /var/log/dve.

The log files use the following name convention:

<module-name>[-<log-type>].log, where:

• <module-name> indicates which module the log file belongs to.
• (Optional) <log-type> can be omitted, or one of the following, depending on what log information is contained:
  • Omitted: contains common log information.
  • audit: contains failed login attempts to the Dynamic Activation.
  • config: contains configuration changes in the GUI, personal data, or both.
  • console: contains console output of this module.
  • karaf: contains special log for NBIA.
  • oauth: contains GUI login attempts and management of GUI users.

  For more information, see Table 8.

Log file name examples: arh-module.log, arh-module-console.log.

Special Log Files on Both SC and PL Nodes

The following log file is available on both SC and PL nodes:

• /var/log/dve/consolidated.log

  Note:  

  Beware that the configurations will be restored to default settings when rebooting the system, or when executing the command:

  $ bootloader.py node activate --host <hostname>

  
  

  This file logs exceptions and errors in the application, and can also be used for debugging the application by changing the log levels in the corresponding log4j.xml, or org.ops4j.pax.logging.cfg file.

  The nbia-module is the module that use the org.ops4j.pax.logging.cfg config file. All other modules use the log4j.xml logging config file.

  These files can contain personal data.

Special Log Files on PL Nodes

The following log files are available on PL nodes:

• /var/log/dve/multi-log-consolidation-console.log

  This log file contains information of the multi-log-consolidation process. Under normal operating conditions, this file is empty.

• /var/log/dve/massiveoperations.log

  Note:  

  This file is only applicable for UDC solutions.

  
  

  This log file contains all messages sent over CLI.

  This file can contain personal data.

• /var/log/dve/request/failedrequests.log

  This log file contains failed requests.

• /var/log/dve/request/partiallysucceededoperations.log

  This log file contains the partially successful request and the subsequent southbound operations.

  If the sequence of operations was not successful, this causes inconsistency in the CUDB. Correct the inconsistency manually. For more information, refer to CUDB Subscription Repair and Remove Procedures, Reference [4].

  This file can contain personal data.

• /var/log/dve/request/eir_partiallysucceededoperations.log

  Note:  

  This file is only applicable for UDC solutions.

  
  

  This log file contains the partially successful CAI3G request and the subsequent southbound operations.

  These southbound operations relate to the logging of IMEIlogdata in the CUDB. The unsuccessful operation causes incomplete data in the CUDB and requires to be manually corrected through direct southbound requests towards the CUDB. For information on how to perform such manual procedures, refer to the EIR section in CUDB Subscription Repair and Remove Procedures, Reference [4].

  This file can contain personal data.

• /var/log/dve/request/successfulrequests.log

  This log file contains successful requests.

  This file can contain personal data.

2.3.2.1   Persistent Log Settings

When rebooting the system or activating the configuration on a node, the configuration will be restored to it's default settings. To avoid overwriting wanted settings, the settings can be made persistent by changing the log levels for the following config files:

• <module_name>-audit.log
• failedrequests.log
• partiallysucceededoperations.log
• eir_partiallysucceededoperations.log
• successfulrequests.log

The following log level parameters can be set:

• @LOG_LEVEL_AUTHORIZATION@ - authorization (default set to OFF)
• @LOG_LEVEL_AUTHENTICATION@ - authentication (default set to TRACE)
• @LOG_LEVEL_FAILED_REQUESTS@ - failed requests (default set to OFF)
• @LOG_LEVEL_SUCCESSFUL_REQUESTS@ - successful requests (default set to OFF)
• @LOG_LEVEL_PARTIALLY_SUCCEEDED_REQUESTS@ - partially succeeded requests (default set to INFO)
• @LOG_LEVEL_PARTIALLY_SUCCEEDED_EIR_OPERATIONS@ - partially succeeded EIR operations (default set to INFO)

The following log level values are supported and can be modified by editing the corresponding config file.

Follow the procedure below to configure the log levels:

1. From an SC node, run the following command to configure log level setting:

   $ bootloader.py config set --parameter <log_level_parameter> --value <log_level_value>

   The values to use are specified above.

2. From an SC node, run the following command, for all PL nodes one by one, to activate the change:
   Attention!

   Wait for each PL node to be activated before starting with the next one, otherwise traffic disturbances occur. The all parameter should only be used when no provisioning traffic is running.

   $ bootloader.py node activate --host <hostname>

2.3.3   Ericsson SNMP Agent

The following log files are used in ESA:

• /var/log/esa/MasterAgent.log

  This log file contains log entries related to the Master Agent operations.

• /var/log/esa/FmAgent.log

  This log file contains log entries related to the FM Agent operations.

• /var/log/esa/PmAgent.log

  This log file contains log entries related to the PM Agent operations.

• /var/log/esa/alarms.log

  This log file contains every alarm and event that has been triggered in the FM Agent, which means all alarm raise, alarm clear and event are logged.

• /var/log/esa/nontranslated/<ipaddress>.log

  This log file is non-translated alarms log and contains every trap and notification that has gone through the alarm translation process in the FM Agent without being successfully matched to a condition. The complete trap content is logged in a log file named <ipaddress>.log, where <ipaddress> is the IP address of the alarming source.

• /usr/local/ssm/log

  This directory holds the log file ssmagent.log, containing information about ESA System Service Monitor (SSM) operations and errors.

2.3.4   Processing Log File Format

Processing log records both provisioning, and GUI operations in Dynamic Activation.

It is possible to view processing logs in the Log Management subtab in Dynamic Activation GUI, see User Guide for Resource Activation, Reference [8]. Processing logs are indexed by instance; for example imsi, msisdn to easier find related processing logs bound to a certain subscriber.

To extract processing logs from the database, use the Processing Log Admin Tool. For detailed information, see Section 5.10.

Log files are strictly following Comma-Separated Value (CSV) file format. CSV uses a comma to separate values, and uses a double quote (") character around fields that contain reserved characters, such as commas or new lines. If the field itself contains double quote, that double quote should be escaped by one double quote (") character.

Attributes not used for provisioning will not be encoded to a readable format in the processing log.

The structure of log files is described as follows:

For each log file, there is a file header describing what value one log entry contains. The head contains the following field names:

LogType(v1.1),RootLogId,SubLogId,TransactionId,Instance,Operation,Status,User,Hostname,Protocol,Target,StartTime,ExecuteTime,ResponseCode,FullRequest,FullResponse,ReplayLogId

The following is an example of a processing log file:

LogType(v1.1),RootLogId,SubLogId,TransactionId,Instance,Operation,Status,User,Hostname,
Protocol,Target,StartTime,ExecuteTime,ResponseCode,FullRequest,FullResponse,ReplayLogId

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1","","","SEARCH","FAILED","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.307","00 00:00:00.001","32",
"dn: serv=Identities,IMSI=264000000510112,dc=imsi,ou=identities,dc=operator,dc=com
Search Scope: OBJECT
Search Filter: (objectClass=*)
Time Limit Milliseconds : 0
Count Limit: 0
Return Object: False
DerefAliases: ALWAYS
DerefLink: False","[LDAP: error code 32 - No Such Object]",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1","","","SEARCH","FAILED","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.309","00 00:00:00.001","32",
"dn: IMSI=264000000510112,dc=imsi,ou=identities,dc=operator,dc=com
Search Scope: OBJECT
Search Filter: (objectClass=*)
Time Limit Milliseconds : 0
Count Limit: 0
Return Object: False
DerefAliases: NEVER
DerefLink: False","[LDAP: error code 32 - No Such Object]",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1/1","","","SEARCH","FAILED","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.311","00 00:00:00.001","32",
"dn: serv=Identities,MSISDN=4919018510112,dc=msisdn,ou=identities,dc=operator,dc=com
Search Scope: OBJECT
Search Filter: (objectClass=*)
Time Limit Milliseconds : 0
Count Limit: 0
Return Object: False
DerefAliases: ALWAYS
DerefLink: False","[LDAP: error code 32 - No Such Object]",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1/1/1","","","SEARCH","FAILED","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.313","00 00:00:00.001","32",
"dn: serv=CSPS,MSISDN=4919018510112,dc=msisdn,ou=identities,dc=operator,dc=com
Search Scope: OBJECT
Search Filter: (objectClass=*)
Time Limit Milliseconds : 0
Count Limit: 0
Return Object: False
DerefAliases: ALWAYS
DerefLink: False","[LDAP: error code 32 - No Such Object]",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1/1/1/1","","","SEARCH","FAILED","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.315","00 00:00:00.001","32",
"dn: MSISDN=4919018510112,dc=msisdn,ou=identities,dc=operator,dc=com
Search Scope: OBJECT
Search Filter: (objectClass=*)
Time Limit Milliseconds : 0
Count Limit: 0
Return Object: False
DerefAliases: NEVER
DerefLink: False","[LDAP: error code 32 - No Such Object]",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1/1/1/1/1","","","HESUI","SUCCESSFUL","",
"vn-1","TELNET","FT_HLR","2015-07-09 13.26.37.317","00 00:00:00.001","0",
"HESUI:IMSI=264000000510112,MSISDN=4919018510112;","HLR LA PROVISIONING PROCESS MODIFIED DATA

PPID
0

MSISDN  CDC
        7

NEID         VALUE
1.1          919458101101
1.2          046200000150FF11
3.1          2
28.14        0
10.2         4
10.3         1
7.52         7
7.28         0
7.39         1234

END",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1/1/1/1/1/1","","","ADD","SUCCESSFUL","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.319","00 00:00:00.032","0",
"dn: MSISDN=4919018510112,dc=msisdn,ou=identities,dc=operator,dc=com
changetype: add
objectClass: top
objectClass: alias
objectClass: MSISDN
aliasedObjectName: mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com","",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1/1/1/1/1/1/1","","","ADD","SUCCESSFUL","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.352","00 00:00:00.002","0",
"dn: mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com
changetype: add
objectClass: CUDBMultiServiceConsumer
objectClass: top
mscId: 4622814443244bbba03ffb88f586f4f3","",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1/1/1/1/1/1/1/1","","","ADD","SUCCESSFUL","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.355","00 00:00:00.012","0",
"dn: serv=Identities,mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com
changetype: add
IMSI: 264000000510112
imsiMask: '0000000000000001'B
MSISDN: 4919018510112
msisdnMask: '0000000000000001'B
CDC: 0
objectClass: top
objectClass: CUDBService
objectClass: mscIdentities","",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1/1/1/1/1/1/1/1/1","","","ADD","SUCCESSFUL","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.370","00 00:00:00.002","0",
"dn: serv=AAA,mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com
changetype: add
serv: AAA
objectClass: top
objectClass: CUDBService","",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1/1/1/1/1/1/1/1/1/1","","","ADD","SUCCESSFUL","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.374","00 00:00:00.007","0",
"dn: IMSI=264000000510112,dc=imsi,ou=identities,dc=operator,dc=com
changetype: add
objectClass: top
objectClass: alias
objectClass: IMSI
aliasedObjectName: mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com","",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1/1/1/1/1/1/1/1/1/1/1","","","ADD","SUCCESSFUL","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.383","00 00:00:00.007","0",
"dn: serv=CSPS,mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com
changetype: add
CDC: 7
MSISDN: 491901851011
IMSI: 26400000051011
CSLOC: 2
GSMUEFEAT: 0
DEMLPP: 4
MEMLPP: 1
SOSDCF: 7
OFA: 0
PWD: 4321
objectClass: top
objectClass: CUDBServiceAuxiliary
objectClass: CP1
objectClass: CP2
objectClass: CP3
objectClass: CP4
objectClass: CP5
objectClass: CP6
objectClass: CP7
objectClass: CP8
objectClass: CP9
objectClass: CPA
objectClass: CPB
objectClass: CPC
objectClass: CPD
objectClass: CPE
objectClass: CPF
objectClass: CPG
objectClass: CPH
objectClass: CPI
objectClass: CPJ
objectClass: CPK
objectClass: CPM
objectClass: CPM1
objectClass: CPM2
objectClass: CPM3
objectClass: CPM4
objectClass: CPZ
objectClass: CP04s
objectClass: CP0A
objectClass: CP11
NAM: 0
CAT: 10
DBSG: 1
SOCB: 0
PWDC: 0
SOCFB: 0
SOCFNRC: 0
SOCFNRY: 0
SOCFU: 0
SODCF: 0
SOCLIP: 0
SOCLIR: 0
SOCOLP: 0","",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1","","","ADD","SUCCESSFUL","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.392","00 00:00:00.006","0",
"dn: ei=gprs,serv=CSPS,mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com
changetype: add
objectClass: top
objectClass: alias
objectClass: CUDBExtensibleObject
aliasedObjectName: PDPCP=0,ou=gprsProfiles,serv=CSPS,ou=mscCommonData,dc=operator,dc=com","",""

"southbound","vn-11507091326370003CAI3G1_2","/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1","","","ADD","SUCCESSFUL","",
"vn-1","LDAP","FT_CUDB","2015-07-09 13.26.37.401","00 00:00:00.002","0",
"dn: ei=camel,serv=CSPS,mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com
changetype: add
objectClass: top
objectClass: alias
objectClass: CUDBExtensibleObject
aliasedObjectName: CSP=0,ou=camelProfiles,serv=CSPS,ou=mscCommonData,dc=operator,dc=com","",""

"northbound","vn-11507091326370003CAI3G1_2","","","msisdn=4919018510112:imsi=264000000510112","CREATE",
"SUCCESSFUL","cai3guser","vn-1","CAI3G1_2","Subscription@http://schemas.ericsson.com/pg/hlr/13.5/",
"2015-07-09 13.26.37.226","00 00:00:00.183","0",
"<soapenv:Envelope xmlns:soapenv=""http://schemas.xmlsoap.org/soap/envelope/""
 xmlns:cai3=""http://schemas.ericsson.com/cai3g1.2/"" xmlns:ns=""http://schemas.ericsson.com/pg/hlr/13.5/"">
<soapenv:Header><cai3:SessionId>99901944f90f4cb492c39aeff5e392c3</cai3:SessionId>
</soapenv:Header><soapenv:Body>
      <cai3:Create>
         <cai3:MOType>Subscription@http://schemas.ericsson.com/pg/hlr/13.5/</cai3:MOType>
         <cai3:MOId>
            <ns:msisdn>4919018510112</ns:msisdn>
            <ns:imsi>264000000510112</ns:imsi>
         </cai3:MOId>
         <cai3:MOAttributes>
            <ns:CreateSubscription imsi=""264000000510112"" msisdn=""4919018510112"">
               <ns:msisdn>4919018510112</ns:msisdn>
               <ns:imsi>264000000510112</ns:imsi>
            </ns:CreateSubscription>
         </cai3:MOAttributes>
      </cai3:Create>
   </soapenv:Body></soapenv:Envelope>","<S:Envelope xmlns:S=""http://schemas.xmlsoap.org/soap/envelope/"">
<S:Header><SessionId xmlns=""http://schemas.ericsson.com/cai3g1.2/"">99901944f90f4cb492c39aeff5e392c3
</SessionId><TransactionId xmlns=""http://schemas.ericsson.com/cai3g1.2/""
 xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance"" xsi:nil=""true""/>
<SequenceId xmlns=""http://schemas.ericsson.com/cai3g1.2/""
 xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance"" xsi:nil=""true""/>
</S:Header><S:Body><CreateResponse xmlns=""http://schemas.ericsson.com/cai3g1.2/"">
<MOId><ns:msisdn xmlns:cai3=""http://schemas.ericsson.com/cai3g1.2/"" 
xmlns:ns=""http://schemas.ericsson.com/pg/hlr/13.5/"" 
xmlns:soapenv=""http://schemas.xmlsoap.org/soap/envelope/"">4919018510112</ns:msisdn>
<ns:imsi xmlns:cai3=""http://schemas.ericsson.com/cai3g1.2/""
 xmlns:ns=""http://schemas.ericsson.com/pg/hlr/13.5/""
 xmlns:soapenv=""http://schemas.xmlsoap.org/soap/envelope/"">264000000510112</ns:imsi>
</MOId></CreateResponse></S:Body></S:Envelope>",""


"southbound","vn-11507091326380001REPLAY","/1/1/1/1/1/1/1/1/1/1/1","","","ADD","FAILED","",
"vn-1","LDAP","FT_CUDB_REPLAY","2015-07-09 13.26.38.313","00 00:00:00.001","68",
"dn: MSISDN=4919018510112,dc=msisdn,ou=id
entities,dc=operator,dc=com
changetype: add
objectClass: top
objectClass: alias
objectClass: MSISDN
aliasedObjectName: mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com","
[LDAP: error code 68 - Entry Already Exists]","vn-11507091326370003CAI3G1_2/1/1/1/1/1/1/1/1/1"

"southbound","vn-11507091326380001REPLAY","/1/1/1/1/1/1/1/1/1/1/1/1","","","ADD","FAILED","",
"vn-1","LDAP","FT_CUDB_REPLAY","2015-07-09 13.26.38.315","00 00:00:00.001","68",
"dn: mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com
changetype: add
objectClass: CUDBMultiServiceConsumer
objectClass: top
mscId: 4622814443244bbba03ffb88f586f4f3","[LDAP: error code 68 - Entry Already Exists]",
"vn-11507091326370003CAI3G1_2/1/1/1/1/1/1/1/1/1/1"

"southbound","vn-11507091326380001REPLAY","/1/1/1/1/1/1/1/1/1/1/1/1/1","","","ADD","FAILED","",
"vn-1","LDAP","FT_CUDB_REPLAY","2015-07-09 13.26.38.317","00 00:00:00.001","68",
"dn: serv=Identities,mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com
changetype: add
IMSI: 264000000510112
imsiMask: '0000000000000001'B
MSISDN: 4919018510112
msisdnMask: '0000000000000001'B
CDC: 0
objectClass: top
objectClass: CUDBService
objectClass: mscIdentities","[LDAP: error code 68 - Entry Already Exists]",
"vn-11507091326370003CAI3G1_2/1/1/1/1/1/1/1/1/1/1/1"

"southbound","vn-11507091326380001REPLAY","/1/1/1/1/1/1/1/1/1/1/1/1/1/1","","","ADD","FAILED","",
"vn-1","LDAP","FT_CUDB_REPLAY","2015-07-09 13.26.38.318","00 00:00:00.001","68",
"dn: serv=AAA,mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com
changetype: add
serv: AAA
objectClass: top
objectClass: CUDBService","[LDAP: error code 68 - Entry Already Exists]",
"vn-11507091326370003CAI3G1_2/1/1/1/1/1/1/1/1/1/1/1/1"

"southbound","vn-11507091326380001REPLAY","/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1","","","ADD","FAILED","",
"vn-1","LDAP","FT_CUDB_REPLAY","2015-07-09 13.26.38.320","00 00:00:00.001","68",
"dn: IMSI=264000000510112,dc=imsi,ou=identities,dc=operator,dc=com
changetype: add
objectClass: top
objectClass: alias
objectClass: IMSI
aliasedObjectName: mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com","
[LDAP: error code 68 - Entry Already Exists]",
"vn-11507091326370003CAI3G1_2/1/1/1/1/1/1/1/1/1/1/1/1/1"

"southbound","vn-11507091326380001REPLAY","/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1","","","ADD","FAILED","",
"vn-1","LDAP","FT_CUDB_REPLAY","2015-07-09 13.26.38.323","00 00:00:00.004","68",
"dn: serv=CSPS,mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com
changetype: add
CDC: 7
MSISDN: 491901851011
IMSI: 26400000051011
CSLOC: 2
GSMUEFEAT: 0
DEMLPP: 4
MEMLPP: 1
SOSDCF: 7
OFA: 0
PWD: 4321
objectClass: top
objectClass: CUDBServiceAuxiliary
objectClass: CP1
objectClass: CP2
objectClass: CP3
objectClass: CP4
objectClass: CP5
objectClass: CP6
objectClass: CP7
objectClass: CP8
objectClass: CP9
objectClass: CPA
objectClass: CPB
objectClass: CPC
objectClass: CPD
objectClass: CPE
objectClass: CPF
objectClass: CPG
objectClass: CPH
objectClass: CPI
objectClass: CPJ
objectClass: CPK
objectClass: CPM
objectClass: CPM1
objectClass: CPM2
objectClass: CPM3
objectClass: CPM4
objectClass: CPZ
objectClass: CP04s
objectClass: CP0A
objectClass: CP11
NAM: 0
CAT: 10
DBSG: 1
SOCB: 0
PWDC: 0
SOCFB: 0
SOCFNRC: 0
SOCFNRY: 0
SOCFU: 0
SODCF: 0
SOCLIP: 0
SOCLIR: 0
SOCOLP: 0","[LDAP: error code 68 - Entry Already Exists]",
"vn-11507091326370003CAI3G1_2/1/1/1/1/1/1/1/1/1/1/1/1/1/1"

"southbound","vn-11507091326380001REPLAY","/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1","","","ADD","FAILED","",
"vn-1","LDAP","FT_CUDB_REPLAY","2015-07-09 13.26.38.328","00 00:00:00.001","68",
"dn: ei=gprs,serv=CSPS,mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com
changetype: add
objectClass: top
objectClass: alias
objectClass: CUDBExtensibleObject
aliasedObjectName: PDPCP=0,ou=gprsProfiles,serv=CSPS,ou=mscCommonData,dc=operator,
dc=com","[LDAP: error code 68 - Entry Already Exists]",
"vn-11507091326370003CAI3G1_2/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1"

"southbound","vn-11507091326380001REPLAY","/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1","","","ADD","FAILED","",
"vn-1","LDAP","FT_CUDB_REPLAY","2015-07-09 13.26.38.330","00 00:00:00.001","68",
"dn: ei=camel,serv=CSPS,mscId=4622814443244bbba03ffb88f586f4f3,ou=multiSCs,dc=operator,dc=com
changetype: add
objectClass: top
objectClass: alias
objectClass: CUDBExtensibleObject
aliasedObjectName: CSP=0,ou=camelProfiles,serv=CSPS,ou=mscCommonData,dc=operator,dc=com",
"[LDAP: error code 68 - Entry Already Exists]",
"vn-11507091326370003CAI3G1_2/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1/1"

"northbound","vn-11507091326380001REPLAY","","","timestamp=1436441194","AUTOREPLAY","SUCCESSFUL",
"replayer","vn-1","","","2015-07-09 13.26.38.076","00 00:00:00.299","0",
"REPLAY:STARTTIME=1436441194,ENDTIME=1436441198;","Automatic Replay operation successful",""

2.3.4.1   Cassandra

Processing logs from all nodes are, upon provisioning, saved in the Cassandra cluster. The logs are stored as an aggregated view, to be viewable in the Log Management subtab in Dynamic Activation GUI. Each processing log is replicated into, by default, three copies that are saved on different nodes. The spread algorithm for where processing logs are replicated too, are handle by Cassandra, and thereby not controlled or possible to know from a Dynamic Activation perspective.

As Processing logs are stored, Cassandra determines and discovers how to efficient compress data, by comparing it. This means that Cassandra will compress both new Processing logs and old Processing logs in a more efficient way, after some usage.

2.3.4.2   Flat Processing Log File

The flat processing log files are named as flatlog-<moduleName>.log_<hostName>, for example: flatlog-nbia.log_node, and are stored in the default directory /var/log/dve/flatlog/ on each node. Each module is corresponding to a specialized Log Type, see Table 11.

When the flat processing log file reaches the maximum file size or after a time interval, the file gets rotated and it is backed up to a CSV formatted file with the name <flatlogfileName>_<data>_<time>_<sequenceNumber>.csv, for example: flatlog-nbia.log_node1_20160407_015025_00001.csv. This CSV file will exist until the end of the lifecycle.

The setting of the flat processing log files can be modified in bootloader, see Section 5.12.4 for all the configurable parameters and their default values.

2.4   Ports

This section includes information about the different traffic, and O&M ports used in Dynamic Activation.

2.4.1   Traffic Ports

The following table contains the traffic ports.

(1)  Means all interfaces.

2.4.2   Operation and Maintenance Ports

The Operation and Maintenance ports are described in Table 13:

(1)  Means all interfaces.

(2)  Configuration instructions are described in Section 6.2.1. PL nodes are only reachable from the SC nodes.

(3)  The HTTP traffic is automatically redirected to HTTPS on port 8383 by tomcat.

2.5   Version Identification

This section describes how to retrieve information about the different component versions and also how to check the hardware present in the system.

Information about Customer Adaptations on Dynamic Activation is presented with the Dynamic Activation component versions.

2.5.1   Software Version

Print all the installed Dynamic Activation RPMs.

From any node, locally:

$ sudo 3ppmon version

Print all installed Dynamic Activation components, and Customer Adaptation System Properties.

From an SC node:

$ bootloader.py system version

2.5.2   BSP Version

This section describes how to retrieve information about the installed BSP version.

Login to BSP 8100:

$ ssh -p 2024 advanced@<BSP_NBI_IP>

> show ManagedElement=1,SystemFunctions=1,SwInventory=1

Look for CXP9025735-XXXXX where XXXXX is the BSP version installed.

Exampel:

SwVersion=CXP9025735-R3A05

2.6   Users

This section gives a brief description of the users of Dynamic Activation.

2.6.1   System Users

The following are the default system users for Dynamic Activation:

• actadm - belongs to the group activation and is used for administering Dynamic Activation.
• dvecli - belongs to the group activation and is used for fetching massive result-files.
• casadm - belongs to the group activation and is used for administering the Cassandra database.
• zooadm - belongs to the group activation and owns the ZooKeeper application.
• syncuser - belongs to the group activation and is used for synchronization of license counters and configurations by SFTP.

The system users can be administered with LDE operations. For more information about LDE, see LDE Product Description, Reference [5].

System Handling

The system user actadm is the user that will own most of the processes and files used by Dynamic Activation. For management of the system, the administrative users administrators need to be created, see section Create Administrative User, Section 2.6.1.1. Administrators are members of the activation group, and are allowed to perform operation and maintenance of the system.

Locking Mechanism

Failed login attempts for the whole system users are logged in /var/log/<hostname>/auth on the SC node and in /var/log/auth on each server.

If too many sequential login attempts fail, the user is locked and an event is raised. For more information about alarms and events, refer to Event and Alarm Handling, Reference [7].

To view the failed login attempts status of a user, enter the following command as user root:

# pam_tally2 --file /home/faillog

If a user is locked, unlock the user by entering the following command as user root:

# pam_tally2 --file /home/faillog --user <user_name> --reset

The number of unsuccessful login attempts required before an event is raised can be edited in the files /etc/pam.d/common-auth and /etc/pam.d/common-account

Change the value of the parameter deny=<deny_value>

2.6.1.1   Create Administrative User

This section contains information on how to create non-root users for administering purposes, such as log file reading, process monitoring, managing Dynamic Activation processes, installation of modules, and more.

To create an administrative user, log in as user root on one of the SC nodes and follow the instructions in the step-list:

1. Create an administrative user that belongs to the global group activation:

   # addActivationUser <administrator_username>

2. Set the password and activate the new user:

   # passwd <administrator_username>

2.6.1.2   Delete Administrative User

To delete an administrative user, log in as user root on one of the SC nodes and follow the instructions in the step-list:

1. # userdel -r <administrator_username>
2. Edit the /cluster/etc/login.allow file. Remove the following line:

   <administrator_username> all

   # vi /cluster/etc/login.allow

2.6.2   Dynamic Activation Users

Dynamic Activation users can be administered through the GUI. For the details, refer to User Guide for Resource Activation, Reference [8].

Failed login attempts for Dynamic Activation users are logged in different files, depending on which interface the login attempt was made for, see Section 2.6.2.1.

2.6.2.1   Failed Login Attempts

• Failed login attempts through the MML, CAI, and CAI3G interfaces, are logged in the /var/log/dve/ema-authentication-module-audit.log file, residing on the SC nodes.

  Example output:

  2017-03-28 11:20:12.923 [acs-service-44] INFO AuditLog - [CAI] Login attempt with username [caiuser] has failed 2017-03-28 11:23:13.270 [acs-service-45] INFO AuditLog - [MML] Login attempt with username [mmltest] has failed 2017-03-28 11:23:44.797 [acs-service-45] INFO AuditLog - [CAI3G1_2] Login attempt with username [cai3gtest] has failed

• Failed login attempts through the CLI interface are logged in the /var/log/dve/dve-application-audit.log file, residing on the PL nodes.

  Example output:

  2017-03-30 08:17:00,957 [AUTHENTICATIONLOG] User [clitest] failed to authenticate 2017-03-30 08:17:00,958 [AUTHENTICATIONLOG] loginAttemptsThreshold reached, locking user clitest

• Failed login attempts through the GUI interface are logged in the /var/log/dve/tomcat-server-audit.log file, residing on the SC nodes.

  Example output:

  2017-03-30 08:24:43,779 [AUTHENTICATIONLOG] User [testuser] failed authentication 2017-03-30 08:24:54,767 [AUTHENTICATIONLOG] User [testuser] is locked

The log files above contain, except for information about failed login attempts, also information about user locking when maximal number of consecutive unsuccessful login attempts is reached.

An event is sent from ESA:

• For every failed login attempt
• When a user is locked (due to threshold reached)

2.7   Crontab

Dynamic Activation uses the following crontab jobs for housekeeping purposes:

Note:  

All crontab jobs are applicable for all nodes, both SC and PL, except for sync_license.sh. This crontab job is only applicable on the PL nodes.

• auto_erase.sh - handles certain log and response files.

  For more information, see Section 5.12.5.

• proclog-retain-maximum-days.sh - this script is used to retain the maximum number of days configured. It removes the N+1:th day configured to retain a steady amount of preferred days with logs in the system. If there is more logs than the N+1:th days, they wont be removed until the next run of the script. This script is run hourly and increments per node, and is only executed by the actadm user.

  For more information, see Section 5.12.2.

• sync_license.sh - used for synchronization of license counters. This crontab job is only executed by the user syncuser.

  For more information, see Section 5.17.6.

To list all crontab jobs, run the following command as user root:

# crontab -l -u actadm

3   User Privileges

Certain system administration commands require superuser privileges to run. It is highly recommended to setup system administrator users part of the wheel group, and run such commands with sudo privileges instead of using the root account.

As an example, the pam_tally2 command :

• (High security risk) If logged in as user root:
  CL22-SC-1:~ # pam_tally2 --file /home/faillog
• (Low security risk/Better choice) If logged in as a system administrator:
  <System Admin>@CL22-SC-1:~> sudo pam_tally2 --file /home/faillog

For security reasons, do not use root user for Operations and Management (O&M) of Dynamic Activation. Instead, create Dynamic Activation administrators, that are part of the activation group, to perform O&M operations. See Section 3.1.

It is highly recommended to disable the root user, and replacing this account with one or more System Administrator users that are part of the wheel group. See Section 3.2.

3.1   Administrator in activation Group

The activation group defines trusted system administrators that are allowed to manage Dynamic Activation.

System administrators in the activation group have privileges to perform the following O&M:

• Use all available bootloader commands, for example, manage application modules, set bootloader configuration, and so on.
• Manage 3PPs with 3ppmon commands by using sudo.
• Run scripts, for example, syncuser/sync_ssh_keys, replay-admin-tool, housekeeping, and so on.

Unlike users in the wheel group, the activation users cannot use sudo to execute other commands that require superuser privileges, except the 3ppmon commands.

For instruction on how to create activation users, see Section 2.6.1.1.

3.2   Administrator in wheel Group

System administrators in the wheel group can use sudo to execute all commands that require superuser privileges. They have privileges to perform:

• Initial deployment and installation
• User management of Linux users, for example, create new users, set passwords, and so on.
• Manage License Keys in Sentinel.
• License Agent administration
• Reboot cluster
• Health check (due to accessing system variables)
• Operating System level management

To keep certain operation-procedures consistent in the Dynamic Activation documentation, some commands (that require superuser privileges to run), are specified to be run as user root with # prompt. These commands can instead, and preferably, be executed as a wheel group user by adding sudo prior to the commands.

4   Operations

This section describes system operation commands, such as starting and stopping the system, and how to perform various system checks.

Note:  

All the commands should be used depending on the state of the system. If the system is running traffic, some of the following commands can disturb the provisioning.

4.1   Operation Commands

This section describes the following commands which are used for checking system status and performing system operations.

• 3ppmon commands – Manage 3rd party applications and services that the Dynamic Activation application processes depend on.
• bootloader.py commands – Manage the Dynamic Activation application progresses (also known as modules).
• systemctl commands – Manage system background processes (also known as daemons).

Note:  

The 3ppmon commands must be run with superuser privileges.

Note:  

bootloader.py commands are exclusively run on SC nodes.

(1)  all is only used for new installations. It should not be used in any other cases.

(2)  For these commands to take effect, they need to be activated. A bootloader.py node activate --host <ip/hostname/all> command is required.

(3)  This command will undeploy per jar-file and on one or several hosts depending on parameter input.

(4)  This is used in Customer Adaptation (CA) development.

Note:  

systemctl commands are run locally on every node, and must be run with superuser privileges.

4.2   Dynamic Activation Property Configuration

Dynamic Activation properties are categorized as follows:

• Site-specific properties must be adapted to specific sites (such as IP address), opposed to non-site specific.
• Adaptable properties implies that the customer can change the value at operation time, opposed to non-adaptable.

This implies the following:

• The configuration of non-adaptable site-specific properties is part of the Dynamic Activation installation procedure, handled by the installation scripts.
• Configuration of adaptable properties is performed at operation time, by use of the bootloader.py command.

To configure a property with a customized value, run the following commands from an SC node:

1. To list the adaptable properties:

   $ bootloader.py config list -A

2. Configure the selected adaptable property:

   $ bootloader.py config set --parameter @<Adaptable Property>@ --value <value>

3. Activate the modification for each node, one by one:
   Attention!

   Wait for each node to be activated before starting with the next one, otherwise traffic disturbances occur. The all parameter should only be used when no provisioning traffic is running.

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the node that is to be activated.

To restore a property to its default (initially installed) value, run the following commands from an SC node:

1. To list the adaptable properties:

   $ bootloader.py config list -A

2. Restore the selected adaptable property:

   $ bootloader.py config remove --parameter @<Adaptable Property>@

3. Activate the modification for each node, one by one:
   Attention!

   Wait for each node to be activated before starting with the next one, otherwise traffic disturbances occur. The all parameter should only be used when no provisioning traffic is running.

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the node that is to be activated.

   Note:  

   Repeat the command for each node affected by this change.

   
   

To check which properties that have been customized, run the following command, from an SC node:

$ bootloader.py config list

Configuration attributes for nbia-module:
   No user configuration exist

Configuration attributes for admin-tool:
   No user configuration exist

Configuration attributes for akka-cluster-service-lib:
   No user configuration exist

Configuration attributes for tomcat-server:
   No user configuration exist

Configuration attributes for ema-faultmgmt-module:
   No user configuration exist

Configuration attributes for NCA-Application:
   No user configuration exist

Configuration attributes for ema-authentication-module:
   No user configuration exist

Configuration attributes for dve-application:
   No user configuration exist

Configuration attributes for lockservice-module:
   No user configuration exist

Configuration attributes for activation-orchestration-module:
   No user configuration exist

Configuration attributes for dve-inbound-interfaces-application:
   No user configuration exist

Configuration attributes for jdv-cudb-config-assembly:
   No user configuration exist

Configuration attributes for cudb-block-proxy:
   No user configuration exist

Configuration attributes for arh-module:
   No user configuration exist

Configuration attributes for NotificationRules:
   No user configuration exist

Configuration attributes for UDC-Application:
   No user configuration exist

Configuration attributes for ema-batch-module:
   No user configuration exist

In the example, the cudb-block-proxy module has a customized parameter, @BLOCK_PROXY_SSLCERT_STORE_PASSWORD@

All other modules have default values for their properties.

4.3   Dynamic Activation System Status

The 3ppmon status command is used to view the status for the 3PP processes in all nodes, or the detailed status for one specified node.

The bootloader.py node status command is used to view the status of Dynamic Activation application processes for all nodes, or the status for one specified node.

4.3.1   All Nodes Status for 3PP Processes

Execute the following command on any node to check the deployment status for all 3PPs in all nodes:

$ sudo 3ppmon status --host all

Host  ESAPrcs  LicSrv  ZooKpr  Cassandra ActvAlrm
SC-1  UP        UP      UP      UP        0
SC-2  UP        UP      UP      UP        0
PL-3  UP        -       UP      UP        0
PL-4  UP                        UP        0

Note:  

A question mark (?) indicates unknown status.

4.3.2   Single Node Status for 3PP Processes

Execute the following command to check the details of the 3PP deployment status, locally:

$ sudo 3ppmon status

Execute the following command to check the details of the 3PP deployment status on a specific node, remote:

$ sudo 3ppmon status --host <hostname>

<hostname> is the hostname of the node that is to be checked.

Example output for the Dynamic Activation system:

ESA
   Master agent                   UP
   Fault mgmt agent               UP
   Performance mgmt agent         UP
   SSM Sub agent                  UP
License Server
   Sentinel LM server             UP
ZooKeeper
   ZooKeeper process              UP
Cassandra
   Cassandra process              UP
Active Alarms
   Number of alarms               0
   Alarm list                     - 

(1)  Shown as unavailable, -, for PL nodes

Note:  

A question mark (?) indicates unknown status.

4.3.3   Status of Dynamic Activation Applications

From an SC node, execute the following command to check the status of the Dynamic Activation applications:

$ bootloader.py node status --host all

Host            Module                        Version                Status          Bindings
===========     ===========               ===========               ===========     ===========
SC-1      akka-cluster-service-lib             3.3.1                  Running            OK (2/2)
SC-1      admin-tool                           3.124
SC-1      activation-orchestration-module          2.2                    Running
SC-1      ema-faultmgmt-module                 1.15                   Running            OK (2/2)
SC-1      ema-authentication-module            1.2.3                  Running
SC-1      tomcat-server                        162.23                 Running (18/0)
SC-1      -> DVE-OM-GUI                        162.23
SC-1      -> Log-Backend-Rest                  162.23
SC-1      -> Log-Consolidation-Web             162.23
SC-1      -> LEH-Backend                       162.23
SC-1      -> LooseErrorHandlingWeb             162.23
SC-1      -> BootloaderRest                    162.23
SC-1      -> Queue-Management                  162.23
SC-1      -> Launchpad                         162.23
SC-1      -> OrderManagement                   162.23
SC-1      -> OrderManagement-Backend           162.23
SC-1      -> SyncSingleEntryGuiFrontend        162.23
SC-1      -> SyncSingleEntryGuiBackend         162.23
SC-1      -> replayer                          162.23
SC-1      -> inbound-interfaces-rest           162.23
SC-1      -> Scm-Web                           162.23
SC-1      -> batchhandler-backend              2.4
SC-1      -> batchhandler-frontend             2.4
SC-1      cudb-block-proxy                     2.0.4                  Running
SC-1      lockservice-module                   1.7.10                 Running
SC-1      ema-batch-module                     2.4                    Running

SC-2      akka-cluster-service-lib             3.3.1                  Running            OK (2/2)
SC-2      admin-tool                           3.124
SC-2      activation-orchestration-module          2.2                    Running
SC-2      ema-faultmgmt-module                 1.15                   Running            OK (2/2)
SC-2      ema-authentication-module            1.2.3                  Running
SC-2      tomcat-server                        162.23                 Running (18/0)
SC-2      -> DVE-OM-GUI                        162.23
SC-2      -> Log-Backend-Rest                  162.23
SC-2      -> Log-Consolidation-Web             162.23
SC-2      -> LEH-Backend                       162.23
SC-2      -> LooseErrorHandlingWeb             162.23
SC-2      -> BootloaderRest                    162.23
SC-2      -> Queue-Management                  162.23
SC-2      -> Launchpad                         162.23
SC-2      -> OrderManagement                   162.23
SC-2      -> OrderManagement-Backend           162.23
SC-2      -> SyncSingleEntryGuiFrontend        162.23
SC-2      -> SyncSingleEntryGuiBackend         162.23
SC-2      -> replayer                          162.23
SC-2      -> inbound-interfaces-rest           162.23
SC-2      -> Scm-Web                           162.23
SC-2      -> batchhandler-backend              2.4
SC-2      -> batchhandler-frontend             2.4
SC-2      lockservice-module                   1.7.10                 Running
SC-2      ema-batch-module                     2.4                    Running

PL-3      admin-tool                           3.124
PL-3      nbia-module                          3.124                  Running            OK (2/2)
PL-3      dve-application                      3.124                  Running (67/0)     OK (2/2)
PL-3      -> NCA-Application                   162.23
PL-3      -> jdv-nca-assembly                  162.23
PL-3      -> UDC-Application                   162.23
PL-3      -> jdv-udc-assembly                  162.23
PL-3      -> jdv-cudb-config-assembly          162.23
PL-3      -> NotificationRules                 162.23
PL-3      arh-module                           3.124                  Running            OK (2/2)
PL-3      dve-inbound-interfaces-application   162.23                 Running

PL-4      admin-tool                           3.124
PL-4      nbia-module                          3.124                  Running            OK (2/2)
PL-4      dve-application                      3.124                  Running (67/0)     OK (2/2)
PL-4      -> NCA-Application                   162.23
PL-4      -> jdv-nca-assembly                  162.23
PL-4      -> UDC-Application                   162.23
PL-4      -> jdv-udc-assembly                  162.23
PL-4      -> jdv-cudb-config-assembly          162.23
PL-4      -> NotificationRules                 162.23
PL-4      arh-module                           3.124                  Running            OK (2/2)
PL-4      dve-inbound-interfaces-application   162.23                 Running

4.3.4   Status of License Agent

From an SC node, execute the following command to check the status of the License Agent:

$ sudo -u actadm lastat

Note:  

It can take up to one hour for the License Agent process to reload new or updated license keys. Forced reload of license keys can be achieved by restarting the License Agent process. See the License Agent command listed in Table 16.

Node       ZooKeeper         Sentinel          Instances License  Cores License     Valid to Run      
SC-1       CONNECTED         CONNECTED         INSTALLED          INSTALLED         YES
SC-2       CONNECTED         CONNECTED         INSTALLED          INSTALLED         YES
PL-3       CONNECTED         CONNECTED         INSTALLED          INSTALLED         YES
PL-4       CONNECTED         CONNECTED         INSTALLED          INSTALLED         YES
PL-5       CONNECTED         CONNECTED         INSTALLED          INSTALLED         YES
PL-6       CONNECTED         CONNECTED         INSTALLED          INSTALLED         YES

Node       ZooKeeper         Sentinel          Instances License  Cores License     Valid to Run      
SC-1       DISCONNECTED      DISCONNECTED      NOT INSTALLED      NOT INSTALLED     NO
SC-2       DISCONNECTED      DISCONNECTED      NOT INSTALLED      NOT INSTALLED     NO
PL-3       DISCONNECTED      DISCONNECTED      NOT INSTALLED      NOT INSTALLED     NO
PL-4       DISCONNECTED      DISCONNECTED      NOT INSTALLED      NOT INSTALLED     NO
PL-5       -                 -                 -                  -                 -
PL-6       DISCONNECTED      DISCONNECTED      NOT INSTALLED      NOT INSTALLED     NO

Required license key [FAT1023848/1] is missing. Retry after installing the required key.
Required license key [FAT1023848/3] is missing. Retry after installing the required key.

lastat: Error occurred while reading status from one or more nodes. Make sure License Agent service is running properly on all nodes.

4.4   GEP Blade Management

To perform a controlled shutdown or startup of a blade, follow the subchapters in this section.

Note:  

Only one node at a time can be down, for a maximum period of three hours. Otherwise consistency in the Cassandra database will be impacted.

4.4.1   Blade Power Off

To power off the blade, execute the following command on the current node:

$ sudo systemctl poweroff

4.4.2   Blade Startup

To turn on the power of a GEP blade, follow the instructions in the step-list below:

Note:  

Due to the sudo systemctl poweroff command previously executed, it is necessary to first lock the node to unlock it.

1. Login with the following command:

   $ ssh -p 2024 advanced@<BSP_NBI_IP>

2. Enter configuration mode, lock the blade, and commit the changes:

   > configure

   (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position of the GEP blade>,Blade=1,administrativeState=LOCKED

   (config)> commit -s

3. Unlock the blade, and commit the changes:

   (config)> ManagedElement=1,Equipment=1,Shelf=0,Slot=<slot position of the GEP blade>,Blade=1,administrativeState=UNLOCKED

   (config)> commit -s

4.5   Controlled Blade Reboot

To perform a controlled reboot of a blade, execute the following command, as user root, from any of the SC nodes:

# cluster reboot -n <nodeId>

4.6   Enabling and Disabling Provisioning Traffic and CAI3G Test Traffic

This section describes how to enable and disable the provisioning traffic, and CAI3G test traffic.

4.6.1   Enabling and Disabling Provisioning Traffic

It is possible to enable and disable provisioning (CAI3G, MML, CAI) traffic on Dynamic Activation. This, to allow maintenance to be performed on a traffic handling PL node, without interfering with live traffic on the Dynamic Activation cluster.

To disable traffic, execute the following commands:

1. From an SC node, run the following command to disable traffic in the cluster:

   $ bootloader.py config set --parameter @REGISTER_SERVICES@ --value false

2. From an SC node, run the following command to activate the configuration:

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the node that is to be activated.

To enable traffic, execute the following commands:

1. From an SC node, run the following command to enable traffic in the cluster:

   $ bootloader.py config set --parameter @REGISTER_SERVICES@ --value true

2. From an SC node, run the following command to activate the configuration:

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the node that is to be activated.

4.6.2   Enabling and Disabling CAI3G Test Traffic Ports and Provisioning Traffic

It is possible to enable and disable CAI3G test traffic ports, over HTTP/HTTPS separated through test ports, using different port numbers, see Table 12.

The test traffic is disabled by default. For a test scenario, one of the SC nodes, and the PL node to test with CAI3G test traffic, is switched into test traffic mode.

To disable provisioning traffic and enable test traffic ports on both nodes, execute the following commands:

1. Test traffic can only be handled by one PL node at a time. Before switching a PL node into test traffic mode, run the following command on an SC node to check the status of the Dynamic Activation applications, to make sure that there is no other PL node running in test mode.

   $ bootloader.py node status --host all

2. From an SC node, run the following command to disable traffic on both nodes:

   $ bootloader.py config set --parameter @REGISTER_SERVICES@ --value false

3. Enable CAI3G test traffic:

   $ bootloader.py config set --parameter @REGISTER_TEST_SERVICES@ --value true

4. From an SC node, run the following commands to activate the configurations:

   $ bootloader.py node activate --host <hostname of an SC-node>

   $ bootloader.py node activate --host <hostname of a PL-node>

5. Run test traffic on test ports 8888 or 8989 (secure).

To enable provisioning traffic and disable CAI3G test traffic ports on the nodes, execute the following commands:

Note:  

This is performed once the test is finished and the nodes previously set to test traffic mode, is to resume live traffic.

1. From an SC node, run the following commands to disable the CAI3G test traffic mode and enable provisioning traffic:

   $ bootloader.py config remove --parameter @REGISTER_TEST_SERVICES@

   $ bootloader.py config remove --parameter @REGISTER_SERVICES@

2. From an SC node, run the following commands to activate the configurations:

   $ bootloader.py node activate --host <hostname of an SC-node>

   $ bootloader.py node activate --host <hostname of a PL-node>

4.7   Starting and Stopping

The commands for starting and stopping processes in the Dynamic Activation system are listed in the following sections.

The different parts of the application-dependent processes are listed in separate subsections.

Dynamic Activation module processes and 3PP processes are supervised by SSM. If any of the processes go down, SSM starts it. The process is added or removed automatically in SSM, when the following commands are used.

Note:  

Stopping processes by using these commands do not trigger any alarms.

4.7.1   Dynamic Activation Application Processes

Starting the Dynamic Activation Application Processes

Note:  

Before starting the application process, ensure that the application-dependent 3PP processes (such as Cassandra or Zookeeper) are started first.

For instructions, see Section 4.7.2.

1. Perform the following command to start the Dynamic Activation application processes on a complete system:

   $ bootloader.py node start --host all

2. Perform the following command to start the Dynamic Activation application processes on a specific node <hostname>:

   $ bootloader.py node start --host <hostname>

3. Verify the status of all the nodes.

   To check the status for all nodes, see Section 4.3.

Stopping the Dynamic Activation Application Processes

1. Perform the following command to stop the Dynamic Activation application processes on a complete system:

   $ bootloader.py node stop --host all

2. Perform the following command to stop the Dynamic Activation application processes on a specific node <hostname>:

   $ bootloader.py node stop --host <hostname>

3. Verify the status of all the nodes.

   To check the status for all nodes, see Section 4.3.

4.7.2   Starting and Stopping 3PP Processes

This section covers how to start and stop 3PP Processes.

The commands in this chapter apply only for the node they are executed on.

Note:  

Before stopping an application-dependent 3PP process (such as Cassandra or Zookeeper), ensure that the Dynamic Activation application process is stopped first.

For an instruction, see Section 4.7.1.

4.7.2.1   Sentinel

This section covers how to start and stop Sentinel RMS. Applies only for SC nodes.

Starting Sentinel RMS (License Server)

To start the license server on an SC node, execute the following command:

$ sudo 3ppmon startlserv

Stopping Sentinel RMS (License Server)

To stop the license server on an SC node, execute the following command:

$ sudo 3ppmon stoplserv

4.7.2.2   Ericsson SNMP Agent

This section covers how to start and stop Ericsson SNMP Agent (ESA) and the System Service Monitors (SSM) agent (comes with ESA).

Starting Ericsson SNMP Agent

To start ESA, execute the following command:

$ sudo 3ppmon startesa

Stopping Ericsson SNMP Agent

To stop ESA, execute the following command:

$ sudo 3ppmon stopesa

Starting SSM

To start the SSM agent, execute the following command:

$ sudo 3ppmon startssm

Stopping SSM

To stop the SSM agent, execute the following command:

$ sudo 3ppmon stopssm

4.7.2.3   ZooKeeper

This section covers how to start and stop the ZooKeeper service.

Note:  

These operations can only be performed on a ZooKeeper node. For an overview of which nodes that have ZooKeeper installed, see Section 2.

Starting ZooKeeper

To start ZooKeeper, execute the following command:

$ sudo 3ppmon startzookeeper

Stopping ZooKeeper

To stop ZooKeeper, execute the following command:

$ sudo 3ppmon stopzookeeper

4.7.2.4   Cassandra

This section covers how to start and stop the Cassandra processes.

Starting Cassandra

To start Cassandra, execute the following command:

$ sudo 3ppmon startcassandra

Stopping Cassandra

To stop Cassandra, execute the following command:

$ sudo 3ppmon stopcassandra

4.7.3   Dynamic Activation Modules

This section covers how to start and stop Dynamic Activation modules.

For modules possible to start and stop on the SC, and PL nodes respectively, see Example 5 in Section 4.3.3.

Note:  

The status-output shows both modules and sub-modules. Sub-modules start with ->.

Starting Dynamic Activation Modules

A module can be started on a specific node, or on all nodes. To start one or all Dynamic Activation modules, execute the following command:

$ bootloader.py module start -n <Module> --host <hostname> | all

Stopping Dynamic Activation Modules

A module can be stopped on a specific node, or on all nodes. To stop one or all Dynamic Activation modules, execute the following command:

$ bootloader.py module stop -n <Module> --host <hostname> | all

4.7.4   Increasing Startup Time-out

When starting a system with many network elements, the startup sequence may fail with a time-out.

To increase the time-out value, execute the following commands:

1. From an SC node, run the following command to disable traffic in the cluster:

   $ bootloader.py config set --parameter @timeout@ --value <timeout_value>

   Note:  

   The <timeout_value> must be lower than four minutes.

   
   
2. From an SC node, run the following command to activate the configuration:

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the node that is to be activated.

4.8   System Checks

This section describes system tools and other utility programs used during problem tracing and for maintaining and verifying the Dynamic Activation system. It also describes how to perform a health check on the system.

4.8.1   Health Check on Dynamic Activation

Health checks are automatically run during installation, update and upgrade, and checks the following:

• Node status of Cassandra.
• The ownership of drbd on SC1 and SC2.
• Node status of Zookeeper
• Status of Sentinel
• The ESA cluster status.
• Active ESA alarms.
• Dynamic Activation status on all nodes.
• The eVIP interfaces alb_0 and alb_1 are shown on all nodes.
• Active cluster alarms

Note:  

The automatic health-check tool does not check the Network Setup for GEP3 or GEP5, if unclear provisioning issues exist or the /var/log/messages file for recent error messages.

It is possible to manually run a system health check on any occasion.

The following health-check options are available:

The step-list below shows an example of a manually run system health-check:

Note:  

All health-check commands need to be run as user root, and can only be executed on SC1 or SC2.

1. Verify that the system status:

   # healthcheck.py run -n <task or group of tasks to be executed>

   Example:

   # healthcheck.py run -n Activation_all_Native

   Output Example:

   ============================================== Name: Activation_all_Native Description: Verify native system status is OK Result stored as: 2017-05-08_14-20-54 ============================================== activation_cassandra PASS activation_drbd PASS activation_esa_cluster PASS activation_esa_alarms PASS activation_lde_cluster PASS activation_zookeeper PASS activation_sentinel PASS activation_network PASS activation_application PASS activation_esa PASS

   To get a list of all possible <task or group of tasks to be executed> options, run the following command:

   # healthcheck.py get

   Output:

   Name Description ====================== ======================================================================== Activation_all_Native Verify native system status is OK Activation_all_Virtual Verify virtual system status is OK activation_application Check that all modules is up and running activation_cassandra Check that cassandra is up and running properly activation_drbd Check ownership of drbd is correct (only for native) activation_esa Check that esa is up and running properly activation_esa_alarms Check if esa alarms exist activation_esa_cluster Check esa cluster status is correct activation_lde_cluster Check lde cluster status (only for native) activation_network Check that the network (evip) is up and running properly (only for native) activation_platform Check that all 3pp's is up and running activation_sentinel Check that sentinel is up and running properly activation_zookeeper Check that zookeeper is up and running properly

2. Run the following command to get detailed information on the health-check:

   # healthcheck.py show -d

   Output Example:

   ============================================== Name: Activation_all_Native Description: Verify native system status is OK Result stored as: 2017-05-08_14-20-54 ============================================== activation_lde_cluster PASS Description : Check lde cluster status (only for native) Messages : LDE cluster verified ok activation_esa_cluster PASS Description : Check esa cluster status is correct Messages : Esa cluster verified ok activation_network PASS Description : Check that the network (evip) is up and running properly (only for native) Messages : evip interface is set up properly on all nodes activation_esa PASS Description : Check that esa is up and running properly Messages : Esa is up and running on all nodes activation_cassandra PASS Description : Check that cassandra is up and running properly Messages : Cassandra is up and running on all nodes activation_esa_alarms PASS Description : Check if esa alarms exist Messages : Active alarms: activation_drbd PASS Description : Check ownership of drbd is correct (only for native) Messages : drbd verified OK activation_application PASS Description : Check that all modules is up and running Messages : All modules is up and running properly on all nodes activation_zookeeper PASS Description : Check that zookeeper is up and running properly Messages : Zookeeper is up and running on all nodes activation_sentinel PASS Description : Check that sentinel is up and running properly Messages : Sentinel is up and running on all nodes

   Note:  

   If any of the statuses are not shown as PASS, check the logs in /var/log/healthcheck/ directory.

   Health-check logs store the executing flow of each task. Check the corresponding log if one or more tasks failed during the health-check.

   The healthcheck.log file stores general information of the health-check execution flow.

   
   
3. Make sure that no unclear provisioning issues exist.

   Note:  

   This step is only applicable on a running system.

   
   

   Check point: Check Dynamic Activation provisioning logs for any unclear provisioning issues, for example partly executed logs, see Section 2.3.2.

4. Check the /var/log/messages file for recent error messages, all nodes:

   Note:  

   This step is only applicable on a running system.

   
   

   # cat /var/log/messages

   Check point: No error messages should exist.

   Note:  

   For a full history of error messages, check the log files according to the Section 2.3.1.

   
   

4.8.2   Network Setup Check

This section contains information on how to check the network setup for GEP3 or GEP5.

1. Login from any SC node:

   # ssh -p 2024 advanced@<BSP_NBI_IP>

2. Enter configure mode:

   > configure

3. Set up SSH value to troubleshootingMode attribute:

   (config)> ManagedElement=1,DmxcFunction=1,SystemInformation=1,troubleshootingMode=ssh

   Note:  

   BSP allows opening of port 22 to enable a login to the BSP node by use of SSH over the NBI network. The port is automatically closed after 5 minutes and no new SSH sessions can be opened. Ongoing SSH sessions will be kept.

   
   
4. Commit changes and exit configure mode:

   (config)> commit -s

   (config)> end

   > exit

5. From any SC node login to the NBI network:

   # ssh advanced@<BSP_NBI_IP>

6. Login to CMX-0-26:

   advanced@SC-2-2:~# ssh advanced@169.254.0.2

7. Run imish to connect to zebOS:

   advanced@blade_0_26:~# imish

8. Login to virtual router om_cn_sp on CMX-0-26:

   blade_0_26> enable

   blade_0_26# login virtual-router om_cn_sp

   blade_0_26> enable

9. Execute the following ping commands on virtual router om_cn_sp on CMX-0-26:

   VRRP:

   # ping <OM_CN_SP_VRRP_GW_IP>

   BFD:

   # ping <OM_CN_SP_CMX_0_26_CE0_GW_IP>

   # ping <OM_CN_SP_CMX_0_28_CE1_GW_IP>

   Check point: Verify that the ping is successful.

10. Verify VRRP in om_cn_sp.

    Verify that VRRP is master on CMX-0-26 by using command:

    # show vrrp

    Note:  

    If using BFD setup, only vlan1.4054 and vlan1.184 is visible.

    
    

    Example printout for vlan1.1162:

    Address family IPv4 VRRP Id: 20 on interface: vlan1.1162 State: AdminUp - Master Virtual Router MAC Address: 00-00-5E-00-01-14 Number of virtual ip addresses configured: 1 Virtual IP address: 10.44.166.92 Not-owner Operations Primary IP address: 10.44.166.93 Operations Master IP address: 10.44.166.93 Priority is 253 Preempt Hold time: 90000 msec Advertisement interval: 30 centi sec Master Advertisement interval: 30 centi sec Skew time: 0 centi sec Accept mode: FALSE Preempt mode: TRUE Multicast membership on IPv4 interface vlan1.1162: JOINED

    Example printout for vlan1.4054:

    Address family IPv4 VRRP Id: 40 on interface: vlan1.4054 State: AdminUp - Master Virtual Router MAC Address: 00-00-5E-00-01-28 Number of virtual ip addresses configured: 1 Virtual IP address: 10.44.178.193 Not-owner Operations Primary IP address: 10.44.178.194 Operations Master IP address: 10.44.178.194 Priority is 253 Preempt Hold time: 90000 msec Advertisement interval: 30 centi sec Master Advertisement interval: 30 centi sec Skew time: 0 centi sec Accept mode: FALSE Preempt mode: TRUE Multicast membership on IPv4 interface vlan1.4054: JOINED

    Example printout for vlan1.184:

    Address family IPv4 VRRP Id: 50 on interface: vlan1.184 State: AdminUp - Master Virtual Router MAC Address: 00-00-5E-00-01-32 Number of virtual ip addresses configured: 1 Virtual IP address: 10.44.186.65 Not-owner Operations Primary IP address: 10.44.186.66 Operations Master IP address: 10.44.186.66 Priority is 253 Preempt Hold time: 90000 msec Advertisement interval: 30 centi sec Master Advertisement interval: 30 centi sec Skew time: 0 centi sec Accept mode: FALSE Preempt mode: TRUE Multicast membership on IPv4 interface vlan1.184: JOINED

    Check point: Verify that the printout matches the example printout above.

11. Verify BFD in om_cn_sp:

    # show bfd session

    Example printout:

    Sess-Idx Remote-Disc Lower-Layer Sess-Type Sess-State UP-Time Remote-Addr 6 8 IPv4 Single-Hop Up 16:14:25 192.168.209.10/32 11 3 IPv4 Single-Hop Up 16:14:36 192.168.100.2/32 12 2 IPv4 Single-Hop Up 16:05:05 192.168.100.3/32 13 2 IPv4 Single-Hop Up 16:05:05 192.168.100.4/32 Number of Sessions: 4

    Check point: Verify that the printout matches the example printout above.

    If BFD is used to external network, the following printout is also visible:

    Sess-Idx Remote-Disc Lower-Layer Sess-Type Sess-State UP-Time Remote-Addr 5 21 IPv4 Single-Hop Up 19:04:06 10.44.166.73/32 6 21 IPv4 Single-Hop Up 19:04:05 10.44.166.74/32

12. Verify that three OSPF neighbors are visible for VLAN 204 in om_cn_sp:

    # show ip ospf neighbor

    Example printout:

    Neighbor ID Pri State Dead Time Address Interface Instance ID 192.168.208.2 252 Full/Backup 00:00:08 192.168.209.10 vlan1.30 0 192.168.208.2 252 Full/Backup 00:00:08 192.168.100.2 vlan1.204 0 192.168.100.3 0 Full/DROther 00:00:06 192.168.100.3 vlan1.204 0 192.168.100.4 0 Full/DROther 00:00:06 192.168.100.4 vlan1.204 0

    Check point: All OSPF areas should be in state Full.

13. Verify that the VIP-OAM-IP address is advertised in om_cn_sp:

    # show ip route <VIP-OAM-IP>

    Example:

    # show ip route 10.64.26.246

    Printout:

    Routing entry for 10.64.26.246/32 Known via "static", distance 130, metric 0, tag 0 192.168.209.10, via vlan1.30 Routing entry for 10.64.26.246/32 Known via "ospf", distance 110, metric 20, tag 0, best Last update 00:33:17 ago * 192.168.100.3, via vlan1.204 * 192.168.100.4, via vlan1.204

    Check point: Verify that the printout matches the example printout above.

14. Exit virtual router om_cn_sp:

    blade_0_26# exit

15. Login to virtual router prov_cn_sp on CMX-0-26:

    blade_0_26# login virtual-router prov_cn_sp

    blade_0_26> enable

16. Execute the following ping commands on virtual router prov_cn_sp on CMX-0-26:

    VRRP:

    # ping <PROV_CN_SP_VRRP_GW_IP>

    BFD:

    # ping <PROV_CN_SP_CMX_0_26_CE0_GW_IP>

    # ping <PROV_CN_SP_CMX_0_28_CE1_GW_IP>

    Check point: Verify that the ping is successful.

17. Verify VRRP in prov_cn_sp.

    Verify that VRRP is master on CMX-0-26 by using command:

    # show vrrp

    Note:  

    If using BFD setup, no vlan is shown.

    
    

    Example printout:

    Address family IPv4 VRRP Id: 10 on interface: vlan1.1161 State: AdminUp - Master Virtual Router MAC Address: 00-00-5E-00-01-0a Number of virtual ip addresses configured: 1 Virtual IP address: 10.44.166.84 Not-owner Operations Primary IP address: 10.44.166.85 Operations Master IP address: 10.44.166.85 Priority is 253 Preempt Hold time: 90000 msec Advertisement interval: 30 centi sec Master Advertisement interval: 30 centi sec Skew time: 0 centi sec Accept mode: FALSE Preempt mode: TRUE Multicast membership on IPv4 interface vlan1.1161: JOINED

    Check point: Verify that the printout matches the example printout above.

18. Verify BFD in prov_cn_sp:

    # show bfd session

    Example printout:

    Sess-Idx Remote-Disc Lower-Layer Sess-Type Sess-State UP-Time Remote-Addr 9 11 IPv4 Single-Hop Up 16:33:19 192.168.208.10/32 10 1 IPv4 Single-Hop Up 16:33:43 192.168.101.2/32 14 1 IPv4 Single-Hop Up 16:23:23 192.168.101.3/32 15 2 IPv4 Single-Hop Up 16:23:15 192.168.101.4/32 Number of Sessions: 4

    Check point: Verify that the printout matches the example printout above.

    If BFD is used to external network, the following printout is also visible:

    Sess-Idx Remote-Disc Lower-Layer Sess-Type Sess-State UP-Time Remote-Addr 8 19 IPv4 Single-Hop Up 19:13:04 10.44.166.65/32 9 19 IPv4 Single-Hop Up 19:13:04 10.44.166.66/32

19. Verify that three OSPF neighbors are visible for VLAN 104 in prov_cn_sp:

    # show ip ospf neighbor

    Example printout:

    OSPF process 1: Neighbor ID Pri State Dead Time Address Interface Instance ID 192.168.208.14 252 Full/Backup 00:00:08 192.168.208.10 vlan1.60 0 192.168.208.14 252 Full/Backup 00:00:08 192.168.101.2 vlan1.104 0 192.168.101.3 0 Full/DROther 00:00:08 192.168.101.3 vlan1.104 0 192.168.101.4 0 Full/DROther 00:00:09 192.168.101.4 vlan1.104 0

    Check point: All OSPF areas should be in state Full.

20. Verify that the VIP-TRAFFIC-IP address is advertised in prov_cn_sp:

    # show ip route <VIP-TRAFFIC-IP>

    Example:

    # show ip route 10.64.26.245

    Printout:

    Routing entry for 10.64.26.245/32 Known via "static", distance 130, metric 0, tag 0 192.168.208.10, via vlan1.60 Routing entry for 10.64.26.245/32 Known via "ospf", distance 110, metric 20, tag 0, best Last update 00:39:41 ago * 192.168.101.3, via vlan1.104 * 192.168.101.4, via vlan1.104

    Check point: Verify that the printout matches the example printout above.

21. Logout from CMX-0-26:

    blade_0_26# exit

    blade_0_26# exit

    advanced@blade_0_26:~# exit

22. Login to CMX-0-28:

    advanced@SC-2-2:~# ssh advanced@169.254.0.4

23. Run imish to connect to zebOS:

    advanced@blade_0_28:~# imish

24. Login to virtual router om_cn_sp on CMX-0-28:

    blade_0_28> enable

    blade_0_28# login virtual-router om_cn_sp

    blade_0_28> enable

25. Execute the following ping commands on virtual router om_cn_sp on CMX-0-28:

    VRRP:

    # ping <OM_CN_SP_VRRP_GW_IP>

    BFD:

    # ping <OM_CN_SP_CMX_0_26_CE0_GW_IP>

    # ping <OM_CN_SP_CMX_0_28_CE1_GW_IP>

    Check point: Verify that the ping is successful.

26. Verify VRRP in om_cn_sp.

    Verify that VRRP is backup on CMX-0-28 by using command:

    # show vrrp

    Note:  

    If using BFD setup, only vlan1.4054 and vlan1.184 is visible.

    
    

    Example printout for vlan1.1162:

    Address family IPv4 VRRP Id: 20 on interface: vlan1.1162 State: AdminUp - Backup Virtual Router MAC Address: 00-00-5E-00-01-14 Number of virtual ip addresses configured: 1 Virtual IP address: 10.44.166.92 Not-owner Operations Primary IP address: 10.44.166.94 Operations Master IP address: 10.44.166.93 Priority is 252 Advertisement interval: 30 centi sec Master Advertisement interval: 30 centi sec Skew time: 0 centi sec Accept mode: FALSE Preempt mode: TRUE Multicast membership on IPv4 interface vlan1.1162: JOINED

    Example printout for vlan1.4054:

    Address family IPv4 VRRP Id: 40 on interface: vlan1.4054 State: AdminUp - Backup Virtual Router MAC Address: 00-00-5E-00-01-28 Number of virtual ip addresses configured: 1 Virtual IP address: 10.44.178.193 Not-owner Operations Primary IP address: 10.44.178.195 Operations Master IP address: 10.44.178.194 Priority is 252 Advertisement interval: 30 centi sec Master Advertisement interval: 30 centi sec Skew time: 0 centi sec Accept mode: FALSE Preempt mode: TRUE Multicast membership on IPv4 interface vlan1.4054: JOINED

    Example printout for vlan1.184:

    Address family IPv4 VRRP Id: 50 on interface: vlan1.184 State: AdminUp - Backup Virtual Router MAC Address: 00-00-5E-00-01-32 Number of virtual ip addresses configured: 1 Virtual IP address: 10.44.186.65 Not-owner Operations Primary IP address: 10.44.186.67 Operations Master IP address: 10.44.186.66 Priority is 252 Advertisement interval: 30 centi sec Master Advertisement interval: 30 centi sec Skew time: 0 centi sec Accept mode: FALSE Preempt mode: TRUE Multicast membership on IPv4 interface vlan1.184: JOINED

    Check point: Verify that the printout matches the example printout above.

27. Verify BFD in om_cn_sp:

    # show bfd session

    Example printout:

    Sess-Idx Remote-Disc Lower-Layer Sess-Type Sess-State UP-Time Remote-Addr 8 6 IPv4 Single-Hop Up 16:38:12 192.168.209.9/32 14 1 IPv4 Single-Hop Up 16:28:52 192.168.100.3/32 15 1 IPv4 Single-Hop Up 16:28:52 192.168.100.4/32 3 11 IPv4 Single-Hop Up 16:38:23 192.168.100.1/32 Number of Sessions: 4

    Check point: Verify that the printout matches the example printout above.

    If BFD is used to external network, the following printout is also visible:

    Sess-Idx Remote-Disc Lower-Layer Sess-Type Sess-State UP-Time Remote-Addr 4 22 IPv4 Single-Hop Up 19:13:09 10.44.166.74/32 5 22 IPv4 Single-Hop Up 19:13:09 10.44.166.73/32

28. Verify that three OSPF neighbors are visible for VLAN 204 in om_cn_sp:

    # show ip ospf neighbor

    Example printout:

    OSPF process 1: Neighbor ID Pri State Dead Time Address Interface Instance ID 192.168.208.1 253 Full/DR 00:00:07 192.168.209.9 vlan1.30 0 192.168.208.1 253 Full/DR 00:00:07 192.168.100.1 vlan1.204 0 192.168.100.3 0 Full/DROther 00:00:07 192.168.100.3 vlan1.204 0 192.168.100.4 0 Full/DROther 00:00:07 192.168.100.4 vlan1.204 0

    Check point: All OSPF areas should be in state Full.

29. Verify that the VIP-OAM-IP address is advertised in om_cn_sp:

    # show ip route <VIP-OAM-IP>

    Example:

    # show ip route 10.64.26.246

    Printout:

    Routing entry for 10.64.26.246/32 Known via "ospf", distance 110, metric 20, tag 0, best Last update 00:40:58 ago * 192.168.100.4, via vlan1.204 * 192.168.100.3, via vlan1.204 Routing entry for 10.64.26.246/32 Known via "static", distance 130, metric 0, tag 0 192.168.209.9, via vlan1.30

    Check point: Verify that the printout matches the example printout above.

30. Exit virtual router om_cn_sp:

    blade_0_28# exit

31. Login to virtual-router prov_cn_sp:

    blade_0_28# login virtual-router prov_cn_sp

    blade_0_28> enable

32. Execute the following ping commands on virtual router prov_cn_sp on CMX-0-28:

    VRRP:

    # ping <PROV_CN_SP_VRRP_GW_IP>

    BFD:

    # ping <PROV_CN_SP_CMX_0_28_CE1_GW_IP>

    # ping <PROV_CN_SP_CMX_0_26_CE0_GW_IP>

    Check point: Verify that the ping is successful.

33. Verify VRRP in prov_cn_sp.

    Verify that VRRP is backup on CMX-0-28 by using command:

    # show vrrp

    Note:  

    If using BFD setup, no vlan is shown.

    
    

    Example printout:

    Address family IPv4 VRRP Id: 10 on interface: vlan1.1161 State: AdminUp - Backup Virtual Router MAC Address: 00-00-5E-00-01-0a Number of virtual ip addresses configured: 1 Virtual IP address: 10.44.166.84 Not-owner Operations Primary IP address: 10.44.166.86 Operations Master IP address: 10.44.166.85 Priority is 252 Advertisement interval: 30 centi sec Master Advertisement interval: 30 centi sec Skew time: 0 centi sec Accept mode: FALSE Preempt mode: TRUE Multicast membership on IPv4 interface vlan1.1161: JOINED

    Check point: Verify that the printout matches the example printout above.

34. Verify BFD in prov_cn_sp:

    # show bfd session

    Example printout:

    Sess-Idx Remote-Disc Lower-Layer Sess-Type Sess-State UP-Time Remote-Addr 1 10 IPv4 Single-Hop Up 16:44:16 192.168.101.1/32 11 9 IPv4 Single-Hop Up 16:43:53 192.168.208.9/32 16 2 IPv4 Single-Hop Up 16:33:57 192.168.101.3/32 17 1 IPv4 Single-Hop Up 16:33:48 192.168.101.4/32 Number of Sessions: 4

    Check point: Verify that the printout matches the example printout above.

    If BFD is used to external network, the following printout is also visible:

    Sess-Idx Remote-Disc Lower-Layer Sess-Type Sess-State UP-Time Remote-Addr 7 20 IPv4 Single-Hop Up 19:15:43 10.44.166.66/32 8 20 IPv4 Single-Hop Up 19:15:43 10.44.166.65/32

35. Verify that three OSPF neighbors are visible for VLAN 104 in prov_cn_sp:

    # show ip ospf neighbor

    Example printout:

    OSPF process 1: Neighbor ID Pri State Dead Time Address Interface Instance ID 192.168.208.13 253 Full/DR 00:00:07 192.168.208.9 vlan1.60 0 192.168.208.13 253 Full/DR 00:00:07 192.168.101.1 vlan1.104 0 192.168.101.3 0 Full/DROther 00:00:06 192.168.101.3 vlan1.104 0 192.168.101.4 0 Full/DROther 00:00:07 192.168.101.4 vlan1.104 0

    Check point: All OSPF areas should be in state Full.

36. Verify that the VIP-TRAFFIC-IP address is advertised in prov_cn_sp:

    # show ip route <VIP-TRAFFIC-IP>

    Example:

    # show ip route 10.64.26.245

    Printout:

    Routing entry for 10.64.26.245/32 Known via "static", distance 130, metric 0, tag 0 192.168.208.9, via vlan1.60 Routing entry for 10.64.26.245/32 Known via "ospf", distance 110, metric 20, tag 0, best Last update 00:43:33 ago * 192.168.101.4, via vlan1.104 * 192.168.101.3, via vlan1.104

    Check point: Verify that the printout matches the example printout above.

37. Logout from CMX-0-28 and SSH on port 22:

    blade_0_28# exit

    blade_0_28# exit

    advanced@blade_0_28:~# exit

    advanced@SC-2-2:~# exit

4.8.3   Maintaining SSL Certificates

Certificates for SSL verification are always time limited. Check the validity of all installed SSL certificates.

On the first SC node (SC-1), go to /home/bootloader/ssl and execute the following command as an administrator user:

$ /usr/java/latest/bin/keytool -list -v -keystore <key_filename> -storepass <password>

If necessary, configure new certificates. Refer to Section 5.5, and Section 6.2.1 for instructions.

4.9   Changing Timezone

When changing the timezone on a fully installed EDA system, the whole cluster needs to be rebooted. This to get full functionality on the crontab jobs. Crontab jobs must run on local time.

Follow the step-list:

1. Edit the cluster.conf file, residing in the /cluster/etc directory, and change the timezone parameter to the new timezone:

   # vi /cluster/etc/cluster.conf

2. Reload the cluster config:

   # cluster config -a -r

3. Reboot the cluster:

   # cluster reboot -a

   Note:  

   During reboot, there will be traffic disturbance.

   
   

5   System Administration

This section holds information on how to administer different parts of the system.

5.1   eVIP Cluster Configuration Administration

This section describes how to edit the eVIP configuration.

5.1.1   eVIP Configuration File

When the eVIP cluster is started, it uses the data in the evip.xml file that is located in the /cluster/storage/system/config/evip-apr9010467 directory. The eVIP configuration is composed of configuration elements that can be edited through a CLI.

5.1.2   Login to the eVIP CLI

To access the eVIP CLI, run the following command from SC1:

$ telnet `/opt/vip/bin/getactivecontrol` 25190

For information on how to change parameters through the CLI, refer to the eVIP on LSB Management Guide, Reference [6].

5.1.3   Logging

The eVIP component logs events to the system log, see Section 2.3.1 for details on how to find the system log output.

5.2   Configuring ESA

This section describes the configuration in ESA.

5.2.1   Alarms

The Ericsson SNMP Agent (ESA) manages the alarm handling in Dynamic Activation. Information about the different alarms can be found in the document Event and Alarm Handling, Reference [7].

5.2.1.1   Trap Destination

The trap destination decides where ESA sends the SNMP alarms. If SNMP v2c is used, the destination is identified with IP address and port number. If SNMP v3 is used, more SNMP v3 configuration is required.

For more information, see Reference [10].

The configuration files are stored in /usr/local/esa/conf

The following file is used for configuring the trap destination - trapDestCfg.xml

The value of the attribute active specifies whether the trap destination configuration is active or not:

• yes - The trap destination definition is active.
• no - The trap destination definition is inactive.

By inactivating the trap destination configuration, ESA disregards reading the configuration and thus does not send any traps to that destination. This setting is useful since the configuration can be kept as is, while it is possible to stop the traps only by deactivating it with a no.

5.2.1.2   SNMP Version

The configuration files are stored in /usr/local/esa/conf

The default SNMP version is v2c.

For more information on how to change SNMP version, see Reference [10].

5.2.2   Fault Management Agent

During runtime, the Fault Management (FM) Agent is ready to accept alarm triggers and hold active alarms and alarm data, which is made available to the OSS using SNMP. The technician working locally with the system could benefit from looking at the data captured and produced by the FM Agent.

A number of CLI commands exist that makes the setup and configuration of the FM Agent easier. Also, they are useful when troubleshooting faults in the system by looking at the FM Agent alarm information.

The following CLI commands are available for the FM Agent:

• fmactivealarms – To view active alarms.
• fmsendmessage – To send message for triggering an alarm or event, or clearing an alarm.

5.2.2.1   Command: View Active Alarms

This command is used for viewing the active alarms that have been triggered in the FM Agent.

Note:  

This command must be run on an SC node.

5.2.2.1.1   Command Format

The command format:

fmactivealarms

The command does not take any arguments.

5.2.2.1.2   Command Output

When running the command on an SC node, if there are active alarms, the command output is a complete list of all active alarms. The following output example shows three active alarms.

# fmactivealarms

Active alarms:
!---------------------------------------------------------------
Module             : ALARMMOD
Error Code         : 11
Resource Id        : 1.1.1.2.72
Timestamp          : Wed Sep 21 09:50:03 CEST 2011
Model Description  : This is fault number 1!
Active Description : The component XXX has an error.
Event Type         : 1
Probable Cause     : 1024
Severity           : minor
Orig Source IP     : 10.1.100.88
---------------------------------------------------------------!
!---------------------------------------------------------------
Module             : ALARMMOD
Error Code         : 22
Resource Id        : 1.1.1.2.9
Timestamp          : Wed Sep 21 09:52:31 CEST 2011
Model Description  : This is fault number 2!
Active Description : The component YYY has an error.
Event Type         : 1
Probable Cause     : 1024
Severity           : minor
Orig Source IP     : 10.1.100.88
---------------------------------------------------------------!
!---------------------------------------------------------------
Module             : ALARMMOD
Error Code         : 33
Resource Id        : 1.1.1.2.21
Timestamp          : Wed Sep 21 09:54:44 CEST 2011
Model Description  : This is fault number 3!
Active Description : The component ZZZ has an error.
Event Type         : 1
Probable Cause     : 1024
Severity           : minor
Orig Source IP     : 10.1.100.88
---------------------------------------------------------------!

If there are no active alarms, the command output is only a heading:

# fmactivealarms

Active alarms:

5.2.2.2   Command: Send Message

This command is used for triggering alarms and events.

5.2.2.2.1   Command Format

The command format:

fmsendmessage <action> <moduleId> <errorCode> <resourceId> ["<activeDescr>"] [<sourceIp>]

The command attributes are the following:

For example, to clear an alarm with Dynamic Activation as module, Error code 4001 and resource ID. 1.1.1.1.2.2.192.168.0.3, issue the following command:

From originating node (the node that triggered the alarm):

# fmsendmessage -c PG 4001 .1.1.1.1.2.2.192.168.0.3 "manually cleared"

From any node:

# fmsendmessage -c PG 4001 .1.1.1.1.2.2.192.168.0.3 "manually cleared" 192.168.0.3

5.2.2.2.2   Command Output

If the specified alarm is successfully triggered, the command does not give any output.

If the specified alarm could not be triggered, the command returns an error message.

5.2.3   CPU Load Monitoring

Ericsson SNMP Agent (ESA) is also used for monitoring the CPU load. The Performance Management Agent (PMA) module in ESA can be used for this purpose.

PMA saves every five minutes a 3GPP XML file containing the CPU load for each core on each node. Saved files older than one day are automatically removed.

The configuration files are stored in the /usr/local/esa/conf directory.

All 3GPP files are stored in the /var/log/esa/pm3gppXml directory. The files can be fetched through SFTP.

For an example of a 3GPP file, see Example 9.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<measCollecFile xmlns="http://www.3gpp.org/ftp/specs/archive/32_series/32.435$measCollec">
  <fileHeader fileFormatVersion="3GPP_PM_32.435_XML_v11.0.0">
    <fileSender/>
    <measCollec beginTime="2014-01-06T19:45:00-05:00"/>
  </fileHeader>
  <measData>
    <managedElement/>
    <measInfo>
      <job jobId="CPULoadCounterNode3"/>
      <granPeriod duration="PT300S" endTime="2014-01-06T19:50:00-05:00"/>
      <repPeriod duration="PT300S"/>
      <measType p="1">CPULoadCounterNode3Core8.max</measType>
      <measType p="2">CPULoadCounterNode3Core5.min</measType>
      <measType p="3">CPULoadCounterNode3Core12.max</measType>
      <measType p="4">CPULoadCounterNode3Core9.actual</measType>
      <measType p="5">CPULoadCounterNode3Core12.actual</measType>
      <measType p="6">CPULoadCounterNode3Core1.actual</measType>
      <measType p="7">CPULoadCounterNode3Core12.min</measType>
      <measType p="8">CPULoadCounterNode3Core7.max</measType>
      <measType p="9">CPULoadCounterNode3Core5.max</measType>
      <measType p="10">CPULoadCounterNode3Core9.max</measType>
      <measType p="11">CPULoadCounterNode3Core9.min</measType>
      <measType p="12">CPULoadCounterNode3Core4.max</measType>
      <measType p="13">CPULoadCounterNode3Core10.max</measType>
      <measType p="14">CPULoadCounterNode3Core6.max</measType>
      <measType p="15">CPULoadCounterNode3Core3.min</measType>
      <measType p="16">CPULoadCounterNode3Core11.min</measType>
      <measType p="17">CPULoadCounterNode3Core6.min</measType>
      <measType p="18">CPULoadCounterNode3Core5.actual</measType>
      <measType p="19">CPULoadCounterNode3Core8.min</measType>
      <measType p="20">CPULoadCounterNode3Core7.actual</measType>
      <measType p="21">CPULoadCounterNode3Core7.min</measType>
      <measType p="22">CPULoadCounterNode3Core11.max</measType>
      <measType p="23">CPULoadCounterNode3Core6.actual</measType>
      <measType p="24">CPULoadCounterNode3Core2.min</measType>
      <measType p="25">CPULoadCounterNode3Core4.actual</measType>
      <measType p="26">CPULoadCounterNode3Core11.actual</measType>
      <measType p="27">CPULoadCounterNode3Core2.max</measType>
      <measType p="28">CPULoadCounterNode3Core4.min</measType>
      <measType p="29">CPULoadCounterNode3Core10.actual</measType>
      <measType p="30">CPULoadCounterNode3Core2.actual</measType>
      <measType p="31">CPULoadCounterNode3Core1.max</measType>
      <measType p="32">CPULoadCounterNode3Core8.actual</measType>
      <measType p="33">CPULoadCounterNode3Core10.min</measType>
      <measType p="34">CPULoadCounterNode3Core3.actual</measType>
      <measType p="35">CPULoadCounterNode3Core1.min</measType>
      <measType p="36">CPULoadCounterNode3Core3.max</measType>
      <measValue measObjLdn="">
        <r p="1">0</r>
        <r p="2">0</r>
        <r p="3">0</r>
        <r p="4">0</r>
        <r p="5">0</r>
        <r p="6">0</r>
        <r p="7">0</r>
        <r p="8">0</r>
        <r p="9">1</r>
        <r p="10">0</r>
        <r p="11">0</r>
        <r p="12">0</r>
        <r p="13">0</r>
        <r p="14">1</r>
        <r p="15">0</r>
        <r p="16">0</r>
        <r p="17">0</r>
        <r p="18">0</r>
        <r p="19">0</r>
        <r p="20">0</r>
        <r p="21">0</r>
        <r p="22">0</r>
        <r p="23">0</r>
        <r p="24">0</r>
        <r p="25">0</r>
        <r p="26">0</r>
        <r p="27">0</r>
        <r p="28">0</r>
        <r p="29">0</r>
        <r p="30">0</r>
        <r p="31">0</r>
        <r p="32">0</r>
        <r p="33">0</r>
        <r p="34">0</r>
        <r p="35">0</r>
        <r p="36">0</r>
      </measValue>
    </measInfo>
  </measData>
  <fileFooter>
    <measCollec endTime="2014-01-06T19:50:00-05:00"/>
  </fileFooter>
</measCollecFile>

The following is to be performed on all nodes.

Configuration of PMA:

• Edit the /usr/local/esa/conf/mainCfg.xml file with the following command:

  $ sudo -u actadm vi /usr/local/esa/conf/mainCfg.xml

  To configure:

  • The output directory.

    Configure the pm3gppXml node to appropriate output directory. For example:

    <output> <pm3gppXml>/var/log/esa/pm3gppXml</pm3gppXml> </output>

  • The time interval in days of how long the 3GPP files are saved.

    Configure the fileRetentionPeriod node. For example:

    <pm> ... <fileRetentionPeriod>1</fileRetentionPeriod> ... </pm>

  • The time interval in minutes of how often the system scans for old 3GPP files.

    Configure the fileRetentionScan node. For example:

    <pm> ... <fileRetentionScan>30</fileRetentionScan> ... </pm>

• Edit the /usr/local/esa/conf/pmCounters/CPUloadCounterNode<nodeId>.xml file with the following command:

  $ sudo -u actadm vi /usr/local/esa/conf/pmCounters/CPUloadCounterNode<nodeId>.xml

  To configure:

  • The time interval in seconds of how often the system reads the data.

    Configure the interval attribute in all dataSource nodes. For example:

    <dataSource interval="60">

    There are limitations on allowed values. They are 10, 15, 20, 30, 60, 120, 300, 600, 900, 1200, 1800, 3600 seconds. For more details, see ESA SAG Performance Management.

• Edit the /usr/local/esa/conf/pmJobs/CPUloadJobNode<nodeid>.xml file with the following command:

  $ sudo -u actadm vi /usr/local/esa/conf/pmJobs/CPUloadJobNode<nodeId>.xml

  To configure:

  • The time interval in minutes of how often the system collects data before any 3GPP file is written to file.

    Configure the <granularityPeriod> node. For example:

    <defGranularityPeriod>5</defGranularityPeriod>

    There are limitations on allowed values. They are 5, 15, 30, 60 minutes. For more details, see ESA SAG Performance Management.

For the new configuration to take effect, restart ESA:

$ sudo 3ppmon stopesa

$ sudo 3ppmon startesa

5.2.4   Configuring Internal Heartbeat Monitoring

The PMA module in ESA is used for monitoring the internal heartbeat. Two PMAs are running, one on each SC node. They monitor all nodes and have defined counters and thresholds for how many SC nodes and PL nodes that are considered as alive, respectively on each SC node.

Counters are fetched by the PMA, by calling an external script, /opt/dve/bin/getNodesNumber.sh

Default interval is 60 seconds. Intervals allowed are 10, 15, 20, 30, 60, 120, 300, 600, 900, 1200, 1800, 3600 seconds.

Configuration of heartbeat interval:

Note:  

Both steps are performed on both SC nodes.

1. Edit the /usr/local/esa/conf/pmCounters/pmNodeCounter.xml file with the following command:

   $ sudo -u actadm vi /usr/local/esa/conf/pmCounters/pmNodeCounter.xml

   Configure the dataSource interval attribute. For example:

   <dataSource interval="120">

2. For the new configuration to take effect, restart ESA:

   $ sudo 3ppmon stopesa

   $ sudo 3ppmon startesa

If one of the PL or SC nodes goes down, an event is sent indicating which particular node that is not responding.

If a counter breaches the falling threshold level, for example the counter for PL nodes, the active ESA administrator (master) raises an alarm. Once the counter breaches the rising threshold level, the active ESA administrator (master) ceases the alarm. Threshold is also defined for SC node counter and raises an alarm if one SC node goes down. Once the counter breaches the rising threshold level, the active ESA administrator (master) ceases the alarm. Default counter thresholds are defined as follows:

Configuration of thresholds:

Note:  

Both steps are performed on both SC nodes.

1. Edit the /usr/local/esa/conf/pmThresholds/pmNodeThreshold.xml file with the following command:

   $ sudo -u actadm vi /usr/local/esa/conf/pmThresholds/pmNodeThreshold.xml

   Configure the default value for rise and fall for PAYLOADCOUNTER

   For example:

   <threshold active="yes">         <counter>             <groupId>CountersForMonitoringOfNodes</groupId>             <counterId>PAYLOADNODECOUNTER</counterId>         </counter>         <type>             <level invert="yes">                 <rise>2</rise>                 <fall>1</fall>             </level>         </type>         <output type="alarm">             <moduleId>PERFORMANCE</moduleId>             <errorCode>6002</errorCode>             <resourceId>.1.1.4.2</resourceId>             <text>Span threshold breached.</text>             <origSourceIp>10.61.235.10</origSourceIp>         </output> </threshold>

   Note:  

   The <origSourceIp> attribute in the above example is a VIP address. The reason for this is that the alarm must be unique in case the raising of an alarm and the ceasing of the same alarm is handled by different PMAs. Appending a VIP address to a <resourceId> is not applicable since the counters are global for the whole cluster, not just for a node. For more details about the address and node name, check the event.

   
   
2. If adding extra PL nodes to the Dynamic Activation cluster, the default threshold values can be updated by the following command:

   $ sudo -u actadm /opt/dve/bin/updatePmThreshold.sh

   Note:  

   The threshold definition file, /usr/local/esa/conf/pmThresholds/pmNodeThreshold.xml, is overwritten.

   
   

Each counter is published through the ERICSSON-ESA-PM-MIB and is possible to fetch through the SNMP Get operations from OSS-RC.

5.3   Firewall

This section describes the basic firewall operations and settings.

5.3.1   Activating and Deactivating

Note:  

The firewall is activated by default on all nodes. The firewall applies to all external interfaces parsed from cluster.conf and is automatically appended to iptables-rules.cfg

To activate the firewall on a node, enter the following command as user root:

# /etc/init.d/firewall start

To deactivate the firewall, enter the following command as user root:

# /etc/init.d/firewall stop

5.3.2   Custom Iptables Rules

Custom iptables rules are supplied by /home/actadm/config/iptables-rules-custom.cfg and are automatically appended to the existing iptables rules upon activation of the firewall.

Refer to /home/actadm/config/iptables-rules-custom.cfg.example

Example on how to add custom iptables rules.

As user root:

# /etc/init.d/firewall stop

# sudo -u actadm cp /home/actadm/config/iptables-rules-custom.cfg.example /home/actadm/config/iptables-rules-custom.cfg

Edit /home/actadm/config/iptables-rules-custom.cfg accordingly.

# /etc/init.d/firewall start

The following printout is displayed:

Starting iptables firewall
Appending custom iptables

5.4   DiffServ

The DiffServ framework defined by the IETF is used to implement a set of forwarding treatments and communicate the classification of packets between UDC nodes and backbone/site infrastructure.

Login as user root on appropriate nodes and perform the following:

1. To activate DiffServ and DSCP, issue the following commands from the appropriate nodes:
   • Default DiffServ Code Point is 16 (CS2):

     # iptables -t mangle -A OUTPUT -j DSCP --set-dscp 16

   • Default DiffServ Code Point for LDAP traffic towards CUDB is 40 (CS5):

     # iptables -t mangle -A OUTPUT -p tcp --dport 389 -j DSCP --set-dscp 40

   • Default DiffServ Code Point for SNMP traps is 32 (CS4):

     # iptables -t mangle -A OUTPUT -p udp --dport 162 -j DSCP --set-dscp 32

To make these rules persistent, do the following:

2. Edit the /home/actadm/config/iptables-rules-custom.cfg file:

   # sudo -u actadm vi /home/actadm/config/iptables-rules-custom.cfg

   If the file does not exist, copy /home/actadm/config/iptables-rules-custom.cfg.example as /home/actadm/config/iptables-rules-custom.cfg and add the following lines between *mangle and COMMIT:

   -A OUTPUT -j DSCP --set-dscp 16

   -A OUTPUT -p tcp -m tcp --dport 5555 -j DSCP --set-dscp 40

   -A OUTPUT -p udp -m udp --dport 162 -j DSCP --set-dscp 40

3. Save the file.
4. Restart the firewall, as user root:

   # /etc/init.d/firewall stop

   # /etc/init.d/firewall start

5.5   Configuring SSL

The SSL administrative procedures described in this section are designated for SSL communication through the following external interfaces:

• tomcat-server (port 8383)
• cudb-block-proxy (ports 8099,8994)
• NBIA/CAI3G (port 8181)
• NBIA/CAI3G (test port 8989)
• CAI (port 3301)
• CLI (port 8992)

5.5.1   General Information

The following guide, throughout these subsections, is a recommendation on how to configure SSL to work on CAI3G (HTTPS/SOAP), CAI (Telnet), CUDB Block Proxy (JMX over RMI), and Dynamic Activation GUI (HTTPS).

Since the Private Key Infrastructure (PKI), and SSL setups vary from one customer installation to another, this instruction is to be seen as a verified guide on how it can be done.

5.5.2   Remarks and Deviations

The following section contains information about critical points, deviations, and remarks regarding the different SSL configurations.

5.5.2.1   Tomcat-server Module

Only one private key and certificate pair is verified for this instruction.

5.5.2.2   Cudb-block-proxy Module

Only one private key and certificate pair is verified for this instruction.

5.5.2.3   NBIA Module

Only one private key and certificate pair is verified for this instruction.

5.5.3   Configuration Instruction

On a high-level, a complete SSL configuration flow contains the following:

• Prerequisite configuration:
  • Create keystore file.
  • Create a Certificate Signing Request (CSR) request.
  • Get Certificate Authority (CA) certificate and signed server certificate from CA organization.
  • Import CA certificate and then the signed server certificate into the keystore file.
  • Populate the system with the keystore files and SSL settings.
  • Activate the SSL settings.

  Note:  

  Two keystore files are maintained: one is traffic keystore for NBIA module, and the other is OAM keystore for other modules.

  Due to several modules use legacy SSL configuration parameters without the keyalias parameter, only one key – one certificate pair can be used for each module.

  
  
• Module/protocol specific configuration for every protocol:
  • NBIA module (port 8181), NBIA module (test port 8989)
  • tomcat-server module (port 8383)
  • cudb-block-proxy module (ports 8099,8994)
  • CLI module (port 8992), see Section 6.2.1.

  Note:  

  The individual module/protocol configuration items are independent of each other.

  A node-by-node activation is required before an individual configuration takes effect in system. Therefore, if multiple modules require SSL communication, it recommends configuring items for all those modules and then performing the activation and verification.

  
  

5.5.3.1   Prerequisite Configuration for NBIA Module

1. Generate a new private key and add it to the pgexternal_trf_keystore.jks keystore.

   The new key is to be used for all external communication.

   Note:  

   Below is an example using example-specific values, such as -dname and similar. Choose your own parameters according to real situation.

   
   

   $ sudo -u actadm /usr/java/latest/bin/keytool -genkey -keyalg RSA -alias pgexternaltrf -keystore /home/bootloader/ssl/pgexternal_trf_keystore.jks -validity 3600 -keysize 2048 -dname 'CN=eg8-vip-trf.ete.ka.sw.ericsson.se, OU=IT, O=Ericsson, L=Unknown, ST=Stockholm, C=SE'

   When prompted for password set keystore password.

2. Generate a CSR.

   $ sudo -u actadm /usr/java/'latest/bin/keytool -keystore /home/bootloader/ssl/pgexternal_trf_keystore.jks -certreq -alias pgexternaltrf -keyalg RSA -file /home/bootloader/ssl/pgexternaltrf.csr

3. Verify the creation of the CSR.

   $ cat /home/bootloader/ssl/pgexternaltrf.csr

   Output:

   -----BEGIN NEW CERTIFICATE REQUEST----- /.../ -----END NEW CERTIFICATE REQUEST-----

4. Send the CSR to your Certificate Authority (CA). Your CA will send you back a number of intermediate certificate and signed server certificate files.

   Note:  

   The number of certificates are differ due to CA vendor.

   
   

   A certificate file looks like the following:

   -----BEGIN CERTIFICATE----- [encoded data] -----END CERTIFICATE-----

5. Save all those files received from CA to /home/bootloader/ssl/.
6. Use the keytool to import the certificate files into the keystore, one by one.

   Note:  

   The following example imports three certificates from CA.

   
   
   • Import Root CA certificate file verisignRoot.crt:

     $ sudo -u actadm /usr/java/latest/bin/keytool -import -keystore /home/bootloader/ssl/pgexternal_trf_keystore.jks -file /home/bootloader/ssl/verisignRoot.crt -alias verisignrootca

     Enter keystore password:

     Certificate was added to keystore

   • Import Intermediate CA/CAs example file intermediate.crt:

     $ sudo -u actadm /usr/java/latest/bin/keytool -import -keystore /home/bootloader/ssl/pgexternal_trf_keystore.jks -file /home/bootloader/ssl/intermediate.crt -alias intermediateca

     Enter keystore password:

     Certificate was added to keystore

   • Import Server certificte file pgexternaltrf.crt:

     $ sudo -u actadm /usr/java/latest/bin/keytool -importcert -keystore /home/bootloader/ssl/pgexternal_trf_keystore.jks -file /home/bootloader/ssl/pgexternaltrf.crt -alias pgexternaltrf

     Enter keystore password:

     Certificate was added to keystore

   Note:  

   If the following error is shown:

   keytool error: java.security.cert.CertificateException:
   java.io.IOException: Incomplete data

   To solve the problem, change the certificate file from .crt to a .der file and then import. For example:

   $ sudo -u actadm openssl x509 -outform der -in pgexternaltrf.crt -out pgexternaltrf.der

   $ sudo -u actadm /usr/java/latest/bin/keytool -importcert -keystore /home/bootloader/ssl/pgexternal_trf_keystore.jks -file /home/bootloader/ssl/pgexternaltrf.der -alias pgexternaltrf

   
   

5.5.3.2   Prerequisite Configuration for Other Modules

For modules other than NBIA module, the prerequisite configuration is the same as NBIA, except those modules use keystore pgexternal_oam_keystore.jks.

An example is given as follows. The differences in commands are using "oam" keystore and certificate files (NBIA module uses "trf" instead). For more instruction details, see Section 5.5.3.1.

1. $ sudo -u actadm /usr/java/latest/bin/keytool -genkey -keyalg RSA -alias pgexternaloam -keystore /home/bootloader/ssl/pgexternal_oam_keystore.jks -validity 3600 -keysize 2048 -dname 'CN=eg8-vip-trf.ete.ka.sw.ericsson.se, OU=IT, O=Ericsson, L=Unknown, ST=Stockholm, C=SE'

   When prompted for password set keystore password.

2. $ sudo -u actadm /usr/java/latest/bin/keytool -keystore /home/bootloader/ssl/pgexternal_oam_keystore.jks -certreq -alias pgexternaloam -keyalg RSA -file /home/bootloader/ssl/pgexternaloam.csr
3. $ cat pgexternal.csr
4. Send the CSR to your CA and get certificate files back. In this example, there three files as follows:
   • Root CA certificate: verisignRoot.crt
   • Intermediate certificate: intermediate.crt
   • Service certificate: pgexternaloam.crt
5. Save the certificate files to /home/bootloader/ssl/.
6. Import those certificate files to the keystore one by one.
   • $ sudo -u actadm /usr/java/latest/bin/keytool -import -keystore /home/bootloader/ssl/pgexternal_oam_keystore.jks -file /home/bootloader/ssl/verisignRoot.crt -alias verisignrootca

     Enter keystore password:

     Certificate was added to keystore

   • $ sudo -u actadm /usr/java/latest/bin/keytool -import -keystore /home/bootloader/ssl/pgexternal_oam_keystore.jks -file /home/bootloader/ssl/intermediate.crt -alias intermediateca

     Enter keystore password:

     Certificate was added to keystore

   • Import Server certificte file pgexternaltrf.crt:

     $ sudo -u actadm /usr/java/latest/bin/keytool -importcert -keystore /home/bootloader/ssl/pgexternal_oam_keystore.jks -file /home/bootloader/ssl/pgexternaloam.crt -alias pgexternaloam

     Enter keystore password:

     Certificate was added to keystore

   Note:  

   If java.io.IOException: Incomplete data error is detected, change the certificate files from .crt to .der files before importing them. For example:

   $ sudo -u actadm openssl x509 -outform der -in pgexternaloam.crt -out pgexternaloam.der

   
   

5.5.3.3   NBIA Module SSL Configuration and Activation

1. Set keystore for NBIA module:

   $ bootloader.py config set --parameter @NBIA_KEYSTOREEXT@ --value /home/bootloader/ssl/pgexternal_trf_keystore.jks

2. Encrypt the clear text password:

   $ java -cp /usr/java/latest/lib/missioncontrol/plugins/org.eclipse.jetty.util_<version>.jar org.eclipse.jetty.util.security.Password <password>

   Example Output:

   OBF:1u2a1toa1w8v1tok1u301u2a1toa1w8v1tok1u30 MD5:f6fdffe48c908deb0f4c3bd36c032e72

3. Set keystore password for the NBIA module:

   $ bootloader.py config set --parameter @NBIA_SSLKEYPASSWORD@ --value OBF:1u2a1toa1w8v1tok1u301u2a1toa1w8v1tok1u30

   $ bootloader.py config set --parameter @NBIA_SSLPASSWORD@ --value OBF:1u2a1toa1w8v1tok1u301u2a1toa1w8v1tok1u30

4. Verify that all parameters have been set:

   $ bootloader.py config list

5. Perform an activation for the configured parameters to take effect on all modules on the system, one by one:
   Attention!

   Wait for each node to be activated before starting with the next one, otherwise traffic disturbances occur. The all parameter should only be used when no provisioning traffic is running.

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the node that is to be activated.

   Note:  

   Repeat the command for each PL node in the cluster.

   
   
6. Verify the certificate from an external node:
   • NBIA CAI3G, NBIA module:

     $ openssl s_client -connect <VIP-TRF-IP>:8181 -showcerts -tls1

     Output:

     CONNECTED(00000003) /.../ <Certificate output>

   • NBIA CAI3G Test Ports, NBIA module:

     $ openssl s_client -connect <VIP-TRF-IP>:8989 -showcerts -tls1

     Output:

     CONNECTED(00000003) /.../ <Certificate output>

5.5.3.4   Tomcat-server Module SSL Configuration and Activation

1. Set keystore for tomcat-server module:

   $ bootloader.py config set --parameter @TOMCAT_KEYSTORE_FILE@ --value /home/bootloader/ssl/pgexternal_oam_keystore.jks

2. Set the keystore password:

   $ bootloader.py config set --parameter @TOMCAT_KEYSTORE_PASSWORD@ --value <password>

3. Set the private alias:

   $ bootloader.py config set --parameter @AUTHN_PRIVATE_ALIAS@ --value pgexternaloam

4. Set the public alias:

   $ bootloader.py config set --parameter @AUTHN_PUBLIC_ALIAS@ --value pgexternaloam

5. Perform an activation for the configured parameters to take effect on all modules on the system, one by one:
   Attention!

   Wait for each node to be activated before starting with the next one, otherwise traffic disturbances occur. The all parameter should only be used when no provisioning traffic is running.

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the node that is to be activated.

   Note:  

   Repeat the command for each SC node in the cluster.

   
   
6. Verify the certificate from an external node:

   $ openssl s_client -connect <VIP-OAM-IP>:8383 -showcerts -tls1

   Expected response example:

   CONNECTED(00000003) depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5 verify error:num=20:unable to get local issuer certificate verify return:0 --- Certificate chain 0 s:/C=SE/ST=Stockholm/L=Stockholm/O=Ericsson/OU=IT/CN=eg8-vip-oam.ete.ka.sw.ericsson.se i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4 -----BEGIN CERTIFICATE----- MIIGCzCCBPOgAwIBAgIQXEKFiasv3kVZjna6Ms73NjANBgkqhkiG9w0BAQsFADB+ MQswCQYDVQQGEwJVUzEdMBsGA1UEChMUU3ltYW50ZWMgQ29ycG9yYXRpb24xHzAd BgNVBAsTFlN5bWFudGVjIFRydXN0IE5ldHdvcmsxLzAtBgNVBAMTJlN5bWFudGVj IENsYXNzIDMgU2VjdXJlIFNlcnZlciBDQSAtIEc0MB4XDTE3MDEwOTAwMDAwMFoX DTE4MDExMDIzNTk1OVowgYExCzAJBgNVBAYTAlNFMRIwEAYDVQQIDAlTdG9ja2hv bG0xEjAQBgNVBAcMCVN0b2NraG9sbTERMA8GA1UECgwIRXJpY3Nzb24xCzAJBgNV ----------------------------------------------

5.5.3.5   Cudb-block-proxy Module SSL Configuration and Activation

1. Set keystore for cudb-block-proxy module:

   $ bootloader.py config set --parameter @BLOCK_PROXY_SSLCERT_STORE@ --value /home/bootloader/ssl/pgexternal_oam_keystore.jks

2. Activate SSL of cudb-block-proxy module:

   $ bootloader.py config set --parameter @BLOCK_PROXY_JMXREMOTE_SSL@ --value true

   $ encrypt.sh '<clear_text_password>'

   Note:  

   The password needs to be inside the single quotation marks (').

   
   

   Output:

   pwqrl5Un3PemL4v88evQ+Q==

   $ bootloader.py config set --parameter @BLOCK_PROXY_SSLCERT_STORE_PASSWORD@ --value pwqrl5Un3PemL4v88evQ+Q==

3. Perform an activation for the configured parameters to take effect on all modules on the system, one by one:
   Attention!

   Wait for each node to be activated before starting with the next one, otherwise traffic disturbances occur. The all parameter should only be used when no provisioning traffic is running.

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the node that is to be activated.

   Note:  

   Repeat the command for each SC node in the cluster.

   
   
4. Verify the certificate from an external node:

   $ openssl s_client -connect <VIP-OAM-IP>:8994 -showcerts -tls1

   Example Output:

   CONNECTED(00000003) /.../ <Certificate output>

5.5.3.6   CAI SSL Configuration and Activation

The CAI SERVER2 with port 3301 is run with SSL by default. To change the SSL configuration for CAI SERVER2, run the following commands:

1. $ bootloader.py config set --parameter @CAI_SERVER_2_ENABLE_SSL@ --value <boolean value>

   Set <boolean value> to true to enable the SSL configuration for CAI SERVER2 with port 3301.

   Set <boolean value> to false to disable the SSL configuration for CAI SERVER2 with port 3301.

2. Perform an activation for the configured parameters to take effect on all modules on the system, one by one:
   Attention!

   Wait for each node to be activated before starting with the next one, otherwise traffic disturbances occur. The all parameter should only be used when no provisioning traffic is running.

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the node that is to be activated.

   Note:  

   Repeat the command for each PL node in the cluster.

   
   
3. Verify the certificate from an external node:

   $ openssl s_client -connect <VIP-OAM-IP>:3301 -showcerts -tls1

   Output:

   CONNECTED(00000003) /.../ <Certificate output>

5.5.3.7   General Cleanup

Backup all SSL working files, including the SSL keystore passwords, from the system to a secure storage area.

Files to be backed up are the:

• Keystore files
• CSR request file
• CA authority certificate files

5.6   Configuring Authentication Rules of GUI Users

This section describes how to configure authentication rules for GUI users, including:

• Password rules that must be fulfilled when setting a password for GUI users.
• Input parameters of Brute Force Protection Algorithm.

5.6.1   Brute Force Protection Algorithm

Brute Force Protection Algorithm is applied on the default admin GUI user because this user can not be permanently locked out from system.

Once the AUTHN_MAX_LOGIN_ATTEMPTS is reached for the default admin GUI user, the user is locked out for a period. The Brute Force Protection Algorithm determines the locking period as follows:

• When the admin user is locked for the 1^st time, the locking period is AUTHN_BRUTE_FORCE_INITIAL_LOCK_PERIOD (5 seconds by default).
• From the 2^nd time, the locking period is doubled each time, until the AUTHN_BRUTE_FORCE_MAX_LOCK_PERIOD is reached. For example:
  • 2^nd locking period is 10 seconds.
  • 3^rd locking period is 20 seconds.
  • 4^th locking period is 40 seconds, and so on.
• Later on, every time when the user is locked out, the locking period is AUTHN_BRUTE_FORCE_MAX_LOCK_PERIOD (15 minutes by default).

5.6.2   View Authentication Parameters

To list all the configuration information for the system, run the following command from an SC node:

$ bootloader.py config list --show_all

Then find the authentication parameters in tomcat section.

Table 20 lists all the parameters for authentication rule configuration.

(1)  The password length must consist of minimum 12 characters.

5.6.3   Modify An Authentication Parameter

To modify the default values of these parameters, do the following from an SC node:

1. Run the bootloader.py command to set a new value for a parameter.

   $ bootloader.py config set --parameter <parameterName> --value <newValue>

   The new value is saved in the user_config file.

   Example of setting a new value for AUTHN_MAX_DAYS_TO_REMEMBER as 30:

   $ bootloader.py config set --parameter @AUTHN_MAX_DAYS_TO_REMEMBER@ --value 30
   INFO - *** @AUTHN_MAX_DAYS_TO_REMEMBER@=30 inserted in user configuration
   INFO - *** To activate the changed parameter, execute "bootloader.py node activate --host <hostname>" command
   

   Check user_config for verification.

   $ cat /home/bootloader/config/user_config
   @AUTHN_MAX_DAYS_TO_REMEMBER@=30
   

2. From each SC node, one by one, run the following command to activate the modification:
   Attention!

   Wait for each SC node to be activated before starting with the next one, otherwise traffic disturbances occur. The all parameter should only be used when no provisioning traffic is running.

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the SC node that is to be activated.

   Note:  

   The new value will take effect after executed the command bootloader node activate.

   GUI users can keep using their current password until expiration.

   
   

5.6.4   Restore An Authentication Parameter

A parameter is restored to default value by removing the modified value from the user_config file.

To restore a parameter, do the following from an SC node:

1. Run the bootloader.py command to remove the modified value.

   $ bootloader.py config remove --parameter <parameterName>

   Example of restoring AUTHN_MAX_DAYS_TO_REMEMBER to default value:

   $ bootloader.py config remove --parameter @AUTHN_MAX_DAYS_TO_REMEMBER@
   INFO - *** Removed @AUTHN_MAX_DAYS_TO_REMEMBER@ from user configuration 
   INFO - *** To activate the changed parameter, execute "bootloader.py node activate --host <hostname>" command
   

2. From each SC node, one by one, run the following command to activate the modification:
   Attention!

   Wait for each SC node to be activated before starting with the next one, otherwise traffic disturbances occur. The all parameter should only be used when no provisioning traffic is running.

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the SC node that is to be activated.

   Note:  

   The restored value will take effect after executed the command bootloader node activate.

   GUI users can keep using their current password until expiration.

   
   

5.7   License Keys Administration

This section holds information on how to manually request and install a new license key and how to perform a rollback.

Note:  

There is no downtime when a new license is installed or a rollback is performed. The Sentinel server is accessed by Dynamic Activation once a day which means that Dynamic Activation has all licenses in cache. To speed up the process, it is possible to trigger an update through the Dynamic Activation GUI. Also, provisioning to the Dynamic Activation is not required to be stopped.

5.7.1   Requesting New License Key

Do the following as user root to request the new license key:

1. Obtain the license code from all SC nodes where the license server is installed:

   # cd /opt/sentinel/bin

   # ./echoid

   Sentinel RMS Development Kit 8.6.2.0053 Host Locking Code Information Utility Copyright (C) 2014 SafeNet, Inc. Locking Code 1 : 2008-*1MS LHEN 9GMR X8EQ Locking Code 1 (Old Style) : 2008-BA44D

   Note:  

   The echoid command needs to be run, as user root, in the /opt/sentinel/bin directory. Otherwise the license provided in Ericsson License Information System (ELIS) will not work.

   
   
2. Provide all Locking Code for the ELIS and get the license file.

13 FAT1022833/5 Ni LONG NORMAL NETWORK EXCL 100000 INFINITE_KEYS
 15 NOV 2013 11 NOV 2014 NO_SHR SLM_CODE 1 NON_COMMUTER NO_GRACE NO_OVERDRAFT 
CL_ND_LCK NON_REDUNDANT AB99F2008,FB97E2008 NO_HLD 5 M2M_Start,_
PS T,fzJ:wrGlhYlTlWOoaZ3VyN5wPB1aJd4HVM505BvjfWZAcemFO1DYYtplY:y90yLgTU2Vw2Z7C1x
FWRupieI93p#AID=fe665bf5-a24a-4c3c-910b-e882f41cb146

5.7.2   Installing License Key

To install a new license key, follow the procedure:

1. Back up the existing licenses if there are any. See Section 5.7.3.
2. Set the environment variable LSHOST to the license server:

   # export LSHOST=nfs

3. Import the new licenses by using a licence file received from ELIS. See Section 5.7.1.

   The license can be imported either from the file directly, or from the string found in the file.

   • Import a license file directly

     # /opt/sentinel/bin/lslic -F <file_name>

     Example printout

     Sentinel RMS Development Kit 8.6.2.0053 License Addition/Deletion Utility Copyright (C) 2014 SafeNet, Inc. License code "13 FAT1022833/5 Ni LONG NORMAL NETWORK EXCL 100000 INFINITE_KEYS 15 NOV 2013 11 NOV 2014 NO_SHR SLM_CODE 1 NON_COMMUTER NO_GRACE NO_OVERDRAFT CL_ND_LCK NON_REDUNDANT AB99F2008,FB97E2008 NO_HLD 5 M2M_Start,_PS T,fzJ:wrGlhYlTlWOoaZ3VyN5wPB1aJd4HVM505BvjfWZAcemFO1DYYtplY: y90yLgTU2Vw2Z7C1xFWRupieI93p$AID= fe665bf5-a24a-4c3c-910b-e882f41cb146" added on nfs

   • Import from a string in the license file
     1. Copy the entire text in the license file as a license string.

        Note:  

        The license string cannot contain any word-wrapping and the whole string needs to be on a single line for Sentinel to be able to read it.

        
        
     2. Import the licences from the license string.

        # /opt/sentinel/bin/lslic -A "<license_string>"

        Example printout

        License Addition/Deletion Utility Copyright (C) 2008 SafeNet, Inc. License code 13 FAT1022833/5 Ni LONG NORMAL NETWORK EXCL 100000 INFINITE_KEYS 15 NOV 2013 11 NOV 2014 NO_SHR SLM_CODE 1 NON_COMMUTER NO_GRACE NO_OVERDRAFT CL_ND_LCK NON_REDUNDANT AB99F2008,FB97E2008 NO_HLD 5 M2M_Start,_ PS T,fzJ:wrGlhYlTlWOoaZ3VyN5wPB1aJd4HVM505BvjfWZAcemFO1DYYtplY:y90yLgTU2Vw2Z7C1x FWRupieI93p$AID=fe665bf5-a24a-4c3c-910b-e882f41cb146

4. Restart the license server:

   # 3ppmon stoplserv

   # 3ppmon startlserv

5. Login as user root, and verify on the localhost, SC1, that the license is installed:

   # /opt/sentinel/bin/lsmon localhost

   Note:  

   Make sure that the response does not contain:

   There is no license in the server

   
   
6. Login to the SC2 node as en administrator.
7. Restart the license server:

   # 3ppmon stoplserv

   # 3ppmon startlserv

8. Login as user root, and verify on the localhost, SC2, that the license is installed:

   # /opt/sentinel/bin/lsmon localhost

   Note:  

   Make sure that the response does not contain:

   There is no license in the server

   
   
9. Verify that the license server NFS contains licenses:

   # /opt/sentinel/bin/lsmon nfs

   Note:  

   Make sure that the response does not contain:

   There is no license in the server

   
   
10. For the new license to take effect, an update on the Dynamic Activation applications needs to be performed in the GUI.

    Login to the Dynamic Activation web GUI. For detailed information, refer to User Guide for Resource Activation, Reference [8].

    Go to the System > Licenses tab.

    Click the update arrows for the appropriate Dynamic Activation application to load the license to the system.

11. If the new license is replacing a license, follow Section 5.7.5 to remove the old license.

5.7.3   Back up License Keys

To back up a license key, follow this procedure:

1. Login to SC1 and create a backup folder.

   # mkdir /home/actadm/licenses/backup

2. Back up the existing license.

   # cp /home/actadm/licenses/lservrc /home/actadm/licenses/backup

5.7.4   Restoring a Backup of License Keys

To restore a backup of a license key, follow this procedure:

1. Login as user root, and stop the Sentinel server on both SC nodes:

   # 3ppmon stoplserv

2. Copy the backed up file lservrc from directory /home/actadm/licenses/backup to the original directory, /home/actadm/licenses/:

   # cp /home/actadm/licenses/backup/lservrc /home/actadm/licenses/

3. Start the Sentinel server on both SC nodes:

   # 3ppmon startlserv

4. For this procedure to take effect, an update on the Dynamic Activation applications needs to be performed in the GUI.

   Login to the Dynamic Activation web GUI. For detailed information, refer to User Guide for Resource Activation, Reference [8].

   Go to the System > Licenses tab.

   Click the update arrows for the appropriate Dynamic Activation application to load the license on the system.

5.7.5   Removing a License Key

To remove a license key, follow the procedure:

1. Login to SC1 as user root, and set the environment variable LSHOST to the license server:

   # export LSHOST=nfs

2. Print existing licenses.

   # /opt/sentinel/bin/lsmon nfs

   Example printout

   Sentinel RMS Development Kit 8.6.2.0053 Application Monitor Copyright (C) 2014 SafeNet, Inc. [Contacting Sentinel RMS Development Kit server on host "nfs"] |- Feature Information |- Feature name : "FAT1022833/1" |- Feature version : "" |- License type : "Normal License" |- License Version : 0x08400000 |- Commuter license allowed : NO |- Maximum concurrent user(s) : Unlimited |- Unreserved tokens in use : 0 |- Available unreserved : Unlimited |- Reserved tokens in use : 0 |- Available reserved : 0 |- Soft limit on users : Unlimited |- License start date : Fri Nov 15 00:00:00 2013 |- Expiration date : Tue Nov 11 23:59:59 2014 |- Log encryption level : 1 |- Check time tamper : Yes |- Application-server locking : Client-locked license. |- Client $1 locking code : 2008-AB99F |- Client $2 locking code : 2008-FB97E |- Additive/exclusive/aggregate : Exclusive license(overrides additive licenses). |- Token lifetime (heartbeat) : 300 secs (5 min(s)) |- Public vendor information : SW_Basic,_PS |- Allowed on VM : YES |- License Information |- License Hash : 1181152AD32E7EBE |- License type : "Normal License" |- License Version : 0x08400000 |- License storage name : /home/actadm/licenses/lservrc |- License status : Active |- Commuter license allowed : NO |- Maximum concurrent user(s) : Unlimited |- Soft limit on users : Unlimited |- License start date : Fri Nov 15 00:00:00 2013 |- Expiration date : Tue Nov 11 23:59:59 2014 |- Log encryption level : 1 |- Check time tamper : Yes |- Application-server locking : Client-locked license. |- Client $1 locking code : 2008-AB99F |- Client $2 locking code : 2008-FB97E |- Additive/exclusive/aggregate : Exclusive license(overrides additive licenses). |- Token lifetime (heartbeat) : 300 secs (5 min(s)) |- Allowed on VM : YES

3. Set the environment variable LSFORCEHOST to the license server:

   # export LSFORCEHOST=nfs

4. Remove the specific License Key:

   # /opt/sentinel/bin/lslic -df “<Feature name>” “<Feature Version>” “<License Hash>”

   Example

   # /opt/sentinel/bin/lslic -df “FAT1022833/1” “” “1181152AD32E7EBE”

   Sentinel RMS Development Kit 8.6.2.0053 License Addition/Deletion Utility Copyright (C) 2014 SafeNet, Inc. This will delete license(s) from the server, do you want to continue? (Y/N):

   Press Y.

   license FAT1022833/1 are deleted.

5. Restart the license server:

   # 3ppmon stoplserv

   # 3ppmon startlserv

6. Verify on the localhost, SC1, that a license is still installed:

   # /opt/sentinel/bin/lsmon localhost

   Note:  

   Make sure that the response does not contain:

   There is no license in the server

   
   
7. Login to the SC2 node as an administrator.
8. Restart the license server:

   # 3ppmon stoplserv

   # 3ppmon startlserv

9. Verify on the localhost, SC2, that a license is still installed:

   # /opt/sentinel/bin/lsmon localhost

   Note:  

   Make sure that the response does not contain:

   There is no license in the server

   
   
10. Verify that the license server NFS contains licenses:

    # /opt/sentinel/bin/lsmon nfs

    Make sure that the response does not contain the text:

    There is no license in the server

11. For this procedure to take effect, an update on the Dynamic Activation applications needs to be performed in the GUI.

    Login to the Dynamic Activation web GUI. For detailed information, refer to User Guide for Resource Activation, Reference [8].

    Go to the System > Licenses tab.

    Click the update arrows to load the license to the system.

5.8   Cassandra Administration

This section outlines useful nodetool commands to administer the Cassandra database.

5.8.1   Cassandra Data Consistency

5.8.1.1   Cassandra Node Outage

A Dynamic Activation cluster consisting of 4 - 12 nodes, replication factor 3, and read/write on consistency level QUORUM, can only have one node down at a time.

• When a node goes down, Cassandra saves incoming writes for that node, in accordance to how the max_hint_window_ms parameter is configured (three hours by default). When the node is up and running again, the cluster starts replicating missed data.

  During node downtime, the node is still able to get consistent reads and writes, though Ericsson recommends getting the node back up and running as soon as possible.

• If the downtime is longer then the max_hint_window_ms configuration, the node requires manual repairing after it is up and running again. For instructions, see Section 5.8.1.2.

5.8.1.2   Repair Cassandra Node

When replacing a blade, or a node goes down for an extended period of time (more than three house by default), inconsistence occurs among Cassandra nodes. To synchronize Cassandra data, the replaced or down node needs to be repaired manually.

To repair a Cassandra node:

1. Log on to the node to be repaired.
2. Run the repair command.

   $ nodetool repair

   Note:  

   Repairing a Cassandra node is a time consuming task. Depending on how much data is available in Cassandra, the nodetool repair command can continue in background up to several hours.

   
   

5.8.2   Cassandra Performance Monitoring

The following commands can be used to monitor the Cassandra cluster:

• nodetool status logconsolidation - Displays information about the Cassandra cluster.
• nodetool compactionstats - Displays the total column and the total number of uncompressed bytes of SSTables being compacted.
• nodetool cfstats - Displays statistics on each table and keyspace.
• nodetool netstats - Displays statistics on network operations and connections within the Cassandra cluster.
• nodetool tpstats - Displays statistics about the number of active, pending, and completed tasks for each stage of Cassandra operations by thread pool.

For more information about nodetool commands, see https://docs.datastax.com/en/cassandra/2.2/cassandra/tools/toolsNodetool.html, Reference [15].

5.9   PC Keys

This section contains information on how to install a new PC key and how to perform a rollback.

Note:  

There is downtime of each node when a new key is installed or a rollback is performed as the packages in the application are redeployed. The Dynamic Activation reads the keys from the key directory every 5 seconds and redeploys when changes are detected. Provisioning is not required to be stopped when adding keys to one node at a time in a cluster, allowing time for each node to get back up.

5.9.1   Installing a PC Key

To install a PC key, follow the steps:

1. Transfer the PC key to a temporary folder and pack it into a .tar.gz file:

   $ tar cvzf PCkey-<Feature_Name>-<Rev>.tar.gz <PcKey>

2. Copy the packed PC key file to the /opt/ericsson/activation/bootloader/repository/ directory:

   $ sudo -u actadm cp PCkey-<Feature_Name>-<Rev>.tar.gz /opt/ericsson/activation/bootloader/repository/

3. If upgrading an old PC key, repeat the following command for all PL nodes to uninstall the old PC key.

   From an SC node:

   $ bootloader.py submodule delete -n PCkey-<Feature_Name> -p dve-application --host <hostname>

   <hostname> is the hostname of the PL node on which the old PC key is to be uninstalled.

4. Repeat the following command for all PL nodes to install the PC key.

   From an SC node:

   $ bootloader.py submodule add -n PCkey-<Feature_Name>-<Rev>.tar.gz -t key -p dve-application --host <hostname>

   <hostname> is the hostname of the PL node on which the PC key is to be installed.

5. From an SC node, run the following command for all PL nodes, one by one, to activate the PC key:
   Attention!

   Wait for each PL node to be activated before starting with the next one, otherwise traffic disturbances occur. The all parameter should only be used when no provisioning traffic is running.

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the PL node that is to be activated.

The keys are active as soon as the application has been redeployed, no other means of activation are necessary unless stated by the PC documentation.

5.9.2   Rolling Back a PC Key

To roll back a PC key, follow the steps:

1. Repeat the following command for all PL nodes to remove the PC key.

   From an SC node:

   $ bootloader.py submodule delete -n PCkey-<Feature_Name> -p dve-application --host <hostname>

   <hostname> is the hostname of the PL node on which the PC key is to be removed.

2. If the current PC key is to be replaced by a previous key, ensure that the previous packed PC key file exists in the /opt/ericsson/activation/bootloader/repository/, and reinstall the previous key.

   From an SC node:

   $ bootloader.py submodule add -n PCkey-<Feature_Name>-<Rev>.tar.gz -t key -p dve-application --host <hostname>

   <hostname> is the hostname of the PL node on which the PC key is to be reinstalled.

3. From an SC node, run the following command, for all PL nodes, one by one, to activate the PC key:
   Attention!

   Wait for each PL node to be activated before starting with the next one, otherwise traffic disturbances occur. The all parameter should only be used when no provisioning traffic is running.

   $ bootloader.py node activate --host <hostname>

   <hostname> is the hostname of the PL node that is to be activated.

4. Remove the packed PC key file from the /opt/ericsson/activation/bootloader/repository/ directory:

   $ cd /opt/ericsson/activation/bootloader/repository/

   $ sudo -u actadm rm PCkey-<Feature_Name>-<Rev>.tar.gz

The keys are deactivated as soon as the application has been redeployed, no other means of deactivation are necessary unless stated by the PC documentation.

5.10   Processing Log Admin Tool

The Processing Log Admin Tool provides maintenance functionality for exporting processing logs from the data storage. This tool is also used by crontab in the Dynamic Activation cluster. When necessary, to release data storage space, it removes log entries in the Cassandra database (starting with the oldest).

The Processing Log Admin Tool can be run locally on one of the nodes within Dynamic Activation. It can also be run from an external node.

5.10.1   Using Processing Log Admin Tool

5.10.1.1   Exporting Logs

The following are ways to export logs using the Processing Log Admin Tool.

• Logs can be exported as days in UTC using the -e option. Then all log entries stored during a day are exported where a day is defined as YYYY-MM-DD 00:00:00.000UTC to YYYY-MM-DD 23:59:59.999UTC. Logs exported as a day are unsorted to optimize export speed for backup purposes.
• Logs can be exported for a given time interval using the -et option. Then all complete logs (a northbound request with all belonging southbound requests) stored during the time interval are exported. The time interval is provided in the system local timezone. The output is sorted by the northbound entry timestamp.
• Complete logs can be exported sorted with the latest log first, using the -el option. This option searches the data store backwards until a provided limit of logs are exported. When exporting logs, a path for storing the export files must be provided.

Export location

The exported log files are compressed to a *.gz file, to minimize disk use. During export, for approximately every 100000 logged entries a compressed file will be created. It is important to take into account that when exporting huge amount of data, these files can fill up the target partition, and it can take a few hours for the export of provisioning data to finish.

When export log files, chose the export location as follows:

• Whenever possible, export the log files to an external disk that is mounted onto the server, or a local path on an external computer that has access to the cluster.
• If only an internal disk is available, the /var/log directory can be used when the size of the log files is small.

  Note:  

  The free space of the /var/log directory can be checked by executing the df -h command.

  
  

5.10.1.1.1   Running Processing Log Admin Tool

Table 21 shows the arguments to use when exporting processing logs:

Run the Processing Log Admin Tool from LDE:

For local use, the Processing Log Admin Tool executable shell script, proclog-admin-tool.sh, is located in the /usr/local/pgngn/admin-tool-<version>/bin directory.

The following example shows how to export the processing logs created during one specified hour:

Example 1:

$ cd /usr/local/pgngn/admin-tool-<version>/bin

$ sudo -u actadm ./proclog-admin-tool.sh -et "2014-06-20 12:00:00 2014-06-23 13:00:00" -p <export_dir_path>

5.10.2   Configuring Processing Log Admin Tool

To update the Processing Log Admin Tool configuration, follow the instructions in Section 4.2.