Contents

1   Introduction

The purpose of this document is to instruct what troubleshooting data is to be collected and enclosed in a Customer Service Request (CSR) or Trouble Report (TR) in case a problem is experienced with MTAS.

This document also describes the procedure to collect the needed information.

Consider the recommended printouts and tracings listed in the document as requirements for meaningful CSR or TR analysis. If necessary data, descriptions, or enclosures are missing, it can result in more data requests from the Ericsson Customer Support.

1.1   Prerequisites

This section describes the possible documents, tools, and required conditions before starting the data collection procedure.

It is expected that the reader has prior knowledge about telecommunication, including knowledge about the virtualized environment and the MTAS. It is assumed that the reader is familiar with concepts, terminology, and abbreviations within these areas.

1.1.1   Parameters

Necessary parameters when performing data collection of the MTAS are shown in Table 1.

1.1.2   Documents

Before starting this procedure, ensure that the following information or documents are available:

• MTAS Alarm List
• MTAS AppTrace
• Create Custom Role
• Create Custom Rule
• Create User Account
• Handling Files
• MTAS Health Check
• Virtual IP Address Management

1.1.3   Tools

A workstation with an SSH client must be available before performing any procedure in this document.

1.1.4   Conditions

For the data collection activities which are needed to run the Health Check, at least the following roles must be assigned for the user: ‘‘Mtas_Application_Administrator’’ and ‘‘SystemTroubleshooter’’. See Create User Account for creating users.

When using ECLI, a CustomRole and CustomRule need to be created to be able to run the CDCLSv commands.

The following roles are needed:

• SystemAdministrator
• SystemSecurityAdministrator
• SystemTroubleshooter
• Mtas_Application_Administrator
• <CustomRole>

The following is an example showing the settings for the CustomRole in ECLI:

(LocalAuthorizationMethod=1)>show CustomRole=RunCDCSLv

CustomRole=RunCDCSLv
   roleName="RunCDCSLv"
   rules
   "ManagedElement=1,SystemFunctions=1,SecM=1,UserManagement=1,⇒
 LocalAuthorizationMethod=1,CustomRule=CLISubshell" 

The following is an example showing the settings for the CustomRule in ECLI:

(LocalAuthorizationMethod=1)>show CustomRule=CLISubshell

CustomRule=CLISubshell
   permission=RWX 
   reservedByRoles 
   "ManagedElement=1,SystemFunctions=1,SecM=1,UserManagement=1,⇒
 LocalAuthorizationMethod=1,CustomRole=RunCDCSLv" 
   ruleData=":cli:regexp:cdclsv.*" 
   ruleName="CLISubshell"

For more information on custom rules and custom roles, see Create Custom Rule and Create Custom Role respectively.

2   Workflow

The workflow for collecting data from the MTAS node is as follows:

1. Collect mandatory data that is needed in connection to any problems experienced. Go to Section 3 Mandatory Data.
2. Collect specific data based on the type of problem that is experienced. Go to Section 3.2 Data Collected Based on Specific Problems and the collecting procedures specified in Section 4.1 MTAS Data Collection Feature.
3. Collect other useful information if it is available within an acceptable amount of time and effort. Go to Section 5 Other Useful Information.

3   Mandatory Data

The data described in this section is always to be included in a CSR or TR.

It is important that data is collected as soon as possible after the problem has occurred as the relevant data can be lost if this activity is postponed.

3.1   General Data to Be Collected

The following types of data are to be collected:

• Version information of the MTAS product and other relevant nodes.
• A detailed description of the problem and for which scenarios the problem has been observed.
• If known, a detailed step by step description of how the fault can be reproduced
• Information about alarms and notifications that can be related to the observed problem.
• References to other CSR or TR problems, which have been observed when the fault occurred.
• Complete node configuration, exported from the Configuration Management browser.
• Information about recent configuration changes, software upgrades, and similar activities that have been performed.
• Values of MTAS counters *Error, *Failed, *NoKE, and ‘ *NoKI; Refer to section Checking Counters in MTAS Health Check.
• Any crashcollector files that can be related to the observed problem.

3.2   Data Collected Based on Specific Problems

This section describes different types of collectable data, based on specific problem types. The data types to be collected (and included in a CSR or TR) depend on the problem experienced.

3.2.1   Software Versions

The version information for the MTAS software components is to be collected.

The software version information can be collected with the following command on one of the SCs:

cmw-repository-list

3.2.2   MTAS Log Files

The logging information that is to be collected is described in Table 2.

Where:

• <X> is the SC number, 1 or 2.
• <Y> is the PL number, starting with 3 and increasing by 1.
• <N> is the name
• <Application> could be axis2, mtasxdms, mtasgenssc and mtasstas
• <Index> is between 1 and 10

3.2.3   Routing Information

The following logging information is to be collected:

• Routing Information on the SC and PL processors by running the route -n command on each of the PLs and SCs.
• Configuration of Evolved Virtual IP (eVIP), collected on active SC node, verified from the:
  /storage/system/config/evip-apr9010467/evip.xml file.

3.2.4   Alarms, Notifications, and Events

MTAS triggers alarms for the most critical events that require operator intervention. The alarm information is accessible through the command lde-alarm. Alerts are also triggered to report relevant events for the operator.

Alarms and notifications information can be found in the FaultManagementLog/alarm and FaultManagementLog/alert directories that are stored on the SC under:
/cluster/storage/no-backup/coremw/var/log/saflog/.

Operating Instructions (OPIs) for each event describe actions to be taken to cease alarms.

A list of the different alarms generated by the MTAS can be found in the MTAS Alarm List.

3.2.5   Performance Measurement Counters

Performance Measurement (PM) counters are useful problem indicators. In particular, this includes the PMF counter reports from the period when the faulty situation occurred.

These counters are available with COM File Management in PerformanceManagementReportFiles file group.

Note:  

For more information on how to transfer counter-files, refer to Handling Files.

Alternatively, it is also possible to collect only counter-information for the service to be reported.

3.2.6   Clients Used

The following must be provided:

• A list of the clients and versions used.

3.2.7   Surrounding Nodes

The following must be provided:

• A list of the surrounding node versions, both Ericsson and other vendors.

3.2.8   Traffic Scenarios

The following must be provided:

• A detailed description of the faulty traffic scenario.
• Network trace (PCAP) including the faulty sequence (if the problem can be repeated) together with information indicating the packet in the trace that provides evidence of incorrect MTAS behavior.
• Specify what the expected traffic behavior of MTAS is, together with a reference to the standard specification or MTAS Function Specification defining the expected behavior.
• Specify for what MTAS Application Server role the problem is experienced (for example MMTel Telephony AS, SCC-AS)
• MTAS service data (XML) provisioned on the subscriber affected by the faulty scenario.
• Application trace using an appropriate trace profile as recommended by MTAS AppTrace (if the problem can be repeated).

3.2.9   Network Configuration

The following must be provided:

• A description of the network configuration.

3.2.10   Configuration Parameter Values

The following must be provided:

• The configuration data exported from Information Model Management (IMM) using the following command:

  cmw-immconfig-export <filename>

3.2.11   Co-located Applications

The following must be provided:

• A list of all the applications and their versions that are co-located on the same node as the MTAS.

3.2.12   AppTrace

Valuable information can be obtained by using the AppTrace function in the MTAS.

For more information about AppTrace, refer to MTAS AppTrace.

3.2.13   Trace Pcap

Traces generated from Wireshark™ or other tracing tools are to be included in a CSR or TR depending on the specific problem type.

3.2.14   License Manager Log Files

License Manager Functionality related events are stored in log files, which are located in the following path of the cluster:

/storage/clear/lm-apr9010503/log/lm.SC-<X>.log

Where <X> is the number designation of the processor.

4   Data Collection Use

4.1   MTAS Data Collection Feature

The data collection tool is used for collecting data to be attached to a CSR, and to collect other useful information, if available in an acceptable amount of time and effort.

The data collection tool is composed of several steps. Steps are specific scripts covering different parts of the MTAS software. Steps are reading states, configurations, and produce logs; or, they collect existing logs into a given place.

Most of the produced logs are readable for the user; for example, EVipAndRouting collects the status of the eVIP controller and the status of ports into a table.

All steps are grouped in the following four predefined profiles in data collection: Limited, Basic, Full, and SLA.

Limited profile

Contains mainly MTAS specific logs, information from approximately the last 1 hour. Some other small steps are added as a plus.

Basic profile

Contains all the steps from Limited profile plus all other logs, except dump files.

Full profile

Contains everything from Basic profile plus dump files. Archive size can be extremely large.

SLA

The Service Level Agreement (SLA) profile is for system internal use, only. It contains only one step: Sla.

See Section 4.4 Profiles and Steps in Data Collection for a detailed specification of the profiles.

Depending on the selected steps/profile, the execution of Data Collection can take several minutes.

The results of the running data collection are stored in:

/cluster/storage/no-backup/dc/

4.2   Start Data Collection

4.2.1   Data Collection Using ECLI

The Data Collection profile packers are listed with the cdclsv-list command and executed with the cdclsv-invoke command. Execution status can be checked with the cdclsv-status command.

Steps

1. Log on to ECLI:

   ssh <username>@<oam-vip> -p 830 -t -s cli

2. Check which profiles are available:

   cdclsv-list

   The following is an example output:

   cdclsPk=DcMtasBasic,cdcls=CDCLSvSite cdclsPk=DcMtasFull,cdcls=CDCLSvSite cdclsPk=DcMtasLimited,cdcls=CDCLSvSite cdclsPk=DcMtasSla,cdcls=CDCLSvSite cdclsPk=HcMtasPostUpgrade,cdcls=CDCLSvSite cdclsPk=HcMtasPreUpgrade,cdcls=CDCLSvSite

3. Select one of the profiles listed in the previous step:

   DcMtasBasic DcMtasFull DcMtasLimited

4. Start data collection with the selected profile:

   cdclsv-invoke cdclsPk=<profile>,cdcls=CDCLSvSite

   Example with the DcMtasBasic profile:

   cdclsv-invoke cdclsPk=DcMtasBasic,cdcls=CDCLSvSite

5. Check the status of the packing:

   cdclsv-status cdclsPk=<profile>,cdcls=CDCLSvSite

   Example: cdclsv-status cdclsPk=DcMtasBasic,cdcls=CDCLSvSite

   The results are found in: /cluster/storage/no-backup/dc/

4.2.2   Data Collection Using CDCLS

Data collection using the CDCLS (Crash Dump and Console Log Collection Service) is performed by executing packer objects for data collection from the command line on either of the SCs.. The Data Collection profile packers are listed with the cdclsv-list-packers command and executed with the cdclsv-pack command.

To start data collection using CDCLS:

1. Check which profiles are available:

   cdclsv-list-packers | grep cdclsPk=DcMtas

   The following results are shown:

   cdclsPk=DcMtasBasic,cdcls=CDCLSvSite cdclsPk=DcMtasFull,cdcls=CDCLSvSite cdclsPk=DcMtasLimited,cdcls=CDCLSvSite cdclsPk=DcMtasSla,cdcls=CDCLSvSite

2. Select one of the following profiles:

   DcMtasBasic
   DcMtasFull
   DcMtasLimited
   DcMtasSla

   Note:  

   If the profile to use is not known, select DcMtasFull.

   
   
3. Start data collection with the selected profile:

   cdclsv-pack cdclsPk=<profile>,cdcls=CDCLSvSite

   Example with the DcMtasBasic profile:

   cdclsv-pack cdclsPk=DcMtasBasic,cdcls=CDCLSvSite

4. Check the status of the packing:

   cdclsv-pack-status cdclsPk=<profile>,cdcls=CDCLSvSite

   Example:

   cdclsv-pack-status cdclsPk=DcMtasBasic,cdcls=CDCLSvSite

   The results are found in:

   /cluster/storage/no-backup/dc/

4.2.3   Data Collection Used by MTAS Health Check

Health check calls the required steps of data collection, and analyzes the logs produced. At the end of the analysis, health check chooses a verdict from the following list:

• PASS
• FAIL
• INFO
• ERROR
• VERIFY

For more information on these verdicts, refer to MTAS Health Check.

If the verdict is PASS, the logs from data collection are removed.

If the verdict is FAIL, INFO, ERROR, or VERIFY, the logs remain available for further analysis.

4.3   Housekeeping of the Results of Data Collection

Housekeeping is required as the results are collected cumulatively (by default). The housekeeping of directory /storage/no-backup/dc is configured in ECLI. The configuration can be changed and checked in ECLI.

Steps

1. Show the current configuration:
   1. Log on to the ECLI:

      ssh <username>@<oam-mip> -p 830 -t -s cli

   2. Navigate to the dataCollection FileGroupPolicy MO, for example:

      >dn ManagedElement=1,SystemFunctions=1,FileM=1,FileGroupPolicy=dataCollection

   3. Show the current configuration:

      (FileGroupPolicy=dataCollection)>show -v

      The following is an output example with the default values:

      fileGroupPolicyId="dataCollection" fullFileGroupAction=DISCARD_OLDEST maxFileGroupSize=0 <default> maxNumberFiles=5 retentionTime=0 <default> userLabel=[] <empty>

      The attribute values are interpreted as follows:

      • maxFileGroupSize: The unit is kilobyte. 0 indicates that no limit is set.
      • maxNumberFiles: 0 indicates that no limit is set. There are also three small system files in the directory counted by the housekeeper. Thus, when the value is set to 5, only two data collection results are available.
      • retentionTime: The unit is minutes. 0 indicates that the files are kept forever.
2. Update the configuration:
   1. Log on to the ECLI:

      ssh <username>@<oam-mip> -p 830 -t -s cli

   2. Navigate to the dataCollection FileGroupPolicy MO, for example:

      >dn ManagedElement=1,SystemFunctions=1,FileM=1,FileGroupPolicy=dataCollection

   3. Enter configure mode:

      (FileGroupPolicy=dataCollection)>configure

   4. Set the new value, for example:

      (config-FileGroupPolicy=dataCollection)>maxNumberFiles=6

      Note:  

      Be careful when changing these values as disk space is limited. Data collection results can be extremely large depending on the size of the cluster, settings, number of users, and so on.

      There are also three small system files in the directory counted by the housekeeper. Thus, when the value is set to 6, only three data collection results are available.

      
      
   5. Commit the changes:

      (config-FileGroupPolicy=dataCollection)>commit

   6. Verify the new values:

      (FileGroupPolicy=dataCollection)>show -v

      The following is an example output:

      fileGroupPolicyId="dataCollection" fullFileGroupAction=DISCARD_OLDEST maxFileGroupSize=0 <default> maxNumberFiles=6 retentionTime=0 <default> userLabel=[] <empty>

4.4   Profiles and Steps in Data Collection

5   Other Useful Information

Other useful information can be included in a CSR or a TR if it is easily available and there is enough time to collect it. An example of useful information is subscriber data, which is collected through the Business Support System (BSS).