Contents

1   Introduction

The purpose of this document is to instruct what troubleshooting data is to be collected and enclosed in a Customer Service Request (CSR) or Trouble Report (TR) in case a problem is experienced with MTAS.

This document also describes the procedure to collect the needed information.

Consider the recommended printouts and tracings listed in the document as requirements for meaningful CSR or TR analysis. If necessary data, descriptions, or enclosures are missing, it can result in more data requests from the Ericsson Customer Support.

1.1   Prerequisites

This section describes the possible documents, tools, and required conditions before starting the data collection procedure.

It is expected that the reader has prior knowledge about telecommunication, including knowledge about the virtualized environment and the MTAS. It is assumed that the reader is familiar with concepts, terminology, and abbreviations within these areas.

1.1.1   Parameters

Necessary parameters when performing data collection of the MTAS are shown in Table 1.

1.1.2   Tools

A workstation with an SSH client must be available before performing any procedure in this document.

2   Workflow

The workflow for collecting data from the MTAS node is as follows:

1. Collect mandatory data that is needed in connection to any problems experienced. Go to Section 3 Mandatory Data.
2. Collect specific data based on the type of problem that is experienced. Go to Section 3.2 Data Collected Based on Specific Problems and the collecting procedures specified in Section 4.1 MTAS Data Collection Feature.
3. Collect other useful information if it is available within an acceptable amount of time and effort. Go to Section 5 Other Useful Information.

3   Mandatory Data

The data described in this section is always to be included in a CSR or TR.

It is important that data is collected as soon as possible after the problem has occurred as the relevant data can be lost if this activity is postponed.

3.1   General Data to Be Collected

The following types of data are to be collected:

• Version information of the MTAS product and other relevant nodes.
• A detailed description of the problem and for which scenarios the problem has been observed.
• If known, a detailed step by step description of how the fault can be reproduced
• Information about alarms and notifications that can be related to the observed problem.
• References to other CSR or TR problems, which have been observed when the fault occurred.
• Complete node configuration, exported from the Configuration Management browser.
• Information about recent configuration changes, software upgrades, and similar activities that have been performed.
• Values of MTAS counters *Error, *Failed, *NoKE, and ‘ *NoKI; Refer to section Checking Counters in MTAS Health Check.
• Any crashcollector files that can be related to the observed problem.

3.2   Data Collected Based on Specific Problems

This section describes different types of data based on specific problem types that can be collected. The data types to be collected and included in a CSR or TR depends on the problem experienced.

This section describes different types of data that can be collected. The problem experienced decides which of the following data types is to be included in a CSR or TR.

3.2.1   Software Versions

The version information for the MTAS software components is to be collected.

3.2.2   MTAS Log Files

The logging information that is to be collected is described in Table 2.

Where:

• <X> is the SC number, 1 or 2.
• <Y> is the PL number, starting with 3 and increases by 1.
• <N> is the instance number, starting with 0 and increases by 1 for every PL; 0 for PL-3, 1 for PL-4, and so on.

3.2.3   Routing Information

The following logging information is to be collected:

• Routing Information on the SC and PL processors by running the route-n command on each of the PLs and SCs.
• Configuration of Evolved Virtual IP (eVIP), verified from the
  /storage/system/config/evip-apr9010467/evip.xml file.

3.2.4   Alarms, Notifications, and Events

The MTAS triggers alarms for the most critical events that require operator intervention. The alarm information is accessible through the command lde-alarm. Alerts are also triggered to report relevant events for the operator.

Alarms and notifications information can be found in the FaultManagementLog/alarm and FaultManagementLog/alert directories that are stored on the SC under:
/cluster/storage/no-backup/coremw/var/log/saflog/.

Operating Instructions (OPIs) for each event describe actions to be taken to cease alarms.

A list of the different alarms generated by the MTAS can be found in MTAS Alarm List.

3.2.5   Performance Measurement Counters

Performance Measurement (PM) counters are useful problem indicators. In particular, this includes the PMF counter reports from the period when the faulty situation occurred.

These counters are available with COM File Management in PerformanceManagementReportFiles file group.

Note:  

For more information on how to transfer counter-files, refer to File Management.

Alternatively, it is also possible to collect only counter-information for the service to be reported.

3.2.6   Clients Used

The following must be provided:

• A list of the clients and versions used.

3.2.7   Surrounding Nodes

The following must be provided:

• A list of the surrounding node versions, both Ericsson and other vendors.

3.2.8   Traffic Scenarios

The following must be provided:

• A detailed description of the faulty traffic scenario.
• Network trace (PCAP) including the faulty sequence (if the problem can be repeated) together with information on what packet in the trace being evidence of the incorrect MTAS behavior.
• Specify what the expected traffic behavior of MTAS is, together with a reference to the standard specification or MTAS Function Specification defining the expected behavior.
• Specify for what MTAS Application Server role the problem is experienced (for example MMTel Telephony AS, SCC-AS)
• MTAS service data (XML) provisioned on the subscriber affected by the faulty scenario.
• Application trace using an appropriate trace profile as recommended by MTAS AppTrace (if the problem can be repeated).

3.2.9   Network Configuration

The following must be provided:

• A description of the network configuration.

3.2.10   Configuration Parameter Values

The following must be provided:

• The configuration data exported from IMM using the following command:

  cmw-immconfig-export <filename>

3.2.11   Co-located Applications

The following must be provided:

• A list of all the applications and their versions that are co-located on the same node as the MTAS.

3.2.12   AppTrace

Valuable information can be obtained by using the AppTrace function in the MTAS.

For more information about AppTrace, refer to MTAS AppTrace.

3.2.13   Trace Pcap

Traces generated from Wireshark™ or other tracing tools are to be included in a CSR or TR depending on the specific problem type.

3.2.14   License Manager Log Files

License Manager Functionality related events are stored in log files, which are located in the following path of the cluster:

/storage/clear/lm-apr9010503/log/lm.SC-<X>.log

Where <X> is the number designation of the processor.

4   Data Collection Use

4.1   MTAS Data Collection Feature

The data collection tool is intended to collect data that is needed to attach to a CSR, and to collect other useful information, if available in an acceptable amount of time and effort.

The data collection tool is composed by several steps. Steps are specific scripts covering different parts of the MTAS software. Steps are reading states, configurations, and produce logs; or, they collect existing logs into a given place.

Most of the produced logs are readable for the user; for example, EVipAndRouting collects the status of the eVIP controller and the status of ports into a table.

All steps are grouped in the following four predefined profiles in data collection: Limited, Basic, Full, and SLA.

Limited profile

Contains mainly MTAS specific logs, information from approximately the last 1 hour. Some other small steps are added as a plus.

Basic profile

Contains all the steps from Limited profile plus all other logs, except dump files.

Full profile

Contains everything from Basic profile plus dump files. Archive size can be extremely large.

SLA

The Service Level Agreement (SLA) profile is for system internally use, only. It contains only one step: Sla.

See Section 4.4 Profiles and Steps in Data Collection for a detailed specification of the profiles.

Depending on the selected steps/profile, the execution of Data Collection can take several minutes.

The results of the running data collection are stored in:

/cluster/storage/no-backup/dc/

4.2   Start Data Collection

4.2.1   Data Collection Using ECLI

The Data Collection profile packers are listed with the cdclsv-list command and executed with the cdclsv-invoke command. Execution status can be checked with the cdclsv-status command.

Steps

1. Log on to ECLI:

   ssh <username>@<oam-vip> -p 830 -t -s cli

2. Check which profiles are available:

   cdclsv-list

   The following is an example output:

   cdclsPk=DcMtasBasic,cdcls=CDCLSvSite cdclsPk=DcMtasFull,cdcls=CDCLSvSite cdclsPk=DcMtasLimited,cdcls=CDCLSvSite cdclsPk=DcMtasSla,cdcls=CDCLSvSite cdclsPk=HcMtasPostUpgrade,cdcls=CDCLSvSite cdclsPk=HcMtasPreUpgrade,cdcls=CDCLSvSite

3. Select one of the profiles listed in the previous step:

   DcMtasBasic DcMtasFull DcMtasLimited

4. Start data collection with the selected profile:

   cdclsv-invoke cdclsPk=<profile>,cdcls=CDCLSvSite

   Example with the DcMtasBasic profile:

   cdclsv-invoke cdclsPk=DcMtasBasic,cdcls=CDCLSvSite

5. Check the status of the packing:

   cdclsv-status cdclsPk=<profile>,cdcls=CDCLSvSite

   Example: cdclsv-status cdclsPk=DcMtasBasic,cdcls=CDCLSvSite

   The results are found in: /cluster/storage/no-backup/dc/

4.2.2   Data Collection Using CDCLS

Data collection using the CDCLS (Crash Dump and Console Log Collection Service) is performed by executing packer objects for data collection. The Data Collection profile packers are listed with the cdclsv-list-packers command and executed with the cdclsv-pack command.

To start data collection using CDCLS:

1. Check which profiles are available:

   cdclsv-list-packers | grep cdclsPk=DcMtas

   The following results are shown:

   cdclsPk=DcMtasBasic,cdcls=CDCLSvSite cdclsPk=DcMtasFull,cdcls=CDCLSvSite cdclsPk=DcMtasLimited,cdcls=CDCLSvSite cdclsPk=DcMtasSla,cdcls=CDCLSvSite

2. Select one of the following profiles:

   DcMtasBasic
   DcMtasFull
   DcMtasLimited
   DcMtasSla

   Note:  

   If the profile to use is not known, select DcMtasFull.

   
   
3. Start data collection with the selected profile:

   cdclsv-pack cdclsPk=<profile>,cdcls=CDCLSvSite

   Example with the DcMtasBasic profile:

   cdclsv-pack cdclsPk=DcMtasBasic,cdcls=CDCLSvSite

4. Check the status of the packing:

   cdclsv-pack-status cdclsPk=<profile>,cdcls=CDCLSvSite

   Example:

   cdclsv-pack-status cdclsPk=DcMtasBasic,cdcls=CDCLSvSite

   The results are found in:

   /cluster/storage/no-backup/dc/

4.2.3   Data Collection Used by MTAS Health Check

Health check calls the required steps of data collection, and analyzes the produced logs. At the end of the analysis, health check chooses a verdict from the following list:

• PASS
• FAIL
• INFO
• ERROR
• VERIFY

For more information on these verdicts, refer to MTAS Health Check.

If the verdict is PASS, the logs from data collection are removed.

If the verdict is FAIL, INFO, ERROR, or VERIFY, the logs remain available for further analysis.

4.3   Housekeeping of the Results of Data Collection

Housekeeping is required as the results are collected cumulative (by default). The housekeeping of directory /storage/no-backup/dc is configured in ECLI. The configuration can be changed and checked in ECLI, for example:

>dn ManagedElement=1,SystemFunctions=1,FileM=1,FileGroupPolicy=dataCollection

>show -v

The following is an output example:

fileGroupPolicyId="dataCollection"
fullFileGroupAction=DISCARD_OLDEST
maxFileGroupSize=10
maxNumberFiles=4
retentionTime=10
userLabel=[] <empty>

4.4   Profiles and Steps in Data Collection

5   Other Useful Information

Other useful information can be included in a CSR or a TR if it is easily available and there is enough time to collect it. An example of useful information is subscriber data, which is collected through the Business Support System.