Contents

1   Introduction

The purpose of this document is to instruct what troubleshooting data is to be collected and enclosed in a Customer Service Request (CSR) or Trouble Report (TR) in case a problem is experienced with MTAS.

This document also describes the procedure to collect the needed information.

Consider the recommended printouts and tracings listed in the document as requirements for meaningful CSR or TR analysis. If necessary data, descriptions, or enclosures are missing, it can result in more data requests from the Ericsson Customer Support.

1.1   Prerequisites

This section describes the possible documents, tools, and required conditions before starting the data collection procedure.

It is expected that the reader has prior knowledge about telecommunication, including knowledge about the virtualized environment and the MTAS. It is assumed that the reader is familiar with concepts, terminology, and abbreviations within these areas.

1.1.1   Parameters

Necessary parameters when performing data collection of the MTAS are shown in Table 1.

Table 1    Installation and Configuration Parameters

Name	Value	Description
oam-vip	____.____.____.____	The external VIP address of the O&M network. For an explanation of the VIP address concept, refer to the following document: Virtual IP Address Management
linux_user_name		The platform administrator name on the SC processors.
linux_user_password		The platform administrator password on the SC processors.

1.1.2   Tools

A workstation with an SSH client must be available before performing any procedure in this document.

2   Workflow

The workflow for collecting data from the MTAS node is as follows:

1. Collect mandatory data that is needed in connection to any problems experienced. Go to Section 3 Mandatory Data.
2. Collect specific data based on the type of problem that is experienced. Go to Section 3.2 Data Collected Based on Specific Problems and the collecting procedures specified in Section 4.1 Collecting Data Using MTAS Data Collection Feature.
3. Collect other useful information if it is available within an acceptable amount of time and effort. Go to Section 5 Other Useful Information.

3   Mandatory Data

The data described in this section is always to be included in a CSR or TR.

It is important that data is collected as soon as possible after the problem has occurred as the relevant data can be lost if this activity is postponed.

3.1   General Data to Be Collected

The following types of data are to be collected:

• Version information of the MTAS product and other relevant nodes.
• A detailed description of the problem and for which scenarios the problem has been observed.
• If known, a detailed step by step description of how the fault can be provoked.
• Information about alarms and notifications that can be related to the observed problem.
• References to other CSR or TR problems, which have been observed when the fault occurred. Slightly rewritten.
• Complete node configuration, exported from the Configuration Management browser.
• Information about recent configuration changes, software upgrades, and similar activities that have been performed.
• Values of MTAS counters *Error, *Failed, *NoKE, and ‘ *NoKI; Refer to section Checking Counters in MTAS Health Check.
• Any crashcollector files that can be related to the observed problem.

3.2   Data Collected Based on Specific Problems

This section describes different types of data based on specific problem types that can be collected. The data types to be collected and included in a CSR or TR depend on the problem experienced.

This section describes different types of data that can be collected. The problem experienced is the source to which of the following data types is to be included in a CSR or TR.

3.2.1   Software Versions

The version information for the MTAS software components is to be collected.

3.2.2   MTAS Log Files

The logging information that is to be collected is described in Table 2.

Table 2    MTAS Log Files

Filename	Log Path
vDicos Virtual Machine Log	SC <x>:/cluster/storage/no-backup/cdclsv/log/lpmsv
Processor Log	SC <x>:/var/log/SC-<X>/messages SC <x>:/var/log/PL-<Y>/messages
MTAS Application Log	SC <x>:/storage/no-backup/coremw/var/log/saflog/MTASAppLogs/vdicos/MTAS_<from-date_to-date>.log
Crash collector Log	SC <x>:/cluster/storage/no-backup/cdclsv/cadump/
MTAS Catalina Log	PL <y>:/opt/mmas/appserver/<instance_name>/logs/catalina.log
MTAS Catalina Out Log	PL <x>:/opt/mmas/appserver/<instance_name>/logs/catalina.out
Access Logs	SC <x>:/storage/no-backup/coremw/var/log/saflog/MMASAccessLogs/
CAI3G Log (AuditLog)	SC <x>:/storage/no-backup/coremw/var/log/saflog/MMASAuditLogs

3.2.3   Routing Information

The following logging information is to be collected:

• Routing Information on the SC and PL processors by running the route-n command on each of the PLs and SCs.
• Configuration of eVIP, verified from the
  /storage/system/config/evip-apr9010467/evip.xml file.

3.2.4   Alarms, Notifications, and Events

The MTAS triggers alarms for the most critical events that require operator intervention. The alarm information is accessible through the command lde-alarm. Alerts are also triggered to report relevant events for the operator.

Alarms and notifications information can be found in the FaultManagementLog/alarm and FaultManagementLog/alert directories that are stored on the SC under:
/cluster/storage/no-backup/coremw/var/log/saflog/.

Operating Instructions (OPIs) for each event describe the actions to be taken to cease alarms.

A list of the different alarms generated by the MTAS can be found in MTAS Alarm List.

3.2.5   Performance Measurement Counters

Performance Measurement (PM) counters are useful problem indicators. In particular, this includes the PMF counter reports from the period when the faulty situation occurred.

These counters are available with COM File Management in PerformanceManagementReportFiles file group.

Note:  

For more information on how to transfer counter-files, refer to File Management.

Alternatively, it is also possible to collect only counter-information for the service to be reported.

3.2.6   Clients Used

The following must be provided:

• A list of the clients and versions used

3.2.7   Surrounding Nodes

The following must be provided:

• A list of the surrounding node versions, both Ericsson and other vendors

3.2.8   Traffic Scenarios

The following must be provided:

• A detailed description of the faulty traffic scenario.
• Network trace (PCAP) including the faulty sequence (if the problem can be repeated) together with information on what packet in the trace being evidence of the incorrect MTAS behavior.
• Specify what the expected traffic behavior of MTAS is together with a reference to the standard specification or MTAS Function Specification defining the expected behavior.
• Specify for what MTAS application server role the problem is experienced (for example MMTel Telephony AS, SCC-AS)
• MTAS service data (XML) provisioned on the subscriber effected by the faulty scenario.
• Application trace using an appropriate trace profile as recommended by MTAS AppTrace (if the problem can be repeated).

3.2.9   Network Configuration

The following must be provided:

• A description of the network configuration

3.2.10   Configuration Parameter Values

The following must be provided:

• The configuration data exported from IMM using the following command:

  cmw-immconfig-export <filename>

3.2.11   Co-located Applications

The following must be provided:

• A list of all the applications and their versions that are co-located on the same node as the MTAS

3.2.12   AppTrace

Valuable information can be obtained by using the AppTrace function in the MTAS.

For more information about AppTrace, refer to MTAS AppTrace.

3.2.13   Trace pcap

Traces generated from Wireshark™ or other tracing tools are to be included in a CSR or TR depending on the specific problem type.

3.2.14   License Manager Log Files

License Manager Functionality related events are stored in log files, which are located in the following path of the cluster:

/storage/clear/lm-apr9010503/log/lm.SC-<X>.log

where <X> is the number designation of the blade.

4   Collecting Data

4.1   Collecting Data Using MTAS Data Collection Feature

Data collection is performed using the CDCLS by executing packer objects for data collection. The data collection profile packers can be listed with cdclsv-list-packers and executed with cdclsv-pack. The results are stored in /cluster/storage/no-backup/dc/. The following data collection profiles are available:

• Basic

  Basic System Information
  CPU Load
  Memory use
  Disk use
  Software Inventory
  Virtual Machine log files
  Processor log files
  Virtual Machine dump files
  Capsule Abortion dump files
  Alarms
  Outage Information
  DRBD Status
  License Manager log files
  

• Medium

  This profile covers all areas from Basic profile and the following ones.

  Disk Status
  MTAS Counters Report
  MTAS PM Report
  CPU Load for MTAS Traffic Pool
  Memory use for MTAS Traffic Pool
  

• Large

  This profile covers all areas from Medium profile and the following ones.

  Upgrade List
  Backup List
  Signalling Status
  MTAS-specific log files

• Full

  This profile covers all areas from Large profile and the following ones.

  Node Configuration
  PM Report Files
  CM Data
  Routing data and eVIP configuration
  MIM files
  MTAS XDMS log files

• SLA

  This profile is for internal use only.

  Moves MtasSla PM Report files periodically to internal location: /storage/no-backup/MtasSla_PmReportFiles/ for consumption by HealthCheck.

  Deletes MtasSla files older than a day in the periodical housekeeping.

The data collection dump files are available through COM File Management in DataCollection file group.

Note:  

For more information on how to transfer data collection files, refer to File Management.

The following example shows how to check for the available profiles and how to collect data with a selected profile:

1. Check which profiles are available:

   cdclsv-list-packers | grep cdclsPk=DcMtas

2. Results are shown:

   cdclsPk=DcMtasBasic,cdcls=CDCLSvSite
   cdclsPk=DcMtasFull,cdcls=CDCLSvSite
   cdclsPk=DcMtasLarge,cdcls=CDCLSvSite
   cdclsPk=DcMtasMedium,cdcls=CDCLSvSite
   cdclsPk=DcMtasSla,cdcls=CDCLSvSite

3. Select a profile and start data collection:

   cdclsv-pack cdclsPk=DcMtasFull,cdcls=CDCLSvSite

4. The next message is displayed:

   Dump is generated in the /storage/no-backup/dc/ directory.

   The user can check the status of the packing with the cdclsv-pack-status command.

5   Other Useful Information

Other useful information could be included in a CSR or TR in case the data is easily available and there is enough time available for collecting it. An example of useful information is subscriber data which is collected through the Business Support System.