Contents

1   Introduction

The purpose of this document is to instruct what troubleshooting data is to be collected and enclosed in a Customer Service Request (CSR) or Trouble Report (TR) in case a problem is experienced with MTAS.

This document also describes the procedure to collect the needed information.

Consider the recommended printouts and tracings listed in the document as requirements for meaningful CSR or TR analysis. If necessary data, descriptions, or enclosures are missing, it can result in more data requests from the Ericsson Customer Support.

1.1   Prerequisites

This section describes the possible documents, tools, and required conditions before starting the data collection procedure.

It is expected that the reader has prior knowledge about telecommunication, including knowledge about the virtualized environment and the MTAS. It is assumed that the reader is familiar with concepts, terminology, and abbreviations within these areas.

1.1.1   Parameters

Necessary parameters when performing data collection of the MTAS are shown in Table 1.

Table 1    Installation and Configuration Parameters

Name	Value	Description
oam-vip	____.____.____.____	The external VIP address of the O&M network. For an explanation of the VIP address concept, refer to the following document: Virtual IP Address Management
linux_user_name		The platform administrator name on the SC processors.
linux_user_password		The platform administrator password on the SC processors.

1.1.2   Tools

A workstation with an SSH client must be available before performing any procedure in this document.

2   Workflow

The workflow for collecting data from the MTAS node is as follows:

1. Collect mandatory data that is needed in connection to any problems experienced. Go to Section 3 Mandatory Data.
2. Collect specific data based on the type of problem that is experienced. Go to Section 3.2 Data Collected Based on Specific Problems and the collecting procedures specified in Section 4.1 MTAS Data Collection Feature.
3. Collect other useful information if it is available within an acceptable amount of time and effort. Go to Section 5 Other Useful Information.

3   Mandatory Data

The data described in this section is always to be included in a CSR or TR.

It is important that data is collected as soon as possible after the problem has occurred as the relevant data can be lost if this activity is postponed.

3.1   General Data to Be Collected

The following types of data are to be collected:

• Version information of the MTAS product and other relevant nodes.
• A detailed description of the problem and for which scenarios the problem has been observed.
• If known, a detailed step by step description of how the fault can be invoked.
• Information about alarms and notifications that can be related to the observed problem.
• References to other CSR or TR problems, which have been observed when the fault occurred.
• Complete node configuration, exported from the Configuration Management browser.
• Information about recent configuration changes, software upgrades, and similar activities that have been performed.
• Values of MTAS counters *Error, *Failed, *NoKE, and ‘ *NoKI; Refer to section Checking Counters in MTAS Health Check.
• Any crashcollector files that can be related to the observed problem.

3.2   Data Collected Based on Specific Problems

This section describes different types of data based on specific problem types that can be collected. The data types to be collected and included in a CSR or TR depends on the problem experienced.

This section describes different types of data that can be collected. The problem experienced decides which of the following data types is to be included in a CSR or TR.

3.2.1   Software Versions

The version information for the MTAS software components is to be collected.

3.2.2   MTAS Log Files

The logging information that is to be collected is described in Table 2.

Table 2    MTAS Log Files

Filename	Log Path
vDicos Virtual Machine Log	SC <x>:/cluster/storage/no-backup/cdclsv/log/lpmsv
Processor Log	SC <x>:/var/log/SC-<X>/messages SC <x>:/var/log/PL-<Y>/messages
MTAS Application Log	SC <x>:/storage/no-backup/coremw/var/log/saflog/MTASAppLogs/vdicos/MTAS_<from-date_to-date>.log
Crash collector Log	SC <x>:/cluster/storage/no-backup/cdclsv/cadump/
MTAS Catalina Log	PL <y>:/opt/mmas/appserver/traffic_instance<N>/logs/catalina.log SC <x>:/var/log/PL-<Y>/mmas/appserver/traffic_instance<N>/catalina.log
MTAS Catalina Out Log	PL <y>:/opt/mmas/appserver/traffic_instance<N>/logs/catalina.out
Access Logs	SC <x>:/storage/no-backup/coremw/var/log/saflog/MMASAccessLogs/
CAI3G Log (AuditLog)	SC <x>:/storage/no-backup/coremw/var/log/saflog/MMASAuditLogs

Where:

• <X> is the SC number, 1 or 2.
• <Y> is the PL number, starting with 3 and increases by 1.
• <N> is the instance number, starting with 0 and increases by 1 for every PL; 0 for PL-3, 1 for PL-4, and so on.

3.2.3   Routing Information

The following logging information is to be collected:

• Routing Information on the SC and PL processors by running the route-n command on each of the PLs and SCs.
• Configuration of Evolved Virtual IP (eVIP), verified from the
  /storage/system/config/evip-apr9010467/evip.xml file.

3.2.4   Alarms, Notifications, and Events

The MTAS triggers alarms for the most critical events that require operator intervention. The alarm information is accessible through the command lde-alarm. Alerts are also triggered to report relevant events for the operator.

Alarms and notifications information can be found in the FaultManagementLog/alarm and FaultManagementLog/alert directories that are stored on the SC under:
/cluster/storage/no-backup/coremw/var/log/saflog/.

Operating Instructions (OPIs) for each event describe the actions to be taken to cease alarms.

A list of the different alarms generated by the MTAS can be found in MTAS Alarm List.

3.2.5   Performance Measurement Counters

Performance Measurement (PM) counters are useful problem indicators. In particular, this includes the PMF counter reports from the period when the faulty situation occurred.

These counters are available with COM File Management in PerformanceManagementReportFiles file group.

Note:  

For more information on how to transfer counter-files, refer to File Management.

Alternatively, it is also possible to collect only counter-information for the service to be reported.

3.2.6   Clients Used

The following must be provided:

• A list of the clients and versions used.

3.2.7   Surrounding Nodes

The following must be provided:

• A list of the surrounding node versions, both Ericsson and other vendors.

3.2.8   Traffic Scenarios

The following must be provided:

• A detailed description of the faulty traffic scenario.
• Network trace (PCAP) including the faulty sequence (if the problem can be repeated) together with information on what packet in the trace being evidence of the incorrect MTAS behavior.
• Specify what the expected traffic behavior of MTAS is, together with a reference to the standard specification or MTAS Function Specification defining the expected behavior.
• Specify for what MTAS Application Server role the problem is experienced (for example MMTel Telephony AS, SCC-AS)
• MTAS service data (XML) provisioned on the subscriber affected by the faulty scenario.
• Application trace using an appropriate trace profile as recommended by MTAS AppTrace (if the problem can be repeated).

3.2.9   Network Configuration

The following must be provided:

• A description of the network configuration.

3.2.10   Configuration Parameter Values

The following must be provided:

• The configuration data exported from IMM using the following command:

  cmw-immconfig-export <filename>

3.2.11   Co-located Applications

The following must be provided:

• A list of all the applications and their versions that are co-located on the same node as the MTAS.

3.2.12   AppTrace

Valuable information can be obtained by using the AppTrace function in the MTAS.

For more information about AppTrace, refer to MTAS AppTrace.

3.2.13   Trace Pcap

Traces generated from Wireshark™ or other tracing tools are to be included in a CSR or TR depending on the specific problem type.

3.2.14   License Manager Log Files

License Manager Functionality related events are stored in log files, which are located in the following path of the cluster:

/storage/clear/lm-apr9010503/log/lm.SC-<X>.log

where <X> is the number designation of the processor.

4   Data Collection Use

4.1   MTAS Data Collection Feature

The data collection tool is intended to collect data that is needed to attach to a CSR, and to collect other useful information, if available in an acceptable amount of time and effort.

The data collection tool is composed by several steps. Steps are specific scripts covering different parts of the MTAS software. Steps are reading states, configurations, and produce logs; or, they collect already existing logs into a given place.

The majority of the produced logs are readable for the user; for example, EVipAndRouting collects the status of the eVIP controller and the status of ports into a table.

All steps are grouped in the following five predefined profiles in data collection: Basic, Medium, Large, Full, and SLA.

Some scripts play a central role; they control the execution of steps and they compress the newly produced logs.

Depending on the selected steps/profile, the execution of Data Collection can take several minutes.

The results of the running data collection are stored in:
/cluster/storage/no-backup/dc/ .

4.2   Start Data Collection

4.2.1   Data Collection Using CDCLS

Data collection using the CDCLS (Crash Dump and Console Log Collection Service) is performed by executing packer objects for data collection. The Data Collection profile packers are listed with cdclsv-list-packers and executed with cdclsv-pack.

To start data collection using CDCLS:

1. Check which profiles are available:

   cdclsv-list-packers | grep cdclsPk=DcMtas

   The following results are shown:

   cdclsPk=DcMtasBasic,cdcls=CDCLSvSite
   cdclsPk=DcMtasFull,cdcls=CDCLSvSite
   cdclsPk=DcMtasLarge,cdcls=CDCLSvSite
   cdclsPk=DcMtasMedium,cdcls=CDCLSvSite
   cdclsPk=DcMtasSla,cdcls=CDCLSvSite

2. Select one of the following profiles:

   DcMtasBasic
   DcMtasMedium
   DcMtasLarge
   DcMtasFull
   DcMtasSla

3. Start data collection with the selected profile:

   cdclsv-pack cdclsPk=<profile>,cdcls=CDCLSvSite

   The following is an example:

   cdclsv-pack cdclsPk=DcMtasBasic,cdcls=CDCLSvSite

4. Check the status of the packing:

   cdclsv-pack-status cdclsPk=DcMtasBasic,cdcls=CDCLSvSite

   The results are found in /cluster/storage/no-backup/dc/.

4.2.2   Data Collection Using Console

For data collection using console, super user rights are required.

To start data collection using console:

1. Execute dcProfileHandler from SC or PL with the proper parameters:

   ./dcProfileHandler --profile=<profile> --output=<output directory> <steps>

   The following is an example:

   ./dcProfileHandler --profile=XXYY --output=
   /cluster/storage/no-backup/dc EVipAndRouting SS7Connections

The following are mandatory parameters:

• Profile: Name of the desired profile. One of the predefined profile names or a free string.
• Output: The path where the results are added to.
• Steps: A list with whitespaces.

For details of other parameters, run dcProfileHandler without parameters:

./dcProfileHandler
Usage: ./dcProfileHandler [--prefix=<prefix>] <--profile=<profile>>
<--output=<output directory>> [--stepMaxSize=<log size>] [--logLastHours=<number of hours>]
[--stepDelay=<number of seconds>] [--HC] [Steps...]

Although profiles can be selected from the console, it is possible to execute any number of steps as well.

The results are found in /cluster/storage/no-backup/dc/.

4.2.3   Data Collection Using MTAS Health Check

Health check calls the required steps of data collection, and analyzes the produced logs. At the end of the analysis, health check chooses a verdict from the following list:

• PASS
• FAIL
• INFO
• ERROR
• VERIFY

For more information on these verdicts, refer to MTAS Health Check.

If the verdict is PASS, the logs from data collection are removed.

If the verdict is FAIL, INFO, ERROR, or VERIFY, the logs remain available for further analysis.

4.3   Profiles and Steps in Data Collection

Every profile listed in Table 3 through Table 6 contains all of the steps mentioned in the preceding one, plus those enumerated in its own table.

4.3.1   Basic Profile

Table 3 lists the basic profile.

Table 3    Basic Profile

Step	Function	Main Commands in the Script of the Step	Related HC Profile / Step
BasicSysInfo	The following is collected: History of the commands executed by root user (history) List of current mount points (mount) Loaded modules (lsmod) NTP status (ntpq -p) Active network connections (netstat -n) Listening sockets (netstart -tulnp) Process list and tree (ps -Afww ,pstree -p) List of cached libraries (ldconfig -v)	history, mount lsmod ntpq -p netstat -n netstat -tulnp ps -Afww, pstree -p ldconfig -v	None
CpuLoad	Creates a table with the current CPU-load for every node.	mpstat -P ALL	MediumPriority / CpuLoadOnSCs, MediumPriority / CpuLoadOnPLs
MemoryUsage	Creates a table with the current memory use for every node.	free -m	MediumPriority / MemoryUsageOnSCs, MediumPriority / MemoryUsageOnPLs
DiskUsage	Creates a table with the current disk use for every SC node.	df -x tmpfs -x devtmpfs	MediumPriority / DiskUsageOnSCs
DrbdStatus	Reads out the DRBD(1) status from the /proc/drbd file on every SC node.	copy	Mandatory / DrbdStatus
SoftwareInventory	Lists the available RPM(2)/SDP(3) files from /cluster/storage/system/software/lpmsv/codearchive/.	ls	LowPriority / SoftwareInventory
VmLogs	Collects logs from the Virtual Machines.	copy, cmw-collect-info, collect_mmas_info	LowPriority / VmLogs
ProcessorLogs	Collects "Linux system logs" from every node in the cluster.	copy	None
DumpFiles	Copies files from /opt/cdclsv/storage/dumps to DC logs.	copy	None
CapsuleAbortions	Collects CA logs from /opt/cdclsv/storage/cadump/.	copy	None
Alarms	Collects the list of alarms from every host in the cluster.	/usr/sbin/lde-alarm --status --full --all	None
NodeOutage	Collects recovery events from /cluster/storage/no-backup/coremw/var/log/saflog/isplog/*	copy	Mandatory / NodeOutage
LmScLogs	Collects LM-, SC logs from /storage/clear/lm-apr9010503/log/	copy	None

(1)  Distributed Replicated Block Device (DRBD)

(2)  Router Policy Manager (RPM)

(3)  Session Description Protocol (SDP)

4.3.2   Medium Profile

The medium profile covers all areas from the basic profile, and the areas listed in Table 4.

Table 4    Medium Profile

Step	Function	Main Commands in the Script of the Step	Related HC Profile / Step
DiskStatus	Collects status of disks from every SC node.	hdparm, dmsetup info, dmsetup status, iostat	None
MtasTrafficPoolCpuLoad	Creates a table with the current CPU-load for every PL node.	mpstat -P ALL	None
MtasTrafficPoolMemoryUsage	Creates a table with the current memory use for every PL node.	/proc/meminfo, top -n 1 -b	None

4.3.3   Large Profile

The large profile covers all areas from the medium profile, and the areas listed in Table 5.

Table 5    Large Profile

Step	Function	Main Commands in the Script of the Step	Related HC Profile / Step
UpgradeList	Creates a list of available campaign files.	cmw-repository-list --campaign	LowPriority / UpgradeList
BackupList	Lists the parameters of the available backup.	lde-brf	LowPriority / BackupList
SignalingStatus	Collects SS7 logs from /opt/sign/.	copy	None
MtasLogs	Collects MTAS-logs from /cluster/storage/no-backup/coremw/var/log/saflog/.	copy	None

4.3.4   Full Profile

The full profile covers all areas from the large profile, and the areas listed in Table 6.

Table 6    Full Profile

Step	Function	Main Commands in the Script of the Step	Related HC Profile / Step
AIT	Collects AIT(1) logs from the following folder: /cluster/storage/no-backup/ait-apr9010496_1.	copy	None
AlarmsAndNotifications	Lists alarm statistic from /storage/no-backup/coremw/var/log/saflog/.	copy	Mandatory / AlarmsAndNotifications
AllMtasPortsStatus	Creates a table with the MTAS ports. The predefined port numbers are as follows: mtasXdmsXCAPPort: 8090 mtasXdmsCCMPPort: 8096 mtasSoapPort: 9080 mtasXdmsCai3GSecurePort: 8443 mtasXdmsCai3GPort: 8095	netstat -an	MediumPriority / AllMtasPortsStatus
CmData	Collects CM configuration into a CmDataLog.xml file.	cmw-immconfig-export	none
CoreMWStatus	Returns with the status of CoreMW.	amf-state	Mandatory / CoreMWStatus
DiameterPortsStatus	Lists the diameter ports with their properties (port number, IP address, and SCTP address).	netstat -apn -A inet,inet6	MediumPriority / DiameterPortsStatus
EVipAndRouting	Collects the routing tables, and the eVIP tables from every ALB(2).	telnet `/opt/vip/bin/getactivecontrol` 25190 show albs show agents [name of alb]	Mandatory / Evip
MIMMOMFiles	Copies the xml files from /opt/com/etc/model into DC logs.	copy	None
Mmas	Collects mmas logs and collects logs from /opt/mmas/appserver/ from PL nodes.	collect_mmas_info	Mandatory / Mmas
NETCONFConnection	Collects the network configuration from every SC node.	netcat localhost 9977	Mandatory / NETCONFConnection
NetworkConnectivity	Checks the communication between SC and PLs one by one.	ping	Mandatory / NetworkConnectivity
NodeConfiguration	Copies the following files from every node into DC logs: xml- files from /opt/cdsv/storage/current/vdcgp_config/*.xml. config-files from /cluster/etc/cluster.conf. config-files from /cluster/nodes/[node]/etc/rpm.conf (list of RPMs).	copy	None
OperationalState	Returns the mtasFunctionAdministrativeState from CM.	read CM through CLI	Mandatory / OperationalState
PMReportFiles	Collects and copies PM report files into the DC logs (from the folder /cluster/storage/no-backup/com-apr9010443/PerformanceManagementReportFiles/). The PM jobs must be started before any PM report files are created.	copy	None
ProcessorLogs	Collects processor-logs (messages-log) from /var/log/.	copy	None
SIPPortsStatus	Lists the IP ports used by SIP and their properties.	netstat -an	Medium / SIPPortsStatus
SS7Connections	Creates a table with the SS7 connections and their properties from /opt/sign/etc/signmgr.cnf	/opt/sign/EABss7050/bin/signmcli	Mandatory / SS7Connections
SecurityStatus	Lists the installed RPMs containing security SW parts.	cmw-rpm-list $node | grep -i sec	LowPriority / SecurityStatus
SoftwareVersionsInstalled	Lists the installed SW components and their versions.	cmw-repository-list	LowPriority / SoftwareVersionsInstalled
SoftwareVersionsRunning	Lists the running SW components and their versions.	cmw-repository-list	LowPriority / SoftwareVersionsInstalled
SystemEnvironmentVariables	Lists the vDicos environmental variables.	vdicos-envdata-get [envEntry]	LowPriority / SystemEnvironmentVariables
SystemStatus	Results a verdict of the following commands: cmw-status app cmw-status csiass cmw-status comp cmw-status node cmw-status sg cmw-status si cmw-status siass cmw-status su cmw-status pm	cmw-status	Mandatory / SystemStatus
VirtualDicosProcessOutage	Creates a detailed list of VMs(3) and their status.	clurun.sh -c vmstatus	Mandatory / VirtualDicosProcessOutage
XdmsInstance	Results a status of the "cmw-status app"	cmw-status app	Mandatory / XdmsInstance
XdmsRpm	Lists the MMAS and XDMS-related packages.	rpm -qa	MediumPriority / XdmsRpm

(1)  Automatic Installation Tool (AIT)

(2)  Abstract Load Balancer (ALB)

(3)  Virtual Machine (VM)

4.3.5   SLA Profile

Note:  

The Service Level Agreement (SLA) profile is for internal use only.

Table 7 lists the SLA profile.

Table 7    SLA Profile

Step	Function	Main Commands in the Script of the Step	Related HC Profile / Step
SLA	Does the following: Verifies the status of the SLA and records the KPI(1) for the VM, Core and, Network Interface under the SLA directory for the last hour. Moves the MtasSla PM Report files periodically to an internal location: /storage/no-backup/MtasSla_PmReportFiles/ for consumption by HealthCheck. Deletes MtasSla files older than a day in the periodical housekeeping.		Mandatory / SLA

(1)   Key Performance Indicator (KPI)

5   Other Useful Information

Other useful information can be included in a CSR or a TR if it is easily available and there is enough time to collect it. An example of useful information is subscriber data, which is collected through the Business Support System.