IPWorks Trace User Guide

Contents

1Introduction
1.1Intended Audience
1.1.1Prerequisite Knowledge
1.2Prerequisites

2
Trace Overview
2.1Configuration Management

3
Use Trace
3.1Enabling Trace Feature
3.2Creating Trace Profile
3.3Activating Trace Profile
3.4Live Viewing Trace Profile
3.5Deactivating Trace Profile
3.6Disabling Trace Feature
3.7Converting Trace Logs
3.8Deleting Trace Profile and Trace Recording
3.8.1Delete Trace Profile
3.8.2Delete Trace Recording

4
IPWorks Trace Event
4.1AAA Diameter
4.2AAA Radius

5
Characteristics

Reference List

1   Introduction

This document describes how to perform subscriber tracing using the trace functionality in IPWorks system.

1.1   Intended Audience

This guide is intended for support personnel who are troubleshooting applications with Trace.

1.1.1   Prerequisite Knowledge

User of this document needs to have knowledge and experience of the following:

1.2   Prerequisites

Working with Trace involves a set of Command-Line Interface commands that allow you to perform Trace operations.

Attention!

Trace CLI commands require elevated privileges. These commands must be executed as root, or a member of the system-ts group using sudo.

2   Trace Overview

Trace provides support for cluster-wide userspace tracing in a Component Based Architecture (CBA) environment. It uses the Linux Trace Toolkit - next generation (LTTng) Userspace Tracer (UST) to collect and process information from traceable software components and applications that are running on the system.

Software tracing is inactive during normal system operation and no trace information is generated. When a problem arises, tracing can be activated to collect additional information that can help troubleshoot the issue.

To start a trace, the troubleshooter activates a Trace Profile that specifies the trace information to collect. When an active tracepoint or logger statement is hit during code execution, it passes information to the userspace tracer. The tracer collects data from participating applications and streams it to the cluster file system where it is recorded in binary log files. These log files remain open until the Trace Profile is deactivated and the final data have been stored.

While a Trace Profile is active, the tracer can be connected to a live viewer. Live view provides immediate access to trace data by monitoring a specified node for trace activity and printing the trace information as it is received.

For more information, refer to Trace User Guide.

2.1   Configuration Management

The Trace information model is an internal model that is integrated with the Core Middleware Information Model Management (IMM) service during installation time. It is divided into the following objects and each class includes a unique set of attributes:

Note:  
For Trace Manager, the default trace manager settings are recommended.

For more information on how to configure the behavior of Trace, such as storage limitation for trace logs, or maximum number of trace session, etc, refer to Trace User Guide.

3   Use Trace

This section describes how to use Trace in IPWorks and the whole procedure using Trace involves the following activities:

  1. Enable Trace feature for IPWorks application. For more information, see Section 3.1 Enabling Trace Feature.
  2. Create a Trace Profile that specifies the trace events and loggers to enable. For more information, see Section 3.2 Creating Trace Profile.
  3. Activate the Trace Profile to start a Trace Recording session. For more information, see Section 3.3 Activating Trace Profile.
  4. If necessary, use live view to monitor trace activity on one or more nodes in the Trace Recording session. For more information, Section 3.4 Live Viewing Trace Profile.
  5. Deactivate the Trace Profile to stop the Trace Recording when enough data has been collected. For more information, Section 3.5 Deactivating Trace Profile.
  6. Disable Trace feature for IPWorks application. For more information, see Section 3.6 Disabling Trace Feature.
  7. Convert the trace logs from the binary Common Trace Format (CTF) to readable ASCII. For more information, see Section 3.7 Converting Trace Logs.

Each operation is conducted with CLI commands as described in the following sections. For more information about how to use Trace, refer to the document Trace User Guide.

3.1   Enabling Trace Feature

For IPWorks Radius and EPC AAA applications, the switch of trace feature is provided. For other IPWorks applications, the trace feature is always enabled.

To enable trace feature, do the following:

  1. Log on to the active SC node.

    #ssh <Username>@<MIP_OAM_IP>

    Password:<Password>

  2. Start an ECLI session.

    #/opt/com/bin/cliss

    For information about how to use ECLI, refer to Ericsson Command-Line Interface User Guide.

  3. Enable trace for Radius and Diameter by using ECLI. 
    >ManagedElement=<Node Name>,IpworksFunction=1,IpworksCommonRoot=1,IpworksTrace=1
(IpworksTrace=1)>configure
(config-IpworksTrace=1)>enableDiameterTrace=true
(config-IpworksTrace=1)>enableRadiusTrace=true 
(config-IpworksTrace=1)>commit
(IpworksTrace=1)>show -v
IpworksTrace=1
   enableDiameterTrace=true
   enableRadiusTrace=true
   ipworksTraceId="1" <default>
(IpworksTrace=1)>exit

    In this example, the trace feature is enabled for both Radius AAA and EPC AAA applications.

3.2   Creating Trace Profile

To create a Trace Profile:

  1. Log on to System Controller Node SC-1 or SC-2.
  2. Create Trace Profile.

    # tracecc-profile-create <profileName> -t '<Domain>[:<Event>], , <Conditional_Filters>''

    Where:

    • <profileName> : The string represents profile name.
    • <Domain>: The trace domain name for application and the available names are shown in Table 1.
    • <Event>: This parameter is optional. The trace event name for application and the available names are shown in Table 1.
    • <Conditional_Filters>: It helps narrow the scope of events captured in trace logs, and the available fields which could be applied in conditional filters are shown in Table 2. For more information on how to work with conditional filters, refer to section Confitional_Filters in Trace User Guide.
      Warning!

      Without the filter, the trace information for all the users over all interfaces will be recorded. This has big impact on traffic performance.

    Example 1: Trace subscriber “460119991000063” over all interfaces

    # tracecc-profile-create test1 -t 'com_ericsson_ipworks_aaa_diameter, , User=="460119991000063" '

    Example 2: Trace subscriber “460119991000063” over “SWm” interface

    # tracecc-profile-create test2 -t 'com_ericsson_ipworks_aaa_diameter, , User=="460119991000063" && Interface=="SWm" '

  3. Check the Trace Profile to make sure no error exists.

    # tracecc-profile-view <profileName>

    Example:

    SC-1:~ # tracecc-profile-view test
captureContextList : ""
consistentLogs : NOT_ENFORCED
contextFilter : ""
description : ""
latestSessionId : 2
liveviewAddress : 
logLevelDefault : TRACE_DEBUG_SYSTEM(7)
participantNode : ""
rateLimit : -1 (default: 100 KByte/s, shared)
resumeSessionOnReboot : INACTIVE
state : INACTIVE
subBufferNumber : TRACE_DEFAULT:4
subBufferSize : TRACE_DEFAULT:16
traceExpressions : com_ericsson_ipworks_aaa_diameter,,Interface == "SWx"&& User == "11111111111"
-------------------------------- 
progress.actionName : deactivateSession
progress.additionalInfo : ""
progress.info : Trace deactivation is complete
progress.percentage : 100
progress.result : SUCCESS
progress.resultInfo : Trace deactivateSession was completed successfully
progress.state : FINISHED
progress.timeActionCompleted : 01/26/17T07:00:12+0100
progress.timeActionStarted : 01/26/17T07:00:10+0100
progress.timeOfLastStatusUpdate : 01/26/17T07:00:12+0100

3.3   Activating Trace Profile

Activating a Trace Profile starts a new Trace Recording session. Only one session can be active for each Trace Profile at the same time.

When a Trace Profile is successfully activated, a new Trace Recording is created in the ACTIVE state. This Trace Recording is controlled by actions on the Trace Profile until the profile is deactivated. At deactivation, the Trace Recording is brought to the COMPLETE state and can be controlled directly.

Trace Recordings have the same name as the activated Trace Profile plus a sequential number, as follows:

<profileName>_<sessionNumber>.

Where:

To activate the Trace Profile:

  1. Log on SC node SC-1 or SC-2.
  2. If necessary, list all Trace Profiles to determine the name of the profile to activate:

    # tracecc-profile-list

  3. Start at least one corresponding AAA server, for example:
    • For EPC AAA server:

      # ipw-ctr start aaa_diameter PL-3

    • For Radius AAA server:

      # ipw-ctr start aaa_radius_backend PL-3

      # ipw-ctr start aaa_radius_stack PL-3

  4. Activate one Trace Profile:

    # tracecc-profile-activate <profileName>

You can refer to the document Trace User Guide to get additional information on usage of Trace Recordings.

3.4   Live Viewing Trace Profile

Live view displays data from an active Trace Recording session in real time. Each live view instance monitors a specific node for trace activity and prints the trace events to standard output as they are hit. Multiple instances of live view can be used to monitor additional nodes simultaneously; however, only one live view instance is allowed per node of the same profile.

Note:  
Live view only prints trace data that is collected while the viewer is active. Data that is collected before live view starts is available in the trace logs.

To live view a Trace Profile:

  1. Log on SC node SC-1 or SC-2.
  2. If necessary, retrieve a list of available nodes for the selected profile by entering tracecc-profile-liveview without specifying a node:

    # tracecc-profile-liveview <profileName>

  3. Start a live view instance of the Trace Recording session, for example:

    # tracecc-profile-liveview -n PL-3 <profileName>

    Example output:

    [07:49:13.689081205] com_ericsson_ipworks_aaa_diameter:DiameterTrace: { 0 }, { "ipwa3d", 5907, 6004 }, { Interface = "S6b", User = "460119991000063", Class = "DiameterStack", Function = "otpdiaHandleRequest", AdditionalInfo = "
DiameterMessage:
<msg function='S6b' name='AAR'>
<ie name='Session-Id'>nas1.nas.com;1701231449;09612491;60119991000063</ie>
<ie name='Origin-Host'>nas1.nas.com</ie>
<ie name='Origin-Realm'>nas.com</ie>
<ie name='Destination-Realm'>ipworks.com</ie>
<ie name='Auth-Application-Id'>16777272</ie>
<ie name='Auth-Request-Type'>2</ie>
<ie name='User-Name'>460119991000063@nai.epc.mnc011.mcc460.3gppnetwork.org</ie>
<ieGroup name='MIP6-Agent-Info'>
    <ieGroup name='MIP-Home-Agent-Host'>
        <ie name='Destination-Host'>topoff.pgw-s2a.gw01-B-er.hz.zj.node.epc.mnc011.mcc460.3gppnetwork.org</ie>
        <ie name='Destination-Realm'>epc.mnc011.mcc460.3gppnetwork.org</ie>
    </ieGroup>
</ieGroup>
<ie name='Service-Selection'>ctnet</ie>
<ie name='MIP6-Feature-Vector'>1099511627776</ie>
</msg>
" }
[07:49:13.689344525] com_ericsson_ipworks_aaa_diameter:DiameterTrace: { 0 }, { "ipwa3d", 5907, 6004 }, { Interface = "S6b", User = "460119991000063", Class = "DiameterStack", Function = "otpdiaHandleRequest", AdditionalInfo = "Store Diameter Request into Message Queue Sucessfully" }
[07:49:13.689724526] com_ericsson_ipworks_aaa_diameter:DiameterTrace: { 0 }, { "ipwa3d", 5907, 5973 }, { Interface = "S6b", User = "460119991000063", Class = "S6bAARMessageHandler", Function = "execute", AdditionalInfo = "enter" }
[07:49:13.689973621] com_ericsson_ipworks_aaa_diameter:DiameterTrace: { 0 }, { "ipwa3d", 5907, 5973 }, { Interface = "S6b", User = "460119991000063", Class = "S6bAARMessageHandler", Function = "execute", AdditionalInfo = "Checking AVP Result is 0" }

3.5   Deactivating Trace Profile

Deactivating a Trace Recording session ends data collection and sends the remaining trace information to trace log files in the globalOutputDirectory.

To view the value of globalOutputDirectory, refer to section globalOutputDirectory in Trace User Guide.

Once the Trace Recording session has been deactivated, the binary trace output can be converted to a readable ASCII format to start troubleshooting.

To deactivate trace profile:

  1. Log on SC node SC-1 or SC-2.
  2. Deactivate the trace profile:

    # tracecc-profile-deactivate <profileName>

3.6   Disabling Trace Feature

For IPWorks Radius and Diameter AAA applications, the switch of trace feature is provided. For other IPWorks applications, the trace feature is always enabled.

To disable trace feature, do the following:

  1. Log on to the active SC node.

    #ssh <Username>@<MIP_OAM_IP>

    Password:<Password>

  2. Start an ECLI session.

    #/opt/com/bin/cliss

    For information about how to use ECLI, refer to Ericsson Command-Line Interface User Guide.

  3. Disable trace for Radius/Diameter by using ECLI. 
    >ManagedElement=<Node Name>,IpworksFunction=1,IpworksCommonRoot=1,IpworksTrace=1
(IpworksTrace=1)>configure
(config-IpworksTrace=1)>enableDiameterTrace=false
(config-IpworksTrace=1)>enableRadiusTrace=false
(config-IpworksTrace=1)>commit
(IpworksTrace=1)>show -v
IpworksTrace=1
   enableDiameterTrace=false
   enableRadiusTrace=false
   ipworksTraceId="1" <default>
(IpworksTrace=1)>exit

    In this example, the trace feature is disabled for both Radius AAA and EPC AAA applications.

3.7   Converting Trace Logs

To convert trace log:

  1. Log on SC node SC-1 or SC-2.
  2. If necessary, list all Trace Recordings to determine the name of the recording to convert:

    # tracecc-recording-list

  3. Convert trace recording:

    # tracecc-recording-convert <recordingName>

  4. The conversion result is stored in <globalOutputDirectory>/<sessionDirectory>/Converted_logs for the Trace Recording, check whether the content is formatted as ASCII.

    Use the below command to find out <globalOutputDirectory>:

    #tracecc-manager-view
globalOutputDirectory           : /storage/no-backup/tracecc-apr9010500/
globalStorageLimit              : 256
globalStorageLowerThreshold     : 75
globalStorageUpperThreshold     : 95
maxStoredSessions               : 10
--------------------------------
progress.actionName             : listTraceEvents
progress.additionalInfo         : ""
progress.info                   : ""
progress.percentage             : 0
progress.result                 : NOT_AVAILABLE
progress.resultInfo             : ""
progress.state                  : RUNNING
progress.timeActionCompleted    : ""
progress.timeActionStarted      : 04/05/17T04:58:03+0200
progress.timeOfLastStatusUpdate : 04/05/17T04:58:03+0200

For more information about how to work trace logs, refer to section Working with Trace Logs in Trace User Guide.

3.8   Deleting Trace Profile and Trace Recording

3.8.1   Delete Trace Profile

Note:  
Only profiles in state INACTIVE can be deleted.

To delete the Trace Profile, do the followings:

  1. Use SSH to connect to SC-1 or SC-2.
  2. If required, list all the Trace Profiles to determine the name of the profile to delete:

    #tracecc-profile-list

  3. If required, verify that the profile state is INACTIVE:

    # tracecc-profile-view [OPTION...] <profileName>

    For example:

    # tracecc-profile-view -a state Profile1

    If the profile state is ACTIVE, deactivate it firstly:

    # tracecc-profile-deactivate <profileName>

  4. Delete the trace profile.

    # tracecc-profile-delete [OPTION...] <profileName>

    For example:

    # tracecc-profile-delete Profile1

    The Trace Profile is deleted.

If you want to know more about how to delete the trace profile, refer to the section Deleting a Trace Profile in Trace User Guide.

3.8.2   Delete Trace Recording

Note:  
Only Trace Recordings in state COMPLETE can be deleted.

To delete the Trace Recording, do the followings:

  1. Use SSH to connect to SC-1 or SC-2.
  2. If required, list all Trace Recordings to determine the name of the recording to delete:

    # tracecc-recording-list

  3. If required, verify that the Trace Recording state is COMPLETE:

    # tracecc-recording-view [OPTION...] <recordingName>

    For example:

    #tracecc-recording-view -a state Profile1_1

    If the state is not COMPLETE, deactivate the trace profile firstly.

    # tracecc-profile-deactivate <profileName>

  4. Delete the trace recording.

    # tracecc-recording-delete <recordingName>

    For example:

    # tracecc-recording-delete Profile1_1

    The Trace Recording and all associated log files are deleted.

If you want to know more about how to delete the trace recording, refer to the section Deleting a Trace Recording in Trace User Guide.

4   IPWorks Trace Event

4.1   AAA Diameter

This section is the introduction for Trace Event for AAA Diameter, You can see the domain name, event name for AAA Diameter in Table 1. The fields and values which could be applied in conditional filters for AAA Diameter are in Table 2.

Table 1    IPWorks Trace Event

Domain

com_ericsson_ipworks_aaa_diameter

Event

DiameterTrace

Table 2    Conditional Filter Fields and Values

Field Name

Field Value

User

The user name or IMSI without prefix.

Interface

The value could be “STa” or “SWm” or “S6b” or “SWx” or "S13" and it is case sensitive.

4.2   AAA Radius

This section is the introduction for Trace Event for AAA Radius, You can see the domain name, event name for AAA Radius in Table 3. The fields and values which could be applied in conditional filters for AAA Radius are in Table 4.

Table 3    IPWorks Trace Event

Domain

com_ericsson_ipworks_aaa_radius

Event

RadiusTrace

Table 4    Conditional Filter Fields and Values

Field Name

Field Value

User

The user name or IMSI without prefix.

Interface

The value could be "PAP_CHAP" or "EAP_SIM_AKA" or "D_Gr" or "LDAP_CUDB" or "Accounting" or "Proxy" or "DM_CoA", and it is case sensitive.

5   Characteristics

The maximum number of Trace profiles activated simultaneously is 10. The test shows that the IPWorks system performance degrades slightly when 10 Trace profiles are activated and each profile traces one subscriber.

Note:  
It is recommended to apply User filter on Trace profile with limited number of subscribers. Otherwise, the IPWorks system performance might degrade significantly.


Reference List

Ericsson Documents
[1] Trace User Guide, 1/1553-APR 901 0500/3 Uen
[2] Trace Command Line Interface (CLI) Reference, 1/1540-APR 901 0500/3 Uen
[3] Ericsson Command-Line Interface User Guide.


Copyright

© Ericsson AB 2018. All rights reserved. No part of this document may be reproduced in any form without the written permission of the copyright owner.

Disclaimer

The contents of this document are subject to revision without notice due to continued progress in methodology, design, and manufacturing. Ericsson shall have no liability for any error or damage of any kind resulting from the use of this document.

Trademark List
All trademarks mentioned herein are the property of their respective owners. These are shown in the document Trademark Information.

    IPWorks Trace User Guide