Performance Data Collection

Ericsson Service-Aware Policy Controller

Performance Data Collection Introduction
Configuration Prerequisites
Performance Data Collection Function
PDC Operational Conditions
PDC Report Data Management
- Daily and Monthly Reports
  - Daily Reports
  - Monthly Reports
- Health Check Reports

1 Performance Data Collection Introduction

The purpose of this document is to provide a guideline to manage the Performance Data Collection (PDC) function in the SAPC.

2 Configuration Prerequisites

Before configuring the SAPC in an operational network, assure that:

CBA Components are installed.
The SAPC product software is installed.
To have a detailed understanding of the function.

3 Performance Data Collection Function

The Performance Data Collection (PDC) function of the SAPC extracts performance information which can be used for performance analysis, dimensioning, and troubleshooting.

The PDC function takes as input:

Performance Management Files (PMFs) of the SAPC. They provide measures related to hardware resources consumption, and measures of parameters that are specific to the SAPC business logic.
Miscellaneous information: number of payload nodes and controller nodes in the cluster, status of the geographical redundancy, virtualization technology used, and the SAPC software version.
Node status information such as, raised alarms, active licenses, and enabled protocol interfaces.

The input information is collected and written into output files, that can be retrieved from the SAPC through its SFTP interface. Four kinds of output files are produced:

Application reports: Contains measures of parameters that are specific to the SAPC business logic
Platform reports: Contains measures of hardware resources consumption
Miscellaneous report: Contains miscellaneous information of the current status of the SAPC.
Heath Check report: Contains node status, node information, and configuration data.

The SAPC is scheduled to:

Execute the PDC function daily.
Create a package at the end of every month containing aggregated information from all report files generated during the month.

The PDC function and health check can be executed manually at any time.

4 PDC Operational Conditions

4.1 PDC Function Administration

4.1.1 Enable Daily Data Collection and Monthly Aggregation

Note:

Daily Performance Data Collection and Monthly Aggregation are enabled by default in the SAPC. The following steps are only needed if the collection has been previously disabled.

To enable the daily execution of the PDC function in the SAPC, and the monthly aggregation of produced files:

Steps

Access to the SAPC executing the following command:
> ssh sapcadmin@<OAM_VIP>
Enable daily execution with the following command:
sapcadmin@SC-X> sapcPdc mode -enable

4.1.2 Disable Daily Data Collection and Monthly Aggregation

To disable the daily execution of the PDC function in the SAPC, and the monthly aggregation of produced files:

Steps

Access to the SAPC executing the following command:
> ssh sapcadmin@<OAM_VIP>
Disable daily execution with the following command:
sapcadmin@SC-X> sapcPdc mode -disable

4.1.3 Manual Data Collection

The PDC function can be executed manually in the SAPC at any time.

4.1.3.1 Manual Collection by Default

To collect data by default, follow these steps:

Steps

Access to the SAPC executing the following command:
> ssh sapcadmin@<OAM_VIP>
Collect data by default executing the following command:
sapcadmin@SC-X> sapcPdc collect

This command generates three daily data reports with the data of the previous day to the execution: miscellaneous, platform, and application. See Daily Reports.

4.1.3.2 Manual Data Collection by Date

To generate reports within a date range, follow these steps:

Steps

Access to the SAPC executing the following command:
> ssh sapcadmin@<OAM_VIP>

Depending on your needs, execute one of the commands described on Table 1.

Table 1 PDC collection by date
Action	Command	Description
Collection by day	sapcPdc collect -day <YYYYMMDD>	The PDC processes PM reports for the specified day and generates a report (no miscellaneous report because it only makes sense in real time). If there are not PM reports for the specified day, reports are not generated for that day.
Collection by interval	sapcPdc collect -interval <YYYYMMDD YYYYMMDD>	The PDC processes PM reports for the specified days within the range and generates a report for each day. If there are not PM reports for a given day, reports are not generated for that day.
Collection by month	sapcPdc collect -month <YYYYMM>	The PDC processes PM reports for each day in the specified month and generates a report for every day. If there are not PM reports for a given day, reports are not generated for that day.

4.1.3.3 Manual Data Collection by Report Type

To generate a report of a specific type (for the previous day to the execution), follow these steps:

Steps

Access to the SAPC executing the following command:
> ssh sapcadmin@<OAM_VIP>

Depending on your needs, execute one of the commands described on Table 2.

Table 2 PDC collection by report type
Report type	Command
Miscellaneous	sapcPdc collect -report misc
Common Components	sapcPdc collect -report platform
Application	sapcPdc collect -report application

4.1.3.4 Manual Data Collection by Date and Report Type

To generate a report of a specific type within a date range, follow these steps:

Steps

Execute following command to access the SAPC:
> ssh sapcadmin@<OAM_VIP>

Depending on your needs, execute one of the commands described on Table 3.

Table 3 PDC collection by date and report type
Action	Command
Collection by day and report type	sapcPdc collect -day <YYYYMMDD> -report {misc \| platform \| application}
Collection by interval and report type	sapcPdc collect -interval <YYYYMMDD YYYYMMDD> -report {misc \| platform \| application}
Collection by month and report type	sapcPdc collect -month <YYYYMM> -report {misc \| platform \| application}

4.1.4 Data Packaging

The PDC processes the report files available and generates a monthly report as tarball output file.

Data packaging includes a cleanup mechanism that automatically deletes PDC trace files and the tarball monthly reports older than one year.

Note:

Before executing data packaging, verify that the execution is needed. Data packaging probably consumes high CPU resources, and must be executed only if necessary.

4.1.4.1 Data Packaging by Default

To package and compress daily files for the previous month, follow these steps:

Steps

Access to the SAPC executing the following command:
> ssh sapcadmin@<OAM_VIP>
Package by default executing the following command:
sapcadmin@SC-X> sapcPdc package

4.1.4.2 Data Packaging by Date

To package daily reports for specific date ranges, follow these steps:

Steps

Access to the SAPC executing the following command:
> ssh sapcadmin@<OAM_VIP>

Depending on your needs, execute one of the commands described on Table 4.

Table 4 Data packaging by date
Action	Command	Description
Packaging by month	sapcPdc package -month <YYYYMM>	Daily report files for a specific month are packaged and compressed, and then, if successful, monthly report files are removed.
Packaging by interval	sapcPdc package -interval <YYYYMMDD YYYYMMDD>	Daily report files within the specific date range are packaged and compressed. If successful, monthly report files are removed.

4.1.4.3 Cleanup

The cleanup option forces deletion of daily report files. The files removed are those used to generate the tarball file.

To clean up daily report files, follow these steps:

Steps

Access to the SAPC executing the following command:
> ssh sapcadmin@<OAM_VIP>

Depending on your needs, execute one of the commands described on Table 5

Table 5 Cleanup options
Command	Description
sapcPdc package -cleanup	The daily report files for the previous month are removed
sapcPdc package -month <YYYYMM> -cleanup	The daily report files for the specified moth are removed
sapcPdc package -interval <YYYYMMDD YYYYMMDD> -cleanup	The daily report files for the specified interval are removed

4.1.5 Health Check

4.1.5.1 Health Check Full Report

To collect the instantaneous status, information, and configuration data of the node execute following steps:

Steps

Access to the SAPC executing the following command:
> ssh sapcadmin@<OAM_VIP>
Collect health check full report executing the following command:
sapcadmin@SC-X> sapcPdc sapcHealthCheck

4.1.5.2 Health Check Report

To collect only the node status:

Steps

Access to the SAPC executing the following command:
> ssh sapcadmin@<OAM_VIP>
Collect health check report executing the following command:
sapcadmin@SC-X> sapcPdc sapcHealthCheck -report healthCheck

4.1.5.3 Node Information Report

To collect only node information as software level and limit capacity license, execute following steps:

Steps

Access to the SAPC executing the following command:
> ssh sapcadmin@<OAM_VIP>
Collect health check node information report executing the following command:
sapcadmin@SC-X> sapcPdc sapcHealthCheck -report nodeInformation

4.1.5.4 Node Configuration Report

To collect only node configuration as enabled capacity license and enabled interfaces, execute following steps:

Steps

Access to the SAPC executing the following command:
> ssh sapcadmin@<OAM_VIP>
Collect health check node configuration report executing the following command:
sapcadmin@SC-X> sapcPdc sapcHealthCheck -report nodeConfiguration

4.2 PDC Configuration

4.2.1 Modify Configuration

Configure the PDC function, executing following steps:

Access to the SAPC executing the following command:

> ssh sapcadmin@<OAM_VIP>
Open /storage/system/config/sapc/pdc.cfg file.
See Table 6 and modify the file depending on your needs.
Save changes in the file.

Table 6 PDC Properties
Property	Description	Definition
daily_processing_hour	Hour and minute when the PDC is launched to generate daily reports.	Type: String Default: 00:06
sapc_measurements_src_path	Paths where the PDC takes SAPC PMs from. Multiple paths must be separated by commas.	Type: String Default: /storage/no-backup/com-apr9010443/PerformanceManagementReportFiles
platform_measurements_src_path	Paths where the PDC takes Common Components PMs from. Multiple paths must be separated by commas.	String Default: /storage/no-backup/com-apr9010443/PerformanceManagementReportFiles
daily_files_output_path	Path where the PDC saves daily reports.	Type: String Default: /storage/no-backup/sapc/PolicyControlPdcFiles/Daily
monthly_files_output_path	Path where the PDC saves monthly reports.	Type: String Default: /storage/no-backup/sapc/PolicyControlPdcFiles/Monthly
sapchealthcheck_output_path	Path where the PDC health check report is saved.	Type: String Default: /storage/no-backup/sapc/PolicyControlPdcFiles/HealthCheck
trace_level	Maximum PDC trace level. In addition, critical messages are also displayed on the console.	Type: String. Possible values: CRITICAL \| ERROR \| WARNING \| INFO \| DEBUG Default CRITICAL
trace_files_output_path	Path where the PDC saves traces.	Type: String Default: /var/log/sapcPdc/traces/
sapchealthcheck_trace_level	Maximum PDC health check trace level. In addition, critical messages are also displayed on the console.	Type: String. Possible values: CRITICAL \| ERROR \| WARNING \| INFO \| DEBUG Default CRITICAL
sapchealthcheck_trace_files_output_path	Path where the PDC health check saves traces.	Type: String Default: /var/log/sapcPdc/traces/
application_counters	List of application counters separated by commas.	Type: String. Values separated by commas. Example: gxCcasInitSuccess,gxCcasSuccess
application_capacity_counters	List of capacity counters separated by commas.	Type: String. Values separated by commas. Example: afActiveSessions,mobileActiveSessions
platform_counters	List of platform counters separated by commas.	Type: String. Values separated by commas. Example: CPULoad.Total, Mem.PercentUsed
nice	Scheduling Linux kernel priority for the PDC execution	Type: Integer. Possible values: From -20 (highest priority) to 19 Default: 5
gx_interface_counters	List of counter names for Gx interface which are used by Health Check to determine if Gx interface is enabled or not in the SAPC.	Type: String. Values separated by commas. Default: gxCcrsInit,gxCcrsUpdate,gxCcrsTerminate
rx_interface_counters	List of counter names for Rx interface which are used by Health Check to determine if Rx interface is enabled or not in the SAPC.	Type: String. Values separated by commas. Default: rxAarsInit,rxAarsUpdate,rxStrs
sy_interface_counters	List of counter names for Sy interface which are used by Health Check to determine if Sy interface is enabled or not in the SAPC.	Type: String. Values separated by commas. Default: sySlrs,sySnrs,syStrs
sms_interface_counters	List of counter names for SMPP interface which are used by Health Check to determine if SMPP interface is enabled or not in the SAPC.	Type: String. Values separated by commas. Default: smsNotificationsSent
soap_interface_counters	List of counter names for SOAP interface which are used by Health Check to determine if SOAP interface is enabled or not in the SAPC.	Type: String. Values separated by commas. Default: soapNotificationsSent
sx_interface_counters	List of counter names for Sx interface which are used by Health Check to determine if Sx interface is enabled or not in the SAPC.	Type: String. Values separated by commas. Default: sxCcrsInit,sxCcrsUpdate,sxCcrsTerminate
extdb_interface_counters	List of counter names for extDb interface which are used by Health Check to determine if extDb interface is enabled or not in the SAPC.	Type: String. Values separated by commas. Default: ldapModifyRequests,ldapSearchRequests,soapExtDbNotificationsReceived
sd_interface_counters	List of counter names for Sd interface which are used by Health Check to determine if Sd interface is enabled or not in the SAPC.	Type: String. Values separated by commas. Default: sdTsrs,sdCcrsUpdate,sdCcrsTerminate

4.2.1.1 Configure Reported Counters

Configure the counters to be collected from the SAPC PM in PDC reports, following these steps:

Steps

Execute steps described in Modify Configuration to modify following properties:
- application_counters
- application_capacity_counters
- platform_counters

Add or remove as many counter names as desired to application_counters, application_capacity_counters or platform_counters values. Counters must be separated by commas.

Note:

When adding or removing counter names, take the following into consideration:

The counter names on the lists are case-sensitive. The PDC does not report counters if the name is not correct. For further information about measurement names, refer to Measurements.
To improve PDC data collection by the SAPC and its subsequent analysis, Ericsson recommends removing the counters related to features that are not configured in the SAPC . For example, if the SAPC does not have Rx configured, the Rx-related counters can be deleted from the pdc.cfg file.

Warning!

If the SAPC does not generate PM measurement for a counter which is configured in PDC to be reported, the counter does not appear in PDC reports.

4.2.1.2 Configure PDC Execution Priority

Modify priority of PDC execution on the system, following these steps:

Steps

Execute steps described in Modify Configuration to modify following property:
- nice

4.2.1.3 Configure Daily Processing Hour

When enabled, PDC launches the generation of daily report. Configure the time when daily report generation is launched following these steps:

Steps

Execute steps described in Modify Configuration to modify following property:

daily_processing_hour

Note:

The daily_processing_hour property must have a value that is not involved in the Daylight Saving Time (DST) change, either the time when DST is enabled or the time when DST is disabled. For example, if DST is enabled at 02:00 A.M and the new hour is 03:00 A.M, the same when DST is disabled, and the time passes from 03:00 A.M to 02:00 A.M, do not use any of these values.

Execute following command to enable automatic PDC execution at new configured time:
sapcadmin@SC-X> sapcPdc mode -enable

4.2.1.4 Configure PDC Traces

The PDC provides a tracing mechanism for troubleshooting purposes. Follow these steps to configure PDC traces for troubleshooting purposes:

Steps

Execute steps described in Modify Configuration to modify following properties:
- trace_files_output_path
- trace_level
Run PDC command to be analyzed.
Connect to SC-1 executing following command:
ssh sapcadmin@<SC-1>
Check generated traces for SC-1 in path indicated in trace_files_output_path property.
Connect to SC-2 executing following command:
ssh sapcadmin@<SC-2>
Check generated traces for SC-2 in path indicated in trace_files_output_path property.

4.2.1.5 Configure PDC Health Check Traces

The PDC health check provides a tracing mechanism for troubleshooting purposes. Follow these steps to configure PDC health check traces for troubleshooting purposes:

Steps

Execute steps described in Modify Configuration to modify following properties:
- sapchealthcheck_trace_files_output_path
- sapchealthcheck_trace_level
Run PDC health check command to be analyzed.
Connect to SC-1 executing following command:
ssh sapcadmin@<SC-1>
Check generated traces for SC-1 in path indicated in sapchealthcheck_trace_files_output_path property.
Connect to SC-2 executing following command:
ssh sapcadmin@<SC-2>
Check generated traces for SC-2 in path indicated in sapchealthcheck_trace_files_output_path property.

4.3 PDC Fault Management

4.3.1 Alarms

Not Applicable.

4.3.2 Notifications

Not Applicable.

4.3.3 Troubleshooting

The following chapters explain typical problems that can appear when executing the PDC. When analyzing a problem, consider if PDC traces could provide additional information and obtain then as described in Configure PDC Execution Priority.

4.3.3.1 Reports Are Not Generated

If the PDC is not generating the reports, it is possible that errors have been introduced on its configuration file. Do the following:

Steps

Look for PDC traces as described in Configure PDC Traces. If there are not traces for the date of the missing reports, it is possible pdc.cfg file is not in the correct path as described in PDC Configuration.
Check ERROR and CRITICAL messages from the trace files as described in Configure PDC Traces.
Open the pdc.cfg file and verify its content:
- All paths exist and there are not problems with permissions rights.
- All properties are configured properly:
  
  property=value1[,value2[,...]]
Check the SAPC is generating PM measurements as described in Measurements.
Check that the SAPC PM reports are generated on the specified path in pdc.cfg file. See Table 6.
Check counters name specified in pdc.cfg file are correct and the same as described in Measurements.
Check that there is enough disk space in paths specified on pdc.cfg file. See Table 6.

4.3.3.2 Newly Added Counter Does Not Appear

If a newly added counter is not shown in PDC reports, follow these steps:

Warning!

If the SAPC does not generate PM measurement for a counter which is configured in PDC to be reported, the counter does not appear in PDC reports.

Steps

Check the SAPC is increasing PM measurements for the newly added counter.
Check that the newly added counter name has been correctly added as described in Configure Reported Counters and check that the counter name is the same as the one in Measurements.

Note: PDC configuration is case-sensitive.

4.3.3.3 CPU Usage Is Too High

Follow steps described in Configure PDC Execution Priority to check that PDC execution priority is properly set.

4.3.3.4 Reports Take Too Long to Be Generated

Follow these steps to avoid reports taking too long to be generated:

Steps

Verify the contents of the SAPC PM measurements files are correct, complete, and have the correct size. See Measurements.
Verify CPU and memory levels for the SAPC. See Preventive Maintenance.
Follow steps described in Configure PDC Execution Priority to change PDC execution priority.
Follow steps described in Configure PDC Traces to configure trace level to CRITICAL.

4.3.3.5 Other Applications Affect PDC Performance

If you detect other applications affect PDC execution, consider:

Steps

Follow steps described in Configure Daily Processing Hour to avoid PDC scheduled execution overlap other applications execution.
Follow steps described in Configure PDC Execution Priority to change PDC execution priority.

4.3.3.6 PDC Execution Interrupted

If PDC execution is not successfully completed, follow these steps:

Steps

Check that all arguments for sapcPdc command are valid and have the correct format.
Execute following steps to verify that there is not a sapcPdc instance already running:
- Connect to SC-1 executing following command:
  
  ssh sapcadmin@<SC-1>
- Check that there are no PDC instances already running on SC-1 executing the following command:
  
  sapcadmin@SC-1> ps –ef | grep sapcPdc
- Connect to SC-2 executing following command:
  
  ssh sapcadmin@<SC-2>
- Check that there are no PDC instances already running on SC-2 executing the following command:
  
  sapcadmin@SC-2> ps –ef | grep sapcPdc

4.4 Other Operational Conditions

4.4.1 Capabilities

The PDC is available on system controllers and runs together with the SAPC. The generation of performance information should not cause any unstability problem in the SAPC. PDC capacity depends on the following factors:

Number of counters to report.
Size of the SAPC PM files (system and platform measures) to process
Granularity period of PM jobs (5 minutes by default)
Other processes running on same node as PDC.

Note: PDC is executed in SC where Backup and Restore function is not running.
PDC collection and health check trace levels.
PDC nice property

4.4.2 External Capabilities

Not Applicable.

4.4.3 Security

Not Applicable.

5 PDC Report Data Management

To collect the PDC report files, use SFTP as explained in System Administrator Guide. Paths where PDC report files are located are described in Table 7.

Table 7 PDC Report Files Path Structure
Path	Description
FileGroup=PolicyControlPdcFiles,FileGroup=Daily	PDC daily report files generated.
FileGroup=PolicyControlPdcFiles,FileGroup=Monthly	PDC monthly report files generated.
FileGroup=PolicyControlPdcFiles,FileGroup=HealthCheck	PDC report for health check.

5.1 Daily and Monthly Reports

All daily and monthly PDC reports have the following format:

Row:

Represents each SAPC PM report as a list of concatenated values.

The first value of the list corresponds with the time stamp of the SAPC PM report.

Following values are the collected data counters (separated by a | Character).
Columns

The first value of each column corresponds with the counter name per peer.

See Example 1.

Note:

Miscellaneous report contains snapshot of the actual status of the system. For that reason, in this report, only one row containing values can be found on it.

Example 1 Reports Format

TimeStamp|ipCanUnauthenticatedEmergencyActiveSessions|mobileActiveSessions|subscribers
2017-07-25 15:15:00|0|100000|3900001
2017-07-25 15:20:00|0|100000|3900001
2017-07-25 15:25:00|0|100000|3900001

5.1.1 Daily Reports

The PDC generates daily reports at scheduled time. Each daily report is composed with:

Application report <YYYYMMDD>_<networkManagedElementId>_sapc_pdc_PMF_application.log
Platform report <YYYYMMDD>_<networkManagedElementId>_sapc_pdc_PMF_platform.log
Miscellaneous report <YYYYMMDD>_<networkManagedElementId>_sapc_pdc_MISC.log

5.1.2 Monthly Reports

The PDC generates monthly reports using as input all generated daily reports from the previous month and collecting them in a unique file.

The reports are packaged to a compressed file. When the package is completed, the daily reports used to make the monthly report are deleted.

Obtained compressed file <month_name>_<networkManagedElementId>_sapc_pdc.tar.gz is the monthly report. After extracting the file, under path /tmp, following files are located:

Monthly application report <month_name>_<networkManagedElementId>_sapc_pdc_PMF_application.log
Monthly platform report <month_name>_<networkManagedElementId>_sapc_pdc_PMF_platform.log
Monthly miscellaneous report <month_name>_<networkManagedElementId>_sapc_pdc_MISC.log

5.2 Health Check Reports

Health check generates a report A<YYYYMMDD.HHMMSS>_HealthCheck.xml in XML format.

This file, contains different sections which group related information:

Node status in <HealthCheck> section
Node information as software level and limit capacity license in<NodeInformation> section
Node configuration as enabled capacity license and enabled interfaces in<NodeConfiguration> section.

Note:

These sections can be generated isolated. See Health Check

Copyright	© Ericsson AB 2018. All rights reserved. No part of this document may be reproduced in any form without the written permission of the copyright owner.
Disclaimer	The contents of this document are subject to revision without notice due to continued progress in methodology, design and manufacturing. Ericsson shall have no liability for any error or damage of any kind resulting from the use of this document.
Trademarks	All trademarks mentioned herein are the property of their respective owners. These are shown in the document Trademark Information.