Contents

1   Introduction

1.1   Document Purpose and Scope

The purpose of this document is to provide a guideline to manage the Performance Data Collection (PDC) function in the SAPC.

2   Configuration Prerequisites

Before configuring the SAPC in an operational network, assure that:

• CBA Components are installed.
• The SAPC product software is installed.
• To have a detailed understanding of the function.

3   Performance Data Collection Function

The Performance Data Collection (PDC) function of the SAPC extracts performance information which can be used for performance analysis, dimensioning, and troubleshooting.

The PDC function takes as input:

• Performance Management Files (PMFs) of the SAPC. They provide measures related to hardware resources consumption, and measures of parameters that are specific to the SAPC business logic.
• Miscellaneous information: number of payload nodes and controller nodes in the cluster, status of the geographical redundancy, virtualization technology used, and the SAPC software version.
• Node status information such as, raised alarms, active licenses, and enabled protocol interfaces.

The input information is collected and written into output files, that can be retrieved from the SAPC through its SFTP interface. Four kinds of output files are produced:

• Application reports: Contains measures of parameters that are specific to the SAPC business logic
• Platform reports: Contains measures of hardware resources consumption
• Miscellaneous report: Contains miscellaneous information of the current status of the SAPC.
• Heath Check report: Contains node status, node information, and configuration data.

The SAPC is scheduled to:

• Execute the PDC function daily.
• Create a package at the end of every month containing aggregated information from all report files generated during the month.

The PDC function and health check can be executed manually at any time.

4   PDC Operational Conditions

4.1   PDC Function Administration

4.1.1   Enable Daily Data Collection and Monthly Aggregation

Note:  

Daily Performance Data Collection and Monthly Aggregation are enabled by default in the SAPC. The following steps are only needed if the collection has been previously disabled.

To enable the daily execution of the PDC function in the SAPC, and the monthly aggregation of produced files:

1. Access to the SAPC executing the following command:

   > ssh sapcadmin@<OAM_VIP>

2. Enable daily execution with the following command:

   sapcadmin@SC-X> sapcPdc mode -enable

4.1.2   Disable Daily Data Collection and Monthly Aggregation

To disable the daily execution of the PDC function in the SAPC, and the monthly aggregation of produced files:

1. Access to the SAPC executing the following command:

   > ssh sapcadmin@<OAM_VIP>

2. Disable daily execution with the following command:

   sapcadmin@SC-X> sapcPdc mode -disable

4.1.3   Manual Data Collection

The PDC function can be executed manually in the SAPC at any time.

4.1.3.1   Manual Collection by Default

To collect data by default, follow these steps:

1. Access to the SAPC executing the following command:

   > ssh sapcadmin@<OAM_VIP>

2. Collect data by default executing the following command:

   sapcadmin@SC-X> sapcPdc collect

   This command generates three daily data reports with the data of the previous day to the execution: miscellaneous, platform, and application. See Section 5.1.1.

4.1.3.2   Manual Data Collection by Date

To generate reports within a date range, follow these steps:

1. Access to the SAPC executing the following command:

   > ssh sapcadmin@<OAM_VIP>

2. Depending on your needs, execute one of the commands described on Table 1.

4.1.3.3   Manual Data Collection by Report Type

To generate a report of a specific type (for the previous day to the execution), follow these steps:

1. Access to the SAPC executing the following command:

   > ssh sapcadmin@<OAM_VIP>

2. Depending on your needs, execute one of the commands described on Table 2.

4.1.3.4   Manual Data Collection by Date and Report Type

To generate a report of a specific type within a date range, follow these steps:

1. Execute following command to access the SAPC:

   > ssh sapcadmin@<OAM_VIP>

2. Depending on your needs, execute one of the commands described on Table 3.

4.1.4   Data Packaging

The PDC processes the report files available and generates a monthly report as tarball output file.

Data packaging includes a cleanup mechanism that automatically deletes PDC trace files and the tarball monthly reports older than one year.

Note:  

Before executing data packaging, verify that the execution is needed. Data packaging probably consumes high CPU resources, and must be executed only if necessary.

4.1.4.1   Data Packaging by Default

To package and compress daily files for the previous month, follow these steps:

1. Access to the SAPC executing the following command:

   > ssh sapcadmin@<OAM_VIP>

2. Package by default executing the following command:

   sapcadmin@SC-X> sapcPdc package

4.1.4.2   Data Packaging by Date

To package daily reports for specific date ranges, follow these steps:

1. Access to the SAPC executing the following command:

   > ssh sapcadmin@<OAM_VIP>

2. Depending on your needs, execute one of the commands described on Table 4.

4.1.4.3   Cleanup

The cleanup option forces deletion of daily report files. The files removed are those used to generate the tarball file.

To clean up daily report files, follow these steps:

1. Access to the SAPC executing the following command:

   > ssh sapcadmin@<OAM_VIP>

2. Depending on your needs, execute one of the commands described on Table 5

4.1.5   Health Check

4.1.5.1   Health Check Full Report

To collect the instantaneous status, information, and configuration data of the node execute following steps:

1. Access to the SAPC executing the following command:

   > ssh sapcadmin@<OAM_VIP>

2. Collect health check full report executing the following command:

   sapcadmin@SC-X> sapcPdc sapcHealthCheck

4.1.5.2   Health Check Report

To collect only the node status:

1. Access to the SAPC executing the following command:

   > ssh sapcadmin@<OAM_VIP>

2. Collect health check report executing the following command:

   sapcadmin@SC-X> sapcPdc -report healthCheck

4.1.5.3   Node Information Report

To collect only node information as software level and limit capacity license, execute following steps:

1. Access to the SAPC executing the following command:

   > ssh sapcadmin@<OAM_VIP>

2. Collect health check node information report executing the following command:

   sapcadmin@SC-X> sapcPdc -report nodeInformation

4.1.5.4   Node Configuration Report

To collect only node configuration as enabled capacity license and enabled interfaces, execute following steps:

1. Access to the SAPC executing the following command:

   > ssh sapcadmin@<OAM_VIP>

2. Collect health check node configuration report executing the following command:

   sapcadmin@SC-X> sapcPdc -report nodeConfiguration

4.2   PDC Configuration

4.2.1   Modify Configuration

Configure the PDC function, executing following steps:

• Access to the SAPC executing the following command:

  > ssh sapcadmin@<OAM_VIP>

• Open /storage/system/config/sapc/pdc.cfg file.
• See Table 6 and modify the file depending on your needs.
• Save changes in the file.

4.2.1.1   Configure Reported Counters

Configure the counters to be collected from the SAPC PM in PDC reports, following these steps:

1. Execute steps described in Section 4.2.1 to modify following properties:
   • application_counters
   • application_capacity_counters
   • platform_counters
2. Add or remove as many counter names as desired to application_counters, application_capacity_counters or platform_counters values. Counters must be separated by commas.

Note:  

When adding or removing counter names, take the following into consideration:

• The counter names on the lists are case-sensitive. The PDC does not report counters if the name is not correct. For further information about measurement names, refer to Measurements.
• To improve PDC data collection by the SAPC and its subsequent analysis, Ericsson recommends removing the counters related to features that are not configured in the SAPC . For example, if the SAPC does not have Rx configured, the Rx-related counters can be deleted from the pdc.cfg file.

4.2.1.2   Configure PDC Execution Priority

Modify priority of PDC execution on the system, following these steps:

1. Execute steps described in Section 4.2.1 to modify following property:
   • nice

4.2.1.3   Configure Daily Processing Hour

When enabled, PDC launches the generation of daily report. Configure the time when daily report generation is launched following these steps:

1. Execute steps described in Section 4.2.1 to modify following property:
   • daily_processing_hour

   Note:  

   The daily_processing_hour property must have a value that is not involved in the Daylight Saving Time (DST) change, either the time when DST is enabled or the time when DST is disabled. For example, if DST is enabled at 02:00 A.M and the new hour is 03:00 A.M, the same when DST is disabled, and the time passes from 03:00 A.M to 02:00 A.M, do not use any of these values.

   
   
2. Execute following command to enable automatic PDC execution at new configured time:

   sapcadmin@SC-X> sapcPdc mode -enable

4.2.1.4   Configure PDC Traces

The PDC provides a tracing mechanism for troubleshooting purposes. Follow these steps to configure PDC traces for troubleshooting purposes:

1. Execute steps described in Section 4.2.1 to modify following properties:
   • trace_files_output_path
   • trace_level
2. Run PDC command to be analyzed.
3. Connect to SC-1 executing following command:

   ssh sapcadmin@SC-1

4. Check generated traces for SC-1 in path indicated in trace_files_output_path property.
5. Connect to SC-2 executing following command:

   ssh sapcadmin@SC-2

6. Check generated traces for SC-2 in path indicated in trace_files_output_path property.

4.2.1.5   Configure PDC Health Check Traces

The PDC health check provides a tracing mechanism for troubleshooting purposes. Follow these steps to configure PDC health check traces for troubleshooting purposes:

1. Execute steps described in Section 4.2.1 to modify following properties:
   • sapchealthcheck_trace_files_output_path
   • sapchealthcheck_trace_level
2. Run PDC health check command to be analyzed.
3. Connect to SC-1 executing following command:

   ssh sapcadmin@SC-1

4. Check generated traces for SC-1 in path indicated in sapchealthcheck_trace_files_output_path property.
5. Connect to SC-2 executing following command:

   ssh sapcadmin@SC-2

6. Check generated traces for SC-2 in path indicated in sapchealthcheck_trace_files_output_path property.

4.3   PDC Fault Management

4.3.1   Alarms

Not Applicable.

4.3.2   Notifications

Not Applicable.

4.3.3   Troubleshooting

The following chapters explain typical problems that can appear when executing the PDC. When analyzing a problem, consider if PDC traces could provide additional information and obtain then as described in Section 4.2.1.2.

4.3.3.1   Reports Are Not Generated

If the PDC is not generating the reports, it is possible that errors have been introduced on its configuration file. Do the following:

1. Look for PDC traces as described in Section 4.2.1.4. If there are not traces for the date of the missing reports, it is possible pdc.cfg file is not in the correct path as described in Section 4.2.
2. Check ERROR and CRITICAL messages from the trace files as described in Section 4.2.1.4.
3. Open the pdc.cfg file and verify its content:
   • All paths exist and there are not problems with permissions rights.
   • All properties are configured properly:

     property=value1[,value2[,...]]

4. Check the SAPC is generating PM measurements as described in Measurements.
5. Check that the SAPC PM reports are generated on the specified path in pdc.cfg file. See Table 6.
6. Check counters name specified in pdc.cfg file are correct and the same as described in Measurements.
7. Check that there is enough disk space in paths specified on pdc.cfg file. See Table 6.

4.3.3.2   Newly Added Counter Does Not Appear

If a newly added counter is not shown in PDC reports, follow these steps:

1. Check the SAPC is increasing PM measurements for the newly added counter.
2. Check that the newly added counter name has been correctly added as described in Section 4.2.1.1 and check that the counter name is the same as the one in Measurements.

   Note:  

   PDC configuration is case-sensitive.

   
   

4.3.3.3   CPU Usage Is Too High

Follow steps described in Section 4.2.1.2 to check that PDC execution priority is properly set.

4.3.3.4   Reports Take Too Long to Be Generated

Follow these steps to avoid reports taking too long to be generated:

1. Verify the contents of the SAPC PM measurements files are correct, complete, and have the correct size. See Measurements.
2. Verify CPU and memory levels for the SAPC. See Preventive Maintenance.
3. Follow steps described in Section 4.2.1.2 to change PDC execution priority.
4. Follow steps described in Section 4.2.1.4 to configure trace level to CRITICAL.

4.3.3.5   Other Applications Affect PDC Performance

If you detect other applications affect PDC execution, consider:

1. Follow steps described in Section 4.2.1.3 to avoid PDC scheduled execution overlap other applications execution.
2. Follow steps described in Section 4.2.1.2 to change PDC execution priority.

4.3.3.6   PDC Execution Interrupted

If PDC execution is not successfully completed, follow these steps:

1. Check that all arguments for sapcPdc command are valid and have the correct format.
2. Execute following steps to verify that there is not a sapcPdc instance already running:
   • Connect to SC-1 executing following command:

     ssh sapcadmin@SC-1

   • Check that there are no PDC instances already running on SC-1 executing the following command:

     sapcadmin@SC-1> ps –ef | grep sapcPdc

   • Connect to SC-2 executing following command:

     ssh sapcadmin@SC-2

   • Check that there are no PDC instances already running on SC-2 executing the following command:

     sapcadmin@SC-2> ps –ef | grep sapcPdc

4.4   Other Operational Conditions

4.4.1   Capabilities

The PDC is available on system controllers and runs together with the SAPC. The generation of performance information should not cause any unstability problem in the SAPC. PDC capacity depends on the following factors:

• Number of counters to report.
• Size of the SAPC PM files (system and platform measures) to process
• Granularity period of PM jobs (5 minutes by default)
• Other processes running on same node as PDC.

  Note:  

  PDC is executed in SC where Backup and Restore function is not running.

  
  
• PDC collection and health check trace levels.
• PDC nice property

4.4.2   External Capabilities

Not Applicable.

4.4.3   Security

Not Applicable.

5   PDC Report Data Management

To collect the PDC report files, use SFTP as explained in System Administrator Guide. Paths where PDC report files are located are described in Table 7.

5.1   Daily and Monthly Reports

All daily and monthly PDC reports have the following format:

• Row:

  Represents each SAPC PM report as a list of concatenated values.

  The first value of the list corresponds with the time stamp of the SAPC PM report.

  Following values are the collected data counters (separated by a | Character).

• Columns

  The first value of each column corresponds with the counter name per peer.

  See Example 1.

Note:  

Miscellaneous report contains snapshot of the actual status of the system. For that reason, in this report, only one row containing values can be found on it.

TimeStamp|ipCanUnauthenticatedEmergencyActiveSessions|mobileActiveSessions|subscribers
2017-07-25 15:15:00|0|100000|3900001
2017-07-25 15:20:00|0|100000|3900001
2017-07-25 15:25:00|0|100000|3900001

5.1.1   Daily Reports

The PDC generates daily reports at scheduled time. Each daily report is composed with:

• Application report <YYYYMMDD>_<networkManagedElementId>_sapc_pdc_PMF_application.log
• Platform report <YYYYMMDD>_<networkManagedElementId>_sapc_pdc_PMF_platform.log
• Miscellaneous report <YYYYMMDD>_<networkManagedElementId>_sapc_pdc_MISC.log

5.1.2   Monthly Reports

The PDC generates monthly reports using as input all generated daily reports from the previous month and collecting them in a unique file.

The reports are packaged to a compressed file. When the package is completed, the daily reports used to make the monthly report are deleted.

Obtained compressed file <month_name>_<networkManagedElementId>_sapc_pdc.tar.gz is the monthly report. After extracting the file, under path /tmp, following files are located:

• Monthly application report <month_name>_<networkManagedElementId>_sapc_pdc_PMF_application.log
• Monthly platform report <month_name>_<networkManagedElementId>_sapc_pdc_PMF_platform.log
• Monthly miscellaneous report <month_name>_<networkManagedElementId>_sapc_pdc_MISC.log

5.2   Health Check Reports

Health check generates a report A<YYYYMMDD.HHMMSS>_HealthCheck.xml in XML format.

This file, contains different sections which group related information:

• Node status in <HealthCheck> section
• Node information as software level and limit capacity license in<NodeInformation> section
• Node configuration as enabled capacity license and enabled interfaces in<NodeConfiguration> section.

Note:  

These sections can be generated isolated. See Section 4.1.5