Contents

1   Introduction

This instruction concerns alarm handling for the Server Platform, Storage Performance Degradation Detected alarm.

1.1   Alarm Description

The alarm is issued when a Ericsson Centralized User Data Base (CUDB) application detects that application components are impacted by a degradation of storage performance.

The alarm is issued in the following situations:

• A monitored I/O heavy process gets stuck uninterruptedly due to missing storage system response.
• Storage system responds to file system probe request with I/O error or timeout.

The possible alarm causes and the corresponding fault reasons, fault locations, and impacts are described in Table 1.

Table 1    Alarm Causes

Alarm Cause	Description	Fault Reason	Fault Location	Impact
File system probe detected an error.	Monitored partition could not be written for a longer period of time (preset timeout) due to I/O error.	Most probably faulty infrastructure.	Storage system.	Performance degradation in the CUDB system.
Lightweight process state check detected an error.	Monitored I/O heavy process got stuck in uninterruptible sleep ("disk sleep").	Most probably faulty infrastructure.	Storage system.	Performance degradation in the CUDB system.

The following are the consequences for the node if the alarm is not solved:

• In case the alarm is raised in a payload blade or Virtual Machine (VM):
  • Performance degradation for the impacted Data Store/Processing Layer Database (DS/PLDB).
  • Lost local redundancy for the impacted DS/PLDB.
  • Lost DS/PLDB geographical redundancy in case both DS or all PLDB blades or VMs fail in a node.
  • Lost node in case all PLDB blades or VMs fail.
• In case the alarm is raised in a System Controller (SC):
  • Service degradation in controlling processes running on the impacted SC.
  • Possible node reboots.
• Unplanned mastership changes which can cause data durability issues.

The alarm attributes are listed and explained in Table 2.

Table 2    Alarm Attributes

Attribute Name	Attribute Value
Auto Cease	No
Module	SERVER-PLATFORM
Error Code	2
Timestamp First	Date and time when the alarm was raised for the first time.
Repeated Counter	Number which indicates how many times the alarm was raised.
Timestamp Last	Date and time of the most recent alarm raised.
Resource ID	.1.3.6.1.4.1.193.169.4.2.<Blade ID>
Alarm Model Description	Storage performance degradation detected, Server Platform
Alarm Active Description	Server Platform: Storage performance degradation detected on host <Blade>. <Additional info>
ITU Alarm Event Type	equipmentAlarm (5)
ITU Alarm Probable Cause	replaceableUnitProblem (69)
ITU Alarm Perceived Severity	Major (4)
Originating Source IP	Node IP where the alarm was raised.
Sequence Number	Number which indicates order in which alarms are raised.

In Table 2, the indicated variables are as follows:

• <Blade ID> is the LDE or LOTC node ID for the blade or VM.
• <Blade> is the LDE or LOTC hostname for the blade or VM.
• <Additional info> is different depending on the CUDB system deployment and blade type:
  • For CUDB systems deployed on native BSP 8100 and payload blade: Automatic shutdown was performed.
  • In all other cases: The variable has no value.

The possible cause is a failure in the storage system.

1.2   Prerequisites

This section provides information on the documents, tools, and conditions that apply to the procedure.

1.2.1   Documents

Before starting this procedure, ensure that you have read the following documents:

• System Safety Information, Reference [5]
• Personal Health and Safety Information, Reference [6]
• CUDB Node Fault Management Configuration Guide, Reference [2]

1.2.2   Tools

Not applicable.

1.2.3   Conditions

Not applicable.

2   Procedure

This section describes the procedure to follow when this alarm is received.

2.1   Procedure for CUDB Systems Deployed on Native BSP 8100

1. Identify the blade.

   For more information, refer to the Identifying the Faulty Blade section of Server Platform, Blade Replacement, Reference [1].

2. Lock the blade.

   For more information, refer to Manage Blade in the BSP 8100 CPI.

3. Perform the blade replacement.

   For more information, refer to the Replacing GEP Boards section of Server Platform, Blade Replacement, Reference [1] or contact the next level of maintenance support.

4. After the blade is replaced, clear the alarm.

   For more information, refer to the Clearing Alarms section of CUDB Node Fault Management Configuration Guide, Reference [2].

2.2   Procedure for CUDB Systems Deployed on a Cloud Infrastructure

In case the Storage Performance Degradation Detected alarm is raised, check the following in the cloud infrastructure:

• Check if there is any ongoing maintenance activity (for example, maintenance of the file systems used by the cloud infrastructure).
• Check if there is a problem with the cloud infrastructure software.
• Check if the cloud infrastructure hardware is hosting a faulty VM.

If everything is working correctly, clear the alarm (refer to the "Clearing Alarms" section of CUDB Node Fault Management Configuration Guide, Reference [2]).

In case problems in the cloud infrastructure are identified:

1. Perform the shutdown of the virtual machine:
   1. Login to the Atlas Dashboard.
   2. Select the appropriate project in the Current Project field, then select Project in the View field.
   3. Choose the Instances category.
   4. Identify the VM to shut down.
   5. In the Actions column of the identified VM, select the action "Shut Off Instance".
2. Make sure that the problems are fixed according to the Actions for Planned Infrastructure Maintenance Activities section of Virtualized CUDB Virtual Machine Recovery, Reference [3].
3. Once the VM is recovered, clear the alarm.

   For more information, refer to the Clearing Alarms section of CUDB Node Fault Management Configuration Guide, Reference [2].

Glossary

For the terms, definitions, acronyms, and abbreviations used in this document, refer to CUDB Glossary of Terms and Acronyms, Reference [4].

Reference List

CUDB Documents
[1] Server Platform, Blade Replacement.
[2] CUDB Node Fault Management Configuration Guide.
[3] Virtualized CUDB Virtual Machine Recovery.
[4] CUDB Glossary of Terms and Acronyms.

Other Ericsson Documents
[5] System Safety Information.
[6] Personal Health and Safety Information.