Contents

1   Introduction

This instruction concerns alarm handling.

1.1   Alarm Description

The NTP Stratum Level Failure alarm is issued by the Managed Object (MO) UpstreamNTPServerConnection. The alarm is issued when the stratum level of the compute host which hosts the virtual Cloud Infrastructure Controller (vCIC) is equal to or lower than the upstream NTP server stratum level, when vCIC NTP is running in orphan mode.

The possible alarm causes and the corresponding fault reasons, fault locations, and impacts are described in Table 1.

Table 1    Alarm Causes

Alarm Cause	Description	Fault Reason	Fault Location	Impact
Stratum on the compute host which hosts the vCIC is less than, or equal to upstream NTP server.	NTP stratum level on any of the compute hosts which host the vCICs is less than, or equal to any of the stratum levels on the upstream NTP servers.	Configuration fault of NTP stratum level on upstream NTP Server	Upstream NTP Server configuration	CEE might not have the correct UTC time of day information.
		Configuration fault of NTP stratum level on the compute hosts which host the vCICs	Configuration of the compute host which hosts the vCIC

Note:  

An alarm can appear as a result of maintenance activity.

The following is the consequence for the node if the alarm is not solved:

• The compute host which hosts the vCIC runs in orphan mode. The NTP hierarchy (based on downstream request time from upstream) is incorrect.

The alarm attributes are listed in Table 2.

Table 2    Alarm Attributes

Attribute Name	Attribute Value
Major Type	193
Minor Type	2031708
Managed Object Class	UpstreamNTPServerConnection
Managed Object Instance	Region=<name_of_the_region>, CeeFunction=1, Node=<hostname_of_the_node>, UpstreamNTPServerConnection=1
Specific Problem	NTP Stratum Level Failure
Event Type	other (1)
Probable Cause	realTimeClockFailure (70)
Additional Text	NTP error
Severity	MINOR (5)

1.2   Prerequisites

This section provides information on the documents, tools, and conditions that apply to the procedure.

1.2.1   Documents

The following documents are needed to solve the alarm:

• NTP Upstream Server Failure

• Data Collection Guideline

1.2.2   Tools

No tools are required.

1.2.3   Conditions

Before starting this procedure, ensure that the following conditions are met:

• SSH credentials for vCIC node and compute node are available.
• An NTP Stratum Level Failure alarm is active.

2   Procedure

This section describes the procedure to follow when this alarm is active.

2.1   Actions for Solving the Alarm

If both NTP Upstream Server Failure and the NTP Stratum Level Failure alarms are active, follow the NTP Upstream Server Failure OPI to fix the alarm. Continue with the following steps after having followed the NTP Upstream Server Failure OPI.

1. Fetch information about the upstream NTP server stratum levels by executing the command:

   root@compute-0-2:~# ntpq -p

   Example of output:

   remote	refid	st	t	when	poll	reach	delay	offset	jitter
   *compute-0-6.domain.	10.35.50.5	5	u	66	1024	377	0.295	-0.139	0.497
   compute-0-5.domain.	10.35.50.5	6	u	888	1024	376	0.180	0.089	0.107
   10.35.50.5	192.168.50.4	9	u	347	1024	377	0.236	1.920	0.039
   seki20-ntp4.k2.	.INIT.	16	u	-	1024	0	0.000	0.000	0.000
   192.168.6.1	10.35.50.6	9	u	68	1024	377	0.123	1.631	0.109
   google-public-d	10.35.50.6	9	u	251	1024	377	0.256	0.928	0.171

   The remote NTP server stratum level is shown in the st column.

2. Check the orphan mode setting for the stratum level of the compute host which hosts the vCIC by entering the following command:

   cat /etc/ntp.conf | grep orphan, the output example is:

   tos orphan 4

   Number 4 is the stratum level on the compute hosts which host the vCICs in this example.

3. The stratum level on the compute hosts which host the vCICs must be higher than that on the upstream NTP servers. If not, change the stratum level value on each compute host which hosts the vCIC in /etc/ntp.conf. Change the value to exceed, or be equal to N+2, where N is the maximum stratum of the upstream NTP servers.

   Save the change and restart the NTP service by executing the command
   service ntp restart
   on the compute hosts which hosts the vCICs.

4. Set the value of orphan_mode_stratum in the config.yaml on Fuel master to the same value as used in previous step.
5. In case the alarm persists, do the following:
   • Collect all files and output from the commands obtained in Step 1– Step 4.
   • Collect troubleshooting data as described in the Data Collection Guideline.

     Note:  

     Alarm logs from Atlas and Linux console as generated from the system when following this OPI.

     
     
   • Consult the next level of maintenance support with all collected information.

   Further actions are outside the scope of this instruction.

6. The job is completed.