CUDB Node Logging Events

1	Introduction
1.1	Document Purpose and Scope
1.2	Revision Information
1.3	Typographic Conventions
2	Log Events
2.1	Infrastructure Log Events
2.2	Middleware and Platform Log Events
2.3	eVIP Log Events
2.4	Fault and Performance Management (ESA) Log Events
2.5	Application Log Events
2.5.1	Log Files
2.5.2	Log Types
2.5.3	Log Format
2.5.4	Log Messages
2.5.4.1	DataAccess
2.5.4.2	DbmsTools
2.5.4.3	Monitoring
2.5.4.4	Notifications
2.5.4.5	OAM
2.5.4.6	Platform
2.5.4.7	Storage
2.5.4.8	3PP
2.6	Security Log Events
3	Sending Logs to a Remote Server
Glossary
Reference List

1 Introduction

This document provides detailed information about the events that the Ericsson Centralized User Database (CUDB) logs.

1.1 Document Purpose and Scope

The purpose of this document is to describe the specific logging events reported by CUDB.

1.2 Revision Information

Rev. A
Rev. B
Rev. C
Rev. D
Rev. E
Rev. F
Rev. G
Rev H

Other than editorial changes, this document has been revised as follows:

Section 2.5.4.5.2: Added new ERR rows to Table 27.
Section 3: Updated procedure, log format information, TIMESTAMP description, and example.

1.3 Typographic Conventions

Typographic conventions can be found in the following document:

Typographic Conventions

2 Log Events

This section contains information about how logs are handled by each CUDB component.

2.1 Infrastructure Log Events

Refer to the "BSP Log Management" document in the BSP 8100 CPI for information on the logging system in BSP 8100.

2.2 Middleware and Platform Log Events

Logs from the platforms, such as, Linux Distribution Extensions (LDE) and Core Middleware (CMW) are stored in the following four separate files on System Controller (SC) blades:

/var/log/<hostname>/kernel: Kernel messages.
/var/log/<hostname>/auth: Messages related to LDE users, groups, and authentication.
/var/log/<hostname>/messages: All messages (except for those found in auth).
/var/log/<hostname>/commandlog: Commands executed by the shell.

For more information about Linux Distribution Extensions (LDE) logs, refer to LDE Management Guide, Reference [6].

CMW logs can be found in /var/log/opensaf.

2.3 eVIP Log Events

eVIP uses syslog as logging system. The eVIP component is configured to log into the file /var/log/messages.

2.4 Fault and Performance Management (ESA) Log Events

Not detailed in this document. Refer to ESA Setup and Configuration, Reference [7] for information related to Ericsson Simple Network Management Protocol (SNMP) Agent (ESA) logs.

2.5 Application Log Events

This section contains detailed information on the application log events.

2.5.1 Log Files

All the application related logging events in CUDB are reported using Linux rsyslog. Refer to The Syslog Protocol (RFC 5424)The Syslog Protocol (RFC 5424), Reference [8] and Transmission of Syslog Messages over UDP (RFC 5426)Transmission of Syslog Messages over UDP (RFC 5426), Reference [9] for further information.

As a general rule, logs are stored in the /var/log/<hostname> directory on the two SC nodes. Exceptions to this location are the Lightweight Directory Access Protocol (LDAP) Front End (FE) logs (see Section 2.5.4.1.1).

Note:: If the Centralized Security Event Logging function is configured, authentication and console commands logs will also be sent to an external logging server (see Section 3 for further details).

Log files are rotated the indicated times before being removed from /var/log/<hostname>.

Following log files are rotated only once:

/var/log/PL_2_*/kernel
/var/log/PL_2_*/auth
/var/log/wtmp

Following log file is rotated three times:

/var/log/PL_2_*/messages_mysqld

Following log files are rotated four times:

/var/log/SC_2_*/systemMonitor
/var/log/SC_2_*/bc_server.log
/var/log/PL_2_5/bc_server.log

Following log file is rotated five times:

/var/log/SC_2_*/consistency_audit_task_history

Following log files are rotated eight times:

/var/log/SC_2_*/messages
/var/log/PL_2_*/messages
/var/log/SC_2_*/kernel
/var/log/SC_2_*/auth

Following log files are rotated nine times:

/var/log/SC_2_*/commandlog
/var/log/PL_2_*/commandlog
/var/log/SC_2_*/sec_events/security_events.log
/var/log/PL_2_*/sec_events/security_events.log

To examine older log files, check the /local/cudb_logarchive directory on the SCs, where the log files removed from /var/log are stored in compressed format for an extended period of time. For each /var/log/<hostname> directory, there is a corresponding /local/cudb_logarchive/<hostname> directory, where the removed logs are stored. These files are also rotated, in sync with the log rotation of their counterparts in /var/log/<hostname>. The last 100 items are kept from each of the following SC blade logs (from /var/log/SC_2_* to /local/cudb_logarchive/SC_2_*):

messages
kernel
auth
commandlog
systemMonitor

The last 100 items are kept from each of the following payload blade logs (from /var/log/PL_2_* to /local/cudb_logarchive/PL_2_*):

messages
kernel
auth
messages_mysqld
commandlog
bc_server.log
sec_events/security_events.log

2.5.2 Log Types

Each logging event has a severity. For severity definitions, refer to The Syslog Protocol (RFC 5424)The Syslog Protocol (RFC 5424), Reference [8].

Only the following severities (sorted from most to least importance) apply to the logging events described in this document:

EMERG: Component or subcomponent is unusable.
CRIT: A critical situation that can cause the component or the system depending on the scope to fail.
ERR: Error conditions.
WARNING: Potentially harmful situation.

Note:: Only those log events that are specially relevant for the operator are listed in the tables in the following sections.

2.5.3 Log Format

All the application related logging information that CUDB logs using syslog has the following format:

<Date> <Time> <Host_Name> <Process_Name[\[PID\]]> <Component> [Subcomponent]: <Severity> - <Message Information>

The meaning of each field is explained as follows:

`<Date>`	Date when the log has been reported.
`<Time>`	Time when the log has been reported.
`<Host Name>`	Physical server where the component is running.
`<Process_Name[\[PID\]]>`	Process that logs the message. Optionally followed by the PID of the process in square brackets. The first square bracket pair in the `<Process_Name[\[PID\]]>` log format indicates optional information and the second square bracket pair escaped by backslash character indicates that, if PID is logged, it must be in square brackets.
`<Component>`	Component that logs the event. Optionally an internal subcomponent can also be shown. See the component description below this table.
`<Severity>`	Indication about the importance of the fault reported by the log (see Section 2.5.2).
`<Message Information>`	Description of the event logged.

The following components and subcomponents are defined:

DataAccess
- LDAP FE (see Section 2.5.4.1.1).
- LDAP FE Monitor (see Section 2.5.4.1.2).
- Schema Management GUI (see Section 2.5.4.1.3).
- LDAP Counters (see Section 2.5.4.1.4).
DbmsTools
- CountersFw (see Section 2.5.4.2.1).
- DataBackupRestore (see Section 2.5.4.2.2).
- Reconciliation (see Section 2.5.4.2.3).
- Replica Consistency Audit: Consistency Check (see Section 2.5.4.2.4).
- Reallocation (see Section 2.5.4.2.5).
- ReplicaRepair: Data Repair (see Section 2.5.4.2.6).
- Selective Replica Check (see Section 2.5.4.2.7).
Monitoring
- System Monitor (see Section 2.5.4.3.1).
- Cluster Supervisor (CS) (see Section 2.5.4.3.2).
- KeepAlive (see Section 2.5.4.3.3).
- High Availability (see Section 2.5.4.3.4).
- Security Logs Monitor (see Section 2.5.4.3.5).
- Replica Consistency (see Section 2.5.4.3.6).
Notifications
- SOAP (see Section 2.5.4.4.1).
Operation and Maintenance (OAM)
- SW_Mgmt: Software Management (see Section 2.5.4.5.1).
- CM: Configuration Management (see Section 2.5.4.5.2).
- FM: Fault Management (see Section 2.5.4.5.3).
- PM: Performance Management (see Section 2.5.4.5.4).
- Security: Security (see Section 2.5.4.5.5).
Platform
- LogHandler (see Section 2.5.4.6.1).
- SmartHWFaultReaction (see Section 2.5.4.6.2).
Storage
- Engine (see Section 2.5.4.7.1).
Third Party Products (3PP)
- MySQL (see Section 2.5.4.8.1).
- BC Server (see Section 2.5.4.8.2).

Example 1 and Example 2 show the typical appearance of a log message.

Example 1 Log Message

Sep 22 14:51:01 PL_2_5 slapd[3340] DataAccess [LDAP]: WARNING - 
Access not granted for required operation

Example 2 Log Message

Jun 18 16:23:13 PL_2_5 cudbReallocate[18372] 
DbmsTools [Reallocation]: 
WARNING - DS replication has not been verified

2.5.4 Log Messages

The messages logged by CUDB are described in the following subsections. Each subsection contains the messages logged by a particular CUDB component.

2.5.4.1 DataAccess

This section describes the subcomponents of the DataAccess component.

2.5.4.1.1 LDAP FE

LDAP FE uses syslog as logging system with the following properties:

The LDAP FE messages are stored in /var/log/ldapfe in the payload blade where the LDAP FE process runs.
/var/log/ldapfe log files are allowed to grow up to 512 KB and are rotated once before being removed.
The LDAP FE error messages related to received LDAP requests are stored in /local2/log/ldapfe_errors in the payload blade where the LDAP FE process runs.
/local2/log/ldapfe_errors log files are allowed to grow up to 10 MB and are rotated 10 times before being removed.

Table 1 shows the LDAP FE messages.

Table 1 LDAP FE Messages
Severity	Message Information	Trigger Event
`EMERG`	`cudb_be_load_table_mappings FAILED`	Back End (BE) initialization cannot be executed successfully.
`EMERG`	`cudbOver_op_modify ldapmodify called with empty attribute`	LDAP modify operation cannot be performed if no attribute is listed.
`EMERG`	`invalid arg number <argument>. One argument is required`	Overlays initialization cannot be executed successfully.
`EMERG`	`Cluster at <connectString> not ready within <seconds> seconds`	Pool is not able to connect within time specified.
`EMERG`	`cudb_dist_conf::<methods> ADL not initialized`	Several methods cannot be executed if ADL is not initialized.
`EMERG`	`Not able to subscribe to Messaging Service`	FE is not able to get mastership info from system monitor.
`ERR`	`cudbOver_main::op_addPL Master PLDB not reachable while Split Brain and provUser false` `cudbOver_main::op_modifyPL Master PLDB not reachable while Split Brain and provUser false` `cudbOver_main::op_addDS Master PLDB not reachable while Split Brain, proxyBehavior is Provisioning and provUser false` `cudbOver_main::op_modifyDS Master PLDB not reachable while Split Brain, proxyBehavior is Provisioning and provUser false` `cudbOver_main::op_search Master PLDB not reachable while Split Brain, proxyBehavior provisioning and isProvisioningUser false`	Add, modify or search operations are not available in symmetrical split situation, and local PLDB partition is not master.
`ERR`	`cudb_branch_cache::createEvent Error in getDictionary` `cudb_branch_cache::createEvent Error in`	`BRANCH_CACHE_CHANGES` tables cannot be initialized.
`ERR`	`cudb_branch_cache::notifyChanges couldn't notify changes`	Changes cannot be executed in `BRANCH_CACHE_CHANGES` table.
`ERR`	`cudb_pm_db_open: failed to register the database with back-monitor`	Database cannot be registered.
`ERR`	`monitoring thread was not able to bind`	Monitoring overload query cannot bind.
`ERR`	`Unable to start CUDB overload control thread. Error code:<ERR_CODE>`	Overload control thread cannot be started.
`ERR`	`cudbOver_main::op_add attempt while even split and Backbone reliable`	Provisioning has been locked due to a potential split-brain situation.
`ERR`	`Entry cannot be built from the object classes info in <tableName> table. Possible data corruption in the entry <dn>`	The length of the column is not a multiple of 4, so the data can be considered corrupted.
`ERR`	`Object class not configured in` `OBJECT_CLASSES` `table`	The length of the column is a multiple of 4, but the object class cannot be found in the cache.
`WARNING`	`Identity key not read`	Identity key cannot be read.
`WARNING`	`cudb_index::getObjectOperation null transaction critical failure` `cudb_index::getObjectOperation couldn't obtain an operation for CUDB_DN_INDEX table` `cudb_index::getObjectOperation couldn't read tuple for CUDB_DN_INDEX table` `cudb_index::getObjectOperation Error setting DnBaseKey`	Different warnings can appear when a transaction is not executed properly.
`WARNING`	`cudb_index::getSubtreeAliasesOperation. Invalid scope.` `cudb_index::getSubtreeAliasesOperation. Error after setting key.` `cudb_index::getSubtreeAliasesOperation. Error after setting alias scan filter.`	Different warnings can appear when the key composition is not according the scope of search operation.
`WARNING`	`cudb_index::getIndexValuesInNdbOp. Error reading operational attribute 'modifyTimestamp'.` `cudb_index::getIndexValuesInNdbOp. Error reading operational attribute 'createTimestamp'.`	Error while reading the operational timestamp attributes.
`WARNING`	`cudb_de::read_de: error reading operational timestamp attributes 'modifyTimestamp'` `cudb_de::read_de: error reading operational timestamp attributes 'createTimestamp'`	Error while reading the operational timestamp attributes of a Distribution entry.
`WARNING`	`Access not granted for required operation`	The access required for the operation is not granted.
`WARNING`	`No space left in DSG <dsg_id>`	It is impossible to add an entry because the Data Store (DS) cluster is full.
`WARNING`	`DSG <dsg_id> returned by distribution external library does not exist. Default distribution algorithm will be applied`	The DS Unit Group (DSG) returned by the distribution external library does not exist.
`WARNING`	`Zone <zone_id> does not exist`	The zone does not exist.
`WARNING`	`No space left in CUDB System`	The CUDB system is full.
`WARNING`	`No space left in zone <zone_id>`	The zone is full.
`WARNING`	`Client disconnected from Messaging Service`	LDAP FE got disconnected from Messaging Service instance and will try to reconnect.
`WARNING`	`Client connection closed to Messaging Service`	LDAP FE got disconnected from Messaging Service instance LDAP FE got disconnected from Messaging Service instance.
`WARNING`	`Client reconnected to Messaging Service in <uri>!`	Reconnection attempt to Messaging Service cluster was successful.

Table 2 shows LDAP FE error message.

Table 2 LDAP FE Error Message Related to Received LDAP Requests
Severity	Message Information	Trigger Event
`WARNING`	`LDAP on_response: err=<ldap_result_code> text=<diagnostic_message> op=<ldap_operation> dn=<distinguished_name> user=<ldap_user_name>`	Error processing a received LDAP request.

Note:: <ldap_operation> can be BIND, SEARCH, MODIFY, ADD, or DELETE.
<distinguished_name> is the distinguished name related to the failed LDAP request. Depending on when in the processing phase the error happened, the distinguished name in the log entry will be the original distinguished name in the LDAP request or one partially modified by CUDB.

Some additional LDAP FE messages are logged within this component, for further information, refer to OpenLDAPOpenLDAP, Reference [10].

2.5.4.1.2 LDAP FE Monitor

The LDAP FE Monitor component running on SC blades or VMs uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, messages with severities info and above are logged.

The messages logged by the LDAP FE Monitor, when running on SC blades or VMs, are summarized in Table 3.

Table 3 LDAP FE Monitor SC Logs
Severity	Message Information	Trigger Event
`EMERG`	`<prog_name> failed to run as daemon. Error:<error>`	LDAP FE Monitor unable to run as daemon.
`EMERG`	`Unable to activate all the evip addresses correctly!`	An error has occurred trying to activate all the eVIP addresses.
`EMERG`	`Unable to deactivate all the evip addresses correctly!`	An error has occurred trying to deactivate all the eVIP addresses.
`ERR`	`Error trying to activate eVip traffic address with port error:<error>`	An error occurred trying to activate the traffic address with the port.
`ERR`	`Error trying to deactivate eVip traffic address with port error:<error>`	An error occurred trying to deactivate the traffic address with the port.
`ERR`	`Error calling evipDtmServerActivate with port error: <port><error>`	An error has occurred calling `evipDtmServerActivate` with port.
`ERR`	`Error calling evipDtmServerDeactivate with port error: <port><error>`	An error has occurred calling `evipDtmServerDeactivate` with port.
`ERR`	`Failed to connect to IMM: too many failed attempts` `Failed to connect to IMM: connection rejected`	Failed to connect to IMM.
`ERR`	`Failed to initialize IMM OM: too many failed attempts` `Failed to initialize IMM OM: unknown error`	Failed to initialize IMM Object Manager.
`ERR`	`Failed to finalize IMM OM accessor` `Failed to finalize IMM OM`	Failure at disconnection from IMM.
`ERR`	`Failed to read <object> object from IMM: object does not exist` `Failed to read <object> object from IMM: invalid parameter` `Failed to read <object> object from IMM: unknown error` `Unsupported type while reading attribute <attribute> from IMM`	The object or the attribute cannot be read from IMM.
`ERR`	`No uuid was read from IMM`	The `uuid` attribute does not exist in IMM.
`ERR`	`Failed to initialize alarms library`	An error occurred trying to initialize the alarms library.
`WARNING`	`<thread> LDAP FE at <IP>:<port>is now down`	LDAP FE thread is down.
`WARNING`	`Failed to establish SNMP session`	Unable to establish SNMP session
`WARNING`	`Could not read LDAP FE's configuration from xml file`	An error occurred trying to read the xml configuration file.
`WARNING`	`Unable to get vip handle. retrying in 5 secs ... cause: <error>`	An error has occurred trying get vip handle.
`WARNING`	`Unable to load libtdm library, not using this capabilities, error: <error>`	An error has occurred trying to load `libtdm` library.
`WARNING`	`<thread> LDAP FE at <IP>:<port>is experiencing a temporary error`	Unable to get the LDAP FE status due to a temporary error.

The LDAP FE Monitor component running on payload blades or VMs uses syslog as logging system with the following properties:

Stored in /var/log/PL_2_*/messages.
By default, messages with severities info and above are logged.

The messages logged by the LDAP FE Monitor, when running on payload blades or VMs, are summarized in Table 4.

Table 4 LDAP FE Monitor Payload Logs
Severity	Message Information	Trigger Event
`CRIT`	`Failed to start LDAP FE process`	It was impossible to start the LDAP FE process.
`ERR`	`immfind for LDAP failed - IMM time out or SA_..._ERR error`	An error occurred while trying to read configuration data.
`ERR`	`LDAP FE is unreachable`	The monitored LDAP FE process cannot be reached.
`ERR`	`Failed to set CPU affinity`	An error occurred while trying to set the CPU affinity for the monitored LDAP FE.
`ERR`	`unable to terminate LDAP FE process`	An error occurred while trying to gracefully terminate the monitored LDAP FE process.
`ERR`	`unable to kill LDAP FE process`	An error occurred while trying to terminate the monitored LDAP FE process.
`ERR`	`inet_ntop: <error>`	An error occurred while trying to convert the IP address into text format.
`ERR`	`host <host> is not resolved`	IP address resolution for host `<host>` failed.
`WARNING`	`LDAP FE process is not running`	The monitored LDAP FE process is not up.
`WARNING`	`LDAP server responded with an error: <error>`	The monitored LDAP FE process is reachable, but the request failed due to `<error>`.
`WARNING`	`LDAP request to <LDAP FE> failed (<X>/<Y>)`	The LDAP request to the monitored LDAP FE process failed `<X>` out of total `<Y>` times.
`WARNING`	`LDAP FE did not shut down in time`	The monitored LDAP FE process did not shut down in time after the graceful termination.
`WARNING`	`LDAP FE process is still running`	The monitored LDAP FE process did not shut down after termination attempt.
`WARNING`	`pre-termination command has failed to finish in <X> milliseconds; sending KILL signal`	The command that was supposed to be executed prior to graceful termination of the monitored LDAP FE process failed to execute. Graceful termination will still occur.
`WARNING`	`could not open config file <file>`	An error occurred when trying to open the configuration file for reading.
`WARNING`	`failed to resolve <host> <X>/5 times, sleeping <Y> milliseconds before retrying`	It was impossible to resolve the IP address for the given `<host>` on attempt `<X>` out of total 5. A retry will occur after `<Y>` milliseconds.

2.5.4.1.3 Schema Management GUI

Schema management Graphic User Interface (GUI) uses log4java as log system with the following properties:

By default, the schema management GUI messages are stored in a /log subdirectory under the directory where the tool has been installed.
The schema management GUI messages file is allowed to grow up to 500 KB and is rotated 10 times before being removed.

The messages logged by the Schema Management GUI are summarized in Table 5.

Table 5 Schema GUI Logs
Severity	Message Information	Trigger Event
`ERR`	`Invalid schema object selected`	Selected an invalid schema object.
`ERR`	`View <id_project_view> not selected`	View not selected.
`ERR`	`Error opening attribute editor for <attr_name>`	An error has occurred trying to open the attribute editor.
`ERR`	`Error opening objectclass editor for <obj_class>`	An error has occurred trying to open the object class editor.
`ERR`	`Error opening schema editor for <schema_name>`	An error has occurred trying to open the schema editor.
`ERR`	`Error trying to execute command <command>`	An error has occurred executing the command, something wrong happened.
`ERR`	`Error renaming schema file`	An error has occurred when a schema file was renamed, probably because the file was removed or modified with external actions.
`ERR`	`The imported file defines attributes or object classes with OID or alias already defined in the project`	The selected file cannot be imported since there is at least one entity already defined.
`ERR`	`Error deleting schema file <schema_name>`	An error has occurred when a schema file was deleted, probably because the file was removed or modified with external actions.
`ERR`	`The selected file is not a valid Open LDAP schema file`	The selected file cannot be imported since it is not well-formed according to the OpenLDAP syntax.
`WARNING`	`Schema <schema_name> is being removed creating error in: <element>`	Due to the deletion of a schema file some inconsistencies appear in the specified elements.
`WARNING`	`ObjectClass <obj_class> is being removed creating error in: <element>`	Due to the deletion of an object class some inconsistencies appear in the specified elements.
`WARNING`	`Internal error - Unknown object type`	Error obtaining the name of the object: unknown object type.
`WARNING`	`AttributeType <attr_type> is being removed creating error in: <element>`	Due to the deletion of an attribute type some inconsistencies appear in the specified elements.

2.5.4.1.4 LDAP Counters

LDAP Counters uses log4java as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, only messages with severity info and above are logged.

The relevant messages logged by the LDAP Counters component are summarized in Table 6.

Note:: The LDAP Counters component uses FATAL severity logging instead of EMERG.

Table 6 LDAP Counters Logs
Severity	Message Information	Trigger Event
`FATAL`	`command must be: ExecuteCounter -s pathCudbSystem.xml -f pathConfigLCA.xml`	Error running LDAP Counters with wrong order or incorrect number of parameters.
`FATAL`	`Bad configuration parameter`	Error running LDAP Counters with bad parameter.
`FATAL`	`ExecuteCounter:main <message description>`	Error executing main LDAP Counters method. See message description.
`ERROR`	`Couldn't update counter value in controller <controller number> because ESA is down`	Error connecting to ESA in controller `<controller number>` due to ESA process is down.
`ERROR`	`Couldn't update counter value in ESA <controller number>. Counter: <counter name>-<counter Id>`	Error updating counters in ESA in controller `<controller number>`
`ERROR`	`Couldn't update <counter type> counters, <message description>` `Counter Type values:` `OVERALL` `OVERALL DROP RATIO` `LOCAL CLUSTER` `LOCAL CLUSTER DROP RATIO` `APPGROUPS`	Exception raised when updating specific type of counter. See message description.
`ERROR`	`ExecuteCounter:obtainTotalOpCompleted: <message description>`	Exception raised when gathering counters into from LDAP FE. See message description.
`ERROR`	`NdbErrorCounterThread:run : <message description>`	Error starting the thread that collects the counters. See message description.
`ERROR`	`NdbErrorCounterThread:obtainCounter : <message description>`	Error running the thread that collects the counters. See message description.
`WARNING`	`CounterHelper:propertiesCudbSystem <message description>`	Error collecting properties from `CudbSystem.xml` file. See message description.
`WARNING`	`CounterHelper:propertiesLCA <message description>`	Error collecting properties from counters configuration file. See message description.
`WARNING`	`XmlHelper:flushNdbCountersToDisk <message description>`	Error parsing configuration or transforming NDB counters. See message description.
`WARNING`	`XmlHelper:cleanUpOldFiles <message description>`	Error deleting old files in the configured directory. See message description.
`WARNING`	`AppGroupCounters.setCounterValues: <message description>`	Error setting the new value of the LDAP Counters. See message description.
`WARNING`	`NdbErrorClassification.setDsgCounterValues: <message description>`	Error updating DSG counters read from the LDAP FE. See message description.
`WARNING`	`NdbErrorClassification.setNdbCounterValues: <message description>`	Exception raised when updating NDB counters read from the LDAP FE. See message description.
`WARNING`	`AppGroupCounters.setCounterValues: <message description>`	Exception raised when updating Applications Group counters. See message description.
`WARNING`	`Overall.setCounterValues: <message description>`	Exception raised when updating Overall counters. See message description.
`WARNING`	`ExecuteCounter:checkInformation <message description>`	Exception raised when error happens checking the counter information. See message description.
`WARNING`	`ExecuteCounter:publishInESA <message description>`	Exception raised when publishing counters into ESA. See message description.

2.5.4.2 DbmsTools

This section describes the subcomponents of the DbmsTools component.

2.5.4.2.1 CountersFw

The CountersFw subcomponent uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, only messages with severity info, warning, or emergency are logged.

The messages logged by the CountersFw subcomponent are summarized in Table 7.

Table 7 CountersFw Logs
Severity	Message Information	Trigger Event
`ERR`	`Error establishing PL Connection.`	In the case of a remote replica, when the communication between CUDB nodes using SSH cannot be established.
`ERR`	`Error establishing DS Connection.`	In the case of a remote replica, when the communication between CUDB nodes using SSH cannot be established.
`ERR`	`Error Calling stored procedure for PL in node <node_id>.` `Error Calling stored procedure for DSG <DSG_id> in node <node_id>.`	It is not possible to execute the stored procedure in charge of performing the count.
`ERR`	`Couldn't get storage engine information about PLDB in node <node_id>.` `Couldn't get storage engine information about DSG <DSG_id> in node <node_id>.` `Couldn't fetch PL counter field.` `Couldn't fetch DS counter field.`	CUDB system configuration file does not have information for the chosen replica.
`ERR`	`Error on changing database to <database_name>`	DataBase is changed to Application Counter's database.
`ERR`	`Couldn't update PL counter field.`	No counter can be stored in PLDB.
`ERR`	`Error Calling stored procedure for DSG <DSG_id> in node <node_id>.`	The called saved procedure doesn't exist in the DSG of the node specified.
`ERR`	`Error Calling stored procedure for PL in node <node_id>.`	The called saved procedure doesn't exist in the PL of the node specified.
`ERR`	`cudbApplicationCounter Error reading SDL list from BC Cluster.`	Error getting the master list and the status of all the replicas of the CUDB system.
`ERR`	`Error converting masterlist to internal data type.`	Internal error when converting masterlist to internal data type.
`ERR`	`Error reading master list from BC Cluster.`	Error getting the master list of the CUDB system.
`ERR`	`Error reading DSG Status List.`	Error getting information about the status of all the replicas of the CUDB system.
`ERR`	`Error converting DSG Status List to internal type.`	Error when converting replica status list to internal data type.
`ERR`	`Error reading Master List.`	Error getting information about the status of all the master replicas of the CUDB system.
`ERR`	`Error getting SDL children, getChildren failed: <error>.`	Error reading status of all the replicas of the CUDB system.
`ERR`	`Error reading DSG Status List for DSG: <dsgId>, error: <error>.`	Error reading status of all the replicas of the CUDB system.
`ERR`	`Error reading master list, getChildrem failed: <error>.`	Error reading the master list of the CUDB system.
`ERR`	`Error getting master info for DSG: <dsgId>, Error: <error>.`	Error reading the master list of the CUDB system.
`ERR`	`Error reading BC entry size for DSG: <dsg>, error: <error>`	Attempt to determine BC entry size failed. See error description.
`ERR`	`Failed to open config file <CONFIG_FILE>.`	The mentioned config file is not available.
`ERR`	`Failed to read all configuration variables (got only <NUM> out of <NUM>), missing: <OPTION_XPATH_LIST>.`	The configuration file (`/home/cudb/common/config/cudbDatabases.xml`) does not contain all the parameters needed by `CountersFW`.
`ERR`	`Unable to obtain connection information of PLDB on node <NODEID>.`	Failed to obtain local connection information of the PLDB on the mentioned node from the system configuration file.
`ERR`	`Unable to obtain connection information of DSG <DSGID> on node <NODEID>.`	Failed to obtain local connection information of the mentioned DSG on the mentioned node from the system configuration file.
`ERR`	`Couldn't connect to PL tables.`	Failed to connect to PLDB.
`ERR`	`PLDB master moved to another node.`	PLDB master moved to another node during the operation.
`ERR`	`Can't connect to <HOST>:<PORT> to access database <DBNAME> with user <USERNAME>: <REASON>.`	Described in reason, but very likely that no MySQL server runs on the mentioned host listening on the mentioned port.
`ERR`	`OAM Command Failed: Couldn't connect to BC Cluster.`	Error, timeout expired while trying to connect to BC Cluster.
`ERR`	`Local node id can't be obtained.`	An error occurred while trying to find local node id.
`ERR`	`Master PLDB node id can't be obtained.`	An error occurred while trying to find master PLDB node.
`WARNING`	`It has been impossible to start <process> process.`	There is a problem trying to start `cudbApplicationCounters` process.
`WARNING`	`Not all the DSGs have an eligible replica. Application counter data might be stale`	Any DSG is masterless or no representative DS replica can be gathered for every DSG, from where the process to update an application counter is launched.
`WARNING`	`Couldn't get slave SDL list. Application counter data might be stale.`	The system has geographical redundancy and there is no slave replica.
`WARNING`	`exception: <description>` `runtime error: <description>` `unknown exception.`	Unexpected error.
`WARNING`	`ADG_ERR: Nodes have master after <num_retries> retries in <counter_name>`	After some tries the configuration to master nodes has been loaded.
`WARNING`	`ADG_ERR: Couldn't get ADL list for <counter_name>. Checking whether to retry. Retry=<num_retries> MaxRetries=<max_retries>`	After some retires the ADL configuration has not been loaded yet and the process will wait for a while and will try again.
`WARNING`	`Error setting PLDB Master List watcher.`	Error when creating a subscription to changes on the status of PLDB master.

2.5.4.2.2 DataBackupRestore

The DataBackupRestore subcomponent uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, only messages with severity info, warning, or emergency are logged.

The messages logged by the DataBackupRestore subcomponent are summarized in Table 8.

Table 8 DataBackupRestore Logs
Severity	Message Information	Trigger Event
`ERR`	`Option -u\|--user-facility must have a value`	The process was launched with the `-u\|--user-facility` flag empty, but this flag needs a value.
`ERR`	`Option -T\|--Timeout must have a value`	The process was launched with the `-T\|--Timeout` flag empty, but this flag needs a value.
`ERR`	`Option -S\|--Slack-backup must have a value`	The process was launched with the `-S\|--Slack-backup` flag empty, but this flag needs a value.
`ERR`	`Option -R\|--Retries-number must have a value`	The process was launched with the `-R\|--Retries-number` flag empty, but this flag needs a value.
`ERR`	`Option -t\|--time must have a value`	The process was launched with the `-t\|--time` flag empty, but this flag needs a value.
`ERR`	`Internal Error: option processing error: <error_description>`	There was a flag command unknown.
`ERR`	`No such file <configuration_file>`	The configuration file does not exist in the specified path.
`ERR`	`ERR IN PARAMETERS : WE MUST SET THE PARAMETERS FOR PATTERN TO BACKUP`	The parameter specified was wrong or it was not specified.
`ERR`	`The node <nodeId> on which restore will be performed has no PLDB configured`	The operator tries to restore a PLDB data backup in a node that has no configured PLDB replica.
`ERR`	`Provisioning Gateway has failed when de-blocking on attempt <attempt_id> - <notification error message>.`	The communication to the Provisioning Gateway was wrong when de-blocking on all of the attempts.
`ERR`	`Not all the DSGs have an eligible replica from which to take the backup`	No representative replica can be gathered for every DSG or PLDB, from where a backup is executed during a system backup procedure.
`ERR`	No master replica for PLDB or it is in degraded mode.	Master replica for PLDB cannot be detected or it is in degraded mode during a system backup procedure.
`WARNING`	`Fault in <SQL_operation> mysql query`	General fault in an SQL sentence, due to a wrong format sentence, data, or connection.
`WARNING`	`It is impossible to establish connection with PL mysql server`	Due to a wrong IP or port, or to a failure in the cluster, the process cannot connect to the database in the Processing Layer (PL).
`WARNING`	`Provisioning Gateway has failed`	The communication to the Provisioning Gateway was wrong.
`WARNING`	`Provisioning Gateway has failed when blocking`	The communication to the Provisioning Gateway was wrong when blocking.
`WARNING`	`Provisioning Gateway has failed when de-blocking on attempt <attempt_id> - <notification error message>`	The communication to the Provisioning Gateway was wrong when de-blocking on one of the attempts. Retrying after 6 seconds.

2.5.4.2.3 Reconciliation

The reconciliation subcomponent uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, only messages with severity info, warning, or emergency are logged.

The messages logged by the reconciliation subcomponent are summarized in Table 9.

Table 9 Reconciliation Logs
Severity	Message Information	Trigger Event
`WARNING`	`Deleted Entry due to Reconciliation Process: [<mscId \| assocId>]`	Error has occurred, entry has been deleted due to the reconciliation process.
`WARNING`	`Failed to add running reconciliation on:<dsg_id>`	Error has occurred while attempting to notify the system about reconciliation task.
`WARNING`	`fork():<pid_id>`	Fork failed. Creating child process responsible for deferred delete has failed.
`WARNING`	`ldap.add():failed` `v_subscribe():failed` `ldap.del():failed` `defered_delete failed` `ev_wait():failed` `ev_unsubscribe():failed`	Error has occurred during reconciliation process; one of the following steps has failed: Adding the unique key through LDAP Subscribing to DELETE events in the PL Removing the unique key Creating child process responsible for deferred delete Waiting for the DELETE events in the PL Un-subscribing the event handler
`WARNING`	`failed to establish 1. checkpoint`	Error has occurred during reconciliation process; one of the steps described above has failed (ldap.add(), ev_subscribe(), ldap.del(), defered_delete, ev_wait() or ev_unsubscribe()).
`WARNING`	`failed to build discrepancy list`	Error has occurred during reconciliation process; scanning the PL/DS pair for identification of dangling references and unreferenced data has failed to build discrepancy list.
`WARNING`	`failed to establish 2. Checkpoint`	Error has occurred during reconciliation process; Preparation for ‘false’ match recheck has failed.
`WARNING`	`double check failed`	Error has occurred during reconciliation process; filtering away false matches has failed.
`WARNING`	`fopen():<undo_file>`	Error has occurred during reconciliation process; creating undo files has failed.
`WARNING`	`database cleanup failed`	Error has occurred during reconciliation process; cleanup of the DS Unit database under reconciliation against the PLDB database has failed.
`WARNING`	`Considering that the delete event <event ID> for which this drp task is waiting for has been lost . Aborting reconciliation...`	In scenarios when the PL master is not located on the same node as the master DS, replication channels are unresponsive during event hooking mechanism and maximum event wait time exceeded, replication is marked as failed.
`WARNING`	`The task <dsg_id> has not been removed from CUDBdcObjet, so we should remove it manually?`	Data reconciliation process has already finished for DS but it is not removed from reconciliation task list.
`WARNING`	`Failed getting next task from pending task list.`	Error has occurred during reconciliation process; looking for new request.
`WARNING`	`<pl:<dsg_id>>: The task <dsg_id> has not been removed from CUDBdcObjet, so we should remove it manually?`	Removing `<dsg_id>` from `CUDBdcObject` after PL mastership change has failed.
`WARNING`	`<not pl:<dsg_id>>: The task <dsg_id> has not been removed from CUDBdcObjet, so we should remove it manually?`	Removing `<dsg_id>` from `CUDBdcObject` has failed; there was no PL mastership change.
`WARNING`	`Subscription to Messaging Service failed <error_id>`	Subscription initialization after PTL setup has failed. Exiting process.
`WARNING`	`Shutting down dared...`	Data Reconciliation Daemon (DARED) has failed getting next task from pending task list. Unexpectedly shutting down daemon.
`WARNING`	`Reconciliation marked as running for '<dsg_id>' but current '<dsg_id>'`	Error has occurred during reconciliation process; running it for different DSG then marked.
`WARNING`	`src/dare-ctx.cpp:672 LDAP Connection failed`	LDAP Connection not defined.
`WARNING`	`src/dare-ctx.cpp:676 Failed to inform Reconciliation running on '<dsg_id>'`	Adding of entry by its distinguished name over LDAP into reconciliation table `CUDBdcObject` of `cudb_user_data` database has failed.
`WARNING`	`src/dare-ctx.cpp:700 Failed to inform Reconciliation finished on '<dsg_id>'`	Deleting of entry by its distinguished name over LDAP from reconciliation table `CUDBdcObject` of `cudb_user_data` database has failed.
`WARNING`	`Node <nodeid> has no new masters to be added to PTL for reconciliation`	After examining the most recently received ADL list, it was determined that the local CUDB node holds no new masters compared to previous ADL list.
`WARNING`	`Not able to begin transaction: INSERT INTO dr_ptl for node <nodeid>`	An error occurred while trying to begin the transaction which writes the PTL to `dr_ptl` table.
`WARNING`	`Not able to commit PTL to dr_ptl table for node <nodeid>`	An error occurred while trying to commit the transaction which writes the PTL to `dr_ptl` table.
`WARNING`	`Some DSGs are not enabled in cudb_system table`	Some DS units are missing from `cudb_system` table.
`WARNING`	`Unexpected error while handling new ADL list`	A situation occurred which is not expected by the system.
`WARNING`	`There is no appropriate Node with PLDB in same Site to use`	An error occurred while attempting to get an appropriate node with PLDB in the same site of a node without PLDB.
`WARNING`	`Failed reading mysql configuration` `Failed reading LDAP local configuration` `Failed reading based configuration`	An error occurred while attempting to read configuration data from the configuration file
`WARNING`	`Master PLDB id not available` `Master PLDB addr not available` `Master PLDB port not available` `Error disabling master PLDB node configuration` `Error getting configuration of master PLDB node` `Unable to update SQL configuration`	An error occurred while attempting to get or set some configuration data related to the Master PLDB in a node without PLDB.
`WARNING`	`Setup of pending task list failed: <message>`	An error occurred while attempting to reconfigure MySQL connections for the pending reconciliation tasks list.
`WARNING`	`Could not initialize the Mysql C API library`	An error occurred while initializing the MySQL C API used for managing the reconciliation tasks list.
`WARNING`	`Client disconnected from Messaging Service`	Reconciliation process got disconnected from Messaging Service instance and will try to reconnect.
`WARNING`	`Client connection closed to Messaging Service`	Reconciliation process got disconnected from Messaging Service instance.
`WARNING`	`Client reconnected to Messaging Service in <uri>!`	Reconnection attempt to Messaging Service cluster was successful.

2.5.4.2.4 Replica Consistency Audit

The Consistency Check function is implemented by the Replica Consistency Audit subcomponent, which contains the following set of executables that perform logging:

The CLI interface (cudbConsistencyMgr).
The manager (CudbDdciManager).
The DSU comparison tool (cudbDsuDiff).

The CLI and the manager part of the Replica Consistency Audit subcomponent uses syslog as logging system with the following properties:

General operational logs are stored in /var/log/<hostname>/messages.
Task history logs are stored in /var/log/<hostname>/consistency_audit_task_history.
By default, only messages with info or higher severity level are logged.
Task history logs use different timestamp format (ISO date) than general operational logs because of different requirements. This timestamp format also includes year and time zone, for example 2014-09-19T16:09:14+02:00.
Task history logs must be read using the cudbConsistencyMgr --taskhistory command. The --taskhistory switch displays the task history of the current CUDB node.
Task history logs are rotated when the log size exceeds 512 kilobytes. 4 old log history files are kept by adding extension .1, .2, .3 and .4. File ending in .4 contains the oldest log entries.

The messages logged as operational logs are summarized in Table 10.

Table 10 Replica Consistency Audit CLI and Manager Logs
Severity	Message Information	Trigger Event
`CRIT`	`Fatal error happened during DDCI AMF component initialization`	Could not initialize AMF client.
`CRIT`	`Waiting for signal failed: <error description>(<error code>)`	Timed waiting for a Unix signal failed, see error description.
`ERR`	`BCClient exception while processing <message type>(<message type ID>): <error description>`	Communication error with or disconnection from the BC cluster, see error description.
`ERR`	`Connection to ZK cluster is lost, trying to reconnect`	Communication error to all servers of the connected BC cluster.
`ERR`	`DSG<ID> on node <node ID> is not alive but <current status>(<current status ID>)`	The given DSG's DSU is not alive.
`ERR`	`Exception while listing tasks: <error description>`	Couldn't perform full task listing, see error description.
`ERR`	`Failed to complete CLI action: <error description>`	Error while performing action specified on command line, see error description.
`ERR`	`Failed to connect to ZK cluster`	Permanent communication error to all servers of the connected BC cluster.
`ERR`	`Failed to connect to ZK cluster`	Permanent communication error to all servers of the local BC cluster.
`ERR`	`Failed to handle data change of <znode path> (value: <value>, result: <result description>): <error description>`	Internal error. (Got an unexpected BC change notification.)
`ERR`	`Failed to set sigmask: <error code>`	Couldn't set Unix signal mask.
`ERR`	`<process ID>: process aborted, but couldn't find related task`	Internal error. (A task's process was aborted but couldn't find out which task the process belongs to.)
`ERR`	`Request execution failed: <error description>`	Failed to complete disk IO operation, see error description.
`ERR`	`<task ID>: failed to start process: <error description>`	Task's process couldn,t be started, see error description.
`ERR`	`<task ID>: timeout while waiting for remote connection to site <site ID>`	Couldn't connect to BC cluster on remote site in time.
`ERR`	`Unexpected call of AmfCB::changeFinished(<znode path>,...)`	Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)
`ERR`	`Unexpected call of AmfCB::getChildrenFinished(<znode path>,...)`	Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)
`ERR`	`Unexpected call of AmfCB::getFinished(<znode path>,...)`	Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)
`ERR`	`Unexpected call of AmfRemoteCB::changeFinished(<znode path>,...)`	Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)
`ERR`	`Unexpected call of AmfRemoteCB::getChildrenFinished(<znode path>,...)`	Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)
`ERR`	`Unexpected call of AmfRemoteCB::getFinished(<znode path>,...)`	Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)
`ERR`	`Unexpected call of CliCB::changeFinished(<znode path>,...)`	Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)
`ERR`	`Unexpected call of CliCB::getChildrenFinished(<znode path>,...)`	Internal error. (A task's process was terminated but couldn't find out which task the process belongs to.)
`ERR`	`Unexpected call of CliCB::getFinished(<znode path>,...)`	Internal error. (A task's process was terminated but couldn't find out which task the process belongs to.)
`ERR`	`<znode path>: child change for which we were not subscribed`	Unexpected `znode` change notification from BC cluster.
`ERR`	`<znode path>: failed to read BC entry size: <error description>`	Attempt to determine BC entry size failed. See error description.
`WARNING`	`<process ID>: couldn't find task for process which just finished with result: <result description> (RC: <exit code>)`	Internal error. (A task's process was terminated but couldn't find out which task the process belongs to.)
`WARNING`	`<process ID>: failed to handle its SIGCHLD: <error description>`	Could not complete operation assigned to the received Unix SIGCHLD signal, see error description.
`WARNING`	`<process ID>: failed to kill: <error description>`	Couldn't kill a task's process, see error description.
`WARNING`	`<process ID>: failed to kill process: <error description>`	Couldn't kill the given process, see error description.
`WARNING`	`<process ID>: no action on SIGCHLD, event: <event name>(<event code>)`	An unexpected SIGCHILD signal is received and discarded.
`WARNING`	`<task ID>: couldn't find process to terminate`	The task's process could not be aborted due to timing or internal error.
`WARNING`	`<task ID>: failed to parse task ID when checking PTL`	Task ID is malformatted.
`WARNING`	`<task ID>: failed to remove from PTL: <error description>`	Task couldn't be removed from PTL, see error description.
`WARNING`	`<task ID>: failed to remove from RTL: <error description>`	Task couldn't be removed from RTL, see error description.
`WARNING`	`<task ID>: failed to remove from RTL: <error description>, try to delete later`	Completed task couldn't be removed from RTL, see error description.
`WARNING`	`<task ID>: found in PTL, but deletion failed with: <error description>. Possibly moved to RTL in the meantime`	See error description.
`WARNING`	`<task ID>: found in PTL, but deletion failed with: <error description>. Possibly moved to RTL in the meantime, attempting to delete from RTL`	See error description.
`WARNING`	`<task ID>: precondition of execution failed: <error description>. Deleting this task from PTL.`	Mastership changed since task put into the pending task list or DSG degraded or replica lag higher than `max-replica-lag`, see error description.
`WARNING`	`<task ID>: process (<process ID>) aborted`	The task's process aborted, couldn't complete its operation.
`WARNING`	`<task ID>: RTL entry is invalid, or previously deleted`	Internal error. (A task's process was terminated but the task was unknown.)
`WARNING`	`<task ID>: task data (<task parameters>) parsing failed: <error description>`	Task parameters are malformed.
`WARNING`	`Unknown and thus unhandled message: <message type>(<message type ID>)`	Internal error. (A thread notified another one, but that thread didn't expect that notification message.)

The DSU comparison tool, which is part of the Replica Consistency Audit subcomponent, uses both syslog and additional separate log files for logging. For information on these additional files, see CUDB Consistency Check, Reference [1].

For messages logged using syslog, the following properties apply:

Stored in /var/log/SC_2_*/messages files on SC blades.
By default, only messages with info or higher severity level are logged.
The LDAP tree logs and the table logs are stored locally on the SC blades where the tool is executed, under /local/cudb_ddci/replica_check/.
The LDAP tree logs and the table logs are kept for a maximum of 14 days or until their total volume exceeds 300 MB.

Relevant messages logged into syslog by the DSU comparison tool are summarized in Table 11.

Table 11 DSU Comparison Tool Logs
Severity	Message Information	Trigger Event
`ERR`	`Invalid option list. Diagnostic info: <internal_diagnostic_info>`	Got invalid command line options. The Consistency Check task exits.
`ERR`	`Invalid Node Id. Diagnostic info: <internal_diagnostic_info>`	CUDB Node ID command line option value is invalid. The Consistency Check task exits.
`ERR`	`Invalid Dsg Id. Diagnostic info: <internal_diagnostic_info>`	DSG ID command line option value is invalid. The Consistency Check task exits.
`ERR`	`Invalid max PPL. Diagnostic info: <internal_diagnostic_info>`	Max. PPL command line option value is invalid. The Consistency Check task exits.
`ERR`	`Invalid alarm severity limit. Diagnostic info: <internal_diagnostic_info>`	Got invalid alarm severity limit command line option value. The Consistency Check task exits.
`ERR`	`Incorrect MySQL Access server information. Diagnostic info: <internal_diagnostic_info>`	Got insufficient information to establish the connection to a MySQL Access server (probable causes: some information is missing or invalid in `cudbSystem.xml`, or invalid command line options are provided). The Consistency Check task exits.
`ERR`	`Unable to establish the connection to MySQL Access server endpoint. Diagnostic info: <internal_diagnostic_info>`	Both MySQL Access servers of one of the specified DS Units are unreachable. The Consistency Check task exits.
`ERR`	`Unrecoverable error. Diagnostic info: <internal_diagnostic_info>`	The Consistency Check task exits due to an error condition which prevents successful task completion. More detailed internal diagnostic information is available in `<internal_diagnostic_info>`.
`ERR`	`MySQL wrapper error. Diagnostic info: <internal_diagnostic_info>`	The Consistency Check task exits due to a failed database query operation. More detailed internal diagnostic information is available in `<internal_diagnostic_info>`.
`ERR`	`MySQL error. Diagnostic info: <internal_diagnostic_info>`	The Consistency Check task exits due to a failed database query operation. More detailed internal diagnostic information is available in `<internal_diagnostic_info>`.
`WARNING`	`Failed to connect to endpoint: <host>:<port> (<num>/<retries>). Diagnostic info: <internal_diagnostic_info>`	Connection failed to MySQL Access server endpoint `<host>:<port>`. The Consistency Check task will try to reestablish the connection to the same MySQL Access server or to its redundant counterpart for `<retries>-<num>` additional times.
`WARNING`	`Failed to execute the <name> SqlPrimitive on endpoint: <host>:<port> (<num>/<retries>). Diagnostic info: <internal_diagnostic_info>`	An SQL query execution failed on MySQL Access server endpoint `<host>:<port>`. The Consistency Check task will disconnect, then tries to reestablish the connection to the same MySQL Access server or to its redundant counterpart for `<retries>-<num>` additional times, and executes the query again if possible.

2.5.4.2.5 Reallocation

The reallocation subcomponent uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, only messages with severity info, warning, or emergency are logged.

The messages logged by the reallocation subcomponent are summarized in Table 12.

Table 12 Reallocation Logs
Severity	Message Information	Trigger Event
`EMERG`	`Not able to subscribe to Messaging Service`	The reallocation procedure is not able to get mastership information from the system monitor.
`ERR`	`Reallocation halted due to master change in source DSG <dsg_id>`	A master change in the source DSG has been detected during reallocation.
`ERR`	`Reallocation halted due to master change in destination DSG <dsg_id>`	A master change in the destination DSG has been detected during reallocation.
`ERR`	`Reallocation halted due to master change in PLDB`	A master change in the Processing Layer Database (PLDB) has been detected during reallocation.
`WARNING`	`Failure when reallocating <de_dn> distributed entry due to <problem>`	A problem arose while a DE was locked, read, or written, or its reference was changed in the PL.
`WARNING`	`DS replication has not been verified`	Verification of a DS block replication has not been completed due to an error or time-out problem.
`WARNING`	`There are no slaves of the current PL`	The PL Master is the only replica in the system. No verification is possible.
`WARNING`	`PL replication not completely verified`	Verification of references in PL replication not completed due to any error or time-out problems.
`WARNING`	`Reached the end of the table, but less than the requested percentage of DEs was reallocated. <percentage>% of the requested amount was reallocated.`	The reallocation procedure reallocated less than the specified amount of DS entries.
`WARNING`	`Client disconnected from Messaging Service`	Reallocation procedure got disconnected from Messaging Service instance and will try to reconnect.
`WARNING`	`Client connection closed to Messaging Service`	Reallocation procedure got disconnected from Messaging Service instance.
`WARNING`	`Client reconnected to Messaging Service in <uri>!`	Reconnection attempt to Messaging Service cluster was successful.

Note:: <problem> can be locking, reading, writing, or writing in PL problems.

2.5.4.2.6 Replica Repair

The Data Repair function is implemented by the Replica Repair subcomponent, which contains the following executable that performs logging:

The Data Repair tool (cudbReplicaRepair).

The Replica Repair subcomponent uses syslog as logging system with the following properties:

Stored in /var/log/PL_2_*/messages.
By default, only messages with info or higher severity level are logged.

The messages logged as operational logs are summarized in Table 13.

Table 13 Replica Repair Logs
Severity	Message Information	Trigger Event
`ERR`	`Failed to raise alarm. Diagnostic info: <internal diagnostic info>`	Data Repair tried to raise an alarm at the end of its execution, but failed.
`ERR`	`LDAP operation error. Diagnostic info: <internal diagnostic info>`	Data Repair received an error from an LDAP interface while trying to execute an LDAP operation.
`ERR`	`LDAP connection error. Diagnostic info: <internal diagnostic info>`	Data Repair received an error from an LDAP interface while trying to connect to an LDAP server.

2.5.4.2.7 Selective Replica Check

The Selective Replica Check function contains the following executable that performs logging:

The binlog analyzer tool (cudbBinlogAnalyzer).

The Selective Replica Check subcomponent uses syslog as logging system with the following properties:

Stored in /var/log/PL_2_*/messages.
By default, only messages with info or higher severity level are logged.

The messages logged as operational logs are summarized in Table 14.

Table 14 Selective Replica Check Logs
Severity	Message Information	Trigger Event
`ERR`	`<line where exception was thrown> invalid --dsg value: <dsg number>`	Selective Replica Check has received an invalid number for the database cluster group identifier when starting.
`ERR`	`<line where exception was thrown> ionice exited with <error code>`	Selective Replica Check failed in setting the `ionice` priority when starting the execution.
`ERR`	`<line where exception was thrown> Search operation results in <internal diagnostic info>`	Selective Replica Check received multiple results for an LDAP search operation.
`ERR`	`<line where exception was thrown> Permanent ldap error.`	An error which cannot be handled occurred in Selective Replica Check while trying to execute an LDAP operation.
`ERR`	`<line where exception was thrown> Unrecoverable error in <internal diagnostic info>`	An error which cannot be handled occurred in Selective Replica Check while trying to execute an LDAP operation.
`ERR`	`<line where exception was thrown> Number of retries exceeded <internal diagnostic info>`	Selective Replica Check exceeded the maximum number of retries for an LDAP search operation.
`ERR`	`<line where exception was thrown> invalid <parameter>`	Selective Replica Check has received an invalid parameter when starting.
`ERR`	`<line where exception was thrown> Unrecoverable <error> occurred in LDAP data fetching process.`	Selective Replica Check encountered an error while generating the output file containing LDAP entries.
`ERR`	`<line where exception was thrown> Directory creation failed with error: <internal diagnostic info>`	Selective Replica Check failed to create a working directory.
`ERR`	`<line where exception was thrown> Directory path exists but it is not a directory.`	Selective Replica Check found a file instead of a directory for writing the output.
`ERR`	`<line where exception was thrown> Failed get the size of file <internal diagnostic info>`	Selective Replica Check could not retrieve the size of the file.
`ERR`	`<line where exception was thrown> Failed to delete file <internal diagnostic info>`	Selective Replica Check could not delete a file.
`ERR`	`<line where exception was thrown> File: <file location> - , status: <internal diagnostic info>`	Selective Replica Check encountered an error when handling a binlog file.
`ERR`	`<line where exception was thrown> Failed get the modify time of file <internal diagnostic info>`	Selective Replica Check could not retrieve the last modification time of a file.
`WARNING`	`<line where exception was thrown> Following ldap exception occured in thread <internal diagnostic info>`	An exception occurred in Selective Replica Check while trying to execute an LDAP operation.
`WARNING`	`<line where exception was thrown> LDAP overload on segment <internal diagnostic info>`	Selective Replica Check received LDAP error code 51 or 80 from the local node.
`WARNING`	`<line where exception was thrown> Problem is probably entry specific. <internal diagnostic info>`	Selective Replica Check could not retrieve a specific DN.
`WARNING`	`<line where exception was thrown> Retry <attempt number> in thread <internal diagnostic info>`	Selective Replica Check is performing a retry on a specific DN.
`WARNING`	`<line where exception was thrown> no binlog files found!`	Selective Replica Check did not find any binlog files.

2.5.4.3 Monitoring

This section describes the subcomponents of the monitoring component.

2.5.4.3.1 System Monitor

System Monitor component uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/systemMonitor.
By default, only messages with severity info, warning or error are logged.

The messages logged by System Monitor are summarized in Table 15.

Table 15 System Monitor Logs
Severity	Message Information	Trigger Event
`ERR`	`Error scheduling automatic Service Continuity`	Temporary error trying to schedule a task to set `serviceContinuity`
`ERR`	`Error while setting automatic Service Continuity`	Temporary error trying to set `serviceContinuity` in BCCluster
`ERR`	`Not possible to send <type_message> message <error_message> <exception_trace>`	Temporary problems sending status information.
`ERR`	`Instance <systemMonitor _id> can’t publish messages in Messaging Service, reporting node as down`	SM is not able to publish messages and gave up on retries.
`ERR`	`SM couldn’t publish the <mesg_subject> message, retrying`	Temporary problem while publishing messages.
`ERR`	`Error connecting to Messaging Service <servers_ip>`	SM is not able to connect to Messaging Service.
`ERR`	`Not possible to handle incident in site <site_id> for D<dsgGroup_id> because is not possible to obtain partition status` `Error while trying to detect a local incident for D <dsgGroup_id> in node <node_id> in site <site_id> . Incident has not been raised <exception_trace>` `There is an error when handling a SDL incident in site <site_id> for D<dsgGroup_id> <exception_trace>` `Not possible to handle incident in site <site_id> for D<dsgGroup_id> <exception_trace>` `Incidents handler error <exception_trace>`	Temporary problems handling incidents.
`ERR`	`Master election or SDL generation for D<dsGroup_id> has being interruped in the middle. Still trying to finish it`	Temporary problems at leader election.
`ERR`	`MastersChecker in site <site_id> can not get SDL info info in masterpath <path> from site <site_id> because node does not exist in BC server. Check your system configuration`	Information path in the BC cluster does not exist.
`WARNING`	`Problem getting SDL info for D<dsgGroup_id> in node <node_id>. Probably CS is not reporting or the node is down` `Problem getting RPI for D<dsgGroup_id> in node <node_id>. CS is not reporting RPI. Active CS is reporting <status>` `Problem getting RPI for D<dsgGroup_id> in node <node_id>. Node <node_id> does not exist in the BC server`	Temporary problem getting status or replication information for a DSG.
`WARNING`	`Not possible elect master (masterless) for D<dsgGroup_id> because there is no RPI available. Probably all the DSG replicas are down or non reporting. Election done by <systemMonitor_id>`	No RPI information is preventing SM to make a master election.
`WARNING`	`It has been not possible to report master election info (<systemMonitor_id>) in site <site_id> because <dsgGroup_id> does not exist (probably DSG it is not configured)`	Problem reporting master election info.
`WARNING`	`Not possible to spread SDL info for D<dsgGroup_id> in site <site_id> because <dsgGroup_id> node does not exists (probably DSG it is not configured)` `SM <systemMonitor_id> PL going down. Messages will not be sent again until PL is alive` `Not possible sending EXPIRE multicast with connection with BC service lost <exception_trace>`	Problems are avoiding to send status information.
`WARNING`	`D <dsgGroup_id> is down (CS reporting down, CS non reporting or local PL is down or no PL alive in site or node in disabledNode list) in node <node_id> (master) in site <site_id>. Raising master incident`	Master incidents triggered.
`WARNING`	`D0 (PL) is down or has stopped reporting in node <node_id> in site <site_id>. All the DSGs in node <node_id> will be considered down`	PLDB is down. All DSGs in node are considered as down.
`WARNING`	`Not possible to evaluate connection state change between sites <site_id> and <site_id> because evaluation has been interrupted. Probably leader is being relinquished` `Not possible to evaluate connection state change between sites <site_id> and <site_id> <exception_trace>` `Not possible to evaluate connection state change between sites <site_id> and <site_id> <exception_trace>` `System Monitor <systemMonitor_id> not able to connect to local BC servers. Retrying...`	Temporary problem in the connection to the BC Clusters.
`WARNING`	`Not valid master value in partition for D <dsgGroup_id>. New master election needed`	New master election is needed.
`WARNING`	`Site <site_id> set as auto-removed for node <node_id>`	The indicated site is down, therefore it is set as auto-removed for the nodes of other sites.
`WARNING`	`Site <site_id> set as non auto-removed for node <node_id>`	The indicated site is up and running, therefore it is set as non auto-removed for the nodes of other sites.
`WARNING`	`Detected Master DSG: <dsg_id> in a disabled node: <remote_node_id> from node: <nodeId>, check configuration if the problem persists.`	The local node configured a remote node with masters as disabled.
`WARNING`	`Detected Master in a disabled DSG: <dsg_id> on node: <remote_node_id> from node: <nodeId>, check configuration if the problem persists.`	The local node configured the master DSG of a remote node as disabled.
`WARNING`	`Service Continuity couldn't be set automatically`	`serviceContinuity` not set because partition status is empty in `BCCluster`
`WARNING`	`Client disconnected from Messaging Service!`	SM got disconnected from Messaging Service instance and will try to reconnect.
`WARNING`	`Client connection closed to Messaging Service!`	SM got disconnected from Messaging Service instance.
`WARNING`	`Client reconnected to Messaging Service in <uri>!`	SM needed reconnection to Messaging Service cluster and it was successful .

Refer to CUDB High Availability, Reference [2] for further information on these logs.

2.5.4.3.2 CS

The CS component uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, messages with severities info and above are logged.

CS logs have the following types of severities:

EMERG (see Table 16).
CRIT (see Table 17).
ERR (see Table 18).
WARNING (see Table 19).

Table 16 shows CS logs with severity EMERG.

Table 16 EMERG CS Logs
`EMERG` Message Information	Trigger Event
`Can not find multicast address. CS EXIT`. `Failed to setup multicast socket for receiving. CS EXIT`.	Failed setup multicast socket.
`Failed to tasket angel ndbd (command via ssh failed)!` `Failed to check ndbd child process`. `Missing ndbd child process of parent ndbd with pid <pid>`. `No MGM available`. `Failed to wait for NDB`. `Failed to wait for NDB, MGM nodes unreachable!` `<node_type> <node_id> on <IP> not ready`. `Failed to start <node_type> <node_id> on <IP> (it died or program not found)!`	Failed starting MySQL cluster processes.
`Can't stop NDBD processes`. `Can't stop <node_type> <node_id> on <IP> (pid=<pid>)`.	Failed stopping MySQL cluster processes.
`Failed to read configuration`. `No multicast channel specified for sending cluster status`.	Failed to read configuration.
`Error creation of binlog index for host=<host_name>:<port>`.	Failure in the creation of binlog index.
`Unknown command`	Unknown command read from command file.

Table 17 shows CS logs with severity CRIT.

Table 17 CRIT CS Logs
`CRIT` Message Information	Trigger Event
`Failed to setup multicast sender`.	Error setting/receiving multicast.
`Unable to set connectstring <connect_string>`. `Unable to create handle to ndb_mgmd`.	Failure connecting to a MGMD.
`Received NULL pointer`. `File not found (errno=<error_code>):<file_name>`. `Path not valid (errno=<error_code>):<file_name>`. `Permission denied (errno=<error_code>):<file_name>`. `Failed loading config`. `No <CudbSystem> root element in <path_file>`. `No local node id found in <path_file>`. `Unhandled exception`. `File name is NULL`. `getDocument returned NULL:<file_name>`. `Failed loading CudbPlDataMemoryGEP3.` `Failed loading CudbDsDataMemoryGEP3.` `Failed loading CudbPlIndexMemoryGEP3.` `Failed loading CudbDsIndexMemoryGEP3.`	Failure reading/parsing configuration file.
`Running out of memory`.	NDBD running out of memory.

Table 18 shows CS logs with severity ERR.

Table 18 ERR CS Logs
`ERR` Message Information	Trigger Event
`Kickstart not allowed in this scenario, can only be done after initial installation, a system data restore or a group data restore.`	Kickstart order is executed in the wrong situation, can be used only at installation and after system or group data restore.
`No valid mysql connection`. `initializeSQLAfterStart initialize failed for node <node_id> on host <host_name> with error:<error_code>`. `initializeSQLAfterStart connection test failed for node <node_id> <server> on host <host_name>`. `ReplicationDistanceCalculator::calcChannelStatus: connection to local slave failed for node <node_id>on host <host> with error: <error message>`. `(Line <line where exception was thrown>)in ReplicationDistanceCalculator::calcSlaveLastEpoch: connection to local slave failed with error: <error message>` `(Line <line where exception was thrown>)in ReplicationDistanceCalculator::calcDelay: getReplicationDistance connection to remote master failed: host <host> with error: <error message>`.	Cannot connect to `mysqld` server.
`connectToMgm: unable to retrive cudb_ds!` `connectToMgm: unable to create MGMD handle!` `connectToMgm: unable to set connection string!` `connectToMgm: unable to connect with con str <connection_string>. Err: <error_code> <error_message>`. `None of MGM nodes available`. `connectToMgm: unable to create mgm handle!` `No MGM`. `connectToMgm: unable to retrive cudb_ds!`	Error connection to MGMD.
`Enter single user failed:<error_messsage>`. `Exit single user failed:<error_messsage>`.	Error entering or exiting single user mode.
`(Line <line where exception was thrown>)in ReplicationDistanceCalculator::calcDelay: mysqlWrapperException getReplicationDistance failed: host <host> with error: <error message>`.	Error fetching information from the `binlog` index of remote master.
`checkClusterStatus connectToMgm failed`. `Error checking cluster status`. `check status failed to query node states`.	Error getting cluster status.
`initializeSQLAfterStart initialize failed for node <node_id> on host <host_name>! Error setting root password!` `initializeSQLAfterStart initialize failed for node <node_id> on host <host_name>`.	Error handling passwords.
`Aborting - No mysqld login information found in XML config file for CUDB-node <node_id>, DS <ds_group_id>`. `DSG <ds_group_id> not present in <path_file>`. `Could not check/read: <file_name>`. `Could not delete: <file_name> (<error_message>)`. `Could not write to <file_name>`.	Error in configuration file.
`API <node_id> not being a master or slave`.	Error in `mysqld` configuration.
`MySQL execution error:<error_code>:<error_message>, <query>`. `Nowhere to store result set`. `Empty resultset:<query>`.	Error in MySQL query.
`Binlog index does not exist for host=<host_name>:<port>`. `Out of memory in ReplicationState::Instance()`.	Error in replication process.
`Setup error: socket not initialized`. `select() socket failed: errno=<error_code>`. `failed receiving: errno=<error_code>`. `Could not create TCP socket with socket(): errno=<error_code>`. `Could not bind TCP socket (port=<port>): errn0=<error_code>`. `Could not get IP address with getsockname(): errno=<error_code>`. `Could not put TCP socket in listening state: errno=<erros_code>`. `select() socket failed: errno=<error_code>`. `Failed to accept() TCP connection: errno=<error_code>`. `Setup error: socket not initialized`.	Error setting/creating a socket.
`Failed to receive on multi-cast socket`.	Error setting/receiving multicast.
`failed to execute: <command>` `Unknown command: <command>`.	Error in ssh command.
`Error received in sigwait, error value EINVAL`. `Error received in sigwait, error value ENOTSIGINIT`.	Error in the thread handling.
`initializeSQLAfterStart initialize failed for node <node_id> on host <host_name> with error:<error_code>`. `initializeSQLAfterStart connection test failed for node <node_id> <server> on host <host_name>`.	Error trying to connect to MySQL server.
`Invalid backup file descriptor`. `Restore failed`. `Restore did not complete successfully`.	Failed data restore.
`Failed to allocate memory, error=<error-code>. Permanent Error. Aborting connection to host=<host_name> port=<port>`.	Failed to allocate memory.
`Unable to determine starting point in binary log`. `replcfg_get_binlog_lastevent: unable to get last event`.	Failure checking binlog file.
`Master for replication channel <channel> in DS group <ds_group_id> not found int CUDB node <node_id> in XML config file`.	MasterDSG not found in configuration.
`COMM_MC: Master-list expired (TTL of <time_in_milliseconds> ms reached) : Master=* NONE *`.	Masterlist info expired.
`(Line <line where exception was thrown>)in ReplicationDistanceCalculator::calcSlaveLastEpoch: curMaster is neither lastEffectiveMaster nor lastSetupMaster [tag_LastEffectiveMaster] = <last effective master id>[tag_LastSetupMaster] = <last setup master id>[tag_CurrentMaster*] = <current master id>`.	Replication status is inconsistent with mastership info.
`Slave for DS group <ds_group_id> not found int CUDB node <node_id> in XML config file`.	Slave DSG not found in configuration.
`Unable to open restore pipe`. `Unable to close pipe`.	Unable to open/close ssh pipe for a command.
`Unable to retrieve last epoch on slave`. `Unable to retrieve last epoch on new master`.	Unable to retrieve last epoch.
`Check Orphan files: Failed to read list of binlogfiles from binlogfolder on host <host_name>:<port>!`. `Check running purge sessions on host <host_name>:<port> failed with error <mysql_error_description>` `Connection to <host_name>:<port> failed with error:<error_description>.` `Load list of binary logs from <host_name>:<port> failed with error:<error_description>.` `Purge on <host_name>:<port> failed with error:<error_description>. Purge file <filename>, purge file parameter <filename>.` `Check 'purge_session' in schema failed on <host>:<port> failed with error: <error_message>."` `Check Number Of Binlog Files By Timestamp: Failed to read list of binlogfiles from binlog folder on host <host_name>:<port>!`	Errors related to the purge and orphan binlog files check
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: <tool> handler fetching data. ERROR DESCRIPTION:: Missing <missing_data>. Cannot continue with <tool>.`	Finite state machine handler was not able to get some data (`node_id`, `timestamp`, `task_id`, ...) needed for proper execution and checking status of tool (Data Repair, Selective Replica Check). Missing data should be saved internally in state machine.
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: SOBR <state>. ERROR DESCRIPTION:: Data fetch error: Missing <missing_data>. Cannot continue with SOBR.`	Finite state machine handler was not able to get some data (`node_id`, `timestamp`, `task_id`, ...) needed for proper execution and checking status of Self-Ordered Backup and Restore in the current state. Missing data should be saved internally in state machine.
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: <tool> finished. ERROR DESCRIPTION:: <tool> did not finish correctly on <node>, exit code was <exit_code>`.	Tool (Data Repair, Selective Replica Check, Self-ordered Backup and Restore) exited with error exit code. Error exit codes for Selective Replica Check are 1, 3, 5, 7. Error exit codes for Data Repair are 1, 3, 5. Error exit code for Self-Ordered Backup and Restore is 1. Error code greater or equal to 126 indicates some other problem (wrong permissions, missing binary, and so on).
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: <tool> running. ERROR DESCRIPTION:: Unable to extract exit code of <tool> on <node>, got <return_string>`.	Tool (Data Repair, Selective Replica Check, Self-Ordered Backup and Restore) finished, but something went wrong parsing return code from output file.
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: <tool> running. ERROR DESCRIPTION:: Unable to get status or exit code of <tool> on <node> after retrying. Giving up.`	Could not get status of currently running tool (Data Repair, Selective Replica Check, Self-Ordered Backup and Restore).
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: <tool> start. ERROR DESCRIPTION:: <tool> unable to start on <node> after retrying. Giving up.`	Could not start tool (Data Repair, Selective Replica Check, Self-Ordered Backup and Restore) over ssh on node.
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: <tool> start. ERROR DESCRIPTION:: Unable to get PID of <tool> on <node>, got <returned_string>.`	Tool (Data Repair, Selective Replica Check, Self-Ordered Backup and Restore) started on node, but could not extract PID from returned string.
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Calculating missing intervals. ERROR DESCRIPTION:: Unable to calculate missing intervals on <node> after retrying. Giving up.` `FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Calculating missing intervals. ERROR DESCRIPTION:: Unable to calculate missing intervals on <node>, got <return_string>`.	Unable to calculate total amount of missing intervals after Selective Replica Check scan execution.
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Reconciliation. ERROR DESCRIPTION:: Can't trigger reconciliation, target PLDB's DSG list is empty`.	Failed to get DSG list for PLDB from Blackboard Coordinator.
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Reconciliation. ERROR DESCRIPTION:: Can't trigger reconciliation, target PLDB hostname list is empty`.	Faild to get PLDB `hostname` from `cudbSystem.xml`.
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Reconciliation. ERROR DESCRIPTION:: Couldn't get list of all DSG's in the system`.	Failed to get list of all DSGs in the system from Blackboard Coordinator.
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Reconciliation. ERROR DESCRIPTION:: Couldn't get master node of incident DSG`.	Failed to get master node of incident DSG from Blackboard Coordinator.
`FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Reconciliation. ERROR DESCRIPTION:: Trigger reconciliation failed, MySQL exception has occurred: <mysql_exception_description>`.	Error in MySQL.
`Repair&Resync: Cannot get data from state machine!`	Incident related data (`timestamp`, `task_id`) needed for the Automatic Handling of Network Isolation cannot be obtained. Data should be present internally in state machine.
`basethread,pthread_condattr_setclock failed` `basethread,pthread_condattr_init failed` `basethread,pthread_cond_init failed`	Error while initializing clock for CS.
`Ndb binlog injector error on Mysql server <host_name>:<port>!`	The ndb `binloginjector` thread failed to subscribe to table events on one or more tables in the cluster. The MySQL server needs to be restarted to fix this problem.
`Start check failed for <node_type> <node_id> on <IP>!` `Process check failed for <node_type> <node_id> on <IP>!`	Error checking process.
`ERROR: Unable to stop ndbmtd <node_id> after two retries, cluster will be put to maintenance`	Error stopping NDB cluster.
`Failed to create thread`	Cannot create thread.
`BackupHandler: Not all the necessary backup files are created.`	Failed to create all the necessary backup files.
`BackupHandler: Not all the necessary backup files are valid.`	Failed to validate all the necessary backup files.
`BackupHandler: Failed to connect to Management server.`	Failed to connect to the NDB Management server.
`BackupHandler: Failed to create logevent handle.`	Failed to create the NDB API logevent handle.
`BackupHandler: Could not get cluster status.`	Could not get cluster status from the NDB API.
`BackupHandlerThread: Failed to recover state.`	Failed to recover backup state during failover.
`BackupHandlerThread: Failed to extract backup location from the command file.`	Failed to extract backup location from the command file.
`BackupHandlerThread: Failed to execute dummy transaction on access nodes.`	Failed to execute dummy transaction on access nodes.
`BackupHandlerThread: Failed to set NDB handles.`	Failed to set NDB handles.
`BackupHandlerThread: Failed to collect connected NDB blades.`	Failed to collect connected NDB blades.
`BackupHandlerThread: Failed to start backup.`	Failed to start backup.
`Failed to connect to IMM: too many failed attempts` `Failed to connect to IMM: connection rejected`	Failed to connect to IMM.
`Failed to initialize IMM OM: too many failed attempts` `Failed to initialize IMM OM: unknown error`	Failed to initialize IMM Object Manager.
`Failed to finalize IMM OM accessor` `Failed to finalize IMM OM`	Failure at disconnection from IMM.
`Failed to read <object> object from IMM: object does not exist` `Failed to read <object> object from IMM: invalid parameter` `Failed to read <object> object from IMM: unknown error` `Unsupported type while reading attribute <attribute> from IMM`	The object or the attribute cannot be read from IMM.
`No uuid was read from IMM`	The `uuid` attribute does not exist in IMM.

Table 19 shows the CS logs with severity WARNING.

Table 19 WARNING CS Logs
`WARNING` Message Information	Trigger Event
`Failed to get cluster state. Management servers might be down`. `Failure getting cluster status from all ndb_mgmd`. `Failure all ndb_mgmd are down`. `Cannot connect to management nodes. Repair skipped for node <node_id>`.	All MGMD down.
`A purge session is already running on <host>:<port>! details: start time:<start time> user:<user_name> Info:<currently_executing_task>`	Binlog purge failure.
`Number of orphan binlogfiles:<number of files not listed in log-bin.index> on <host_ip>:<port>!` `Removing orphan binlogfile <binlogfile>, session_id <session_id>, on <host_ip>:<port>`. `Ndb_binlog_index record count for <binlogfile> is:<record_count>, on <host_ip>:<port>!`	Binlog files not listed in log-bin.index file found.
`Error, CS node doesn't exist, trying to repair`. `Error, DSG node doesn't exist, trying to repair`.	BC server nodes do not exist.
`Unable to connect to mysqld`. `Can't connect to mysql server`. `Failed to connect to mysqld server, error=<error_message> Aborting connection to host=<host_name> port=<port> timeout=<timeout_in_seconds>`. `Failed to connect to mysqld server. Aborting epoch check for host=<host_name> port=<port> and master server-id in (<servr_id>)`. `Failed to connect to mysqld server. Aborting \stop slave\ for host=<host_name>port=<port>`. `Failed to connect to mysqld slave server. Aborting replication start for slave=<host_name>:<port> to master=<host_name>:<port>`. `Failed to connect to mysqld master server. Aborting replication start for slave=<host_name>:<port> to master=<host_name>:<port>`. `Unable to remove local epoch info's from ndb_apply_status, <error_message>`. `ReplicationControl::removeAppliedEpochOnMaster Failed to remove applied epochs from master '<error_message>'`.	Cannot connect to `mysqld` server.
`Start mysqld process failed for node <node_id> on host <host_name>! Unable to connect to cluster management nodes!` `Start mysqld process failed for node <node_id> on host <host_name>! mysqld process not connected to cluster!`	CS could start the `mysqld` process but the process did not connect to the `mysql` cluster within time-out so the start process failed.
`Start mysqld process failed for node <node_id> on host <host_name>! mysqld process cluster connection check failed!`	Newly started `mysqld` process reported to be connected to the `mysql` cluster but it is not reachable (it did not allow connections OR it had lost the connection to the cluster).
`start: Check Available NDBS: NOT enough NDBDs`. `start: wait for cluster failed, no more retries left!` `MySQL slaves are * down `. `Cluster is down `. `Cluster is degraded `. `MGM node is down ` `MySQL masters are down ` `NDB is down *`	Cluster down/degraded.
`Connection with BC-cluster has been broken permanently`. `Connection with BC-cluster has been broken temporarily`.	Connection to BC server broken.
`errono:<error_code>, msg:<error_message>`. `Connection failed`. `Cluster <connect_string> was not ready within <seconds> secs`. `Master <node_id> is not writing to binlog or has not yet receievd any traffic`. `No binlog epoch info on masters is available`.	Error accessing to a database/table.
`Process check failed for <process_name> on <host_name> !`	Error checking `ndb_restore` process.
`Error getting info for CS node`. `Master information received with wrong DSG`.	Error getting information from BC server.
`setSQlPasswords` `initialize failed for node <node_id> on host <host_name> with error:<error_code>`.	CS failed to set the proper user grants during `mysqld` server initialization.
`API <node_id>: check/repair process failed, unable to set 'root' user password`.	CS failed to set the root password. If this happens in CS "ready" state, it will retry the fix in the next iteration. If it happens during cluster "initialization", execute the `cudbManageStore -o initialize` command again.
`API <node_id>: check/repair process failed, unable to set access/replication user password`.	CS failed to set the password for "access" or "replication" user. If this happens in CS "ready" state, it will retry the fix in the next iteration. If it happens during cluster "initialization", execute the `cudbManageStore -o initialize` command again.
`Failed to connect to mysqld server. Aborting \DELETE FROM mysql.ndb_apply_status WHERE server_id=<server_id> for host=<host_name>, port=<port>`. `Failed to delete mysql.ndb_apply_status entries in node=<host_name>:<port> for server=<server_id>`.	Error handling replication in the initialization of the cluster.
`Replication info not gotten`. `Failed to start replication for slave=<host_name>:<port> from master=<host_name>:<port>, start epoch=<epoch>`. `Failed to connect to mysqld slave server. Aborting replication start for slave=<host_name>:<port> to master=<host_name>:<port>`. `Failed to kickstart replication for slave=<host_name>:<port> from master=<host_name>:<port>`. `Failed to connect to mysqld slave server. Aborting replication start for slave=<host_name>:<port>: to master=<host_name>:<port>` `Failed to connect to mysqld master server. Aborting replication start for slave=<host_name>:<port>: to master=<host_name>:<port>` `Started replication to slave=<host_name>:<port>: from master=<host_name>:<port>` `Failed to start replication for slave=<host_name>:<port>: from master=<host_name>:<port>. The epoch of the new master is behind the epoch of the slave`. `Failed to start replication for slave=<host_name>:<port>: from master=<host_name>:<port>` `Failed to connect to mysqld slave server. Aborting replication start for slave=<host_name>:<port>: to master=<host_name>:<port>` `Failed to connect to mysqld master server. Aborting replication start for slave=<host_name>:<port>: to master=<host_name>:<port>` `Can't stop replication on slave Server-id=<server_id>. Slave might be down, continues replication setup anyway`. `Can't get replication start parameters for slaves. Aborting replication setup`. `Can't kickstart replication on slave. Aborting replication setup`. `Failed to get the last binlog event on Slave Node - Temporary error.` `Old master server doesn't have any epoch information - Permanent error.` `Can't start replication. Flow through program. Aborting replication setup`. `No slave server online. Aborting replication setup`. `Sync replication failed - Permanent error.` `Can't get replication start parameters for slaves. Aborting replication setup`. `Can't start replication on slave. Aborting replication setup`. `Can't stop replication on slave Server-id=<server_id>. Slave might be down, continues replication setup anyway`. `New master at <host>:<port> is behind slave at <host>;<port>, leave slave as it is`. `Replication already running on the slave`. `LOST EVENT occurred as last event on Master Node - Permanent error.` `Channels stopped after kickstart, another kickstart is required to fix replication`. `Sql Slave unavailable - Temporary error.` `LOST EVENT occurred in binlog as last event on Slave Node - Permanent error.` `Failed to get the last replicated epoch - Temporary error.` `Sync replication failed - Temporary error.` `Can't get valid serverId. Aborting replication setup.`	Error in replication handling.
`<thread_name> is hanging, finalize`. `No Thread=<thread_id> could be found corresponding to thread`. `The thread has allready been detached OR Another thread is already waiting on termination of the thread`. `The thread argument refers to the calling thread. One can not destroy one self`. `Unkown pthread_join return code`. `MgmThread stop run method early`. `MgmThread stop run method early, standby mode`. `No Thread=<thread_id> could be found corresponding to thread`. `The thread has allready been detached OR Another thread is already waiting on termination of the thread`. `The thread argument refers to the calling thread. One can not destroy one self`. `Unkown pthread_join return code <error_code>`.	Error in thread handling.
`Error in masterList watcher, value read:<master_value>, value stored:<master_info>`. `Error in sdlList watcher, value read:<master_value>, value stored:<master_info>`. `Error in workingMode watcher, value read:<master_value>, value stored:<master_info>`. `Error setting SdlList watcher`. `Error setting Working mode watcher`. `Watcher result not ok, ignoring working mode received`. `Watcher result not ok, ignoring SdlList received`. `Wrong format Sdl list received, ignoring SdlList watcher`. `Watcher result not ok, ignoring Provisioning Assurance info received` `Error setting Provisioning assurance watcher` `Trying to repair Provisioning Assurance watcher` `Error in SOBR info watcher, value read: <SOBR value>, value stored: <SOBR info>` `Error setting SOBR info watcher` `Watcher result not ok, ignoring SOBR info received`	Error in watcher.
`tablespaceMonitoringParams are invalid - defaulting`. `Invalid hostname=<host_name>`. `Empty ESA list in xml config file`. `Error reading configuration information`. `Slave is NULL, slave for DS group <ds_group_id> not found in CUDB node <node_id> in XML config file`. `Slave for DS group <ds_group_id> not found in CUDB node <node_id> in XML config file`. `Master for replication channel in DS group <ds_group_id> not found in CUDB node <node_id> in XML config file`. `Local master servers not found in XML config file for <ds_group_id> <node_id>`.	Error reading configuration.
`Could not fetch cudb_cluster_supervisor.last_master content for node id:<node_id>. Returning zeros`. `Could not store data in cudb_cluster_supervisor.last_master table for node id:<node_id>. Returning zeros`. `Could not clear data from cudb_cluster_supervisor.last_master table`. `Last master table is empty:<node_id>`. `Permanent error when trying to access last_master table for node id:<node_id>`. `Permanent error when trying to clear last_master table`. `Error reading last master info.`	Error reading last master information from the local NDB cluster.
`Send error: errno=<error_code>`.	Error sending a TCP message.
`No multicast channel specified for cluster status`. `Failed to setup multicast receiver`.	Error setting/receiving multicast.
`Failed the disable of the events on binlog, error=<error_message>. Aborting conection to host=<host_name> port=<port>`.	Error setting events on binlog in MySQL.
`Error getting info for DSG node`. `Error creating DSG reporting node in BC-cluster`. `error creating CS node`. `Error deleting DSG node`. `Error updating DSG node with value:<dsgroup_id>, node does not exist`. `Error updating DSG node with value:<dsgroup_id>`. `Error setting MasterList watcher`. `Masterlist watcher result not ok, exit updateCSMasterState`. `Master information received for DSG <dsgroup_id> has wrong format:<info>`. `Error updating CS node with value:<status>`. `BC connection is down, cannot sendRDMStatus()` `Unable to update node with RDM status: <RDM_status>` `Error creating SOBR node in BC-cluster` `Error updating SOBR info node with value: <SOBR info>`	Error updating information to BC Server.
`Unexpected restore output:<message>`. `Error while creating databases`.	Failed data restore.
`Error trying to delete entries in ndb_apply_status for server_id not equal to <server_id>`.	Failed to clear `ndb_apply_status` table.
`Could not fetch Index/Data memory usage`. `Failed to get cluster memory usage`.	Failed to get cluster memory usage.
`Unable to read last binlog event`. `Failed to connect to mysql server. Aborting checking of binlog index for host=<host_name>:<port>`. `Failed checking of binlog index for host=<host_name>:<port>`. `Failed to connect to mysql server. Aborting creation of binlog index for host=<host_name>:<port>`.	Failure checking binlog index.
`NDB <node_id>:check/repair failed`. `API <node_id>:check/repair process skipped, DataPath not found <data_path>`.	Failure checking/repairing MySQL processes.
`Failed to create NdbMgmHandle`. `Failed to set connect string. Connect string=<connect_string>`. `Failed to set timeout. Timeout=<time_in_seconds>`. `Failed to connect to Management server. Connect string=<connect_string>`. `Failed to connect to ndb_mgmd <connect_string>`.	Failure connecting to MGMD.
`Failure in command: du --summarize -BK <path_name>.output:<error_message>`. `Binlog_is_consistent(const char* Host, const char* binlog_path) - command <command> failed`. `Failed to start <node_type> <node_id> on <IP> (command via ssh failed)!` `Kill -9 on <node_type> <node_id> at <IP> (pid=<pid>)`. `Kill -9 on <node_type> <node_id> at <IP> (pid=<pid>) (nameprocess =<name_process>)`.	Failure executing a command.
`Failed to enable corefiles`. `Failed to open file <path_file>`. `Failed to write data to file <path_file>`. `Failed to close file <path_file>`.	Failure handling a file.
`Trying to repair CS node value`. `Couldn't repair CS node value`. `Trying to repair MasterList get`. `Couldn't repair MasterList get`. `Trying to repair MasterList watcher`. `Couldn't repair MasterList watcher`. `Trying to repair SdlList watcher`. `Couldn't repair SdlList get`. `Trying to repair SdlList get`. `Couldn't repair SdlList watcher`. `Trying to repair WorkingMode watcher`. `Couldn't repair WorkingMode watcher`. `Trying to repair SOBR info watcher` `Couldn't repair SOBR info watcher`	Failure repairing BC server node.
`cl_control,Mount point <data_path> not mounted!` `Failed to check if <data_path> is mounted!`	Mount point checking failure.
`New master config recived CUDB_NODE=<node_id> DS_GROUP=<ds_group_id>`.	New master config received.
`COMM_MC: Not receiving any config : Master = * NONE `. `COMM_MC: Received empty master-list: EXPIRE *.`	No information received from multicast.
`Unable to trigger UPDATE dummy operation in CUDB_REPLICATION_DUMMY on server <host_name>:<port>, <error_message>`.	Unable to execute dummy operation.
`Event standby invoked but we are already in standby state`. `Event active invoked but we are already in active state`.	Unexpected AMF change received.
`Received signal not expected, ignoring it`.	Unexpected signal received.
`Cluster status info is not available for DSG <dsgroup_id>`.	CS was not able to ask cluster status from MGM nodes.
`Failed Triggering dummy operation on master DS <masterCUDB>`.	After a master `mysqld` is restarted, the system tries to avoid having lost events in the same position on both replication channels. In this case it failed to generate a dummy event on the DSG master replica.
`Clean stop procedure for Mysql server with ndb node id <node_id> failed on <host_ip> with error: <error_message>. Killing the server.`	CS failed to shut down the `mysqld` instance in a graceful way. The `mysqld` process is stopped in a forced way (using `'kill -15'`, followed by `'kill -9'`).
`Ndb Binlog Index is corrupted! Node <cudb_node_id>, DSG <ds_group_id> API node ID <node_id>, Server ID <server_id>. Starting reinitalization.`	The check of ndb binlog index table found inconsistency after starting or restarting master MySQL server. Inconsistency can have been caused by a non-graceful stop of the `mysqld` process or a file system error.
`Error executing Ndb Binlog Index check. Binlog state unknown.`	CS could not execute the check statement on master MySQL server after starting or restarting it.
`FSM: State Machine not running`. `FSM: No transition from state <state> with event <event>`. `FSM: State already exists`.	Error in finite state machine.
`FSM <handler>: Problem executing <command> remotely on <node>.`	Error executing command over ssh.
`Repair&Resync: Timestamp value is 0!`	Attempt to start Automatic Handling of Network Isolation without a valid incident timestamp. Incident timestamp is read from Blackboard Coordinator.
`Repair&Resync: Cannot start finite state machine, exiting from thread.`	Attempt to start the state machine for the Automatic Handling of Network Isolation when it is already running.
`Repair&Resync: Cannot initialize csDbIo object`. `Repair&Resync: Cannot get info about last master`.	Failed attempt to fetch information about last master from `ClusterSupervisor` database (MySQL).
`Failed to start <node_type> <node_id> on <IP> (it died or program not found)!` `No MGM available` `Failed to wait for NDB` `Failed to wait for NDB, MGM nodes unreachable!` `Missing ndbd child process of parent ndbd with pid <pid_process>` `Failed to check ndbd child process`	Failed to start MySQL cluster processes.
`Can't stop NDBD processes` `Can't stop <node_type> <node_id> on <IP> (pid=<pid_process>)`	Failed to stop MySQL cluster processes.
`Cluster state is down! No nodes have been started!`	Error getting cluster status.
`RDM status is unknown, not updating BC cluster`	Error getting RDM status.
`Error Calculating timestamp to PG, binlog epoch in master is lower than last epoch in slave table apply status, check if master has been initialized or a system restore is in place.`	Error calculating timestamp to PG
`Rollover has not been successfully completed. Another rollover attempt will be made.`	`ClusterSupervisor` instance rollover cannot be completed because BC connection cannot be established or DSG node cannot be created in BC. Reattempting another rollover!
`BackupHandler: Error getting next event`	Can not get the next event from the NBD API.
`BackupHandler: Error writing backup information into file: <meta file path> with error: <error message>`	An error occurred while trying to write backup information into the meta file.
`BackupHandler: Validation failed for node <node id> with MGM hosted on <ip>.`	Failed to verification the backup for the given node.
`BackupHandler: Problem executing <command>.`	Problem executing the NBD restore command to verify the backup.

2.5.4.3.3 KeepAlive

KeepAlive component uses syslog as logging system with the following properties:

Stored in /var/log/<hostname>/messages.
By default, only messages with severity info, warning, or emergency are logged.

The messages logged by the KeepAlive components are summarized in Table 20.

Table 20 KeepAlive Logs
Severity	Message Information	Trigger Event
`ERR`	`Error running <description>`	Error getting `KA_Cycle` value from `keepAlive.xml`.
`ERR`	`<error_description>`	Error getting `KA_CHECK` value from `keepAlive.xml`.
`ERR`	`Incorrect execution, it must NOT begin with '.'`	The process was launched in a wrong way.
`WARNING`	`Monitoring disabled`	Warning about the monitor is not enabled and has not check anything.
`WARNING`	`Component '<component_name>' has capability 1_ACTIVE: it should NOT receive standby assignment from AMF`	The component cannot change its state to standby.
`WARNING`	`Component '<component_name>' has capability 1_ACTIVE: this library does not support assingment removal for this capability`	No assignment is given to the component.
`WARNING`	`Command '<command>' could not be executed: system() call failed with result: <result_code>` `Command '<command>' was not terminated normally`.	Error executing the given command.

2.5.4.3.4 High Availability

High availability component uses syslog as logging system with the following properties:

Stored in /var/log/<hostname>/messages.
By default, only messages with severity info, warning, or emergency are logged.

The messages logged by the high availability components are summarized in Table 21.

Table 21 High Availability Logs
Severity	Message Information	Trigger Event
`WARNING`	`Command <command> could not be executed: system call failed with result: <result_code>`	A system call has failed when executing a command.
`WARNING`	`Command <command> was not terminated normally`	A command execution has failed for some reason.
`WARNING`	`Component <component_name> has capability 1_ACTIVE: it should NOT receive standby assignment from AMF`	Component `CudbPmAgentMonitor` received a `STANDBY` assignment whereas its configuration is `ACTIVE`.
`WARNING`	`Component <component_name> has capability 1_ACTIVE: this library does not support assignment removal for this capability`	`NO ASSIGNMENT` state is not supported for this component.

2.5.4.3.5 Security Logs Monitor

The Security Logs Monitor component uses syslog as logging system with the following properties:

Logs are stored in /var/log/SC_2_*/messages.
By default, only messages with severity info and error are logged.

Security Logs Monitor uses the Availability Management Framework (AMF) service. The messages logged by this component are summarized in Table 22.

Table 22 Security Logs Monitor
Severity	Message Information	Trigger Event
`ERR`	`AmfListener.cpp::callWithRetryNoChk <line number> - Too many failed connect attempts`	An error occurred while using the AMF service.
`ERR`	`not SA_AIS_OK: <value>`	An error occurred while using the AMF service.
`ERR`	`Exception caught in SecLogMonitor.cpp::operator <line number> - Cannot configure syslog for Security Log streaming`	Unexpected error while the external security events logging is being enabled.
`ERR`	`Exception caught in SecLogMonitor.cpp::operator <line number> - Cannot delete configuration for syslog for Security Log streaming`	Unexpected error while the external security events logging is being disabled.
`ERR`	`Exception caught in SecLogMonitor.cpp::operator <line number> : <exception>`	Unexpected error while the external security events logging is being enabled or disabled.
`ERR`	`Exception caught in main.cpp::main <line number> - Unhandled exception occurred`	Unexpected error during Security Log Monitor process execution.

2.5.4.3.6 Replica Consistency

The Replica Consistency component uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, only messages with severity info and above are logged.

Replica Consistency logs have the following types of severities:

ERR (see Table 23).

Table 23 shows the Replica Consistency logs with severity ERR.

Table 23 ERR Replica Consistency Logs
`ERR` Message Information	Trigger Event
`Unable to get the system configuration file on node <node_id> - <res_string> - <node_IP>`	Can not get the `cudbSystem.xml` from the remote Node.
`No database is specified. Use [-p \| -d <dsgId>] or [--pl \| --dsg <dsgId>] or see help`	No DSG to backup was specified with the command line arguments.
`DSG <dsg_id> does not exist in the CUDB system`	The specified DSG ID to backup does not exist in the CUDB system.
`No destination node is specified. Use -n or --node or see help`	No destination node was specified with the command line arguments.
`Node <node_id> does not exist in the CUDB system`	The specified Node does not exist in the CUDB system.
`DSG <dsg_id> has no replica on source node <node_id>`	The specified DSG ID to backup has no replica on the source Node.
`DSG <dsg_id> has no replica on destination node <node_id>`	The specified DSG ID to backup has no replica on the destination Node.
`Unable to determine the holder of Master replica for DSG <dsg_id>, no information in the BC cluster`	Can not get any information from the BC cluster.
`Unable to determine the holder of Master replica for DSG <dsg_id> from the BCClient info`	Can not determine the holder of Master replica for the given DSG from the information available in the BC cluster.
`Unable to determine the holder of Master replica for DSG <dsg_id> from the MySQL info`	Can not to determine the holder of Master replica for the given DSG from the MySQL info.
`Node <node_id> has the Master replica for DSG <dsg_id>`	The destination Node has the Master replica for the specified DSG.
`Unable to determine the holder of Master replicas, no information in the BC cluster`	Can not determine the holder of Master replicas from the information available in the BC cluster.
`The local node has Master replica(s)`	The local node has Master replicas before the backup transfer or restore procedure while processing PLDB replica.
`Some of the blades hosting master replica for DSG <dsg_id> are unreachable via ssh but their ndbs are running`	Some of the blades hosting master replica for the specified DSG are unreachable via ssh but their ndbs are running.
`Unable to copy backup files from <blade_id>`	Can not copy backup files on the source blade from the backup location to the temporary directory.
`DS Mastership changed`	DS Mastership changed before the backup transfer or restore procedure.
`Backup creation was unsuccessful`	Failed to parse the backup location from the `cudbManageStore` output.
`Unable to get the number of running ndb processes for DSG <dsg_id>`	Can not get the number of running ndb processes for the specified DSG.
`Command execution on <local/remote> node <node_IP> failed: Unable to create backup on <node_id>`	Can not create backup on the Master node.
`Command execution on <local/remote> node <node_IP> failed: Unable to create backup directory <backup_location> on <node_id> on blade <blade_id>`	Can not create backup directory on the destination Node.
`Command execution on <local/remote> node <node_IP> failed: Unable to create backup dir under /cluster/tmp/backups on Master Node <node_id>`	Can not create backup directory on the Master Node.
`Command execution on <local/remote> node <node_IP> failed: Unable to create backup dir under /cluster/tmp/backups on Node <node_id>`	Can not create backup directory on the destination Node.
`Command execution on <local/remote> node <node_IP> failed: Unable to copy backup files to <node_id>`	Can not copy backup files to the destination Node.
`Command execution on <local/remote> node <node_IP> failed: Unable to copy backup files from the temporary dir to the backup location on <blade_id>`	Can not copy backup files from the temporary directory to the backup location on the target blade.
`Command execution on <local/remote> node <node_IP> failed: Unable to remove backup dir under /cluster/tmp/backups on <node_id>`	Can not remove the backup directory on the destination Node.
`Command execution on <local/remote> node <node_IP> failed: Unable to remove backup dir <backup_location> on <blade_id>`	Can not remove backup directory on the source Blade.
`Command execution on <local/remote> node <node_IP> failed: Unable to remove backup dir under /cluster/tmp/backups on <node_id>`	Can not remove backup directory on the Master Node.
`Command execution on <local/remote> node <node_IP> failed: Unable to restore backup on <node_id>`	Failed to restore backup on the destination Node.
`Unable to remove backup transfer flag on node <node_id>`	Can not remove the backup copy flag on the Master Node.
`Unable to remove backup transfer flag dir on node <node_id>`	Can not remove the backup copy flag directory on the Master Node.
`Unable to create lock directory on node <node_id>`	Can not create lock directory on the Master Node.
`Unable to get the copy lock on node <node_id>`	Can not get the copy lock on the Master Node.
`Unable to create the backup ready file on node <node_id>`	Can not create the backup ready file on the Master Node.
`Unable to create transfer flag file on node <node_id>`	Can not create transfer flag file on the Master Node.
`Unable to create transfer flag directory on node <node_id>`	Can not create transfer flag file directory on the Master Node.

2.5.4.3.7 Licensing

Licensing component uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, only messages with severity info, warning, or emergency are logged.

The messages logged by the Licensing subcomponent are summarized in Table 24.

Table 24 Licensing Logs
Severity	Message Information	Trigger Event
`CRIT`	`Problem reading cudbSystem.xml, can't start process! Exiting!`	System configuration file `/cluster/home/cudb/common/config/cudbSystem.xml` cannot be read.
`ERR`	`Capacity license for <profile_name> profile not found!` `Problem with getting capacity license for <profile_name> profile.`	Error getting capacity license for requested profile.
`ERR`	`Capacity limit for <profile_name> profile is exceeded!`	Capacity limit for requested profile is exceeded.
`ERR`	`Capacity license for <profile_name> profile is rejected for the following operational mode: <operational_mode>`	License Manager (LM) is in the operational state that enforces rejection of request for capacity license.
`ERR`	`License Manager handler couldn't be initialized, can't check licenses`	An error occurred while initializing LM handler.
`ERR`	`Requested capacity license doesn't exist`	Capacity license for requested profile is not loaded.
`ERR`	`Capacity license for <profile_name> profile is rejected for the license type: <license_type>`	LM has license type that enforces rejection of request for capacity license.
`ERR`	`Wrong version of LM! Can't use License Manager API!`	Wrong LM version detected.
`ERR`	`Application ID is invalid. Problem with License Manager.`	Attempt to use API version not supported by LM.
`ERR`	`Registration to License Manager failed!`	An error occurred trying to register LM.
`ERR`	`Could not find any license type!`	None of the allowed license types are loaded.
`ERR`	`AMF initialization failed, shutting down process.`	Could not initialize AMF service.
`ERR`	`AMF problem, shutting down process.`	An error occurred while using the AMF service.
`ERR`	`Could not connect to LDAP!`	An error occurred while connecting to the CUDB LDAP interface to fetch accounting information.
`ERR`	`Failed to set signal mask for thread, exiting!`	An error occurred while modifying the signal mask for the calling thread.
`ERR`	`Error in sigwait.`	An error occurred while waiting for a signal.
`ERR`	`Failed to raise an alarm. Diagnostic info: <diagnostic_information>` `Failed to clear an alarm. Diagnostic info: <diagnostic_information>`	Diagnostic info: `<diagnostic_information>`.
`ERR`	`init - saImmOmInitialize returned error`	Failed to initialize IMM Object Manager.
`ERR`	`init - saImmOmAccessorInitialize returned error`	Failed to initialize IMM Object Manager Accessor.
`ERR`	`readFingerprint - error fingerprint could not be read or is empty!`	Fingerprint cannot be read or it is empty.
`WARNING`	`init - saImmOmInitialize returned SA_AIS_ERR_TRY_AGAIN`	Temporary problem initializing an IMM Object Manager.
`WARNING`	`init - saImmOmAccessorInitialize returned SA_AIS_ERR_TRY_AGAIN`	Temporary problem initializing an IMM Object Manager Accessor.
`WARNING`	`readFingerprint - saImmOmAccessorGet_2 returned SA_AIS_ERR_TRY_AGAIN`	Temporary problem reading fingerprint.

2.5.4.4 Notifications

This section describes the subcomponent of the notifications component.

2.5.4.4.1 SOAP

The notifications component uses syslog as logging system with the following properties:

Stored in /var/log/PL_2_X/messages of the payload blades or Virtual Machines (VMs) where the notifications processes are running (X being the ID of the payload blade or VM).
By default, only messages with severity info, warning, or emergency are logged.

The messages logged by the notifications component are summarized in Table 25.

Table 25 Notifications Logs
Severity	Message Information	Trigger Event
`ERR`	`<error_message> <file_name> <number_line_file>`	This generic message is returned mainly by CUDB checker when an error has occurred retrieving some objects from database.
`ERR`	`Configuration Configuration Error! Source:<component> Description:<error_description>`	Error occurred while the configuration was being parsed. Information on configuration component and detailed description of the error are attached. `component` denotes the `source/component` context.
`ERR`	`Amf listener Error while initializing AMF component! <error_description>`	Error occurred while AMF service is configured.
`ERR`	`Cannot get instance of CudbConfig class. Aborting.`	Error occurred in attempt to get configuration data from data model.
`ERR`	`Not possible to subscribe to SM.`	The Notification component is not able to get info from system monitor.
`WARNING`	`Error received from Notification <endpoint>`	Error has occurred while evaluating the notification result: either a wrong HTTP code from FE has been received or a SOAP client or server fault is encountered.
`WARNING`	`Broadcast Notification EP (<EP>) is marked as unavailable`	Broadcast endpoint is marked as unavailable due to too many connection errors.
`WARNING`	`Notification was not sent to broadcast EP (<EP>) because it was marked unreachable`	Notification was not sent to broadcast endpoint because it was unreachable.
`WARNING`	`Internal Notification error`	CUDB Database Events internal notification error.
`WARNING`	`Internal Notification error: attrList size is 0`	CUDB Database Events internal notification error: attribute list is empty.
`WARNING`	`Internal Notification error: dnObjectList size is 0`	CUDB Database Events internal notification error: object list is empty.
`WARNING`	`No received response from Notification endpoint <endpoint>`	No response from endpoint received.
`WARNING`	`Notification endpoint invalid:<endpoint>`	Invalid notification endpoint was specified.
`WARNING`	`Notification EP (<endpoint>) is marked as unavailable`	Endpoint is marked as unavailable due to too many connection errors.
`WARNING`	`Notification FE connection error to <endpoint>`	Connection error to endpoint.
`WARNING`	`Notification FE SOAP other error at <endpoint>`	Connection error to endpoint.
`WARNING`	`Notification FEs not configured`	The SOAP interface was unable to send notification as SOAP message, or accessing the configuration to get endpoints has failed, or no notification FEs are configured.
`WARNING`	`Due to unavailable SOAP EPs notification message is discarded: <DN>`	Notification message was not delivered to any WRR SOAP endpoints as all of them were unavailable.
`WARNING`	`Due to the unavailable SOAP EP notification message is discarded: <DN>`	Notification message was not delivered to the single configured WRR SOAP endpoint as it was unavailable.
`WARNING`	`<error_message> <file_name> <number_line_file>`	This generic message is returned mainly by CUDB checker when an error has occurred retrieving some objects from database.
`WARNING`	`Client disconnected from Messaging Service`	Notification process got disconnected from Messaging Service instance and it will try to reconnect.
`WARNING`	`Client connection closed to Messaging Service`	Notification process got disconnected from Messaging Service instance.
`WARNING`	`Client reconnected to Messaging Service in <uri>!`	Reconnection attempt to Messaging Service cluster was successful.

2.5.4.5 OAM

This section describes the subcomponents of the OAM component.

2.5.4.5.1 Software Management

Software Management component uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, only messages with severity info, warning, or emergency are logged.

The messages logged by the software management subcomponent are summarized in Table 26.

Table 26 Software Management Logs
Severity	Message Information	Trigger Event
`ERR`	`Incorrect number of parameters`	There was a flag or parameter command unknown.
`ERR`	`Error has occurred`	An error happened in the lines above in the SQL sentences.
`ERR`	`create::ERR: A backup name must be given, aborting`	In the create procedure, the backup name parameter was wrong or it was not specified.
`ERR`	`check_lock::ERROR: applyConfig action is ongoing, aborting.`	Software and configuration backup/restore is started while `applyConfig` action is ongoing.
`ERR`	`check_lock::ERROR: /home/cudb/oam/configMgmt/commands/config/cudbOiImmChanges.txt exists, aborting.`	Software and configuration backup/restore is started while there are pending configuration changes that are not applied with `applyConfig` action.
`ERR`	`check_lock::ERROR: Timeout while waiting OIWorker to write in cudbOiImmChanges.txt and generate configuration files.`	Timeout expired while software and configuration backup/restore was waiting for commit of the configuration changes to release the lock.
`ERR`	`check_lock::ERROR: OIWorker failed to generate configuration files.`	Generation of the configuration files failed.
`ERR`	`create::ERR: Backup lock file exists`	The file to lock a backup procedure exists. This file avoids a new backup when there is another one running.
`ERR`	`create::ERR: backup failed, could not pushd <root_path_backup>, failed`	The path where all backups are going to be created does not exist.
`ERR`	`create::ERR: backup failed, creation of cudb backup location, mkdir <backup_file_location> failed`	The path where the backup is going to be stored could not be created.
`ERR`	`create::ERR: cudb complete backup <backup_id> already exists, aborting`	A new backup with the same identifier as a previous one cannot be created. The name given to the `backup file (<backup_id>)` already exists, therefore it cannot be created
`ERR`	`create::ERR: Incomplete cudb backup named <backup_id> already exists, file <backup_file_location> exists, aborting`	A file is blocked to avoid another backup process being launched at the same time.
`ERR`	`create::ERR: configuration read failed, aborting`	An error occurred in reading node configuration.
`ERR`	`create::ERR: compress failed, some error, aborting`	Compressing the files under `/home/cudb` failed.
`ERR`	`restore::ERR: A backup name must be given, aborting`	In the restore procedure, the backup name parameter is wrong or it is not specified.
`ERR`	`restore ::ERR: backup failed, could not pushd <root_path_backup>, failed`	The path where the backups are stored does not exist.
`ERR`	`restore::ERR: cudb backup failed, cudb backup location directory does not exist, aborting`	The path where the backup is going to be stored could not be created.
`ERR`	`restore::ERR: cudb backup file <backup_id> does not exist, aborting`	The backup to be restored does not exist.
`ERR`	`restore::ERR: configuration read failed, aborting`	An error occurred in reading node configuration.
`ERR`	`restore::ERR: Node Id and HW type check fails, aborting`	The current node ID or hardware type do not match with the backup.
`ERR`	`restore::ERR: cudb sql backup file <backup_file>-cudbSmpConfig.sql does not exist, aborting`	The SQL file related to the backup specified does not exist.
`ERR`	`restore::ERR: CMW backup file <backup_file_tar> does not exist, aborting`	The Core MW backup file does not exist.
`ERR`	`create::ERR: A backup name must be given, aborting`	In the create procedure the backup name parameter was wrong or it was not specified.
`ERR`	`restore::ERR: decompress failed, some error, aborting`	Decompressing the files under `/home/cudb` failed.
`ERR`	`restore::ERR: mysql tables backup failed, aborting`	An attempt to make a restore of the MySQL tables failed.
`ERR`	`delete::ERR: A backup name must be given, aborting`	The file name backup parameter is wrong or not specified.
`ERR`	`IMM file not found <imm_config> . <vars>`	The file `.CudbOamModel_Instances_config_imm.xml` is not in the required path: `"/cluster/home/coremw_appdata/incoming/cudb-install-temp"`.
`ERR`	`IMM file not found <imm_config_path_file> . <vars>`	The file `.CudbOamModel_Instances_config_imm.xml` is not in the required path: `"/cluster/home/coremw_appdata/incoming/cudb-install-temp"`.
`ERR`	`CudbLocalNode::hwType not found in <imm_instances_file> . <vars>`	There is no instance of `hwType` in the file `CudbOamModel_Instances_config_imm.xml`.
`ERR`	`Unsupported HW type: "'+vars.hw_type+'".', vars <hw_type> . <vars>`	The HwType set in the file `CudbOamModel_Instances_config_imm.xml` is not supported or it is wrong.
`ERR`	`CMSMHT SDP handling tools RPM file not found in repository . <vars>`	CSMHT rpm not found in the repository _
`ERR`	`Unable to uncompress CSMHT SDP handling tools RPM file . <vars>`	An error occurred trying to uncompress the CSMHT rpm _
`ERR`	`Configuration file not found`	`cluster.conf` file not found in `/cluster/etc`/.
`ERR`	`Unable to uncompress CSMHT SDP handling tools RPM file . <vars>`	An error occurred trying to uncompress the CSMHT rpm _
`ERR`	`Reference <reference_file> file not found`	`cudbReference` file not found under `"home/coremw_appdata/incoming/cudb-schema-tools-install-temp"`.
`ERR`	`Incorrect number of parameters`	The number of parameters are wrong.
`ERR`	`Package<tmp_path> / <sdp_name> file not found`	`sdp` not found in the directory _
`ERR`	`Not Supported. Install in all nodes necessary`	The installation is not supported for a concrete node. Only Supported for all nodes.
`ERR`	`Unable to retrieve installation order for package <sdp_id>`	An error occurred trying to obtain the `sdp` information of the `sdp`.
`WARNING`	`restore::WARNING: failed to extract cudbReference file from tar`	`cudbReference` file cannot be extracted from the tar file.

2.5.4.5.2 Configuration Management

Configuration Management component uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, only messages with severity info, warning, or emergency are logged.

The messages logged by the Configuration Management subcomponent are summarized in Table 27.

Table 27 Configuration Management Logs
Severity	Message Information	Trigger Event
`EMERG`	`Problem while updating applyConfigStatus in CudbOiWorker sigHandler`	`applyConfig` was running when the active `CudbOiWorker` instance was terminated, and there was an issue while updating `applyConfigStatus`.
`EMERG`	`Problem while updating updateUserInfoStatus in CudbOiWorker sigHandler`	`updateUserInfo` was running when the active `CudbOiWorker` instance was terminated, and there was an issue while updating `updateUserInfoStatus`.
`EMERG`	`CudbAdminOperationApplyConfig::executeAdminOperation - Problem while starting applyConfigThread. Result code was: <result_code>`	There was an issue while trying to start the administrative operation `applyConfig`.
`EMERG`	`CudbAdminOperationApplyConfig::cancelApplyConfig - Failed to update applyConfigStatus with cancellation information.`	There was an issue while trying to update `applyConfigStatus` upon a cancellation request for `applyConfig`.
`EMERG`	`CudbAdminOperationApplyConfig::cancelApplyConfig - There was a problem while canceling thread. pthread_cancel returned: <result_code>`	There was an issue while trying to cancel the thread in which `applyConfig` was executed.
`EMERG`	`CudbAsyncAdminOperationBase::updateStatus - Unable to find id=<id>`	The id does not match any of the existing `CudbAsyncActionProgress` class ids.
`EMERG`	`CUDBAPPLYCONFIG::signalHandler - Failed to update applyConfigStatus.`	Error while updating `applyConfigStatus` values after received signal.
`EMERG`	`CUDBAPPLYCONFIG::Main- Another instance seems to be running.`	Another instance of the process is running.
`EMERG`	`CUDBAPPLYCONFIG::Main- Failed to update applyConfigStatus.`	Error while updating `applyConfigStatus` values when starting the `cudbApplyConfig` command.
`EMERG`	`CUDBAPPLYCONFIG::Main- Failed to update applyConfigStatus after execution. Operation finished with: result = <result_string>`	Error while updating `applyConfigStatus` after execution of the `cudbApplyConfig` command has finished.
`EMERG`	`CUDBAPPLYCONFIG::getInfoFromCudbSystemConfig - Exception while parsing cudbSystem.xml: <error_string>`	Error while parsing `cudbSystem.xml`.
`EMERG`	`ImmConnector::init - Generation of initial config files is not allowed`	Missing licenses for the requested initial configuration.
`EMERG`	`ImmConnector::init - Missing license - Payload Blades/ Virtual Machines license for the number of installed payload blades`	License for a requested number of blades is not loaded at initial configuration.
`EMERG`	`ImmConnector::init - Missing license - <value_pack_name> value pack that contains <feature_name> feature`	Missing license for `<value_pack_name>` value pack at initial configuration.
`EMERG`	`ImmConnector::updateAttr - CommitError: Missing license - Payload Blades/ Virtual Machines license for the number of installed payload blades`	Attempt to configure additional DSU when the license for requested number of blades is not loaded.
`EMERG`	`ImmConnector::updateAttr - CommitError: Missing license - Deployment Flexibility value pack that contains Multiple geographic zones feature` `ImmConnector::updateAttr - CommitError: Missing license - Advanced Network Protection value pack that contains Triple geographic redundancy feature` `ImmConnector::createUpdateCudbLdapView - CommitError: Missing license - Applications Facilitator value pack that contains Ldap views feature`	Configuring feature at runtime without the needed license for it.
`EMERG`	`CudbAdminOperationUpdateUserInfo::executeAdminOperation - Failed to update status for updateUserInfo prior to execution.`	Error while updating `updateUserInfoStatus` when starting administrative operation `updateUserInfo`.
`EMERG`	`CudbAdminOperationApplyConfig::executeAdminOperation - Failed to update status for applyConfig prior to execution.`	Error while updating `applyConfigStatus` when starting administrative operation `applyConfig`.
`EMERG`	`CudbAdminOperationApplyConfig::executeAdminOperation -Precheck for file generation failed. File generation didn't finish successfully. Return console`	There was a problem with configuration file generation after the last commit.
`EMERG`	`CUDBAPPLYCONFIG::Main - Something went wrong with configuration file generation before command was invoked ERROR`	There was a problem with configuration file generation after the last commit, but before command was invoked.
`EMERG`	`CUDBAPPLYCONFIG::Main - applyConfig waited for configuration file generation to finish successfully, but it failed ERROR`	`applyConfig` was started while configuration file generation was running and it waited for the generation to finish successfully, but it failed.
`EMERG`	`CUDBAPPLYCONFIG::Main - cudbSwBackup is running and it is not possible to apply configuration`	`applyConfig` was started while `cudbSwBackup` was running.
`EMERG`	`CUDBAPPLYCONFIG::doProcess - Problem reading information from cudbSystem.xml configuration file. ERROR`	There was a problem while reading `cudbSystem.xml`.
`EMERG`	`CUDBAPPLYCONFIG::doProcess - Problem reading information from cluster.conf configuration file. ERROR`	There was a problem while reading `cluster.conf`.
`EMERG`	`SaImmOiCcbObjectCreateCallback::cudbSwBackup is running!`	A configuration change was attempted while software backup was running.
`EMERG`	`SaImmOiCcbObjectDeleteCallback::cudbSwBackup is running!`	A configuration change was attempted while software backup was running.
`EMERG`	`SaImmOiCcbObjectModifyCallback::cudbSwBackup is running!`	A configuration change was attempted while software backup was running.
`EMERG`	`SaImmOiCcbApplyCallback - Unable to get OiChanges lock`	It was not possible to acquire the `OiChanges` lock after the commit was executed.
`EMERG`	`EsaJobFilesGenerator::getConfigFileNames - PM job directory doesn't exist!`	The directory containing the job files is inaccessible.
`EMERG`	`EsaJobFilesGenerator::generate - failed to access local node object!`	Error occurred when attempting to get configuration data from the data model.
`EMERG`	`EsaJobFilesGenerator::modifyElements - bad XML format <detailed_message>!`	One of the required attributes is missing in a job file.
`EMERG`	`EsaJobFilesGenerator::modifyGranularityPeriod - Failed to parse document!`	Unable to read the requested xml file.
`EMERG`	`EsaJobFilesGenerator::modifyGranularityPeriod - Invalid document format!`	A job file has bad xml format.
`EMERG`	`Process::restartProcess - <process_name> is not up, no more retries.`	After restart, process is not up.
`EMERG`	`Process::restartProcess - <process_name> is not up on <host_ip>, no more retries.`	After restart, process is not up on a specific host.
`EMERG`	`Process::restartProcess - Restarting <process_name> was not successful.`	Restarting process was not successfully executed.
`EMERG`	`Process::restartProcessSingleInstance - <process_name> is not up, no more retries.`	After restarting process running as a single instance, process is not up.
`EMERG`	`Process::restartProcessSingleInstance - <process_name> is not up on <host_ip>, no more retries.`	After restarting process running as a single instance, process is not up on a specific host.
`EMERG`	`Process::restartProcessSingleInstance - Restarting <process_name> was not successful.`	Restarting process running as a single instance was not successfully executed.
`EMERG`	`Process::restartProcessSingleInstance - Starting <process_name> was not successful.`	Starting process running as a single instance was not successfully executed.
`EMERG`	`Process::restartProcessSingleInstance - Stopping <process_name> was not successful.`	Stopping process running as a single instance was not successfully executed.
`ERR`	`Error trying to apply dynamic iptables rules, error: [<error_code>]. Retrying!`	Error executing `iptables` commands on eVIP Front-End Elements.
`ERR`	`Error trying to set iptables startup command, error: [<error_code>]. Retrying!`	Error setting the `iptables` commands to be executed on eVIP Front-End Element startup.
`ERR`	`Error calling evipDtmExecuteIptablesCommand to [<alb_name>] [<ip_address>] eVip traffic address... Cause: [<error_code>]`	Error calling method to execute `iptables` commands on eVIP Front-End Elements.
`ERR`	`Error calling evipDtmSetIptablesStartCommands to [<alb_name>] [<ip_address>] eVip traffic address... Cause: [<error_code>]`	Error calling method to set `iptables` commands that must be executed on eVIP Front-End Element startup.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Incorrect OI changes file. This indicates a major problem generating configuration files that shall be managed by Ericsson support.`	The file with the changes of the CUDB configuration model is incorrect.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Encountered an error reading information from cudb system configuration file.`	Error reading information from the CUDB system configuration file.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Encountered an error reading cluster.conf required values.`	Error reading the `cluster.conf` file.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Encountered an error reading OI changes file.`	Error reading the OI changes file.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Encountered an error handling OI changes file.`	Error handling the OI changes file.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Encountered an error handling SQL configuration file.`	Error handling the SQL configuration file.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Encountered an error handling process configuration file.`	Error handling the process configuration file.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Encountered an error updating SQL schema files.`	Error updating SQL schema files.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Encountered an error updating tablespace monitoring parameters.`	Error updating the BLOB disk space monitoring parameters.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Encountered at least an error executing commands.`	Error executing any command when the changes in the CUDB configuration model are applied.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Encountered at least an error restarting processes`.	Error restarting any process when the changes in the CUDB configuration model are applied.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Encountered an error while executing command [commnad].`	Error executing the indicated command.
`ERR`	`CUDBAPPLYCONFIG::doProcess - Lock file [<lockFileName>] cannot be removed. Delete it manually.`	Error trying to delete the lock file specified.
`ERR`	`CUDBAPPLYCONFIG::<method_name> finished with ERRORS` `CUDBAPPLYCONFIG::<method_name> cannot create fileRreader`	Some of the actions carried out in the method specified to update the system with the changes inserted in the CUDB configuration data model have failed.
`ERR`	`CUDBAPPLYCONFIG::updateRedundancyLevel - <rdnUniqueCudbLdapAccessInstance>_redundancyLevel attribute not found.`	The `rdn` attribute specified is not found.
`ERR`	`CUDBAPPLYCONFIG::updateRedundancyLevel - <rdnUniqueCudbLdapAccessInstance>_redundancyLevel attribute value [<value>] cannot be parsed to an integer.`	The `rdn` attribute value specified cannot be parsed to an integer.
`ERR`	`CUDBAPPLYCONFIG::updateClusterConfNetkworkInfo - Error reading cluster.conf fle. Error: <error_description>`	Error reading the `cluster.conf` file. The error description is shown, too.
`ERR`	`CudbApplyConfig::getChanges - Error reading changes file.`	Error reading the changes file.
`ERR`	`CUDBAPPLYCONFIG::getCommandsToLaunch - Modified parameter [<parameter>] not found in any operation instances table.`	The parameter modified is not found in any operation instances table.
`ERR`	`CUDBAPPLYCONFIG::getProcessesAffected - Object with Oid [<object_id>] not found calculating affected processes.`	The object is not found, calculating the affected processes.
`ERR`	`CUDBAPPLYCONFIG::getProcessesAffected - Rule for affected process [<process>] is missing values to be replaced in commands.`	Rule for the process specified is missing, values have to be replaced in commands.
`ERR`	`CUDBAPPLYCONFIG::lauchSQLQueriesToDs - Connection FAILED, skipping server [<host>:<port>], detailed information: <error_description>.`	Connection to server has failed with the host and port specified. The error description is shown.
`ERR`	`CUDBAPPLYCONFIG::launchSQLQueriesToConnectedHost - SQL statements [<sql_statements>] failed, detailed information: <error_description>.`	SQL statements have failed. The error description is shown.
`ERR`	`CUDBAPPLYCONFIG::launchCommands - System call failed, result code=[<error_code>]`	System call failed. The result code is shown.
`ERR`	`CUDBAPPLYCONFIG::launchCommands - Executed process was not terminated normally, result code=[<error_code>]`	Process was not terminated normally. The result code is shown.
`ERR`	`CUDBAPPLYCONFIG::launchCommands - Command failed, result code=[<error_code>]`	Command failed. The result code is shown.
`ERR`	`CUDBAPPLYCONFIG::insertTextInSqlFile - <type_operation>operation in file [<file_name>] failed. Cause: <error_description>.`	The operation in file failed. The error description is shown.
`ERR`	`CUDBAPPLYCONFIG::insertTextInSqlFile - Open file [<file_name>] for <type_operation> failed. Cause: <error_description>.`	Opening the file has failed. The error description is shown.
`ERR`	`CUDBAPPLYCONFIG::readFiles2Detect Error - unable to open files2Detect` `CUDBAPPLYCONFIG::readFiles2Detect Error - cannot allocate oneEntry`	Error handling `files2Detect`.
`ERR`	`CUDBAPPLYCONFIG::tableSpaceMonitoringChanges Error - Failed to apply Table Space Monitor changes`	Failed to apply BLOB disk space monitoring changes.
`ERR`	`CUDBAPPLYCONFIG::Main -,Finished with error(s): <error_code>.`	Error launching `cudbApplyConfig` process.
`ERR`	`CUDBAPPLYCONFIG::Main - Note that OI changes file was not deleted, remove it manually if necessary`.	The file with the changes to be applied to the CUDB configuration data model has not deleted.
`ERR`	`FileGeneratorsExecutor::populateGeneratorList - Factory does not know config file generator [<file_generator>]` `FileGeneratorsExecutor::earlyChecks - At least a check failed.` `FileGeneratorsExecutor::generateConfigFiles - At least a file generator failed.` `FileGeneratorsExecutor::generateConfigFiles - Failure inttializing information providers.` `FileGeneratorsExecutor::generateConfigFiles - Failure building file generators list.` `FileGeneratorsExecutor::doGeneration - Config file generator [<file_generator>} failed.` `FileGeneratorsExecutor::doGeneration - Exception cause start: ------------------------ <error_message>` `FileGeneratorsExecutor::doGeneration - Exception cause end ---------------------------`	Errors in the generation of the configuration files using the data inserted in the CUDB configuration data model.
`ERR`	`OiChanges::commitMark - Can't write file <path_file>` `OiChanges::commitMark - Error at writing <mark> to file <path_file>` `OiChanges::commit Error in stored process` `OiChanges::moveFileToBackup - Failed to make a backup copy of OI changes file [<path_file>] to [<path_backup_file>]` `OiChanges::checkMarks Error: two not expected consecutive <mark> in file <file_name>` `OiChanges::checkMarks Error: unmatched matched <begin_mark>/<end_mark> in file <file_name>` `OiChanges::readFile Can't read file <file_name>`	Errors handling the file with the data changed in the CUDB configuration data model.
`ERR`	`Process::execCommand - System call failed, result code=[<result_code>]` `Process::execCommand - Command failed, result code=[<result_code>]` `Process::execCommand - Executed process was not terminated normally, result code=[<result_code>]`	Error executing some processes to update the system with the changes inserted in the CUDB configuration data model.
`ERR`	`ImmConnector::ImmConnector - CRITICAL Error: Could not read or parse cudbOI.xml file. Cause: <error_code>` `ImmConnector::ImmConnector - CRITICAL Error: Could not initialize database accessor`	Errors in the initialization of process to handle the CUDB configuration data model.
`ERR`	`Error: something went wrong validating files generation` `Error: something went wrong generating config files`	Error in the validation or creation of the configuration files.
`ERR`	`Error: changes log file wasn't correctly written`	There was an error updating the log file needed for the application of committed configuration changes.
`ERR`	`CudbPlGroupOI::checkThreshold - memoryWarningThreshold must be lower than memoryFullThreshold`	Failure when trying to set illegal threshold values.
`ERR`	`Error: new value for numAssignedNodes is out of range for [hwType]`	Failure when trying to set `numAssignedNodes` value which is out of range for that `hwType`.
`ERR`	`ImmConnector::createUpdateCudbLdapView - CommitError: The attribute ldapViewName cannot be empty.` `ImmConnector::createUpdateCudbLdapView - CommitError: The attribute ldapViewName cannot be shared among multiple CudbLdapView objects.` `ImmConnector::createUpdateCudbLdapView - CommitError: The attribute ldapViewName cannot be shared among multiple CudbLdapView objects. Non-applied CudbLdapView object with same ldapViewName attribute already exists on the node.` `ImmConnector:: createUpdateCudbExternalAuthServer - CommitError: cudbExternalAuthServerId must be set to 1` `Ds Group Id must be greater than 0 or less than 256` `ImmConnector:: createUpdateCudbProvisioningGateway - CommitError: ProvisioningGatewayConfigId must be set to 1` `ImmConnector:: createUpdateCudbLogCertificates - CommitError: cudbLogCertificatesId must be set to 1` `ImmConnector:: createUpdateCudbExternalLogServer - CommitError: cudbExternalLogServerId must be set to 1` `ImmConnector:: createUpdateCudbLdapAccess - CommitError: ldap Access Id must be set to 1` `ImmConnector:: createUpdateCudbRemoteNode - CommitError: remote Node Id must be greater than 0 and less than 256` `ImmConnector:: createUpdateCudbRemoteNode - CommitError: two CUDB nodes can not have the same Id` `ImmConnector:: createUpdateCudbRemotePl - CommitError: remote Pl Id must be set to 1` `ImmConnector:: createUpdateCudbLocalDs - CommitError: local Ds Id must be greater than 0` `ImmConnector:: createUpdateCudbLocalDs - CommitError: local Ds Id should be one more than the last one` `ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbExternalLogMgmt` `ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbExternalAuthMgmt` `ImmConnector::updateAttr - CommitError: invalid value of attribute pgNodeIpAddresses in object CudbProvisioningGatewayConfig` `ImmConnector::updateAttr - CommitError: invalid value of attribute reallocationBlockSize in class CudbSystem` `ImmConnector::updateAttr - CommitError: trafficVIP, oamVIP, or cudbVIP attributes are equal between Local and Remote Node` `ImmConnector::updateAttr - CommitError: The attribute cudbUserPassword can not be an empty string. In class cudbLdapUser` `• ImmConnector::updateAttr - CommitError: Invalid value of attribute cudbUserGroup in class CudbLdapUser` `• ImmConnector::updateAttr - CommitError: two Application Services can not have the same name` `ImmConnector::updateAttr - CommitError: trafficVIP, oamVIP, or cudbVIP attributes does not have ip address format` `ImmConnector::updateAttr - CommitError: The attribute overloadRejectionWeight must be set to one of the predefined values. In class cudbLdapUser` `ImmConnector::updateAttr - CommitError: The attribute readModeInDS must be set to one of the predefined values 'MA', 'MP', 'LP' cudbLdapUser` `ImmConnector::updateAttr - CommitError: The attribute readModeInPL must be set to one of the predefined values 'MA', 'MP', 'LP' cudbLdapUser` `ImmConnector::updateAttr - CommitError: another LocalDs from the same LocalNode has the same group` `ImmConnector::updateAttr - CommitError: two Ds nodes from the same group can not have the same priority` `ImmConnector::updateAttr - CommitError: two local Ds nodes can not have the same dsGroupId` `ImmConnector::updateAttr - CommitError: another RemoteDs from the same RemoteNode has the same group` `ImmConnector::updateAttr - CommitError: the same dsGroup can not be present in two different zones` `ImmConnector::updateAttr - CommitError: incorrect format of attribute smpMgrAddr in object CudbLocalNode` `ImmConnector::updateAttr - CommitError: ip address format -> <ipaddress\|hostname>[:<port>]` `ImmConnector::updateAttr - CommitError: each masterReplicationChannel1Port/masterReplicationChannel2Port/accessPort for a CudbPlGroup/CudbDsGroup must be unique` `ImmConnector::updateAttr - CommitError: there is no instance of CudbDsGroup object for that group` `ImmConnector::updateAttr - CommitError: two remote Ds nodes from the same remoteNode can not have the same dsGroupId` `ImmConnector::checkCudbLdapView - Cannot assign LDAP View to provisioning user!` `ImmConnector::checkCudbLdapView - Cannot assign LDAP View to re-provisioning user!` `ImmConnector::checkCudbLdapView - The attribute cudbLdapViewId cannot be assigned to non-exisiting view in cudbLdapUser` `ImmConnector::updateAttr - CommitError: this default zone is not defined in any CudbLocalNode or CudbRemoteNode` `ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbProvisioningGatewayConfig` `ImmConnector::updateAttr - CommitError: there is no instance of CudbDsGroup object for that group` `ImmConnector::updateAttr - CommitError: two Pl nodes can not have the same priority`	Attempt to set up invalid attribute value.
`ERR`	`ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbExternalLogMgmt` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbExternalLogServer` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object cudbAutomaticMasterChange` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbPlGroup` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbDsGroup` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbLdapUser` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbRemoteNode` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbRemoteDs` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbLocalDs` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbAppService` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbLogCertificates` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbExternalAuthServer` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbLdapAccess` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbLocalNode` `ImmConnector::updateAttr - CommitError: attributes cannot be modified for object cudbSystem`	Attempt to modify restricted attributes.
`ERR`	`ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbExternalAuthMgmt`	Attempt to enable `CudbExternalAuthMgmt` without server parameters set on `CudbExternalAuthServer`.
`ERR`	`ImmConnector::updateAttr - CommitError: attributes do not have the correct format for object CudbLocalPl`	Attempt to set up invalid value of the `numAssignedNodes` attribute.
`ERR`	`ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbLogCertificates` `ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbSystemSecurity` `ImmConnector::updateAttr - CommitError: some Application Service files do not exist`	Attempt to set up value for attribute without the corresponding file stored.
`ERR`	`ImmConnector::updateAttr - CommitError: invalid value of attribute ldapAttrIndexes in object cudbLdapAccess`	Attempt to delete existing LDAP indexes, or set up a new index that does not exist as column in the corresponding PLDB table or any user database table.
`ERR`	`ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbExternalAuthServer`	Attempt to set up invalid value of the attribute `primaryServer` or `secondaryServer` for `CudbExternalAuthServer`.
`ERR`	`ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbExternalLogServer`	Attempt to set up value of the attribute `externalLogServerIp` without strict IP address syntax.
`ERR`	`ImmConnector:: createUpdateCudbLocalDs - CommitError: local Ds Id value not supported`	Attempt to set up value for `cudbLocalDsId` that is out of range for the configured HW.
`ERR`	`Number of requested blades = <number_of_blades>, number of granted blades = <blade_capacity>. Requested blade capacity was not granted.`	Attempt to configure additional DSU when the license for the requested number of blades is not loaded.
`ERR`	`Number of requested blades = <number_of_blades>, number of granted blades = <blade_capacity>. Requested blade capacity was not granted. License manager is in autonomous operational mode, if Payload Blades/ Virtual Machines license has been updated while in autonomous mode it couldn't be read.` `Payload Blades/ Virtual Machines license could not be retrieved. License manager is in autonomous operational mode, if new licenses have been installed they couldn't be read`	Attempt to configure additional DSU while in Autonomous Mode.
`ERR`	`Payload Blades/ Virtual Machines license could not be retrieved.`	An error occurred while requesting payload blades/VMs license from License Manager.
`ERR`	`Payload Blades/ Virtual Machines license is rejected for the following operational mode: <operational_mode>`	License Manager is in the operational state that enforces rejection of request for payload blades/VMs license.
`ERR`	`Payload Blades/ Virtual Machines license is rejected for the license type: <license_type>`	License Manager has license type that enforces rejection of request for payload blades/VMs license.
`ERR`	`License Manager handler couldn't be initialized, can't check licenses`	An error occurred while initializing License Manager handler.
`ERR`	`Feature license with name <feature_name> could not be retrieved. License manager is in autonomous operational mode, if new licenses have been installed they couldn't be read`	Attempt to feature license from License Manager while License Manager is in Autonomous Mode.
`ERR`	`Feature license with name <feature_name> could not be retrieved.`	An error occurred while requesting feature license.
`ERR`	`Feature license with name <feature_name> is rejected for the following operational mode: <operational_mode>`	License Manager is in the operational state that enforces rejection of request for feature license.
`ERR`	`Requested feature license doesn't exist`	Attempt to request non-existing feature license from License Manager.
`ERR`	`Feature license with name <feature_name> is rejected for the license type: <license_type>`	License Manager has license type that enforces rejection of request for feature license.
`ERR`	`Wrong version of LM! Can't use License Manager API!`	Wrong version of License Manager detected.
`ERR`	`Application ID is invalid. Problem with License Manager.`	Attempt to use API version not supported by License Manager.
`ERR`	`Registration to License Manager failed!`	An error occurred trying to register License Manager.
`ERR`	`Could not find any license type!`	None of the allowed license types are loaded.
`ERR`	`CudbLdapCertificatesOI::checkAttrFormat tlsCertificateFile can't be set as an empty string.` `CudbLdapCertificatesOI::checkAttrFormat tlsCertificateKeyFile can't be set as an empty string.` `CudbSoapCertificatesOI::checkAttrFormat tlsCertificateFile can't be set as an empty string.` `CudbSoapCertificatesOI::checkAttrFormat tlsCertificateKeyFile can't be set as an empty string.` `CudbSystemSecurityOi::checkAttrFormat tlsCaCertificatesFile can't be set as an empty string.`	Error trying to set a certificate-related attribute to an empty string value after it was already assigned.
`ERR`	`Failed to raise an alarm. Diagnostic info: <diagnostic_information>` `Failed to clear an alarm. Diagnostic info: <diagnostic_information>`	Diagnostic info: `<diagnostic_information>`.
`ERR`	`init - saImmOmInitialize returned error`	Failed to initialize IMM Object Manager.
`ERR`	`init - saImmOmAccessorInitialize returned error`	Failed to initialize IMM Object Manager Accessor.
`ERR`	`readFingerprint - error fingerprint could not be read or is empty!`	Fingerprint cannot be read or it is empty.
`ERR`	`input file not found <file>`	`updateUserInfo` could not find input file.
`ERR`	`Could not retrieve users and groups from database (ldapsearch command failed)<ldap_search_result>`	`ldapsearch` error during `updateUserInfo`.
`ERR`	`Failed. Unexpected error with immcfg -d <CudbLdapUser or CudbLdapUserGroup>`	Error during `updateUserInfo` while trying to delete LDAP users and/or groups from configuration model.
`ERR`	`Failed. Unexpected error with immcfg -f CreateLdapUsers.xml`	Error during `updateUserInfo` while trying to add LDAP users and/or groups to configuration model.
`ERR`	`User information updated, but there was a problem with config file generation in node <node_id>`	Error occurred during the commit phase after deleting or inserting `CudbLdapUser` or `CudbLdapUserGroup`.
`WARNING`	`CudbExternalLogMgmtOI::checkAttrMods - Attempt to enable CudbExternalLogMgmt without server parameters set on CudbExternalLogServer`	Attempt to enable `CudbExternalLogMgmt` without server parameters set.
`WARNING`	`CudbExternalAuthMgmtOI::checkAttrMods - Attempt to enable CudbExternalAuthMgmt without server parameters set on CudbExternalAuthServer`	Attempt to enable `CudbExternalAuthMgmt` without server parameters set.
`WARNING`	`FileGenerator::backupConfigFile Failed to make a backup copy of the config file [<path_config_file_name>] to [<path_backup_file_name>]. Cause: <error_code>`	Failed trying to make a backup of the configuration file.
`WARNING`	`init - saImmOmInitialize returned SA_AIS_ERR_TRY_AGAIN`	Temporary problem initializing an IMM Object Manager.
`WARNING`	`init - saImmOmAccessorInitialize returned SA_AIS_ERR_TRY_AGAIN`	Temporary problem initializing an IMM Object Manager Accessor.
`WARNING`	`readFingerprint - saImmOmAccessorGet_2 returned SA_AIS_ERR_TRY_AGAIN`	Temporary problem reading fingerprint.
`WARNING`	`CUDBAPPLYCONFIG::doProcess - OI changes file not found, if there were no changes to the IMM model this is OK.`	The OI changes file is not found. This is correct when nothing changed in the CUDB configuration data model.
`WARNING`	`CUDBAPPLYCONFIG::getCommandsToLaunch - Commands to run rule for [<final_className_ParamName>] is missing values to be replaced [<partil_results>].`	Warning that a command to launch is missing.
`WARNING`	`CudbAsyncActionProgressOI::updateStructAttributes - <attribute_name> doesn't match to any of CudbAsyncActionProgress attributes`	An attribute was provided for update which does not match any known attribute in `CudbAsyncActionProgress` structure.

2.5.4.5.3 Fault Management

Fault management component uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, only messages with severity info, warning, or emergency are logged.

The messages logged by the Fault management subcomponent are summarized in Table 28.

Table 28 Fault Management Logs
Severity	Message Information	Trigger Event
`CRIT`	`No such file <configuration_system_file>`	The CUDB system configuration file does not exist in the path specified.
`CRIT`	`No such file <configuration_database_file>`	The database configuration file does not exist in the path specified.
`ERR`	`It is impossible to establish connection with PL mysql server <host_ip> <port>`	Due to a wrong IP, port, or failure in the cluster, the process cannot connect to the database in the PL.
`ERR`	`some mandatory argument has been missed ...`	The parameter was wrong or it was not specified.
`ERR`	`Fault in mysql query`	Fault in the SQL sentence when trying to get the memory use from the system monitor tables.
`ERR`	`Invalid DG group called by ESA: <counter>`	The wanted DS does not exist.
`WARNING`	`It has been impossible to get the counter value`	The `snmpget` command did not get values for the OID related to the counter.

2.5.4.5.4 Performance Management

Performance management component uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages and /var/log/PL_2_*/messages.
By default, only messages with severity info, warning, or error are logged.

The messages logged by the performance management subcomponent are summarized in Table 29 and Table 30.

Table 29 Performance Management Logs located on controller blades or VMs
Severity	Message Information	Trigger Event
`WARNING`	`ESA PM agent did not start successfully in <ipEsaAddress>`	The ESA PM Agent could not be reloaded successfully.
`WARNING`	`Format of current and previous NDB data for <blade> does not match.`	The NDB data retrieved from a blade has a faulty structure.
`WARNING`	`Data retrieval from <blade> timed out. Taking data from the last interval as current.`	Retrieving data from a blade took longer than expected. Data from the last interval will be used as current interval data.
`WARNING`	`There was a problem parsing data for <blade>. Invalid data:<data>`	Data retrieved from a blade has a different format than the one expected by the process.
`ERR`	`Error cleaning up child processes. Exiting.`	An error occurred when the process tried to clean up its child processes.
`ERR`	`Unable to open file for writing! <error>`	Opening the output file has failed. The error description is shown.
`ERR`	`There was a problem reading data retrieved from <blade>. Error:<error>`	An error occurred when trying to read data retrieved from a certain blade. Error description is shown.
`ERR`	`An error occurred retrieving data from <blade> Error:<error>`	An error occurred when trying to obtain data from a certain blade. Error description is shown.
`ERR`	`DSG <dsg_id> not found in <config_file>`	The DSG with the supplied id was not found in CUDB system configuration file.
`ERR`	`Error occurred while executing <imm_command>`	Error occurred when trying to obtain information from the configuration model.

Table 30 Performance Management Logs located on payload blades or VMs
Severity	Message Information	Trigger Event
`ERR`	`Unable to determine pid of <process_name>. Pidof return code: <return_code>`	An error occurred while attempting to determine the PID of the process.
`ERR`	`Unable to determine bitmask for process with pid <process_pid>. Error: <return_code>`	An error occurred while attempting to determine the bitmask of the process based on its PID.
`ERR`	`Unable to read <config_ini> file.`	Failed to read the database cluster configuration file.
`ERR`	`Unable to read <system_config> file.`	Failed to read the CUDB system configuration file.
`ERR`	`Error occurred while retrieving process information <error>`	An error occurred while trying to gather uptime information from a process. The error description is shown.
`ERR`	`Error occurred while retrieving parent process information <error>`	An error occurred while trying to gather parent information from a process. The error description is shown.
`ERR`	`CudbLocalNode not found in <system_config>`	An error occurred while attempting to read the `CudbLocalNode` attribute from the system configuration file.

2.5.4.5.5 Security

Security component uses syslog as logging system with the following properties:

Stored in /var/log/<hostname>/messages.
By default, only messages with severity warning and emergency are logged.

Note:: This section describes the operational logs for security components. For information about logs describing important security related events that occur in the node, see Section 2.6.

The messages logged by the security subcomponent are summarized in Table 31.

Table 31 Security Logs
Severity	Message Information	Trigger Event
`ERR`	`Could not do persistent mode without all parameter.`	`cudbRemoteTrust` parameter `--persistent (-p)` was used without parameter `--all (-a)`.
`ERR`	`Incorrect number of parameters.`	`cudbRemoteTrust` was invoked with incorrect number of parameters. No more than 2 parameters should be used.
`ERR`	`Only one instance of the script can be running on the node! Check /cluster/home/cudb/cudbRemoteTrust.lock if you think you shouldn't be seeing this message.`	Another instance of `cudbRemoteTrust` was already executing on the node. Check both controllers for running `cudbRemoteTrust` processes and, if there aren't any, delete lockfile `/cluster/home/cudb/cudbRemoteTrust.lock`.
`ERR`	`Cannot parse cluster config file.`	`cudbRemoteTrust` was unable to parse cluster configuration file `/cluster/etc/cluster.conf`.
`ERR`	`Cannot parse system config file.`	`cudbRemoteTrust` was unable to parse system configuration file `/cluster/home/cudb/common/config/cudbSystem.xml`.
`ERR`	`Unknown parameter.`	The specified parameter is not supported by `cudbRemoteTrust`.
`ERR`	`Kill signal caught. Exiting.`	Kill signal was received by `cudbRemoteTrust`. Exiting.
`ERR`	`Cannot read files /home/cudb/common/config/cudbLocalNodeId.txt and /home/cudb/common/config/cudbLocalNodeId.txt.bak.`	`cudbRemoteTrust` was unable to read files containing `cudbLocalNodeId`.
`ERR`	`Error at loading new SSHD config on blade <blade>`	`cudbRemoteTrust` could not load new SSHD configuration on the blade.
`ERR`	`Error at restarting SSH daemon on blade <blade>`	There was a problem when `cudbRemoteTrust` restarted SSH daemon on the blade.
`WARNING`	`Cannot copy known_hosts to blade <blade>.`	There was a problem with `cudbRemoteTrust` when copying `known_hosts` file to a certain blade.
`WARNING`	`Cannot copy id_dsa.pub to blade <blade>.`	There was a problem with `cudbRemoteTrust` when copying `id_dsa.pub` file to a certain blade.
`WARNING`	`Some trust file failed, not persistent trust file.`	There was a problem with `cudbRemoteTrust` when copying one of the required trust files. Persistence mode will not be applied.
`WARNING`	`Couldn't get SITE-VIP information from CUDB Node <node>`	`cudbRemoteTrust` could not get SITE_VIP information from the node.
`WARNING`	`Invalid SITE-VIP address detected at CUDB Node <node>: <IP address>`	`cudbRemoteTrust` detected an IP address with invalid format.

2.5.4.6 Platform

As mentioned in Section 2.2, platform events are logged in certain files. Some of the logs, those considered of relevance, are also copied to the Security file (see Section 2.6 for further details).

2.5.4.6.1 LogHandler

The LogHandler component uses syslog as logging system with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, only messages with severity info or error are logged.

The relevant messages logged by the LogHandler component are summarized in Table 32.

Table 32 LogHandler Logs
Severity	Message Information	Trigger Event
`ERR`	`Logrotate with archive rules returned '<rotate_err_code>'. Verify that the logs are being properly rotated.`	There is some problem trying to rotate with archive rules.
`ERR`	`Logrotate with live rules returned '<rotate_err_code>'. Verify that the logs are being properly rotated.`	There is some problem trying to rotate with live rules.
`ERR`	`Startup check file is invalid (unknown action '<action>'). Exiting.`	The action specified is wrong or not supported.

2.5.4.6.2 SmartHWFaultReaction

Logs related to HW state checking are logged with the following parameters:

Stored in /var/log/<hostname>/messages.
By default, only messages with severity info, warning, or error are logged.

The messages logged by the SmartHWFaultReaction component are summarized in Table 33.

Table 33 SmartHWFaultReaction Logs
Severity	Message Information	Trigger Event
`ERR`	`Failed to read necessary parameters.`	Collecting the necessary system parameters failed so the process cannot run.
`ERR`	`Error reading config file < configuration file >.`	Error reading configuration file .
`ERR`	`Invalid configuration: < error message >.`	Configuration data change has been detected, but the data was invalid.
`ERR`	`Touch command failed.`	File System probe detected File System error.
`ERR`	`Halting blade.`	Halting the blade where hardware fault was detected.
`ERR`	`Failed to halt blade.`	Halting commands failed, faulty blade was not powered off successfully.
`WARNING`	`Test limit exceeded, there are still threads in D state for excessive time. CHECK_INTERVAL: < CHECK_INTERVAL > s, D_state timeout: < D_state timeout > s.`	Monitored process has been detected as being stuck in D state more than the configured timeout.
`WARNING`	`Setting up default values.`	Default parameters are applied due to invalid configuration data.
`WARNING`	`Touch command did not complete until timeout.`	File System probe timeout has been exceeded due to missing response.

2.5.4.7 Storage

This section describes the subcomponent of the storage component.

2.5.4.7.1 Engine

The storage engine uses syslog as logging system, with the following properties:

Stored in /var/log/SC_2_*/messages.
By default, only messages with severity info, warning, or emergency are logged.

The messages logged by the storage engine subcomponent are summarized in Table 34.

Table 34 Storage Engine Logs
Severity	Message Information	Trigger Event
`ERR`	`Incorrect number of parameters`	The process got a wrong number of parameters.
`ERR`	`Unknown parameter`	There was a flag or parameter command unknown.
`ERR`	`Unable to increase the object class space in cluster`	Increasing the disk space assigned to one object class failed.
`ERR`	`Unable to obtain data from cluster : <store_id> <cluster_type>`	The process did not have access to the cluster through one of the two IPs to obtain the object classes that have table space in the cluster.

2.5.4.8 3PP

This section describes the subcomponent of the 3PP component.

2.5.4.8.1 MySQL

MySQL processes use syslog as logging system with the following properties:

By default, the mysqld messages are stored in /var/log/PL_2_*/messages_mysqld.
/var/log/PL_2_*/messages_mysqld log files are allowed to grow up to 512 KB and are rotated three times before being removed.

MySQL Cluster Carrier Grade Edition (CGE) has its own log files, these logs are allocated in each ndb component and in the Controller Nodes where the mysql mgmt agent is running. Following are the directories used by mysql to store information:

/local/cudb/mysql/mgmt
/local/cudb/mysql/mysqldAccess/
/local/cudb/mysql/mysqldMaster
/local/cudb/mysql/mysqldSlave/
/local/cudb/mysql/ndbd

For more details related to MySQL log messages, refer to MySQL Server LogsMySQL Server Logs, Reference [11].

2.5.4.8.2 BC Server

BC Server uses log4j as logging system with the following properties:

The processes are logging on the controller blades. By default, the messages are stored in the following locations:
- /var/log/<BC server host>/bc_server.log
- File bc_server.err is stored locally and contains logs of the failing process on the blade.
  /var/log/bc_server.err

2.6 Security Log Events

The security events are logged with the following properties:

Stored in /var/log/sec_events/security_events.log.
By default, only messages with severity info, warning, error, or critical are logged.
Only users with system administrator rights have access to the security events log file.

The messages logged in the security log file are summarized in Table 35 (refer to CUDB Security and Privacy Management, Reference [3] for further information on security events).

The logs from this table are written either by the Security component or by any other component or subcomponent. When written by another component, the event is logged also in the file where the component usually logs other events.

Table 35 Security Related Logs
Severity	Message Information	Trigger Event	Component / Subcomponent
`ERR`	`TLS Certificate NOT modified for SOAP`	SOAP certificate change was not applied because Notifications restart failed.	Notifications / SOAP
`ERR`	`TLS Certificate NOT modified for CA`	Certification Authority certificate change was not applied because Notifications restart failed	Platform
`WARNING`	`Failed login for <user> from <source_IP>`	User tries to log in with an incorrect password.	Platform
`WARNING`	`Failed login. Account locked for <user>`	User has exceeded the maximum number of login attempts.	Platform
`WARNING`	`Invalid user <user> from <source_ip>`	User tries to log in with an incorrect user name.	Platform
`WARNING`	`Failed LDAP Bind Password <source_ip> <bind_dn>`	An LDAP bind has failed because of a wrong user or password.	DataAccess / LDAP

3 Sending Logs to a Remote Server

The Centralized Security Event Logging function allows a CUDB node to send log information to an external logging server. The log information sent with the function are the following:

Log events related to the authentication and management of LDE users and LDE user groups on that CUDB node.
The log of commands executed in the CUDB CLI on that node.

The required information regarding the external logging server must be introduced in the CUDB node using the CUDB Configuration Data Model. Refer to the procedure described in the Configuration Modification Procedure section of CUDB Node Configuration Data Model Description, Reference [4] for creating an instance of the CudbExternalLogServer class. Then, configure Centralized Security Event Logging, either with or without Transport Layer Security (TLS), by following the Configuring Secure Centralized Security Event Logging section of CUDB Security and Privacy Management, Reference [3]. The mentioned section also contains configuration examples for both cases.

Once configured, enable the function by setting the enabled attribute of the CudbExternalLogMgmt class to true, also as described in CUDB Node Configuration Data Model Description, Reference [4].

Consider the following when configuring the function:

The enabled attribute on CudbExternalLogMgmt cannot be set to true, if the CudbExternalLogServer instance does not exist.
The instance of the CudbExternalLogServer class cannot be deleted until the feature is disabled (that is, the enabled attribute of the CudbExternalLogMgmt class is set to false).
The changes are not applied until the cudbApplyConfig command is executed.

Logs sent to the Remote Server have the following format:

<PRI>TIMESTAMP CUDB_NODE_ID-FROMHOST TAG MSG

Where:

PRI – syslog priority of the received message.
TIMESTAMP – timestamp of the collected logs, according to RFC 3339, Reference [12].
CUDB_NODE_ID – identity of the CUDB node on which logs are generated.
FROMHOST – identity of the host on which logs are generated.
TAG – identity of the process which sends the log.
MSG – content of the message.

Example:

<182>2017-11-07T10:14:07.974797+01:00 CUDB_82-SC_2_1 commandlog: [info] 7557 root@SC_2_1:/root 10.82.225.157: ["cluster reboot -n 1"]

Glossary

For the terms, definitions, acronyms and abbreviations used in this document, refer to CUDB Glossary of Terms and Acronyms, Reference [5].

Reference List

CUDB Documents
[1] CUDB Consistency Check.
[2] CUDB High Availability.
[3] CUDB Security and Privacy Management.
[4] CUDB Node Configuration Data Model Description.
[5] CUDB Glossary of Terms and Acronyms.

Other Ericsson Documents
[6] LDE Management Guide.
[7] ESA Setup and Configuration.

Other Documents and Online References
[8] The Syslog Protocol (RFC 5424) http://www.rfc-editor.org/rfc/rfc5424.txt.
[9] Transmission of Syslog Messages over UDP (RFC 5426) http://www.rfc-editor.org/rfc/rfc5426.txt.
[10] OpenLDAP http://www.openldap.org.
[11] MySQL Server Logs http://dev.mysql.com/doc/refman/5.5/en/server-logs.html.
[12] Date and Time on the Internet: Timestamps. IETF RFC 3339 https://www.ietf.org/rfc/rfc3339.txt.