CUDB Node Logging Events

Contents

1Introduction
1.1Document Purpose and Scope
1.2Revision Information
1.3Typographic Conventions

2
Log Events
2.1Infrastructure Log Events
2.2Middleware and Platform Log Events
2.3eVIP Log Events
2.4Fault and Performance Management (ESA) Log Events
2.5Application Log Events
2.5.1Log Files
2.5.2Log Types
2.5.3Log Format
2.5.4Log Messages
2.5.4.1DataAccess
2.5.4.2DbmsTools
2.5.4.3Monitoring
2.5.4.4Notifications
2.5.4.5OAM
2.5.4.6Platform
2.5.4.7Storage
2.5.4.83PP
2.6Security Log Events

3
Sending Logs to a Remote Server

Glossary

Reference List

1   Introduction

This document provides detailed information about the events that the Ericsson Centralized User Database (CUDB) logs.

1.1   Document Purpose and Scope

The purpose of this document is to describe the specific logging events reported by CUDB.

1.2   Revision Information


Rev. A
Rev. B
Rev. C
Rev. D
Rev. E
Rev. F
Rev. G
Rev H
Rev J

Other than editorial changes, this document has been revised as follows:

1.3   Typographic Conventions

Typographic conventions can be found in the following document:

2   Log Events

This section contains information about how logs are handled by each CUDB component.

2.1   Infrastructure Log Events

Refer to the "BSP Log Management" document in the BSP 8100 CPI for information on the logging system in BSP 8100.

2.2   Middleware and Platform Log Events

Logs from the platforms, such as, Linux Distribution Extensions (LDE) and Core Middleware (CMW) are stored in the following four separate files on System Controller (SC) blades:

For more information about Linux Distribution Extensions (LDE) logs, refer to LDE Management Guide, Reference [6].

CMW logs can be found in /var/log/opensaf.

2.3   eVIP Log Events

eVIP uses syslog as logging system. The eVIP component is configured to log into the file /var/log/messages.

2.4   Fault and Performance Management (ESA) Log Events

Not detailed in this document. Refer to ESA Setup and Configuration, Reference [7] for information related to Ericsson Simple Network Management Protocol (SNMP) Agent (ESA) logs.

2.5   Application Log Events

This section contains detailed information on the application log events.

2.5.1   Log Files

All the application related logging events in CUDB are reported using Linux rsyslog. Refer to The Syslog Protocol (RFC 5424)The Syslog Protocol (RFC 5424), Reference [8] and Transmission of Syslog Messages over UDP (RFC 5426)Transmission of Syslog Messages over UDP (RFC 5426), Reference [9] for further information.

As a general rule, logs are stored in the /var/log/<hostname> directory on the two SC nodes. Exceptions to this location are the Lightweight Directory Access Protocol (LDAP) Front End (FE) logs (see Section 2.5.4.1.1).

Note:  
If the Centralized Security Event Logging function is configured, authentication and console commands logs will also be sent to an external logging server (see Section 3 for further details).

Log files are rotated the indicated times before being removed from /var/log/<hostname>.

Following log files are rotated only once:

Following log file is rotated three times:

Following log files are rotated four times:

Following log file is rotated five times:

Following log files are rotated eight times:

Following log files are rotated nine times:

To examine older log files, check the /local/cudb_logarchive directory on the SCs, where the log files removed from /var/log are stored in compressed format for an extended period of time. For each /var/log/<hostname> directory, there is a corresponding /local/cudb_logarchive/<hostname> directory, where the removed logs are stored. These files are also rotated, in sync with the log rotation of their counterparts in /var/log/<hostname>. The last 100 items are kept from each of the following SC blade logs (from /var/log/SC_2_* to /local/cudb_logarchive/SC_2_*):

The last 100 items are kept from each of the following payload blade logs (from /var/log/PL_2_* to /local/cudb_logarchive/PL_2_*):

2.5.2   Log Types

Each logging event has a severity. For severity definitions, refer to The Syslog Protocol (RFC 5424)The Syslog Protocol (RFC 5424), Reference [8].

Only the following severities (sorted from most to least importance) apply to the logging events described in this document:

Note:  
Only those log events that are specially relevant for the operator are listed in the tables in the following sections.

2.5.3   Log Format

All the application related logging information that CUDB logs using syslog has the following format:

<Date> <Time> <Host_Name> <Process_Name[\[PID\]]> <Component> [Subcomponent]: <Severity> - <Message Information>

The meaning of each field is explained as follows:

<Date>

Date when the log has been reported.

<Time>

Time when the log has been reported.

<Host Name>

Physical server where the component is running.

<Process_Name[\[PID\]]>

Process that logs the message. Optionally followed by the PID of the process in square brackets. The first square bracket pair in the <Process_Name[\[PID\]]> log format indicates optional information and the second square bracket pair escaped by backslash character indicates that, if PID is logged, it must be in square brackets.

<Component>

Component that logs the event. Optionally an internal subcomponent can also be shown. See the component description below this table.

<Severity>

Indication about the importance of the fault reported by the log (see Section 2.5.2).

<Message Information>

Description of the event logged.

The following components and subcomponents are defined:

Example 1 and Example 2 show the typical appearance of a log message.

Example 1   Log Message

Sep 22 14:51:01 PL_2_5 slapd[3340] DataAccess [LDAP]: WARNING - 
Access not granted for required operation

Example 2   Log Message

Jun 18 16:23:13 PL_2_5 cudbReallocate[18372] 
DbmsTools [Reallocation]: 
WARNING - DS replication has not been verified

2.5.4   Log Messages

The messages logged by CUDB are described in the following subsections. Each subsection contains the messages logged by a particular CUDB component.

2.5.4.1   DataAccess

This section describes the subcomponents of the DataAccess component.

2.5.4.1.1   LDAP FE

LDAP FE uses syslog as logging system with the following properties:

Table 1 shows the LDAP FE messages.

Table 1    LDAP FE Messages

Severity

Message Information

Trigger Event

EMERG

cudb_be_load_table_mappings FAILED

Back End (BE) initialization cannot be executed successfully.

EMERG

cudbOver_op_modify ldapmodify called with empty attribute

LDAP modify operation cannot be performed if no attribute is listed.

EMERG

invalid arg number <argument>. One argument is required

Overlays initialization cannot be executed successfully.

EMERG

Cluster at <connectString> not ready within <seconds> seconds

Pool is not able to connect within time specified.

EMERG

cudb_dist_conf::<methods> ADL not initialized

Several methods cannot be executed if ADL is not initialized.

EMERG

Not able to subscribe to Messaging Service

FE is not able to get mastership info from system monitor.

ERR

  • cudbOver_main::op_addPL Master PLDB not reachable while Split Brain and provUser false

  • cudbOver_main::op_modifyPL Master PLDB not reachable while Split Brain and provUser false

  • cudbOver_main::op_addDS Master PLDB not reachable while Split Brain, proxyBehavior is Provisioning and provUser false

  • cudbOver_main::op_modifyDS Master PLDB not reachable while Split Brain, proxyBehavior is Provisioning and provUser false

  • cudbOver_main::op_search Master PLDB not reachable while Split Brain, proxyBehavior provisioning and isProvisioningUser false

Add, modify or search operations are not available in symmetrical split situation, and local PLDB partition is not master.

ERR

  • cudb_branch_cache::createEvent Error in getDictionary

  • cudb_branch_cache::createEvent Error in

BRANCH_CACHE_CHANGES tables cannot be initialized.

ERR

cudb_branch_cache::notifyChanges couldn't notify changes

Changes cannot be executed in BRANCH_CACHE_CHANGES table.

ERR

cudb_pm_db_open: failed to register the database with back-monitor

Database cannot be registered.

ERR

monitoring thread was not able to bind

Monitoring overload query cannot bind.

ERR

Unable to start CUDB overload control thread. Error code:<ERR_CODE>

Overload control thread cannot be started.

ERR

cudbOver_main::op_add attempt while even split and Backbone reliable

Provisioning has been locked due to a potential split-brain situation.

ERR

Entry cannot be built from the object classes info in <tableName> table. Possible data corruption in the entry <dn>

The length of the column is not a multiple of 4, so the data can be considered corrupted.

ERR

Object class not configured in OBJECT_CLASSES table

The length of the column is a multiple of 4, but the object class cannot be found in the cache.

WARNING

Identity key not read

Identity key cannot be read.

WARNING

  • cudb_index::getObjectOperation null transaction critical failure

  • cudb_index::getObjectOperation couldn't obtain an operation for CUDB_DN_INDEX table

  • cudb_index::getObjectOperation couldn't read tuple for CUDB_DN_INDEX table

  • cudb_index::getObjectOperation Error setting DnBaseKey

Different warnings can appear when a transaction is not executed properly.

WARNING

  • cudb_index::getSubtreeAliasesOperation. Invalid scope.

  • cudb_index::getSubtreeAliasesOperation. Error after setting key.

  • cudb_index::getSubtreeAliasesOperation. Error after setting alias scan filter.

Different warnings can appear when the key composition is not according the scope of search operation.

WARNING

  • cudb_index::getIndexValuesInNdbOp. Error reading operational attribute 'modifyTimestamp'.

  • cudb_index::getIndexValuesInNdbOp. Error reading operational attribute 'createTimestamp'.

Error while reading the operational timestamp attributes.

WARNING

  • cudb_de::read_de: error reading operational timestamp attributes 'modifyTimestamp'

  • cudb_de::read_de: error reading operational timestamp attributes 'createTimestamp'

Error while reading the operational timestamp attributes of a Distribution entry.

WARNING

Access not granted for required operation

The access required for the operation is not granted.

WARNING

No space left in DSG <dsg_id>

It is impossible to add an entry because the Data Store (DS) cluster is full.

WARNING

DSG <dsg_id> returned by distribution external library does not exist. Default distribution algorithm will be applied

The DS Unit Group (DSG) returned by the distribution external library does not exist.

WARNING

Zone <zone_id> does not exist

The zone does not exist.

WARNING

No space left in CUDB System

The CUDB system is full.

WARNING

No space left in zone <zone_id>

The zone is full.

WARNING

Client disconnected from Messaging Service

LDAP FE got disconnected from Messaging Service instance and will try to reconnect.

WARNING

Client connection closed to Messaging Service

LDAP FE got disconnected from Messaging Service instance LDAP FE got disconnected from Messaging Service instance.

WARNING

Client reconnected to Messaging Service in <uri>!

Reconnection attempt to Messaging Service cluster was successful.

Table 2 shows LDAP FE error message.

Table 2    LDAP FE Error Message Related to Received LDAP Requests

Severity

Message Information

Trigger Event

WARNING

LDAP on_response: err=<ldap_result_code> text=<diagnostic_message> op=<ldap_operation> dn=<distinguished_name> user=<ldap_user_name>

Error processing a received LDAP request.

Note:  
<ldap_operation> can be BIND, SEARCH, MODIFY, ADD, or DELETE.

<distinguished_name> is the distinguished name related to the failed LDAP request. Depending on when in the processing phase the error happened, the distinguished name in the log entry will be the original distinguished name in the LDAP request or one partially modified by CUDB.


Some additional LDAP FE messages are logged within this component, for further information, refer to OpenLDAPOpenLDAP, Reference [10].

2.5.4.1.2   LDAP FE Monitor

The LDAP FE Monitor component running on SC blades or Virtual Machines (VMs) uses syslog as logging system with the following properties:

The messages logged by the LDAP FE Monitor, when running on SC blades or VMs, are summarized in Table 3.

Table 3    LDAP FE Monitor SC Logs

Severity

Message Information

Trigger Event

EMERG

<prog_name> failed to run as daemon. Error:<error>

LDAP FE Monitor unable to run as daemon.

EMERG

Unable to activate all the evip addresses correctly!

An error has occurred trying to activate all the eVIP addresses.

EMERG

Unable to deactivate all the evip addresses correctly!

An error has occurred trying to deactivate all the eVIP addresses.

ERR

Error trying to activate eVip traffic address with port error:<error>

An error occurred trying to activate the traffic address with the port.

ERR

Error trying to deactivate eVip traffic address with port error:<error>

An error occurred trying to deactivate the traffic address with the port.

ERR

Error calling evipDtmServerActivate with port error: <port><error>

An error has occurred calling evipDtmServerActivate with port.

ERR

Error calling evipDtmServerDeactivate with port error: <port><error>

An error has occurred calling evipDtmServerDeactivate with port.

ERR

  • Failed to connect to IMM: too many failed attempts

  • Failed to connect to IMM: connection rejected

Failed to connect to IMM.

ERR

  • Failed to initialize IMM OM: too many failed attempts

  • Failed to initialize IMM OM: unknown error

Failed to initialize IMM Object Manager.

ERR

  • Failed to finalize IMM OM accessor

  • Failed to finalize IMM OM

Failure at disconnection from IMM.

ERR

  • Failed to read <object> object from IMM: object does not exist

  • Failed to read <object> object from IMM: invalid parameter

  • Failed to read <object> object from IMM: unknown error

  • Unsupported type while reading attribute <attribute> from IMM

The object or the attribute cannot be read from IMM.

ERR

No uuid was read from IMM

The uuid attribute does not exist in IMM.

ERR

Failed to initialize alarms library

An error occurred trying to initialize the alarms library.

WARNING

<thread> LDAP FE at <IP>:<port>is now down

LDAP FE thread is down.

WARNING

Failed to establish SNMP session

Unable to establish SNMP session

WARNING

Could not read LDAP FE's configuration from xml file

An error occurred trying to read the xml configuration file.

WARNING

Unable to get vip handle. retrying in 5 secs ... cause: <error>

An error has occurred trying get vip handle.

WARNING

Unable to load libtdm library, not using this capabilities, error: <error>

An error has occurred trying to load libtdm library.

WARNING

<thread> LDAP FE at <IP>:<port>is experiencing a temporary error

Unable to get the LDAP FE status due to a temporary error.

The LDAP FE Monitor component running on payload blades or VMs uses syslog as logging system with the following properties:

The messages logged by the LDAP FE Monitor, when running on payload blades or VMs, are summarized in Table 4.

Table 4    LDAP FE Monitor Payload Logs

Severity

Message Information

Trigger Event

CRIT

Failed to start LDAP FE process

It was impossible to start the LDAP FE process.

ERR

immfind for LDAP failed - IMM time out or SA_..._ERR error

An error occurred while trying to read configuration data.

ERR

LDAP FE is unreachable

The monitored LDAP FE process cannot be reached.

ERR

Failed to set CPU affinity

An error occurred while trying to set the CPU affinity for the monitored LDAP FE.

ERR

unable to terminate LDAP FE process

An error occurred while trying to gracefully terminate the monitored LDAP FE process.

ERR

unable to kill LDAP FE process

An error occurred while trying to terminate the monitored LDAP FE process.

ERR

inet_ntop: <error>

An error occurred while trying to convert the IP address into text format.

ERR

host <host> is not resolved

IP address resolution for host <host> failed.

WARNING

LDAP FE process is not running

The monitored LDAP FE process is not up.

WARNING

LDAP server responded with an error: <error>

The monitored LDAP FE process is reachable, but the request failed due to <error>.

WARNING

LDAP request to <LDAP FE> failed (<X>/<Y>)

The LDAP request to the monitored LDAP FE process failed <X> out of total <Y> times.

WARNING

LDAP FE did not shut down in time

The monitored LDAP FE process did not shut down in time after the graceful termination.

WARNING

LDAP FE process is still running

The monitored LDAP FE process did not shut down after termination attempt.

WARNING

pre-termination command has failed to finish in <X> milliseconds; sending KILL signal

The command that was supposed to be executed prior to graceful termination of the monitored LDAP FE process failed to execute. Graceful termination will still occur.

WARNING

could not open config file <file>

An error occurred when trying to open the configuration file for reading.

WARNING

failed to resolve <host> <X>/5 times, sleeping <Y> milliseconds before retrying

It was impossible to resolve the IP address for the given <host> on attempt <X> out of total 5. A retry will occur after <Y> milliseconds.

2.5.4.1.3   Schema Management GUI

Schema management Graphic User Interface (GUI) uses log4java as log system with the following properties:

The messages logged by the Schema Management GUI are summarized in Table 5.

Table 5    Schema GUI Logs

Severity

Message Information

Trigger Event

ERR

Invalid schema object selected

Selected an invalid schema object.

ERR

View <id_project_view> not selected

View not selected.

ERR

Error opening attribute editor for <attr_name>

An error has occurred trying to open the attribute editor.

ERR

Error opening objectclass editor for <obj_class>

An error has occurred trying to open the object class editor.

ERR

Error opening schema editor for <schema_name>

An error has occurred trying to open the schema editor.

ERR

Error trying to execute command <command>

An error has occurred executing the command, something wrong happened.

ERR

Error renaming schema file

An error has occurred when a schema file was renamed, probably because the file was removed or modified with external actions.

ERR

The imported file defines attributes or object classes with OID or alias already defined in the project

The selected file cannot be imported since there is at least one entity already defined.

ERR

Error deleting schema file <schema_name>

An error has occurred when a schema file was deleted, probably because the file was removed or modified with external actions.

ERR

The selected file is not a valid Open LDAP schema file

The selected file cannot be imported since it is not well-formed according to the OpenLDAP syntax.

WARNING

Schema <schema_name> is being removed creating error in: <element>

Due to the deletion of a schema file some inconsistencies appear in the specified elements.

WARNING

ObjectClass <obj_class> is being removed creating error in: <element>

Due to the deletion of an object class some inconsistencies appear in the specified elements.

WARNING

Internal error - Unknown object type

Error obtaining the name of the object: unknown object type.

WARNING

AttributeType <attr_type> is being removed creating error in: <element>

Due to the deletion of an attribute type some inconsistencies appear in the specified elements.

2.5.4.1.4   LDAP Counters

LDAP Counters uses log4java as logging system with the following properties:

The relevant messages logged by the LDAP Counters component are summarized in Table 6.

Note:  
The LDAP Counters component uses FATAL severity logging instead of EMERG.

Table 6    LDAP Counters Logs

Severity

Message Information

Trigger Event

FATAL

command must be: ExecuteCounter -s pathCudbSystem.xml -f pathConfigLCA.xml

Error running LDAP Counters with wrong order or incorrect number of parameters.

FATAL

Bad configuration parameter

Error running LDAP Counters with bad parameter.

FATAL

ExecuteCounter:main <message description>

Error executing main LDAP Counters method. See message description.

ERROR

Couldn't update counter value in controller <controller number> because ESA is down

Error connecting to ESA in controller <controller number> due to ESA process is down.

ERROR

Couldn't update counter value in ESA <controller number>. Counter: <counter name>-<counter Id>

Error updating counters in ESA in controller <controller number>

ERROR

Couldn't update <counter type> counters, <message description>


    Counter Type values:

  • OVERALL

  • OVERALL DROP RATIO

  • LOCAL CLUSTER

  • LOCAL CLUSTER DROP RATIO

  • APPGROUPS

Exception raised when updating specific type of counter. See message description.

ERROR

ExecuteCounter:obtainTotalOpCompleted: <message description>

Exception raised when gathering counters into from LDAP FE. See message description.

ERROR

NdbErrorCounterThread:run : <message description>

Error starting the thread that collects the counters. See message description.

ERROR

NdbErrorCounterThread:obtainCounter : <message description>

Error running the thread that collects the counters. See message description.

WARNING

CounterHelper:propertiesCudbSystem <message description>

Error collecting properties from CudbSystem.xml file. See message description.

WARNING

CounterHelper:propertiesLCA <message description>

Error collecting properties from counters configuration file. See message description.

WARNING

XmlHelper:flushNdbCountersToDisk <message description>

Error parsing configuration or transforming NDB counters. See message description.

WARNING

XmlHelper:cleanUpOldFiles <message description>

Error deleting old files in the configured directory. See message description.

WARNING

AppGroupCounters.setCounterValues: <message description>

Error setting the new value of the LDAP Counters. See message description.

WARNING

NdbErrorClassification.setDsgCounterValues: <message description>

Error updating DSG counters read from the LDAP FE. See message description.

WARNING

NdbErrorClassification.setNdbCounterValues: <message description>

Exception raised when updating NDB counters read from the LDAP FE. See message description.

WARNING

AppGroupCounters.setCounterValues: <message description>

Exception raised when updating Applications Group counters. See message description.

WARNING

Overall.setCounterValues: <message description>

Exception raised when updating Overall counters. See message description.

WARNING

ExecuteCounter:checkInformation <message description>

Exception raised when error happens checking the counter information. See message description.

WARNING

ExecuteCounter:publishInESA <message description>

Exception raised when publishing counters into ESA. See message description.

2.5.4.2   DbmsTools

This section describes the subcomponents of the DbmsTools component.

2.5.4.2.1   CountersFw

The CountersFw subcomponent uses syslog as logging system with the following properties:

The messages logged by the CountersFw subcomponent are summarized in Table 7.

Table 7    CountersFw Logs

Severity

Message Information

Trigger Event

ERR

Error establishing PL Connection.

In the case of a remote replica, when the communication between CUDB nodes using SSH cannot be established.

ERR

Error establishing DS Connection.

In the case of a remote replica, when the communication between CUDB nodes using SSH cannot be established.

ERR

  • Error Calling stored procedure for PL in node <node_id>.

  • Error Calling stored procedure for DSG <DSG_id> in node <node_id>.

It is not possible to execute the stored procedure in charge of performing the count.

ERR

  • Couldn't get storage engine information about PLDB in node <node_id>.

  • Couldn't get storage engine information about DSG <DSG_id> in node <node_id>.

  • Couldn't fetch PL counter field.

  • Couldn't fetch DS counter field.

CUDB system configuration file does not have information for the chosen replica.

ERR

Error on changing database to <database_name>

DataBase is changed to Application Counter's database.

ERR

Couldn't update PL counter field.

No counter can be stored in PLDB.

ERR

Error Calling stored procedure for DSG <DSG_id> in node <node_id>.

The called saved procedure doesn't exist in the DSG of the node specified.

ERR

Error Calling stored procedure for PL in node <node_id>.

The called saved procedure doesn't exist in the PL of the node specified.

ERR

cudbApplicationCounter Error reading SDL list from BC Cluster.

Error getting the master list and the status of all the replicas of the CUDB system.

ERR

Error converting masterlist to internal data type.

Internal error when converting masterlist to internal data type.

ERR

Error reading master list from BC Cluster.

Error getting the master list of the CUDB system.

ERR

Error reading DSG Status List.

Error getting information about the status of all the replicas of the CUDB system.

ERR

Error converting DSG Status List to internal type.

Error when converting replica status list to internal data type.

ERR

Error reading Master List.

Error getting information about the status of all the master replicas of the CUDB system.

ERR

Error getting SDL children, getChildren failed: <error>.

Error reading status of all the replicas of the CUDB system.

ERR

Error reading DSG Status List for DSG: <dsgId>, error: <error>.

Error reading status of all the replicas of the CUDB system.

ERR

Error reading master list, getChildrem failed: <error>.

Error reading the master list of the CUDB system.

ERR

Error getting master info for DSG: <dsgId>, Error: <error>.

Error reading the master list of the CUDB system.

ERR

Error reading BC entry size for DSG: <dsg>, error: <error>

Attempt to determine BC entry size failed. See error description.

ERR

Failed to open config file <CONFIG_FILE>.

The mentioned config file is not available.

ERR

Failed to read all configuration variables (got only <NUM> out of <NUM>), missing: <OPTION_XPATH_LIST>.

The configuration file (/home/cudb/common/config/cudbDatabases.xml) does not contain all the parameters needed by CountersFW.

ERR

Unable to obtain connection information of PLDB on node <NODEID>.

Failed to obtain local connection information of the PLDB on the mentioned node from the system configuration file.

ERR

Unable to obtain connection information of DSG <DSGID> on node <NODEID>.

Failed to obtain local connection information of the mentioned DSG on the mentioned node from the system configuration file.

ERR

Couldn't connect to PL tables.

Failed to connect to PLDB.

ERR

PLDB master moved to another node.

PLDB master moved to another node during the operation.

ERR

Can't connect to <HOST>:<PORT> to access database <DBNAME> with user <USERNAME>: <REASON>.

Described in reason, but very likely that no MySQL server runs on the mentioned host listening on the mentioned port.

ERR

OAM Command Failed: Couldn't connect to BC Cluster.

Error, timeout expired while trying to connect to BC Cluster.

ERR

Local node id can't be obtained.

An error occurred while trying to find local node id.

ERR

Master PLDB node id can't be obtained.

An error occurred while trying to find master PLDB node.

WARNING

It has been impossible to start <process> process.

There is a problem trying to start cudbApplicationCounters process.

WARNING

Not all the DSGs have an eligible replica. Application counter data might be stale

Any DSG is masterless or no representative DS replica can be gathered for every DSG, from where the process to update an application counter is launched.

WARNING

Couldn't get slave SDL list. Application counter data might be stale.

The system has geographical redundancy and there is no slave replica.

WARNING

  • exception: <description>

  • runtime error: <description>

  • unknown exception.

Unexpected error.

WARNING

ADG_ERR: Nodes have master after <num_retries> retries in <counter_name>

After some tries the configuration to master nodes has been loaded.

WARNING

ADG_ERR: Couldn't get ADL list for <counter_name>. Checking whether to retry. Retry=<num_retries> MaxRetries=<max_retries>

After some retires the ADL configuration has not been loaded yet and the process will wait for a while and will try again.

WARNING

Error setting PLDB Master List watcher.

Error when creating a subscription to changes on the status of PLDB master.

2.5.4.2.2   DataBackupRestore

The DataBackupRestore subcomponent uses syslog as logging system with the following properties:

The messages logged by the DataBackupRestore subcomponent are summarized in Table 8.

Table 8    DataBackupRestore Logs

Severity

Message Information

Trigger Event

ERR

Option -u|--user-facility must have a value

The process was launched with the -u|--user-facility flag empty, but this flag needs a value.

ERR

Option -T|--Timeout must have a value

The process was launched with the -T|--Timeout flag empty, but this flag needs a value.

ERR

Option -S|--Slack-backup must have a value

The process was launched with the -S|--Slack-backup flag empty, but this flag needs a value.

ERR

Option -R|--Retries-number must have a value

The process was launched with the -R|--Retries-number flag empty, but this flag needs a value.

ERR

Option -t|--time must have a value

The process was launched with the -t|--time flag empty, but this flag needs a value.

ERR

Internal Error: option processing error: <error_description>

There was a flag command unknown.

ERR

No such file <configuration_file>

The configuration file does not exist in the specified path.

ERR

ERR IN PARAMETERS : WE MUST SET THE PARAMETERS FOR PATTERN TO BACKUP

The parameter specified was wrong or it was not specified.

ERR

The node <nodeId> on which restore will be performed has no PLDB configured

The operator tries to restore a PLDB data backup in a node that has no configured PLDB replica.

ERR

Provisioning Gateway has failed when de-blocking on attempt <attempt_id> - <notification error message>.

The communication to the Provisioning Gateway was wrong when de-blocking on all of the attempts.

ERR

Not all the DSGs have an eligible replica from which to take the backup

No representative replica can be gathered for every DSG or PLDB, from where a backup is executed during a system backup procedure.

ERR

No master replica for PLDB or it is in degraded mode.

Master replica for PLDB cannot be detected or it is in degraded mode during a system backup procedure.

ERR

Cannot schedule data backup, error executing [immfind <sentence>]

Failure to schedule data backup due to an error while looking for the active SC blade or VM.

WARNING

Fault in <SQL_operation> mysql query

General fault in an SQL sentence, due to a wrong format sentence, data, or connection.

WARNING

It is impossible to establish connection with PL mysql server

Due to a wrong IP or port, or to a failure in the cluster, the process cannot connect to the database in the Processing Layer (PL).

WARNING

Provisioning Gateway has failed

The communication to the Provisioning Gateway was wrong.

WARNING

Provisioning Gateway has failed when blocking

The communication to the Provisioning Gateway was wrong when blocking.

WARNING

Provisioning Gateway has failed when de-blocking on attempt <attempt_id> - <notification error message>

The communication to the Provisioning Gateway was wrong when de-blocking on one of the attempts. Retrying after 6 seconds.

2.5.4.2.3   Reconciliation

The reconciliation subcomponent uses syslog as logging system with the following properties:

The messages logged by the reconciliation subcomponent are summarized in Table 9.

Table 9    Reconciliation Logs

Severity

Message Information

Trigger Event

WARNING

Deleted Entry due to Reconciliation Process: [<mscId | assocId>]

Error has occurred, entry has been deleted due to the reconciliation process.

WARNING

Failed to add running reconciliation on:<dsg_id>

Error has occurred while attempting to notify the system about reconciliation task.

WARNING

fork():<pid_id>

Fork failed. Creating child process responsible for deferred delete has failed.

WARNING

  • ldap.add():failed

  • v_subscribe():failed

  • ldap.del():failed

  • defered_delete failed

  • ev_wait():failed

  • ev_unsubscribe():failed

Error has occurred during reconciliation process; one of the following steps has failed:


  • Adding the unique key through LDAP

  • Subscribing to DELETE events in the PL

  • Removing the unique key

  • Creating child process responsible for deferred delete

  • Waiting for the DELETE events in the PL

  • Un-subscribing the event handler

WARNING

failed to establish 1. checkpoint

Error has occurred during reconciliation process; one of the steps described above has failed (ldap.add(), ev_subscribe(), ldap.del(), defered_delete, ev_wait() or ev_unsubscribe()).

WARNING

failed to build discrepancy list

Error has occurred during reconciliation process; scanning the PL/DS pair for identification of dangling references and unreferenced data has failed to build discrepancy list.

WARNING

failed to establish 2. Checkpoint

Error has occurred during reconciliation process; Preparation for ‘false’ match recheck has failed.

WARNING

double check failed

Error has occurred during reconciliation process; filtering away false matches has failed.

WARNING

fopen():<undo_file>

Error has occurred during reconciliation process; creating undo files has failed.

WARNING

database cleanup failed

Error has occurred during reconciliation process; cleanup of the DS Unit database under reconciliation against the PLDB database has failed.

WARNING

Considering that the delete event <event ID> for which this drp task is waiting for has been lost . Aborting reconciliation...

In scenarios when the PL master is not located on the same node as the master DS, replication channels are unresponsive during event hooking mechanism and maximum event wait time exceeded, replication is marked as failed.

WARNING

The task <dsg_id> has not been removed from CUDBdcObjet, so we should remove it manually?

Data reconciliation process has already finished for DS but it is not removed from reconciliation task list.

WARNING

Failed getting next task from pending task list.

Error has occurred during reconciliation process; looking for new request.

WARNING

<pl:<dsg_id>>: The task <dsg_id> has not been removed from CUDBdcObjet, so we should remove it manually?

Removing <dsg_id> from CUDBdcObject after PL mastership change has failed.

WARNING

<not pl:<dsg_id>>: The task <dsg_id> has not been removed from CUDBdcObjet, so we should remove it manually?

Removing <dsg_id> from CUDBdcObject has failed; there was no PL mastership change.

WARNING

Subscription to Messaging Service failed <error_id>

Subscription initialization after PTL setup has failed. Exiting process.

WARNING

Shutting down dared...

Data Reconciliation Daemon (DARED) has failed getting next task from pending task list. Unexpectedly shutting down daemon.

WARNING

Reconciliation marked as running for '<dsg_id>' but current '<dsg_id>'

Error has occurred during reconciliation process; running it for different DSG then marked.

WARNING

src/dare-ctx.cpp:672 LDAP Connection failed

LDAP Connection not defined.

WARNING

src/dare-ctx.cpp:676 Failed to inform Reconciliation running on '<dsg_id>'

Adding of entry by its distinguished name over LDAP into reconciliation table CUDBdcObject of cudb_user_data database has failed.

WARNING

src/dare-ctx.cpp:700 Failed to inform Reconciliation finished on '<dsg_id>'

Deleting of entry by its distinguished name over LDAP from reconciliation table CUDBdcObject of cudb_user_data database has failed.

WARNING

Node <nodeid> has no new masters to be added to PTL for reconciliation

After examining the most recently received ADL list, it was determined that the local CUDB node holds no new masters compared to previous ADL list.

WARNING

Not able to begin transaction: INSERT INTO dr_ptl for node <nodeid>

An error occurred while trying to begin the transaction which writes the PTL to dr_ptl table.

WARNING

Not able to commit PTL to dr_ptl table for node <nodeid>

An error occurred while trying to commit the transaction which writes the PTL to dr_ptl table.

WARNING

Some DSGs are not enabled in cudb_system table

Some DS units are missing from cudb_system table.

WARNING

Unexpected error while handling new ADL list

A situation occurred which is not expected by the system.

WARNING

There is no appropriate Node with PLDB in same Site to use

An error occurred while attempting to get an appropriate node with PLDB in the same site of a node without PLDB.

WARNING

  • Failed reading mysql configuration

  • Failed reading LDAP local configuration

  • Failed reading based configuration

An error occurred while attempting to read configuration data from the configuration file

WARNING

  • Master PLDB id not available

  • Master PLDB addr not available

  • Master PLDB port not available

  • Error disabling master PLDB node configuration

  • Error getting configuration of master PLDB node

  • Unable to update SQL configuration

An error occurred while attempting to get or set some configuration data related to the Master PLDB in a node without PLDB.

WARNING

Setup of pending task list failed: <message>

An error occurred while attempting to reconfigure MySQL connections for the pending reconciliation tasks list.

WARNING

Could not initialize the Mysql C API library

An error occurred while initializing the MySQL C API used for managing the reconciliation tasks list.

WARNING

Client disconnected from Messaging Service

Reconciliation process got disconnected from Messaging Service instance and will try to reconnect.

WARNING

Client connection closed to Messaging Service

Reconciliation process got disconnected from Messaging Service instance.

WARNING

Client reconnected to Messaging Service in <uri>!

Reconnection attempt to Messaging Service cluster was successful.

2.5.4.2.4   Replica Consistency Audit

The Consistency Check function is implemented by the Replica Consistency Audit subcomponent, which contains the following set of executables that perform logging:

The CLI and the manager part of the Replica Consistency Audit subcomponent uses syslog as logging system with the following properties:

The messages logged as operational logs are summarized in Table 10.

Table 10    Replica Consistency Audit CLI and Manager Logs

Severity

Message Information

Trigger Event

CRIT

Fatal error happened during DDCI AMF component initialization

Could not initialize AMF client.

CRIT

Waiting for signal failed: <error description>(<error code>)

Timed waiting for a Unix signal failed, see error description.

ERR

BCClient exception while processing <message type>(<message type ID>): <error description>

Communication error with or disconnection from the BC cluster, see error description.

ERR

Connection to ZK cluster is lost, trying to reconnect

Communication error to all servers of the connected BC cluster.

ERR

DSG<ID> on node <node ID> is not alive but <current status>(<current status ID>)

The given DSG's DSU is not alive.

ERR

Exception while listing tasks: <error description>

Couldn't perform full task listing, see error description.

ERR

Failed to complete CLI action: <error description>

Error while performing action specified on command line, see error description.

ERR

Failed to connect to ZK cluster

Permanent communication error to all servers of the connected BC cluster.

ERR

Failed to connect to ZK cluster

Permanent communication error to all servers of the local BC cluster.

ERR

Failed to handle data change of <znode path> (value: <value>, result: <result description>): <error description>

Internal error. (Got an unexpected BC change notification.)

ERR

Failed to set sigmask: <error code>

Couldn't set Unix signal mask.

ERR

<process ID>: process aborted, but couldn't find related task

Internal error. (A task's process was aborted but couldn't find out which task the process belongs to.)

ERR

Request execution failed: <error description>

Failed to complete disk IO operation, see error description.

ERR

<task ID>: failed to start process: <error description>

Task's process couldn,t be started, see error description.

ERR

<task ID>: timeout while waiting for remote connection to site <site ID>

Couldn't connect to BC cluster on remote site in time.

ERR

Unexpected call of AmfCB::changeFinished(<znode path>,...)

Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)

ERR

Unexpected call of AmfCB::getChildrenFinished(<znode path>,...)

Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)

ERR

Unexpected call of AmfCB::getFinished(<znode path>,...)

Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)

ERR

Unexpected call of AmfRemoteCB::changeFinished(<znode path>,...)

Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)

ERR

Unexpected call of AmfRemoteCB::getChildrenFinished(<znode path>,...)

Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)

ERR

Unexpected call of AmfRemoteCB::getFinished(<znode path>,...)

Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)

ERR

Unexpected call of CliCB::changeFinished(<znode path>,...)

Internal error. (Got a notification about the completion of an asynchronous BC operation which was not expected.)

ERR

Unexpected call of CliCB::getChildrenFinished(<znode path>,...)

Internal error. (A task's process was terminated but couldn't find out which task the process belongs to.)

ERR

Unexpected call of CliCB::getFinished(<znode path>,...)

Internal error. (A task's process was terminated but couldn't find out which task the process belongs to.)

ERR

<znode path>: child change for which we were not subscribed

Unexpected znode change notification from BC cluster.

ERR

<znode path>: failed to read BC entry size: <error description>

Attempt to determine BC entry size failed. See error description.

WARNING

<process ID>: couldn't find task for process which just finished with result: <result description> (RC: <exit code>)

Internal error. (A task's process was terminated but couldn't find out which task the process belongs to.)

WARNING

<process ID>: failed to handle its SIGCHLD: <error description>

Could not complete operation assigned to the received Unix SIGCHLD signal, see error description.

WARNING

<process ID>: failed to kill: <error description>

Couldn't kill a task's process, see error description.

WARNING

<process ID>: failed to kill process: <error description>

Couldn't kill the given process, see error description.

WARNING

<process ID>: no action on SIGCHLD, event: <event name>(<event code>)

An unexpected SIGCHILD signal is received and discarded.

WARNING

<task ID>: couldn't find process to terminate

The task's process could not be aborted due to timing or internal error.

WARNING

<task ID>: failed to parse task ID when checking PTL

Task ID is malformatted.

WARNING

<task ID>: failed to remove from PTL: <error description>

Task couldn't be removed from PTL, see error description.

WARNING

<task ID>: failed to remove from RTL: <error description>

Task couldn't be removed from RTL, see error description.

WARNING

<task ID>: failed to remove from RTL: <error description>, try to delete later

Completed task couldn't be removed from RTL, see error description.

WARNING

<task ID>: found in PTL, but deletion failed with: <error description>. Possibly moved to RTL in the meantime

See error description.

WARNING

<task ID>: found in PTL, but deletion failed with: <error description>. Possibly moved to RTL in the meantime, attempting to delete from RTL

See error description.

WARNING

<task ID>: precondition of execution failed: <error description>. Deleting this task from PTL.

Mastership changed since task put into the pending task list or DSG degraded or replica lag higher than max-replica-lag, see error description.

WARNING

<task ID>: process (<process ID>) aborted

The task's process aborted, couldn't complete its operation.

WARNING

<task ID>: RTL entry is invalid, or previously deleted

Internal error. (A task's process was terminated but the task was unknown.)

WARNING

<task ID>: task data (<task parameters>) parsing failed: <error description>

Task parameters are malformed.

WARNING

Unknown and thus unhandled message: <message type>(<message type ID>)

Internal error. (A thread notified another one, but that thread didn't expect that notification message.)

The DSU comparison tool, which is part of the Replica Consistency Audit subcomponent, uses both syslog and additional separate log files for logging. For information on these additional files, see CUDB Consistency Check, Reference [1].

For messages logged using syslog, the following properties apply:

Relevant messages logged into syslog by the DSU comparison tool are summarized in Table 11.

Table 11    DSU Comparison Tool Logs

Severity

Message Information

Trigger Event

ERR

Invalid option list. Diagnostic info:
<internal_diagnostic_info>

Got invalid command line options. The Consistency Check task exits.

ERR

Invalid Node Id. Diagnostic info:
<internal_diagnostic_info>

CUDB Node ID command line option value is invalid. The Consistency Check task exits.

ERR

Invalid Dsg Id. Diagnostic info:
<internal_diagnostic_info>

DSG ID command line option value is invalid. The Consistency Check task exits.

ERR

Invalid max PPL. Diagnostic info:
<internal_diagnostic_info>

Max. PPL command line option value is invalid. The Consistency Check task exits.

ERR

Invalid alarm severity limit. Diagnostic info:
<internal_diagnostic_info>

Got invalid alarm severity limit command line option value. The Consistency Check task exits.

ERR

Incorrect MySQL Access server information. Diagnostic info:
<internal_diagnostic_info>

Got insufficient information to establish the connection to a MySQL Access server (probable causes: some information is missing or invalid in cudbSystem.xml, or invalid command line options are provided). The Consistency Check task exits.

ERR

Unable to establish the connection to MySQL Access server endpoint. Diagnostic info:
<internal_diagnostic_info>

Both MySQL Access servers of one of the specified DS Units are unreachable. The Consistency Check task exits.

ERR

Unrecoverable error. Diagnostic info:
<internal_diagnostic_info>

The Consistency Check task exits due to an error condition which prevents successful task completion. More detailed internal diagnostic information is available in <internal_diagnostic_info>.

ERR

MySQL wrapper error. Diagnostic info:
<internal_diagnostic_info>

The Consistency Check task exits due to a failed database query operation. More detailed internal diagnostic information is available in <internal_diagnostic_info>.

ERR

MySQL error. Diagnostic info:
<internal_diagnostic_info>

The Consistency Check task exits due to a failed database query operation. More detailed internal diagnostic information is available in <internal_diagnostic_info>.

WARNING

Failed to connect to endpoint: <host>:<port> (<num>/<retries>). Diagnostic info:
<internal_diagnostic_info>

Connection failed to MySQL Access server endpoint <host>:<port>. The Consistency Check task will try to reestablish the connection to the same MySQL Access server or to its redundant counterpart for <retries>-<num> additional times.

WARNING

Failed to execute the <name> SqlPrimitive on endpoint: <host>:<port> (<num>/<retries>). Diagnostic info:
<internal_diagnostic_info>

An SQL query execution failed on MySQL Access server endpoint <host>:<port>. The Consistency Check task will disconnect, then tries to reestablish the connection to the same MySQL Access server or to its redundant counterpart for <retries>-<num> additional times, and executes the query again if possible.

2.5.4.2.5   Reallocation

The reallocation subcomponent uses syslog as logging system with the following properties:

The messages logged by the reallocation subcomponent are summarized in Table 12.

Table 12    Reallocation Logs

Severity

Message Information

Trigger Event

EMERG

Not able to subscribe to Messaging Service

The reallocation procedure is not able to get mastership information from the system monitor.

ERR

Reallocation halted due to master change in source DSG <dsg_id>

A master change in the source DSG has been detected during reallocation.

ERR

Reallocation halted due to master change in destination DSG <dsg_id>

A master change in the destination DSG has been detected during reallocation.

ERR

Reallocation halted due to master change in PLDB

A master change in the Processing Layer Database (PLDB) has been detected during reallocation.

WARNING

Failure when reallocating <de_dn> distributed entry due to <problem>

A problem arose while a DE was locked, read, or written, or its reference was changed in the PL.

WARNING

DS replication has not been verified

Verification of a DS block replication has not been completed due to an error or time-out problem.

WARNING

There are no slaves of the current PL

The PL Master is the only replica in the system. No verification is possible.

WARNING

PL replication not completely verified

Verification of references in PL replication not completed due to any error or time-out problems.

WARNING

Reached the end of the table, but less than the requested percentage of DEs was reallocated. <percentage>% of the requested amount was reallocated.

The reallocation procedure reallocated less than the specified amount of DS entries.

WARNING

Client disconnected from Messaging Service

Reallocation procedure got disconnected from Messaging Service instance and will try to reconnect.

WARNING

Client connection closed to Messaging Service

Reallocation procedure got disconnected from Messaging Service instance.

WARNING

Client reconnected to Messaging Service in <uri>!

Reconnection attempt to Messaging Service cluster was successful.

Note:  
<problem> can be locking, reading, writing, or writing in PL problems.

2.5.4.2.6   Replica Repair

The Data Repair function is implemented by the Replica Repair subcomponent, which contains the following executable that performs logging:

The Replica Repair subcomponent uses syslog as logging system with the following properties:

The messages logged as operational logs are summarized in Table 13.

Table 13    Replica Repair Logs

Severity

Message Information

Trigger Event

ERR

Failed to raise alarm. Diagnostic info: <internal diagnostic info>

Data Repair tried to raise an alarm at the end of its execution, but failed.

ERR

LDAP operation error. Diagnostic info: <internal diagnostic info>

Data Repair received an error from an LDAP interface while trying to execute an LDAP operation.

ERR

LDAP connection error. Diagnostic info: <internal diagnostic info>

Data Repair received an error from an LDAP interface while trying to connect to an LDAP server.

2.5.4.2.7   Selective Replica Check

The Selective Replica Check function contains the following executable that performs logging:

The Selective Replica Check subcomponent uses syslog as logging system with the following properties:

The messages logged as operational logs are summarized in Table 14.

Table 14    Selective Replica Check Logs

Severity

Message Information

Trigger Event

ERR

<line where exception was thrown> invalid --dsg value: <dsg number>

Selective Replica Check has received an invalid number for the database cluster group identifier when starting.

ERR

<line where exception was thrown> ionice exited with <error code>

Selective Replica Check failed in setting the ionice priority when starting the execution.

ERR

<line where exception was thrown> Search operation results in <internal diagnostic info>

Selective Replica Check received multiple results for an LDAP search operation.

ERR

<line where exception was thrown> Permanent ldap error.

An error which cannot be handled occurred in Selective Replica Check while trying to execute an LDAP operation.

ERR

<line where exception was thrown> Unrecoverable error in <internal diagnostic info>

An error which cannot be handled occurred in Selective Replica Check while trying to execute an LDAP operation.

ERR

<line where exception was thrown> Number of retries exceeded <internal diagnostic info>

Selective Replica Check exceeded the maximum number of retries for an LDAP search operation.

ERR

<line where exception was thrown> invalid <parameter>

Selective Replica Check has received an invalid parameter when starting.

ERR

<line where exception was thrown> Unrecoverable <error> occurred in LDAP data fetching process.

Selective Replica Check encountered an error while generating the output file containing LDAP entries.

ERR

<line where exception was thrown> Directory creation failed with error: <internal diagnostic info>

Selective Replica Check failed to create a working directory.

ERR

<line where exception was thrown> Directory path exists but it is not a directory.

Selective Replica Check found a file instead of a directory for writing the output.

ERR

<line where exception was thrown> Failed get the size of file <internal diagnostic info>

Selective Replica Check could not retrieve the size of the file.

ERR

<line where exception was thrown> Failed to delete file <internal diagnostic info>

Selective Replica Check could not delete a file.

ERR

<line where exception was thrown> File: <file location> - , status: <internal diagnostic info>

Selective Replica Check encountered an error when handling a binlog file.

ERR

<line where exception was thrown> Failed get the modify time of file <internal diagnostic info>

Selective Replica Check could not retrieve the last modification time of a file.

WARNING

<line where exception was thrown> Following ldap exception occured in thread <internal diagnostic info>

An exception occurred in Selective Replica Check while trying to execute an LDAP operation.

WARNING

<line where exception was thrown> LDAP overload on segment <internal diagnostic info>

Selective Replica Check received LDAP error code 51 or 80 from the local node.

WARNING

<line where exception was thrown> Problem is probably entry specific. <internal diagnostic info>

Selective Replica Check could not retrieve a specific DN.

WARNING

<line where exception was thrown> Retry <attempt number> in thread <internal diagnostic info>

Selective Replica Check is performing a retry on a specific DN.

WARNING

<line where exception was thrown> no binlog files found!

Selective Replica Check did not find any binlog files.

2.5.4.3   Monitoring

This section describes the subcomponents of the monitoring component.

2.5.4.3.1   System Monitor

System Monitor component uses syslog as logging system with the following properties:

The messages logged by System Monitor are summarized in Table 15.

Table 15    System Monitor Logs

Severity

Message Information

Trigger Event

ERR

Error scheduling automatic Service Continuity

Temporary error trying to schedule a task to set serviceContinuity

ERR

Error while setting automatic Service Continuity

Temporary error trying to set serviceContinuity in BCCluster

ERR

Not possible to send <type_message> message <error_message> <exception_trace>

Temporary problems sending status information.

ERR

Instance <systemMonitor _id> can’t publish messages in Messaging Service, reporting node as down

SM is not able to publish messages and gave up on retries.

ERR

SM couldn’t publish the <mesg_subject> message, retrying

Temporary problem while publishing messages.

ERR

Error connecting to Messaging Service <servers_ip>

SM is not able to connect to Messaging Service.

ERR

  • Not possible to handle incident in site <site_id> for D<dsgGroup_id> because is not possible to obtain partition status

  • Error while trying to detect a local incident for D <dsgGroup_id> in node <node_id> in site <site_id> . Incident has not been raised <exception_trace>

  • There is an error when handling a SDL incident in site <site_id> for D<dsgGroup_id> <exception_trace>

  • Not possible to handle incident in site <site_id> for D<dsgGroup_id> <exception_trace>

  • Incidents handler error <exception_trace>

Temporary problems handling incidents.

ERR

Master election or SDL generation for D<dsGroup_id> has being interruped in the middle. Still trying to finish it

Temporary problems at leader election.

ERR

MastersChecker in site <site_id> can not get SDL info info in masterpath <path> from site <site_id> because node does not exist in BC server. Check your system configuration

Information path in the BC cluster does not exist.

WARNING

  • Problem getting SDL info for D<dsgGroup_id> in node <node_id>. Probably CS is not reporting or the node is down

  • Problem getting RPI for D<dsgGroup_id> in node <node_id>. CS is not reporting RPI. Active CS is reporting <status>

  • Problem getting RPI for D<dsgGroup_id> in node <node_id>. Node <node_id> does not exist in the BC server

Temporary problem getting status or replication information for a DSG.

WARNING

Not possible elect master (masterless) for D<dsgGroup_id> because there is no RPI available. Probably all the DSG replicas are down or non reporting. Election done by <systemMonitor_id>

No RPI information is preventing SM to make a master election.

WARNING

It has been not possible to report master election info (<systemMonitor_id>) in site <site_id> because <dsgGroup_id> does not exist (probably DSG it is not configured)

Problem reporting master election info.

WARNING

  • Not possible to spread SDL info for D<dsgGroup_id> in site <site_id> because <dsgGroup_id> node does not exists (probably DSG it is not configured)

  • SM <systemMonitor_id> PL going down. Messages will not be sent again until PL is alive

  • Not possible sending EXPIRE multicast with connection with BC service lost <exception_trace>

Problems are avoiding to send status information.

WARNING

D <dsgGroup_id> is down (CS reporting down, CS non reporting or local PL is down or no PL alive in site or node in disabledNode list) in node <node_id> (master) in site <site_id>. Raising master incident

Master incidents triggered.

WARNING

D0 (PL) is down or has stopped reporting in node <node_id> in site <site_id>. All the DSGs in node <node_id> will be considered down

PLDB is down. All DSGs in node are considered as down.

WARNING

  • Not possible to evaluate connection state change between sites <site_id> and <site_id> because evaluation has been interrupted. Probably leader is being relinquished

  • Not possible to evaluate connection state change between sites <site_id> and <site_id> <exception_trace>

  • Not possible to evaluate connection state change between sites <site_id> and <site_id> <exception_trace>

  • System Monitor <systemMonitor_id> not able to connect to local BC servers. Retrying...

Temporary problem in the connection to the BC Clusters.

WARNING

Not valid master value in partition for D <dsgGroup_id>. New master election needed

New master election is needed.

WARNING

Site <site_id> set as auto-removed for node <node_id>

The indicated site is down, therefore it is set as auto-removed for the nodes of other sites.

WARNING

Site <site_id> set as non auto-removed for node <node_id>

The indicated site is up and running, therefore it is set as non auto-removed for the nodes of other sites.

WARNING

Detected Master DSG: <dsg_id> in a disabled node: <remote_node_id> from node: <nodeId>, check configuration if the problem persists.

The local node configured a remote node with masters as disabled.

WARNING

Detected Master in a disabled DSG: <dsg_id> on node: <remote_node_id> from node: <nodeId>, check configuration if the problem persists.

The local node configured the master DSG of a remote node as disabled.

WARNING

Service Continuity couldn't be set automatically

serviceContinuity not set because partition status is empty in BCCluster

WARNING

Client disconnected from Messaging Service!

SM got disconnected from Messaging Service instance and will try to reconnect.

WARNING

Client connection closed to Messaging Service!

SM got disconnected from Messaging Service instance.

WARNING

Client reconnected to Messaging Service in <uri>!

SM needed reconnection to Messaging Service cluster and it was successful .

WARNING

An error occurred while changing DSG mastership: <reason>

Mastership Change could not be performed due to issues specified in <reason> part of the log message.

Refer to CUDB High Availability, Reference [2] for further information on these logs.

2.5.4.3.2   CS

The CS component uses syslog as logging system with the following properties:

CS logs have the following types of severities:

Table 16 shows CS logs with severity EMERG.

Table 16    EMERG CS Logs

EMERG Message Information

Trigger Event

  • Can not find multicast address. CS EXIT.

  • Failed to setup multicast socket for receiving. CS EXIT.

Failed setup multicast socket.

  • Failed to tasket angel ndbd (command via ssh failed)!

  • Failed to check ndbd child process.

  • Missing ndbd child process of parent ndbd with pid <pid>.

  • No MGM available.

  • Failed to wait for NDB.

  • Failed to wait for NDB, MGM nodes unreachable!

  • <node_type> <node_id> on <IP> not ready.

  • Failed to start <node_type> <node_id> on <IP> (it died or program not found)!

Failed starting MySQL cluster processes.

  • Can't stop NDBD processes.

  • Can't stop <node_type> <node_id> on <IP> (pid=<pid>).

Failed stopping MySQL cluster processes.

  • Failed to read configuration.

  • No multicast channel specified for sending cluster status.

Failed to read configuration.

Error creation of binlog index for host=<host_name>:<port>.

Failure in the creation of binlog index.

  • Unknown command

Unknown command read from command file.

Table 17 shows CS logs with severity CRIT.

Table 17    CRIT CS Logs

CRIT Message Information

Trigger Event

Failed to setup multicast sender.

Error setting/receiving multicast.

  • Unable to set connectstring <connect_string>.

  • Unable to create handle to ndb_mgmd.

Failure connecting to a MGMD.

  • Received NULL pointer.

  • File not found (errno=<error_code>):<file_name>.

  • Path not valid (errno=<error_code>):<file_name>.

  • Permission denied (errno=<error_code>):<file_name>.

  • Failed loading config.

  • No <CudbSystem> root element in <path_file>.

  • No local node id found in <path_file>.

  • Unhandled exception.

  • File name is NULL.

  • getDocument returned NULL:<file_name>.

  • Failed loading CudbPlDataMemoryGEP3.

  • Failed loading CudbDsDataMemoryGEP3.

  • Failed loading CudbPlIndexMemoryGEP3.

  • Failed loading CudbDsIndexMemoryGEP3.

Failure reading/parsing configuration file.

Running out of memory.

NDBD running out of memory.

Table 18 shows CS logs with severity ERR.

Table 18    ERR CS Logs

ERR Message Information

Trigger Event

Kickstart not allowed in this scenario, can only be done after initial installation, a system data restore or a group data restore.

Kickstart order is executed in the wrong situation, can be used only at installation and after system or group data restore.

  • No valid mysql connection.

  • initializeSQLAfterStart initialize failed for node <node_id> on host <host_name> with error:<error_code>.

  • initializeSQLAfterStart connection test failed for node <node_id> <server> on host <host_name>.

  • ReplicationDistanceCalculator::calcChannelStatus: connection to local slave failed for node <node_id>on host <host> with error: <error message>.

  • (Line <line where exception was thrown>)in ReplicationDistanceCalculator::calcSlaveLastEpoch: connection to local slave failed with error: <error message>

  • (Line <line where exception was thrown>)in ReplicationDistanceCalculator::calcDelay: getReplicationDistance connection to remote master failed: host <host> with error: <error message>.

Cannot connect to mysqld server.

  • connectToMgm: unable to retrive cudb_ds!

  • connectToMgm: unable to create MGMD handle!

  • connectToMgm: unable to set connection string!

  • connectToMgm: unable to connect with con str <connection_string>. Err: <error_code> <error_message>.

  • None of MGM nodes available.

  • connectToMgm: unable to create mgm handle!

  • No MGM.

  • connectToMgm: unable to retrive cudb_ds!

Error connection to MGMD.

  • Enter single user failed:<error_messsage>.

  • Exit single user failed:<error_messsage>.

Error entering or exiting single user mode.

(Line <line where exception was thrown>)in ReplicationDistanceCalculator::calcDelay: mysqlWrapperException getReplicationDistance failed: host <host> with error: <error message>.

Error fetching information from the binlog index of remote master.

  • checkClusterStatus connectToMgm failed.

  • Error checking cluster status.

  • check status failed to query node states.

Error getting cluster status.

  • initializeSQLAfterStart initialize failed for node <node_id> on host <host_name>! Error setting root password!

  • initializeSQLAfterStart initialize failed for node <node_id> on host <host_name>.

Error handling passwords.

  • Aborting - No mysqld login information found in XML config file for CUDB-node <node_id>, DS <ds_group_id>.

  • DSG <ds_group_id> not present in <path_file>.

  • Could not check/read: <file_name>.

  • Could not delete: <file_name> (<error_message>).

  • Could not write to <file_name>.

Error in configuration file.

API <node_id> not being a master or slave.

Error in mysqld configuration.

  • MySQL execution error:<error_code>:<error_message>, <query>.

  • Nowhere to store result set.

  • Empty resultset:<query>.

Error in MySQL query.

  • Binlog index does not exist for host=<host_name>:<port>.

  • Out of memory in ReplicationState::Instance().

Error in replication process.

  • Setup error: socket not initialized.

  • select() socket failed: errno=<error_code>.

  • failed receiving: errno=<error_code>.

  • Could not create TCP socket with socket(): errno=<error_code>.

  • Could not bind TCP socket (port=<port>): errn0=<error_code>.

  • Could not get IP address with getsockname(): errno=<error_code>.

  • Could not put TCP socket in listening state: errno=<erros_code>.

  • select() socket failed: errno=<error_code>.

  • Failed to accept() TCP connection: errno=<error_code>.

  • Setup error: socket not initialized.

Error setting/creating a socket.

Failed to receive on multi-cast socket.

Error setting/receiving multicast.

  • failed to execute: <command>

  • Unknown command: <command>.

Error in ssh command.

  • Error received in sigwait, error value EINVAL.

  • Error received in sigwait, error value ENOTSIGINIT.

Error in the thread handling.

  • initializeSQLAfterStart initialize failed for node <node_id> on host <host_name> with error:<error_code>.

  • initializeSQLAfterStart connection test failed for node <node_id> <server> on host <host_name>.

Error trying to connect to MySQL server.

  • Invalid backup file descriptor.

  • Restore failed.

  • Restore did not complete successfully.

Failed data restore.

Failed to allocate memory, error=<error-code>. Permanent Error. Aborting connection to host=<host_name> port=<port>.

Failed to allocate memory.

  • Unable to determine starting point in binary log.

  • replcfg_get_binlog_lastevent: unable to get last event.

Failure checking binlog file.

Master for replication channel <channel> in DS group <ds_group_id> not found int CUDB node <node_id> in XML config file.

MasterDSG not found in configuration.

COMM_MC: Master-list expired (TTL of <time_in_milliseconds> ms reached) : Master=*** NONE ***.

Masterlist info expired.

(Line <line where exception was thrown>)in ReplicationDistanceCalculator::calcSlaveLastEpoch: curMaster is neither lastEffectiveMaster nor lastSetupMaster [tag_LastEffectiveMaster*] = <last effective master id>[tag_LastSetupMaster*] = <last setup master id>[tag_CurrentMaster*] = <current master id>.

Replication status is inconsistent with mastership info.

Slave for DS group <ds_group_id> not found int CUDB node <node_id> in XML config file.

Slave DSG not found in configuration.

  • Unable to open restore pipe.

  • Unable to close pipe.

Unable to open/close ssh pipe for a command.

  • Unable to retrieve last epoch on slave.

  • Unable to retrieve last epoch on new master.

Unable to retrieve last epoch.

  • Check Orphan files: Failed to read list of binlogfiles from binlogfolder on host <host_name>:<port>!.

  • Check running purge sessions on host <host_name>:<port> failed with error <mysql_error_description>

  • Connection to <host_name>:<port> failed with error:<error_description>.

  • Load list of binary logs from <host_name>:<port> failed with error:<error_description>.

  • Purge on <host_name>:<port> failed with error:<error_description>. Purge file <filename>, purge file parameter <filename>.

  • Check 'purge_session' in schema failed on <host>:<port> failed with error: <error_message>."

  • Check Number Of Binlog Files By Timestamp: Failed to read list of binlogfiles from binlog folder on host <host_name>:<port>!

Errors related to the purge and orphan binlog files check

FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: <tool> handler fetching data. ERROR DESCRIPTION:: Missing <missing_data>. Cannot continue with <tool>.

Finite state machine handler was not able to get some data (node_id, timestamp, task_id, ...) needed for proper execution and checking status of tool (Data Repair, Selective Replica Check). Missing data should be saved internally in state machine.

FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: SOBR <state>. ERROR DESCRIPTION:: Data fetch error: Missing <missing_data>. Cannot continue with SOBR.

Finite state machine handler was not able to get some data (node_id, timestamp, task_id, ...) needed for proper execution and checking status of Self-Ordered Backup and Restore in the current state. Missing data should be saved internally in state machine.

FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: <tool> finished. ERROR DESCRIPTION:: <tool> did not finish correctly on <node>, exit code was <exit_code>.

Tool (Data Repair, Selective Replica Check, Self-ordered Backup and Restore) exited with error exit code. Error exit codes for Selective Replica Check are 1, 3, 5, 7. Error exit codes for Data Repair are 1, 3, 5. Error exit code for Self-Ordered Backup and Restore is 1. Error code greater or equal to 126 indicates some other problem (wrong permissions, missing binary, and so on).

FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: <tool> running. ERROR DESCRIPTION:: Unable to extract exit code of <tool> on <node>, got <return_string>.

Tool (Data Repair, Selective Replica Check, Self-Ordered Backup and Restore) finished, but something went wrong parsing return code from output file.

FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: <tool> running. ERROR DESCRIPTION:: Unable to get status or exit code of <tool> on <node> after retrying. Giving up.

Could not get status of currently running tool (Data Repair, Selective Replica Check, Self-Ordered Backup and Restore).

FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: <tool> start. ERROR DESCRIPTION:: <tool> unable to start on <node> after retrying. Giving up.

Could not start tool (Data Repair, Selective Replica Check, Self-Ordered Backup and Restore) over ssh on node.

FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: <tool> start. ERROR DESCRIPTION:: Unable to get PID of <tool> on <node>, got <returned_string>.

Tool (Data Repair, Selective Replica Check, Self-Ordered Backup and Restore) started on node, but could not extract PID from returned string.

  • FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Calculating missing intervals. ERROR DESCRIPTION:: Unable to calculate missing intervals on <node> after retrying. Giving up.

  • FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Calculating missing intervals. ERROR DESCRIPTION:: Unable to calculate missing intervals on <node>, got <return_string>.

Unable to calculate total amount of missing intervals after Selective Replica Check scan execution.

FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Reconciliation. ERROR DESCRIPTION:: Can't trigger reconciliation, target PLDB's DSG list is empty.

Failed to get DSG list for PLDB from Blackboard Coordinator.

FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Reconciliation. ERROR DESCRIPTION:: Can't trigger reconciliation, target PLDB hostname list is empty.

Faild to get PLDB hostname from cudbSystem.xml.

FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Reconciliation. ERROR DESCRIPTION:: Couldn't get list of all DSG's in the system.

Failed to get list of all DSGs in the system from Blackboard Coordinator.

FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Reconciliation. ERROR DESCRIPTION:: Couldn't get master node of incident DSG.

Failed to get master node of incident DSG from Blackboard Coordinator.

FSM ErrorHandler: Orchestrator execution failed. ERROR STATE:: Reconciliation. ERROR DESCRIPTION:: Trigger reconciliation failed, MySQL exception has occurred: <mysql_exception_description>.

Error in MySQL.

Repair&Resync: Cannot get data from state machine!

Incident related data (timestamp, task_id) needed for the Automatic Handling of Network Isolation cannot be obtained. Data should be present internally in state machine.

basethread,pthread_condattr_setclock failed
basethread,pthread_condattr_init failed
basethread,pthread_cond_init failed

Error while initializing clock for CS.

Ndb binlog injector error on Mysql server <host_name>:<port>!

The ndb binloginjector thread failed to subscribe to table events on one or more tables in the cluster. The MySQL server needs to be restarted to fix this problem.

  • Start check failed for <node_type> <node_id> on <IP>!

  • Process check failed for <node_type> <node_id> on <IP>!

Error checking process.

ERROR: Unable to stop ndbmtd <node_id> after two retries, cluster will be put to maintenance

Error stopping NDB cluster.

Failed to create thread

Cannot create thread.

BackupHandler: Not all the necessary backup files are created.

Failed to create all the necessary backup files.

BackupHandler: Not all the necessary backup files are valid.

Failed to validate all the necessary backup files.

BackupHandler: Failed to connect to Management server.

Failed to connect to the NDB Management server.

BackupHandler: Failed to create logevent handle.

Failed to create the NDB API logevent handle.

BackupHandler: Could not get cluster status.

Could not get cluster status from the NDB API.

BackupHandlerThread: Failed to recover state.

Failed to recover backup state during failover.

BackupHandlerThread: Failed to extract backup location from the command file.

Failed to extract backup location from the command file.

BackupHandlerThread: Failed to execute dummy transaction on access nodes.

Failed to execute dummy transaction on access nodes.

BackupHandlerThread: Failed to set NDB handles.

Failed to set NDB handles.

BackupHandlerThread: Failed to collect connected NDB blades.

Failed to collect connected NDB blades.

BackupHandlerThread: Failed to start backup.

Failed to start backup.

  • Failed to connect to IMM: too many failed attempts

  • Failed to connect to IMM: connection rejected

Failed to connect to IMM.

  • Failed to initialize IMM OM: too many failed attempts

  • Failed to initialize IMM OM: unknown error

Failed to initialize IMM Object Manager.

  • Failed to finalize IMM OM accessor

  • Failed to finalize IMM OM

Failure at disconnection from IMM.

  • Failed to read <object> object from IMM: object does not exist

  • Failed to read <object> object from IMM: invalid parameter

  • Failed to read <object> object from IMM: unknown error

  • Unsupported type while reading attribute <attribute> from IMM

The object or the attribute cannot be read from IMM.

No uuid was read from IMM

The uuid attribute does not exist in IMM.

Table 19 shows the CS logs with severity WARNING.

Table 19    WARNING CS Logs

WARNING Message Information

Trigger Event

  • Failed to get cluster state. Management servers might be down.

  • Failure getting cluster status from all ndb_mgmd.

  • Failure all ndb_mgmd are down.

  • Cannot connect to management nodes. Repair skipped for node <node_id>.

All MGMD down.

A purge session is already running on <host>:<port>! details: start time:<start time> user:<user_name> Info:<currently_executing_task>

Binlog purge failure.

  • Number of orphan binlogfiles:<number of files not listed in log-bin.index> on <host_ip>:<port>!

  • Removing orphan binlogfile <binlogfile>, session_id <session_id>, on <host_ip>:<port>.

  • Ndb_binlog_index record count for <binlogfile> is:<record_count>, on <host_ip>:<port>!

Binlog files not listed in log-bin.index file found.

  • Error, CS node doesn't exist, trying to repair.

  • Error, DSG node doesn't exist, trying to repair.

BC server nodes do not exist.

  • Unable to connect to mysqld.

  • Can't connect to mysql server.

  • Failed to connect to mysqld server, error=<error_message> Aborting connection to host=<host_name> port=<port> timeout=<timeout_in_seconds>.

  • Failed to connect to mysqld server. Aborting epoch check for host=<host_name> port=<port> and master server-id in (<servr_id>).

  • Failed to connect to mysqld server. Aborting \stop slave\ for host=<host_name>port=<port>.

  • Failed to connect to mysqld slave server. Aborting replication start for slave=<host_name>:<port> to master=<host_name>:<port>.

  • Failed to connect to mysqld master server. Aborting replication start for slave=<host_name>:<port> to master=<host_name>:<port>.

  • Unable to remove local epoch info's from ndb_apply_status, <error_message>.

  • ReplicationControl::removeAppliedEpochOnMaster Failed to remove applied epochs from master '<error_message>'.

Cannot connect to mysqld server.

  • Start mysqld process failed for node <node_id> on host <host_name>! Unable to connect to cluster management nodes!

  • Start mysqld process failed for node <node_id> on host <host_name>! mysqld process not connected to cluster!

CS could start the mysqld process but the process did not connect to the mysql cluster within time-out so the start process failed.

Start mysqld process failed for node <node_id> on host <host_name>! mysqld process cluster connection check failed!

Newly started mysqld process reported to be connected to the mysql cluster but it is not reachable (it did not allow connections OR it had lost the connection to the cluster).

  • start: Check Available NDBS: NOT enough NDBDs.

  • start: wait for cluster failed, no more retries left!

  • MySQL slaves are *** down ***.

  • Cluster is *** down ***.

  • Cluster is *** degraded ***.

  • MGM node is *** down ***

  • MySQL masters are *** down ***

  • NDB is *** down ***

Cluster down/degraded.

  • Connection with BC-cluster has been broken permanently.

  • Connection with BC-cluster has been broken temporarily.

Connection to BC server broken.

  • errono:<error_code>, msg:<error_message>.

  • Connection failed.

  • Cluster <connect_string> was not ready within <seconds> secs.

  • Master <node_id> is not writing to binlog or has not yet receievd any traffic.

  • No binlog epoch info on masters is available.

Error accessing to a database/table.

Process check failed for <process_name> on <host_name> !

Error checking ndb_restore process.

  • Error getting info for CS node.

  • Master information received with wrong DSG.

Error getting information from BC server.

setSQlPasswords initialize failed for node <node_id> on host <host_name> with error:<error_code>.

CS failed to set the proper user grants during mysqld server initialization.

API <node_id>: check/repair process failed, unable to set 'root' user password.

CS failed to set the root password. If this happens in CS "ready" state, it will retry the fix in the next iteration. If it happens during cluster "initialization", execute the cudbManageStore -o initialize command again.

API <node_id>: check/repair process failed, unable to set access/replication user password.

CS failed to set the password for "access" or "replication" user. If this happens in CS "ready" state, it will retry the fix in the next iteration. If it happens during cluster "initialization", execute the cudbManageStore -o initialize command again.

  • Failed to connect to mysqld server. Aborting \DELETE FROM mysql.ndb_apply_status WHERE server_id=<server_id> for host=<host_name>, port=<port>.

  • Failed to delete mysql.ndb_apply_status entries in node=<host_name>:<port> for server=<server_id>.

Error handling replication in the initialization of the cluster.

  • Replication info not gotten.

  • Failed to start replication for slave=<host_name>:<port> from master=<host_name>:<port>, start epoch=<epoch>.

  • Failed to connect to mysqld slave server. Aborting replication start for slave=<host_name>:<port> to master=<host_name>:<port>.

  • Failed to kickstart replication for slave=<host_name>:<port> from master=<host_name>:<port>.

  • Failed to connect to mysqld slave server. Aborting replication start for slave=<host_name>:<port>: to master=<host_name>:<port>

  • Failed to connect to mysqld master server. Aborting replication start for slave=<host_name>:<port>: to master=<host_name>:<port>

  • Started replication to slave=<host_name>:<port>: from master=<host_name>:<port>

  • Failed to start replication for slave=<host_name>:<port>: from master=<host_name>:<port>. The epoch of the new master is behind the epoch of the slave.

  • Failed to start replication for slave=<host_name>:<port>: from master=<host_name>:<port>

  • Failed to connect to mysqld slave server. Aborting replication start for slave=<host_name>:<port>: to master=<host_name>:<port>

  • Failed to connect to mysqld master server. Aborting replication start for slave=<host_name>:<port>: to master=<host_name>:<port>

  • Can't stop replication on slave Server-id=<server_id>. Slave might be down, continues replication setup anyway.

  • Can't get replication start parameters for slaves. Aborting replication setup.

  • Can't kickstart replication on slave. Aborting replication setup.

  • Failed to get the last binlog event on Slave Node - Temporary error.

  • Old master server doesn't have any epoch information - Permanent error.

  • Can't start replication. Flow through program. Aborting replication setup.

  • No slave server online. Aborting replication setup.

  • Sync replication failed - Permanent error.

  • Can't get replication start parameters for slaves. Aborting replication setup.

  • Can't start replication on slave. Aborting replication setup.

  • Can't stop replication on slave Server-id=<server_id>. Slave might be down, continues replication setup anyway.

  • New master at <host>:<port> is behind slave at <host>;<port>, leave slave as it is.

  • Replication already running on the slave.

  • LOST EVENT occurred as last event on Master Node - Permanent error.

  • Channels stopped after kickstart, another kickstart is required to fix replication.

  • Sql Slave unavailable - Temporary error.

  • LOST EVENT occurred in binlog as last event on Slave Node - Permanent error.

  • Failed to get the last replicated epoch - Temporary error.

  • Sync replication failed - Temporary error.

  • Can't get valid serverId. Aborting replication setup.

Error in replication handling.

  • <thread_name> is hanging, finalize.

  • No Thread=<thread_id> could be found corresponding to thread.

  • The thread has allready been detached OR Another thread is already waiting on termination of the thread.

  • The thread argument refers to the calling thread. One can not destroy one self.

  • Unkown pthread_join return code.

  • MgmThread stop run method early.

  • MgmThread stop run method early, standby mode.

  • No Thread=<thread_id> could be found corresponding to thread.

  • The thread has allready been detached OR Another thread is already waiting on termination of the thread.

  • The thread argument refers to the calling thread. One can not destroy one self.

  • Unkown pthread_join return code <error_code>.

Error in thread handling.

  • Error in masterList watcher, value read:<master_value>, value stored:<master_info>.

  • Error in sdlList watcher, value read:<master_value>, value stored:<master_info>.

  • Error in workingMode watcher, value read:<master_value>, value stored:<master_info>.

  • Error setting SdlList watcher.

  • Error setting Working mode watcher.

  • Watcher result not ok, ignoring working mode received.

  • Watcher result not ok, ignoring SdlList received.

  • Wrong format Sdl list received, ignoring SdlList watcher.

  • Watcher result not ok, ignoring Provisioning Assurance info received

  • Error setting Provisioning assurance watcher

  • Trying to repair Provisioning Assurance watcher

  • Error in SOBR info watcher, value read: <SOBR value>, value stored: <SOBR info>

  • Error setting SOBR info watcher

  • Watcher result not ok, ignoring SOBR info received

Error in watcher.

  • tablespaceMonitoringParams are invalid - defaulting.

  • Invalid hostname=<host_name>.

  • Empty ESA list in xml config file.

  • Error reading configuration information.

  • Slave is NULL, slave for DS group <ds_group_id> not found in CUDB node <node_id> in XML config file.

  • Slave for DS group <ds_group_id> not found in CUDB node <node_id> in XML config file.

  • Master for replication channel in DS group <ds_group_id> not found in CUDB node <node_id> in XML config file.

  • Local master servers not found in XML config file for <ds_group_id> <node_id>.

Error reading configuration.

  • Could not fetch cudb_cluster_supervisor.last_master content for node id:<node_id>. Returning zeros.

  • Could not store data in cudb_cluster_supervisor.last_master table for node id:<node_id>. Returning zeros.

  • Could not clear data from cudb_cluster_supervisor.last_master table.

  • Last master table is empty:<node_id>.

  • Permanent error when trying to access last_master table for node id:<node_id>.

  • Permanent error when trying to clear last_master table.

  • Error reading last master info.

Error reading last master information from the local NDB cluster.

Send error: errno=<error_code>.

Error sending a TCP message.

  • No multicast channel specified for cluster status.

  • Failed to setup multicast receiver.

Error setting/receiving multicast.

Failed the disable of the events on binlog, error=<error_message>. Aborting conection to host=<host_name> port=<port>.

Error setting events on binlog in MySQL.

  • Error getting info for DSG node.

  • Error creating DSG reporting node in BC-cluster.

  • error creating CS node.

  • Error deleting DSG node.

  • Error updating DSG node with value:<dsgroup_id>, node does not exist.

  • Error updating DSG node with value:<dsgroup_id>.

  • Error setting MasterList watcher.

  • Masterlist watcher result not ok, exit updateCSMasterState.

  • Master information received for DSG <dsgroup_id> has wrong format:<info>.

  • Error updating CS node with value:<status>.

  • BC connection is down, cannot sendRDMStatus()

  • Unable to update node with RDM status: <RDM_status>

  • Error creating SOBR node in BC-cluster

  • Error updating SOBR info node with value: <SOBR info>

Error updating information to BC Server.

  • Unexpected restore output:<message>.

  • Error while creating databases.

Failed data restore.

Error trying to delete entries in ndb_apply_status for server_id not equal to <server_id>.

Failed to clear ndb_apply_status table.

  • Could not fetch Index/Data memory usage.

  • Failed to get cluster memory usage.

Failed to get cluster memory usage.

  • Unable to read last binlog event.

  • Failed to connect to mysql server. Aborting checking of binlog index for host=<host_name>:<port>.

  • Failed checking of binlog index for host=<host_name>:<port>.

  • Failed to connect to mysql server. Aborting creation of binlog index for host=<host_name>:<port>.

Failure checking binlog index.

  • NDB <node_id>:check/repair failed.

  • API <node_id>:check/repair process skipped, DataPath not found <data_path>.

Failure checking/repairing MySQL processes.

  • Failed to create NdbMgmHandle.

  • Failed to set connect string. Connect string=<connect_string>.

  • Failed to set timeout. Timeout=<time_in_seconds>.

  • Failed to connect to Management server. Connect string=<connect_string>.

  • Failed to connect to ndb_mgmd <connect_string>.

Failure connecting to MGMD.

  • Failure in command: du --summarize -BK <path_name>.output:<error_message>.

  • Binlog_is_consistent(const char* Host, const char* binlog_path) - command <command> failed.

  • Failed to start <node_type> <node_id> on <IP> (command via ssh failed)!

  • Kill -9 on <node_type> <node_id> at <IP> (pid=<pid>).

  • Kill -9 on <node_type> <node_id> at <IP> (pid=<pid>) (nameprocess =<name_process>).

Failure executing a command.

  • Failed to enable corefiles.

  • Failed to open file <path_file>.

  • Failed to write data to file <path_file>.

  • Failed to close file <path_file>.

Failure handling a file.

  • Trying to repair CS node value.

  • Couldn't repair CS node value.

  • Trying to repair MasterList get.

  • Couldn't repair MasterList get.

  • Trying to repair MasterList watcher.

  • Couldn't repair MasterList watcher.

  • Trying to repair SdlList watcher.

  • Couldn't repair SdlList get.

  • Trying to repair SdlList get.

  • Couldn't repair SdlList watcher.

  • Trying to repair WorkingMode watcher.

  • Couldn't repair WorkingMode watcher.

  • Trying to repair SOBR info watcher

  • Couldn't repair SOBR info watcher

Failure repairing BC server node.

  • cl_control,Mount point <data_path> not mounted!

  • Failed to check if <data_path> is mounted!

Mount point checking failure.

New master config recived CUDB_NODE=<node_id> DS_GROUP=<ds_group_id>.

New master config received.

  • COMM_MC: Not receiving any config : Master = *** NONE ***.

  • COMM_MC: Received empty master-list: *** EXPIRE ***.

No information received from multicast.

Unable to trigger UPDATE dummy operation in CUDB_REPLICATION_DUMMY on server <host_name>:<port>, <error_message>.

Unable to execute dummy operation.

  • Event standby invoked but we are already in standby state.

  • Event active invoked but we are already in active state.

Unexpected AMF change received.

Received signal not expected, ignoring it.

Unexpected signal received.

Cluster status info is not available for DSG <dsgroup_id>.

CS was not able to ask cluster status from MGM nodes.

Failed Triggering dummy operation on master DS <masterCUDB>.

After a master mysqld is restarted, the system tries to avoid having lost events in the same position on both replication channels. In this case it failed to generate a dummy event on the DSG master replica.

Clean stop procedure for Mysql server with ndb node id <node_id> failed on <host_ip> with error: <error_message>. Killing the server.

CS failed to shut down the mysqld instance in a graceful way. The mysqld process is stopped in a forced way (using 'kill -15', followed by 'kill -9').

Ndb Binlog Index is corrupted! Node <cudb_node_id>, DSG <ds_group_id> API node ID <node_id>, Server ID <server_id>. Starting reinitalization.

The check of ndb binlog index table found inconsistency after starting or restarting master MySQL server. Inconsistency can have been caused by a non-graceful stop of the mysqld process or a file system error.

Error executing Ndb Binlog Index check. Binlog state unknown.

CS could not execute the check statement on master MySQL server after starting or restarting it.

  • FSM: State Machine not running.

  • FSM: No transition from state <state> with event <event>.

  • FSM: State already exists.

Error in finite state machine.

FSM <handler>: Problem executing <command> remotely on <node>.

Error executing command over ssh.

Repair&Resync: Timestamp value is 0!

Attempt to start Automatic Handling of Network Isolation without a valid incident timestamp. Incident timestamp is read from Blackboard Coordinator.

Repair&Resync: Cannot start finite state machine, exiting from thread.

Attempt to start the state machine for the Automatic Handling of Network Isolation when it is already running.

  • Repair&Resync: Cannot initialize csDbIo object.

  • Repair&Resync: Cannot get info about last master.

Failed attempt to fetch information about last master from ClusterSupervisor database (MySQL).

  • Failed to start <node_type> <node_id> on <IP> (it died or program not found)!

  • No MGM available

  • Failed to wait for NDB

  • Failed to wait for NDB, MGM nodes unreachable!

  • Missing ndbd child process of parent ndbd with pid <pid_process>

  • Failed to check ndbd child process

Failed to start MySQL cluster processes.

  • Can't stop NDBD processes

  • Can't stop <node_type> <node_id> on <IP> (pid=<pid_process>)

Failed to stop MySQL cluster processes.

Cluster state is down! No nodes have been started!

Error getting cluster status.

RDM status is unknown, not updating BC cluster

Error getting RDM status.

Error Calculating timestamp to PG, binlog epoch in master is lower than last epoch in slave table apply status, check if master has been initialized or a system restore is in place.

Error calculating timestamp to PG

Rollover has not been successfully completed. Another rollover attempt will be made.

ClusterSupervisor instance rollover cannot be completed because BC connection cannot be established or DSG node cannot be created in BC. Reattempting another rollover!

BackupHandler: Error getting next event

Can not get the next event from the NBD API.

BackupHandler: Error writing backup information into file: <meta file path> with error: <error message>

An error occurred while trying to write backup information into the meta file.

BackupHandler: Validation failed for node <node id> with MGM hosted on <ip>.

Failed to verification the backup for the given node.

BackupHandler: Problem executing <command>.

Problem executing the NBD restore command to verify the backup.

2.5.4.3.3   KeepAlive

KeepAlive component uses syslog as logging system with the following properties:

The messages logged by the KeepAlive components are summarized in Table 20.

Table 20    KeepAlive Logs

Severity

Message Information

Trigger Event

ERR

Error running <description>

Error getting KA_Cycle value from keepAlive.xml.

ERR

<error_description>

Error getting KA_CHECK value from keepAlive.xml.

ERR

Incorrect execution, it must NOT begin with '.'

The process was launched in a wrong way.

WARNING

Monitoring disabled

Warning about the monitor is not enabled and has not check anything.

WARNING

Component '<component_name>' has capability 1_ACTIVE: it should NOT receive standby assignment from AMF

The component cannot change its state to standby.

WARNING

Component '<component_name>' has capability 1_ACTIVE: this library does not support assingment removal for this capability

No assignment is given to the component.

WARNING

  • Command '<command>' could not be executed: system() call failed with result: <result_code>

  • Command '<command>' was not terminated normally.

Error executing the given command.

2.5.4.3.4   High Availability

High availability component uses syslog as logging system with the following properties:

The messages logged by the high availability components are summarized in Table 21.

Table 21    High Availability Logs

Severity

Message Information

Trigger Event

WARNING

Command <command> could not be executed: system call failed with result: <result_code>

A system call has failed when executing a command.

WARNING

Command <command> was not terminated normally

A command execution has failed for some reason.

WARNING

Component <component_name> has capability 1_ACTIVE: it should NOT receive standby assignment from AMF

Component CudbPmAgentMonitor received a STANDBY assignment whereas its configuration is ACTIVE.

WARNING

Component <component_name> has capability 1_ACTIVE: this library does not support assignment removal for this capability

NO ASSIGNMENT state is not supported for this component.

2.5.4.3.5   Security Logs Monitor

The Security Logs Monitor component uses syslog as logging system with the following properties:

Security Logs Monitor uses the Availability Management Framework (AMF) service. The messages logged by this component are summarized in Table 22.

Table 22    Security Logs Monitor

Severity

Message Information

Trigger Event

ERR

AmfListener.cpp::callWithRetryNoChk <line number> - Too many failed connect attempts

An error occurred while using the AMF service.

ERR

not SA_AIS_OK: <value>

An error occurred while using the AMF service.

ERR

Exception caught in SecLogMonitor.cpp::operator <line number> - Cannot configure syslog for Security Log streaming

Unexpected error while the external security events logging is being enabled.

ERR

Exception caught in SecLogMonitor.cpp::operator <line number> - Cannot delete configuration for syslog for Security Log streaming

Unexpected error while the external security events logging is being disabled.

ERR

Exception caught in SecLogMonitor.cpp::operator <line number> : <exception>

Unexpected error while the external security events logging is being enabled or disabled.

ERR

Exception caught in main.cpp::main <line number> - Unhandled exception occurred

Unexpected error during Security Log Monitor process execution.

2.5.4.3.6   Replica Consistency

The Replica Consistency component uses syslog as logging system with the following properties:

Replica Consistency logs have the following types of severities:

Table 23 shows the Replica Consistency logs with severity ERR.

Table 23    ERR Replica Consistency Logs

ERR Message Information

Trigger Event

Unable to get the system configuration file on node <node_id> - <res_string> - <node_IP>

Can not get the cudbSystem.xml from the remote Node.

No database is specified. Use [-p | -d <dsgId>] or [--pl | --dsg <dsgId>] or see help

No DSG to backup was specified with the command line arguments.

DSG <dsg_id> does not exist in the CUDB system

The specified DSG ID to backup does not exist in the CUDB system.

No destination node is specified. Use -n or --node or see help

No destination node was specified with the command line arguments.

Node <node_id> does not exist in the CUDB system

The specified Node does not exist in the CUDB system.

DSG <dsg_id> has no replica on source node <node_id>

The specified DSG ID to backup has no replica on the source Node.

DSG <dsg_id> has no replica on destination node <node_id>

The specified DSG ID to backup has no replica on the destination Node.

Unable to determine the holder of Master replica for DSG <dsg_id>, no information in the BC cluster

Can not get any information from the BC cluster.

Unable to determine the holder of Master replica for DSG <dsg_id> from the BCClient info

Can not determine the holder of Master replica for the given DSG from the information available in the BC cluster.

Unable to determine the holder of Master replica for DSG <dsg_id> from the MySQL info

Can not to determine the holder of Master replica for the given DSG from the MySQL info.

Node <node_id> has the Master replica for DSG <dsg_id>

The destination Node has the Master replica for the specified DSG.

Unable to determine the holder of Master replicas, no information in the BC cluster

Can not determine the holder of Master replicas from the information available in the BC cluster.

The local node has Master replica(s)

The local node has Master replicas before the backup transfer or restore procedure while processing PLDB replica.

Some of the blades hosting master replica for DSG <dsg_id> are unreachable via ssh but their ndbs are running

Some of the blades hosting master replica for the specified DSG are unreachable via ssh but their ndbs are running.

Unable to copy backup files from <blade_id>

Can not copy backup files on the source blade from the backup location to the temporary directory.

DS Mastership changed

DS Mastership changed before the backup transfer or restore procedure.

Backup creation was unsuccessful

Failed to parse the backup location from the cudbManageStore output.

Unable to get the number of running ndb processes for DSG <dsg_id>

Can not get the number of running ndb processes for the specified DSG.

Command execution on <local/remote> node <node_IP> failed: Unable to create backup on <node_id>

Can not create backup on the Master node.

Command execution on <local/remote> node <node_IP> failed: Unable to create backup directory <backup_location> on <node_id> on blade <blade_id>

Can not create backup directory on the destination Node.

Command execution on <local/remote> node <node_IP> failed: Unable to create backup dir under /cluster/tmp/backups on Master Node <node_id>

Can not create backup directory on the Master Node.

Command execution on <local/remote> node <node_IP> failed: Unable to create backup dir under /cluster/tmp/backups on Node <node_id>

Can not create backup directory on the destination Node.

Command execution on <local/remote> node <node_IP> failed: Unable to copy backup files to <node_id>

Can not copy backup files to the destination Node.

Command execution on <local/remote> node <node_IP> failed: Unable to copy backup files from the temporary dir to the backup location on <blade_id>

Can not copy backup files from the temporary directory to the backup location on the target blade.

Command execution on <local/remote> node <node_IP> failed: Unable to remove backup dir under /cluster/tmp/backups on <node_id>

Can not remove the backup directory on the destination Node.

Command execution on <local/remote> node <node_IP> failed: Unable to remove backup dir <backup_location> on <blade_id>

Can not remove backup directory on the source Blade.

Command execution on <local/remote> node <node_IP> failed: Unable to remove backup dir under /cluster/tmp/backups on <node_id>

Can not remove backup directory on the Master Node.

Command execution on <local/remote> node <node_IP> failed: Unable to restore backup on <node_id>

Failed to restore backup on the destination Node.

Unable to remove backup transfer flag on node <node_id>

Can not remove the backup copy flag on the Master Node.

Unable to remove backup transfer flag dir on node <node_id>

Can not remove the backup copy flag directory on the Master Node.

Unable to create lock directory on node <node_id>

Can not create lock directory on the Master Node.

Unable to get the copy lock on node <node_id>

Can not get the copy lock on the Master Node.

Unable to create the backup ready file on node <node_id>

Can not create the backup ready file on the Master Node.

Unable to create transfer flag file on node <node_id>

Can not create transfer flag file on the Master Node.

Unable to create transfer flag directory on node <node_id>

Can not create transfer flag file directory on the Master Node.

2.5.4.3.7   Licensing

Licensing component uses syslog as logging system with the following properties:

The messages logged by the Licensing subcomponent are summarized in Table 24.

Table 24    Licensing Logs

Severity

Message Information

Trigger Event

CRIT

Problem reading cudbSystem.xml, can't start process! Exiting!

System configuration file /cluster/home/cudb/common/config/cudbSystem.xml cannot be read.

ERR

  • Capacity license for <profile_name> profile not found!

  • Problem with getting capacity license for <profile_name> profile.

Error getting capacity license for requested profile.

ERR

Capacity limit for <profile_name> profile is exceeded!

Capacity limit for requested profile is exceeded.

ERR

Capacity license for <profile_name> profile is rejected for the following operational mode: <operational_mode>

License Manager (LM) is in the operational state that enforces rejection of request for capacity license.

ERR

License Manager handler couldn't be initialized, can't check licenses

An error occurred while initializing LM handler.

ERR

Requested capacity license doesn't exist

Capacity license for requested profile is not loaded.

ERR

Capacity license for <profile_name> profile is rejected for the license type: <license_type>

LM has license type that enforces rejection of request for capacity license.

ERR

Wrong version of LM! Can't use License Manager API!

Wrong LM version detected.

ERR

Application ID is invalid. Problem with License Manager.

Attempt to use API version not supported by LM.

ERR

Registration to License Manager failed!

An error occurred trying to register LM.

ERR

Could not find any license type!

None of the allowed license types are loaded.

ERR

AMF initialization failed, shutting down process.

Could not initialize AMF service.

ERR

AMF problem, shutting down process.

An error occurred while using the AMF service.

ERR

Could not connect to LDAP!

An error occurred while connecting to the CUDB LDAP interface to fetch accounting information.

ERR

Failed to set signal mask for thread, exiting!

An error occurred while modifying the signal mask for the calling thread.

ERR

Error in sigwait.

An error occurred while waiting for a signal.

ERR

  • Failed to raise an alarm. Diagnostic info: <diagnostic_information>

  • Failed to clear an alarm. Diagnostic info: <diagnostic_information>

Diagnostic info: <diagnostic_information>.

ERR

init - saImmOmInitialize returned error

Failed to initialize IMM Object Manager.

ERR

init - saImmOmAccessorInitialize returned error

Failed to initialize IMM Object Manager Accessor.

ERR

readFingerprint - error fingerprint could not be read or is empty!

Fingerprint cannot be read or it is empty.

WARNING

init - saImmOmInitialize returned SA_AIS_ERR_TRY_AGAIN

Temporary problem initializing an IMM Object Manager.

WARNING

init - saImmOmAccessorInitialize returned SA_AIS_ERR_TRY_AGAIN

Temporary problem initializing an IMM Object Manager Accessor.

WARNING

readFingerprint - saImmOmAccessorGet_2 returned SA_AIS_ERR_TRY_AGAIN

Temporary problem reading fingerprint.

2.5.4.4   Notifications

This section describes the subcomponent of the notifications component.

2.5.4.4.1   SOAP

The notifications component uses syslog as logging system with the following properties:

The messages logged by the notifications component are summarized in Table 25.

Table 25    Notifications Logs

Severity

Message Information

Trigger Event

ERR

<error_message> <file_name> <number_line_file>

This generic message is returned mainly by CUDB checker when an error has occurred retrieving some objects from database.

ERR

Configuration Configuration Error! Source:<component> Description:<error_description>

Error occurred while the configuration was being parsed. Information on configuration component and detailed description of the error are attached. component denotes the source/component context.

ERR

Amf listener Error while initializing AMF component! <error_description>

Error occurred while AMF service is configured.

ERR

Cannot get instance of CudbConfig class. Aborting.

Error occurred in attempt to get configuration data from data model.

ERR

Not possible to subscribe to SM.

The Notification component is not able to get info from system monitor.

WARNING

Error received from Notification <endpoint>

Error has occurred while evaluating the notification result: either a wrong HTTP code from FE has been received or a SOAP client or server fault is encountered.

WARNING

Broadcast Notification EP (<EP>) is marked as unavailable

Broadcast endpoint is marked as unavailable due to too many connection errors.

WARNING

Notification was not sent to broadcast EP (<EP>) because it was marked unreachable

Notification was not sent to broadcast endpoint because it was unreachable.

WARNING

Internal Notification error

CUDB Database Events internal notification error.

WARNING

Internal Notification error: attrList size is 0

CUDB Database Events internal notification error: attribute list is empty.

WARNING

Internal Notification error: dnObjectList size is 0

CUDB Database Events internal notification error: object list is empty.

WARNING

No received response from Notification endpoint <endpoint>

No response from endpoint received.

WARNING

Notification endpoint invalid:<endpoint>

Invalid notification endpoint was specified.

WARNING

Notification EP (<endpoint>) is marked as unavailable

Endpoint is marked as unavailable due to too many connection errors.

WARNING

Notification FE connection error to <endpoint>

Connection error to endpoint.

WARNING

Notification FE SOAP other error at <endpoint>

Connection error to endpoint.

WARNING

Notification FEs not configured

The SOAP interface was unable to send notification as SOAP message, or accessing the configuration to get endpoints has failed, or no notification FEs are configured.

WARNING

Due to unavailable SOAP EPs notification message is discarded: <DN>

Notification message was not delivered to any WRR SOAP endpoints as all of them were unavailable.

WARNING

Due to the unavailable SOAP EP notification message is discarded: <DN>

Notification message was not delivered to the single configured WRR SOAP endpoint as it was unavailable.

WARNING

<error_message> <file_name> <number_line_file>

This generic message is returned mainly by CUDB checker when an error has occurred retrieving some objects from database.

WARNING

Client disconnected from Messaging Service

Notification process got disconnected from Messaging Service instance and it will try to reconnect.

WARNING

Client connection closed to Messaging Service

Notification process got disconnected from Messaging Service instance.

WARNING

Client reconnected to Messaging Service in <uri>!

Reconnection attempt to Messaging Service cluster was successful.

2.5.4.5   OAM

This section describes the subcomponents of the OAM component.

2.5.4.5.1   Software Management

Software Management component uses syslog as logging system with the following properties:

The messages logged by the software management subcomponent are summarized in Table 26.

Table 26    Software Management Logs

Severity

Message Information

Trigger Event

ERR

Incorrect number of parameters

There was a flag or parameter command unknown.

ERR

Error has occurred

An error happened in the lines above in the SQL sentences.

ERR

create::ERR: A backup name must be given, aborting

In the create procedure, the backup name parameter was wrong or it was not specified.

ERR

check_lock::ERROR: applyConfig action is ongoing, aborting.

Software and configuration backup/restore is started while applyConfig action is ongoing.

ERR

check_lock::ERROR: /home/cudb/oam/configMgmt/commands/config/cudbOiImmChanges.txt exists, aborting.

Software and configuration backup/restore is started while there are pending configuration changes that are not applied with applyConfig action.

ERR

check_lock::ERROR: Timeout while waiting OIWorker to write in cudbOiImmChanges.txt and generate configuration files.

Timeout expired while software and configuration backup/restore was waiting for commit of the configuration changes to release the lock.

ERR

check_lock::ERROR: OIWorker failed to generate configuration files.

Generation of the configuration files failed.

ERR

create::ERR: Backup lock file exists

The file to lock a backup procedure exists. This file avoids a new backup when there is another one running.

ERR

create::ERR: backup failed, could not pushd <root_path_backup>, failed

The path where all backups are going to be created does not exist.

ERR

create::ERR: backup failed, creation of cudb backup location, mkdir <backup_file_location> failed

The path where the backup is going to be stored could not be created.

ERR

create::ERR: cudb complete backup <backup_id> already exists, aborting

A new backup with the same identifier as a previous one cannot be created.


The name given to the backup file (<backup_id>) already exists, therefore it cannot be created

ERR

create::ERR: Incomplete cudb backup named <backup_id> already exists, file <backup_file_location> exists, aborting

A file is blocked to avoid another backup process being launched at the same time.

ERR

create::ERR: configuration read failed, aborting

An error occurred in reading node configuration.

ERR

create::ERR: compress failed, some error, aborting

Compressing the files under /home/cudb failed.

ERR

restore::ERR: A backup name must be given, aborting

In the restore procedure, the backup name parameter is wrong or it is not specified.

ERR

restore ::ERR: backup failed, could not pushd <root_path_backup>, failed

The path where the backups are stored does not exist.

ERR

restore::ERR: cudb backup failed, cudb backup location directory does not exist, aborting

The path where the backup is going to be stored could not be created.

ERR

restore::ERR: cudb backup file <backup_id> does not exist, aborting

The backup to be restored does not exist.

ERR

restore::ERR: configuration read failed, aborting

An error occurred in reading node configuration.

ERR

restore::ERR: Node Id and HW type check fails, aborting

The current node ID or hardware type do not match with the backup.

ERR

restore::ERR: cudb sql backup file <backup_file>-cudbSmpConfig.sql does not exist, aborting

The SQL file related to the backup specified does not exist.

ERR

restore::ERR: CMW backup file <backup_file_tar> does not exist, aborting

The Core MW backup file does not exist.

ERR

create::ERR: A backup name must be given, aborting

In the create procedure the backup name parameter was wrong or it was not specified.

ERR

restore::ERR: decompress failed, some error, aborting

Decompressing the files under /home/cudb failed.

ERR

restore::ERR: mysql tables backup failed, aborting

An attempt to make a restore of the MySQL tables failed.

ERR

delete::ERR: A backup name must be given, aborting

The file name backup parameter is wrong or not specified.

ERR

IMM file not found <imm_config> . <vars>

The file .CudbOamModel_Instances_config_imm.xml is not in the required path: "/cluster/home/coremw_appdata/incoming/cudb-install-temp".

ERR

IMM file not found <imm_config_path_file> . <vars>

The file .CudbOamModel_Instances_config_imm.xml is not in the required path: "/cluster/home/coremw_appdata/incoming/cudb-install-temp".

ERR

CudbLocalNode::hwType not found in <imm_instances_file> . <vars>

There is no instance of hwType in the file CudbOamModel_Instances_config_imm.xml.

ERR

Unsupported HW type: "'+vars.hw_type+'".', vars <hw_type> . <vars>

The HwType set in the file CudbOamModel_Instances_config_imm.xml is not supported or it is wrong.

ERR

CMSMHT SDP handling tools RPM file not found in repository . <vars>

CSMHT rpm not found in the repository _

ERR

Unable to uncompress CSMHT SDP handling tools RPM file . <vars>

An error occurred trying to uncompress the CSMHT rpm _

ERR

Configuration file not found

cluster.conf file not found in /cluster/etc/.

ERR

Unable to uncompress CSMHT SDP handling tools RPM file . <vars>

An error occurred trying to uncompress the CSMHT rpm _

ERR

Reference <reference_file> file not found

cudbReference file not found under "home/coremw_appdata/incoming/cudb-schema-tools-install-temp".

ERR

Incorrect number of parameters

The number of parameters are wrong.

ERR

Package<tmp_path> / <sdp_name> file not found

sdp not found in the directory _

ERR

Not Supported. Install in all nodes necessary

The installation is not supported for a concrete node. Only Supported for all nodes.

ERR

Unable to retrieve installation order for package <sdp_id>

An error occurred trying to obtain the sdp information of the sdp.

ERR

Cannot schedule software backup, error executing immfind <imm_sentence>

Failure to schedule software backup due to an error while looking for the active SC blade or VM.

WARNING

restore::WARNING: failed to extract cudbReference file from tar

cudbReference file cannot be extracted from the tar file.

2.5.4.5.2   Configuration Management

Configuration Management component uses syslog as logging system with the following properties:

The messages logged by the Configuration Management subcomponent are summarized in Table 27.

Table 27    Configuration Management Logs

Severity

Message Information

Trigger Event

EMERG

Problem while updating applyConfigStatus in CudbOiWorker sigHandler

applyConfig was running when the active CudbOiWorker instance was terminated, and there was an issue while updating applyConfigStatus.

EMERG

Problem while updating updateUserInfoStatus in CudbOiWorker sigHandler

updateUserInfo was running when the active CudbOiWorker instance was terminated, and there was an issue while updating updateUserInfoStatus.

EMERG

CudbAdminOperationApplyConfig::executeAdminOperation - Problem while starting applyConfigThread. Result code was: <result_code>

There was an issue while trying to start the administrative operation applyConfig.

EMERG

CudbAdminOperationApplyConfig::cancelApplyConfig - Failed to update applyConfigStatus with cancellation information.

There was an issue while trying to update applyConfigStatus upon a cancellation request for applyConfig.

EMERG

CudbAdminOperationApplyConfig::cancelApplyConfig - There was a problem while canceling thread. pthread_cancel returned: <result_code>

There was an issue while trying to cancel the thread in which applyConfig was executed.

EMERG

CudbAsyncAdminOperationBase::updateStatus - Unable to find id=<id>

The id does not match any of the existing CudbAsyncActionProgress class ids.

EMERG

CUDBAPPLYCONFIG::signalHandler - Failed to update applyConfigStatus.

Error while updating applyConfigStatus values after received signal.

EMERG

CUDBAPPLYCONFIG::Main- Another instance seems to be running.

Another instance of the process is running.

EMERG

CUDBAPPLYCONFIG::Main- Failed to update applyConfigStatus.

Error while updating applyConfigStatus values when starting the cudbApplyConfig command.

EMERG

CUDBAPPLYCONFIG::Main- Failed to update applyConfigStatus after execution. Operation finished with: result = <result_string>

Error while updating applyConfigStatus after execution of the cudbApplyConfig command has finished.

EMERG

CUDBAPPLYCONFIG::getInfoFromCudbSystemConfig - Exception while parsing cudbSystem.xml: <error_string>

Error while parsing cudbSystem.xml.

EMERG

ImmConnector::init - Generation of initial config files is not allowed

Missing licenses for the requested initial configuration.

EMERG

ImmConnector::init - Missing license - Payload Blades/ Virtual Machines license for the number of installed payload blades

License for a requested number of blades is not loaded at initial configuration.

EMERG

ImmConnector::init - Missing license - <value_pack_name> value pack that contains <feature_name> feature

Missing license for <value_pack_name> value pack at initial configuration.

EMERG

ImmConnector::updateAttr - CommitError: Missing license - Payload Blades/ Virtual Machines license for the number of installed payload blades

Attempt to configure additional DSU when the license for requested number of blades is not loaded.

EMERG

  • ImmConnector::updateAttr - CommitError: Missing license - Deployment Flexibility value pack that contains Multiple geographic zones feature

  • ImmConnector::updateAttr - CommitError: Missing license - Advanced Network Protection value pack that contains Triple geographic redundancy feature

  • ImmConnector::createUpdateCudbLdapView - CommitError: Missing license - Applications Facilitator value pack that contains Ldap views feature

Configuring feature at runtime without the needed license for it.

EMERG

CudbAdminOperationUpdateUserInfo::executeAdminOperation - Failed to update status for updateUserInfo prior to execution.

Error while updating updateUserInfoStatus when starting administrative operation updateUserInfo.

EMERG

CudbAdminOperationApplyConfig::executeAdminOperation - Failed to update status for applyConfig prior to execution.

Error while updating applyConfigStatus when starting administrative operation applyConfig.

EMERG

CudbAdminOperationApplyConfig::executeAdminOperation -Precheck for file generation failed. File generation didn't finish successfully. Return console

There was a problem with configuration file generation after the last commit.

EMERG

CUDBAPPLYCONFIG::Main - Something went wrong with configuration file generation before command was invoked ERROR

There was a problem with configuration file generation after the last commit, but before command was invoked.

EMERG

CUDBAPPLYCONFIG::Main - applyConfig waited for configuration file generation to finish successfully, but it failed ERROR

applyConfig was started while configuration file generation was running and it waited for the generation to finish successfully, but it failed.

EMERG

CUDBAPPLYCONFIG::Main - cudbSwBackup is running and it is not possible to apply configuration

applyConfig was started while cudbSwBackup was running.

EMERG

CUDBAPPLYCONFIG::doProcess - Problem reading information from cudbSystem.xml configuration file. ERROR

There was a problem while reading cudbSystem.xml.

EMERG

CUDBAPPLYCONFIG::doProcess - Problem reading information from cluster.conf configuration file. ERROR

There was a problem while reading cluster.conf.

EMERG

SaImmOiCcbObjectCreateCallback::cudbSwBackup is running!

A configuration change was attempted while software backup was running.

EMERG

SaImmOiCcbObjectDeleteCallback::cudbSwBackup is running!

A configuration change was attempted while software backup was running.

EMERG

SaImmOiCcbObjectModifyCallback::cudbSwBackup is running!

A configuration change was attempted while software backup was running.

EMERG

SaImmOiCcbApplyCallback - Unable to get OiChanges lock

It was not possible to acquire the OiChanges lock after the commit was executed.

EMERG

EsaJobFilesGenerator::getConfigFileNames - PM job directory doesn't exist!

The directory containing the job files is inaccessible.

EMERG

EsaJobFilesGenerator::generate - failed to access local node object!

Error occurred when attempting to get configuration data from the data model.

EMERG

EsaJobFilesGenerator::modifyElements - bad XML format <detailed_message>!

One of the required attributes is missing in a job file.

EMERG

EsaJobFilesGenerator::modifyGranularityPeriod - Failed to parse document!

Unable to read the requested xml file.

EMERG

EsaJobFilesGenerator::modifyGranularityPeriod - Invalid document format!

A job file has bad xml format.

EMERG

Process::restartProcess - <process_name> is not up, no more retries.

After restart, process is not up.

EMERG

Process::restartProcess - <process_name> is not up on <host_ip>, no more retries.

After restart, process is not up on a specific host.

EMERG

Process::restartProcess - Restarting <process_name> was not successful.

Restarting process was not successfully executed.

EMERG

Process::restartProcessSingleInstance - <process_name> is not up, no more retries.

After restarting process running as a single instance, process is not up.

EMERG

Process::restartProcessSingleInstance - <process_name> is not up on <host_ip>, no more retries.

After restarting process running as a single instance, process is not up on a specific host.

EMERG

Process::restartProcessSingleInstance - Restarting <process_name> was not successful.

Restarting process running as a single instance was not successfully executed.

EMERG

Process::restartProcessSingleInstance - Starting <process_name> was not successful.

Starting process running as a single instance was not successfully executed.

EMERG

Process::restartProcessSingleInstance - Stopping <process_name> was not successful.

Stopping process running as a single instance was not successfully executed.

ERR

Error trying to apply dynamic iptables rules, error: [<error_code>]. Retrying!

Error executing iptables commands on eVIP Front-End Elements.

ERR

Error trying to set iptables startup command, error: [<error_code>]. Retrying!

Error setting the iptables commands to be executed on eVIP Front-End Element startup.

ERR

Error calling evipDtmExecuteIptablesCommand to [<alb_name>] [<ip_address>] eVip traffic address... Cause: [<error_code>]

Error calling method to execute iptables commands on eVIP Front-End Elements.

ERR

Error calling evipDtmSetIptablesStartCommands to [<alb_name>] [<ip_address>] eVip traffic address... Cause: [<error_code>]

Error calling method to set iptables commands that must be executed on eVIP Front-End Element startup.

ERR

CUDBAPPLYCONFIG::doProcess - Incorrect OI changes file. This indicates a major problem generating configuration files that shall be managed by Ericsson support.

The file with the changes of the CUDB configuration model is incorrect.

ERR

CUDBAPPLYCONFIG::doProcess - Encountered an error reading information from cudb system configuration file.

Error reading information from the CUDB system configuration file.

ERR

CUDBAPPLYCONFIG::doProcess - Encountered an error reading cluster.conf required values.

Error reading the cluster.conf file.

ERR

CUDBAPPLYCONFIG::doProcess - Encountered an error reading OI changes file.

Error reading the OI changes file.

ERR

CUDBAPPLYCONFIG::doProcess - Encountered an error handling OI changes file.

Error handling the OI changes file.

ERR

CUDBAPPLYCONFIG::doProcess - Encountered an error handling SQL configuration file.

Error handling the SQL configuration file.

ERR

CUDBAPPLYCONFIG::doProcess - Encountered an error handling process configuration file.

Error handling the process configuration file.

ERR

CUDBAPPLYCONFIG::doProcess - Encountered an error updating SQL schema files.

Error updating SQL schema files.

ERR

CUDBAPPLYCONFIG::doProcess - Encountered an error updating tablespace monitoring parameters.

Error updating the BLOB disk space monitoring parameters.

ERR

CUDBAPPLYCONFIG::doProcess - Encountered at least an error executing commands.

Error executing any command when the changes in the CUDB configuration model are applied.

ERR

CUDBAPPLYCONFIG::doProcess - Encountered at least an error restarting processes.

Error restarting any process when the changes in the CUDB configuration model are applied.

ERR

CUDBAPPLYCONFIG::doProcess - Encountered an error while executing command [commnad].

Error executing the indicated command.

ERR

CUDBAPPLYCONFIG::doProcess - Lock file [<lockFileName>] cannot be removed. Delete it manually.

Error trying to delete the lock file specified.

ERR

  • CUDBAPPLYCONFIG::<method_name> finished with ERRORS

  • CUDBAPPLYCONFIG::<method_name> cannot create fileRreader

Some of the actions carried out in the method specified to update the system with the changes inserted in the CUDB configuration data model have failed.

ERR

CUDBAPPLYCONFIG::updateRedundancyLevel - <rdnUniqueCudbLdapAccessInstance>_redundancyLevel attribute not found.

The rdn attribute specified is not found.

ERR

CUDBAPPLYCONFIG::updateRedundancyLevel - <rdnUniqueCudbLdapAccessInstance>_redundancyLevel attribute value [<value>] cannot be parsed to an integer.

The rdn attribute value specified cannot be parsed to an integer.

ERR

CUDBAPPLYCONFIG::updateClusterConfNetkworkInfo - Error reading cluster.conf fle. Error: <error_description>

Error reading the cluster.conf file. The error description is shown, too.

ERR

CudbApplyConfig::getChanges - Error reading changes file.

Error reading the changes file.

ERR

CUDBAPPLYCONFIG::getCommandsToLaunch - Modified parameter [<parameter>] not found in any operation instances table.

The parameter modified is not found in any operation instances table.

ERR

CUDBAPPLYCONFIG::getProcessesAffected - Object with Oid [<object_id>] not found calculating affected processes.

The object is not found, calculating the affected processes.

ERR

CUDBAPPLYCONFIG::getProcessesAffected - Rule for affected process [<process>] is missing values to be replaced in commands.

Rule for the process specified is missing, values have to be replaced in commands.

ERR

CUDBAPPLYCONFIG::lauchSQLQueriesToDs - Connection FAILED, skipping server [<host>:<port>], detailed information: <error_description>.

Connection to server has failed with the host and port specified. The error description is shown.

ERR

CUDBAPPLYCONFIG::launchSQLQueriesToConnectedHost - SQL statements [<sql_statements>] failed, detailed information: <error_description>.

SQL statements have failed. The error description is shown.

ERR

CUDBAPPLYCONFIG::launchCommands - System call failed, result code=[<error_code>]

System call failed. The result code is shown.

ERR

CUDBAPPLYCONFIG::launchCommands - Executed process was not terminated normally, result code=[<error_code>]

Process was not terminated normally. The result code is shown.

ERR

CUDBAPPLYCONFIG::launchCommands - Command failed, result code=[<error_code>]

Command failed. The result code is shown.

ERR

CUDBAPPLYCONFIG::insertTextInSqlFile - <type_operation>operation in file [<file_name>] failed. Cause: <error_description>.

The operation in file failed. The error description is shown.

ERR

CUDBAPPLYCONFIG::insertTextInSqlFile - Open file [<file_name>] for <type_operation> failed. Cause: <error_description>.

Opening the file has failed. The error description is shown.

ERR

  • CUDBAPPLYCONFIG::readFiles2Detect Error - unable to open files2Detect

  • CUDBAPPLYCONFIG::readFiles2Detect Error - cannot allocate oneEntry

Error handling files2Detect.

ERR

CUDBAPPLYCONFIG::tableSpaceMonitoringChanges Error - Failed to apply Table Space Monitor changes

Failed to apply BLOB disk space monitoring changes.

ERR

CUDBAPPLYCONFIG::Main -,Finished with error(s): <error_code>.

Error launching cudbApplyConfig process.

ERR

CUDBAPPLYCONFIG::Main - Note that OI changes file was not deleted, remove it manually if necessary.

The file with the changes to be applied to the CUDB configuration data model has not deleted.

ERR

  • FileGeneratorsExecutor::populateGeneratorList - Factory does not know config file generator [<file_generator>]

  • FileGeneratorsExecutor::earlyChecks - At least a check failed.

  • FileGeneratorsExecutor::generateConfigFiles - At least a file generator failed.

  • FileGeneratorsExecutor::generateConfigFiles - Failure inttializing information providers.

  • FileGeneratorsExecutor::generateConfigFiles - Failure building file generators list.

  • FileGeneratorsExecutor::doGeneration - Config file generator [<file_generator>} failed.

  • FileGeneratorsExecutor::doGeneration - Exception cause start: ------------------------ <error_message>

  • FileGeneratorsExecutor::doGeneration - Exception cause end ---------------------------

Errors in the generation of the configuration files using the data inserted in the CUDB configuration data model.

ERR

  • OiChanges::commitMark - Can't write file <path_file>

  • OiChanges::commitMark - Error at writing <mark> to file <path_file>

  • OiChanges::commit Error in stored process

  • OiChanges::moveFileToBackup - Failed to make a backup copy of OI changes file [<path_file>] to [<path_backup_file>]

  • OiChanges::checkMarks Error: two not expected consecutive <mark> in file <file_name>

  • OiChanges::checkMarks Error: unmatched matched <begin_mark>/<end_mark> in file <file_name>

  • OiChanges::readFile Can't read file <file_name>

Errors handling the file with the data changed in the CUDB configuration data model.

ERR

  • Process::execCommand - System call failed, result code=[<result_code>]

  • Process::execCommand - Command failed, result code=[<result_code>]

  • Process::execCommand - Executed process was not terminated normally, result code=[<result_code>]

Error executing some processes to update the system with the changes inserted in the CUDB configuration data model.

ERR

  • ImmConnector::ImmConnector - CRITICAL Error: Could not read or parse cudbOI.xml file. Cause: <error_code>

  • ImmConnector::ImmConnector - CRITICAL Error: Could not initialize database accessor

Errors in the initialization of process to handle the CUDB configuration data model.

ERR

  • Error: something went wrong validating files generation

  • Error: something went wrong generating config files

Error in the validation or creation of the configuration files.

ERR

Error: changes log file wasn't correctly written

There was an error updating the log file needed for the application of committed configuration changes.

ERR

CudbPlGroupOI::checkThreshold - memoryWarningThreshold must be lower than memoryFullThreshold

Failure when trying to set illegal threshold values.

ERR

Error: new value for numAssignedNodes is out of range for [hwType]

Failure when trying to set numAssignedNodes value which is out of range for that hwType.

ERR

  • ImmConnector::createUpdateCudbLdapView - CommitError: The attribute ldapViewName cannot be empty.

  • ImmConnector::createUpdateCudbLdapView - CommitError: The attribute ldapViewName cannot be shared among multiple CudbLdapView objects.

  • ImmConnector::createUpdateCudbLdapView - CommitError: The attribute ldapViewName cannot be shared among multiple CudbLdapView objects. Non-applied CudbLdapView object with same ldapViewName attribute already exists on the node.

  • ImmConnector:: createUpdateCudbExternalAuthServer - CommitError: cudbExternalAuthServerId must be set to 1

  • Ds Group Id must be greater than 0 or less than 256

  • ImmConnector:: createUpdateCudbProvisioningGateway - CommitError: ProvisioningGatewayConfigId must be set to 1

  • ImmConnector:: createUpdateCudbLogCertificates - CommitError: cudbLogCertificatesId must be set to 1

  • ImmConnector:: createUpdateCudbExternalLogServer - CommitError: cudbExternalLogServerId must be set to 1

  • ImmConnector:: createUpdateCudbLdapAccess - CommitError: ldap Access Id must be set to 1

  • ImmConnector:: createUpdateCudbRemoteNode - CommitError: remote Node Id must be greater than 0 and less than 256

  • ImmConnector:: createUpdateCudbRemoteNode - CommitError: two CUDB nodes can not have the same Id

  • ImmConnector:: createUpdateCudbRemotePl - CommitError: remote Pl Id must be set to 1

  • ImmConnector:: createUpdateCudbLocalDs - CommitError: local Ds Id must be greater than 0

  • ImmConnector:: createUpdateCudbLocalDs - CommitError: local Ds Id should be one more than the last one

  • ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbExternalLogMgmt

  • ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbExternalAuthMgmt

  • ImmConnector::updateAttr - CommitError: invalid value of attribute pgNodeIpAddresses in object CudbProvisioningGatewayConfig

  • ImmConnector::updateAttr - CommitError: invalid value of attribute reallocationBlockSize in class CudbSystem

  • ImmConnector::updateAttr - CommitError: trafficVIP, oamVIP, or cudbVIP attributes are equal between Local and Remote Node

  • ImmConnector::updateAttr - CommitError: The attribute cudbUserPassword can not be an empty string. In class cudbLdapUser

  • • ImmConnector::updateAttr - CommitError: Invalid value of attribute cudbUserGroup in class CudbLdapUser

  • • ImmConnector::updateAttr - CommitError: two Application Services can not have the same name

  • ImmConnector::updateAttr - CommitError: trafficVIP, oamVIP, or cudbVIP attributes does not have ip address format

  • ImmConnector::updateAttr - CommitError: The attribute overloadRejectionWeight must be set to one of the predefined values. In class cudbLdapUser

  • ImmConnector::updateAttr - CommitError: The attribute readModeInDS must be set to one of the predefined values 'MA', 'MP', 'LP' cudbLdapUser

  • ImmConnector::updateAttr - CommitError: The attribute readModeInPL must be set to one of the predefined values 'MA', 'MP', 'LP' cudbLdapUser

  • ImmConnector::updateAttr - CommitError: another LocalDs from the same LocalNode has the same group

  • ImmConnector::updateAttr - CommitError: two Ds nodes from the same group can not have the same priority

  • ImmConnector::updateAttr - CommitError: two local Ds nodes can not have the same dsGroupId

  • ImmConnector::updateAttr - CommitError: another RemoteDs from the same RemoteNode has the same group

  • ImmConnector::updateAttr - CommitError: the same dsGroup can not be present in two different zones

  • ImmConnector::updateAttr - CommitError: incorrect format of attribute smpMgrAddr in object CudbLocalNode

  • ImmConnector::updateAttr - CommitError: ip address format -> <ipaddress|hostname>[:<port>]

  • ImmConnector::updateAttr - CommitError: each masterReplicationChannel1Port/masterReplicationChannel2Port/accessPort for a CudbPlGroup/CudbDsGroup must be unique

  • ImmConnector::updateAttr - CommitError: there is no instance of CudbDsGroup object for that group

  • ImmConnector::updateAttr - CommitError: two remote Ds nodes from the same remoteNode can not have the same dsGroupId

  • ImmConnector::checkCudbLdapView - Cannot assign LDAP View to provisioning user!

  • ImmConnector::checkCudbLdapView - Cannot assign LDAP View to re-provisioning user!

  • ImmConnector::checkCudbLdapView - The attribute cudbLdapViewId cannot be assigned to non-exisiting view in cudbLdapUser

  • ImmConnector::updateAttr - CommitError: this default zone is not defined in any CudbLocalNode or CudbRemoteNode

  • ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbProvisioningGatewayConfig

  • ImmConnector::updateAttr - CommitError: there is no instance of CudbDsGroup object for that group

  • ImmConnector::updateAttr - CommitError: two Pl nodes can not have the same priority

Attempt to set up invalid attribute value.

ERR

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbExternalLogMgmt

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbExternalLogServer

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object cudbAutomaticMasterChange

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbPlGroup

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbDsGroup

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbLdapUser

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbRemoteNode

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbRemoteDs

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbLocalDs

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbAppService

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbLogCertificates

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbExternalAuthServer

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbLdapAccess

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbLocalNode

  • ImmConnector::updateAttr - CommitError: attributes cannot be modified for object cudbSystem

Attempt to modify restricted attributes.

ERR

ImmConnector::updateAttr - CommitError: attributes cannot be modified for object CudbExternalAuthMgmt

Attempt to enable CudbExternalAuthMgmt without server parameters set on CudbExternalAuthServer.

ERR

ImmConnector::updateAttr - CommitError: attributes do not have the correct format for object CudbLocalPl

Attempt to set up invalid value of the numAssignedNodes attribute.

ERR

  • ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbLogCertificates

  • ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbSystemSecurity

  • ImmConnector::updateAttr - CommitError: some Application Service files do not exist

Attempt to set up value for attribute without the corresponding file stored.

ERR

ImmConnector::updateAttr - CommitError: invalid value of attribute ldapAttrIndexes in object cudbLdapAccess

Attempt to delete existing LDAP indexes, or set up a new index that does not exist as column in the corresponding PLDB table or any user database table.

ERR

ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbExternalAuthServer

Attempt to set up invalid value of the attribute primaryServer or secondaryServer for CudbExternalAuthServer.

ERR

ImmConnector::updateAttr - CommitError: invalid value of attributes in object CudbExternalLogServer

Attempt to set up value of the attribute externalLogServerIp without strict IP address syntax.

ERR

ImmConnector:: createUpdateCudbLocalDs - CommitError: local Ds Id value not supported

Attempt to set up value for cudbLocalDsId that is out of range for the configured HW.

ERR

Number of requested blades = <number_of_blades>, number of granted blades = <blade_capacity>. Requested blade capacity was not granted.

Attempt to configure additional DSU when the license for the requested number of blades is not loaded.

ERR

  • Number of requested blades = <number_of_blades>, number of granted blades = <blade_capacity>. Requested blade capacity was not granted. License manager is in autonomous operational mode, if Payload Blades/ Virtual Machines license has been updated while in autonomous mode it couldn't be read.

  • Payload Blades/ Virtual Machines license could not be retrieved. License manager is in autonomous operational mode, if new licenses have been installed they couldn't be read

Attempt to configure additional DSU while in Autonomous Mode.

ERR

Payload Blades/ Virtual Machines license could not be retrieved.

An error occurred while requesting payload blades or VMs license from License Manager.

ERR

Payload Blades/ Virtual Machines license is rejected for the following operational mode: <operational_mode>

License Manager is in the operational state that enforces rejection of request for payload blades or VMs license.

ERR

Payload Blades/ Virtual Machines license is rejected for the license type: <license_type>

License Manager has license type that enforces rejection of request for payload blades or VMs license.

ERR

License Manager handler couldn't be initialized, can't check licenses

An error occurred while initializing License Manager handler.

ERR

Feature license with name <feature_name> could not be retrieved. License manager is in autonomous operational mode, if new licenses have been installed they couldn't be read

Attempt to feature license from License Manager while License Manager is in Autonomous Mode.

ERR

Feature license with name <feature_name> could not be retrieved.

An error occurred while requesting feature license.

ERR

Feature license with name <feature_name> is rejected for the following operational mode: <operational_mode>

License Manager is in the operational state that enforces rejection of request for feature license.

ERR

Requested feature license doesn't exist

Attempt to request non-existing feature license from License Manager.

ERR

Feature license with name <feature_name> is rejected for the license type: <license_type>

License Manager has license type that enforces rejection of request for feature license.

ERR

Wrong version of LM! Can't use License Manager API!

Wrong version of License Manager detected.

ERR

Application ID is invalid. Problem with License Manager.

Attempt to use API version not supported by License Manager.

ERR

Registration to License Manager failed!

An error occurred trying to register License Manager.

ERR

Could not find any license type!

None of the allowed license types are loaded.

ERR

  • CudbLdapCertificatesOI::checkAttrFormat tlsCertificateFile can't be set as an empty string.

  • CudbLdapCertificatesOI::checkAttrFormat tlsCertificateKeyFile can't be set as an empty string.

  • CudbSoapCertificatesOI::checkAttrFormat tlsCertificateFile can't be set as an empty string.

  • CudbSoapCertificatesOI::checkAttrFormat tlsCertificateKeyFile can't be set as an empty string.

  • CudbSystemSecurityOi::checkAttrFormat tlsCaCertificatesFile can't be set as an empty string.

Error trying to set a certificate-related attribute to an empty string value after it was already assigned.

ERR

  • Failed to raise an alarm. Diagnostic info: <diagnostic_information>

  • Failed to clear an alarm. Diagnostic info: <diagnostic_information>

Diagnostic info: <diagnostic_information>.

ERR

init - saImmOmInitialize returned error

Failed to initialize IMM Object Manager.

ERR

init - saImmOmAccessorInitialize returned error

Failed to initialize IMM Object Manager Accessor.

ERR

readFingerprint - error fingerprint could not be read or is empty!

Fingerprint cannot be read or it is empty.

ERR

input file not found <file>

updateUserInfo could not find input file.

ERR

Could not retrieve users and groups from database (ldapsearch command failed)<ldap_search_result>

ldapsearch error during updateUserInfo.

ERR

Failed. Unexpected error with immcfg -d <CudbLdapUser or CudbLdapUserGroup>

Error during updateUserInfo while trying to delete LDAP users and/or groups from configuration model.

ERR

Failed. Unexpected error with immcfg -f CreateLdapUsers.xml

Error during updateUserInfo while trying to add LDAP users and/or groups to configuration model.

ERR

User information updated, but there was a problem with config file generation in node <node_id>

Error occurred during the commit phase after deleting or inserting CudbLdapUser or CudbLdapUserGroup.

WARNING

CudbExternalLogMgmtOI::checkAttrMods - Attempt to enable CudbExternalLogMgmt without server parameters set on CudbExternalLogServer

Attempt to enable CudbExternalLogMgmt without server parameters set.

WARNING

CudbExternalAuthMgmtOI::checkAttrMods - Attempt to enable CudbExternalAuthMgmt without server parameters set on CudbExternalAuthServer

Attempt to enable CudbExternalAuthMgmt without server parameters set.

WARNING

FileGenerator::backupConfigFile Failed to make a backup copy of the config file [<path_config_file_name>] to [<path_backup_file_name>]. Cause: <error_code>

Failed trying to make a backup of the configuration file.

WARNING

init - saImmOmInitialize returned SA_AIS_ERR_TRY_AGAIN

Temporary problem initializing an IMM Object Manager.

WARNING

init - saImmOmAccessorInitialize returned SA_AIS_ERR_TRY_AGAIN

Temporary problem initializing an IMM Object Manager Accessor.

WARNING

readFingerprint - saImmOmAccessorGet_2 returned SA_AIS_ERR_TRY_AGAIN

Temporary problem reading fingerprint.

WARNING

CUDBAPPLYCONFIG::doProcess - OI changes file not found, if there were no changes to the IMM model this is OK.

The OI changes file is not found. This is correct when nothing changed in the CUDB configuration data model.

WARNING

CUDBAPPLYCONFIG::getCommandsToLaunch - Commands to run rule for [<final_className_ParamName>] is missing values to be replaced [<partil_results>].

Warning that a command to launch is missing.

WARNING

CudbAsyncActionProgressOI::updateStructAttributes - <attribute_name> doesn't match to any of CudbAsyncActionProgress attributes

An attribute was provided for update which does not match any known attribute in CudbAsyncActionProgress structure.

2.5.4.5.3   Fault Management

Fault management component uses syslog as logging system with the following properties:

The messages logged by the Fault management subcomponent are summarized in Table 28.

Table 28    Fault Management Logs

Severity

Message Information

Trigger Event

CRIT

No such file <configuration_system_file>

The CUDB system configuration file does not exist in the path specified.

CRIT

No such file <configuration_database_file>

The database configuration file does not exist in the path specified.

ERR

It is impossible to establish connection with PL mysql server <host_ip> <port>

Due to a wrong IP, port, or failure in the cluster, the process cannot connect to the database in the PL.

ERR

some mandatory argument has been missed ...

The parameter was wrong or it was not specified.

ERR

Fault in mysql query

Fault in the SQL sentence when trying to get the memory use from the system monitor tables.

ERR

Invalid DG group called by ESA: <counter>

The wanted DS does not exist.

WARNING

It has been impossible to get the counter value

The snmpget command did not get values for the OID related to the counter.

2.5.4.5.4   Performance Management

Performance management component uses syslog as logging system with the following properties:

The messages logged by the performance management subcomponent are summarized in Table 29 and Table 30.

Table 29    Performance Management Logs located on controller blades or VMs

Severity

Message Information

Trigger Event

WARNING

ESA PM agent did not start successfully in <ipEsaAddress>

The ESA PM Agent could not be reloaded successfully.

WARNING

Format of current and previous NDB data for <blade> does not match.

The NDB data retrieved from a blade has a faulty structure.

WARNING

Data retrieval from <blade> timed out. Taking data from the last interval as current.

Retrieving data from a blade took longer than expected. Data from the last interval will be used as current interval data.

WARNING

There was a problem parsing data for <blade>. Invalid data:<data>

Data retrieved from a blade has a different format than the one expected by the process.

ERR

Error cleaning up child processes. Exiting.

An error occurred when the process tried to clean up its child processes.

ERR

Unable to open file for writing! <error>

Opening the output file has failed. The error description is shown.

ERR

There was a problem reading data retrieved from <blade>. Error:<error>

An error occurred when trying to read data retrieved from a certain blade. Error description is shown.

ERR

An error occurred retrieving data from <blade> Error:<error>

An error occurred when trying to obtain data from a certain blade. Error description is shown.

ERR

DSG <dsg_id> not found in <config_file>

The DSG with the supplied id was not found in CUDB system configuration file.

ERR

Error occurred while executing <imm_command>

Error occurred when trying to obtain information from the configuration model.

Table 30    Performance Management Logs located on payload blades or VMs

Severity

Message Information

Trigger Event

ERR

Unable to determine pid of <process_name>. Pidof return code: <return_code>

An error occurred while attempting to determine the PID of the process.

ERR

Unable to determine bitmask for process with pid <process_pid>. Error: <return_code>

An error occurred while attempting to determine the bitmask of the process based on its PID.

ERR

Unable to read <config_ini> file.

Failed to read the database cluster configuration file.

ERR

Unable to read <system_config> file.

Failed to read the CUDB system configuration file.

ERR

Error occurred while retrieving process information <error>

An error occurred while trying to gather uptime information from a process. The error description is shown.

ERR

Error occurred while retrieving parent process information <error>

An error occurred while trying to gather parent information from a process. The error description is shown.

ERR

CudbLocalNode not found in <system_config>

An error occurred while attempting to read the CudbLocalNode attribute from the system configuration file.

2.5.4.5.5   Security

Security component uses syslog as logging system with the following properties:

Note:  
This section describes the operational logs for security components. For information about logs describing important security related events that occur in the node, see Section 2.6.

The messages logged by the security subcomponent are summarized in Table 31.

Table 31    Security Logs

Severity

Message Information

Trigger Event

ERR

Could not do persistent mode without all parameter.

cudbRemoteTrust parameter --persistent (-p) was used without parameter --all (-a).

ERR

Incorrect number of parameters.

cudbRemoteTrust was invoked with incorrect number of parameters. No more than 2 parameters should be used.

ERR

Only one instance of the script can be running on the node! Check /cluster/home/cudb/cudbRemoteTrust.lock if you think you shouldn't be seeing this message.

Another instance of cudbRemoteTrust was already executing on the node. Check both controllers for running cudbRemoteTrust processes and, if there aren't any, delete lockfile /cluster/home/cudb/cudbRemoteTrust.lock.

ERR

Cannot parse cluster config file.

cudbRemoteTrust was unable to parse cluster configuration file /cluster/etc/cluster.conf.

ERR

Cannot parse system config file.

cudbRemoteTrust was unable to parse system configuration file /cluster/home/cudb/common/config/cudbSystem.xml.

ERR

Unknown parameter.

The specified parameter is not supported by cudbRemoteTrust.

ERR

Kill signal caught. Exiting.

Kill signal was received by cudbRemoteTrust. Exiting.

ERR

Cannot read files /home/cudb/common/config/cudbLocalNodeId.txt and /home/cudb/common/config/cudbLocalNodeId.txt.bak.

cudbRemoteTrust was unable to read files containing cudbLocalNodeId.

ERR

Error at loading new SSHD config on blade <blade>

cudbRemoteTrust could not load new SSHD configuration on the blade.

ERR

Error at restarting SSH daemon on blade <blade>

There was a problem when cudbRemoteTrust restarted SSH daemon on the blade.

WARNING

Cannot copy known_hosts to blade <blade>.

There was a problem with cudbRemoteTrust when copying known_hosts file to a certain blade.

WARNING

Cannot copy id_dsa.pub to blade <blade>.

There was a problem with cudbRemoteTrust when copying id_dsa.pub file to a certain blade.

WARNING

Some trust file failed, not persistent trust file.

There was a problem with cudbRemoteTrust when copying one of the required trust files. Persistence mode will not be applied.

WARNING

Couldn't get SITE-VIP information from CUDB Node <node>

cudbRemoteTrust could not get SITE_VIP information from the node.

WARNING

Invalid SITE-VIP address detected at CUDB Node <node>: <IP address>

cudbRemoteTrust detected an IP address with invalid format.

2.5.4.6   Platform

As mentioned in Section 2.2, platform events are logged in certain files. Some of the logs, those considered of relevance, are also copied to the Security file (see Section 2.6 for further details).

2.5.4.6.1   LogHandler

The LogHandler component uses syslog as logging system with the following properties:

The relevant messages logged by the LogHandler component are summarized in Table 32.

Table 32    LogHandler Logs

Severity

Message Information

Trigger Event

ERR

Logrotate with archive rules returned '<rotate_err_code>'. Verify that the logs are being properly rotated.

There is some problem trying to rotate with archive rules.

ERR

Logrotate with live rules returned '<rotate_err_code>'. Verify that the logs are being properly rotated.

There is some problem trying to rotate with live rules.

ERR

Startup check file is invalid (unknown action '<action>'). Exiting.

The action specified is wrong or not supported.

2.5.4.6.2   SmartHWFaultReaction

Logs related to HW state checking are logged with the following parameters:

The messages logged by the SmartHWFaultReaction component are summarized in Table 33.

Table 33    SmartHWFaultReaction Logs

Severity

Message Information

Trigger Event

ERR

Failed to read necessary parameters.

Collecting the necessary system parameters failed so the process cannot run.

ERR

Error reading config file < configuration file >.

Error reading configuration file .

ERR

Invalid configuration: < error message >.

Configuration data change has been detected, but the data was invalid.

ERR

Touch command failed.

File System probe detected File System error.

ERR

Halting blade.

Halting the blade where hardware fault was detected.

ERR

Failed to halt blade.

Halting commands failed, faulty blade was not powered off successfully.

WARNING

Test limit exceeded, there are still threads in D state for excessive time. CHECK_INTERVAL: < CHECK_INTERVAL > s, D_state timeout: < D_state timeout > s.

Monitored process has been detected as being stuck in D state more than the configured timeout.

WARNING

Setting up default values.

Default parameters are applied due to invalid configuration data.

WARNING

Touch command did not complete until timeout.

File System probe timeout has been exceeded due to missing response.

2.5.4.7   Storage

This section describes the subcomponent of the storage component.

2.5.4.7.1   Engine

The storage engine uses syslog as logging system, with the following properties:

The messages logged by the storage engine subcomponent are summarized in Table 34.

Table 34    Storage Engine Logs

Severity

Message Information

Trigger Event

ERR

Incorrect number of parameters

The process got a wrong number of parameters.

ERR

Unknown parameter

There was a flag or parameter command unknown.

ERR

Unable to increase the object class space in cluster

Increasing the disk space assigned to one object class failed.

ERR

Unable to obtain data from cluster : <store_id> <cluster_type>

The process did not have access to the cluster through one of the two IPs to obtain the object classes that have table space in the cluster.

2.5.4.8   3PP

This section describes the subcomponent of the 3PP component.

2.5.4.8.1   MySQL

MySQL processes use syslog as logging system with the following properties:

MySQL Cluster Carrier Grade Edition (CGE) has its own log files, these logs are allocated in each ndb component and in the Controller Nodes where the mysql mgmt agent is running. Following are the directories used by mysql to store information:

For more details related to MySQL log messages, refer to MySQL Server LogsMySQL Server Logs, Reference [11].

2.5.4.8.2   BC Server

BC Server uses log4j as logging system with the following properties:

2.6   Security Log Events

The security events are logged with the following properties:

The messages logged in the security log file are summarized in Table 35 (refer to CUDB Security and Privacy Management, Reference [3] for further information on security events).

The logs from this table are written either by the Security component or by any other component or subcomponent. When written by another component, the event is logged also in the file where the component usually logs other events.

Table 35    Security Related Logs

Severity

Message Information

Trigger Event

Component / Subcomponent

ERR

TLS Certificate NOT modified for SOAP

SOAP certificate change was not applied because Notifications restart failed.

Notifications / SOAP

ERR

TLS Certificate NOT modified for CA

Certification Authority certificate change was not applied because Notifications restart failed

Platform

WARNING

Failed login for <user> from <source_IP>

User tries to log in with an incorrect password.

Platform

WARNING

Failed login. Account locked for <user>

User has exceeded the maximum number of login attempts.

Platform

WARNING

Invalid user <user> from <source_ip>

User tries to log in with an incorrect user name.

Platform

WARNING

Failed LDAP Bind Password <source_ip> <bind_dn>

An LDAP bind has failed because of a wrong user or password.

DataAccess / LDAP

3   Sending Logs to a Remote Server

The Centralized Security Event Logging function allows a CUDB node to send log information to an external logging server. The log information sent with the function are the following:

The required information regarding the external logging server must be introduced in the CUDB node using the CUDB Configuration Data Model. Refer to the procedure described in the Configuration Modification Procedure section of CUDB Node Configuration Data Model Description, Reference [4] for creating an instance of the CudbExternalLogServer class. Then, configure Centralized Security Event Logging, either with or without Transport Layer Security (TLS), by following the Configuring Secure Centralized Security Event Logging section of CUDB Security and Privacy Management, Reference [3]. The mentioned section also contains configuration examples for both cases.

Once configured, enable the function by setting the enabled attribute of the CudbExternalLogMgmt class to true, also as described in CUDB Node Configuration Data Model Description, Reference [4].

Consider the following when configuring the function:

Logs sent to the Remote Server have the following format:

<PRI>TIMESTAMP CUDB_NODE_ID-FROMHOST TAG MSG

Where:

Example:

<182>2017-11-07T10:14:07.974797+01:00 CUDB_82-SC_2_1 commandlog: [info] 7557 root@SC_2_1:/root 10.82.225.157: ["cluster reboot -n 1"]

Glossary

For the terms, definitions, acronyms and abbreviations used in this document, refer to CUDB Glossary of Terms and Acronyms, Reference [5].


Reference List

CUDB Documents
[1] CUDB Consistency Check.
[2] CUDB High Availability.
[3] CUDB Security and Privacy Management.
[4] CUDB Node Configuration Data Model Description.
[5] CUDB Glossary of Terms and Acronyms.
Other Ericsson Documents
[6] LDE Management Guide.
[7] ESA Setup and Configuration.
Other Documents and Online References
[8] The Syslog Protocol (RFC 5424) http://www.rfc-editor.org/rfc/rfc5424.txt.
[9] Transmission of Syslog Messages over UDP (RFC 5426) http://www.rfc-editor.org/rfc/rfc5426.txt.
[10] OpenLDAP http://www.openldap.org.
[11] MySQL Server Logs http://dev.mysql.com/doc/refman/5.5/en/server-logs.html.
[12] Date and Time on the Internet: Timestamps. IETF RFC 3339 https://www.ietf.org/rfc/rfc3339.txt.


Copyright

© Ericsson AB 2016–2018. All rights reserved. No part of this document may be reproduced in any form without the written permission of the copyright owner.

Disclaimer

The contents of this document are subject to revision without notice due to continued progress in methodology, design and manufacturing. Ericsson shall have no liability for any error or damage of any kind resulting from the use of this document.

Trademark List
All trademarks mentioned herein are the property of their respective owners. These are shown in the document Trademark Information.

    CUDB Node Logging Events