Contents

1   Overview

This User Guide describes how to configure and use Crash Dump and Console Log Collection Service (CDCLSv) for logging and crash dump handling in LEM. CDCLSv collects component-specific console logs when a Linux core dump is generated. The list of files and directories to be collected and the log file size and the number of kept files are configurable.

CDCLSv is a service that is used for the following tasks:

• Handling CDCLSv, CDSv, CLUSv, ESHSv, and SCSv component-specific console logs and rotate them when a configurable file size is reached. For further details, refer to LEM System Architecture Description, Reference [1].
• Collecting the crash dumps created by Linux for the Core MW components of LEM together with all the generated log files. Placing these dumps and log files in a time-stamped and compressed package.

2   Configuring CDCLSv

CDCLSv comes with a default configuration that, in general, suits the LEM services needs. If any configuration changes are to be done, refer to LEM Parameter Description.

2.1   Custom Packers

CDCLSv supports creating custom packers. This function is also used by the LEM and vDicos to collect logs when a LEM or vDicos process crashed and dump is generated.

The packers can be defined through the IMM with CdclsPacker and CdclsPackerOption classes, see Figure 1.

The main idea is that the user can configure a command in the packer object that is called periodically with the options that are defined in the packer option object. That user-defined script can determine what to do. For example, in vDicos the packer script checks some predefined directory where dump files can be created, and if new dump files are found, then it starts the packing procedure.

2.1.1   CdclsPacker Object

There are two types of the packers: master and slave. Only the master packer scripts are called periodically. The master can have slave packers that are called before the master is called. The master script is only called when all its slave scripts exited, see Table 1.

2.1.2   CdclsPackerOption Object

When a packer is executed by the CDCLSv the arguments come from these objects. This class is child of the CdclsPacker class, see Table 2.

2.1.3   Running a Packer

When the execution period expires of a packer, then the CdclsvDirector starts the packing procedure.

1. Determine the name of the packing activity: <packer_name>-<timestamp>. The name comes from the cdclsPk attribute of the CdclsPacker object, the time stamp is the current time when the activity starts.
2. Check if the packer has prerequisite packer. If so, then call that. The arguments from the packer option for the slave packer are part of the executed command. Important: the first two arguments are always -d <packing_activity_name>.
3. When all scripts belong to the prerequisite are exited, then the master script is called. The arguments from the packer option for the master packer are part of the executed command. Important: the first two arguments are always -d <packing_activity_name>.

There can be three ways how a packing activity ends in normal case:

1. Time-out in the slave packers.
2. Time-out in the master packer.
3. Master script ends in time.

When the activity does not conclude in time, that is described in the cdclsPkExecTimeout value of the packer object, an alarm is raised. The alarm stops the activity and kills the command if started the execution.

The master scripts exit the return code and are analyzed by the CdclsvDirector. Zero means that the script exited successfully and it did packing activity. In this case, if there was alarm raised before, it is cleared. On other return code, the alarm is untouched.

2.1.4   Running The Packer By The User

The user can also perform a packing activity by running the cdclsv-pack <packer> command. In this case an additional parameter is passed to the packer scripts: -force.

3   Logging

All the software components of the vDicos solution log, by using CDCLSv, into a dedicated directory on the service control nodes. This directory can be reached on each node in the cluster through the following link: " /opt/cdclsv/storage/log/". The log filenames, created under this directory, follow the next template:
<process_name>-<log_source><-instance_id><-timestamp>

Where:

• <process_name>: The name of the process or AMF component writing logs in the log file.
• <log_source>: The identity of the node, where the process or AMF component generating the log is running on (the source of the log file).
• <-instance_id>: An optional tag that appears if the running component instance number on the same node is more than one.
• <-timestamp>: This information is displayed only for rotated log files and represents the file creation time stamp. For a log file that has not yet been rotated, the <-timestamp> tag is blank (for example: <process_name>-<log_source>).

The <process_name> tag of a log file can take the following names in LEM:

• CDCLSv:
  • cdclsvdirector: Contains the output of the CdclsvDirector component.
  • cdclsvagent: Contains the output of the CdclsvAgent and CdclsvLogRotator components.
• CDSv:
  • cdsvdirector: Contains the output of the CdsvDirector component.
• CLUSv:
  • cludirector: Contains the output of the CluDirector component.
• ESHSv:
  • eshsv: Contains the output of the Eshsv component.
• SCSv:
  • scsvcollector: Contains the output of the ScsvCollector component.
  • scsvstreamer: Contains the output of the ScsvStreamer component.

The default configuration for these log files is the following:

• cdclsLrMaxLogFileSize: 52428800. The maximum log file size is 50 MB. If this size is exceeded, the file is rotated. That is, a new log file is created and the old one is closed and preserved until the number of preserved log files reach a certain threshold.
• cdclsLrMaxLogFileNr: 2. The maximum number of rotated files preserved by CDCLSv for a certain logger source.

To change both the number and size of log files for all directors, agents, vms, and so on, modify the attributes of log rotators in IMM, as shown in Example 1. In this example, one log file is set with a size of 30 MB for the CDSv director.

immlist cdclsLr=cdsvdirector,cdcls=CDCLSvSite
immcfg -a cdclsLrMaxLogFileNr=1 cdclsLr=cdsvdirector,cdcls=CDCLSvSite
immcfg -a cdclsLrMaxLogFileSize=31457280 cdclsLr=cdsvdirector,cdcls=CDCLSvSite
immlist cdclsLr=cdsvdirector,cdcls=CDCLSvSite

4   Crash Dumps

The LEM applications are Core MW processes. When a process of LEM crashes, a Linux core dump is generated by the Linux system. These dumps are generated in the /cluster/dumps directory.

4.1   Packer Objects for LEM

The CdclsPacker object tells the CDCLSv what to do with a crash file. The CDCLSv implementation comes with a default Packer task implementation that effectuates the following:

• Checks the apparition of a file or collection of files on the file system matching a predefined file type template. The template description is configurable through the CdclsPkOption child object of the packer.
• Upon fulfillment of theis trigger condition, for example, crash dump apparition, it collects the CDCLSv logs and LEM-related crash dumps. It then packs them into a time-stamped *.tgz file. Backtraces for the LEM-related crash dumps are generated as well.

The CdclsPacker objects for the LEM components are configured, as shown in the following example, where two packer objects are created at installation of the CDCLSv.

cdclsPk=vDicos,cdcls=CDCLSvSite

Name                               Type         Value(s)
==================================================================================
cdclsPkType                        SA_UINT32_T  1 (0x1)
cdclsPkTool                        SA_STRING_T  /opt/cdclsv/bin/cdclsv_packer.sh
cdclsPkPrereq                      SA_NAME_T    vDicosPrereq (12)
cdclsPkExecTimeout                 SA_UINT32_T  295 (0x127)
cdclsPkExecPeriod                  SA_UINT32_T  60 (0x3c)
cdclsPkExecMode                    SA_UINT32_T  1 (0x1)
cdclsPk                            SA_NAME_T    cdclsPk=vDicos (14)
SaImmAttrImplementerName           SA_STRING_T  CdclsPackerImplementer
SaImmAttrClassName                 SA_STRING_T  CdclsPacker
SaImmAttrAdminOwnerName            SA_STRING_T  <Empty>

cdclsPk=vDicosPrereq,cdcls=CDCLSvSite

Name                               Type         Value(s)
==================================================================================
cdclsPkType                        SA_UINT32_T  2 (0x2)
cdclsPkTool                        SA_STRING_T  /opt/cdclsv/bin/cdclsv_prereq_packer.sh
cdclsPkPrereq                      SA_NAME_T    <Empty>
cdclsPkExecTimeout                 SA_UINT32_T  295 (0x127)
cdclsPkExecPeriod                  SA_UINT32_T  60 (0x3c)
cdclsPkExecMode                    SA_UINT32_T  2 (0x2)
cdclsPk                            SA_NAME_T    cdclsPk=vDicosPrereq (20)
SaImmAttrImplementerName           SA_STRING_T  CdclsPackerImplementer
SaImmAttrClassName                 SA_STRING_T  CdclsPacker
SaImmAttrAdminOwnerName            SA_STRING_T  <Empty>

The first object is the master (cdclsPkExecMode attribute is one), and the second is the slave (cdclsPkExecMode attribute is two).

These predefined packers are also configurable through the CdclsPkOption objects. Each component has its own CdclsPkOption object to determine the core dump files that trigger a collection of a dump through the option attribute. This attribute is the argument of the cdclsPkTool command in the CdclsPacker object.

An example of the CDSv:

cdclsPkOption=cdsvOptions,cdclsPk=vDicos,cdcls=CDCLSvSite

Name                             Type         Value(s)
========================================================================
option                           SA_STRING_T  -w /cluster/dumps/Cdsv* -c /opt/cdsv/storage/*
cdclsPkOption                    SA_NAME_T    cdclsPkOption=cdsvOptions (25)
SaImmAttrImplementerName         SA_STRING_T  CdclsPackerOptionImplementer
SaImmAttrClassName               SA_STRING_T  CdclsPackerOption
SaImmAttrAdminOwnerName          SA_STRING_T  <Empty>

4.2   Collection

In every minute, the CDCLSv checks if a crash occurs (cdclsPkExecPeriod attribute of the packer object). If so, then it collects the logs in a time-stamped, compressed file in the /opt/cdclsv/storage/dumps/ directory. The cdclsv_packer.sh script collects the logs and dumps if a file appears with a name that matches at least one of the filters in the watch list (-w or -wi parameter). Before collection, it runs the slave packers, defined in the cdclsPrereq. The cdclsv_prereq_packer.sh script also matches the filenames. It differentiates the following two kinds of files:

The analyzed files and the output of the analysis is placed in the dump directory that gets compressed. The master packer also watches the file in this directory so it collects the analyzed files.

A manual way is available to trigger the collection of the logs on to a compressed package, as described in Section 5.

4.2.1   Structure of the Collected Compressed File

The content of the compressed file depends on the packer configuration by the CdclsPackerOption objects. By default, it contains the following information:

• /opt/cdclsv/storage/log/: Log of the LEM components.
• /var/log/<node_name>/messages: Messages files from all nodes.
• /var/log/<node_name>/messages.1: First rotated messages files from all nodes.
• /opt/cdsv/storage/: Cdsv configuration of the cluster.

4.3   Limiting the Number of Stored Dumps

The default value for the number of stored dumps is 50. To change this limit, set attribute -k, as shown in the following example.

immcfg -a option="-k 49 -w /cluster/dumps/Cdclsv* -wi /opt/cdclsv/storage/autocc/* \
-c /opt/cdclsv/storage/log -c /var/log/*/messages -c /var/log/*/messages.1 \
-c /cluster/storage/no-backup/coremw/var/log/saflog/vdicos/vdlog/*” \
cdclsPkOption=baseCdclsvOptions,cdclsPk=vDicos,cdcls=CDCLSvSite

Note:  

Setting this attribute to a too low value (less than 10) can be disadvantageous. It is possible that the logs of a trigger event that caused a chain of crashes later can be lost.

5   Commands

This section briefly describes the most useful commands to control the CDCLSv.

5.1   Log Related Commands

This section describes commands that are in relation with logs.

5.2   Packer Related Commands

This section describes commands that are in relation with packers:

The following commands are available from within the COM CLI:

Reference List