Contents

1   Overview

This User Guide describes how to configure and use Crash Dump and Console Log Collection Service (CDCLSv) for logging and crash dump handling in LEM. CDCLSv collects component-specific console logs when a Linux core dump is generated. The list of files and directories to be collected and the log file size and the number of kept files are configurable.

CDCLSv is a service that is used for the following tasks:

• Handling CDCLSv, CDSv, CLUSv, ESHSv, and SCSv component-specific console logs and rotate them when a configurable file size is reached. For further details, refer to LEM System Architecture Description, Reference [2].
• Collecting the crash dumps created by Linux for the Core MW components of LEM together with all the generated log files. Placing these dumps and log files in a time-stamped and compressed package.

2   Configuring CDCLSv

CDCLSv comes with a default configuration that, in general, suits the LEM services needs. If any configuration changes are to be done, refer to LEM Parameter Description.

2.1   Custom Packers

CDCLSv supports creating custom packers. This function is also used by the LEM and vDicos to collect logs when a LEM or vDicos process crashed and dump is generated.

The packers can be defined through the IMM with CdclsPacker and CdclsPackerOption classes, see Figure 1.

The main idea is that the user can configure a command in the packer object that is called periodically with the options that are defined in the packer option object. That user-defined script can determine what to do. For example, in vDicos the packer script checks some predefined directory where dump files can be created, and if new dump files are found, then it starts the packing procedure.

2.1.1   CdclsPacker Object

There are two types of the packers: master and slave. Only the master packer scripts are called periodically. The master can have slave packers that are called before the master is called. The master script is only called when all its slave scripts exited, see Table 1.

2.1.2   CdclsPackerOption Object

When a packer is executed by the CDCLSv, the arguments come from these objects. There is one special cdclsPkOption that is not used as an argument: if the cdclsPkOption is " dumpprefix", then the value given in the option is used as prefix of the name of the packing activity.

This class is child of the CdclsPacker class, see Table 2.

2.1.3   Running a Packer

When the execution period expires of a packer, then the CdclsvDirector starts the packing procedure.

1. Determine the name of the packing activity: <packer_name>-<timestamp>.

   The name comes from the option attribute of the CdclsPackerOption object, where the cdclsPkOption is " dumpprefix". If there is no such object, then the cdclsPk attribute of the CdclsPacker object is used as name. If manual packing was triggered by calling the cdclsv-pack script, then the caller can specify this name through an argument, see Section 5.3.

   The time stamp is the current time when the activity starts.

2. Check if the packer has prerequisite packer. If so, then call that. The arguments from the packer option for the slave packer are part of the executed command. Important: the first two arguments are always -d <packing_activity_name>.
3. When all scripts belong to the prerequisite are exited, then the master script is called. The arguments from the packer option for the master packer are part of the executed command. Important: the first two arguments are always -d <packing_activity_name>.

There can be three ways how a packing activity ends in normal case:

1. Time-out in the slave packers.
2. Time-out in the master packer.
3. Master script ends in time.

When the activity does not conclude in time, that is described in the cdclsPkExecTimeout value of the packer object, an alarm is raised. The alarm stops the activity and kills the command if started the execution.

The master scripts exit the return code and are analyzed by the CdclsvDirector. Zero means that the script exited successfully and it did packing activity. In this case, if there was alarm raised before, it is cleared. On other return code, the alarm is untouched.

2.1.4   Running The Packer By The User

The user can also perform a packing activity by running the cdclsv-pack <packer> command. In this case an additional parameter is passed to the packer scripts: -force.

3   Logging

All the software components of the vDicos solution log, by using CDCLSv, into a dedicated directory on the service control nodes. This directory can be reached on each node in the cluster through the following link: " /opt/cdclsv/storage/log/". The log filenames, created under this directory, follow the next template:
<process_name>-<log_source><-instance_id><-timestamp>

Where:

• <process_name>: The name of the process or AMF component writing logs in the log file.
• <log_source>: The identity of the node, where the process or AMF component generating the log is running on (the source of the log file).
• <-instance_id>: An optional tag that appears if the running component instance number on the same node is more than one.
• <-timestamp>: This information is displayed only for rotated log files and represents the file creation time stamp. For a log file that has not yet been rotated, the <-timestamp> tag is blank (for example: <process_name>-<log_source>).

The <process_name> tag of a log file can take the following names in LEM:

• CDCLSv:
  • cdclsvdirector: Contains the output of the CdclsvDirector component.
  • cdclsvagent: Contains the output of the CdclsvAgent and CdclsvLogRotator components.
• CDSv:
  • cdsvdirector: Contains the output of the CdsvDirector component.
• CLUSv:
  • cludirector: Contains the output of the CluDirector component.
• ESHSv:
  • eshsv: Contains the output of the Eshsv component.
• SCSv:
  • scsvcollector: Contains the output of the ScsvCollector component.
  • scsvstreamer: Contains the output of the ScsvStreamer component.

The default configuration for these log files is the following:

• cdclsLrMaxLogFileSize: 52428800. The maximum log file size is 50 MB. If this size is exceeded, the file is rotated. That is, a new log file is created and the old one is closed and preserved until the number of preserved log files reach a certain threshold.
• cdclsLrMaxLogFileNr: 2. The maximum number of rotated files preserved by CDCLSv for a certain logger source.

To change both the number and size of log files for all directors, agents, vms, and so on, modify the attributes of log rotators in IMM, as shown in Example 1. In this example, one log file is set with a size of 30 MB for the CDSv director.

immlist cdclsLr=cdsvdirector,cdcls=CDCLSvSite
immcfg -a cdclsLrMaxLogFileNr=1 cdclsLr=cdsvdirector,cdcls=CDCLSvSite
immcfg -a cdclsLrMaxLogFileSize=31457280 cdclsLr=cdsvdirector,cdcls=CDCLSvSite
immlist cdclsLr=cdsvdirector,cdcls=CDCLSvSite

3.1   Log Sizes

Though it is possible to configure both the number and size of log files, most of the system runs with the default values. On a fully installed system (with LEM, DBS, and vDicos), the number of logs can reach a high amount, and they can easily populate the available disk space.

To examine the space required for the logs, do the following:

1. Get the number of the nodes and blades (NODENUM). It is assumed that all components are running on these nodes.
2. Get all of the log rotators:

   immfind -c CdclsLogRotator

3. Go through all the log rotator objects and get the cdclsLrMaxLogFileSize and the cdclsLrMaxLogFileNr attributes.
4. Use the following formula to determine the maximum amount of space, in bytes, that the log rotator can log:

   cdclsLrMaxLogFileSize×(cdclsLrMaxLogFileNr+1)× NODENUM

   Summarize these numbers for all log rotators to get the maximum size of the logs.

Example 1 shows a script that can give an estimation on the following:

• The size of the logs without rotation (calculating only with the current log files).
• The maximum size of the logs, including the rotated files.
• If -c is used as argument, then the current log size (if available).

#!/bin/bash

MINSIZE=0
MAXSIZE=0
CURSIZE=0
PRINTCUR=0

if [ "$1" == "-c" ]
then
    PRINTCUR=1
fi

NODENUM=`immfind | grep ^safSu= | grep CdclsvAgent | wc -l`

for LR in `immfind -c CdclsLogRotator`
do
    LR_SIZE=`immlist ${LR} -a cdclsLrMaxLogFileSize | cut -d = -f 2-`
    LR_NUM=`immlist ${LR} -a cdclsLrMaxLogFileNr | cut -d = -f 2-`
    LR_MSG=`immlist ${LR} -a cdclsLrLogProperties | cut -d = -f 2-`
    if [ ${LR_MSG} != "msgwriter" ]
    then
        MINSIZE=$((MINSIZE + LR_SIZE))
        MAXSIZE=$((MAXSIZE + ( LR_SIZE * ( LR_NUM + 1 ) ) ))
        if [ $PRINTCUR -eq 1 ]
        then
            LR_FILES=`immlist ${LR} -a cdclsLrLogFile | cut -d = -f 2-`
            for FS in `du -sb  ${LR_FILES}* 2> /dev/null | cut -f1`
            do
                CURSIZE=$((CURSIZE + FS))
            done
        fi
    fi
done
echo "Max log size without rotation: ${MINSIZE} / node, ⇒
        sum: $((MINSIZE * NODENUM)) with ${NODENUM} nodes"
echo "Max log size:                  ${MAXSIZE} / node, ⇒
        sum: $((MAXSIZE * NODENUM)) with ${NODENUM} nodes"
if [ $PRINTCUR -eq 1 ]
then
echo "Current log size:              ${CURSIZE}"
fi

For more accurate calculation, determine the exact node number (NODENUM) for each log rotator object.

To keep the size of the logs within a reasonable limit, configure the cdclsSizeLimit. That parameter tries to keep the log size below the limit by removing the old rotated files. For more information, refer to LEM Parameter Description.

If the cdclsSizeLimit is set below the maximum log size without rotation, and if the size of all log files is near to cdclsLrMaxLogFileSize, and if there are no more rotated log files, then the size of the log can go over the limit without the file getting deleted. Only rotated files are deleted if the size of the log is above limit.

cdclsSizeLimit can be set below that value, because the components can log in different quantity. If one component produces larger logs and other components produce smaller logs (or none at all), then the rotated log files of the component with the larger logs get deleted. This way the size of the log files can be kept below the limit.

4   Crash Dumps

The LEM applications are Core MW processes. When a process of LEM crashes, a Linux core dump is generated by the Linux system. These dumps are generated in a dedicated directory, for example: /cluster/dumps or /storage/dumps.

4.1   Packer Objects for LEM

The CdclsPacker object tells the CDCLSv what to do with a crash file. The CDCLSv implementation comes with a default Packer task implementation that effectuates the following:

• Checks the apparition of a file or collection of files on the file system matching a predefined file type template. The template description is configurable through the CdclsPkOption child object of the packer.
• Upon fulfillment of the trigger condition, for example, crash dump apparition, it collects the CDCLSv logs. It then packs them into a time-stamped *.tgz file. Backtraces for the LEM-related crash dumps are generated as well.

  Additionally, if the collectCoreDumps parameter is enabled, then CDCLSv also collects and packs the LEM-related crash dumps, see section "collectCoreDumps" in LEM Parameter Description, Reference [1].

The CdclsPacker objects for the LEM components are configured, as shown in the following example, where two packer objects are created at installation of the CDCLSv.

cdclsPk=vDicos,cdcls=CDCLSvSite

Name                               Type         Value(s)
==================================================================================
cdclsPkType                        SA_UINT32_T  1 (0x1)
cdclsPkTool                        SA_STRING_T  /opt/cdclsv/bin/cdclsv_packer.sh
cdclsPkPrereq                      SA_NAME_T    vDicosPrereq (12)
cdclsPkExecTimeout                 SA_UINT32_T  295 (0x127)
cdclsPkExecPeriod                  SA_UINT32_T  60 (0x3c)
cdclsPkExecMode                    SA_UINT32_T  1 (0x1)
cdclsPk                            SA_NAME_T    cdclsPk=vDicos (14)
SaImmAttrImplementerName           SA_STRING_T  CdclsPackerImplementer
SaImmAttrClassName                 SA_STRING_T  CdclsPacker
SaImmAttrAdminOwnerName            SA_STRING_T  <Empty>

cdclsPk=vDicosPrereq,cdcls=CDCLSvSite

Name                               Type         Value(s)
==================================================================================
cdclsPkType                        SA_UINT32_T  2 (0x2)
cdclsPkTool                        SA_STRING_T  /opt/cdclsv/bin/cdclsv_prereq_packer.sh
cdclsPkPrereq                      SA_NAME_T    <Empty>
cdclsPkExecTimeout                 SA_UINT32_T  295 (0x127)
cdclsPkExecPeriod                  SA_UINT32_T  60 (0x3c)
cdclsPkExecMode                    SA_UINT32_T  2 (0x2)
cdclsPk                            SA_NAME_T    cdclsPk=vDicosPrereq (20)
SaImmAttrImplementerName           SA_STRING_T  CdclsPackerImplementer
SaImmAttrClassName                 SA_STRING_T  CdclsPacker
SaImmAttrAdminOwnerName            SA_STRING_T  <Empty>

The first object is the master (cdclsPkExecMode attribute is one), and the second is the slave (cdclsPkExecMode attribute is two).

These predefined packers are also configurable through the CdclsPkOption objects. Each component has its own CdclsPkOption object to determine the core dump files that trigger a collection of a dump through the option attribute. This attribute is the argument of the cdclsPkTool command in the CdclsPacker object.

An example of the CDSv:

cdclsPkOption=cdsvOptions,cdclsPk=vDicos,cdcls=CDCLSvSite

Name                             Type         Value(s)
========================================================================
option                           SA_STRING_T  -w /cluster/dumps/Cdsv* -c /opt/cdsv/storage/*
cdclsPkOption                    SA_NAME_T    cdclsPkOption=cdsvOptions (25)
SaImmAttrImplementerName         SA_STRING_T  CdclsPackerOptionImplementer
SaImmAttrClassName               SA_STRING_T  CdclsPackerOption
SaImmAttrAdminOwnerName          SA_STRING_T  <Empty>

4.2   Collection

In every minute, the CDCLSv checks if a crash occurs (cdclsPkExecPeriod attribute of the packer object). If so, then it collects the logs in a time-stamped, compressed file in the /opt/cdclsv/storage/dumps/ directory. The cdclsv_packer.sh script collects the logs and dumps if a file appears with a name that matches at least one of the filters in the watch list (-w or -wi parameter). Before collection, it runs the slave packers, defined in the cdclsPrereq. The cdclsv_prereq_packer.sh script also matches the filenames. It differentiates the following two kinds of files:

The analyzed files and the output of the analysis is placed in the dump directory that gets compressed. The master packer also watches the file in this directory so it collects the analyzed files.

A manual way is available to trigger the collection of the logs on to a compressed package, as described in Section 5.

4.2.1   Structure of the Collected Compressed File

The content of the compressed file depends on the packer configuration by the CdclsPackerOption objects. By default, it contains the following information:

• /opt/cdclsv/storage/log/: Log of the LEM components.
• /var/log/<node_name>/messages: Messages files from all nodes.
• /var/log/<node_name>/messages.1: First rotated messages files from all nodes.
• /opt/cdsv/storage/: Cdsv configuration of the cluster.

4.3   Limiting the Number of Stored Dumps

The default value for the number of stored dumps is 50. To change this limit, set attribute -k, as shown in the following example.

immcfg -a option="-k 49 -w /cluster/dumps/Cdclsv* -wi /opt/cdclsv/storage/autocc/* \
-c /opt/cdclsv/storage/log -c /var/log/*/messages -c /var/log/*/messages.1 \
-c /cluster/storage/no-backup/coremw/var/log/saflog/vdicos/vdlog/*” \
cdclsPkOption=baseCdclsvOptions,cdclsPk=vDicos,cdcls=CDCLSvSite

Setting this attribute to a too low value (less than 10) can be disadvantageous. It is possible that the logs of a trigger event that caused a chain of crashes later can be lost.

Note:  

This method is deprecated by the new CdclsConfigAttributecalled maxNumOfDumps.

5   Commands

This section briefly describes the most useful commands to control the CDCLSv.

5.1   CDCLSv Configuration Related Commands

This section describes commands that are in relation with CDCLSv configuration.

5.2   Log Related Commands

This section describes commands that are in relation with logs.

5.3   Packer Related Commands

This section describes commands that are in relation with packers:

The following commands are available from within the COM CLI:

Reference List