WEBVTT

0:00:04.200000 --> 0:00:08.880000
 Linux logging. So now that we have gotten
 an understanding, both theoretical

0:00:08.880000 --> 0:00:16.360000
 and practical of the Windows event logging
 process, we can turn our attention

0:00:16.360000 --> 0:00:22.620000
 to the other operating system under
 the category of endpoint logs, as

0:00:22.620000 --> 0:00:24.800000
 it were. That is Linux, right?

0:00:24.800000 --> 0:00:29.880000
 And Linux is very important because
 you may not use Linux on a day-to

0:00:29.880000 --> 0:00:37.300000
-day basis, but the organization you are
 protecting or defending will probably

0:00:37.300000 --> 0:00:42.980000
 have Linux servers, our Linux system
 set up for a plethora of users or

0:00:42.980000 --> 0:00:46.920000
 use cases. And as a result, it's very
 important to understand how Linux

0:00:46.920000 --> 0:00:49.360000
 handles logging.

0:00:49.360000 --> 0:00:53.020000
 So what is Linux logging?

0:00:53.020000 --> 0:00:58.340000
 Well, it's fairly similar to Windows
 event logging, Windows logging in

0:00:58.340000 --> 0:01:06.680000
 general, but there are a couple of differences
 or nuances which will become

0:01:06.680000 --> 0:01:10.800000
 apparent. So Linux logging refers to
 the process by which Linux-based

0:01:10.800000 --> 0:01:16.760000
 systems record events, messages,
 and system activity, right?

0:01:16.760000 --> 0:01:20.980000
 And key point here is that these logs are
 essential for system administration

0:01:20.980000 --> 0:01:25.340000
 troubleshooting, performance monitoring,
 and of course, security monitoring,

0:01:25.340000 --> 0:01:29.920000
 analysis as part of that.

0:01:29.920000 --> 0:01:34.420000
 And Linux relies on a combination
 of text-based log files.

0:01:34.420000 --> 0:01:40.980000
 So it doesn't have the spiffy format
 that you have with Windows, EVTX

0:01:40.980000 --> 0:01:46.460000
 in this case. It's a simple
 text-based log file.

0:01:46.460000 --> 0:01:48.740000
 You also have logging demons,
 which are much better.

0:01:48.740000 --> 0:01:54.900000
 Think of SystemD, which as a phenomenal,
 or I know I'm going to get a

0:01:54.900000 --> 0:01:59.900000
 lot of, there's not going to be a lot
 of love for me when I say that,

0:01:59.900000 --> 0:02:04.880000
 but SystemD handles logging quite well,
 as well as kernel-level messages

0:02:04.880000 --> 0:02:08.160000
 to capture and store logs.

0:02:08.160000 --> 0:02:12.100000
 So how does logging work on Linux?

0:02:12.100000 --> 0:02:17.640000
 Similar to Windows, it starts with
 event or log generation, right?

0:02:17.640000 --> 0:02:22.460000
 So whenever a system component application
 or user action occurs, an event

0:02:22.460000 --> 0:02:23.820000
 message is generated.

0:02:23.820000 --> 0:02:28.980000
 And these events or logs can
 be system-level messages.

0:02:28.980000 --> 0:02:34.460000
 So think of service start, stop, kernel
 messages, application output,

0:02:34.460000 --> 0:02:38.300000
 examples of these would be
 errors, access logs, etc.

0:02:38.300000 --> 0:02:41.280000
 And of course, security-related activity.


0:02:41.280000 --> 0:02:46.780000
 So think of login attempts from
 a classical point of view.

0:02:46.780000 --> 0:02:48.560000
 You then have the logging demons.

0:02:48.560000 --> 0:02:53.080000
 So this is the primary distinction
 now, between Windows and Linux.

0:02:53.080000 --> 0:02:58.440000
 And obviously so, but Linux handles it
 in a very unique way, because Linux

0:02:58.440000 --> 0:03:00.620000
 is indeed unique.

0:03:00.620000 --> 0:03:07.620000
 So the logging system on Linux or the
 demons, as it were, the logging

0:03:07.620000 --> 0:03:14.100000
 system routes and stores these events
 using login, logging demons or demons,

0:03:14.100000 --> 0:03:16.100000
 depending on how you pronounce it.

0:03:16.100000 --> 0:03:18.460000
 And these include the following.

0:03:18.460000 --> 0:03:21.180000
 Now, this will differ.

0:03:21.180000 --> 0:03:26.760000
 Or the logging demon will differ, depending
 on the distribution being

0:03:26.760000 --> 0:03:38.660000
 used, right? So, our sys log, this
 is the default demon or Debian.

0:03:38.660000 --> 0:03:42.400000
 I would say Debian-based, Debian
 and Debian-based distribution.

0:03:42.400000 --> 0:03:45.580000
 So Debian, Ubuntu, Swan and so forth.

0:03:45.580000 --> 0:03:50.920000
 You then have an improved version of
 sys log or sys log, which is sys

0:03:50.920000 --> 0:03:59.040000
 log NGE or next generation, which is
 really something that would, users

0:03:59.040000 --> 0:04:01.320000
 typically set up themselves.

0:04:01.320000 --> 0:04:06.300000
 It offers quite, it offers some additional
 features that you may find

0:04:06.300000 --> 0:04:12.700000
 useful. But we have the class of the
 classic journal D, which is on most

0:04:12.700000 --> 0:04:17.880000
 system D-based systems, which sort of
 encompasses Debian-based distros.

0:04:17.880000 --> 0:04:21.260000
 And I know I said that with regards to
 our sys log, I was thinking they're

0:04:21.260000 --> 0:04:32.340000
 historically. But any distribution
 that utilizes system D as its init

0:04:32.340000 --> 0:04:35.920000
 system will have journal D, right?

0:04:35.920000 --> 0:04:39.040000
 And what do these demons or demons do?

0:04:39.040000 --> 0:04:46.080000
 Well, they handle message formatting,
 priority or severity tagging or

0:04:46.080000 --> 0:04:52.700000
 labeling, routing to files, remote servers
 or other logging systems, etc.

0:04:52.700000 --> 0:04:54.780000
 So, storage and formatting.

0:04:54.780000 --> 0:05:02.920000
 I'll wear an in what format are the
 logs stored or saved as it were.

0:05:02.920000 --> 0:05:06.980000
 So, logs on Linux are stored as plain
 text files, typically under the

0:05:06.980000 --> 0:05:09.720000
 var log directory.

0:05:09.720000 --> 0:05:14.820000
 So, here are some examples of some common
 logs that you're likely to find

0:05:14.820000 --> 0:05:17.440000
 on all Linux systems.

0:05:17.440000 --> 0:05:21.920000
 You have the auth.log or
 auth log as it were.

0:05:21.920000 --> 0:05:25.200000
 And this contains authentication
 and authorization logs.

0:05:25.200000 --> 0:05:27.420000
 So, think log in attempts.

0:05:27.420000 --> 0:05:32.860000
 sys log, general system activity logs,
 the kernel log or kernel log, these,

0:05:32.860000 --> 0:05:34.440000
 you know, you find kernel messages.

0:05:34.440000 --> 0:05:41.560000
 So, very similar organization to that,
 off windows with a few differences.

0:05:41.560000 --> 0:05:43.580000
 And then of course, you have messages.

0:05:43.580000 --> 0:05:45.500000
 These are general system messages.

0:05:45.500000 --> 0:05:48.500000
 So, think non-system D.

0:05:48.500000 --> 0:05:50.900000
 So, how do you access these logs?

0:05:50.900000 --> 0:05:53.500000
 You know, do you have an event
 viewer or log viewer?

0:05:53.500000 --> 0:05:55.680000
 Well, you do. There are
 programs for that.

0:05:55.680000 --> 0:05:59.740000
 I don't want to sound like I'm anti
-Linux here, but windows by default

0:05:59.740000 --> 0:06:03.640000
 on every window system, the event viewer,
 the event viewer is pre-packaged

0:06:03.640000 --> 0:06:05.540000
 or pre-installed with it.

0:06:05.540000 --> 0:06:11.920000
 On Linux, you typically would rely on
 your standard terminal based text

0:06:11.920000 --> 0:06:18.980000
 utilities, text viewing and editing
 utilities like cat, less or tail.

0:06:18.980000 --> 0:06:24.740000
 As I said, there's log viewers for
 Linux, really great ones, you know,

0:06:24.740000 --> 0:06:26.100000
 with a GUI, etc.

0:06:26.100000 --> 0:06:31.120000
 But you need to install those ones
 manually or after the fact.

0:06:31.120000 --> 0:06:40.900000
 But you also have great utilities like
 if used correctly or you know how

0:06:40.900000 --> 0:06:47.020000
 to use them, pretty much defeat
 the filtering system on windows.

0:06:47.020000 --> 0:06:52.680000
 So, you can use grep or org to search
 and filter for what you're looking

0:06:52.680000 --> 0:06:57.520000
 for. And you can also utilize general
 control or journal CTL on system

0:06:57.520000 --> 0:07:00.740000
 D systems to query the system journal.

0:07:00.740000 --> 0:07:04.060000
 So, an example of this would be, you
 know, given you two examples, you

0:07:04.060000 --> 0:07:11.280000
 have using journal CTL with the options
 XE, and then using the tail utility.

0:07:11.280000 --> 0:07:14.620000
 So, tail file var log auth dot log.

0:07:14.620000 --> 0:07:19.480000
 So, you start at the logs at the bottom
 because remember, it's a text

0:07:19.480000 --> 0:07:22.520000
 file. So, you're not going to have
 the latest logs at the top.

0:07:22.520000 --> 0:07:24.580000
 You're going to have them
 at the bottom of the file.

0:07:24.580000 --> 0:07:28.260000
 Hence, the reason you use
 tail, tail pretty much.

0:07:28.260000 --> 0:07:33.260000
 Outputs allows you to view the file,
 the entries at the bottom of the

0:07:33.260000 --> 0:07:36.800000
 file, which in the case of Linux based
 on how they are saved or the file

0:07:36.800000 --> 0:07:41.300000
 format in which they're saved
 in will be at the bottom.

0:07:41.300000 --> 0:07:45.180000
 So, what are the types
 of Linux log files?

0:07:45.180000 --> 0:07:47.940000
 So, as I mentioned, you have
 authentication logs.

0:07:47.940000 --> 0:07:55.040000
 These are typically stored under var
 log auth log or var log secure.

0:07:55.040000 --> 0:08:03.140000
 These log files contain authentication
 attempts or login attempts, and

0:08:03.140000 --> 0:08:09.320000
 pseudo usage. So, invocation of
 pseudo is also logged here.

0:08:09.320000 --> 0:08:14.200000
 And it obviously goes without saying
 very, very useful for detecting brute

0:08:14.200000 --> 0:08:19.600000
 force attacks, privilege escalation,
 or pseudo usage as it were.

0:08:19.600000 --> 0:08:24.940000
 And obviously unauthorized logins,
 you know, so fairly straightforward

0:08:24.940000 --> 0:08:30.260000
 system logs, as I mentioned again,
 are typically under var log sys log

0:08:30.260000 --> 0:08:31.860000
 or var log messages.

0:08:31.860000 --> 0:08:36.320000
 They include general system and daemon
 activity, you know, very useful

0:08:36.320000 --> 0:08:40.520000
 for spotting service crashes,
 system errors, reboots, etc.

0:08:40.520000 --> 0:08:41.960000
 All that good stuff.

0:08:41.960000 --> 0:08:46.860000
 You then have your kernel logs, which
 are stored under var log kern.log.

0:08:46.860000 --> 0:08:50.560000
 These, this log captures messages
 from the Linux kernel.

0:08:50.560000 --> 0:08:55.040000
 So, think of hardware errors,
 driver issues, etc.

0:08:55.040000 --> 0:08:58.920000
 And then you have your application logs,
 which is, you know, are the most

0:08:58.920000 --> 0:09:01.680000
 common types of logs that you're going
 to be dealing with or analyzing

0:09:01.680000 --> 0:09:03.140000
 in the context of Linux.

0:09:03.140000 --> 0:09:10.120000
 Of course, I'm generalizing here, but
 given what Linux is used for, you

0:09:10.120000 --> 0:09:15.240000
 know, as a server to host services like
 Apache web server or MySQL database,

0:09:15.240000 --> 0:09:19.140000
 the application logs are
 very important, right?

0:09:19.140000 --> 0:09:23.880000
 So the locations will vary by the application,
 but typically they will

0:09:23.880000 --> 0:09:29.800000
 utilize the predefined convention on
 Linux or storing them under var log.

0:09:29.800000 --> 0:09:32.680000
 But they can also have custom parts.

0:09:32.680000 --> 0:09:36.500000
 So in the case of, you know, the var
 log directory, there's a further

0:09:36.500000 --> 0:09:41.760000
 categorization organization, and you'll
 typically see these programs or

0:09:41.760000 --> 0:09:47.460000
 services or using their own folder
 name for their own logs.

0:09:47.460000 --> 0:09:50.280000
 So under var log, you'll
 have a separation.

0:09:50.280000 --> 0:09:54.800000
 You'll see that the Apache logs are
 stored under the Apache 2 folder.

0:09:54.800000 --> 0:09:57.060000
 And then under that, you
 have access to log.

0:09:57.060000 --> 0:10:01.020000
 In the case of MySQL, var log, MySQL,
 you can pick a look at the error

0:10:01.020000 --> 0:10:03.260000
 log as an example.

0:10:03.260000 --> 0:10:07.240000
 Key point to note is that custom or
 third party applications may write

0:10:07.240000 --> 0:10:12.380000
 to their own log stored in, you know,
 in the location as defined by the

0:10:12.380000 --> 0:10:19.120000
 developer or developers of that program
 or service or application.

0:10:19.120000 --> 0:10:21.400000
 You then have the audit logs.

0:10:21.400000 --> 0:10:25.540000
 So the tool, in this case, would be,
 you know, a tool like ordered D.

0:10:25.540000 --> 0:10:30.040000
 So the Linux ordered demon, the location
 is again under var log, in this

0:10:30.040000 --> 0:10:33.940000
 case now under its own folder
 called audit and audit.log.

0:10:33.940000 --> 0:10:41.260000
 This track system calls file, you know,
 files being accessed, use activity

0:10:41.260000 --> 0:10:46.420000
 all at a granular level is generally
 used for regulatory compliance and

0:10:46.420000 --> 0:10:48.420000
 detailed forensics.

0:10:48.420000 --> 0:10:50.940000
 And then the system D journal,
 if applicable.

0:10:50.940000 --> 0:10:55.260000
 So the tool obviously is going to be
 journal D, the command is general

0:10:55.260000 --> 0:11:01.920000
 control. So I can, yeah, system D and
 I'm not going to get into that right

0:11:01.920000 --> 0:11:03.460000
 now, because it's hugely debated.

0:11:03.460000 --> 0:11:07.920000
 In any case, you know, this captures the
 kernel logs, service logs, authentication

0:11:07.920000 --> 0:11:09.960000
 events and more.

0:11:09.960000 --> 0:11:14.020000
 This time not in the standard text
 format, but in a centralized binary

0:11:14.020000 --> 0:11:17.000000
 journal, which is pretty cool.

0:11:17.000000 --> 0:11:20.940000
 So here have a reference, you know,
 Linux logging architecture, so you

0:11:20.940000 --> 0:11:24.080000
 can better understand how it works, because
 it can be a little bit confusing,

0:11:24.080000 --> 0:11:28.920000
 you know, that saying something in comparison
 to a monolithic operating

0:11:28.920000 --> 0:11:30.800000
 system like Windows.

0:11:30.800000 --> 0:11:36.060000
 But over here, you know, a very simple
 diagram is not as complicated as

0:11:36.060000 --> 0:11:39.780000
 it looks, but just showing you, you know,
 that you have your log messages,

0:11:39.780000 --> 0:11:44.600000
 and in this case, kernel messages, they're
 not going directly into syslog

0:11:44.600000 --> 0:11:48.520000
 ng, that's just showing that they're
 going into this layer where you have

0:11:48.520000 --> 0:11:53.980000
 the demons here, which can be any could
 be our syslog, it could be syslog

0:11:53.980000 --> 0:11:56.580000
 ng or journal D done really matter.

0:11:56.580000 --> 0:12:02.200000
 The bottom line is that they all then
 handle, you know, the, they both

0:12:02.200000 --> 0:12:07.720000
 then facilitate the process of, you
 know, saving or logging those log

0:12:07.720000 --> 0:12:13.420000
 as it were into, you know, the different
 respective log files, so syslog,

0:12:13.420000 --> 0:12:24.640000
 auth.log, etc. So you just wanted to,
 you know, clarify how you need to

0:12:24.640000 --> 0:12:31.060000
 play. So what are some use cases of,
 you know, Linux logs or Linux logging

0:12:31.060000 --> 0:12:33.460000
 in incident response?

0:12:33.460000 --> 0:12:38.440000
 You know, here I've sort of listed the
 logs that you look at or the relevant

0:12:38.440000 --> 0:12:43.280000
 logs that you take a look at or analyze
 when dealing with a, you know,

0:12:43.280000 --> 0:12:44.840000
 specific type of instance.

0:12:44.840000 --> 0:12:47.860000
 So in this case, in the case of brute
 force detection, you want to be

0:12:47.860000 --> 0:12:51.300000
 monitoring the auth.log file, right?

0:12:51.300000 --> 0:12:57.560000
 For malware, you typically want to, want
 to be monitoring syslog messages

0:12:57.560000 --> 0:13:03.360000
 and, you know, journal control, root
 access tracking, obviously auth.log

0:13:03.360000 --> 0:13:06.460000
 audit.log as well.

0:13:06.460000 --> 0:13:08.940000
 Natural movement, most likely SSH.

0:13:08.940000 --> 0:13:12.220000
 So you're going to be monitoring
 auth.log primarily.

0:13:12.220000 --> 0:13:14.960000
 Of course, this is not something
 you did here to.

0:13:14.960000 --> 0:13:16.960000
 This is just an example.

0:13:16.960000 --> 0:13:22.380000
 For data exfiltration, activity or
 scripts, you know, you can, you can

0:13:22.380000 --> 0:13:27.040000
 have custom script logs,
 process auditing, etc.

0:13:27.040000 --> 0:13:28.520000
 And so why does this matter?

0:13:28.520000 --> 0:13:33.320000
 So the reason this matters is because
 Linux provides, Linux logs provide

0:13:33.320000 --> 0:13:36.680000
 deep visibility into system
 behavior and user actions.

0:13:36.680000 --> 0:13:41.120000
 And these logs are critical, really
 critical, especially in the context

0:13:41.120000 --> 0:13:46.300000
 of Linux, or I should say specifically
 in this context, because they can

0:13:46.300000 --> 0:13:50.800000
 be used for forensics or, you know,
 audit trails and of course detection

0:13:50.800000 --> 0:13:55.520000
 of compromise. And the other great thing
 about Linux logs is they're fairly

0:13:55.520000 --> 0:13:59.380000
 lightweight and customizable just based
 on their format, the fact that

0:13:59.380000 --> 0:14:03.500000
 they're stored in a relatively
 centralized location.

0:14:03.500000 --> 0:14:09.120000
 But they can easily then subsequently
 be centralized for theme integration

0:14:09.120000 --> 0:14:12.560000
 or for shipping as it were.

0:14:12.560000 --> 0:14:18.540000
 And here I've also added a quick reference,
 you know, Linux log files,

0:14:18.540000 --> 0:14:20.220000
 just as a cheat sheet.

0:14:20.220000 --> 0:14:24.000000
 So pretty much summarizes
 where all the logs are.

0:14:24.000000 --> 0:14:26.980000
 So system logs, var log sys log.

0:14:26.980000 --> 0:14:33.040000
 So general system messages, this is typical
 of Ubuntu Debian based distros,

0:14:33.040000 --> 0:14:37.200000
 I should say, then general system messages,
 var log messages, system wide

0:14:37.200000 --> 0:14:43.160000
 logs. So think of red at enterprise Linux
 or CentOS, CentOS authentication

0:14:43.160000 --> 0:14:49.420000
 logs that's standard on all Linux distros
 be under var log auth dot log,

0:14:49.420000 --> 0:14:52.500000
 the secure log, this is
 an alternative, right?

0:14:52.500000 --> 0:14:56.420000
 So this is var log secure, it's
 an alternative to auth dot log.

0:14:56.420000 --> 0:15:02.420000
 This is sometimes the case in, so the
 purpose is auth and access logs,

0:15:02.420000 --> 0:15:08.860000
 but most likely on a red at enterprise
 Linux or CentOS kernel logs, these

0:15:08.860000 --> 0:15:14.180000
 will be var log kernel log audit logs,
 var log audit audit audit or log.

0:15:14.180000 --> 0:15:21.160000
 So detail system call, detail system
 call level security events boot logs.

0:15:21.160000 --> 0:15:24.820000
 So if you have interested in, you know,
 analyzing what happens when your

0:15:24.820000 --> 0:15:29.580000
 Linux system boots up, take a look at
 the var log boot or log, that needs

0:15:29.580000 --> 0:15:31.660000
 to be configured, I believe.

0:15:31.660000 --> 0:15:37.940000
 And then of course your cron logs for
 cron jobs, var, that if configured

0:15:37.940000 --> 0:15:44.080000
 as well. You then have D message,
 which is, you know, ring buffer.

0:15:44.080000 --> 0:15:48.720000
 So D message, you're dealing with,
 you know, the kernel here.

0:15:48.720000 --> 0:15:53.460000
 So kernel messages from current boot
 and then application logs, typically

0:15:53.460000 --> 0:15:55.840000
 var log and then the app name.

0:15:55.840000 --> 0:15:59.880000
 So an example here would be var
 log Apache to as I stated.

0:15:59.880000 --> 0:16:02.140000
 So hopefully that reference helps you.

0:16:02.140000 --> 0:16:06.380000
 Now I have some essential Linux log
 commands that you can use in your

0:16:06.380000 --> 0:16:13.920000
 analysis. So for example, using the
 less command line utility there and

0:16:13.920000 --> 0:16:19.840000
 then real time monitoring, you can use
 tail or general control searching

0:16:19.840000 --> 0:16:22.560000
 here. I've used an example of grep.

0:16:22.560000 --> 0:16:26.000000
 So you can, you know, if you're looking
 for failed authentication attempts,

0:16:26.000000 --> 0:16:30.340000
 grep, in this case, the string on filtering
 for is failed or pattern,

0:16:30.340000 --> 0:16:32.620000
 I should say, is failed password.

0:16:32.620000 --> 0:16:38.220000
 The file I'm searching for it in is
 var log, also log, or you can use

0:16:38.220000 --> 0:16:42.420000
 journal, controller, journal, CTL,
 and then pipe it to grep, and then

0:16:42.420000 --> 0:16:47.020000
 use grep to look for patterns like
 SS, SSH attempts, in this case, SSH

0:16:47.020000 --> 0:16:55.700000
 authentication attempts, SSH, these
 referring to the open SSH daemon.

0:16:55.700000 --> 0:16:59.380000
 And then you can, you know, for example,
 filter by time using general

0:16:59.380000 --> 0:17:04.000000
 control. So you can say general control,
 use the sense flag and then specify,

0:17:04.000000 --> 0:17:07.060000
 you know, the time frame.

0:17:07.060000 --> 0:17:11.600000
 So one hour ago, or you can be very specific
 and you can specify a starting

0:17:11.600000 --> 0:17:14.480000
 point and a stop point.

0:17:14.480000 --> 0:17:23.700000
 So general control since 2024, 0501
 until 2024, 0515, so 15th of May,

0:17:23.700000 --> 0:17:27.480000
 or first of May to 15th of May.

0:17:27.480000 --> 0:17:29.500000
 And then view the kernel messages.

0:17:29.500000 --> 0:17:32.960000
 Typically, I do it using the
 message and then less.

0:17:32.960000 --> 0:17:36.880000
 All right, so that brings us
 to the end of this video.

0:17:36.880000 --> 0:17:39.140000
 And with that being said,
 that's going to be it.

0:17:39.140000 --> 0:17:41.880000
 And I will be seeing you
 in the next video.