WEBVTT 0:00:03.560000 --> 0:00:09.200000 So, introduction to the ELK stack or the Elastic Stack, Elastic Security 0:00:09.200000 --> 0:00:14.180000 as it's called. So, now that we've gotten an introduction to SEEMS and 0:00:14.180000 --> 0:00:17.880000 we understand how they work, we're going to take a look or get an introduction 0:00:17.880000 --> 0:00:24.720000 to ELK. Or, as I said, it's commonly, I was commonly referred to as today, 0:00:24.720000 --> 0:00:27.180000 the Elastic Stack. 0:00:27.180000 --> 0:00:31.900000 And, you know, the reason why this is sort of important in the context 0:00:31.900000 --> 0:00:35.340000 of SEEMS is, you know, because this is the one you're most likely going 0:00:35.340000 --> 0:00:40.800000 to be interacting with as you get to grips with, you know, how to use 0:00:40.800000 --> 0:00:45.600000 a SEEMS. You know, as I mentioned, given the fact that ELK has been there 0:00:45.600000 --> 0:00:50.240000 for a while, a lot of the other SEEMS are built on ELK or at least some 0:00:50.240000 --> 0:00:53.260000 of the core components that make up this stack. 0:00:53.260000 --> 0:00:56.540000 But, you know, regardless of that, let's get started. 0:00:56.540000 --> 0:00:59.200000 There's quite a bit that I want to cover here that's very important. 0:00:59.200000 --> 0:01:03.260000 You may not think it's important to understand how the Elastic Stack or 0:01:03.260000 --> 0:01:05.180000 the ELK Stack works. 0:01:05.180000 --> 0:01:08.320000 But, hopefully, by the end of this video, that will be apparent. 0:01:08.320000 --> 0:01:14.580000 And, hopefully, the theoretical explanations I provide here will be very 0:01:14.580000 --> 0:01:22.100000 useful in the next video when we'll be actually using ELK in a lab environment 0:01:22.100000 --> 0:01:25.340000 with the objective of getting familiar with it and we'll be learning how 0:01:25.340000 --> 0:01:29.660000 to write specific searches, so on and so forth. 0:01:29.660000 --> 0:01:32.880000 So, what is the Elastic Stack or ELK? 0:01:32.880000 --> 0:01:38.140000 The ELK Stack is a powerful open source suite of tools used for searching, 0:01:38.140000 --> 0:01:43.320000 analyzing and visualizing large volumes of data, especially log data in 0:01:43.320000 --> 0:01:47.940000 real time. ELK is an abbreviation of the term Elastic Stack. 0:01:47.940000 --> 0:01:52.040000 More specifically, it is an abbreviation of the names of the components 0:01:52.040000 --> 0:01:57.680000 that make up the technology stack, namely Elasticsearch, Logstash and 0:01:57.680000 --> 0:02:03.660000 Kibana. So, E for Elasticsearch, EL for Logstash and K for Kibana. 0:02:03.660000 --> 0:02:06.180000 So, that's where ELK comes from. 0:02:06.180000 --> 0:02:12.080000 So, these tools or components, these three Elasticsearch, Elasticsearch, 0:02:12.080000 --> 0:02:15.920000 Logstash and Kibana are developed by the same company. 0:02:15.920000 --> 0:02:21.480000 So, Elastic and their website is elastic .co and are commonly used together 0:02:21.480000 --> 0:02:25.740000 to process and analyze massive amounts of data efficiently. 0:02:25.740000 --> 0:02:29.160000 The combination of these components is the reason it is referred to as 0:02:29.160000 --> 0:02:33.140000 a stack. So, when you hear people say ELK Stack and they don't say, you 0:02:33.140000 --> 0:02:37.600000 know, Qradar Stack or something like this, if you're familiar with what 0:02:37.600000 --> 0:02:40.740000 a technology stack is, you know, you're referring to, you know, multiple 0:02:40.740000 --> 0:02:44.860000 pieces of technology or components that are working together to deliver 0:02:44.860000 --> 0:02:49.220000 something. In this case, it just so happens that the three are, you know, 0:02:49.220000 --> 0:02:51.800000 developed by the same company. 0:02:51.800000 --> 0:02:57.300000 And you may also be asking yourself, where does the term Elastic Stack 0:02:57.300000 --> 0:03:01.860000 come from? Well, that was really a modern sort of evolution of the name 0:03:01.860000 --> 0:03:09.700000 or, you know, it went from ELK so today or colloquially, it's you typically 0:03:09.700000 --> 0:03:13.900000 hear these two used interchangeably, ELK and Elastic. 0:03:13.900000 --> 0:03:16.600000 Elastic Stack is more accurate and up to date. 0:03:16.600000 --> 0:03:22.540000 That's because the stack as a whole now goes beyond the three Elasticsearch, 0:03:22.540000 --> 0:03:23.940000 Logstash and Kibana. 0:03:23.940000 --> 0:03:27.860000 And it now includes additional tools that I mentioned earlier on in this 0:03:27.860000 --> 0:03:33.860000 course, like Beats and the Elastic Agent, which are used to collection 0:03:33.860000 --> 0:03:37.060000 and of course monitoring capabilities. 0:03:37.060000 --> 0:03:40.080000 So why was ELK created? 0:03:40.080000 --> 0:03:44.060000 Well, in modern IT environments, systems, you know, it goes without saying 0:03:44.060000 --> 0:03:49.960000 generate huge amounts or volumes of logs from web servers, applications, 0:03:49.960000 --> 0:03:54.540000 operating systems, security tools, examples of which are firewalls, antiviruses, 0:03:54.540000 --> 0:03:58.740000 etc. As a result of that, you know, managing searching and making sense 0:03:58.740000 --> 0:04:01.480000 of this data manually is nearly impossible. 0:04:01.480000 --> 0:04:07.040000 And so ELK was created to centralize log data from multiple systems, index 0:04:07.040000 --> 0:04:11.640000 and search through data quickly or provide that ability to do so, visualize 0:04:11.640000 --> 0:04:16.380000 trends, patterns and anomalies and enable real time monitoring and alerting. 0:04:16.380000 --> 0:04:20.420000 So the only reason why I'm telling you this, because this seems like a 0:04:20.420000 --> 0:04:29.140000 quite a fundamental or rudimentary question is to give other problems 0:04:29.140000 --> 0:04:31.580000 that it's sought to solve. 0:04:31.580000 --> 0:04:35.080000 Okay, not to say that other solutions hadn't solved it, but you know, 0:04:35.080000 --> 0:04:39.500000 it's very important that you have a historical context as to where or 0:04:39.500000 --> 0:04:44.800000 why ELK was created and what problems it was trying to solve. 0:04:44.800000 --> 0:04:49.360000 So what is the core purpose of the ELK stack if we are to sort of, you 0:04:49.360000 --> 0:04:53.540000 know, try and summarize it or come up with a succinct explanation? 0:04:53.540000 --> 0:04:58.980000 Well, the best way to understand what the ELK stack is or what it's used 0:04:58.980000 --> 0:05:04.600000 for is to think of ELK as a data pipeline, which comprises of number one, 0:05:04.600000 --> 0:05:07.300000 ingesting logs from various sources. 0:05:07.300000 --> 0:05:12.640000 This is typically via log stash or beats, two, to store and index the 0:05:12.640000 --> 0:05:18.220000 data efficiently, the logs in this case, which is typically done using 0:05:18.220000 --> 0:05:23.760000 elastic search, and three, explore and data, which is done using kibana. 0:05:23.760000 --> 0:05:28.660000 So these three components that make up ELK or the ELK stack, each handle, 0:05:28.660000 --> 0:05:32.500000 you know, different aspects of the data pipeline or the logging pipeline 0:05:32.500000 --> 0:05:37.240000 from ingestion to exploration and visualization. 0:05:37.240000 --> 0:05:41.980000 So the flexibility and compatibility between the components that make 0:05:41.980000 --> 0:05:47.740000 up the stack makes ELK a goal, you know, a go to solution for log management 0:05:47.740000 --> 0:05:53.280000 security monitoring application performance monitoring APM as it is abbreviated 0:05:53.280000 --> 0:05:57.380000 as business intelligence, as well as operational analytics. 0:05:57.380000 --> 0:06:03.380000 Again, why am I pointing this out as well, you know, similar to Splunk, 0:06:03.380000 --> 0:06:07.120000 you know, where you have the Splunk enterprise security version, which 0:06:07.120000 --> 0:06:11.360000 is much more, which is pretty much customized or designed to be a seam 0:06:11.360000 --> 0:06:20.820000 in the you can use it not just for security monitoring, but it, you know, 0:06:20.820000 --> 0:06:24.400000 going beyond log management, you can also use it for application performance 0:06:24.400000 --> 0:06:29.440000 monitoring, business intelligence, as well as operational analytics, very, 0:06:29.440000 --> 0:06:33.920000 very similar to what the basic or standard version of vanilla version 0:06:33.920000 --> 0:06:35.520000 of Splunk allows you to do. 0:06:35.520000 --> 0:06:39.620000 So you know, you can pretty much get data from wherever you want from 0:06:39.620000 --> 0:06:46.060000 an API endpoint, you can also upload spreadsheets, you can connect databases. 0:06:46.060000 --> 0:06:51.020000 And so when we talk about ELK and Splunk, one of the reasons why they're 0:06:51.020000 --> 0:06:54.880000 so popular is not only because you can sort of build security on top of 0:06:54.880000 --> 0:06:59.660000 them or a security layer like what was or was who did, but because, you 0:06:59.660000 --> 0:07:03.400000 know, they can be used for, you know, they can be used to analyze or to 0:07:03.400000 --> 0:07:09.700000 collect aggregate and analyze, you know, different type of different types 0:07:09.700000 --> 0:07:11.620000 of data in different ways. 0:07:11.620000 --> 0:07:15.220000 It doesn't necessarily need to be geared towards security monitoring, 0:07:15.220000 --> 0:07:18.420000 for example, or just logs from an operating system. 0:07:18.420000 --> 0:07:22.380000 That's the point that I wanted to point out or to make very clear here. 0:07:22.380000 --> 0:07:26.500000 So that then leads us to the next logical question, which is, okay, given 0:07:26.500000 --> 0:07:31.420000 these multitude of users or use cases, where and how is ELK typically 0:07:31.420000 --> 0:07:33.680000 used in the real world? 0:07:33.680000 --> 0:07:37.560000 Well, you typically see it in enterprise IT departments for infrastructure 0:07:37.560000 --> 0:07:42.920000 monitoring or uptime, so on and so forth, DevOps teams for troubleshooting 0:07:42.920000 --> 0:07:47.520000 applications, obviously security teams and security operation centers 0:07:47.520000 --> 0:07:51.720000 for incident detection and response, and of course cloud environments 0:07:51.720000 --> 0:07:53.500000 for centralized log management. 0:07:53.500000 --> 0:07:55.520000 So it serves a lot of purposes. 0:07:55.520000 --> 0:07:59.420000 And this list pretty much represents what I've seen in the wild. 0:07:59.420000 --> 0:08:05.280000 You know, in real socks, having worked as a developer, as a system administrator, 0:08:05.280000 --> 0:08:10.540000 I remember using ELK for system monitoring or infrastructure monitoring 0:08:10.540000 --> 0:08:12.980000 quite extensively. 0:08:12.980000 --> 0:08:15.420000 In fact, it really was the go to solution. 0:08:15.420000 --> 0:08:21.680000 If you had different, if you had an S software as a service business, 0:08:21.680000 --> 0:08:25.100000 or you're running your own infrastructure and you have multiple deployments 0:08:25.100000 --> 0:08:32.600000 and so on and so forth, ELK was absolutely invaluable in being able to 0:08:32.600000 --> 0:08:37.160000 monitor exactly what's going on, on different deployments, ensuring that 0:08:37.160000 --> 0:08:42.420000 load is, you know, ensuring that your services or your web applications 0:08:42.420000 --> 0:08:48.560000 are operating correctly, identifying, you know, potential issues, you 0:08:48.560000 --> 0:08:55.060000 know, creating alerts for specific load thresholds or ratios, so on and 0:08:55.060000 --> 0:08:58.220000 so forth. So there's a lot that you can use ELK for. 0:08:58.220000 --> 0:09:02.280000 So, you know, I briefly mentioned or touched upon the core components 0:09:02.280000 --> 0:09:06.240000 of the ELK stack, which again are represented by the abbreviated form 0:09:06.240000 --> 0:09:09.900000 there of the actual stack. 0:09:09.900000 --> 0:09:15.520000 So elastic search, log stash and kibana, let's take a deeper or closer 0:09:15.520000 --> 0:09:18.960000 look at them because you need to understand exactly what each one of these 0:09:18.960000 --> 0:09:23.760000 is used for. And if you understand this using ELK or the ELK stack will 0:09:23.760000 --> 0:09:29.320000 be much easier. So the ELK stack is made up as I mentioned of three core 0:09:29.320000 --> 0:09:34.100000 components. Now, of course, we have others like beats, but these are elastic 0:09:34.100000 --> 0:09:36.540000 search log stash and kibana. 0:09:36.540000 --> 0:09:41.060000 As I said, I repeated myself three times now, but there's also additional 0:09:41.060000 --> 0:09:44.980000 tools like beats which are often integrated for enhanced log collection. 0:09:44.980000 --> 0:09:48.620000 What that means or why this is so important is because you don't need 0:09:48.620000 --> 0:09:53.500000 beats or, you know, which are described as lightweight data shippers or, 0:09:53.500000 --> 0:09:57.480000 you know, agents, if you will, and we already explored what they are and 0:09:57.480000 --> 0:09:58.580000 the beats protocol. 0:09:58.580000 --> 0:10:04.740000 We're talking about the log protocols and formats. 0:10:04.740000 --> 0:10:09.680000 But what you need to understand here is elastic search, which I've sort 0:10:09.680000 --> 0:10:11.360000 of summarized in this image. 0:10:11.360000 --> 0:10:14.300000 So search and analytics engine log stash. 0:10:14.300000 --> 0:10:16.840000 This is the data processing pipeline. 0:10:16.840000 --> 0:10:20.340000 Kibana is really what you describe as the visualization layer. 0:10:20.340000 --> 0:10:24.340000 So that's where you now start making sense of the logs that you have, 0:10:24.340000 --> 0:10:28.240000 you know, that are being collected. 0:10:28.240000 --> 0:10:31.840000 So let's start off with elastic search. 0:10:31.840000 --> 0:10:36.200000 So each component plays a specific role in the data log processing pipeline. 0:10:36.200000 --> 0:10:39.040000 And let's take a closer look at each of them. 0:10:39.040000 --> 0:10:44.020000 So for each of the components of sort of added an orange-ish card to the 0:10:44.020000 --> 0:10:48.240000 right that sort of explains its use or relevance to security operations 0:10:48.240000 --> 0:10:51.580000 generally, which you can sort of adapt to in response. 0:10:51.580000 --> 0:10:53.680000 In any case, elastic search. 0:10:53.680000 --> 0:10:57.380000 So elastic search is a distributed, restful search and analytics engine 0:10:57.380000 --> 0:11:02.360000 designed for horizontal scalability and real-time performance. 0:11:02.360000 --> 0:11:05.720000 And some of its key features are that, you know, it stores structured 0:11:05.720000 --> 0:11:07.100000 and unstructured data. 0:11:07.100000 --> 0:11:09.220000 So think of logs, metrics, text, etc. 0:11:09.220000 --> 0:11:14.040000 So very robust with regards to the type of data that it can store. 0:11:14.040000 --> 0:11:17.220000 B, it allows for near instantaneous search and filtering. 0:11:17.220000 --> 0:11:21.240000 As you'll see in the next video and will actually use elk. 0:11:21.240000 --> 0:11:26.420000 See, it's built on a patchy lucine, which offers powerful full-text search. 0:11:26.420000 --> 0:11:29.980000 Again, something that's not usually understood, but hopefully will become 0:11:29.980000 --> 0:11:31.840000 apparent in the next video. 0:11:31.840000 --> 0:11:36.180000 And D, highly scalable and fault tolerant, which I can actually speak 0:11:36.180000 --> 0:11:40.280000 to. In any case, what is its use or relevance in security operations? 0:11:40.280000 --> 0:11:44.740000 Well, it's used to index and retrieve logs quickly, enabling security 0:11:44.740000 --> 0:11:48.920000 analysts to search for compromises of compromise, for example, track, 0:11:48.920000 --> 0:11:53.460000 use activity and pivot across data sets rapidly, which again is very important 0:11:53.460000 --> 0:11:56.080000 when you talk about correlation. 0:11:56.080000 --> 0:12:00.060000 You then have log stash, right, which is fall intents and purposes, the 0:12:00.060000 --> 0:12:01.400000 data processing pipeline. 0:12:01.400000 --> 0:12:07.320000 So log stash is a server-side data ingestion tool that collects, transforms 0:12:07.320000 --> 0:12:10.200000 and sends data to elastic search. 0:12:10.200000 --> 0:12:14.660000 And the key features here are a, it supports multiple input sources or 0:12:14.660000 --> 0:12:15.840000 log sources, if you will. 0:12:15.840000 --> 0:12:21.720000 So think of syslog files, APIs, pretty much anything that's out there 0:12:21.720000 --> 0:12:27.320000 really. There's usually an extension or plugin that can allow for you 0:12:27.320000 --> 0:12:33.380000 to actually ingest that data into elk through log stash. 0:12:33.380000 --> 0:12:38.260000 Secondly, it can pass filter and enrich log data using rock patterns or 0:12:38.260000 --> 0:12:40.980000 conditionals, very, very powerful. 0:12:40.980000 --> 0:12:45.040000 And it outputs data to elastic search files or even message queues like 0:12:45.040000 --> 0:12:49.200000 Kafka. Something that's not really important may not be important to you, 0:12:49.200000 --> 0:12:53.180000 but let's sort of contextualize this in the context of security operations 0:12:53.180000 --> 0:12:57.400000 or its applicability or importance in security operations. 0:12:57.400000 --> 0:13:01.340000 Well, log stash is important because it helps normalize logs from different 0:13:01.340000 --> 0:13:05.980000 sources, extract important fields like usernames, IPs, processes, names, 0:13:05.980000 --> 0:13:10.680000 etc. And this is very important and sure clean searchable data in elastic 0:13:10.680000 --> 0:13:16.640000 search. So hopefully this is starting to contextualize why it's important 0:13:16.640000 --> 0:13:21.080000 to understand elastic search log stash and kibana, which we'll get to 0:13:21.080000 --> 0:13:27.680000 shortly. Because again, all of these three primary or core components 0:13:27.680000 --> 0:13:34.460000 that make up the elk stack are pretty much the features that we talked 0:13:34.460000 --> 0:13:38.160000 about when we were talking about the core features. 0:13:38.160000 --> 0:13:40.300000 Of a C, or how a C works. 0:13:40.300000 --> 0:13:47.780000 So, you know, if you want to be able to if you want to be able to normalize 0:13:47.780000 --> 0:13:51.600000 logs or get logs in a format that's understandable with key fields like 0:13:51.600000 --> 0:13:55.140000 usernames, IPs, processes, names, etc. 0:13:55.140000 --> 0:13:59.720000 Then in the context of the elk stack, you know, you you probably want 0:13:59.720000 --> 0:14:03.800000 to you probably will want to thank log stash for that. 0:14:03.800000 --> 0:14:09.240000 And the other important reason or the other reason why this is important 0:14:09.240000 --> 0:14:13.780000 is because when you understand what each of these three components does 0:14:13.780000 --> 0:14:18.260000 with regards to elk in general, but in the context of a seam, you're able 0:14:18.260000 --> 0:14:22.060000 then to fine tune each of these components to give you exactly what you 0:14:22.060000 --> 0:14:23.980000 want, how you want it. 0:14:23.980000 --> 0:14:25.480000 So on and so forth. 0:14:25.480000 --> 0:14:27.260000 So very, very important. 0:14:27.260000 --> 0:14:30.600000 And then kibana, which is the visualization layer, right? 0:14:30.600000 --> 0:14:33.560000 So kibana is fall in tens and purposes. 0:14:33.560000 --> 0:14:37.160000 The best way of saying it or the easiest way of understanding it is this 0:14:37.160000 --> 0:14:41.180000 is the front end interface that allows users to visualize data that is 0:14:41.180000 --> 0:14:43.460000 stored in Elasticsearch. 0:14:43.460000 --> 0:14:47.220000 And its key features are a, you know, the ability to create interactive 0:14:47.220000 --> 0:14:49.320000 dashboards and data visualization. 0:14:49.320000 --> 0:14:52.740000 So think of charts, graphs, tables, etc. 0:14:52.740000 --> 0:14:57.640000 It gives you the ability to leverage or to use kql, which is its very 0:14:57.640000 --> 0:14:59.220000 own query language. 0:14:59.220000 --> 0:15:04.020000 It's called the kibana query language, supremely powerful to filter and 0:15:04.020000 --> 0:15:05.480000 search for data. 0:15:05.480000 --> 0:15:09.940000 It also supports real time exploration of data, enables alerting detection 0:15:09.940000 --> 0:15:14.340000 rules and even timeline based investigations in Elastic Security. 0:15:14.340000 --> 0:15:18.280000 And kibana is arguably the most important in terms of you being able to 0:15:18.280000 --> 0:15:23.500000 use it effectively when it comes to when it comes down to instant response. 0:15:23.500000 --> 0:15:27.680000 And that's because of what I just mentioned in the final key feature, 0:15:27.680000 --> 0:15:33.220000 the ability to, you know, create alerts, detection rules, and more importantly, 0:15:33.220000 --> 0:15:39.060000 utilize, you know, the, the timeline functionality or those capabilities 0:15:39.060000 --> 0:15:44.160000 to, you know, essentially build a timeline or to refine the timeline that 0:15:44.160000 --> 0:15:46.060000 you've already sort of established. 0:15:46.060000 --> 0:15:51.140000 And so that leads into the use or the importance or relevance with regards 0:15:51.140000 --> 0:15:58.420000 to the security operations, but over here, you can see, you know, allows 0:15:58.420000 --> 0:16:02.680000 for allows you to monitor real time dashboards for suspicious activity. 0:16:02.680000 --> 0:16:07.280000 So whenever you see a SOC using the ELK stack, regardless of whether it's 0:16:07.280000 --> 0:16:12.660000 Wazoo or whatever, those dashboards are being facilitated by kibana, you 0:16:12.660000 --> 0:16:15.960000 know, the dashboards that, you know, sort of summarize everything that's 0:16:15.960000 --> 0:16:21.900000 going on. That's being performed or facilitated by kibana. 0:16:21.900000 --> 0:16:25.000000 It allows you to visualize trends in log data. 0:16:25.000000 --> 0:16:28.820000 So for example, you can, you know, whenever you see, if you look at a 0:16:28.820000 --> 0:16:34.460000 graph, you know, in kibana, one that's showing you authentication attempts, 0:16:34.460000 --> 0:16:41.080000 you know, using, you know, visualizations like a graph, you're easily 0:16:41.080000 --> 0:16:49.780000 able to pick up anomalies or suspicious activity by, you know, sort of 0:16:49.780000 --> 0:16:53.780000 entry examples, but very important for you to understand, you know, the 0:16:53.780000 --> 0:16:58.200000 power of the ELK stack, but more importantly, how to actually use it effectively. 0:16:58.200000 --> 0:17:03.260000 And finally, last but not least, it provides you with the ability to drill 0:17:03.260000 --> 0:17:06.420000 down into specific events for investigation. 0:17:06.420000 --> 0:17:09.620000 As we did to a certain extent with Plunk in the previous video, where 0:17:09.620000 --> 0:17:14.160000 we performed, or we wrote a search query, we got results and then we drilled 0:17:14.160000 --> 0:17:20.840000 down, you know, based on the results we got, we were able to drill down 0:17:20.840000 --> 0:17:26.100000 and identify the events associated with that particular, with the results 0:17:26.100000 --> 0:17:28.280000 of a particular search. 0:17:28.280000 --> 0:17:31.780000 That's a very basic example of a drill down, but there you go. 0:17:31.780000 --> 0:17:35.040000 And then finally, this is not really a core component, but I thought it'd 0:17:35.040000 --> 0:17:38.540000 be quite important for me to mention it, because I mentioned it prior, 0:17:38.540000 --> 0:17:42.360000 or previously early on in this course, and that is Beats, which is really 0:17:42.360000 --> 0:17:44.920000 an add on. These are lightweight data shippers. 0:17:44.920000 --> 0:17:48.420000 So Beats are lightweight agents or data shippers that are installed on 0:17:48.420000 --> 0:17:52.720000 endpoints to collect specific, that's the keyword here, specific types 0:17:52.720000 --> 0:17:56.180000 of data and forded to log stash or elastic search. 0:17:56.180000 --> 0:18:00.960000 Now, Beats is not a component in and of itself, it's referring to a data 0:18:00.960000 --> 0:18:07.200000 shipper of, you know, varying kind, which is why I'm not listing out the 0:18:07.200000 --> 0:18:10.580000 features here, but I'm listing out the popular and common beats that you 0:18:10.580000 --> 0:18:14.800000 have out there. So you have whenever you see the suffix beat associated 0:18:14.800000 --> 0:18:19.320000 with a particular shipper for the or agent, just know you're dealing with 0:18:19.320000 --> 0:18:23.400000 the elk stack. So when you see file beat, what is file beat used for? 0:18:23.400000 --> 0:18:25.180000 It's used to read log files. 0:18:25.180000 --> 0:18:28.840000 So think of things like Apache log system logs, etc. 0:18:28.840000 --> 0:18:31.320000 You then have the infamous win log beat. 0:18:31.320000 --> 0:18:34.020000 This is used to collect Windows event logs. 0:18:34.020000 --> 0:18:37.440000 So whenever you see win log beat installed on a Windows system, just know 0:18:37.440000 --> 0:18:41.260000 this is all going back to a nelk stack. 0:18:41.260000 --> 0:18:46.980000 And you then have packet beat some, you know, one, a type of beat that 0:18:46.980000 --> 0:18:50.160000 not a lot of people are familiar with, but this is used to capture network 0:18:50.160000 --> 0:18:54.200000 traffic and then metric beat, which is again, in the context of security, 0:18:54.200000 --> 0:18:57.540000 you're probably not familiar with it, but this is used to gather system 0:18:57.540000 --> 0:19:00.060000 metrics. So if you remember in the previous video, when we were setting 0:19:00.060000 --> 0:19:05.920000 up the Splunk Universal Ford on the Windows system, when we were specifying 0:19:05.920000 --> 0:19:09.740000 the types of logs that we wanted to forward to the Splunk server, you 0:19:09.740000 --> 0:19:15.140000 saw that on the right over there, you also had the ability to to forward 0:19:15.140000 --> 0:19:20.200000 system metrics or to gather system metrics with universal forda, it's 0:19:20.200000 --> 0:19:22.460000 sort of built into that agent. 0:19:22.460000 --> 0:19:27.120000 But the advantage you have with beats is that, you know, you set up the 0:19:27.120000 --> 0:19:30.880000 beats depending on the operating system, but also depending on the types 0:19:30.880000 --> 0:19:35.480000 of logs that you want to, you know, gather or to ship back to log stash, 0:19:35.480000 --> 0:19:41.520000 as it were. So what is the relevance of beats in security operations or 0:19:41.520000 --> 0:19:43.400000 to you as an instant responder? 0:19:43.400000 --> 0:19:47.300000 Well, beats are often used on endpoints and servers to ensure continuous 0:19:47.300000 --> 0:19:51.260000 and structured log forwarding from a wide range of sources. 0:19:51.260000 --> 0:19:56.760000 Now, I've sort of this whole first two section of the course, you may 0:19:56.760000 --> 0:20:00.400000 have been a little bit over funk as you know, asking, so how is this important 0:20:00.400000 --> 0:20:01.960000 to be as an instant responder? 0:20:01.960000 --> 0:20:06.140000 Well, if you're working in an organization that uses any one of these 0:20:06.140000 --> 0:20:10.400000 technologies, let's keep logging to the side, like, you know, a seem like 0:20:10.400000 --> 0:20:15.980000 Elk or Waza, Wazoo, Q radar, Splunk, whatever it is. 0:20:15.980000 --> 0:20:19.060000 The reason that's important to you as an instant responder is because 0:20:19.060000 --> 0:20:22.160000 you're going to need to interact with these systems, you're going to need 0:20:22.160000 --> 0:20:26.060000 to continuously improve detection or perform what I call detection engineering, 0:20:26.060000 --> 0:20:30.280000 which will actually touch on in this course, because what a lot of people 0:20:30.280000 --> 0:20:33.820000 forget to do if you remember in the previous course, when we're talking 0:20:33.820000 --> 0:20:38.380000 about the preparation phase, is after an incident, so the post incident 0:20:38.380000 --> 0:20:44.760000 activity, one of those key activities within post incident activity or 0:20:44.760000 --> 0:20:49.600000 key tasks is to continually update or improve detection. 0:20:49.600000 --> 0:20:55.380000 And you know, you find IOCs or, you know, as a result of analyzing an 0:20:55.380000 --> 0:20:58.360000 incident, you're able to learn a little bit more. 0:20:58.360000 --> 0:21:03.600000 You're based on that able to, you know, get these IOCs or generate IOCs 0:21:03.600000 --> 0:21:07.720000 and send them back to the SOC team or the detection engineering team for 0:21:07.720000 --> 0:21:11.300000 them to continuously improve detection so that that instant or that particular 0:21:11.300000 --> 0:21:14.260000 type of activity is detected much better. 0:21:14.260000 --> 0:21:17.800000 In most cases, you probably, you know, will need to have an understanding 0:21:17.800000 --> 0:21:24.000000 of how or of what systems, what detection or security monitoring systems, 0:21:24.000000 --> 0:21:28.060000 you know, your company's using or the SOC is using in order for you to 0:21:28.060000 --> 0:21:32.200000 generate IOCs that are fairly easy to integrate back into that pipeline. 0:21:32.200000 --> 0:21:37.020000 So the bottom line is your, you know, you may think that you know, you're 0:21:37.020000 --> 0:21:41.580000 not going to be dealing with the CIM, but you are not always not, you 0:21:41.580000 --> 0:21:44.380000 know, if you were a SOC analyst, it's not going to be to that extent, 0:21:44.380000 --> 0:21:49.900000 but you know, you're going to need to perform additional log analysis, 0:21:49.900000 --> 0:21:55.560000 develop your timeline, perform correlation, all of this comes back to 0:21:55.560000 --> 0:21:59.660000 the CIM. And you know, when it comes down to understanding, you know, 0:21:59.660000 --> 0:22:04.220000 things like the, like beats here, you may ask yourself, well, why exactly 0:22:04.220000 --> 0:22:05.660000 is that important to me? 0:22:05.660000 --> 0:22:09.240000 Well, again, remember, you're going to be interacting with the systems, 0:22:09.240000 --> 0:22:13.160000 let's say the organization that you work for as an incident responder 0:22:13.160000 --> 0:22:16.900000 uses the ELK stack and on all of the endpoints, let's say within a particular 0:22:16.900000 --> 0:22:22.180000 subnet, you know, you have windlog beats, so on and so forth, you need 0:22:22.180000 --> 0:22:27.060000 to understand how these beats pass and format their logs in order for 0:22:27.060000 --> 0:22:29.160000 you to be effective in analyzing them. 0:22:29.160000 --> 0:22:34.140000 There's sort of a lot of carryover or overlap between the responsibilities 0:22:34.140000 --> 0:22:38.020000 when it comes down to detection or the detection side of things. 0:22:38.020000 --> 0:22:42.860000 And from my perspective, I became a much better incident responder, or 0:22:42.860000 --> 0:22:46.740000 I was a much better incident responder because I had this background of 0:22:46.740000 --> 0:22:51.460000 having worked as a system administrator, having gone through the whole, 0:22:51.460000 --> 0:22:59.100000 you know, formalization of what you today call information security before 0:22:59.100000 --> 0:23:00.500000 we even had SOCs. 0:23:00.500000 --> 0:23:06.240000 And I went through the whole security engineering evolution in IT, detection 0:23:06.240000 --> 0:23:08.900000 engineering, all the way to the first seams. 0:23:08.900000 --> 0:23:16.560000 And so I sort of had that background, I knew how to harden systems, I 0:23:16.560000 --> 0:23:20.260000 knew what different logs look like, you know, Windows event logs. 0:23:20.260000 --> 0:23:24.740000 When you understand this, trust me, your job as an incident responder 0:23:24.740000 --> 0:23:29.960000 with regards to this specific phase detection and analysis becomes much 0:23:29.960000 --> 0:23:30.780000 more straightforward. 0:23:30.780000 --> 0:23:34.460000 Whereas you can imagine as an incident responder, if you're not familiar 0:23:34.460000 --> 0:23:38.800000 with how to use, let's say the most common of them all, the ELK stack 0:23:38.800000 --> 0:23:42.600000 to search for something, which we will be doing in the next video. 0:23:42.600000 --> 0:23:46.260000 And you'll see just how important it is. 0:23:46.260000 --> 0:23:49.900000 Imagine not being able to do it or being ineffective. 0:23:49.900000 --> 0:23:53.520000 You're actually causing more damage. 0:23:53.520000 --> 0:23:59.500000 When you're ineffective, remember an incident is ongoing or could be ongoing. 0:23:59.500000 --> 0:24:03.480000 Your ineffectiveness is actually causing damage. 0:24:03.480000 --> 0:24:08.600000 You need to be efficient, you need to be quick, and you can be daily daling. 0:24:08.600000 --> 0:24:12.460000 So understand the key outcome or the key point I'm trying to make with 0:24:12.460000 --> 0:24:20.080000 this first two or three sections of this course is please be efficient 0:24:20.080000 --> 0:24:24.820000 or proficient with the tools that you'll be interacting with or that are 0:24:24.820000 --> 0:24:29.640000 involved in your overall process, but whatever that may be. 0:24:29.640000 --> 0:24:37.420000 And again, this course can be consumed by an incident responder, a practitioner, 0:24:37.420000 --> 0:24:42.460000 a SOC analyst, a CISO, hopefully you're all finding value at different 0:24:42.460000 --> 0:24:45.660000 levels. In any case, I just wanted to point that out. 0:24:45.660000 --> 0:24:49.600000 And that sort of sets the stage for this section of this video, which 0:24:49.600000 --> 0:24:56.940000 is the ELK stack in a SOC, let's say, but if I'm speaking clearly, in 0:24:56.940000 --> 0:24:59.740000 security operations, generally speaking. 0:24:59.740000 --> 0:25:05.420000 So the ELK stack plays a central role in modern SOCs, its ability to ingest, 0:25:05.420000 --> 0:25:10.620000 store, analyze and visualize log data, makes it indispensable for monitoring 0:25:10.620000 --> 0:25:14.380000 and responding to cyber threats. 0:25:14.380000 --> 0:25:17.880000 So over here, I've sort of listed out the use cases with regards to security 0:25:17.880000 --> 0:25:23.220000 operations and sort of elaborated on those use cases because you need 0:25:23.220000 --> 0:25:27.560000 to understand just in the event, ELK is being used, you need to understand 0:25:27.560000 --> 0:25:30.760000 what, you know, just how much it can do. 0:25:30.760000 --> 0:25:34.800000 So in the case of threat detection, ELK enables the detection of malicious 0:25:34.800000 --> 0:25:38.860000 behaviors by passing and correlating logs across the environment. 0:25:38.860000 --> 0:25:42.780000 And analysts can create detection rules custom or using elastic security 0:25:42.780000 --> 0:25:45.740000 to flag suspicious activities. 0:25:45.740000 --> 0:25:49.260000 In the content, you know, when we're talking about instant response, when 0:25:49.260000 --> 0:25:54.360000 let's take a look at an example here, when an alert is triggered, responders 0:25:54.360000 --> 0:25:59.120000 can pivot quickly to search logs in elastic search via kibana. 0:25:59.120000 --> 0:26:04.300000 And ELK or the elastic stack allows for deep dive investigations, tracing 0:26:04.300000 --> 0:26:07.340000 the origin method and impact of attacks. 0:26:07.340000 --> 0:26:11.020000 You then have general security monitoring or what the SOC does, right? 0:26:11.020000 --> 0:26:15.360000 So real time dashboards track critical security metrics like failed logins, 0:26:15.360000 --> 0:26:20.120000 abnormal outbound traffic or changes in file integrity, which is quite 0:26:20.120000 --> 0:26:24.560000 important. And alert thresholds can be set to escalate high risk events 0:26:24.560000 --> 0:26:27.800000 automatically. You then have threat hunting, right? 0:26:27.800000 --> 0:26:31.300000 So analysts this is based on what I've seen. 0:26:31.300000 --> 0:26:35.500000 Analysts use kibana search and filter capabilities to proactively look 0:26:35.500000 --> 0:26:38.840000 for anomalies or indicators of compromise. 0:26:38.840000 --> 0:26:44.320000 And they also utilize custom queries to help identify patterns that don't 0:26:44.320000 --> 0:26:45.920000 match known signatures. 0:26:45.920000 --> 0:26:48.360000 So ELK is extremely powerful. 0:26:48.360000 --> 0:26:55.840000 And in my opinion, you should be at least fairly comfortable with using 0:26:55.840000 --> 0:27:02.200000 kibana, you know, writing your own search query, so on and so forth. 0:27:02.200000 --> 0:27:04.420000 It's very important. 0:27:04.420000 --> 0:27:10.700000 So here I have an example of how ELK would be used, you know, in the very 0:27:10.700000 --> 0:27:15.200000 basic scenario or case where, you know, talking about detecting brute 0:27:15.200000 --> 0:27:17.160000 force login attempts. 0:27:17.160000 --> 0:27:21.140000 So here we have, you know, Windows event logs being sent into filebeat, 0:27:21.140000 --> 0:27:25.260000 log stash, elastic search, and then kibana, you have your dashboard. 0:27:25.260000 --> 0:27:29.640000 And someone has created a graph, you know, that essentially provides a 0:27:29.640000 --> 0:27:32.280000 visualization of authentication attempts. 0:27:32.280000 --> 0:27:36.980000 And in this case, there is a spike in the number fail login attempts within 0:27:36.980000 --> 0:27:40.140000 a specific time frame, right? 0:27:40.140000 --> 0:27:46.600000 And so we start with step one, file beat forwards Windows event logs to 0:27:46.600000 --> 0:27:48.840000 log stash, or it could be win log beat. 0:27:48.840000 --> 0:27:53.500000 If you remember, I explained what file beat here, file beat is when we're 0:27:53.500000 --> 0:27:58.780000 talking about the popular common beats, file beat reads log files. 0:27:58.780000 --> 0:28:03.320000 It's not really specific to Windows, but you know, you can use them interchangeably. 0:28:03.320000 --> 0:28:05.900000 On Windows, you will typically want to use win log beat. 0:28:05.900000 --> 0:28:09.560000 In any case, I'm sort of going on different tangents. 0:28:09.560000 --> 0:28:12.600000 So file beat forwards the Windows event logs to log stash. 0:28:12.600000 --> 0:28:17.020000 This is the Windows system that's experiencing the brute force attack, 0:28:17.020000 --> 0:28:20.780000 right? So what does log stash do? 0:28:20.780000 --> 0:28:26.200000 Log stash passes the event ID associated with a fail log on, which is 0:28:26.200000 --> 0:28:30.740000 4625. We covered that when we're talking about when we went through the 0:28:30.740000 --> 0:28:34.760000 demo of the Windows event logging as a process. 0:28:34.760000 --> 0:28:36.380000 And then what happens next? 0:28:36.380000 --> 0:28:40.180000 Well, we have a last elastic search, which I already explained what it 0:28:40.180000 --> 0:28:43.740000 does. And it pretty much stores the data or, you know, stores the logs 0:28:43.740000 --> 0:28:48.860000 in an index. Step four, kibana, the kibana dashboard visualizes spikes 0:28:48.860000 --> 0:28:50.660000 in fail login attempts. 0:28:50.660000 --> 0:28:54.940000 And then step five, the analyst sets an alert, or creates an alert that 0:28:54.940000 --> 0:28:59.980000 is triggered if one user or IP exceeds 10 failures in five minutes. 0:28:59.980000 --> 0:29:04.460000 So you can sort of see it end to end right from the logs being collected 0:29:04.460000 --> 0:29:09.620000 and aggregated all the way to how, you know, how they're stored, sorry, 0:29:09.620000 --> 0:29:14.880000 how they're passed, the key info that's important to you are indicator 0:29:14.880000 --> 0:29:17.280000 of, you know, malicious activity. 0:29:17.280000 --> 0:29:24.200000 And then in the context of kibana, you essentially, you're essentially 0:29:24.200000 --> 0:29:31.100000 using kibana to, to make sense of the data that you're gathering, right? 0:29:31.100000 --> 0:29:35.940000 And you know, when you start out, you don't really know what exactly you 0:29:35.940000 --> 0:29:37.120000 want to be monitoring. 0:29:37.120000 --> 0:29:40.520000 But as I said, you start with the very basic authentication attempts, 0:29:40.520000 --> 0:29:42.680000 you create a visualization for it. 0:29:42.680000 --> 0:29:45.740000 The advantage that you have with visualizations, like in the form of a 0:29:45.740000 --> 0:29:50.260000 graph, is you're, you know, it's much easier for a human being to understand, 0:29:50.260000 --> 0:29:53.380000 you know, what you're dealing with in terms of severity. 0:29:53.380000 --> 0:29:57.940000 So if you look at a spike that is highly anomalous over, let's say, a 0:29:57.940000 --> 0:30:01.220000 month or something, you know, you're dealing with something fairly interesting 0:30:01.220000 --> 0:30:06.180000 here. Now, over a year, it may, you know, it may be normal to experience 0:30:06.180000 --> 0:30:17.420000 brute force attacks every now and then, all the features that it offers. 0:30:17.420000 --> 0:30:24.200000 And it can be extended or improved upon or augmented through various plugins, 0:30:24.200000 --> 0:30:27.100000 it can be customized to meet your requirements. 0:30:27.100000 --> 0:30:31.060000 And as I said, a lot of, it's one of the reasons you have a tool which, 0:30:31.060000 --> 0:30:36.240000 you know, is now considered an EDR slash XDR, like Waza or Wazoo. 0:30:36.240000 --> 0:30:40.640000 It's one of the reasons why, you know, Wazoo is actually built on the 0:30:40.640000 --> 0:30:46.240000 elk stack. So what Wazoo is essentially a security layer or plugin as 0:30:46.240000 --> 0:30:48.980000 it were into elk. 0:30:48.980000 --> 0:30:51.020000 Now, of course, I'm sort of simplifying it. 0:30:51.020000 --> 0:30:54.820000 There's a lot more that Waza or Wazoo does with regards to the agent and 0:30:54.820000 --> 0:30:59.200000 the EDR capabilities, file integrity, monitoring, threat hunting, threat 0:30:59.200000 --> 0:31:00.940000 intelligence, all that good stuff. 0:31:00.940000 --> 0:31:07.920000 But there's a reason why I'm, you know, I'm covering elk to, to this extent. 0:31:07.920000 --> 0:31:14.620000 So, to sort of bring this to a close, why elk is important for instant 0:31:14.620000 --> 0:31:17.320000 responsive, it isn't already clear already. 0:31:17.320000 --> 0:31:21.680000 So the elk stack isn't just a tool for log management or even log analysis, 0:31:21.680000 --> 0:31:26.340000 really. It's a critical enabler for detecting, investigating and responding 0:31:26.340000 --> 0:31:27.800000 to security incidents. 0:31:27.800000 --> 0:31:31.860000 And here's why every instant responder should understand it and master 0:31:31.860000 --> 0:31:41.980000 it eventually. I'm not faster detection and triage, be better context 0:31:41.980000 --> 0:31:46.080000 during investigation, see more efficient threat hunting and reporting 0:31:46.080000 --> 0:31:50.280000 the reduced dwell time of adversaries in the network. 0:31:50.280000 --> 0:31:52.760000 In essence, you just become better. 0:31:52.760000 --> 0:31:58.340000 So you become effective or you master, you know, a tool you don't need 0:31:58.340000 --> 0:32:01.760000 to master everything with regards to how elk works. 0:32:01.760000 --> 0:32:06.180000 Just, you know, focus on searches, being able to navigate around, find 0:32:06.180000 --> 0:32:10.540000 what you're looking for, create visualizations, so on and so forth. 0:32:10.540000 --> 0:32:16.100000 And you just become better at risk, you know, detection and response or 0:32:16.100000 --> 0:32:19.820000 detection and analysis as it were in this particular case. 0:32:19.820000 --> 0:32:23.100000 And I've added some references at the end of the slides here, where you 0:32:23.100000 --> 0:32:29.560000 can take a look at the elastic, you know, elk or elastics of visual websites. 0:32:29.560000 --> 0:32:31.780000 So it's elastic.co. 0:32:31.780000 --> 0:32:33.480000 There's the official documentation. 0:32:33.480000 --> 0:32:38.200000 And then there's also elastic security, which is essentially what they 0:32:38.200000 --> 0:32:42.000000 would consider the seem built on top of the elk stack. 0:32:42.000000 --> 0:32:46.100000 And we're not going to dive into that right now, because again, this video 0:32:46.100000 --> 0:32:47.360000 has been quite lengthy. 0:32:47.360000 --> 0:32:54.560000 And I think I've I have paid homage to elastic or the elk stack, as I 0:32:54.560000 --> 0:32:56.860000 still call it, as best as I can. 0:32:56.860000 --> 0:33:00.220000 And hopefully, you come out of this video, understanding it a little bit 0:33:00.220000 --> 0:33:06.500000 better. But the only direction from here is up, because now we're going 0:33:06.500000 --> 0:33:11.140000 to take this theoretical knowledge with regards to the elk stack or elastic 0:33:11.140000 --> 0:33:17.040000 stack, and we're going to put it into context by using the elk stack practically, 0:33:17.040000 --> 0:33:22.560000 and not just for, you know, finding generic logs, it's going to be, you 0:33:22.560000 --> 0:33:26.660000 know, we're going to be dealing with a real data set that has quite a 0:33:26.660000 --> 0:33:28.760000 bit of malicious activity. 0:33:28.760000 --> 0:33:30.280000 So our own is spoiled too much. 0:33:30.280000 --> 0:33:31.860000 That's going to be it for this video. 0:33:31.860000 --> 0:33:34.080000 And I will be seeing you in the next video.