1 00:00:00,000 --> 00:00:10,000 Next area to investigate is CPU performance. 2 00:00:10,000 --> 00:00:13,000 So what can you do to monitor CPU performance? 3 00:00:13,000 --> 00:00:17,000 Well, of course, apart from the obvious thing like top command, 4 00:00:17,000 --> 00:00:20,000 which is revealing a lot of information about CPUs, 5 00:00:20,000 --> 00:00:22,000 there is proc CPU info, 6 00:00:22,000 --> 00:00:25,000 which gives detailed information about the CPUs, 7 00:00:25,000 --> 00:00:28,000 like flags that are supported. 8 00:00:28,000 --> 00:00:33,000 If, for instance, you want to use hardware virtualization, 9 00:00:33,000 --> 00:00:38,000 you need to make sure that either the SVM or the VMX flag is available, 10 00:00:38,000 --> 00:00:41,000 and proc CPU info will tell you about it. 11 00:00:41,000 --> 00:00:44,000 Uptime is showing how long the system has been up, 12 00:00:44,000 --> 00:00:47,000 and it also shows load averages. 13 00:00:47,000 --> 00:00:52,000 All the uptime output is also integrated in the top command output. 14 00:00:52,000 --> 00:00:55,000 And SAR is a very convenient command that can be used 15 00:00:55,000 --> 00:00:58,000 to gather long-time usage statistics. 16 00:00:58,000 --> 00:01:04,000 The nice thing about SAR is that it gathers performance data every X minutes. 17 00:01:04,000 --> 00:01:07,000 And once performance data has been gathered, 18 00:01:07,000 --> 00:01:09,000 you can analyze it using the SAR command. 19 00:01:09,000 --> 00:01:14,000 But you need to enable it, because by default it won't do anything. 20 00:01:14,000 --> 00:01:18,000 And finally, there are CCTL interface parameters in slashproc 21 00:01:18,000 --> 00:01:21,000 to tune different CPU-related settings. 22 00:01:21,000 --> 00:01:23,000 Let me show you. 23 00:01:23,000 --> 00:01:26,000 So to start it, I'm going to show you systemd timers. 24 00:01:26,000 --> 00:01:34,000 Let me use systemctl list unit files minus t timer. 25 00:01:34,000 --> 00:01:39,000 And what I'm looking for is the sysstat collect timer. 26 00:01:39,000 --> 00:01:42,000 So what is in the sysstat collect timer? 27 00:01:42,000 --> 00:01:54,000 Well, let's use systemctl get on sysstat collect dot timer. 28 00:01:54,000 --> 00:01:58,000 And there we can see that this is a job that is running every 10 minutes 29 00:01:58,000 --> 00:02:03,000 to collect information that the SAR utility is using. 30 00:02:03,000 --> 00:02:06,000 Now, the critical question here is what time is it? 31 00:02:06,000 --> 00:02:09,000 It's 1.33. Good. 32 00:02:09,000 --> 00:02:15,000 I installed the SAR utility earlier when I've shown you the iostat command output. 33 00:02:15,000 --> 00:02:22,000 And that means that I'm expecting this sysstat utility to have collected some data. 34 00:02:22,000 --> 00:02:28,000 If you go in the var log directory, then you can find an SA subdirectory. 35 00:02:28,000 --> 00:02:34,000 And this SA subdirectory is the directory that is normally filled by the sysstat collect utility. 36 00:02:34,000 --> 00:02:38,000 And oh, boy, I can see no files in there. 37 00:02:38,000 --> 00:02:39,000 What does that mean? 38 00:02:39,000 --> 00:02:43,000 That means that SAR minus uppercase A, which is a good start. 39 00:02:43,000 --> 00:02:51,000 Well, as you can see, it's telling me it cannot open var log SA SA11 node switch file or directory. 40 00:02:51,000 --> 00:02:54,000 It's looking for a file, but the file does not exist. 41 00:02:54,000 --> 00:02:59,000 So apparently, data collection has not been happening yet. 42 00:02:59,000 --> 00:03:07,000 If I want to know more about it, then I can use systemctl status on sysstat collect dot timer. 43 00:03:07,000 --> 00:03:12,000 And there we can see that it is enabled, enabled, and active. 44 00:03:12,000 --> 00:03:15,000 It is inactive and triggers sysstat collect service. 45 00:03:15,000 --> 00:03:18,000 Well, apparently, it just hasn't started yet. 46 00:03:18,000 --> 00:03:19,000 But you know what? 47 00:03:19,000 --> 00:03:23,000 We can also start sysstat collect service manually. 48 00:03:23,000 --> 00:03:33,000 So systemctl start on sysstat collect dot service is going to run the process for me manually. 49 00:03:33,000 --> 00:03:39,000 So now that the sysstat information is starting to gather, we need to give it a little bit of time. 50 00:03:39,000 --> 00:03:42,000 So let's have a look at the proc parameters. 51 00:03:42,000 --> 00:03:48,000 In slash proc slash sys, you find information about tunables. 52 00:03:48,000 --> 00:03:55,000 And these kernel tunables, well, there's a couple of tunables for CPU as well. 53 00:03:55,000 --> 00:03:58,000 You find them in proc sys kernel. 54 00:03:58,000 --> 00:04:03,000 Because in the end, the Linux kernel determines what the CPU is doing. 55 00:04:03,000 --> 00:04:09,000 And here you can find a couple of these options, which are the scheduler options. 56 00:04:09,000 --> 00:04:16,000 The scheduler determines how much time every process gets that wants to do something on the CPU. 57 00:04:16,000 --> 00:04:18,000 So how does it work? 58 00:04:18,000 --> 00:04:26,000 Well, let's say that you want to change the amount of time for your SCAD RT runtime microseconds. 59 00:04:26,000 --> 00:04:29,000 Now, first, let's investigate what is in there. 60 00:04:29,000 --> 00:04:32,000 So SCAD RT, which is for real-time processes. 61 00:04:32,000 --> 00:04:39,000 And every real-time process is getting this amount of microseconds as its runtime, 62 00:04:39,000 --> 00:04:46,000 meaning that the remaining amount of microseconds on a second is for all the other processes. 63 00:04:46,000 --> 00:04:53,000 And if you think this is too much or not enough, you echo the new value into the file. 64 00:04:53,000 --> 00:05:01,000 So echo 900,000 greater than SCAD RT runtime US. 65 00:05:01,000 --> 00:05:04,000 And that will set the setting right now. 66 00:05:04,000 --> 00:05:11,000 And if you're happy with it, you can make it persistent using sysctl, as we have discussed before. 67 00:05:11,000 --> 00:05:13,000 So how does that work? 68 00:05:13,000 --> 00:05:17,000 Well, I would suggest you do the following. 69 00:05:17,000 --> 00:05:24,000 You copy the SCAD RT runtime US, because that's a difficult parameter name, isn't it? 70 00:05:24,000 --> 00:05:26,000 Well, here we have it copied. 71 00:05:26,000 --> 00:05:30,000 Now I'm going into etc.sysctl.d. 72 00:05:30,000 --> 00:05:34,000 And there we have the 99.sysctl.conf. 73 00:05:34,000 --> 00:05:38,000 And you can just add it to this. 74 00:05:38,000 --> 00:05:40,000 So what am I going to edit? 75 00:05:40,000 --> 00:05:51,000 Well, I am going to edit kernel.parameter that I want to change equals value that I want to give to the parameter. 76 00:05:51,000 --> 00:05:59,000 And that makes sure that next time your system is started, the sysctl service is picking up the value. 77 00:05:59,000 --> 00:06:05,000 And in case you forgot the exact naming of this parameter, well, this here is a relative file name. 78 00:06:05,000 --> 00:06:09,000 It's a relative file name where the dot is used as a separator. 79 00:06:09,000 --> 00:06:14,000 And it is relative to the slash proc slash sys directory. 80 00:06:14,000 --> 00:06:17,000 So this will make it persistent. 81 00:06:17,000 --> 00:06:19,000 Now let's get back to SAR. 82 00:06:19,000 --> 00:06:24,000 SAR minus minus help is showing all the different SAR utilities. 83 00:06:24,000 --> 00:06:31,000 And as you can see, many, many command line options are available. 84 00:06:31,000 --> 00:06:34,000 Here we can see the minus U option. 85 00:06:34,000 --> 00:06:37,000 After all, we're talking about CPU utilization. 86 00:06:37,000 --> 00:06:46,000 So SAR minus U all should show information that has been gathered about all of my CPUs. 87 00:06:46,000 --> 00:06:50,000 So a little bit of troubleshooting about this service collection. 88 00:06:50,000 --> 00:06:59,000 If you do a systemctl cat on sysstat collect dot service, you can see what it is actually doing. 89 00:06:59,000 --> 00:07:03,000 And here you can see that this is one shot, so it will run it once. 90 00:07:03,000 --> 00:07:08,000 And what it runs is uselib64sa1, et cetera. 91 00:07:08,000 --> 00:07:12,000 So if you just copy this command and you run it manually, 92 00:07:12,000 --> 00:07:17,000 then the result is that it is going to gather some system information for you. 93 00:07:17,000 --> 00:07:20,000 Now the timer is doing this every 10 minutes. 94 00:07:20,000 --> 00:07:25,000 But if you're impatient and if you don't want to work, this allows you to get started. 95 00:07:25,000 --> 00:07:28,000 And next, you can use your SAR command. 96 00:07:28,000 --> 00:07:32,000 So SAR minus minus help is giving all these different options. 97 00:07:32,000 --> 00:07:39,000 Probably a good way to get started is by using CPU information only. 98 00:07:39,000 --> 00:07:41,000 That will be minus U all. 99 00:07:41,000 --> 00:07:45,000 So try SAR minus U all. 100 00:07:45,000 --> 00:07:52,000 And here you see the statistics over time that have been gathered for your CPU utilization. 101 00:07:52,000 --> 00:07:55,000 So what do we see? 102 00:07:55,000 --> 00:07:59,000 Well, we see the user and the nice and the system space. 103 00:07:59,000 --> 00:08:03,000 And it is capturing this information every 10 minutes. 104 00:08:03,000 --> 00:08:10,000 And that is useful because if every day you want to know which are your peak moments and your low moments, 105 00:08:10,000 --> 00:08:17,000 well, you can clearly figure out if there are moments when your CPUs are hit harder than at other moments. 106 00:08:17,000 --> 00:08:21,000 And that is how SAR can be an absolutely very useful utility.