1 00:00:00,000 --> 00:00:10,120 In this video, we'll talk about a couple of commands that can help in troubleshooting 2 00:00:10,120 --> 00:00:17,160 storage problems. So what can you do? Well, first there is fscheck. Fsck or fscheck is 3 00:00:17,160 --> 00:00:24,000 what you use to monitor file system integrity. You can use it manually on any device to figure 4 00:00:24,000 --> 00:00:29,760 out if there is any trouble. Fartprobe is an interesting command. It can be used to 5 00:00:29,760 --> 00:00:35,080 update the kernel partition table with real storage information. If ever you feel that 6 00:00:35,080 --> 00:00:42,040 your storage information is not visible by the kernel, fartprobe might be your fix. du 7 00:00:42,040 --> 00:00:47,639 is for disk usage. It's the utility that you use to get an overview of available disk space 8 00:00:47,639 --> 00:00:55,000 for each directory, and df will show available disk space per device. Iostat can generate 9 00:00:55,000 --> 00:01:01,119 a wide range of storage-related statistics, and you can see a device activity overview. 10 00:01:01,119 --> 00:01:07,639 Ioping is an optional utility which is used to test IOPerformance, but in many cases it's 11 00:01:07,639 --> 00:01:14,599 not installed by default. And to optimize IOPerformance, the IOScheduler can be tuned 12 00:01:14,599 --> 00:01:20,559 through the slash sys file system. Let me show you. So to start with the last one, in 13 00:01:20,559 --> 00:01:29,400 the sys directory you have interfaces to all your different devices. And if we go to block, 14 00:01:29,400 --> 00:01:35,519 in block, you can find the subdirectory for your block devices. And in there I can find 15 00:01:35,519 --> 00:01:42,760 my NVMe 0 and 1. Apparently on this system we have five different NVMe disks, and for 16 00:01:42,760 --> 00:01:50,800 each of them you can optimize the IOScheduler. And you do that through the queue directory 17 00:01:50,800 --> 00:01:58,919 in this directory. And in this queue directory there is different files, including the filescheduler. 18 00:01:58,919 --> 00:02:06,000 And there we can see four different schedulers, non-MD, deadline, kyber, and bfq. Now if ever 19 00:02:06,000 --> 00:02:13,000 you think that your disk is poorly behaving, you might change these schedulers. To do so 20 00:02:13,000 --> 00:02:19,960 you echo the desired value, so like bfq, into the scheduler file. Once again, like slash 21 00:02:19,960 --> 00:02:25,839 proc slash sys is one of these directories where you cannot directly edit files, so use 22 00:02:25,839 --> 00:02:31,440 this approach to change the contents of the file. And there we can see that now it is 23 00:02:31,440 --> 00:02:37,679 set to bfq. And you can test if this is leading to a better behavior. If it is, you can make 24 00:02:37,679 --> 00:02:45,320 it persistent using one of the systemd startup units. If it's not, better get back by echoing 25 00:02:45,320 --> 00:02:51,919 the original value, or just reboot, which will always get you back to the original value. 26 00:02:51,919 --> 00:02:59,960 Talking about which, if you are on NVMe, NVMe is SSD on steroids. It's a really fast SSD. 27 00:02:59,960 --> 00:03:06,759 You don't need to optimize anything on your IO scheduler to have it work the right way. 28 00:03:06,759 --> 00:03:13,759 IOstat is also an interesting utility. Let me use IOstat 2.5, where two means I want 29 00:03:13,759 --> 00:03:19,520 a two-second interval, and five means I want to see it five times. And oh, I need to install 30 00:03:19,520 --> 00:03:25,839 the sysstat package. Sysstat is a very convenient package that I think you should have at all 31 00:03:25,839 --> 00:03:32,919 time. And what do we see here in IOstat? Well, we see an overview of device activity. 32 00:03:32,919 --> 00:03:37,759 The nice thing about it is that it is giving a list of all the devices, so you can easily 33 00:03:37,759 --> 00:03:45,000 identify the busiest device, which in this case apparently is devnvme0 and 3. Ignore 34 00:03:45,000 --> 00:03:53,119 about the dm0, because dm0 is a device mapper, and that is what is used by LVM. So dm0 is 35 00:03:53,119 --> 00:04:00,199 hosted by NVMe0 and 3, as we see more or less the same values here. And there you can 36 00:04:00,199 --> 00:04:09,440 see the stats, transaction per second, kilobyte read per second, kilobyte written per second, 37 00:04:09,440 --> 00:04:16,799 kilobyte discarded, the total kilobyte read and written, and the total kilobyte discarded. 38 00:04:16,799 --> 00:04:22,600 That's a nice overview of IO performance. Talking about performance, let me also use 39 00:04:22,600 --> 00:04:33,040 dnfinstall-y io-top. That's one of my personal favorites. So when you use io-top, then what 40 00:04:33,040 --> 00:04:41,279 do you see? Well, you see current IO activity. And what I like about io-top is that like 41 00:04:41,279 --> 00:04:46,600 within the top utility, you can see the busiest utility on the top, which apparently in this 42 00:04:46,600 --> 00:04:56,920 case is the ptyxis. ptyxis is what is behind your GNOME terminals on modern systems. Use 43 00:04:56,920 --> 00:05:04,480 Q to get out of here. Then how about io-ping? Well, io-ping is a nice utility, but as I 44 00:05:04,480 --> 00:05:10,519 was mentioning, it's a utility that is not often available in the default distribution 45 00:05:10,839 --> 00:05:17,359 repositories. But do remember that io-ping can be used to ping IO devices and get more 46 00:05:17,359 --> 00:05:22,079 information about their performance. You might see questions about it in the exam because 47 00:05:22,079 --> 00:05:24,160 it's in the Linux Plus objectives.