1 00:00:00,000 --> 00:00:08,640 Sometimes while booting, it goes wrong. 2 00:00:08,640 --> 00:00:10,360 Now, what can you do? 3 00:00:10,360 --> 00:00:12,600 Well, first, you can boot into one 4 00:00:12,600 --> 00:00:16,360 of the troubleshooting targets, like systemd.unit 5 00:00:16,360 --> 00:00:19,160 is emergency.target or systemd.unit 6 00:00:19,160 --> 00:00:22,760 is rescue.target, just the way we have just seen. 7 00:00:22,760 --> 00:00:26,280 It's also possible to use initis-bin-bash 8 00:00:26,280 --> 00:00:27,559 as an alternative. 9 00:00:27,559 --> 00:00:31,920 That dives in before you even get to your systemd. 10 00:00:31,920 --> 00:00:34,000 But if a bootloader no longer shows, 11 00:00:34,000 --> 00:00:36,360 a rescue disk can be used. 12 00:00:36,360 --> 00:00:38,400 Now, the good news is that normally, 13 00:00:38,400 --> 00:00:42,520 any Linux installation disk can be used as a rescue disk. 14 00:00:42,520 --> 00:00:45,080 Before showing you, let me draw an overview 15 00:00:45,080 --> 00:00:47,200 of the different options of what you 16 00:00:47,200 --> 00:00:50,240 can use in which specific case. 17 00:00:50,240 --> 00:00:52,400 So before we explore the boot procedure, 18 00:00:52,400 --> 00:00:54,520 what can you do if things go wrong? 19 00:00:54,520 --> 00:00:58,799 Well, if you don't see grub at all, there's only one option, 20 00:00:58,799 --> 00:01:02,400 and that is going to be the rescue disk. 21 00:01:02,400 --> 00:01:04,480 So you need to boot from an external media 22 00:01:04,480 --> 00:01:06,080 to get access to your system. 23 00:01:06,080 --> 00:01:08,720 If you do see grub, but your kernel doesn't load, 24 00:01:08,720 --> 00:01:14,040 there's another option, and that is the rescue kernel. 25 00:01:14,040 --> 00:01:16,879 The rescue kernel is offered in the grub boot menu, 26 00:01:16,879 --> 00:01:19,199 and it offers an alternative. 27 00:01:19,199 --> 00:01:23,000 Then your kernel and your init-ramfs are loading, 28 00:01:23,000 --> 00:01:26,000 and you can jump into the system directly 29 00:01:26,000 --> 00:01:28,400 after loading the init-ramfs. 30 00:01:28,400 --> 00:01:32,040 On Red Hat, there is an option rd.break, 31 00:01:32,040 --> 00:01:35,160 but that option is Red Hat only, and for that reason, 32 00:01:35,160 --> 00:01:37,080 not so very powerful. 33 00:01:37,080 --> 00:01:40,800 On generic Linux, you can also use the boot argument, 34 00:01:40,800 --> 00:01:45,959 init is slash bin slash bash. 35 00:01:45,959 --> 00:01:49,120 So both of these options are added as kernel arguments, 36 00:01:49,120 --> 00:01:53,000 and the most generic kernel argument is init is bin bash. 37 00:01:53,000 --> 00:01:56,400 It skips your systemd, it throws you a bash shell, 38 00:01:56,400 --> 00:01:59,639 and you don't even have to provide a root user password. 39 00:01:59,639 --> 00:02:03,080 Once systemd is loaded, things are getting more complicated. 40 00:02:03,080 --> 00:02:11,600 So you would use systemd.unit equals emergency.target, 41 00:02:11,600 --> 00:02:14,559 and this emergency target will get you in right 42 00:02:14,559 --> 00:02:15,720 before early boot. 43 00:02:15,720 --> 00:02:18,600 So you really have a minimum of services 44 00:02:18,600 --> 00:02:21,119 which maximizes the chance of you getting back 45 00:02:21,119 --> 00:02:23,479 in a normal operational state. 46 00:02:23,479 --> 00:02:25,000 Now, in troubleshooting, it's always 47 00:02:25,000 --> 00:02:28,720 good to get into as much of your system as you can, 48 00:02:28,720 --> 00:02:34,119 and that is why there is also the rescue target. 49 00:02:34,119 --> 00:02:37,080 And a rescue target is bringing you all the way up 50 00:02:37,080 --> 00:02:40,759 to the early boot, you just don't load the services here. 51 00:02:40,759 --> 00:02:43,399 So these are your different options for troubleshooting, 52 00:02:43,399 --> 00:02:46,080 and in most cases, using one of them 53 00:02:46,080 --> 00:02:49,119 will get you back into your system. 54 00:02:49,119 --> 00:02:51,880 All right, let me demonstrate, and I'm 55 00:02:51,880 --> 00:02:56,039 going to demonstrate how you can use a rescue disk. 56 00:02:56,039 --> 00:02:59,960 In order to do so, I need to connect my disk, 57 00:02:59,960 --> 00:03:02,279 and I'm doing that in VMware. 58 00:03:02,279 --> 00:03:05,520 Your virtualization software is where you present your disk, 59 00:03:05,520 --> 00:03:08,279 and normally, it will find the disk 60 00:03:08,279 --> 00:03:10,240 that you have used for installation, 61 00:03:10,240 --> 00:03:12,440 as you can see right here grayed out. 62 00:03:12,440 --> 00:03:15,559 And when you use connect CD, DVD, 63 00:03:15,559 --> 00:03:17,600 it should be connected. 64 00:03:17,600 --> 00:03:20,919 And then you need to make sure that you boot from it. 65 00:03:20,919 --> 00:03:23,360 Now, there are different ways to boot 66 00:03:23,360 --> 00:03:26,559 into your specific environment. 67 00:03:26,559 --> 00:03:30,000 I am going to use sudo minus I, and I'm 68 00:03:30,000 --> 00:03:33,679 going to show you the EFI boot manager. 69 00:03:33,679 --> 00:03:37,399 So EFI boot manager is showing me what? 70 00:03:37,399 --> 00:03:39,800 Showing me all these different options, 71 00:03:39,800 --> 00:03:43,919 and I want to boot from option number one. 72 00:03:43,919 --> 00:03:51,559 So EFI boot MGR minus N0001. 73 00:03:51,559 --> 00:03:54,119 And then I'm going to reboot. 74 00:03:54,119 --> 00:03:57,919 And as you can see, I'm booting from my rescue disk. 75 00:03:57,919 --> 00:04:01,679 So the alternative of using EFI boot manager 76 00:04:01,679 --> 00:04:03,759 is to press Escape at the right moment. 77 00:04:03,759 --> 00:04:07,559 And let's face it, the reason you are going to a rescue disk 78 00:04:07,559 --> 00:04:09,279 is probably because you don't get 79 00:04:09,279 --> 00:04:12,399 into a fully operational environment anymore. 80 00:04:12,399 --> 00:04:14,960 And then you need to press Escape at the right moment 81 00:04:14,960 --> 00:04:18,399 while booting, or F2, F10, or F12. 82 00:04:18,399 --> 00:04:22,200 The key that you are using depends on the hardware 83 00:04:22,200 --> 00:04:24,079 that you are using. 84 00:04:24,079 --> 00:04:26,119 Now, for rescue disk, the option that we need 85 00:04:26,119 --> 00:04:27,519 is troubleshooting. 86 00:04:27,519 --> 00:04:30,000 So in troubleshooting, you select 87 00:04:30,000 --> 00:04:34,040 a rescue CentOS stream system, in case it is CentOS, 88 00:04:34,040 --> 00:04:34,519 of course. 89 00:04:34,519 --> 00:04:37,239 With the other rescue disks, Ubuntu and Red Hat 90 00:04:37,239 --> 00:04:41,559 and other distributions are offering similar solutions. 91 00:04:41,559 --> 00:04:43,279 So the idea is that you are going 92 00:04:43,279 --> 00:04:47,079 to load your operating system from the installation disk. 93 00:04:47,079 --> 00:04:50,000 And then while booting from the installation disk, 94 00:04:50,000 --> 00:04:53,959 you are going to try to address your hard disk. 95 00:04:53,959 --> 00:04:57,399 Some distributions require you to do a completely manual 96 00:04:57,399 --> 00:05:00,679 procedure where you need to find your information. 97 00:05:00,679 --> 00:05:05,040 Some distributions are making it a lot easier for you. 98 00:05:05,040 --> 00:05:08,320 Now, I want to show you in a generic way 99 00:05:08,320 --> 00:05:12,760 how you can access your hard disk from the rescue disk. 100 00:05:12,760 --> 00:05:17,119 And there, well, the rest of the troubleshooting would follow. 101 00:05:17,119 --> 00:05:18,679 Let's not dive into that. 102 00:05:18,679 --> 00:05:22,239 Let's just see if we can access the hard disk again. 103 00:05:22,239 --> 00:05:24,959 So here, CentOS is telling me what it is doing. 104 00:05:24,959 --> 00:05:28,559 It's attempting to find the installation 105 00:05:28,559 --> 00:05:31,920 and mount it on the directory, mnt-sysroot. 106 00:05:31,920 --> 00:05:33,279 I don't want to do that. 107 00:05:33,279 --> 00:05:35,720 I want to do option number three, 108 00:05:35,720 --> 00:05:39,239 because not all distributors are giving a nice option 109 00:05:39,239 --> 00:05:40,480 like this. 110 00:05:40,480 --> 00:05:43,359 Option number three, that is where you go to a shell. 111 00:05:43,359 --> 00:05:46,119 And that is where you can try to find your hardware. 112 00:05:46,119 --> 00:05:48,440 So LSBLK is what I'm using. 113 00:05:48,440 --> 00:05:49,799 And what do I see? 114 00:05:49,799 --> 00:05:52,839 I see my block devices. 115 00:05:52,839 --> 00:05:55,559 So that's good news already to start with. 116 00:05:55,559 --> 00:05:58,239 So I have my SCA1, 2, and 3. 117 00:05:58,239 --> 00:06:01,959 Now, I'm going to use VGScan. 118 00:06:01,959 --> 00:06:05,920 VGScan is LVM volume group scan. 119 00:06:05,920 --> 00:06:11,359 This is going to try to find LVM information on your system. 120 00:06:11,359 --> 00:06:15,239 Now, when I use LSBLK again, I still see the same thing. 121 00:06:15,239 --> 00:06:18,519 But when I use VGS, I see my volume group. 122 00:06:18,519 --> 00:06:23,119 And when I use LVS, I see my logical volumes. 123 00:06:23,119 --> 00:06:27,440 And with that, there is my root logical volume. 124 00:06:27,440 --> 00:06:34,720 So let's try to give it a mount dev CS slash root on slash MNT. 125 00:06:34,720 --> 00:06:36,799 That's a temporary mount. 126 00:06:36,799 --> 00:06:41,880 And uh-oh, we have a cannot lookup block device. 127 00:06:41,880 --> 00:06:45,160 DMessage may have more information about it. 128 00:06:45,160 --> 00:06:46,239 What is going on? 129 00:06:46,239 --> 00:06:51,040 It doesn't know about dev CS root yet. 130 00:06:51,040 --> 00:06:53,559 Now, if you get back to LSBLK, that's 131 00:06:53,559 --> 00:06:57,480 understandable because LSBLK can't show it as well. 132 00:06:57,480 --> 00:07:01,119 And if you use LS on slash dev slash CS, 133 00:07:01,119 --> 00:07:04,000 we see no such root or directory. 134 00:07:04,000 --> 00:07:09,799 And also, LS on dev mapper is showing no LVM information. 135 00:07:09,799 --> 00:07:11,679 Now, what do we need to do next? 136 00:07:11,679 --> 00:07:13,799 We need LVChange. 137 00:07:13,799 --> 00:07:17,079 Now, it's maybe wise to use LVChange minus minus help 138 00:07:17,079 --> 00:07:19,160 to find out how to use it. 139 00:07:19,160 --> 00:07:22,839 And here in the help, we can see that, oh my goodness, 140 00:07:22,839 --> 00:07:25,559 there are so many options. 141 00:07:25,559 --> 00:07:32,359 And in this so many options, I'm looking for the minus A. 142 00:07:32,359 --> 00:07:34,079 Minus A is activate. 143 00:07:34,079 --> 00:07:36,600 And that has a Y. And we need the volume group name. 144 00:07:36,600 --> 00:07:37,559 Aha. 145 00:07:37,559 --> 00:07:43,160 So LVChange CS minus A Y. 146 00:07:43,160 --> 00:07:44,920 That's what I'm looking at. 147 00:07:44,920 --> 00:07:47,160 And now I'm using LSBLK again. 148 00:07:47,160 --> 00:07:48,200 And what do we see? 149 00:07:48,200 --> 00:07:50,679 We see the LVM logical volumes. 150 00:07:50,679 --> 00:07:56,239 Because of the VG scan first and the LVChange volume group 151 00:07:56,239 --> 00:07:59,760 name minus A Y, I now have my volume group available. 152 00:07:59,760 --> 00:08:06,200 And I should be able to mount dev CS root on MNT. 153 00:08:06,200 --> 00:08:09,160 And now I'm in my MNT directory. 154 00:08:09,160 --> 00:08:12,320 And from there, I can continue working 155 00:08:12,320 --> 00:08:15,799 and try to recover my system. 156 00:08:15,799 --> 00:08:19,200 Now that we are here, let's do the next step as well. 157 00:08:19,200 --> 00:08:23,839 And the next step would be that we make the other file systems 158 00:08:23,839 --> 00:08:25,399 available as well. 159 00:08:25,399 --> 00:08:29,760 My problem is that the PROC file system in MNT is empty. 160 00:08:29,760 --> 00:08:31,959 The sys file system is empty. 161 00:08:31,959 --> 00:08:34,359 And the dev file system is empty. 162 00:08:34,359 --> 00:08:37,320 But we do have these guys on the rescue system. 163 00:08:37,320 --> 00:08:38,919 Now, the thing with these file systems 164 00:08:38,919 --> 00:08:42,440 is that they are automatically created while booting. 165 00:08:42,440 --> 00:08:45,200 And I need to make sure that the file systems on the rescue 166 00:08:45,200 --> 00:08:49,239 system are present in my recovered system. 167 00:08:49,239 --> 00:08:53,840 And in order to do so, I'm using mount minus O bind. 168 00:08:53,840 --> 00:08:55,359 That's a bind mount. 169 00:08:55,359 --> 00:08:58,239 And in a bind mount, you can connect directly 170 00:08:58,239 --> 00:09:00,080 to another directory. 171 00:09:00,080 --> 00:09:02,239 I do it for PROC on MNT PROC. 172 00:09:02,239 --> 00:09:05,320 I do that for dev on MNT dev. 173 00:09:05,320 --> 00:09:11,280 And I do that for sys on MNT sys. 174 00:09:11,280 --> 00:09:14,559 And there, we are a bit further. 175 00:09:14,559 --> 00:09:18,119 Now, I'm going to use chroot on slash MNT. 176 00:09:18,119 --> 00:09:21,960 And that brings me into the root of my normal operational 177 00:09:21,960 --> 00:09:23,039 system. 178 00:09:23,039 --> 00:09:24,359 I can even write. 179 00:09:24,359 --> 00:09:27,359 And that means that I can do my troubleshooting commands. 180 00:09:27,359 --> 00:09:30,440 Let's just try one common troubleshooting command 181 00:09:30,440 --> 00:09:33,760 to reset the root user password. 182 00:09:33,760 --> 00:09:34,679 Does that work? 183 00:09:34,679 --> 00:09:36,400 That's the only thing I care about. 184 00:09:36,400 --> 00:09:37,760 And yes, it works. 185 00:09:37,760 --> 00:09:41,640 So this concludes my demo about rescuing. 186 00:09:41,640 --> 00:09:46,400 Final step, use exit to exit the chroot environment. 187 00:09:46,400 --> 00:09:50,960 And use reboot to reboot in your normal operational system.