While working with text files, you are going to like regular expressions.
Regular expressions or regex are text patterns that are used by tools like grep and others.
When you use regular expressions, always put it between single quotes.
That is because regular expressions contain special characters and you want
to avoid these special characters to be interpreted as shell metacharacters.
In regular expressions you have these special characters.
They look the same as wildcard characters, also known as globbing.
Make sure not to confuse them.
A star in a regular expression is a completely different way as a star in a wildcard character.
So that means that you can use a command like grab a, where the first part identified
by the single quotes is the regular expression and the second part is is the globing pattern.
Regular expresses are for use with specific tools only, and these
are utilities like grep and vim and awk and sed.
Mainly regular expresses are for utilities that are doing something with text files.
For more details, you can consult the man page man page in section number 7 for regex.
Regular expresses are often considered to be confusing, and I can understand that now. Why is that so?
That is because there are basic regular expresses that work with most tools.
There are also extended regular expressions that don't always work.
If it's an extended regular expression, you need
grep to identify it as an extended regex.
And some scripting languages like Perl come with their own regular expresses as well.
Now, as a result, you might be googling for regular expressions and find something
that looks very convenient, but it might be a Perl regular expression, and that's
not going to work if you run it with commands like grep.
Now let's talk about using regular expressions.
Now let me demonstrate regular expressions for your convenience. The commands that I'm going
to use are on the slide as well, so let me show you.
So for these demos, I created a file with the name regex with a lot of text, which you can see right here.
Now let's demonstrate some regular expressions starting
with grab, carrot l on regex.
What is that? Well, that is looking for lines that are starting with an link.
The caret is what we call an anchor, and it looks for lines that
they're starting with. We also have the opposite, and that is the dollar.
So anadolar is looking for lines that are ending with.
Then we have the one character regular expression.
So what can we do? Well, we can
do something like grab, carat dot dollar.
That means that it is looking for lines that have one character and only one character.
And as you can see, there is one line that is matching.
You can look for words using grep between single quotes.
B, anna, B on the vowel, and that is looking
for lines that have the text Anna is a word.
It might be starting with. It might be in the middle of the line, but for sure it's a word.
Now, as you notice, the B followed by the text that you're looking
for is not very clear, but that's the way how it works.
Then you have the zero or more times a regular expression.
So grep n.star
x on MyFal.
The star here is a modifier of the character right before. So we are looking for a line
that starts at an end. Then we have zero or more characters, and at the end of the
line we have an X. That's my val, it's regex and there you can see the result.
Also pretty convenient is a regular expression for spaces
or tabs. You can use grab single quote, dot,
dot star, double square bracket, colon, space colon double
square bracket, dot dot star single quote in regex.
Now that looks complicated. How do we interpret? Well, the first dot is it should have at least one character.
Then there's a dot star that means followed by zero or more characters. But then we need a
space or a tab or anything like that. And that is followed again by at least one character.
And this is the result.
Sadly, only one result, but one is enough here.
So these are examples of base regular expressions.
Let's have a look at the next part of this demo.
In the next example, we have a few extended regular expressions.
And extended regular expressions need to be called with the grep minus uppercase e.
So I'm going to look for bi t on regex. What
is that? Well, the preceding character needs to occur one or
more times, and as you can see. Oh, one time.
Well, one time is enough.
Then we also have the zero or one time, and that will be a question mark.
So bit is showing a match on BT as well as bit.
Then we have the repeating operator.
And in the repeating operator you can use indicators to tell the
regular expression that a character should be occurring a couple of times.
Now, the syntax is a little bit awkward because this is what
you use to indicate that we have three ends in our text.
So we are looking for the n, and the n is followed by the repeating operator. Now, the
challenge is that in the repeating operator there is a curly brace and we need to escape the
special meaning of the curly brace. And that is why in front of the opening, as well as
a closing curly brace, there's Backslash and oh boy, nothing in there. Well, let's use echo bollen.
Great. Then great. Then to my regex file and then we try it again.
Oh boy. What is going wrong? Well, I can see what is going wrong. I make a
beginner error and that is I'm using grep e and grep e is for extended regular expressions.
But if the pattern you are looking for is a
standard regular expression, you get no match. No match.
Now next let's look for a string. That must
be a word. So grep on Anna in regex.
I think we already saw that.
But what is going on? Well, I'm making a typo. It shouldn't be N,
should be B. And the final option is a pretty powerful option as well.
Grep e extended regular expression this time.
And we are looking for either SVM
or VMX in the file regex.
So that will give a match if either one of these texts occurs. And if one of them doesn't occur, then it's okay as well.
And that is also an extended regular expression.
That's fun, isn't it? These extended regular expressions and base regular expressions
that need to be addressed separately. But it's the way it is.
Now, as I know that you can't get enough of it. In
this final slide we have some terrible advanced examples of regular expressions.
Not because I'm expecting you to learn immediately what this is all about, but because it's a
part of the power of regular expressions and it doesn't hurt to have seen some of it.
So the first one is going to search for IP addresses in log
files, for instance, so sudo grab minus e. It's an extended regular expression.
And Here we go. Range 09 and repeating Operator 13 at least 1
and maximum 3 followed by a dot, but the dot is escaped. So
we are really looking for a dot and it for any character.
And then we repeat this 09 again one up to three
times. And we need to repeat that a few more times.
Let's do that in var log messages.
Oh no, no result. That can't happen.
Let me do a logger 192.68.29.11
and do it again and aha.
Do you see that?
It's small but it's there. The so called typo. And now it's looking so much better.
I add a dot instead of a comma between the one and the three and of course then it doesn't work.
So that's example number one. Now do we have any Mac addresses
in the log sudo grep e. Now what is a Mac address?
Well, a Mac address is another pattern.
And the pattern that we are looking for is A F A09.
Oh my goodness, that's a complex pattern. So we're looking for uppercase
a till F, lowercase A till F and 0 followed by 9.
And that's what we do two times.
So AB a 09, A, stuff like that would all work.
And all of that is what we put between parentheses.
And we tell the regular expression to repeat that five times
in total, followed by a f once more, a FN092 times.
And we check for that in var log messages
and there we can see the Mac address.
So that's a regular expression that will look for Mac address.
Then let's see how we can look for dates in the
4 digit year, 2 digit month and 2 digit date format.
Well, that will be sudo grab e again and this time
b. So we are looking for a word 09 and we
will do that four times followed by 09 and now we're
looking for months. So two times only and again a 09,
two times only as a word in var log messages.
And I'm getting an error here. Do you see the typo? Well, we have an invalid range end and that means that
we have something in the square brackets, curly braces and so on that doesn't match. And now it's working so much better.
And here we see.
Now the final one is sudo grab minus
O minus E followed by HTTPs question mark,
colon, slash, slash, a Z a Z09.
That's a pattern that we have seen before.
Dot, slash, question mark equals underscore minus
square bracket plus on. Well, let's do
that on var log messages again.
And that is looking for URL format.
Now what is this? Well, in this regular expression, the plus at the end continues the regular expression as long as
any character between the square bracket is encountered and it will stop on finding an invalid character like space. That's the
end of the URL. And that brings us a result in the way that you can see it right here.
And of course I don't expect you to be able to produce irregular expresses like this
immediately, but it does make sense having seen them and knowing how to interpret them.