{"id":22958,"date":"2020-09-29T17:07:49","date_gmt":"2020-09-29T15:07:49","guid":{"rendered":"http:\/\/cornay.co.za\/wp\/?p=22958"},"modified":"2020-09-29T17:07:49","modified_gmt":"2020-09-29T15:07:49","slug":"linux-fu-making-awk-a-bit-easier","status":"publish","type":"post","link":"http:\/\/cornay.co.za\/wp\/2020\/09\/29\/linux-fu-making-awk-a-bit-easier\/","title":{"rendered":"Linux-Fu: Making AWK a Bit Easier"},"content":{"rendered":"

awk<\/code> is a kind of Swiss Army knife for text files. However, some of its limitations are often a bit annoying. I’ve used a simple set of functions to make awk<\/code> a bit better, although I will warn you: it does require GNU extensions to awk<\/code>. That is, you must use gawk<\/code> and not other versions. Your system probably maps \/usr\/bin\/awk<\/code> to something and that something might be gawk<\/code>. But it could also be mawk<\/code> or some other flavor. If you use a Debian-based distro, update-alternatives is your friend here. But for the purposes of this post, I’m going to assume you are using gawk<\/code>.<\/p>\n

By the end of the post, you’ll see how to use my awk<\/code> add-on functions<\/a> to split up a line into fields even when there is no single character to separate all fields. In addition, you’ll be able to refer to the fields using names you decide. You won’t have to remember that $2 is the time field. You’ll say Fields_fields[\"time\"]<\/code> instead.<\/p>\n

The Problem<\/h2>\n

awk<\/code> does a lot of common work for you when you use it to process text files. It reads files a record at a time. Normally, a record is a single line. Then it splits the line on fields using whitespace, or some other choice of field separators. You can write code that manipulates the line or individual fields. This default behavior is great, especially since you can change the end of record character and the field separator. A surprising number of files fit this sort of format.<\/p>\n

Until, of course, they don’t. If you have data coming from a data logging instrument or some database, it could be formatted in a variety of ways. Some fields might have structured data with a variety of separators. This isn’t a deal-breaker. Since you can get at the whole line, you can do almost anything you want, but the logic is harder and the whole point to using awk<\/code> is to make things easier.<\/p>\n

For example, suppose you had a file from a data recorder that had an eight-digit serial number, followed by a six-character tag, and then two floating point numbers separated by colons. The pattern might look like<\/p>\n

\n^([0-9]{8})([a-zA-Z0-9]{6})([-+.0-9]+),([-+.0-9]+)$\n<\/pre>\n
This would be hard to handle with the conventional field splitting and you’d normally just write code to split everything apart.<\/p>\n
<\/span><\/p>\n
If you have regular fields, but don’t know how many, you probably want to set FS<\/code> or FPAT<\/code>, instead. We talked about FPAT<\/code> a little before when we were abusing awk<\/code> to read hex files<\/a>. This library is a little different. You can use it to pick apart a line totally. For example, you might have part of the line with a fixed field length and then multiple types of separators. That can be hard to handle with the other methods.<\/p>\n
Regular Expressions<\/h2>\n
\"\"<\/a>To make things easier, I’ll wrap up the gawk<\/code> match<\/code> function. That function exists in regular awk<\/code>, of course, but gawk<\/code> adds an extension that makes things much easier. Normally, the function performs a regular expression match on a string and tells you where the match starts, if there was a match, and how many characters matched.<\/p>\n
With the GNU extensions in gawk<\/code>, you can provide an extra array argument. That array will get some information about the match. In particular, the zero item of the array will contain the entire match. If the regular expression contains sub-expressions in parenthesis, the array will contain those, numbering by the order of the parenthesis. It will also contain start and length information.<\/p>\n
For example, if your regular expression were \"^([0-9]+)([a-z]+)$\"<\/code> and your input string is 123abc<\/code>, the array would look like this:<\/p>\n
\n\narray[0] - 123abc\narray[1] - 123\narray[2] - abc\narray[0start] - 1\narray[0length] - 6\narray[1start] - 1\narray[1length] - 3\narray[2start] - 4\narray[2length] - 3\n\n<\/pre>\n
You can even have nested expressions, so \"^(([xyz])[0-9]+)([a-z]+)$\"<\/code> with an input of z1x gives array[1]=z1<\/code>, array[2]=z<\/code>, and array[3]=x<\/code>.<\/p>\n
Theory vs Practice<\/h2>\n
In theory, that’s all you need. You can write a regular expression to pick apart a line, parse it, and then access the pieces using the array. In practice, it is much nicer to have everything done so you can use plain names to access the data.<\/p>\n
As an example data format, consider a line like this:<\/p>\n
\n\n11\/10\/2020 07:00 The Best of Bradbury, 14.95 *****\n\n<\/pre>\n
There is a date in US format, a time in 24-hour format, an item name, a price, and a rating from 1 to 5 stars that may not be present. Writing a regular expression to grab each field is a bit complex, but not very hard. Here is one way to do it:<\/p>\n
\n\n"^(([01][0-9])\/([0-3][0-9])\/(2[01][0-9][0-9]))[[:space:]]*(([0-2][0-9]):([0-5][0-9]))[[:space:]]+([^,]+),[[:space:]]*([0-9.]+)[[:space:]]*([*]{1,5})?[[:space:]]*$"\n\n<\/pre>\n
That’s a mouthful, but it works. Note that each item is in parenthesis and some of those are nested. So the date is one field, but the month, day, and year are also fields.<\/p>\n
The Library<\/h2>\n
Once you grab the files on GitHub<\/a>, you could put the fields_* functions into your code. You need to do some setup in the BEGIN tag. Then you process each line using fields_process. Here’s a small example (with the functions omitted):<\/p>\n
\n\nBEGIN {\nfields_setup("^(([01][0-9])\/([0-3][0-9])\/(2[01][0-9][0-9]))[[:space:]]*(([0-2][0-9]):([0-5][0-9]))[[:space:]]+([^,]+),\u00a0 \u00a0 \u00a0[[:space:]]*([0-9.]+)[[:space:]]*([*]{1,5})?[[:space:]]*$")\nfields_setupN(1,"date")\nfields_setupN(2,"month")\nfields_setupN(3,"day")\nfields_setupN(4,"year")\nfields_setupN(5,"time")\nfields_setupN(6,"hours")\nfields_setupN(7,"minutes")\nfields_setupN(8,"item")\nfields_setupN(9,"price")\nfields_setupN(10,"star")\n\n}\n\n{\nv=fields_process()\n\n... your code here...\n\n}\n<\/pre>\n
In your code you can write something like:<\/p>\n
\n\ncost=Fields_fields["price"] * 3\n\n<\/pre>\n
Simple, right? The fields_process<\/code> function returns false if there was no match. You can still access the normal awk<\/code> fields like $0 or $2 if you want.<\/p>\n
Inside<\/h2>\n
The extra functions rely on two things: the extensions to the gawk<\/code> match<\/code> function and awk<\/code>‘s associative array mechanism. In the past, I’ve added the named keys to the existing match array so you could get data out either way. However, I’ve modified it so that the match array is local because I almost never really want that capability and then you have to filter out the extra fields if you want to dump the entire array.<\/p>\n
It is frequently useful to start the regular expression with ^<\/code> and end it with $<\/code> to anchor the entire string. Just don’t forget that the regular expression needs to handle white space consumption, as the example does. This is often a benefit when you have fields that can contain spaces, but if you wanted spaces to break fields anyway, you are probably better off with the original parsing scheme.<\/p>\n
Another trick is to get “the rest of the line” after you parsed off the first fields. You can do that by adding \"(.*)$\"<\/code> to the end of the regular expression. Just don’t forget to set up a tag for it using fields_setupN<\/code> so that you can fetch the value later.<\/p>\n
An easy extension to this library would be to make the pattern an array. The processing function could try each pattern in turn until one matches. Then it would return the index of the matching pattern or false if there were no matches. This would let you define multiple types of lines if you had a complex file format. You’d probably want to have different sets of field tags for each one, too.<\/p>\n
I have a long history of abusing tools like awk<\/code> to do things, like build cross assemblers<\/a>. Even so, I’m probably not the worst offender<\/a>.<\/p>\n
source https:\/\/hackaday.com\/2020\/09\/29\/linux-fu-making-awk-a-bit-easier\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
awk is a kind of Swiss Army knife for text files. However, some of its limitations are often a bit annoying. I’ve used a simple set of functions to make awk a bit better, although I will warn you: it…<\/span><\/p>\n
Read more Linux-Fu: Making AWK a Bit Easier<\/span> ›<\/span><\/a><\/div>\n
<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[66],"tags":[],"class_list":["post-22958","post","type-post","status-publish","format-standard","hentry","category-news-2"],"_links":{"self":[{"href":"http:\/\/cornay.co.za\/wp\/wp-json\/wp\/v2\/posts\/22958","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/cornay.co.za\/wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/cornay.co.za\/wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/cornay.co.za\/wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/cornay.co.za\/wp\/wp-json\/wp\/v2\/comments?post=22958"}],"version-history":[{"count":1,"href":"http:\/\/cornay.co.za\/wp\/wp-json\/wp\/v2\/posts\/22958\/revisions"}],"predecessor-version":[{"id":22963,"href":"http:\/\/cornay.co.za\/wp\/wp-json\/wp\/v2\/posts\/22958\/revisions\/22963"}],"wp:attachment":[{"href":"http:\/\/cornay.co.za\/wp\/wp-json\/wp\/v2\/media?parent=22958"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/cornay.co.za\/wp\/wp-json\/wp\/v2\/categories?post=22958"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/cornay.co.za\/wp\/wp-json\/wp\/v2\/tags?post=22958"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}