Shell script – Suspension of Disbelief

I’m in love with the Crafting Interpreters book. In it, Bob Nystrom teach us how to writer an interpreter by implementing a little programming language called Lox. It was a long time since I had so much fun programming! Besides being well-written, the book is funny and teach way more than I would expect. But I have a problem.

The snippets in the bug are written in a way we can copy and paste them. However, the book has challenges at the end of each chapter, these challenges have no source code and sometime they force us to change the interpreter a lot. I do every one of these exercises and as a result my interpreter diverges too much from the source in the book. Consequently, I often break some part of my interpreter.

How to solve that?

Unity tests would be brittle since the code structure changes frequently. End-to-end tests seem more practical in this case. So, for each new feature of the language, I wrote a little program. For example, my interpreter should create closures, and to ensure that I copied the Lox program below to the file counter.lox:

This program result should be the numbers 1 and 2 printed in different lines. So I put these values in a file called counter.lox.out. The program cannot fail either, so I created an empty file called counter.lox.err. (In some cases, it is necessary to ensure the Lox program will fail. In these cases, the file .lox.err should have content.)

Well, I wrote programs and output files for various examples; now I need to compare the programs’ results to the expected outputs. I decided to use the tool that helps me the most in urgent times: shell script. I did a Bash script with a for iterating over all examples:

For each example, I executed the Lox program, redirecting the outputs to temporary files:

Now, we compare the real output with the expected output through diff. When it compares two files, diff returns 0 if there is no difference, 1 if there exists a difference or 2 in case of error. Since in Bash the conditional if considers 0 as true, we just check the negation of diff‘s exit code.

If the program prints something in standard output that is different from what is in its .lox.out file, we have a failure:

We also check the standard error and the .lox.err file:

Finally, I check if there was some failure and report the result:

Not all of my Lox programs can be checked, though. For example, there is a program which times loop executions, it is impossible to anticipate the value it will print. Because of that, I added the possibility to jump some programs: we need just to create a file with the .lox.skip extension:

If, however, I have a Lox example and it does not have expected output files (nor the .lox.skip file) then I have a problem and the entire script fails:

With that, my test script is done. Let us see how it behaves:

$ ./lcheck.sh PASS examples/attr.lox PASS examples/bacon.lox PASS examples/badfun.lox PASS examples/badret.lox PASS examples/bagel.lox PASS examples/bostoncream.lox PASS examples/cake.lox PASS examples/checkuse.lox PASS examples/circle2.lox PASS examples/circle.lox 1d0 < 3 1c1 < --- > [line 1] Error at ',': Expect ')' after expression. FAIL examples/comma.lox PASS examples/counter.lox PASS examples/devonshinecream.lox PASS examples/eclair.lox PASS examples/fibonacci2.lox PASS examples/fibonacci.lox PASS examples/func.lox PASS examples/funexprstmt.lox PASS examples/hello2.lox PASS examples/hello3.lox PASS examples/hello.lox PASS examples/math.lox PASS examples/notaclass.lox PASS examples/noteveninaclass.lox PASS examples/point.lox PASS examples/retthis.lox PASS examples/scope1.lox PASS examples/scope.lox PASS examples/supersuper.lox PASS examples/thisout.lox PASS examples/thrice.lox SKIP examples/timeit.lox PASS examples/twovars.lox PASS examples/usethis.lox PASS examples/varparam.lox

Oops, apparently I removed the support for the comma operator by accident. Good thing I wrote this script, right?

I hope this post was minimally interesting! Now, I am going to repair my comma operator and keep reading this wonderful book.

(This post is a translation of Não me Interprete Mal: Improvisando Testes para um Interpretador.)

One of my hobbies during this recent World Cup was to collect stickers. Actually, I’ve built the sticker album because my son wanted it but I had fun, too, I guess.

2018 sticker album showing France team missing three pictures. — Sadly, not completed yet

An important part of collecting stickers is to exchange the repeated ones. Through messages in WhatsApp groups, we report which repeated stickers we have and which ones we still need. As a programmer, I refused to compare the lists myself, so I wrote a little program em Python (with doctests and all) to find intersections.

The missing laptop

Last week, a person came to my home to exchange stickers. I had the lists of repeated and needed cards, both mine and hers, but my script was in another laptop. I did not even know where this machine was and my guest was in a hurry.

There was no time to find the computer, or rewriting the program. Or even to compare manually.

It’s Unix time!

The list format

In general, the lists had this format:

15, 18, 26, 31, 40, 45 (2), 49, 51, 110, 115, 128, 131 (2), 143, 151, 161, 162, 183 (2), 216 (2), 221, 223, 253, 267 (3), 269, 280, 287, 296, 313, 325, 329, 333 (2), 353 (3), 355, 357, 359, 362, 365, 366, 371, 373, 384, 399, 400, 421 (2), 445, 457, 469, 470, 498 (2), 526, 536, 553, 560, 568, 570, 585, 591 (2), 604 (2), 639 (2), 660.

Basically, I needed to remove everything which were not digits, alongside with the numbers in parentheses, and to compare both lists. Easy, indeed.

Pre-processing with sed

First, I had to remove the counters between parentheses:

$ cat list.txt | sed 's/([^)]*)//g' 15, 18, 26, 31, [...] 591 , 604 , 639 , 660.

(I know, UUOC. Whatever.)

Then, I put each number in its own line:

$ cat list.txt | sed 's/([^)]*)//g' | sed 's/, */\n/g'

Later, I clean up every line removing any character that is not a digit:

cat list.txt | sed 's/([^)]*)//g' | sed 's/, */\n/g' | sed 's/[^0-9]*$[0-9]*$[^0-9]*/\1/g'

In practice, I only call sed once, passing up both expressions. Here, I believe it would be clearer to invoke sed many times.)

Finally, I sort the values:

$ cat list.txt | sed 's/([^)]*)//g' | sed 's/, */\n/g' | sed 's/[^0-9]*$[0-9]*$[^0-9]*/\1/g' | sort -n > mine-needed.txt

I do it with the list of needed stickers, and also with the list of repeated stickers, getting two files.

Finding intersections with grep

Now, I need to compare them. There are many options, and I choose to use grep.

In this case, I called grep with one of the files as an input, and the other file as a list of patterns to match, through the -f option. Also, only the complete match matters here, so we are going to use the -x flag. Finally, I asked grep to compare strings directly (instead of treating them as regular expressions) with the -F flag.

$ fgrep -Fxf mine-needed.txt theirs-repeated.txt 253 269 333 470 639

Done! In a minute, I already know which stickers I want. I just need to do the same with my repeated ones.

Why is this interesting?

These one-liners are not really a big deal to me, today. The interesting thing is that when I started to use the terminal, they would be incredible. Really, look how many pipes we use to pre-process the files! And this grep trick? I suffered to merely create a regex which worked! Actually, until solving this problem, I did not even know the -x option.

I once helped a friend to process a good number of files. He already spent more than two hours trying to do it with Java, and we solved it together in ten minutes with shell script. He then asked me how much he wanted to know shell script and asked me how to learn it.

Well, little examples (like this one), as simple as they seem, taught me a lot. This is how I learned to script: trying to solve problems, knowing new commands and options in small batches. In the end, this is a valuable skill.

So, I hope this little toying enrich your day, too. I certainly enriched mine — I’d like to think about it before spending three times more time with my Python script!

This post is a translation of Trocando figurinhas sobre o terminal.

Category: Shell script

Don’t Interpret Me Wrong: Improvising Tests for an Interpreter

Exchanging World Cup’s sticker figures with the terminal

The missing laptop

The list format

Pre-processing with sed

Finding intersections with grep

Why is this interesting?