{"id":58,"date":"2018-08-07T14:57:44","date_gmt":"2018-08-07T14:57:44","guid":{"rendered":"http:\/\/suspensao.blog.br\/disbelief\/?p=58"},"modified":"2021-12-08T14:07:50","modified_gmt":"2021-12-08T17:07:50","slug":"dont-interpret-me-wrong-improvising-tests-for-an-interpreter","status":"publish","type":"post","link":"https:\/\/suspensao.blog.br\/disbelief\/dont-interpret-me-wrong-improvising-tests-for-an-interpreter\/","title":{"rendered":"Don&#8217;t Interpret Me Wrong: Improvising Tests for an Interpreter"},"content":{"rendered":"<p>I&#8217;m in love with the <a href=\"http:\/\/craftinginterpreters.com\/\"><em>Crafting Interpreters<\/em><\/a> book. In it, <a href=\"http:\/\/journal.stuffwithstuff.com\/\">Bob Nystrom<\/a> teach us how to writer an interpreter by implementing a little programming language called <a href=\"http:\/\/craftinginterpreters.com\/the-lox-language.html\">Lox<\/a>. It was a long time since I had so much fun programming! Besides being well-written, the book is funny and teach way more than I would expect. But I have a problem.<\/p>\n<p>The snippets in the bug are written in a way we can copy and paste them. However, the book has challenges at the end of each chapter, these challenges have no source code and sometime they force us to change the interpreter a lot. I do every one of these exercises and as a result my interpreter diverges too much from the source in the book. Consequently, I often break some part of my interpreter.<\/p>\n<p>How to solve that?<\/p>\n<p>Unity tests would be brittle since the code structure changes frequently. <a href=\"https:\/\/www.techopedia.com\/definition\/7035\/end-to-end-test\">End-to-end tests<\/a> seem more practical in this case. So, for each new feature of the language, I wrote a little program. For example, my interpreter should create <a href=\"http:\/\/craftinginterpreters.com\/functions.html#local-functions-and-closures\">closures<\/a>, and to ensure that I copied the Lox program below to the file <a href=\"https:\/\/bitbucket.org\/brandizzi\/jlox\/src\/default\/examples\/counter.lox?at=default&amp;fileviewer=file-view-default\"><code>counter.lox<\/code><\/a>:<\/p>\n<p><script src=\"https:\/\/pastebin.com\/embed_js\/fzCFuC1k\"><\/script><noscript>&lt;\/p&gt;<br \/>\n&lt;pre&gt;&lt;code&gt;fun makeCounter() {<br \/>\n  var i = 0;<br \/>\n  fun count() {<br \/>\n    i = i + 1;<br \/>\n    print i;<br \/>\n  }<\/p>\n<p>  return count;<br \/>\n}<\/p>\n<p>var counter = makeCounter();<br \/>\ncounter(); \/\/ &#8220;1&#8221;.<br \/>\ncounter(); \/\/ &#8220;2&#8221;.&lt;\/code&gt;&lt;\/pre&gt;<br \/>\n&lt;p&gt;<\/noscript><\/p>\n<p>This program result should be the numbers 1 and 2 printed in different lines. So I put these values in a file called <a href=\"https:\/\/bitbucket.org\/brandizzi\/jlox\/src\/default\/examples\/counter.lox.out\"><code>counter.lox.out<\/code><\/a>. The program cannot fail either, so I created an empty file called <a href=\"https:\/\/bitbucket.org\/brandizzi\/jlox\/src\/default\/examples\/counter.lox.out\"><code>counter.lox.err<\/code><\/a>. (In some cases, it is necessary to ensure the Lox program will fail. In these cases, the file <code>.lox.err<\/code> should have content.)<\/p>\n<p>Well, I wrote programs and output files for various examples; now I need to compare the programs&#8217; results to the expected outputs. I decided to use the tool that <a href=\"http:\/\/suspensao.blog.br\/disbelief\/exchanging-sticker-cards-with-the-terminal\/\">helps me the most in urgent times<\/a>: shell script. I did a Bash script with a <code>for<\/code> iterating over all examples:<\/p>\n<p><script src=\"https:\/\/pastebin.com\/embed_js\/hwW7nCM0\"><\/script><noscript>&lt;\/p&gt;<br \/>\n&lt;pre&gt;&lt;code&gt;for l in examples\/*.lox<br \/>\ndo<\/p>\n<p>done&lt;\/code&gt;&lt;\/pre&gt;<br \/>\n&lt;p&gt;<\/noscript><\/p>\n<p>For each example, I executed the Lox program, redirecting the outputs to temporary files:<br \/>\n<script src=\"https:\/\/pastebin.com\/embed_js\/LDD3LjEk\"><\/script><noscript>&lt;\/p&gt;<br \/>\n&lt;pre&gt;&lt;code&gt;for l in examples\/*.lox<br \/>\ndo<br \/>\n  out=$(mktemp)<br \/>\n  err=$(mktemp)<br \/>\n  java -classpath target\/classes\/ br.com.brandizzi.adam.myjlox.Lox $l &amp;gt; $out 2&amp;gt; $err<br \/>\ndone&lt;\/code&gt;&lt;\/pre&gt;<br \/>\n&lt;p&gt;<\/noscript><\/p>\n<p>Now, we compare the real output with the expected output through <a href=\"https:\/\/www.gnu.org\/software\/diffutils\/\"><code>diff<\/code><\/a>. When it compares two files, <a href=\"https:\/\/stackoverflow.com\/questions\/6971284\/what-are-the-error-exit-values-for-diff\"><code>diff<\/code> returns 0 if there is no difference, 1 if there exists a difference or 2 in case of error<\/a>. Since in Bash the conditional <code>if<\/code> considers 0 as true, we just check the negation of <code>diff<\/code>&#8216;s exit code.<\/p>\n<p>If the program prints something in standard output that is different from what is in its <code>.lox.out<\/code> file, we have a failure:<\/p>\n<p><script src=\"https:\/\/pastebin.com\/embed_js\/Z8Fk2EyN\"><\/script><noscript>&lt;\/p&gt;<br \/>\n&lt;pre&gt;&lt;code&gt;for l in examples\/*.lox<br \/>\ndo<br \/>\n  out=$(mktemp)<br \/>\n  err=$(mktemp)<br \/>\n  java -classpath target\/classes\/ br.com.brandizzi.adam.myjlox.Lox $l &amp;gt; $out 2&amp;gt; $err<\/p>\n<p>  if ! diff $l.out $out<br \/>\n  then<br \/>\n    FAIL=1<br \/>\n  fi<br \/>\ndone&lt;\/code&gt;&lt;\/pre&gt;<br \/>\n&lt;p&gt;<\/noscript><\/p>\n<p>We also check the standard error and the <code>.lox.err<\/code> file:<\/p>\n<p><script src=\"https:\/\/pastebin.com\/embed_js\/GbBJE4ff\"><\/script><noscript>&lt;\/p&gt;<br \/>\n&lt;pre&gt;&lt;code&gt;for l in examples\/*.lox<br \/>\ndo<br \/>\n  out=$(mktemp)<br \/>\n  err=$(mktemp)<br \/>\n  java -classpath target\/classes\/ br.com.brandizzi.adam.myjlox.Lox $l &amp;gt; $out 2&amp;gt; $err<\/p>\n<p>  if ! diff $l.out $out<br \/>\n  then<br \/>\n    FAIL=1<br \/>\n  fi<\/p>\n<p>  if ! diff $l.err $err<br \/>\n  then<br \/>\n    FAIL=1<br \/>\n  fi<br \/>\ndone&lt;\/code&gt;&lt;\/pre&gt;<br \/>\n&lt;p&gt;<\/noscript><\/p>\n<p>Finally, I check if there was some failure and report the result:<br \/>\n<script src=\"https:\/\/pastebin.com\/embed_js\/XagVcuG2\"><\/script><noscript>&lt;\/p&gt;<br \/>\n&lt;pre&gt;&lt;code&gt;for l in examples\/*.lox<br \/>\ndo<br \/>\n  out=$(mktemp)<br \/>\n  err=$(mktemp)<br \/>\n  java -classpath target\/classes\/ br.com.brandizzi.adam.myjlox.Lox $l &amp;gt; $out 2&amp;gt; $err<\/p>\n<p>  if ! diff $l.out $out<br \/>\n  then<br \/>\n    FAIL=1<br \/>\n  fi<\/p>\n<p>  if ! diff $l.err $err<br \/>\n  then<br \/>\n    FAIL=1<br \/>\n  fi<\/p>\n<p>  if [ &amp;quot;$FAIL&amp;quot; = &amp;quot;1&amp;quot; ]<br \/>\n  then<br \/>\n    echo &amp;quot;FAIL&amp;quot; $l<br \/>\n  else<br \/>\n    echo &amp;quot;PASS&amp;quot; $l<br \/>\n  fi<br \/>\ndone&lt;\/code&gt;&lt;\/pre&gt;<br \/>\n&lt;p&gt;<\/noscript><\/p>\n<p>Not all of my Lox programs can be checked, though. For example, there is <a href=\"https:\/\/bitbucket.org\/brandizzi\/jlox\/src\/default\/examples\/timeit.lox\">a program which times loop execution<\/a>s, it is impossible to anticipate the value it will print. Because of that, I added the possibility to jump some programs: we need just to create <a href=\"https:\/\/bitbucket.org\/brandizzi\/jlox\/src\/default\/examples\/timeit.lox.skip\">a file with the <code>.lox.skip<\/code> extension<\/a>:<\/p>\n<p><script src=\"https:\/\/pastebin.com\/embed_js\/MM30zc01\"><\/script><noscript>&lt;\/p&gt;<br \/>\n&lt;pre&gt;&lt;code&gt;for l in examples\/*.lox<br \/>\ndo<br \/>\n  if [ -e $l.skip ]<br \/>\n  then<br \/>\n    echo SKIP $l<br \/>\n    continue<br \/>\n  fi<\/p>\n<p>  out=$(mktemp)<br \/>\n  err=$(mktemp)<br \/>\n  java -classpath target\/classes\/ br.com.brandizzi.adam.myjlox.Lox $l &amp;gt; $out 2&amp;gt; $err<\/p>\n<p>  if ! diff $l.out $out<br \/>\n  then<br \/>\n    FAIL=1<br \/>\n  fi<\/p>\n<p>  if ! diff $l.err $err<br \/>\n  then<br \/>\n    FAIL=1<br \/>\n  fi<\/p>\n<p>  if [ &amp;quot;$FAIL&amp;quot; = &amp;quot;1&amp;quot; ]<br \/>\n  then<br \/>\n    echo &amp;quot;FAIL&amp;quot; $l<br \/>\n  else<br \/>\n    echo &amp;quot;PASS&amp;quot; $l<br \/>\n  fi<br \/>\ndone&lt;\/code&gt;&lt;\/pre&gt;<br \/>\n&lt;p&gt;<\/noscript><\/p>\n<p>If, however, I have a Lox example and it does not have expected output files (nor the <code>.lox.skip<\/code> file) then I have a problem and the entire script fails:<\/p>\n<p><script src=\"https:\/\/pastebin.com\/embed_js\/arAxZYAN\"><\/script><noscript>&lt;\/p&gt;<br \/>\n&lt;pre&gt;&lt;code&gt;for l in examples\/*.lox<br \/>\ndo<br \/>\n  if [ -e $l.skip ]<br \/>\n  then<br \/>\n    echo SKIP $l<br \/>\n    continue<br \/>\n  elif [ ! -e $l.out ] || [ ! -e $l.err ]<br \/>\n  then<br \/>\n    echo missing $l.out or $l.err<br \/>\n    exit 1<br \/>\n  fi<\/p>\n<p>  out=$(mktemp)<br \/>\n  err=$(mktemp)<br \/>\n  java -classpath target\/classes\/ br.com.brandizzi.adam.myjlox.Lox $l &amp;gt; $out 2&amp;gt; $err<\/p>\n<p>  if ! diff $l.out $out<br \/>\n  then<br \/>\n    FAIL=1<br \/>\n  fi<\/p>\n<p>  if ! diff $l.err $err<br \/>\n  then<br \/>\n    FAIL=1<br \/>\n  fi<\/p>\n<p>  if [ &amp;quot;$FAIL&amp;quot; = &amp;quot;1&amp;quot; ]<br \/>\n  then<br \/>\n    echo &amp;quot;FAIL&amp;quot; $l<br \/>\n  else<br \/>\n    echo &amp;quot;PASS&amp;quot; $l<br \/>\n  fi<br \/>\ndone&lt;\/code&gt;&lt;\/pre&gt;<br \/>\n&lt;p&gt;<\/noscript><\/p>\n<p>With that, my test script is done. Let us see how it behaves:<\/p>\n<p><code>$ .\/lcheck.sh<br \/>\nPASS examples\/attr.lox<br \/>\nPASS examples\/bacon.lox<br \/>\nPASS examples\/badfun.lox<br \/>\nPASS examples\/badret.lox<br \/>\nPASS examples\/bagel.lox<br \/>\nPASS examples\/bostoncream.lox<br \/>\nPASS examples\/cake.lox<br \/>\nPASS examples\/checkuse.lox<br \/>\nPASS examples\/circle2.lox<br \/>\nPASS examples\/circle.lox<br \/>\n1d0<br \/>\n&lt; 3<br \/>\n1c1<br \/>\n&lt;<br \/>\n---<br \/>\n&gt; [line 1] Error at ',': Expect ')' after expression.<br \/>\nFAIL examples\/comma.lox<br \/>\nPASS examples\/counter.lox<br \/>\nPASS examples\/devonshinecream.lox<br \/>\nPASS examples\/eclair.lox<br \/>\nPASS examples\/fibonacci2.lox<br \/>\nPASS examples\/fibonacci.lox<br \/>\nPASS examples\/func.lox<br \/>\nPASS examples\/funexprstmt.lox<br \/>\nPASS examples\/hello2.lox<br \/>\nPASS examples\/hello3.lox<br \/>\nPASS examples\/hello.lox<br \/>\nPASS examples\/math.lox<br \/>\nPASS examples\/notaclass.lox<br \/>\nPASS examples\/noteveninaclass.lox<br \/>\nPASS examples\/point.lox<br \/>\nPASS examples\/retthis.lox<br \/>\nPASS examples\/scope1.lox<br \/>\nPASS examples\/scope.lox<br \/>\nPASS examples\/supersuper.lox<br \/>\nPASS examples\/thisout.lox<br \/>\nPASS examples\/thrice.lox<br \/>\nSKIP examples\/timeit.lox<br \/>\nPASS examples\/twovars.lox<br \/>\nPASS examples\/usethis.lox<br \/>\nPASS examples\/varparam.lox<\/code><\/p>\n<p>Oops, apparently I removed the support for the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Comma_operator\">comma operator<\/a> by accident. Good thing I wrote this script, right?<\/p>\n<p>I hope this post was minimally interesting! Now, I am going to repair my comma operator and keep reading <a href=\"http:\/\/craftinginterpreters.com\/\">this wonderful book<\/a>.<\/p>\n<p><em>(This post is a translation of <a href=\"http:\/\/suspensao.blog.br\/descrenca\/nao-me-interprete-mal-improvisando-testes-para-um-interpretador\/\">N\u00e3o me Interprete Mal: Improvisando Testes para um Interpretador<\/a>.)<\/em> <em><br \/>\n<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;m in love with the Crafting Interpreters book. In it, Bob Nystrom teach us how to writer an interpreter by implementing a little programming language called Lox. It was a long time since I had so much fun programming! Besides being well-written, the book is funny and teach way more than I would expect. But &hellip; <a href=\"https:\/\/suspensao.blog.br\/disbelief\/dont-interpret-me-wrong-improvising-tests-for-an-interpreter\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Don&#8217;t Interpret Me Wrong: Improvising Tests for an Interpreter&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[21,23,22,10,20,1],"tags":[27,25,28,26,24,17,5],"class_list":["post-58","post","type-post","status-publish","format-standard","hentry","category-end-to-end-test","category-interpreters","category-programming-languages","category-shell-script","category-testing","category-uncategorized","tag-bash","tag-crafting-interpreters","tag-end-to-end-tests","tag-interpreters","tag-lox","tag-shell-script","tag-testing"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9Ru6q-W","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/suspensao.blog.br\/disbelief\/wp-json\/wp\/v2\/posts\/58","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/suspensao.blog.br\/disbelief\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/suspensao.blog.br\/disbelief\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/suspensao.blog.br\/disbelief\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/suspensao.blog.br\/disbelief\/wp-json\/wp\/v2\/comments?post=58"}],"version-history":[{"count":7,"href":"https:\/\/suspensao.blog.br\/disbelief\/wp-json\/wp\/v2\/posts\/58\/revisions"}],"predecessor-version":[{"id":92,"href":"https:\/\/suspensao.blog.br\/disbelief\/wp-json\/wp\/v2\/posts\/58\/revisions\/92"}],"wp:attachment":[{"href":"https:\/\/suspensao.blog.br\/disbelief\/wp-json\/wp\/v2\/media?parent=58"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/suspensao.blog.br\/disbelief\/wp-json\/wp\/v2\/categories?post=58"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/suspensao.blog.br\/disbelief\/wp-json\/wp\/v2\/tags?post=58"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}