“Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.”
–Martin Golding

Quotes like above tell us the importance of testing the modules that we develop. Testing for different languages is done using various tools for example JUNIT for java, FunSuite for scala and TestRig (grun on command line) for grammar(.g4). In this blog our focus will be on understanding what ANTLR is and mainly testing the correctness of an input(string generally) with respect to the developed grammar using Grun.

ANTLR (ANother Tool For Language Recognition) is a tool that converts grammars into programs (java programs for now) that recognize sentences in the language described by the grammar. For example, given a grammar for JSON, the ANTLR tool generates java files that recognizes JSON input using some support classes from the ANTLR run time library.

The blog is divided into 2 sections :

1). The first section covers testing of the input JSON string  with respect to the grammar rules defined in ‘JSONGrammar.g4’ file (control your nerves and keep reading) using Ubuntu shell.

2). In second section we do the unit testing of ‘JSONGrammarParser.java’ and JSONGrammarLexer.java (will be covered in the next blog) using IntelliJ Idea

Section 1 :

In This section we have following prerequisites :
1). Java 1.6 or above
2). Latest ANTLR4 JAR (download here)[http://www.antlr.org/download/antlr-4.5.3-complete.jar]

Following steps will guide you to the glory of grammar testing via Ubuntu shell :

STEP 1 “Checking prerequisites” ->

a). Open console and run following command

$ java -version

it yields (on my system) :

java version "1.8.0_72"

if the version is below 1.6, install the latest java version from here [Hit me for java 8]

b). You can add ANTLR4 jar by following ways:

[I]. Add jar by downloading manually. Latest version being ‘antlr-4.5.3-complete.jar'[Hit me for ANTLR4 jar]. Add it to ‘/usr/local/lib’ (just a convention put it anywhere you like but take care of the classpath of the jar)

[II]. Download using shell via following command

$ cd /usr/local/lib          //here our jars will be downloaded
$ curl -O http://www.antlr.org/download/antlr-4.5-complete.jar          //curl request to dowload the latest version

If some error like this appear :

curl: (23) Failed writing body (0 != 847)
grant permission to the access the folder and sub folders of the lib directory by changing the owner from root :

 $ cd (go to root folder)
$ cd /usr/local
$ sudo chown -R sahil lib          //here sahil is the owner on my system with r-w-x permission
$ cd
$ cd /usr/local/lib
$ curl -O http://www.antlr.org/download/antlr-4.5-complete.jar         //Yes i see the jar in lib folder 'happy...'

STEP 2 “First Example” (generating JSON grammar) ->

Lets make a temporary directory in home and name it ‘antlr4_demo’ now create a new file in it and save it as ‘JSONGrammar.g4’. The file contains the grammar with rules for a valid JSON statement.

Content of ‘JSONGrammar.g4’ ->

grammar JSONGrammar;

json
: object
| array
;

object
: '{' pair (',' pair)* '}'
| '{' '}'
;

pair
: STRING ':' value
;

array
: '[' value (',' value)* ']'
| '[' ']'
;

value
: STRING
| NUMBER
| object
| array
| 'true'
| 'false'
| 'null'
;


STRING
: '"' (ESC | ~ ["\\])* '"'
;
fragment ESC
: '\\' (["\\/bfnrt] | UNICODE)
;
fragment UNICODE
: 'u' HEX HEX HEX HEX
;
fragment HEX
: [0-9a-fA-F]
;
NUMBER
: '-'? INT '.' [0-9] + EXP? | '-'? INT EXP | '-'? INT
;
fragment INT
: '0' | [1-9] [0-9]*
;
// no leading zeros
fragment EXP
: [Ee] [+\-]? INT
;
// \- since - means "range" inside [...]
WS
: [ \t\n\r] + -> skip
;

After defining the grammar, open Terminal and write :

$ cd antlr4_demo          //enter in the folder in which our grammar file is present

STEP 3 “Exporting classpath of the jar and creating alias” ->

After installing the jar we can use it in any folder where our grammar file (.g4) is present to do so we export the class path for the antlr4 jar as follows :

$ export CLASSPATH=".:/usr/local/lib/antlr-4.5-complete.jar:$CLASSPATH"

Here since we have our jar file in /usr/local/lib, we exported the classpath like this. Adjust the class path on basis of the location of jar in your system.

Now we define some alias to save some typing (alias are my heroes)

$ alias antlr4='java -Xmx500M -cp "/usr/local/lib/antlr-4.5-complete.jar:$CLASSPATH" org.antlr.v4.Tool'
$ alias grun='java org.antlr.v4.runtime.misc.TestRig'

STEP 4 “Testing if jar is accessible or not” ->

Either launch org.antlr.v4.Tool directly:

 $ java org.antlr.v4.Tool

or use -jar option on java:

$ java -jar /usr/local/lib/antlr-4.5-complete.jar

or just write:

$ antlr4

and it shows :

ANTLR Parser Generator Version 4.5
-o ___ and so on…

STEP 5 “running the antlr tool on file” ->

In this step we run the antlr tool on the .g4 file and generate java files which have the corresponding java code for the grammar. It is done by :

$ antlr4 JSONGrammar.g4            //here antlr4 is the alias which we created above

and it will auto generate the following files in ‘antlr4_demo’ folder (dont belive me? check the folder yourself).

JSONGrammar.tokens
JSONGrammarBaseListener.java
JSONGrammarLexer.java
JSONGrammarLexer.tokens
JSONGrammarListener.java
JSONGrammarParser.java

Here corresponding parser and lexer classes for the grammar are created on basis of which Grun will test the inputs for being grammatically correct or not. Now we compile all the java files by :

$ javac JSONGrammar*.java

STEP 6 “all set to test” ->

Here TestRig or Grun (for shell) is used to test the grammar. Once you’ve compiled a lexer/parser (the auto generated classes), you can use ANTLR’s TestRig to display various views of the parse tree. The first argument to TestRig is the name of a grammar (i.e. the name of the java package implementing the parser); the second argument is the name of a rule to be used to start parsing. This means you can test parts of your grammar independently. The optional third parameter is a file name; if you provide it, TestRig will feed its contents to the parser.

To test a string for being a valid JSON follow to voyage :

$ grun JSONGrammar json -tree           //here grun is the alias which we created above

Now enter the json string in the shell to validate it. For example :-

{“name”:”sahil”,”age”:24}

After entering the json string hit enter and ctrl+d which signifies end of line (and Boom!!! you get your result).
Since here the the grammar we entered is correct it will generate parse tree for the grammar and prints it on the console :

(json (object { (pair “name” : (value “sahil”)) , (pair “age” : (value 24)) }))
Now let us test a negative json string and find out what happens :

$ grun JSONGrammar json -tree
{"name"="sahil","age"=24}         //replacing ':' with '=' which makes it an invalid json
ctrl+d

It yields :

line 1:7 token recognition error at: '='
line 1:21 token recognition error at: '='
line 1:8 missing ':' at '"sahil"'
line 1:22 missing ':' at '24'
(json (object { (pair "name" <missing ':'> (value "sahil")) , (pair "age" <missing ':'> (value 24)) }))

Yeah that is awesome, it tells where things went wrong and why they went wrong

You can also pick the input from a file. To do so create a file say ‘json_file.txt’ in ‘antlr4_demo’ folder which contains following JSON text :

{“name”:”sahil”,”age”:24}

$ grun JSONGrammar json json_file.txt -tree

it will give following output:

(json (object { (pair "name" : (value "sahil")) , (pair "age" : (value 24)) }))

Instead of building a textual tree we can also have the GUI version of the tree which can be achieved by:

$ grun JSONGrammar json -gui            //we just replaced '-tree' with '-gui'

rest process is same ie entering the json string and ‘ctrl+d’

Now, after successfully completing the first section we now prepare to kill section 2 in the next blog where we use Intellij and do unit testing of the grammar using TestRig.

References:
https://github.com/antlr/antlr4/blob/master/doc/getting-started.md

happy coding…