TantrixAuto/yantra: A C++ parser generator: lexer, parser, and AST technology in a single software. · GitHub
Yantra is a robust compiler compiler and LALR(1) parser generator written in C++, with the next core options:
- An built-in lexer
- Built-in help for UNICODE/UTF8 enter
- Built-in AST builder
- Built-in AST walker(s)
- Bottom-up parsing (being LALR), and top-down strolling (traversal)
- Multi-mode lexer, helpful for implementing nested multi-line feedback, and many others.
- Lexer-driven (push-based) parser: reads enter one character at a time and feeds tokens to the parser as they full, helpful for processing enter because it arrives (e.g. from a socket).
- An non-compulsory amalgamated mode, the place your entire parser is generated as a single cpp file, together with a full-featured foremost() operate.
- Or, in non-amalgamated mode, the parser is generated as separate .hpp and .cpp information, able to drop into an present undertaking.
The title Yantra is Sanskrit for machine, as in state machine on this context.
Yantra has no dependencies past the C++ customary library, so constructing it’s a plain CMake construct:
git clone [email protected]:TantrixAuto/yantra.git
cd yantra
mkdir construct && cd construct
cmake ..
cmake --build .
This produces the ycc executable in bin/. Save a grammar file, good day.y:
begin := stmts;
stmts := stmts stmt;
stmts := stmt;
stmt := ID;
ID := "[A-Za-z]+";
WS := "s"!;
Then generate a parser from it:
bin/ycc -c ascii -f good day.y -a
This writes good day.cpp (an amalgamated, self-contained parser with its personal foremost()) and good day.log. Compile it with any C++23 compiler:
# clang
clang++ --std=c++23 -o good day good day.cpp
# gcc
g++ --std=c++23 -o good day good day.cpp
# MSVC (cl.exe, from a Developer Command Prompt)
cl /std:c++23 /EHsc /nologo good day.cpp
The grammar above acknowledges a number of whitespace-separated alphabetic phrases. -s feeds that string on to the parser as enter (versus -f , which reads from a file, or -i, which reads interactively from the console):
# succeeds silently
$ ./good day -s "good day global community"
$ echo $?
0
# -t1 prints the parsed AST
$ ./good day -s "good day global community" -t1
0:start_1(1:stmts_1(2:stmts_2(3:stmt_1(4:ID(good day))) 2:stmt_1(3:ID(global community))) 1:_tEND())
# fails: ID solely matches letters, "123" is not legitimate enter for this grammar
$ ./good day -s "good day 123"
s1-err:?a1.in(001,007):TOKEN_ERROR{{token: }}
good day 123
$ echo $?
1
Yantra parses your entire enter into an AST first, then walks it top-down calling your semantic actions, not like most parser mills, the place actions run bottom-up as every rule is decreased. That ordering is what lets a mum or dad rule’s motion run earlier than its kids are visited. Save this as calc.y:
%class Calculator;
begin := expr;
expr := expr(a) PLUS expr(b)
%{
std::cout << "Adding" << std::endl;
%}
expr := NUMBER(N)
%{
std::cout << "Number: " << N.textual content << std::endl;
%}
NUMBER := "d+";
PLUS := "+";
WS := "s+"!;
Generate and compile it the identical means as above (bin/ycc -c ascii -f calc.y -a, then any of the three compiler instructions), then run it:
$ ./calc -s "1 + 2 + 3"
Adding
Number: 1
Adding
Number: 2
Number: 3
1 + 2 + 3 parses left-associatively as (1 + 2) + 3. So the outer Adding, the foundation of the tree, prints first, adopted by its left little one (Number: 1) after which its proper little one, which is itself one other Adding node with its personal two kids. A hand-written recursive-descent or bottom-up parser must construct additional AST lessons and a separate strolling cross to get this ordering. Here it falls out of the grammar straight.
See the Build Instructions and Tutorial beneath for an actual walk-through of the grammar syntax.
- vs. Bison / Yacc / Lemon (the basic LALR(1) household):
- Lemon, from SQLite, is Yantra’s direct said inspiration.
- These run semantic actions throughout parsing, as every rule reduces, bottom-up.
- Yantra all the time builds the total AST first, then walks it top-down in a separate cross, so a mum or dad rule’s motion can run earlier than its kids are visited.
- A single grammar can even outline multiple walker (e.g. one which emits C++, one other that emits Java, from the identical parse).
- Getting both of these out of the Bison household means hand-building your personal AST and walker on prime.
- vs. ANTLR:
- ANTLR walks a fully-built parse tree too, however that comes free of charge from its LL(*) algorithm, which already builds the tree top-down because it parses.
- Yantra will get the identical top-down stroll out of LALR(1), a bottom-up algorithm with no pure “entire tree exists but” second throughout parsing, whereas maintaining LALR(1)’s time and house effectivity over adaptive LL(*).
- Beyond that, Yantra targets C++ solely (ANTLR generates for a lot of languages) and ships its personal built-in lexer with mode-stack help as an alternative of a separate lexer generator.
- ANTLR’s personal generator software is Java, so utilizing it from a C++ undertaking means including a JVM to the construct toolchain simply to run the generator. Yantra is a local C++ executable with no such dependency.
- ANTLR is much extra mature and extensively used. Yantra is a a lot smaller, newer, single-maintainer undertaking.
- vs. tree-sitter:
- A distinct downside totally. It’s constructed for incremental, error-tolerant parsing embedded in editors and IDEs (what GitHub, Neovim, and many others. use it for), not for producing a compiler/codegen backend.
- Yantra would not do incremental reparsing and is not attempting to.
See Known Limitations for an sincere record of what Yantra would not do but.
The following are a set of key hyperlinks to get aware of Yantra.
It is advisable that they be learn within the given order.
See https://github.com/TantrixAuto/lingo for standalone pattern undertaking that makes use of yantra.
This is a language server extension created by Raj Chaudhuri that gives syntax highlighting for Yantra information in vscode, qtcreator, and some other IDE that helps the Language Server Protocol.
https://github.com/rajware/yantra-language-server
Yantra is licensed below the MIT License.
Renji Panicker (@renjipanicker)
