6f81608535
The lexer needs to always return braces and brackets separately so that
the parser can decide if they are part of some construct like an array
subscript or a variable expansion. This means that there was no point
in moving bare-word tokenization into the external scanner. I've moved
it back into the normal scanner.
The tricky part is how to deal with the separate '}' and ']' tokens
in the case where they are *not* part of a subscript or an expansion.
For example, in code like `echo {hi}`, the syntax tree should still
clearly indicate that only *one* argument is being passed to echo.
For now, we achieve this by grouping the '{', hi, and '}' tokens into
a single `concatenation` node, which is a bit odd, but it's the best
we can do.
|
||
|---|---|---|
| .. | ||
| tree_sitter | ||
| binding.cc | ||
| grammar.json | ||
| parser.c | ||
| scanner.cc | ||