shadowfacts
/
tree-sitter-bash
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
tree-sitter-bash/corpus/literals.txt

291 lines
5.9 KiB
Plaintext
Raw Normal View History

Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
=============================
Literal words
=============================
echo a
echo a b
---
(program
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command (command_name (word)) (word))
(command (command_name (word)) (word) (word)))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
Allow special characters in bare word arguments I've moved tokenization of bare words into the external scanner. This way we can keep the grammar simple, but support some fancy rules that I've inferred from experimenting with bash: - Only allow '}' inside of a bare word if '}' isn't a valid lookahead token (i.e. we're not inside of a variable expansion). - Only allow ']' at the *start* of a bare word if neither ']' nor ']]' are valid lookahead tokens (i.e. we're not inside of a square bracket command or an array subscript). - Parentheses seem to never be allowed in bare words. You have to quote them. For alphanumeric words, I fall through to the normal scanner so that it can continue to distinguish reserved words from other words. Fixes #5
2018-02-27 18:54:40 +00:00
=============================
Words with special characters
=============================
Always lex braces and brackets as separate tokens The lexer needs to always return braces and brackets separately so that the parser can decide if they are part of some construct like an array subscript or a variable expansion. This means that there was no point in moving bare-word tokenization into the external scanner. I've moved it back into the normal scanner. The tricky part is how to deal with the separate '}' and ']' tokens in the case where they are *not* part of a subscript or an expansion. For example, in code like `echo {hi}`, the syntax tree should still clearly indicate that only *one* argument is being passed to echo. For now, we achieve this by grouping the '{', hi, and '}' tokens into a single `concatenation` node, which is a bit odd, but it's the best we can do.
2018-02-28 19:13:49 +00:00
echo {o[k]}
Allow special characters in bare word arguments I've moved tokenization of bare words into the external scanner. This way we can keep the grammar simple, but support some fancy rules that I've inferred from experimenting with bash: - Only allow '}' inside of a bare word if '}' isn't a valid lookahead token (i.e. we're not inside of a variable expansion). - Only allow ']' at the *start* of a bare word if neither ']' nor ']]' are valid lookahead tokens (i.e. we're not inside of a square bracket command or an array subscript). - Parentheses seem to never be allowed in bare words. You have to quote them. For alphanumeric words, I fall through to the normal scanner so that it can continue to distinguish reserved words from other words. Fixes #5
2018-02-27 18:54:40 +00:00
echo }}}
echo ]]] ===
---
(program
Always lex braces and brackets as separate tokens The lexer needs to always return braces and brackets separately so that the parser can decide if they are part of some construct like an array subscript or a variable expansion. This means that there was no point in moving bare-word tokenization into the external scanner. I've moved it back into the normal scanner. The tricky part is how to deal with the separate '}' and ']' tokens in the case where they are *not* part of a subscript or an expansion. For example, in code like `echo {hi}`, the syntax tree should still clearly indicate that only *one* argument is being passed to echo. For now, we achieve this by grouping the '{', hi, and '}' tokens into a single `concatenation` node, which is a bit odd, but it's the best we can do.
2018-02-28 19:13:49 +00:00
(command (command_name (word)) (concatenation (word) (word)))
Allow special characters in bare word arguments I've moved tokenization of bare words into the external scanner. This way we can keep the grammar simple, but support some fancy rules that I've inferred from experimenting with bash: - Only allow '}' inside of a bare word if '}' isn't a valid lookahead token (i.e. we're not inside of a variable expansion). - Only allow ']' at the *start* of a bare word if neither ']' nor ']]' are valid lookahead tokens (i.e. we're not inside of a square bracket command or an array subscript). - Parentheses seem to never be allowed in bare words. You have to quote them. For alphanumeric words, I fall through to the normal scanner so that it can continue to distinguish reserved words from other words. Fixes #5
2018-02-27 18:54:40 +00:00
(command (command_name (word)) (word))
Add special lexing for regexes after `=~`
2018-03-01 17:53:10 +00:00
(command (command_name (word)) (word) (word)))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
=============================
Simple variable expansions
=============================
echo $abc
---
(program
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command (command_name (word)) (simple_expansion (variable_name))))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
Add special variable @, expressions in declarations
2018-03-01 01:32:48 +00:00
=============================
Special variable expansions
=============================
echo $# $* $@
---
(program
(command
(command_name (word))
(simple_expansion (special_variable_name))
(simple_expansion (special_variable_name))
(simple_expansion (special_variable_name))))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
=============================
Variable expansions
=============================
echo ${abc}
echo ${abc:-def}
Handle empty default value during variable expansion (#1) * Handle empty default value during variable expansion * Make expression after colon optional As suggested by @maxbrunsfeld
2018-02-24 21:40:43 +00:00
echo ${abc:- }
echo ${abc:
}
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
---
(program
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command (command_name (word)) (expansion (variable_name)))
Handle empty default value during variable expansion (#1) * Handle empty default value during variable expansion * Make expression after colon optional As suggested by @maxbrunsfeld
2018-02-24 21:40:43 +00:00
(command (command_name (word)) (expansion (variable_name) (word)))
(command (command_name (word)) (expansion (variable_name)))
(command (command_name (word)) (expansion (variable_name))))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
Generalize parsing of expansion operators
2018-03-01 00:28:55 +00:00
===================================
Variable expansions with operators
===================================
A="${B[0]# }"
C="${D/#* -E /}"
F="${G%% *}"
---
(program
(variable_assignment
(variable_name)
(string (expansion (subscript (variable_name) (word)))))
(variable_assignment
(variable_name)
Add regex syntax inside of expansions
2018-03-01 18:41:16 +00:00
(string (expansion (variable_name) (regex) (word) (word))))
Generalize parsing of expansion operators
2018-03-01 00:28:55 +00:00
(variable_assignment
(variable_name)
(string (expansion (variable_name) (word) (word)))))
Always lex braces and brackets as separate tokens The lexer needs to always return braces and brackets separately so that the parser can decide if they are part of some construct like an array subscript or a variable expansion. This means that there was no point in moving bare-word tokenization into the external scanner. I've moved it back into the normal scanner. The tricky part is how to deal with the separate '}' and ']' tokens in the case where they are *not* part of a subscript or an expansion. For example, in code like `echo {hi}`, the syntax tree should still clearly indicate that only *one* argument is being passed to echo. For now, we achieve this by grouping the '{', hi, and '}' tokens into a single `concatenation` node, which is a bit odd, but it's the best we can do.
2018-02-28 19:13:49 +00:00
===================================
Variable expansions in strings
===================================
A="${A:-$B/c}"
Add regex syntax inside of expansions
2018-03-01 18:41:16 +00:00
A="${b=$c/$d}"
Always lex braces and brackets as separate tokens The lexer needs to always return braces and brackets separately so that the parser can decide if they are part of some construct like an array subscript or a variable expansion. This means that there was no point in moving bare-word tokenization into the external scanner. I've moved it back into the normal scanner. The tricky part is how to deal with the separate '}' and ']' tokens in the case where they are *not* part of a subscript or an expansion. For example, in code like `echo {hi}`, the syntax tree should still clearly indicate that only *one* argument is being passed to echo. For now, we achieve this by grouping the '{', hi, and '}' tokens into a single `concatenation` node, which is a bit odd, but it's the best we can do.
2018-02-28 19:13:49 +00:00
---
(program
(variable_assignment
(variable_name)
(string
(expansion
(variable_name)
(concatenation (simple_expansion (variable_name)) (word)))))
(variable_assignment
(variable_name)
(string
(expansion
(variable_name)
(concatenation
(simple_expansion (variable_name))
(word)
(simple_expansion (variable_name)))))))
Add regex syntax inside of expansions
2018-03-01 18:41:16 +00:00
===================================
Variable expansions with regexes
===================================
A=${B//:;;/$'\n'}
# escaped space
C=${D/;\ *;|}
---
(program
(variable_assignment (variable_name) (expansion (variable_name) (regex)))
(comment)
(variable_assignment (variable_name) (expansion (variable_name) (regex))))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
===================================
Other variable expansion operators
===================================
cat ${BAR} ${ABC=def} ${GHI:?jkl}
---
(program
(command
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command_name (word))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
(expansion (variable_name))
(expansion (variable_name) (word))
(expansion (variable_name) (word))))
=============================
Command substitutions
=============================
echo `echo hi`
echo $(echo $(echo hi))
---
(program
(command
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command_name (word))
(command_substitution (command (command_name (word)) (word))))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
(command
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command_name (word))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
(command_substitution (command
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command_name (word))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
(command_substitution (command
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command_name (word))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
(word)))))))
Add process substitutions
2017-07-16 05:22:38 +00:00
=============================
Process substitutions
=============================
wc -c <(echo abc && echo def)
echo abc > >(wc -c)
---
(program
(command
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command_name (word))
Add process substitutions
2017-07-16 05:22:38 +00:00
(word)
(process_substitution (list
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command (command_name (word)) (word))
(command (command_name (word)) (word)))))
Add process substitutions
2017-07-16 05:22:38 +00:00
(command
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command_name (word))
Add process substitutions
2017-07-16 05:22:38 +00:00
(word)
(file_redirect (process_substitution
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command (command_name (word)) (word))))))
Add process substitutions
2017-07-16 05:22:38 +00:00
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
=============================
Single quoted strings
=============================
echo 'a b' 'c d'
---
(program
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command (command_name (word)) (raw_string) (raw_string)))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
=============================
Double quoted strings
=============================
echo "a" "b"
echo "a ${b} c" "d $e"
---
(program
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command (command_name (word))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
(string)
(string))
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command (command_name (word))
Reorganize, rename a bunch of things
2017-07-16 05:13:55 +00:00
(string (expansion (variable_name)))
(string (simple_expansion (variable_name)))))
Handle strings containing command substitutions
2017-07-16 05:41:56 +00:00
=========================================
Strings containing command substitutions
=========================================
find "`dirname $file`" -name "$base"'*'
---
(program
(command
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(command_name (word))
(string (command_substitution (command (command_name (word)) (simple_expansion (variable_name)))))
Handle strings containing command substitutions
2017-07-16 05:41:56 +00:00
(word)
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(concatenation
(string (simple_expansion (variable_name)))
(raw_string))))
Add length operator in variable expansions
2017-07-16 06:12:22 +00:00
Handle escape sequences in strings (#7) This should fix #6
2018-02-28 20:15:05 +00:00
=========================================
Strings containing escape sequence
=========================================
echo "\"The great escape\`\${var}"
---
(program (command (command_name (word)) (string)))
Allow $ in strings
2018-03-01 17:54:08 +00:00
======================================
Strings containing special characters
======================================
echo "s/$/'/"
echo "#"
---
(program
(command (command_name (word)) (string))
(command (command_name (word)) (string)))
Add length operator in variable expansions
2017-07-16 06:12:22 +00:00
=========================================
Arrays and array expansions
=========================================
a=()
b=(1 2 3)
echo ${a[@]}
echo ${#b[@]}
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
a[$i]=50
Add missing array syntax
2017-12-26 22:55:37 +00:00
a+=(foo "bar" $(baz))
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
Add length operator in variable expansions
2017-07-16 06:12:22 +00:00
---
(program
Add support for exporting environment variables (#4) * Add support for exporting environment variables * Support multiple exports in one command Also improves the structure of the AST so the variable name and the assigned expression aren't free-floating * Allow for multiple declarations in local declarations as well Simplify the generated nodes. * Commit generated code * Consolidate declarations into a single declaration_command This also adds support for declare, typeset, and readonly * Rename environment_variable_assignment to variable_assignment
2018-02-27 21:22:28 +00:00
(variable_assignment (variable_name) (array))
(variable_assignment (variable_name) (array (word) (word) (word)))
Always lex braces and brackets as separate tokens The lexer needs to always return braces and brackets separately so that the parser can decide if they are part of some construct like an array subscript or a variable expansion. This means that there was no point in moving bare-word tokenization into the external scanner. I've moved it back into the normal scanner. The tricky part is how to deal with the separate '}' and ']' tokens in the case where they are *not* part of a subscript or an expansion. For example, in code like `echo {hi}`, the syntax tree should still clearly indicate that only *one* argument is being passed to echo. For now, we achieve this by grouping the '{', hi, and '}' tokens into a single `concatenation` node, which is a bit odd, but it's the best we can do.
2018-02-28 19:13:49 +00:00
(command (command_name (word)) (expansion (subscript (variable_name) (word))))
(command (command_name (word)) (expansion (subscript (variable_name) (word))))
Add support for exporting environment variables (#4) * Add support for exporting environment variables * Support multiple exports in one command Also improves the structure of the AST so the variable name and the assigned expression aren't free-floating * Allow for multiple declarations in local declarations as well Simplify the generated nodes. * Commit generated code * Consolidate declarations into a single declaration_command This also adds support for declare, typeset, and readonly * Rename environment_variable_assignment to variable_assignment
2018-02-27 21:22:28 +00:00
(variable_assignment
Handle words concatenated with strings and expansions
2017-07-17 17:19:35 +00:00
(subscript (variable_name) (simple_expansion (variable_name)))
Add missing array syntax
2017-12-26 22:55:37 +00:00
(word))
Add support for exporting environment variables (#4) * Add support for exporting environment variables * Support multiple exports in one command Also improves the structure of the AST so the variable name and the assigned expression aren't free-floating * Allow for multiple declarations in local declarations as well Simplify the generated nodes. * Commit generated code * Consolidate declarations into a single declaration_command This also adds support for declare, typeset, and readonly * Rename environment_variable_assignment to variable_assignment
2018-02-27 21:22:28 +00:00
(variable_assignment
Add missing array syntax
2017-12-26 22:55:37 +00:00
(variable_name)
(array
(word)
(string)
(command_substitution (command (command_name (word)))))))