Repository navigation
parser: refuse a modifier-rescue and a not/nested-! command in an arg position, and require every clause terminator - #38
Merged
Merged
Conversation
…d as a nested `!` operand
`arg` has no `modifier_rescue` alternative in parse.y v3_4_0 — only `stmt`
(3184) and the assignment right-hand sides `arg_rhs` (4162), `command_rhs` and
`endless_arg` do. So `f(x rescue nil)`, `[x rescue nil]`, `a[x rescue nil]`,
`{k: x rescue nil}`, `case x; when y rescue nil` and, because `expr` has no such
arm either, `if x rescue nil; end` are all SyntaxErrors in MRI, while
`f((x rescue nil))` and `f(a = 1 rescue nil)` are not.
`expr: '!' command_call` (3350) admits a paren-less command as the operand of
`!`, but the operand is a `command_call`, not an `expr`, so the privilege does
not nest: `!foo bar` parses and `!!foo bar` does not.
`opt_rescue` (5926) and `case_body` (5397) both REQUIRE the `then` nonterminal.
Without it the stray token of an `arg_value` we had already correctly refused —
the `bar` of `rescue foo bar` — was read as the clause's first statement and the
clause parsed anyway.
Mechanism: rescueModOK inverts noRescueMod so the zero value is the refusing
one, permitRescueMod is its only writer and withRescueModifier its only reader,
mirroring argOnly/permitCommand/commandArgsFollow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…and the clause terminator
- `defined?` unparenthesised is `arg: keyword_defined '\n'? begin_defined arg`
(parse.y v3_4_0 4021) — an ARG production, so it must NOT go through
commandArgsFollow, whose first act is to refuse everything in an `arg`
position. `[defined? x]`, `{k: defined? x}`, `1 + defined? x` and
`def m(a = defined? x); end` are legal MRI and v0.6.0 refused them.
- `arg` has no `keyword_not` arm (only `expr: keyword_not expr`, 3345), so
`[not x]`, `{k: not x}`, `f(1, not x)`, `[*not x]`, `f(¬ x)`,
`def m(a = not x); end` and `case y; when not x; end` are SyntaxErrors in both
of MRI's parsers.
- `then`/`do` — `term | KEYWORD | term KEYWORD` — is REQUIRED by every clause
head (`opt_rescue` 5926, `case_body` 5397, `k_if`, `k_unless`, `expr_value_do`,
`p_case_body`), so `if x end`, `while x end`, `for i in a end`,
`case x; when 1 end` and `begin; x; rescue Foo end` are SyntaxErrors. Requiring
it is also what makes a refused `arg_value` OBSERVABLE: the stray `bar` of
`rescue foo bar` was being read as the clause's first statement.
- modifier-`rescue` refused in the remaining `arg_value` positions: hash literal
entries, a `when`/`rescue` list, a parameter default, a keyword-argument value.
Tables: 319 enumerated shapes, each judged by BOTH `ruby -c` and
`ruby --parser=parse.y -c` on ruby 4.0.5. 115 accepted -> REFUSED, 7 REFUSED ->
accepted (the `defined?` family), 0 regressions, 13 shapes where MRI's two
parsers disagree and which are therefore asserted in neither direction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this refuses, and why the gate is a table and not a corpus
v0.6.0 taught the parser that an
arg_valueposition has nocommandalternative. Five shapes of that family survived, reported against v0.6.0. The
bibliography turned them into four productions, one of which is not about
commandat all, and the enumerated table grew from 76 shapes to 319.!!foo barbegin; x; rescue foo bar; y; endbegin; x; rescue Foo, bar baz; y; endf(x rescue nil)f(1, x rescue nil)The productions, mapped to our functions
Read from
parse.yatruby/rubytagv3_4_0. Line numbers are that file's.expr: '!' command_call(3350) vsarg: '!' arg(4009)parseUnary,token.BANGarmexpr: keyword_not '\n'? expr(3345) — and noarg: keyword_notarmparseExprOrAssign, thenotprefixstmt: stmt modifier_rescue after_rescue stmt(3184)parseStatements→permitRescueMod(true)arg_rhs: arg modifier_rescue after_rescue arg(4157-4166)parseAssignRhs,assignRhsValue→permitRescueMod(true)arg_value: arg(4133),args(4314),call_args(4223),aref_args(4140)parseArgList/parseCallArgs/parseArrayElems→permitRescueMod(false)command_args: call_args(4250)parseCommandArgs→permitRescueMod(false)exc_list: arg_value(5950),case_args: arg_value(5397)parseSplatOrExprassoc: arg_value tASSOC arg_value | tLABEL arg_value | tDSTAR arg_value(6789)parseHashLiteral,parseKwArgValuef_opt: f_arg_asgn f_eq arg_valueparseParamDefaultexpr_value(the if/unless/while/until/for/case headers)parseCond,parseCase's subjectopt_rescue: k_rescue exc_list exc_var then compstmt(5926),case_body: k_when case_args then compstmt(5397),k_if/k_unless/if_tail,expr_value_do,p_case_bodyclauseHeadEndarg: keyword_defined '\n'? begin_defined arg(4021)parseIdentExpr, thedefined?armFour findings, and two of them contradict the brief I was given
1. There is no
arg: arg modifier_rescue argproduction. Line 4162 belongs toarg_rhs, notarg. A modifier-rescuelives in exactly two families:stmt(3184) and the assignment right-hand sides (arg_rhs4162,command_rhs3323,mrhs_arg3219,endless_arg4081). Sof(x rescue nil)is not "a separate rule" from the
arg_valuefamily — it is the same rule,and one flag expresses both. The carve-out is what makes the picture falsifiable:
[a = x rescue nil],f(a = 1 rescue nil),f(a.b = 1 rescue nil),f(a ||= 1 rescue nil)anda[a = 1 rescue nil]are all legal MRI in thevery positions that refuse the bare form.
2.
exc_listalready hadpermitCommand(false); the defect was elsewhere.parseSplatOrExprhas carried that call since v0.6.0, and it worked:commandArgsFollowdid decline to givefoothe argumentbarinrescue foo bar. The leftoverbarwas then read as the clause's firststatement, and the clause parsed.
opt_rescue'sthen—term | keyword_then | term keyword_then— is not optional, and neither iscase_body's, nork_if's, norexpr_value_do's.begin; x; rescue Foo end,case x; when 1 end,if x end,while x endandfor i in a endare all SyntaxErrors in MRI, and weaccepted every one of them. A refused production only surfaces as an error when
something downstream insists on what must come next.
3. The
!asymmetry is real and decides where the fix goes.!foo barislegal,
!!foo baris not, becauseexpr: '!' command_calltakes acommand_call— not anexpr. Acommand_callis never itself a unaryexpression, so the privilege cannot nest. The fix belongs at the operator
(one level of
!consumes it), not at the operand.4.
defined?unparenthesised is anargproduction, and v0.6.0 refused it.arg: keyword_defined '\n'? begin_defined arg(4021) is anarg, not acommand, so it is available in anargposition:[defined? x],{k: defined? x},1 + defined? x,f(1, defined? x)anddef m(a = defined? x); endare legal MRI. v0.6.0 gated it behindcommandArgsFollow, whose first act is to refuse everything in anargposition — a second regression of the same shape as the
[defined?(foo bar)]one v0.6.0 caught before shipping. Seven table shapes and two real stdlib
files were refused because of it.
One mechanism, extended — not a second one
rescueModOKreplacesnoRescueModwith the sense inverted, so the zero valueis the refusing one and only the positions MRI gives a production to turn it
on.
permitRescueModis its only writer andwithRescueModifierits onlyreader, exactly mirroring
argOnly/permitCommand/commandArgsFollow. The!,notanddefined?fixes are allargOnly; they add no flag.Gate: the enumerated table
319 shapes, each judged on ruby 4.0.5 by both
ruby -candruby --parser=parse.y -c. Any shape whose legality is contextual is writteninside the context that makes it legal, so no
-cverdict here is a semanticrefusal dressed as a grammar one.
defined?family)A/B: the new table, dropped unchanged onto a clean
origin/maincheckout in aseparate worktree, fails 122 = 115 + 7 assertions and compiles.
The 13 disagreements are all
notin a position that admits anexpr: prismaccepts
x = not y,f(not x),a[not x],p(not x),a.b(not x),x ||= not y,return not x,yield not x,foo not x,p not x,!not x,!not foo bar;parse.yrefuses them. We keep prism's answer and assertnothing. (
a[foo bar do end]andf(foo bar do end), the two already known, arelikewise absent from both tables.)
The tables in
arg_position_command_test.goare now 219 refused and 259accepted shapes, extended in place.
Floor: two corpus sweeps, with their populations named
The sweeper is its own integrity guard and exits non-zero on any of: a row with
an empty path column, distinct paths ≠ input count, or fewer than
-min-okfiles parsing — the floor being the baseline's own OK count, so a runthat read nothing cannot pass. All six runs printed
INTEGRITY OK.87b1631,*_spec.rbunderlanguage/+core/87b1631, every.rbin the corpusrubylibdirrecursive (ruby 4.0.5)The two stdlib files that moved the other way are
ripper/filter.rbandripper/lexer.rb, and they are the nicest result here: both use— MRI's third spelling of
then,term keyword_then. v0.6.0 triedp.accept(token.THEN)before skipping the newline and left thethenas a straystatement. Implementing the clause terminator as the required nonterminal it
is refused 115 shapes and accepted two real files at the same time.
Controls, all still holding (re-measured against both MRI parsers)
while begin; y do end; end; end— still a SyntaxError; so arewhile y do end; endandwhile x = y do end; end.foo :a, (y do end)andx = (y do end)— still accepted.while (y do end); end,while [y do end]; end,while a[y do end]; end,while {k: y do end}; end,until [y do end]; end,foo :a, [y do end],foo :a, {k: y do end},foo :a, (y do end)— all still accepted.a[foo bar],a[foo bar] = 1,a[foo bar] += 1,f(foo bar, 1),a[foo bar, 1]— still accepted. The rule stays head-only.[defined?(foo bar)]— still accepted.raise "x" rescue 42still parses as(raise "x") rescue 42, asserted on the AST, not on acceptance.Coverage, four parts
ast/ast.gonode()marker method. No
parser.gofunction is below 100%.go tool cover -funcwas resolved in the tree that produced itsprofile (
origin/mainin its own worktree, the branch in its own).(
assignRhsValue,clauseHeadEnd,permitRescueMod) and zero removals,so nothing was orphaned out of the set.
gofmt -l .lists onlyparen_singleton_def_test.go, which it already listed onorigin/main; both files this branch touches are clean.Not fixed (3 shapes MRI refuses and we still accept)
not a = foo bar—not's operand is anexpr, anda = foo bariscommand_asgn, reachable fromstmtonly. Needs the assignment RHS to loseits
commandright undernot; a narrower question than this change.begin; x; else; foo bar; end— anelsewith norescue. A different rule(
bodystmt), unrelated toarg_value.f(&b, )— a trailing comma after a block-pass argument.opt_block_argislast in
call_argsand admits no comma after it.🤖 Generated with Claude Code