diff --git a/GLR.md b/GLR.md index 0d7d6b9..a51d308 100644 --- a/GLR.md +++ b/GLR.md @@ -97,6 +97,8 @@ Initialize the generated context with initial user data (or `with_default_userda In GLR mode, user data is branch-local. When the parser forks because of a conflict, the current semantic stack and user data are cloned, and each resulting branch owns and mutates its own `UserData` value independently. Token values, semantic return values, and user data must therefore implement `Clone` for GLR branch-local parsing. Results returned from `accept_all()` include the final user data for each successful branch. +`NoAction` is the only feed failure that means the CFG cannot consume the lookahead terminal. `ReduceAction` means the terminal was grammatically feedable, but a runtime semantic action failed. GLR recovery is entered only when every active branch fails with `NoAction`; if at least one branch survives, the feed succeeds and any sibling branch failures are returned in `FeedSuccess::errors`. + For example, this ambiguous grammar creates two branches for the same input terminal symbol. Each branch mutates its own cloned `Vec<&'static str>`: ```rust @@ -143,15 +145,26 @@ fn main() { // Feed tokens to the GLR parser for token in input { // basic feeding: - if let Err(e) = context.feed(token) { - eprintln!("Fatal parse error: {}", e); - return; + match context.feed(token) { + Ok(success) => { + if let Some(branch_errors) = success.errors { + eprintln!("Some GLR branches were pruned: {}", branch_errors); + } + } + Err(e) => { + eprintln!("Fatal parse error: {}", e); + return; + } } // or location-aware feeding (if %location is configured in the grammar): // let span = MySpan { start: ..., end: ... }; - // if let Err(e) = context.feed_location(token, span) { - // eprintln!("Fatal parse error: {}", e); + // if let Ok(success) = context.feed_location(token, span) { + // if let Some(branch_errors) = success.errors { + // eprintln!("Some GLR branches were pruned: {}", branch_errors); + // } + // } else { + // eprintln!("Fatal parse error"); // return; // } } @@ -172,7 +185,8 @@ fn main() { ### Key API Components - **`EContext::new(userdata)`**: Initializes a new GLR state context with initial user data. Use `EContext::with_default_userdata()` when `UserData: Default`. -- **`context.feed(token)`**: Feeds a token into all active parsing stacks. -- **`context.feed_location(token, location)`**: Feeds a token with its location span into all active parsing stacks (requires `%location` in the grammar). +- **`context.feed(token)`**: Feeds a token into all active parsing stacks and returns `FeedSuccess` when at least one branch survives. +- **`context.feed_location(token, location)`**: Feeds a token with its location span into all active parsing stacks (requires `%location` in the grammar) and returns `FeedSuccess` when at least one branch survives. +- **`FeedSuccess::errors`**: Optional branch-failure information from GLR branches pruned while the feed still succeeded. - **`context.accept()`**: Finalizes parsing (feeding the end-of-file symbol) and returns the first successful `(parse_result, userdata)` pair. - **`context.accept_all()`**: Finalizes parsing and returns an iterator over successful `(parse_result, userdata)` pairs from all active branches. diff --git a/README.md b/README.md index afec0b1..fad871f 100644 --- a/README.md +++ b/README.md @@ -188,12 +188,16 @@ context.feed(token); context.feed_location(token, token_location); ``` +Feed failures are classified by cause. `NoAction` means the CFG cannot consume the lookahead terminal; it leaves the parser stack unchanged and is the only failure kind that can enter panic-mode recovery. `ReduceAction` means the terminal was grammatically feedable, but a runtime semantic action failed. In GLR mode, recovery is attempted only when every active branch fails with `NoAction`. + --- ## GLR Parsing RustyLR provides native support for Generalized LR (GLR) parsing. When you add the `%glr;` directive to a grammar, RustyLR generates a non-deterministic parser that forks state branches upon encountering shift/reduce or reduce/reduce conflicts. This is particularly useful for ambiguous grammars or complex programming languages. +GLR `feed()` and `feed_location()` return a success value that may contain errors from branches pruned while another branch survived. Treat the feed as failed only when the returned `Result` is `Err`; inspect the success value when branch-level diagnostics matter. + In GLR mode, user data is branch-local. When the parser forks, the current user data is cloned so each active branch owns and mutates an independent `UserData` value. For more details, see [GLR.md](GLR.md). diff --git a/SYNTAX.md b/SYNTAX.md index 12bea9e..4d3e397 100644 --- a/SYNTAX.md +++ b/SYNTAX.md @@ -395,7 +395,9 @@ Expr: '{' Stmt* '}' ; ``` -The reserved terminal `error` is used to implement panic-mode recovery. When the parser encounters an unexpected token: +The reserved terminal `error` is used to implement panic-mode recovery. Recovery is entered only for `NoAction`, which means the CFG cannot consume the lookahead terminal. `ReduceAction` means the terminal was grammatically feedable, but a runtime semantic action failed. + +When the parser encounters an unexpected token: 1. **Pop Stack**: The parser pops states off the parser stack until it finds a state that can shift the `error` symbol. If no such state is found, parsing aborts immediately with a `ParseError::NoAction`. 2. **Shift `error`**: The parser shifts the `error` symbol onto the stack. The initial span/location of the `error` token is set to the span of the popped tokens. 3. **Handle Lookahead & Merge**: @@ -408,7 +410,7 @@ The reserved terminal `error` is used to implement panic-mode recovery. When the #### Notes - The `error` token does not carry a semantic value. - You can access the entire span of all discarded/merged tokens using the location binder `@error`. -- In GLR mode, the parser will prefer non-error paths and only trigger error recovery if all other active branches fail. +- In GLR mode, the parser will prefer non-error paths and only trigger error recovery if all active branches fail with `NoAction`. - **Comparison to Bison**: Unlike Bison, which runs a blocking pull-loop internally to discard tokens until it finds a synchronizing token, RustyLR integrates with a push-based model (`feed()`). If a token cannot follow `error`, RustyLR discards it and merges its location span into the `error` token's span, returning control to the caller. This enables reactive parsing and easy diagnostics via `@error` containing the full range of discarded tokens. ### Operator Precedence @@ -461,6 +463,8 @@ Expr Defines a custom error type returned by reduce actions. If your reduce action returns a `Result`, returning an `Err(custom_error)` will stop execution (or prune the GLR branch) and bubble the error up wrapped in `ParseError::ReduceAction`. +A reduce-action error is semantic failure, not a syntax `NoAction`. It does not enter panic-mode error recovery. In GLR mode, if another branch successfully shifts the same terminal, the feed succeeds and reports the pruned-branch error in the GLR feed success value. + ## No Optimization ``` @@ -500,6 +504,8 @@ To feed locations, use `feed_location` instead of `feed`: context.feed_location(token, span); ``` +`NoAction` means the CFG cannot consume the lookahead terminal and leaves the parser stack unchanged. `ReduceAction` means the terminal was grammatically feedable, but runtime execution failed. Only `NoAction` can enter panic-mode recovery. + Within reduce actions, access symbol spans using the `@` prefix: ```rust Expr(i32) : e1=Expr '+' e2=Expr { diff --git a/USING_RUSTYLR_WITH_AI.md b/USING_RUSTYLR_WITH_AI.md index c12862b..428d9b8 100644 --- a/USING_RUSTYLR_WITH_AI.md +++ b/USING_RUSTYLR_WITH_AI.md @@ -110,6 +110,8 @@ fn parse(tokens: impl IntoIterator) -> Result } ``` +Feed failures have two main runtime causes. `NoAction` means the CFG cannot consume the lookahead terminal, leaves the parser stack unchanged, and is the only failure kind that can enter panic-mode recovery. `ReduceAction` means the terminal was grammatically feedable, but runtime execution failed. + ## Agent Checklist Before implementing: @@ -129,6 +131,7 @@ While implementing: - Use `%left`, `%right`, or `%precedence` before changing grammar shape solely to fix expression conflicts. - Use the `error` terminal for recovery points such as delimiters, statements, or object members. - Include `error` in precedence declarations if recovery productions introduce a shift/reduce conflict. +- Treat reduce-action `Err` values as semantic failures, not recovery triggers. In GLR mode, a successful feed may still return branch errors through its success value when other branches were pruned. - Keep generated parser files committed only if the project convention does so. When debugging: diff --git a/llms.txt b/llms.txt index e444803..828ee49 100644 --- a/llms.txt +++ b/llms.txt @@ -25,6 +25,10 @@ RustyLR targets Rust 2024 and requires Rust 1.85 or newer. Generated `Context` types are the normal parser contexts downstream code should use. They initialize parsing for the selected start symbol and return the typed start value from `accept()` or `accept_all()`. +Feed failures have two main runtime causes. `NoAction` means the CFG cannot consume the lookahead terminal, leaves the parser stack unchanged, and is the only failure kind that can enter panic-mode recovery. `ReduceAction` means the terminal was grammatically feedable, but runtime execution failed. + +Reduce-action errors are semantic failures, not panic-mode recovery triggers. GLR feed success values may contain branch errors when one branch survives and sibling branches are pruned. + Generated contexts implement `Clone` when their runtime storage and user data satisfy the required `Clone` bounds. Choose token and semantic value types that implement `Clone` when cloning contexts or using GLR branch-local user data; user data must also implement `Clone` in those cases. Context `Debug` output reports parser stack state and user data; the root parser state is implicit, and GLR output is grouped by active branch. Use the explicit tree APIs for syntax-tree inspection when the `tree` feature is enabled. diff --git a/rusty_lr/tests/lr_glr_operation.rs b/rusty_lr/tests/lr_glr_operation.rs index cd9efdf..2c4d6e3 100644 --- a/rusty_lr/tests/lr_glr_operation.rs +++ b/rusty_lr/tests/lr_glr_operation.rs @@ -376,4 +376,93 @@ mod reduce_action_errors { } assert!(too_large.accept().is_err()); } + + mod glr_recovery { + use rusty_lr::lr1; + + // Keep optimization disabled so `Rejected` remains a distinct reduction that fails before + // the following terminal is shifted. This makes the recovery boundary observable. + lr1! { + %glr; + %nooptim; + %tokentype char; + %userdata Vec<&'static str>; + %error &'static str; + %start Item; + + Item(char) + : Rejected 'y' { + Rejected + } + | error 'y' { + data.push("recovered"); + '?' + } + ; + + Rejected(char): 'x' { + if true { + return Err("rejected token"); + } + 'x' + }; + } + + #[test] + fn glr_reduce_action_error_does_not_enter_recovery_mode() { + let mut ctx = ItemContext::new(Vec::new()); + ctx.feed('x').unwrap(); + + // `error 'y'` could recover from a grammatical `NoAction`, but the failing branch is + // grammatically valid and dies from a reduce-action error instead. + let err = ctx.feed('y').unwrap_err(); + + assert_eq!(err.reduce_action_errors, ["rejected token"]); + assert!(ctx.userdata_all().all(|userdata| userdata.is_empty())); + } + } + + mod glr_partial_errors { + use rusty_lr::lr1; + + // This grammar keeps one shift path alive while a competing reduce path fails + // semantically, exercising successful feeds with pruned-branch diagnostics. + lr1! { + %glr; + %nooptim; + %tokentype char; + %error &'static str; + %start S; + + S(&'static str) + : A 'y' { + "reduce path" + } + | 'x' 'y' { + "shift path" + } + ; + + A(char): 'x' { + if true { + return Err("bad reduce path"); + } + 'x' + }; + } + + #[test] + fn glr_feed_success_reports_pruned_branch_errors() { + let mut ctx = SContext::with_default_userdata(); + assert!(ctx.feed('x').unwrap().errors.is_none()); + + // Feeding `y` succeeds through the direct shift branch, while the reduce branch reports + // its semantic failure through `FeedSuccess::errors`. + let success = ctx.feed('y').unwrap(); + let errors = success.errors.expect("reduce branch should be reported"); + + assert_eq!(errors.reduce_action_errors, ["bad reduce path"]); + assert_eq!(ctx.accept().unwrap(), ("shift path", ())); + } + } } diff --git a/rusty_lr_core/src/parser/deterministic/context.rs b/rusty_lr_core/src/parser/deterministic/context.rs index 66a5b34..8c78625 100644 --- a/rusty_lr_core/src/parser/deterministic/context.rs +++ b/rusty_lr_core/src/parser/deterministic/context.rs @@ -1,3 +1,4 @@ +use std::cell::RefCell; use std::collections::BTreeSet; use super::ParseError; @@ -13,6 +14,35 @@ use crate::parser::terminalclass::TerminalClass; use crate::TerminalSymbol; +/// One deterministic reduction captured during pre-feed simulation. +/// +/// This stores the production and goto decision so the commit phase does not re-query the LR table +/// for the same reduce step. +struct PlannedReduction { + rule_index: usize, + tokens_len: usize, + #[allow(dead_code)] + lhs: NonTerm, + next_nonterm_shift: crate::parser::table::ShiftTarget, +} + +/// Commit record produced by the deterministic pre-feed CFG simulation. +/// +/// `feed_location` builds this before mutating semantic stacks. If terminal shift is impossible, +/// no reduce action has run and the parser stack is left unchanged. +struct FeedPlan { + reductions: Vec>, + terminal_shift: crate::parser::table::ShiftTarget, +} + +/// Minimal syntax failure captured by the simulation before the original terminal is moved. +/// +/// Keeping only the failed state lets the caller build `NoAction` without cloning the terminal +/// during planning. +struct FeedPlanError { + state: usize, +} + /// A struct that maintains the current state and the values associated with each symbol. pub struct Context< P: Parser, @@ -30,6 +60,11 @@ pub struct Context< /// Keeping this reference in the context avoids repeated `Parser::get_tables()` calls in the /// token-feeding hot path. pub(crate) tables: &'static P::Tables, + /// Reusable simulation stack for deterministic pre-feed planning. + /// + /// `can_feed` only has `&self`, so this scratch buffer uses interior mutability to avoid + /// allocating a fresh `Vec` for each grammatical simulation. + pub(crate) feed_extra_state_stack: RefCell>, /// Tree stack for tree representation of the parse. #[cfg(feature = "tree")] pub(crate) tree_stack: crate::tree::TreeList, @@ -54,6 +89,7 @@ impl< location_stack: Vec::new(), userdata, tables: P::get_tables(), + feed_extra_state_stack: RefCell::new(Vec::new()), #[cfg(feature = "tree")] tree_stack: crate::tree::TreeList::new(), @@ -99,6 +135,7 @@ impl< location_stack: Vec::with_capacity(capacity), userdata, tables: P::get_tables(), + feed_extra_state_stack: RefCell::new(Vec::with_capacity(capacity)), #[cfg(feature = "tree")] tree_stack: crate::tree::TreeList::new(), @@ -190,13 +227,8 @@ impl< } pub fn can_accept(&self) -> bool { - let mut extra_state_stack = Vec::new(); - - self.can_feed_impl( - self.state_stack.len(), - &mut extra_state_stack, - P::TermClass::EOF, - ) + // EOF acceptance uses the same grammatical feed simulation as regular terminals. + self.plan_feed_location(P::TermClass::EOF).is_ok() } /// Get current state index @@ -337,38 +369,31 @@ impl< return Err(ParseError::NoAction(err)); } - let mut extra_state_stack = Vec::new(); - use crate::TriState; let mut pop_count = 0; - let mut found = false; + let mut error_plan = None; for &s in self.state_stack.iter().rev() { match self.tables.can_accept_error(s.into_usize()) { TriState::False => {} - TriState::Maybe => { - extra_state_stack.clear(); - - if self.can_feed_impl( + TriState::Maybe | TriState::True => { + // Reuse the same CFG simulation used by normal feeds when deciding + // whether this stack prefix can actually shift the recovery token. + if let Ok(plan) = self.plan_feed_location_from_stack_len( self.state_stack.len() - pop_count, - &mut extra_state_stack, P::TermClass::ERROR, ) { - found = true; + error_plan = Some(plan); break; } } - TriState::True => { - found = true; - break; - } } pop_count += 1; } - if !found { + let Some(error_plan) = error_plan else { return Err(ParseError::NoAction(err)); - } + }; let error_location = Data::Location::new(self.location_stack.iter().rev(), pop_count); @@ -384,11 +409,7 @@ impl< self.tree_stack.truncate(l); } - match self.feed_location_impl( - TerminalSymbol::Error, - P::TermClass::ERROR, - error_location, - ) { + match self.apply_feed_plan(error_plan, TerminalSymbol::Error, error_location) { Ok(()) => { // try shift given term again // to check if the given terminal should be merged with `error` token @@ -442,12 +463,62 @@ impl< P::Term: Clone, P::NonTerm: std::fmt::Debug, { + // First build a complete grammatical feed plan. No semantic action runs until this + // succeeds, so late `NoAction` cannot leave partially reduced stacks behind. + let plan = match self.plan_feed_location(class) { + Ok(plan) => plan, + Err(err) => { + return Err(ParseError::NoAction(super::error::NoActionError { + term, + location, + state: err.state, + })); + } + }; + self.apply_feed_plan(plan, term, location) + } + + /// Simulate the deterministic LR stack operations needed to feed one terminal. + /// + /// The returned plan is a commit log: every reduction records its production, pop length, and + /// non-terminal goto, followed by the final terminal shift. Returning `NoAction` means the + /// caller has not mutated parser state. + fn plan_feed_location( + &self, + class: P::TermClass, + ) -> Result, FeedPlanError> { + self.plan_feed_location_from_stack_len(self.state_stack.len(), class) + } + + /// Build a feed plan against a retained prefix of the current state stack. + /// + /// Error recovery uses this to test candidate pop depths and then commit the selected `error` + /// token feed without repeating the table simulation. + fn plan_feed_location_from_stack_len( + &self, + stack_len: usize, + class: P::TermClass, + ) -> Result, FeedPlanError> { + let mut extra_state_stack = self.feed_extra_state_stack.borrow_mut(); + extra_state_stack.clear(); + + self.plan_feed_location_with_extra_stack(stack_len, class, &mut extra_state_stack) + } + + fn plan_feed_location_with_extra_stack( + &self, + mut stack_len: usize, + class: P::TermClass, + extra_state_stack: &mut Vec, + ) -> Result, FeedPlanError> { use super::super::table::ReduceRules; - use crate::Location; - let shift_to = loop { - let state = self.state(); + let mut reductions = Vec::new(); + // Walk the same reduce chain that a real feed would perform, but keep all new states in + // `extra_state_stack` so the real parser stack remains untouched. + let terminal_shift = loop { + let state = self.simulated_state(&extra_state_stack, stack_len); let (shift, reduce) = match self.tables.term_action(state, class) { Some(action) => (action.shift(), action.reduce()), None => (None, None), @@ -457,105 +528,145 @@ impl< let rule = *self.tables.rule(reduce_rule.into_usize()); let tokens_len = rule.len; - // pop state stack - self.state_stack - .truncate(self.state_stack.len() - tokens_len); - - let mut shift = false; - - let mut new_location = - Data::Location::new(self.location_stack.iter().rev(), tokens_len); + // Simulate popping the RHS of the production across already-simulated states and + // the original stack prefix. + if tokens_len <= extra_state_stack.len() { + extra_state_stack.truncate(extra_state_stack.len() - tokens_len); + } else { + let left = tokens_len - extra_state_stack.len(); + extra_state_stack.clear(); + stack_len -= left; + } - let Some(next_nonterm_shift) = - self.tables.shift_goto_nonterm(self.state(), rule.lhs) - else { + let Some(next_nonterm_shift) = self.tables.shift_goto_nonterm( + self.simulated_state(&extra_state_stack, stack_len), + rule.lhs, + ) else { unreachable!( - "Failed to shift nonterminal: {:?} in state {}", - rule.lhs, - self.state() + "Failed to shift nonterminal: {} in state {}", + rule.lhs.as_str(), + self.simulated_state(&extra_state_stack, stack_len) ); }; - // call reduce action - match Data::reduce_action( - &mut self.data_stack, - &mut self.location_stack, - next_nonterm_shift.push, - reduce_rule.into_usize(), - &mut shift, - &term, - &mut self.userdata, - &mut new_location, - ) { - Ok(_) => {} - Err(err) => { - return Err(ParseError::ReduceAction(super::error::ReduceActionError { - term, - location, - state: self.state(), - source: err, - })); - } - }; - self.location_stack.push(new_location); - - // construct tree - #[cfg(feature = "tree")] - { - let mut children = Vec::with_capacity(rule.len); - for _ in 0..rule.len { - let tree = self.tree_stack.pop().unwrap(); - children.push(tree); - } - children.reverse(); - - self.tree_stack.push(crate::tree::Tree::new_nonterminal( - rule.lhs.clone(), - children, - )); - } - - // shift with reduced nonterminal - self.state_stack.push(next_nonterm_shift.state); + // Record the reduce and goto decision now so commit can replay it without + // re-running table selection. + reductions.push(PlannedReduction { + rule_index: reduce_rule.into_usize(), + tokens_len, + lhs: rule.lhs, + next_nonterm_shift, + }); + extra_state_stack.push(next_nonterm_shift.state); } else { break shift; } }; - // shift with terminal - if let Some(next_state_id) = shift_to { - self.state_stack.push(next_state_id.state); + // A feed is grammatically possible only if the simulated reduce chain reaches a terminal + // shift. Otherwise this is a syntax `NoAction`, not a semantic failure. + match terminal_shift { + Some(terminal_shift) => Ok(FeedPlan { + reductions, + terminal_shift, + }), + None => Err(FeedPlanError { + state: self.simulated_state(&extra_state_stack, stack_len), + }), + } + } - #[cfg(feature = "tree")] - self.tree_stack - .push(crate::tree::Tree::new_terminal(term.clone())); - - if next_state_id.push { - match term { - TerminalSymbol::Terminal(t) => self.data_stack.push(Data::new_terminal(t)), - TerminalSymbol::Error - | TerminalSymbol::Eof - | TerminalSymbol::VirtualStart(_) => self.data_stack.push(Data::new_empty()), + /// Apply a previously simulated deterministic feed plan to the real parser stacks. + /// + /// At this point CFG-level success is known. Remaining errors can only come from user reduce + /// actions, so they are reported as semantic reduce-action failures instead of recovery input. + fn apply_feed_plan( + &mut self, + plan: FeedPlan, + term: TerminalSymbol, + location: Data::Location, + ) -> Result<(), ParseError> + where + P::Term: Clone, + P::NonTerm: std::fmt::Debug, + { + use crate::Location; + + for reduction in plan.reductions { + // Replay the simulated RHS pop before invoking the generated reduce action, matching + // the stack shape that the generated action expects. + self.state_stack + .truncate(self.state_stack.len() - reduction.tokens_len); + + let mut shift = false; + let mut new_location = + Data::Location::new(self.location_stack.iter().rev(), reduction.tokens_len); + + match Data::reduce_action( + &mut self.data_stack, + &mut self.location_stack, + reduction.next_nonterm_shift.push, + reduction.rule_index, + &mut shift, + &term, + &mut self.userdata, + &mut new_location, + ) { + Ok(_) => {} + Err(err) => { + return Err(ParseError::ReduceAction(super::error::ReduceActionError { + term, + location, + state: self.state(), + source: err, + })); } - } else { - match term { - TerminalSymbol::Terminal(_) - | TerminalSymbol::Error - | TerminalSymbol::Eof - | TerminalSymbol::VirtualStart(_) => self.data_stack.push(Data::new_empty()), + }; + self.location_stack.push(new_location); + + #[cfg(feature = "tree")] + { + // Mirror the same reduction in the optional syntax tree stack. + let mut children = Vec::with_capacity(reduction.tokens_len); + for _ in 0..reduction.tokens_len { + let tree = self.tree_stack.pop().unwrap(); + children.push(tree); } + children.reverse(); + + self.tree_stack + .push(crate::tree::Tree::new_nonterminal(reduction.lhs, children)); } - self.location_stack.push(location); + self.state_stack.push(reduction.next_nonterm_shift.state); + } - Ok(()) + // Commit the terminal shift only after all planned reductions have been replayed. + self.state_stack.push(plan.terminal_shift.state); + + #[cfg(feature = "tree")] + self.tree_stack + .push(crate::tree::Tree::new_terminal(term.clone())); + + if plan.terminal_shift.push { + match term { + TerminalSymbol::Terminal(t) => self.data_stack.push(Data::new_terminal(t)), + TerminalSymbol::Error | TerminalSymbol::Eof | TerminalSymbol::VirtualStart(_) => { + self.data_stack.push(Data::new_empty()) + } + } } else { - Err(ParseError::NoAction(super::error::NoActionError { - term, - location, - state: self.state(), - })) + match term { + TerminalSymbol::Terminal(_) + | TerminalSymbol::Error + | TerminalSymbol::Eof + | TerminalSymbol::VirtualStart(_) => self.data_stack.push(Data::new_empty()), + } } + + self.location_stack.push(location); + + Ok(()) } /// Check if `term` can be feeded to current state. @@ -563,10 +674,9 @@ impl< /// So this function will return `false` even if term can be shifted as `error` token, /// and will return `true` if `Err` variant is returned by `reduce_action`. pub fn can_feed(&self, term: &Data::Term) -> bool { - let mut extra_state_stack = Vec::new(); let class = P::TermClass::from_term(term); - self.can_feed_impl(self.state_stack.len(), &mut extra_state_stack, class) + self.plan_feed_location(class).is_ok() } /// Check if current context can enter panic mode @@ -576,12 +686,14 @@ impl< return false; } - let mut extra_state_stack = Vec::new(); let error = P::TermClass::ERROR; let mut stack_len = self.state_stack.len(); loop { - match self.can_feed_impl(stack_len, &mut extra_state_stack, error) { + match self + .plan_feed_location_from_stack_len(stack_len, error) + .is_ok() + { true => break true, // successfully shifted `error` false => { if stack_len == 0 { @@ -591,66 +703,9 @@ impl< } } } - extra_state_stack.clear(); } } - fn can_feed_impl( - &self, - mut stack_len: usize, - extra_state_stack: &mut Vec, - class: P::TermClass, - ) -> bool { - let shift_to = loop { - let state = self.simulated_state(extra_state_stack, stack_len); - - let (shift, reduce) = match self.tables.term_action(state, class) { - Some(action) => (action.shift(), action.reduce()), - None => (None, None), - }; - if let Some(reduce_rule) = reduce { - use super::super::table::ReduceRules; - let reduce_rule = reduce_rule.to_iter().next().unwrap(); - let rule = *self.tables.rule(reduce_rule.into_usize()); - let tokens_len = rule.len; - - // pop state stack - if tokens_len <= extra_state_stack.len() { - extra_state_stack.truncate(extra_state_stack.len() - tokens_len); - } else { - let left = tokens_len - extra_state_stack.len(); - extra_state_stack.clear(); - stack_len -= left; - } - - // shift with reduced nonterminal - let last_state = extra_state_stack - .last() - .copied() - .map(Index::into_usize) - .unwrap_or_else(|| self.state_from_len(stack_len)); - if let Some(next_state_id) = self.tables.shift_goto_nonterm(last_state, rule.lhs) { - extra_state_stack.push(next_state_id.state); - } else { - unreachable!( - "unreachable: nonterminal shift should always succeed after reduce operation. Failed to shift nonterminal '{}' in state {}.", - rule.lhs.as_str(), - extra_state_stack - .last() - .copied() - .map(Index::into_usize) - .unwrap_or_else(|| self.state_from_len(stack_len)) - ); - } - } else { - break shift; - } - }; - - // shift with terminal - shift_to.is_some() - } - fn feed_eof( &mut self, ) -> Result<(), ParseError> @@ -699,6 +754,7 @@ where #[cfg(feature = "tree")] tree_stack: self.tree_stack.clone(), + feed_extra_state_stack: RefCell::new(Vec::new()), _phantom: std::marker::PhantomData, } } @@ -724,3 +780,369 @@ where .finish() } } + +#[cfg(test)] +mod tests { + use super::*; + use crate::DefaultLocation; + use crate::TriState; + use crate::parser::table::ShiftTarget; + use crate::parser::table::TermActionRef; + + #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] + enum TestTermClass { + Error, + Eof, + Start, + A, + B, + } + + impl TerminalClass for TestTermClass { + type Term = char; + + const ERROR: Self = TestTermClass::Error; + const EOF: Self = TestTermClass::Eof; + + fn as_str(&self) -> &'static str { + match self { + TestTermClass::Error => "error", + TestTermClass::Eof => "eof", + TestTermClass::Start => "start", + TestTermClass::A => "a", + TestTermClass::B => "b", + } + } + + fn to_usize(&self) -> usize { + *self as usize + } + + fn from_term(term: &Self::Term) -> Self { + match term { + 'a' => TestTermClass::A, + 'b' => TestTermClass::B, + _ => TestTermClass::Error, + } + } + + fn from_virtual_start(_branch_idx: u32) -> Self { + TestTermClass::Start + } + } + + #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] + enum TestNonTerm { + N, + } + + impl NonTerminal for TestNonTerm { + fn nonterm_type(&self) -> Option { + None + } + + fn as_str(&self) -> &'static str { + "N" + } + + fn to_usize(&self) -> usize { + 0 + } + } + + struct TestTables; + + impl ParserTables for TestTables { + type TermClass = TestTermClass; + type NonTerm = TestNonTerm; + type ReduceRules = usize; + type StateIndex = usize; + + fn term_action( + &self, + state: usize, + class: Self::TermClass, + ) -> Option> { + // This table intentionally reduces on lookahead `b`, but the state after reducing has + // no `b` shift. It exercises late `NoAction` after a planned reduction. + match (state, class) { + (0, TestTermClass::Start) => Some(TermActionRef::Shift(ShiftTarget::new(1, false))), + (1, TestTermClass::A) => Some(TermActionRef::Shift(ShiftTarget::new(2, true))), + (2, TestTermClass::B) => Some(TermActionRef::Reduce(&0)), + _ => None, + } + } + + fn shift_goto_nonterm( + &self, + state: usize, + nonterm: Self::NonTerm, + ) -> Option> { + match (state, nonterm) { + (1, TestNonTerm::N) => Some(ShiftTarget::new(3, true)), + _ => None, + } + } + + fn is_accept(&self, _state: usize) -> bool { + false + } + + fn expected_shift_term(&self, _state: usize) -> impl Iterator + '_ { + std::iter::empty() + } + + fn expected_shift_nonterm( + &self, + _state: usize, + ) -> impl Iterator + '_ { + std::iter::empty() + } + + fn expected_reduce_rule(&self, _state: usize) -> impl Iterator + '_ { + std::iter::empty::() + } + + fn can_accept_error(&self, _state: usize) -> TriState { + TriState::False + } + + fn rule(&self, rule: usize) -> &crate::parser::table::RuleInfo { + static RULES: [crate::parser::table::RuleInfo; 1] = + [crate::parser::table::RuleInfo { + lhs: TestNonTerm::N, + len: 1, + }]; + &RULES[rule] + } + + fn state_count(&self) -> usize { + 4 + } + + fn rule_count(&self) -> usize { + 1 + } + } + + struct TestParser; + + impl Parser for TestParser { + const ERROR_USED: bool = false; + + type Term = char; + type TermClass = TestTermClass; + type NonTerm = TestNonTerm; + type StateIndex = usize; + type ReduceRules = usize; + type Tables = TestTables; + + fn get_tables() -> &'static Self::Tables { + &TestTables + } + + fn __rusty_lr_parser_version() -> (usize, usize, usize) { + crate::versions::EXPECTED_RUSTY_LR_PARSER_VERSION + } + } + + struct RecoveryTables; + + impl ParserTables for RecoveryTables { + type TermClass = TestTermClass; + type NonTerm = TestNonTerm; + type ReduceRules = usize; + type StateIndex = usize; + + fn term_action( + &self, + state: usize, + class: Self::TermClass, + ) -> Option> { + // Recovery from state 2 must reduce before the synthetic `error` token can shift. + // Marking the state as `TriState::True` below ensures recovery still builds a plan. + match (state, class) { + (0, TestTermClass::Start) => Some(TermActionRef::Shift(ShiftTarget::new(1, false))), + (1, TestTermClass::A) => Some(TermActionRef::Shift(ShiftTarget::new(2, true))), + (2, TestTermClass::Error) => Some(TermActionRef::Reduce(&0)), + (3, TestTermClass::Error) => Some(TermActionRef::Shift(ShiftTarget::new(4, false))), + (4, TestTermClass::B) => Some(TermActionRef::Shift(ShiftTarget::new(5, true))), + _ => None, + } + } + + fn shift_goto_nonterm( + &self, + state: usize, + nonterm: Self::NonTerm, + ) -> Option> { + match (state, nonterm) { + (1, TestNonTerm::N) => Some(ShiftTarget::new(3, true)), + _ => None, + } + } + + fn is_accept(&self, _state: usize) -> bool { + false + } + + fn expected_shift_term(&self, _state: usize) -> impl Iterator + '_ { + std::iter::empty() + } + + fn expected_shift_nonterm( + &self, + _state: usize, + ) -> impl Iterator + '_ { + std::iter::empty() + } + + fn expected_reduce_rule(&self, _state: usize) -> impl Iterator + '_ { + std::iter::empty::() + } + + fn can_accept_error(&self, state: usize) -> TriState { + match state { + 2 => TriState::True, + _ => TriState::False, + } + } + + fn rule(&self, rule: usize) -> &crate::parser::table::RuleInfo { + static RULES: [crate::parser::table::RuleInfo; 1] = + [crate::parser::table::RuleInfo { + lhs: TestNonTerm::N, + len: 1, + }]; + &RULES[rule] + } + + fn state_count(&self) -> usize { + 6 + } + + fn rule_count(&self) -> usize { + 1 + } + } + + struct RecoveryParser; + + impl Parser for RecoveryParser { + const ERROR_USED: bool = true; + + type Term = char; + type TermClass = TestTermClass; + type NonTerm = TestNonTerm; + type StateIndex = usize; + type ReduceRules = usize; + type Tables = RecoveryTables; + + fn get_tables() -> &'static Self::Tables { + &RecoveryTables + } + + fn __rusty_lr_parser_version() -> (usize, usize, usize) { + crate::versions::EXPECTED_RUSTY_LR_PARSER_VERSION + } + } + + #[derive(Clone, Debug, PartialEq, Eq)] + enum TestData { + Empty, + Terminal(char), + N, + } + + impl SemanticValue for TestData { + type Term = char; + type NonTerm = TestNonTerm; + type UserData = Vec<&'static str>; + type ReduceActionError = (); + type Location = DefaultLocation; + + fn new_empty() -> Self { + TestData::Empty + } + + fn new_terminal(term: Self::Term) -> Self { + TestData::Terminal(term) + } + + fn reduce_action( + data_stack: &mut Vec, + location_stack: &mut Vec, + push_data: bool, + rule_index: usize, + _shift: &mut bool, + _lookahead: &TerminalSymbol, + userdata: &mut Self::UserData, + _location0: &mut Self::Location, + ) -> Result<(), Self::ReduceActionError> { + assert_eq!(rule_index, 0); + assert_eq!(data_stack.pop(), Some(TestData::Terminal('a'))); + location_stack.pop(); + userdata.push("reduced"); + if push_data { + data_stack.push(TestData::N); + } else { + data_stack.push(TestData::Empty); + } + Ok(()) + } + } + + struct TestStart; + + impl StartExtractor for TestStart { + type StartType = (); + + const BRANCH_INDEX: u32 = 0; + + fn extract(_value: TestData) -> Option { + Some(()) + } + } + + #[test] + fn feed_failure_after_planned_reductions_does_not_commit_stack_changes() { + let mut ctx = Context::::new(Vec::new()); + ctx.feed('a').unwrap(); + + // `b` would trigger a reduction before discovering that it cannot be shifted. The context + // must remain byte-for-byte equivalent to its pre-feed state. + let state_stack = ctx.state_stack.clone(); + let data_stack = ctx.data_stack.clone(); + let location_stack = ctx.location_stack.clone(); + + let err = ctx.feed('b').unwrap_err(); + + assert_eq!(err.state(), 3); + assert_eq!(ctx.state_stack, state_stack); + assert_eq!(ctx.data_stack, data_stack); + assert_eq!(ctx.location_stack, location_stack); + assert!(ctx.userdata.is_empty()); + } + + #[test] + fn error_token_recovery_commits_the_precomputed_plan() { + let mut ctx = Context::::new(Vec::new()); + ctx.feed('a').unwrap(); + + // Feeding `b` has no direct action, so recovery shifts `error`. That recovery shift must + // use the reduction discovered by pre-feed simulation. + ctx.feed('b').unwrap(); + + assert_eq!(ctx.userdata, vec!["reduced"]); + assert_eq!(ctx.state_stack, vec![1, 3, 4, 5]); + assert_eq!( + ctx.data_stack, + vec![ + TestData::Empty, + TestData::N, + TestData::Empty, + TestData::Terminal('b') + ] + ); + } +} diff --git a/rusty_lr_core/src/parser/deterministic/error.rs b/rusty_lr_core/src/parser/deterministic/error.rs index 79a1c7a..9f2c082 100644 --- a/rusty_lr_core/src/parser/deterministic/error.rs +++ b/rusty_lr_core/src/parser/deterministic/error.rs @@ -17,14 +17,6 @@ pub struct ReduceActionError { pub source: Source, } -#[derive(Clone, Debug)] -pub struct NoPrecedenceError { - pub term: TerminalSymbol, - pub location: Location, - pub state: usize, - pub rule: usize, -} - /// Error type for feed() #[derive(Clone, Debug)] pub enum ParseError { @@ -33,10 +25,6 @@ pub enum ParseError { /// Error from reduce action. ReduceAction(ReduceActionError), - - /// Rule index when shift/reduce conflict occur with no shift/reduce precedence defined. - /// This is same as when setting %nonassoc in Bison. - NoPrecedence(NoPrecedenceError), } impl ParseError { @@ -44,7 +32,6 @@ impl ParseError { match self { ParseError::NoAction(err) => &err.location, ParseError::ReduceAction(err) => &err.location, - ParseError::NoPrecedence(err) => &err.location, } } @@ -52,7 +39,6 @@ impl ParseError { match self { ParseError::NoAction(err) => &err.term, ParseError::ReduceAction(err) => &err.term, - ParseError::NoPrecedence(err) => &err.term, } } @@ -60,7 +46,6 @@ impl ParseError { match self { ParseError::NoAction(err) => err.state, ParseError::ReduceAction(err) => err.state, - ParseError::NoPrecedence(err) => err.state, } } } @@ -82,9 +67,6 @@ where err.term, err.state, err.source ) } - ParseError::NoPrecedence(err) => { - write!(f, "NoPrecedence: {}, State: {}", err.rule, err.state) - } } } } diff --git a/rusty_lr_core/src/parser/nondeterministic/context.rs b/rusty_lr_core/src/parser/nondeterministic/context.rs index c1d3a8e..cb0e35a 100644 --- a/rusty_lr_core/src/parser/nondeterministic/context.rs +++ b/rusty_lr_core/src/parser/nondeterministic/context.rs @@ -3,6 +3,7 @@ use std::num::NonZeroUsize; use super::Node; use super::NodeId; use super::ParseError; +use super::error::FeedSuccess; use crate::Location; use crate::TerminalSymbol; @@ -91,6 +92,15 @@ struct Branch { userdata: UserData, } +/// Branch state returned when a GLR feed path reaches no terminal shift. +/// +/// The caller only needs the graph node and user data to keep or discard that branch; the lookahead +/// terminal and location are owned by the failed attempt and do not need to be returned. +struct UnshiftedBranch { + node: NodeId, + userdata: UserData, +} + impl Branch { fn new(node: NodeId, userdata: UserData) -> Self { Branch { node, userdata } @@ -122,9 +132,8 @@ pub struct Context< /// But we don't want to reallocate every `feed` call pub(crate) reduce_errors: Vec, - /// For temporary use. - /// store rule indices where shift/reduce conflicts occured with no precedence defined. - pub(crate) no_precedences: Vec, + // Future optimization point: GLR can also reuse simulation state stacks, but branch-local + // recursion and cloned alternatives need a separate scratch ownership strategy. /// Decoded parser tables shared by every active GLR branch. /// /// Branches clone stack/userdata state, but they all read the same immutable runtime tables. @@ -151,7 +160,6 @@ impl< next_branches: Default::default(), reduce_errors: Default::default(), fallback_branches: Default::default(), - no_precedences: Default::default(), tables: P::get_tables(), _phantom: std::marker::PhantomData, }; @@ -873,7 +881,10 @@ impl< pub fn feed( &mut self, term: Data::Term, - ) -> Result<(), ParseError> + ) -> Result< + FeedSuccess, + ParseError, + > where P::Term: Clone, P::NonTerm: std::fmt::Debug, @@ -906,15 +917,7 @@ impl< class: P::TermClass, location: Data::Location, userdata: Data::UserData, - ) -> Result< - (), - ( - NodeId, - TerminalSymbol, - Data::Location, - Data::UserData, - ), - > + ) -> Result<(), UnshiftedBranch> where Data: Clone, Data::UserData: Clone, @@ -934,7 +937,7 @@ impl< let mut shifted = false; let mut reduced_node = node; - let mut reduced_userdata = userdata.clone(); + let mut reduced_userdata = None; // Each GLR branch receives its own user data copy before running reduce actions. let mut shift_ = false; @@ -969,9 +972,9 @@ impl< Ok(_) => { shifted = true; } - Err((reduced_node_, _, _, userdata_)) => { - reduced_node = reduced_node_; - reduced_userdata = userdata_; + Err(unshifted) => { + reduced_node = unshifted.node; + reduced_userdata = Some(unshifted.userdata); } } } @@ -1040,7 +1043,10 @@ impl< } else if shifted { Ok(()) } else { - Err((reduced_node, term, location, reduced_userdata)) + Err(UnshiftedBranch { + node: reduced_node, + userdata: reduced_userdata.unwrap_or(userdata), + }) } } else if let Some(shift) = shift_state { let node_ = self.node_mut(node); @@ -1077,7 +1083,7 @@ impl< Ok(()) } else { // no reduce, no shift - Err((node, term, location, userdata)) + Err(UnshiftedBranch { node, userdata }) } } fn panic_mode( @@ -1085,8 +1091,7 @@ impl< mut node: NodeId, userdata: Data::UserData, extra_state_stack: &mut Vec, - ) -> Data::Location - where + ) where Data: Clone, Data::UserData: Clone, P::Term: Clone, @@ -1100,10 +1105,8 @@ impl< let pop_count = loop { let node_ = self.node(node); if !node_.is_leaf() { - let loc = error_location_preserved - .unwrap_or_else(|| Data::Location::new(self.location_iter(node), 0)); self.try_remove_node(node); - return loc; + return; } let mut pop_count = 0; @@ -1143,14 +1146,11 @@ impl< Data::Location::new(self.location_iter(node), pop_count) }); - let loc = error_location_preserved - .clone() - .unwrap_or_else(|| Data::Location::new(self.location_iter(node), 0)); if let Some(parent) = self.try_remove_node(node) { node = parent; continue; } else { - return loc; + return; } } else { break pop_count; @@ -1179,22 +1179,30 @@ impl< node, TerminalSymbol::Error, P::TermClass::ERROR, - error_location.clone(), + error_location, userdata, ) { Ok(()) => {} - Err((err_node, _, _, _)) => { - self.try_remove_node(err_node); + Err(unshifted) => { + self.try_remove_node(unshifted.node); } // other errors } - error_location } /// Feed one terminal with location to parser, and update state stack. + /// + /// Each active GLR branch is first checked with the same CFG simulation used by `can_feed`. + /// Branches rejected by that simulation are `NoAction` branches and are the only branches + /// eligible for panic-mode recovery. If at least one branch can grammatically shift the + /// terminal, recovery is not entered; failed sibling branches are returned in + /// `FeedSuccess::errors`. pub fn feed_location( &mut self, term: P::Term, location: Data::Location, - ) -> Result<(), ParseError> + ) -> Result< + FeedSuccess, + ParseError, + > where P::Term: Clone, P::NonTerm: std::fmt::Debug, @@ -1204,22 +1212,42 @@ impl< use crate::Location; self.reduce_errors.clear(); - self.no_precedences.clear(); self.fallback_branches.clear(); self.next_branches.clear(); let class = P::TermClass::from_term(&term); let mut current_branches = std::mem::take(&mut self.current_branches); + let mut feedable_branch_seen = false; for Branch { node, userdata } in current_branches.drain(..) { - if let Err((node, _, _, userdata)) = self.feed_location_impl( - node, - TerminalSymbol::Terminal(term.clone()), - class, - location.clone(), - userdata, - ) { - // store to fallback nodes in case of all nodes failed to shift + // Classify this branch before mutating the graph-structured stack. Only branches that + // fail this CFG simulation are considered `NoAction` branches. + let mut extra_state_stack = Vec::new(); + let node_and_len = { + let node_ = self.node(node); + if node_.len() == 0 { + None + } else { + Some((node, NonZeroUsize::new(node_.len()).unwrap())) + } + }; + + if self.can_feed_impl(&mut extra_state_stack, node_and_len, class) { + feedable_branch_seen = true; + + // The CFG admits this terminal on this branch. From here on, failures are caused + // by semantic reduce actions or runtime conflict decisions, so they must not + // trigger error recovery. + let _ = self.feed_location_impl( + node, + TerminalSymbol::Terminal(term.clone()), + class, + location.clone(), + userdata, + ); + } else { + // This branch cannot grammatically shift the lookahead. Keep it only as a + // recovery candidate in case every active branch is also `NoAction`. self.fallback_branches.push(Branch::new(node, userdata)); } } @@ -1230,6 +1258,19 @@ impl< // check for panic mode // and restore nodes to original state from fallback_branches if self.next_branches.is_empty() { + if feedable_branch_seen { + // A branch was grammatically able to shift the terminal, but failed while + // committing semantic/runtime actions. That is not syntax recovery input. + std::mem::swap(&mut self.current_branches, &mut self.fallback_branches); + + return Err(ParseError { + term: TerminalSymbol::Terminal(term), + location, + reduce_action_errors: std::mem::take(&mut self.reduce_errors), + states: self.states().collect(), + }); + } + // early return if `error` token is not used in the grammar if !P::ERROR_USED { std::mem::swap(&mut self.current_branches, &mut self.fallback_branches); @@ -1238,52 +1279,30 @@ impl< term: TerminalSymbol::Terminal(term), location, reduce_action_errors: std::mem::take(&mut self.reduce_errors), - no_precedences: std::mem::take(&mut self.no_precedences), states: self.states().collect(), }); } let mut fallback_branches = std::mem::take(&mut self.fallback_branches); - let root_userdata = fallback_branches - .first() - .map(|branch| branch.userdata.clone()); let mut extra_state_stack = Vec::new(); - let mut error_location = if let Some(first_branch) = fallback_branches.first() { - Data::Location::new(self.location_iter(first_branch.node), 0) - } else { - location.clone() - }; // try enter panic mode and store error nodes to next_branches for Branch { node, userdata } in fallback_branches.drain(..) { - error_location = self.panic_mode(node, userdata, &mut extra_state_stack); + // GLR recovery only searches concrete branch stacks. The implicit root state 0 is + // before the virtual start symbol and cannot accept a synthetic `error` token. + self.panic_mode(node, userdata, &mut extra_state_stack); } // put back for reuse allocated memory self.fallback_branches = fallback_branches; - if self.next_branches.is_empty() { - // for-loop above doesn't check for root state (0). - // check for 0 state here - if self.tables.can_accept_error(0) == crate::TriState::True { - if let Some(userdata) = root_userdata { - // all nodes were deleted, so create new - let node = self.new_node(); - if let Err(_) = self.feed_location_impl( - node, - TerminalSymbol::Error, - P::TermClass::ERROR, - error_location, - userdata, - ) { - return Err(ParseError { - term: TerminalSymbol::Terminal(term), - location, - reduce_action_errors: std::mem::take(&mut self.reduce_errors), - no_precedences: std::mem::take(&mut self.no_precedences), - states: self.states().collect(), - }); - } - } - } + if !self.reduce_errors.is_empty() { + // Shifting the synthetic `error` token may execute reduce actions. If one fails, + // recovery itself failed semantically and must be reported. + return Err(ParseError { + term: TerminalSymbol::Terminal(term), + location, + reduce_action_errors: std::mem::take(&mut self.reduce_errors), + states: self.states().collect(), + }); } // if next_node is still empty, then no panic mode was entered, this is an error @@ -1293,7 +1312,6 @@ impl< term: TerminalSymbol::Terminal(term), location, reduce_action_errors: std::mem::take(&mut self.reduce_errors), - no_precedences: std::mem::take(&mut self.no_precedences), states: self.states().collect(), }) } else { @@ -1340,11 +1358,31 @@ impl< } } self.next_branches = next_branches; - Ok(()) + // Recovery consumed or merged the lookahead. This is a successful feed, and the + // original `NoAction` branches have been transformed into recovery branches. + Ok(FeedSuccess { errors: None }) } } else { + // At least one original branch accepted the terminal. Report sibling branch failures + // to callers without treating this feed as fatal. + let errors = if self.fallback_branches.is_empty() && self.reduce_errors.is_empty() { + None + } else { + let states = self + .fallback_branches + .iter() + .map(|branch| self.state(branch.node)) + .collect(); + self.fallback_branches.clear(); + Some(ParseError { + term: TerminalSymbol::Terminal(term), + location, + reduce_action_errors: std::mem::take(&mut self.reduce_errors), + states, + }) + }; std::mem::swap(&mut self.current_branches, &mut self.next_branches); - Ok(()) + Ok(FeedSuccess { errors }) } } /// Feed one terminal with location to parser, and update state stack. @@ -1570,9 +1608,8 @@ Failed to shift nonterminal '{}' after reducing rule '{}'. This indicates a pars } } - // check root node - extra_state_stack.clear(); - self.can_feed_impl(&mut extra_state_stack, None, P::TermClass::ERROR) + // State 0 is the implicit root before virtual start and cannot accept `error`. + false }) } @@ -1589,7 +1626,6 @@ Failed to shift nonterminal '{}' after reducing rule '{}'. This indicates a pars use crate::location::Location; self.reduce_errors.clear(); - self.no_precedences.clear(); self.fallback_branches.clear(); self.next_branches.clear(); @@ -1602,14 +1638,15 @@ Failed to shift nonterminal '{}' after reducing rule '{}'. This indicates a pars let mut current_branches = std::mem::take(&mut self.current_branches); for Branch { node, userdata } in current_branches.drain(..) { let node_eof_location = Data::Location::new(self.location_iter(node), 0); - if let Err((node, _, _, userdata)) = self.feed_location_impl( + if let Err(unshifted) = self.feed_location_impl( node, TerminalSymbol::Eof, P::TermClass::EOF, node_eof_location, userdata, ) { - self.fallback_branches.push(Branch::new(node, userdata)); + self.fallback_branches + .push(Branch::new(unshifted.node, unshifted.userdata)); } } self.current_branches = current_branches; @@ -1624,7 +1661,6 @@ Failed to shift nonterminal '{}' after reducing rule '{}'. This indicates a pars term: TerminalSymbol::Eof, location: eof_location, reduce_action_errors: std::mem::take(&mut self.reduce_errors), - no_precedences: std::mem::take(&mut self.no_precedences), states: self.states().collect(), }) } else { @@ -1687,7 +1723,6 @@ where next_branches: Default::default(), reduce_errors: Default::default(), fallback_branches: Default::default(), - no_precedences: Default::default(), tables: self.tables, _phantom: std::marker::PhantomData, } diff --git a/rusty_lr_core/src/parser/nondeterministic/error.rs b/rusty_lr_core/src/parser/nondeterministic/error.rs index 2a47641..bd752b2 100644 --- a/rusty_lr_core/src/parser/nondeterministic/error.rs +++ b/rusty_lr_core/src/parser/nondeterministic/error.rs @@ -10,14 +10,25 @@ pub struct ParseError { pub location: Location, /// Error from reduce action (from every diverged paths) pub reduce_action_errors: Vec, - /// Rule indices when shift/reduce conflict occur with no shift/reduce precedence defined. - /// This is same as when setting %nonassoc in Bison. - pub no_precedences: Vec, /// States when the error occurred (from all diverged paths) pub(crate) states: Vec, } +/// Successful GLR feed result. +/// +/// GLR parsing can keep one branch alive while other branches fail. In that case the feed is a +/// success, but `errors` records the branches that were pruned by `NoAction` or reduce-action +/// failure. +#[derive(Clone, Debug)] +pub struct FeedSuccess { + /// Branch errors observed while at least one GLR branch survived this feed. + /// + /// This is intentionally optional so callers that only care about fatal parse failure can keep + /// using `context.feed(token)?` or `context.feed(token).unwrap()`. + pub errors: Option>, +} + impl ParseError { pub fn location(&self) -> &Location { &self.location diff --git a/rusty_lr_core/src/parser/nondeterministic/mod.rs b/rusty_lr_core/src/parser/nondeterministic/mod.rs index 9411e2d..358cff5 100644 --- a/rusty_lr_core/src/parser/nondeterministic/mod.rs +++ b/rusty_lr_core/src/parser/nondeterministic/mod.rs @@ -3,6 +3,7 @@ mod error; mod node; pub use context::Context; +pub use error::FeedSuccess; pub use error::ParseError; pub use node::Node; pub(crate) use node::NodeId; diff --git a/rusty_lr_parser/src/parser/lexer.rs b/rusty_lr_parser/src/parser/lexer.rs index 0a81011..fe3ef05 100644 --- a/rusty_lr_parser/src/parser/lexer.rs +++ b/rusty_lr_parser/src/parser/lexer.rs @@ -214,7 +214,7 @@ impl std::fmt::Display for Lexed { } } -fn ident_to_keyword(ident: Ident) -> Option { +fn ident_to_keyword(ident: Ident) -> Option { match ident.to_string().as_str() { "left" => Some(Lexed::Left(ident)), "right" => Some(Lexed::Right(ident)), @@ -233,10 +233,20 @@ fn ident_to_keyword(ident: Ident) -> Option { "location" => Some(Lexed::Location(ident)), "allow" => Some(Lexed::Allow(ident)), _ => None, - } -} - -/// lex & feed stream to parser + } +} + +fn feed_location( + context: &mut GrammarContext, + term: Lexed, + location: Location, +) -> Result<(), ParseError> { + // The grammar lexer only needs fatal parse errors here. In GLR builds, successful feeds may + // carry pruned-branch diagnostics, but lexer fallback decisions are driven by `can_feed`. + context.feed_location(term, location).map(|_| ()) +} + +/// lex & feed stream to parser /// For '%' directives and 'Group' variants, /// First tries to feed the Compound token /// if it failed, then feed the internal splitted tokens recursively @@ -252,13 +262,13 @@ pub fn feed_recursive(input: TokenStream, context: &mut GrammarContext) -> Resul match ident_to_keyword(ident.clone()) { Some(keyword) => { if context.can_feed(&keyword) { - context.feed_location(keyword, location)?; + feed_location(context, keyword, location)?; } else { - context.feed_location(Lexed::Ident(ident), location)?; + feed_location(context, Lexed::Ident(ident), location)?; } } _ => { - context.feed_location(Lexed::Ident(ident), location)?; + feed_location(context, Lexed::Ident(ident), location)?; } } } @@ -267,21 +277,21 @@ pub fn feed_recursive(input: TokenStream, context: &mut GrammarContext) -> Resul let location = Location::Range(location, location + 1); match punct.as_char() { - ':' => context.feed_location(Lexed::Colon(punct), location)?, - ';' => context.feed_location(Lexed::Semicolon(punct), location)?, - '|' => context.feed_location(Lexed::Pipe(punct), location)?, - '+' => context.feed_location(Lexed::Plus(punct), location)?, - '*' => context.feed_location(Lexed::Star(punct), location)?, - '?' => context.feed_location(Lexed::Question(punct), location)?, - '^' => context.feed_location(Lexed::Caret(punct), location)?, - '-' => context.feed_location(Lexed::Minus(punct), location)?, - '=' => context.feed_location(Lexed::Equal(punct), location)?, - '!' => context.feed_location(Lexed::Exclamation(punct), location)?, - '.' => context.feed_location(Lexed::Dot(punct), location)?, - '%' => context.feed_location(Lexed::Percent(punct), location)?, - '$' => context.feed_location(Lexed::Dollar(punct), location)?, - ',' => context.feed_location(Lexed::Comma(punct), location)?, - _ => context.feed_location(Lexed::OtherPunct(punct), location)?, + ':' => feed_location(context, Lexed::Colon(punct), location)?, + ';' => feed_location(context, Lexed::Semicolon(punct), location)?, + '|' => feed_location(context, Lexed::Pipe(punct), location)?, + '+' => feed_location(context, Lexed::Plus(punct), location)?, + '*' => feed_location(context, Lexed::Star(punct), location)?, + '?' => feed_location(context, Lexed::Question(punct), location)?, + '^' => feed_location(context, Lexed::Caret(punct), location)?, + '-' => feed_location(context, Lexed::Minus(punct), location)?, + '=' => feed_location(context, Lexed::Equal(punct), location)?, + '!' => feed_location(context, Lexed::Exclamation(punct), location)?, + '.' => feed_location(context, Lexed::Dot(punct), location)?, + '%' => feed_location(context, Lexed::Percent(punct), location)?, + '$' => feed_location(context, Lexed::Dollar(punct), location)?, + ',' => feed_location(context, Lexed::Comma(punct), location)?, + _ => feed_location(context, Lexed::OtherPunct(punct), location)?, } } TokenTree::Group(group) => match group.delimiter() { @@ -290,7 +300,7 @@ pub fn feed_recursive(input: TokenStream, context: &mut GrammarContext) -> Resul if context.can_feed(&token) { let location = context.userdata_mut().span_manager.add_span(span); let location = Location::Range(location, location + 1); - context.feed_location(token, location)?; + feed_location(context, token, location)?; } else { let Lexed::ParenGroup(group) = token else { unreachable!(); @@ -304,7 +314,7 @@ pub fn feed_recursive(input: TokenStream, context: &mut GrammarContext) -> Resul let open_location = Location::Range(open_location, open_location + 1); // feed the splitted tokens - context.feed_location(Lexed::LParen, open_location)?; + feed_location(context, Lexed::LParen, open_location)?; feed_recursive(group.stream(), context)?; let close_span = group.span_close(); @@ -313,21 +323,21 @@ pub fn feed_recursive(input: TokenStream, context: &mut GrammarContext) -> Resul .span_manager .add_span(close_span.clone()); let close_location = Location::Range(close_location, close_location + 1); - context.feed_location(Lexed::RParen, close_location)?; + feed_location(context, Lexed::RParen, close_location)?; } } Delimiter::Brace => { let location = context.userdata_mut().span_manager.add_span(span); let location = Location::Range(location, location + 1); // for now, splitted for brace is not in syntax, so ignore it - context.feed_location(Lexed::BraceGroup(group), location)?; + feed_location(context, Lexed::BraceGroup(group), location)?; } Delimiter::Bracket => { let token = Lexed::BracketGroup(group); if context.can_feed(&token) { let location = context.userdata_mut().span_manager.add_span(span); let location = Location::Range(location, location + 1); - context.feed_location(token, location)?; + feed_location(context, token, location)?; } else { let Lexed::BracketGroup(group) = token else { unreachable!(); @@ -339,7 +349,7 @@ pub fn feed_recursive(input: TokenStream, context: &mut GrammarContext) -> Resul .add_span(open_span.clone()); let open_location = Location::Range(open_location, open_location + 1); // feed the splitted tokens - context.feed_location(Lexed::LBracket, open_location)?; + feed_location(context, Lexed::LBracket, open_location)?; feed_recursive(group.stream(), context)?; let close_span = group.span_close(); @@ -348,14 +358,14 @@ pub fn feed_recursive(input: TokenStream, context: &mut GrammarContext) -> Resul .span_manager .add_span(close_span.clone()); let close_location = Location::Range(close_location, close_location + 1); - context.feed_location(Lexed::RBracket, close_location)?; + feed_location(context, Lexed::RBracket, close_location)?; } } _ => { let location = context.userdata_mut().span_manager.add_span(span); let location = Location::Range(location, location + 1); // for now, compound for nonegroup is not in syntax, so ignore it - context.feed_location(Lexed::NoneGroup(group), location)?; + feed_location(context, Lexed::NoneGroup(group), location)?; } }, TokenTree::Literal(literal) => { @@ -379,7 +389,7 @@ pub fn feed_recursive(input: TokenStream, context: &mut GrammarContext) -> Resul }; let location = context.userdata_mut().span_manager.add_span(span); let location = Location::Range(location, location + 1); - context.feed_location(term, location)? + feed_location(context, term, location)? } }; }