diff --git a/GLR.md b/GLR.md index a51d308e..ec904627 100644 --- a/GLR.md +++ b/GLR.md @@ -93,12 +93,22 @@ E(i32) ## Parsing with the GLR Parser -Initialize the generated context with initial user data (or `with_default_userdata()` when the user data type implements `Default`), and feed your terminal symbols (tokens) to it. The GLR parser shares a similar interface to the deterministic parser: `accept()` returns the first successful `(parse_result, userdata)` pair, while `accept_all()` returns an iterator over all successful pairs with the start type already extracted. You can feed terminal symbols (tokens) using either `feed` (basic) or `feed_location` (location-aware). +Initialize the generated context with initial user data (or `with_default_userdata()` when the user data type implements `Default`), and feed your terminal symbols (tokens) to it. The GLR parser shares a similar interface to the deterministic parser: `accept()` mutably borrows the context and returns the first successful `(parse_result, userdata)` pair, while `accept_all()` returns an iterator over all successful pairs with the start type already extracted. Successful acceptance moves branch results out and consumes the context; later feeds or accepts return a consumed-context error. You can feed terminal symbols (tokens) using either `feed` (basic) or `feed_location` (location-aware). In GLR mode, user data is branch-local. When the parser forks because of a conflict, the current semantic stack and user data are cloned, and each resulting branch owns and mutates its own `UserData` value independently. Token values, semantic return values, and user data must therefore implement `Clone` for GLR branch-local parsing. Results returned from `accept_all()` include the final user data for each successful branch. `NoAction` is the only feed failure that means the CFG cannot consume the lookahead terminal. `ReduceAction` means the terminal was grammatically feedable, but a runtime semantic action failed. GLR recovery is entered only when every active branch fails with `NoAction`; if at least one branch survives, the feed succeeds and any sibling branch failures are returned in `FeedSuccess::errors`. +GLR parse errors keep branch diagnostics grouped. `ParseError::branch_errors` is a list of `ParseErrorBranch` values; each value represents one failed nondeterministic branch for the current lookahead and preserves that branch's parser state, branch-local user data, and reduce-action error when the failure was semantic. When at least one branch survives, the context remains usable and the failed branch user data is available only through the reported branch error. When no branch survives, branches that only saw `NoAction` are restored for reuse and branches that failed with `ReduceAction` are consumed. Later feeds or accepts report a consumed-context error only if no reusable branch remains. + +Branch reuse follows the same rule for `feed()`, `feed_location()`, `accept()`, and `accept_all()`: + +- A branch that fails with `NoAction` has not run a semantic action for the lookahead, so its stack and user data are unchanged. If no branch succeeds, that branch is restored into the context and can be fed again. +- A branch that fails with `ReduceAction` may have partially mutated semantic state. That branch is consumed, and its user data is moved into `ParseErrorBranch::ReduceAction`. +- If at least one branch succeeds, the successful branches become the next active context. Failed sibling branches are pruned and reported through `FeedSuccess::errors`. +- If no branch succeeds but at least one `NoAction` branch remains reusable, the call returns `Err(ParseError)` while keeping those reusable branches active for the next call. +- If every branch has been consumed, the context itself is consumed. Later `feed()` or `accept()` calls return a consumed-context error. + For example, this ambiguous grammar creates two branches for the same input terminal symbol. Each branch mutates its own cloned `Vec<&'static str>`: ```rust @@ -188,5 +198,6 @@ fn main() { - **`context.feed(token)`**: Feeds a token into all active parsing stacks and returns `FeedSuccess` when at least one branch survives. - **`context.feed_location(token, location)`**: Feeds a token with its location span into all active parsing stacks (requires `%location` in the grammar) and returns `FeedSuccess` when at least one branch survives. - **`FeedSuccess::errors`**: Optional branch-failure information from GLR branches pruned while the feed still succeeded. -- **`context.accept()`**: Finalizes parsing (feeding the end-of-file symbol) and returns the first successful `(parse_result, userdata)` pair. -- **`context.accept_all()`**: Finalizes parsing and returns an iterator over successful `(parse_result, userdata)` pairs from all active branches. +- **`ParseError::branch_errors`**: Branch-grouped diagnostics. Each `ParseErrorBranch` is either `NoAction { state, userdata }` or `ReduceAction { state, source, userdata }`. +- **`context.accept()`**: Finalizes parsing (feeding the end-of-file symbol) and returns the first successful `(parse_result, userdata)` pair. Branches that only see `NoAction` remain reusable; success consumes the context. +- **`context.accept_all()`**: Finalizes parsing and returns an iterator over successful `(parse_result, userdata)` pairs from all active branches. Branches that only see `NoAction` remain reusable; success consumes the context. diff --git a/README.md b/README.md index 3e9f2c58..16844649 100644 --- a/README.md +++ b/README.md @@ -167,7 +167,7 @@ context.feed(token); context.feed_location(token, token_location); ``` -Feed failures are classified by cause. `NoAction` means the CFG cannot consume the lookahead terminal; it leaves the parser stack unchanged and is the only failure kind that can enter panic-mode recovery. `ReduceAction` means the terminal was grammatically feedable, but a runtime semantic action failed. In GLR mode, recovery is attempted only when every active branch fails with `NoAction`. +Feed failures are classified by cause. `NoAction` means the CFG cannot consume the lookahead terminal; it leaves the parser stack and user data unchanged, and the same context can be fed again. `NoAction` is the only failure kind that can enter panic-mode recovery. `ReduceAction` means the terminal was grammatically feedable, but a runtime semantic action failed. In deterministic mode, `ReduceAction` moves user data into the error and consumes the context; later `feed()` or `accept()` attempts return `ParseError::ConsumedContext`. `accept()` borrows the context mutably: `NoAction` leaves it reusable, while successful acceptance moves the result and user data out and consumes the context. In GLR mode, `ReduceAction` consumes only the branch that failed. If every branch fails, branches that only saw `NoAction` are restored and the context remains reusable; the whole context is consumed only when no reusable branch remains. --- @@ -175,7 +175,7 @@ Feed failures are classified by cause. `NoAction` means the CFG cannot consume t RustyLR provides native support for Generalized LR (GLR) parsing. When you add the `%glr;` directive to a grammar, RustyLR generates a non-deterministic parser that forks state branches upon encountering shift/reduce or reduce/reduce conflicts. This is particularly useful for ambiguous grammars or complex programming languages. -GLR `feed()` and `feed_location()` return a success value that may contain errors from branches pruned while another branch survived. Treat the feed as failed only when the returned `Result` is `Err`; inspect the success value when branch-level diagnostics matter. +GLR `feed()` and `feed_location()` return a success value that may contain errors from branches pruned while another branch survived. Treat the feed as failed only when the returned `Result` is `Err`; inspect the success value when branch-level diagnostics matter. GLR `ParseError::branch_errors` stores `ParseErrorBranch` values, each preserving the failed branch's state, branch-local user data, and, for semantic failures, its reduce-action error. Surviving GLR branches remain usable after a successful feed with pruned-branch errors. When no branch survives, `NoAction` branches are restored for reuse and consumed branches are reported in the error; later operations return a consumed-context error only if no reusable branch remains. Successful `accept()`/`accept_all()` moves accepted branch results out and consumes the context. In GLR mode, user data is branch-local. When the parser forks, the current user data is cloned so each active branch owns and mutates an independent `UserData` value. diff --git a/SYNTAX.md b/SYNTAX.md index 4d3e3974..b1cb8883 100644 --- a/SYNTAX.md +++ b/SYNTAX.md @@ -463,7 +463,7 @@ Expr Defines a custom error type returned by reduce actions. If your reduce action returns a `Result`, returning an `Err(custom_error)` will stop execution (or prune the GLR branch) and bubble the error up wrapped in `ParseError::ReduceAction`. -A reduce-action error is semantic failure, not a syntax `NoAction`. It does not enter panic-mode error recovery. In GLR mode, if another branch successfully shifts the same terminal, the feed succeeds and reports the pruned-branch error in the GLR feed success value. +A reduce-action error is semantic failure, not a syntax `NoAction`. It does not enter panic-mode error recovery. In deterministic mode, `NoAction` leaves the context unchanged and reusable, while `ReduceAction` moves user data into the error because the reduce action may have partially mutated semantic state; later `feed()` or `accept()` attempts on that context return `ParseError::ConsumedContext`. `accept()` and `accept_all()` mutably borrow the context: `NoAction` leaves it reusable, while success moves the accepted result and user data out and consumes it. In GLR mode, if another branch successfully shifts the same terminal, the feed succeeds and reports the pruned-branch error in the GLR feed success value. If no branch survives, branches that only saw `NoAction` are restored for reuse and branches that failed with `ReduceAction` are consumed; later `feed()` or `accept()` reports a consumed-context error only if no reusable branch remains. GLR parse errors expose `branch_errors`, where each `ParseErrorBranch` is either `NoAction { state, userdata }` or `ReduceAction { state, source, userdata }` for one nondeterministic branch. ## No Optimization diff --git a/USING_RUSTYLR_WITH_AI.md b/USING_RUSTYLR_WITH_AI.md index 428d9b8a..744a3fb2 100644 --- a/USING_RUSTYLR_WITH_AI.md +++ b/USING_RUSTYLR_WITH_AI.md @@ -131,7 +131,7 @@ While implementing: - Use `%left`, `%right`, or `%precedence` before changing grammar shape solely to fix expression conflicts. - Use the `error` terminal for recovery points such as delimiters, statements, or object members. - Include `error` in precedence declarations if recovery productions introduce a shift/reduce conflict. -- Treat reduce-action `Err` values as semantic failures, not recovery triggers. In GLR mode, a successful feed may still return branch errors through its success value when other branches were pruned. +- Treat reduce-action `Err` values as semantic failures, not recovery triggers. Deterministic `NoAction` leaves the context reusable, but deterministic `ReduceAction` moves user data into the error and consumes the context; later `feed()` or `accept()` returns `ParseError::ConsumedContext`. `accept()`/`accept_all()` borrow mutably: `NoAction` can be retried after more input, while success consumes the context. In GLR mode, a successful feed may still return branch errors through its success value when other branches were pruned; inspect `ParseError::branch_errors` to distinguish per-branch `NoAction { state, userdata }` and `ReduceAction { state, source, userdata }` failures. If no GLR branch survives, `NoAction` branches are restored for reuse and consumed branches are reported in the error; subsequent operations return consumed-context errors only if no reusable branch remains. - Keep generated parser files committed only if the project convention does so. When debugging: diff --git a/llms.txt b/llms.txt index 828ee490..660c7fb7 100644 --- a/llms.txt +++ b/llms.txt @@ -27,7 +27,7 @@ Generated `Context` types are the normal parser contexts downstream code Feed failures have two main runtime causes. `NoAction` means the CFG cannot consume the lookahead terminal, leaves the parser stack unchanged, and is the only failure kind that can enter panic-mode recovery. `ReduceAction` means the terminal was grammatically feedable, but runtime execution failed. -Reduce-action errors are semantic failures, not panic-mode recovery triggers. GLR feed success values may contain branch errors when one branch survives and sibling branches are pruned. +Reduce-action errors are semantic failures, not panic-mode recovery triggers. Deterministic `NoAction` leaves the context reusable, but deterministic `ReduceAction` moves user data into the error and consumes the context; later `feed()` or `accept()` returns `ParseError::ConsumedContext`. `accept()`/`accept_all()` borrow mutably: `NoAction` can be retried after more input, while success consumes the context. GLR feed success values may contain branch errors when one branch survives and sibling branches are pruned. GLR `ParseError::branch_errors` stores per-branch `NoAction { state, userdata }` or `ReduceAction { state, source, userdata }` failures. If no GLR branch survives, `NoAction` branches are restored for reuse and consumed branches are reported in the error; subsequent operations return consumed-context errors only if no reusable branch remains. Generated contexts implement `Clone` when their runtime storage and user data satisfy the required `Clone` bounds. Choose token and semantic value types that implement `Clone` when cloning contexts or using GLR branch-local user data; user data must also implement `Clone` in those cases. diff --git a/rusty_lr/tests/lr_glr_operation.rs b/rusty_lr/tests/lr_glr_operation.rs index 95fc13e4..6716f7fe 100644 --- a/rusty_lr/tests/lr_glr_operation.rs +++ b/rusty_lr/tests/lr_glr_operation.rs @@ -123,7 +123,7 @@ mod pattern_operators { #[test] fn separated_pattern_rejects_missing_required_item() { - let ctx = CsvContext::with_default_userdata(); + let mut ctx = CsvContext::with_default_userdata(); assert!(ctx.accept().is_err()); let mut ctx = CsvContext::with_default_userdata(); @@ -477,6 +477,113 @@ mod reduce_action_errors { assert!(too_large.accept().is_err()); } + #[test] + fn deterministic_accept_no_action_is_reusable_but_success_consumes_context() { + let mut ctx = NumberContext::with_default_userdata(); + + let too_early = ctx.accept().unwrap_err(); + assert_eq!(too_early.reduce_action_error(), None); + assert!(!too_early.is_consumed_context()); + + ctx.feed('7').unwrap(); + assert_eq!(ctx.accept().unwrap(), (7, ())); + + let consumed = ctx.feed('8').unwrap_err(); + assert!(consumed.is_consumed_context()); + } + + mod deterministic_error_userdata { + use rusty_lr::lr1; + + lr1! { + %nooptim; + %tokentype char; + %userdata Vec<&'static str>; + %error &'static str; + %start S; + + S(()): A 'y' { () }; + + A(()): 'x' { + data.push("reduced a"); + if true { + return Err("bad a"); + } + () + }; + } + + #[test] + fn deterministic_reduce_action_error_returns_userdata() { + let mut ctx = SContext::new(vec!["seed"]); + ctx.feed('x').unwrap(); + + let err = ctx.feed('y').unwrap_err(); + + assert_eq!(err.reduce_action_error().unwrap(), &"bad a"); + assert_eq!(err.userdata(), Some(&vec!["seed", "reduced a"])); + assert!(ctx.userdata_all().next().is_none()); + assert!(!ctx.can_feed(&'y')); + assert!(!ctx.can_accept()); + + let consumed = ctx.feed('y').unwrap_err(); + assert!(consumed.is_consumed_context()); + assert!(consumed.userdata().is_none()); + } + } + + mod glr_accept_consume { + use rusty_lr::lr1; + + lr1! { + %glr; + %tokentype char; + %start S; + + S(char): 'x' { 'x' }; + } + + #[test] + fn glr_accept_success_consumes_context() { + let mut ctx = SContext::with_default_userdata(); + ctx.feed('x').unwrap(); + assert_eq!(ctx.accept().unwrap(), ('x', ())); + + let consumed = ctx.feed('x').unwrap_err(); + assert!(consumed.is_consumed_context()); + assert!(!ctx.can_accept()); + } + + #[test] + fn glr_feed_no_action_is_reusable() { + let mut ctx = SContext::with_default_userdata(); + + let err = ctx.feed('y').unwrap_err(); + assert_eq!(err.branch_errors.len(), 1); + assert!(err.branch_errors[0].reduce_action_error().is_none()); + assert!(!err.is_consumed_context()); + assert!(ctx.can_feed(&'x')); + + ctx.feed('x').unwrap(); + assert_eq!(ctx.accept().unwrap(), ('x', ())); + } + + #[test] + fn glr_accept_no_action_is_reusable() { + let mut ctx = SContext::with_default_userdata(); + + let err = ctx.accept().unwrap_err(); + assert_eq!(err.branch_errors.len(), 1); + assert!(err.branch_errors[0].reduce_action_error().is_none()); + assert!(!err.is_consumed_context()); + assert!(!ctx.can_accept()); + assert!(ctx.can_feed(&'x')); + + ctx.feed('x').unwrap(); + assert_eq!(ctx.accept().unwrap(), ('x', ())); + } + } + mod glr_recovery { use rusty_lr::lr1; @@ -501,6 +608,7 @@ mod reduce_action_errors { ; Rejected(char): 'x' { + data.push("rejected branch"); if true { return Err("rejected token"); } @@ -517,8 +625,20 @@ mod reduce_action_errors { // grammatically valid and dies from a reduce-action error instead. let err = ctx.feed('y').unwrap_err(); - assert_eq!(err.reduce_action_errors, ["rejected token"]); + assert_eq!(err.branch_errors.len(), 1); + assert_eq!( + err.branch_errors[0].reduce_action_error(), + Some(&"rejected token") + ); + assert_eq!(err.branch_errors[0].userdata(), &vec!["rejected branch"]); assert!(ctx.userdata_all().all(|userdata| userdata.is_empty())); + assert!(!ctx.can_feed(&'y')); + assert!(!ctx.can_accept()); + + let consumed = ctx.feed('y').unwrap_err(); + assert!(consumed.is_consumed_context()); + let consumed = ctx.accept().unwrap_err(); + assert!(consumed.is_consumed_context()); } } @@ -561,11 +681,210 @@ mod reduce_action_errors { let success = ctx.feed('y').unwrap(); let errors = success.errors.expect("reduce branch should be reported"); - assert_eq!(errors.reduce_action_errors, ["bad reduce path"]); + assert_eq!(errors.branch_errors.len(), 1); + assert_eq!( + errors.branch_errors[0].reduce_action_error(), + Some(&"bad reduce path") + ); assert_eq!(ctx.accept().unwrap(), ("shift path", ())); } } + mod glr_multiple_branch_errors { + use rusty_lr::lr1; + + // Both reduce alternatives are grammatically valid for the lookahead, but each semantic + // action rejects its own GLR branch. The resulting parse error should preserve that branch + // boundary instead of flattening the reduce-action errors into one parser-wide list. + lr1! { + %glr; + %nooptim; + %tokentype char; + %error &'static str; + %start S; + + S(&'static str) + : A 'y' { + "a" + } + | B 'y' { + "b" + } + ; + + A(char): 'x' { + if true { + return Err("bad a"); + } + 'a' + }; + + B(char): 'x' { + if true { + return Err("bad b"); + } + 'b' + }; + } + + #[test] + fn glr_parse_error_keeps_reduce_errors_grouped_by_branch() { + let mut ctx = SContext::with_default_userdata(); + ctx.feed('x').unwrap(); + + let err = ctx.feed('y').unwrap_err(); + + assert_eq!(err.branch_errors.len(), 2); + for branch in &err.branch_errors { + assert_eq!(branch.states().count(), 1); + assert!(branch.reduce_action_error().is_some()); + } + + let mut reduce_errors: Vec<_> = err + .branch_errors + .iter() + .map(|branch| *branch.reduce_action_error().unwrap()) + .collect(); + reduce_errors.sort(); + assert_eq!(reduce_errors, ["bad a", "bad b"]); + + assert!(!err.is_consumed_context()); + assert!(!ctx.can_feed(&'y')); + assert!(!ctx.can_accept()); + + let consumed = ctx.feed('y').unwrap_err(); + assert!(consumed.is_consumed_context()); + let consumed = ctx.accept().unwrap_err(); + assert!(consumed.is_consumed_context()); + } + } + + mod glr_mixed_branch_failure_reuse { + mod feed { + use rusty_lr::lr1; + + lr1! { + %glr; + %nooptim; + %tokentype char; + %error &'static str; + %start S; + + S(&'static str) + : A 'q' { + "a" + } + | B 'r' { + "b" + } + ; + + A(()): X 'p' { + if true { + return Err("bad reduce"); + } + () + }; + + B(()): Y 'p' { () }; + X(()): 'x' { () }; + Y(()): 'x' { () }; + } + + #[test] + fn glr_feed_restores_no_action_branch_when_sibling_reduce_fails() { + let mut ctx = SContext::with_default_userdata(); + ctx.feed('x').unwrap(); + ctx.feed('p').unwrap(); + + let err = ctx.feed('q').unwrap_err(); + + assert!(!err.is_consumed_context()); + assert_eq!(err.branch_errors.len(), 2); + assert_eq!( + err.branch_errors + .iter() + .filter(|branch| branch.reduce_action_error().is_none()) + .count(), + 1 + ); + assert_eq!( + err.branch_errors + .iter() + .filter(|branch| branch.reduce_action_error() == Some(&"bad reduce")) + .count(), + 1 + ); + assert!(ctx.can_feed(&'r')); + + ctx.feed('r').unwrap(); + assert_eq!(ctx.accept().unwrap(), ("b", ())); + } + } + + mod accept { + use rusty_lr::lr1; + + lr1! { + %glr; + %nooptim; + %tokentype char; + %error &'static str; + %start S; + + S(&'static str) + : A { + "a" + } + | B 'r' { + "b" + } + ; + + A(()): X 'p' { + if true { + return Err("bad reduce"); + } + () + }; + + B(()): Y 'p' { () }; + X(()): 'x' { () }; + Y(()): 'x' { () }; + } + + #[test] + fn glr_accept_restores_no_action_branch_when_sibling_reduce_fails() { + let mut ctx = SContext::with_default_userdata(); + ctx.feed('x').unwrap(); + ctx.feed('p').unwrap(); + + let err = ctx.accept().unwrap_err(); + + assert!(!err.is_consumed_context()); + assert_eq!(err.branch_errors.len(), 2); + assert_eq!( + err.branch_errors + .iter() + .filter(|branch| branch.reduce_action_error().is_none()) + .count(), + 1 + ); + assert_eq!( + err.branch_errors + .iter() + .filter(|branch| branch.reduce_action_error() == Some(&"bad reduce")) + .count(), + 1 + ); + assert!(ctx.can_feed(&'r')); + + ctx.feed('r').unwrap(); + assert_eq!(ctx.accept().unwrap(), ("b", ())); + } + } + } + mod glr_recovery_partial_errors { use rusty_lr::lr1; @@ -605,7 +924,11 @@ mod reduce_action_errors { let errors = success .errors .expect("failed recovery branch should be reported"); - assert_eq!(errors.reduce_action_errors, ["bad recovery branch"]); + assert_eq!(errors.branch_errors.len(), 1); + assert_eq!( + errors.branch_errors[0].reduce_action_error(), + Some(&"bad recovery branch") + ); ctx.debug_check(); ctx.feed('s').unwrap(); @@ -656,7 +979,11 @@ mod reduce_action_errors { let errors = success .errors .expect("failed sibling branch should be reported"); - assert_eq!(errors.reduce_action_errors, ["bad nested branch"]); + assert_eq!(errors.branch_errors.len(), 1); + assert_eq!( + errors.branch_errors[0].reduce_action_error(), + Some(&"bad nested branch") + ); ctx.debug_check(); assert_eq!(ctx.accept().unwrap(), ("good", ())); diff --git a/rusty_lr/tests/ui/context_clone_non_clone_storage_compiles.rs b/rusty_lr/tests/ui/context_clone_non_clone_storage_compiles.rs index de67f143..2505e402 100644 --- a/rusty_lr/tests/ui/context_clone_non_clone_storage_compiles.rs +++ b/rusty_lr/tests/ui/context_clone_non_clone_storage_compiles.rs @@ -10,6 +10,6 @@ lr1! { } fn main() { - let context = SContext::with_default_userdata(); + let mut context = SContext::with_default_userdata(); let (_value, _userdata) = context.accept().unwrap(); } diff --git a/rusty_lr_core/src/parser/deterministic/context.rs b/rusty_lr_core/src/parser/deterministic/context.rs index 8060da53..26865f10 100644 --- a/rusty_lr_core/src/parser/deterministic/context.rs +++ b/rusty_lr_core/src/parser/deterministic/context.rs @@ -43,6 +43,17 @@ struct FeedPlanError { state: usize, } +struct NoActionFailure { + term: TerminalSymbol, + location: Location, + state: usize, +} + +enum FeedLocationError { + NoAction(NoActionFailure), + ReduceAction(ParseError), +} + /// A struct that maintains the current state and the values associated with each symbol. pub struct Context< P: Parser, @@ -54,7 +65,12 @@ pub struct Context< pub state_stack: Vec, pub(crate) data_stack: Vec, pub(crate) location_stack: Vec, - pub(crate) userdata: Data::UserData, + /// User data owned by this deterministic parse path. + /// + /// `NoAction` is detected before semantic actions run, so it leaves this value in place and + /// the context can be fed again. A reduce-action error may leave semantic state partially + /// mutated, so it moves this value into `ParseError::ReduceAction` and consumes the context. + pub(crate) userdata: Option, /// Decoded parser tables initialized when the context is created. /// /// Keeping this reference in the context avoids repeated `Parser::get_tables()` calls in the @@ -87,7 +103,7 @@ impl< data_stack: Vec::new(), location_stack: Vec::new(), - userdata, + userdata: Some(userdata), tables: P::get_tables(), feed_extra_state_stack: RefCell::new(Vec::new()), @@ -133,7 +149,7 @@ impl< data_stack: Vec::with_capacity(capacity), location_stack: Vec::with_capacity(capacity), - userdata, + userdata: Some(userdata), tables: P::get_tables(), feed_extra_state_stack: RefCell::new(Vec::with_capacity(capacity)), @@ -151,24 +167,46 @@ impl< { Self::with_capacity(capacity, Default::default()) } + fn userdata_ref(&self) -> &Data::UserData { + self.userdata + .as_ref() + .expect("parser context userdata was moved into a previous reduce-action parse error") + } + + fn userdata_mut_ref(&mut self) -> &mut Data::UserData { + self.userdata + .as_mut() + .expect("parser context userdata was moved into a previous reduce-action parse error") + } + + fn take_userdata(&mut self) -> Data::UserData { + self.userdata + .take() + .expect("parser context userdata was moved into a previous reduce-action parse error") + } + + fn is_consumed(&self) -> bool { + self.userdata.is_none() + } + /// Borrow the user data owned by this context. pub fn userdata(&self) -> &Data::UserData { - &self.userdata + self.userdata_ref() } /// Borrow the user data owned by this context as an iterator. pub fn userdata_all(&self) -> impl Iterator { - std::iter::once(&self.userdata) + self.userdata.iter() } /// Mutably borrow the user data owned by this context. pub fn userdata_mut(&mut self) -> &mut Data::UserData { - &mut self.userdata + self.userdata_mut_ref() } /// Mutably borrow the user data owned by this context as an iterator. pub fn userdata_all_mut(&mut self) -> impl Iterator { - std::iter::once(&mut self.userdata) + self.userdata.iter_mut() } fn state_from_len(&self, stack_len: usize) -> usize { @@ -188,11 +226,15 @@ impl< } /// End this context and pop the value of the start symbol from the data stack. + /// + /// `ParseError::NoAction` leaves the context reusable, so callers may feed more tokens and + /// call `accept` again. Successful acceptance moves the start value and user data out of the + /// context, consuming it. A reduce-action failure also consumes the context. pub fn accept( - mut self, + &mut self, ) -> Result< (Start::StartType, Data::UserData), - ParseError, + ParseError, > where Data::Term: Clone, @@ -209,15 +251,20 @@ impl< Start::BRANCH_INDEX ) }); - Ok((start, self.userdata)) + Ok(( + start, + self.userdata.take().expect( + "parser context userdata was moved into a previous reduce-action parse error", + ), + )) } /// End this context and return an iterator with the start symbol and final user data. pub fn accept_all( - self, + &mut self, ) -> Result< impl Iterator, - ParseError, + ParseError, > where Data::Term: Clone, @@ -228,6 +275,9 @@ impl< pub fn can_accept(&self) -> bool { // EOF acceptance uses the same grammatical feed simulation as regular terminals. + if self.is_consumed() { + return false; + } self.plan_feed(P::TermClass::EOF).is_ok() } @@ -337,7 +387,7 @@ impl< pub fn feed( &mut self, term: Data::Term, - ) -> Result<(), ParseError> + ) -> Result<(), ParseError> where P::Term: Clone, P::NonTerm: std::fmt::Debug, @@ -347,26 +397,35 @@ impl< } /// Feed one terminal with location to parser, and update stacks. + /// + /// If this returns `ParseError::NoAction`, no reduce action has run for the lookahead and the + /// context remains reusable. If this returns `ParseError::ReduceAction`, the user data has + /// been moved into the error and this context is consumed. Calling `feed` or `accept` on a + /// consumed context returns `ParseError::ConsumedContext`. pub fn feed_location( &mut self, term: P::Term, location: Data::Location, - ) -> Result<(), ParseError> + ) -> Result<(), ParseError> where P::Term: Clone, P::NonTerm: std::fmt::Debug, { use crate::Location; + if self.is_consumed() { + return Err(self.consumed_context_error(TerminalSymbol::Terminal(term), location)); + } + let class = P::TermClass::from_term(&term); match self.feed_location_impl(TerminalSymbol::Terminal(term), class, location) { Ok(()) => Ok(()), - Err(ParseError::NoAction(err)) => { + Err(FeedLocationError::NoAction(err)) => { // nothing shifted; enters panic mode // if `error` token was not used in the grammar, early return here if !P::ERROR_USED { - return Err(ParseError::NoAction(err)); + return Err(Self::no_action_error(err)); } let mut pop_count = 0; @@ -385,7 +444,7 @@ impl< } let Some(error_plan) = error_plan else { - return Err(ParseError::NoAction(err)); + return Err(Self::no_action_error(err)); }; let error_location = @@ -423,7 +482,7 @@ impl< } } } - Err(err) => Err(err), + Err(FeedLocationError::ReduceAction(err)) => Err(err), } } @@ -432,7 +491,10 @@ impl< term: TerminalSymbol, class: P::TermClass, location: Data::Location, - ) -> Result<(), ParseError> + ) -> Result< + (), + FeedLocationError, + > where P::Term: Clone, P::NonTerm: std::fmt::Debug, @@ -442,7 +504,7 @@ impl< let plan = match self.plan_feed(class) { Ok(plan) => plan, Err(err) => { - return Err(ParseError::NoAction(super::error::NoActionError { + return Err(FeedLocationError::NoAction(NoActionFailure { term, location, state: err.state, @@ -450,6 +512,31 @@ impl< } }; self.apply_feed_plan(plan, term, location) + .map_err(FeedLocationError::ReduceAction) + } + + fn no_action_error( + err: NoActionFailure, + ) -> ParseError { + // NoAction is raised by CFG-only planning before any semantic action runs, so user data + // stays in the context and the caller may recover by feeding another token. + ParseError::NoAction(super::error::NoActionError { + term: err.term, + location: err.location, + state: err.state, + }) + } + + fn consumed_context_error( + &self, + term: TerminalSymbol, + location: Data::Location, + ) -> ParseError { + ParseError::ConsumedContext(super::error::ConsumedContextError { + term, + location, + state: self.state(), + }) } /// Simulate the deterministic LR stack operations needed to feed one terminal. @@ -556,7 +643,7 @@ impl< plan: FeedPlan, term: TerminalSymbol, location: Data::Location, - ) -> Result<(), ParseError> + ) -> Result<(), ParseError> where P::Term: Clone, P::NonTerm: std::fmt::Debug, @@ -580,16 +667,23 @@ impl< reduction.rule_index, &mut shift, &term, - &mut self.userdata, + self.userdata.as_mut().expect( + "parser context userdata was moved into a previous reduce-action parse error", + ), &mut new_location, ) { Ok(_) => {} Err(err) => { + let state = self.state(); + // A user reduce action may have mutated stacks or user data before failing. + // Move user data into the error and leave the context unusable. + let userdata = self.take_userdata(); return Err(ParseError::ReduceAction(super::error::ReduceActionError { term, location, - state: self.state(), + state, source: err, + userdata, })); } }; @@ -636,6 +730,9 @@ impl< /// So this function will return `false` even if term can be shifted as `error` token, /// and will return `true` if `Err` variant is returned by `reduce_action`. pub fn can_feed(&self, term: &Data::Term) -> bool { + if self.is_consumed() { + return false; + } let class = P::TermClass::from_term(term); self.plan_feed(class).is_ok() @@ -643,6 +740,9 @@ impl< /// Check if current context can enter panic mode pub fn can_panic(&self) -> bool { + if self.is_consumed() { + return false; + } // if `error` token was not used in the grammar, early return here if !P::ERROR_USED { return false; @@ -667,13 +767,21 @@ impl< fn feed_eof( &mut self, - ) -> Result<(), ParseError> + ) -> Result<(), ParseError> where P::Term: Clone, P::NonTerm: std::fmt::Debug, { let eof_location = Data::Location::new(self.location_stack.iter().rev(), 0); - self.feed_location_impl(TerminalSymbol::Eof, P::TermClass::EOF, eof_location) + if self.is_consumed() { + return Err(self.consumed_context_error(TerminalSymbol::Eof, eof_location)); + } + + match self.feed_location_impl(TerminalSymbol::Eof, P::TermClass::EOF, eof_location) { + Ok(()) => Ok(()), + Err(FeedLocationError::NoAction(err)) => Err(Self::no_action_error(err)), + Err(FeedLocationError::ReduceAction(err)) => Err(err), + } } } @@ -1067,7 +1175,8 @@ mod tests { assert_eq!(ctx.state_stack, state_stack); assert_eq!(ctx.data_stack, data_stack); assert_eq!(ctx.location_stack, location_stack); - assert!(ctx.userdata.is_empty()); + assert!(err.userdata().is_none()); + assert!(ctx.userdata.as_ref().unwrap().is_empty()); } #[test] @@ -1079,7 +1188,7 @@ mod tests { // use the reduction discovered by pre-feed simulation. ctx.feed('b').unwrap(); - assert_eq!(ctx.userdata, vec!["reduced"]); + assert_eq!(ctx.userdata.as_ref().unwrap(), &vec!["reduced"]); assert_eq!(ctx.state_stack, vec![1, 3, 4, 5]); assert_eq!( ctx.data_stack, diff --git a/rusty_lr_core/src/parser/deterministic/error.rs b/rusty_lr_core/src/parser/deterministic/error.rs index 9f2c0822..1ece50a7 100644 --- a/rusty_lr_core/src/parser/deterministic/error.rs +++ b/rusty_lr_core/src/parser/deterministic/error.rs @@ -10,28 +10,51 @@ pub struct NoActionError { pub state: usize, } #[derive(Clone, Debug)] -pub struct ReduceActionError { +pub struct ReduceActionError { pub term: TerminalSymbol, pub location: Location, pub state: usize, pub source: Source, + pub userdata: UserData, +} +#[derive(Clone, Debug)] +pub struct ConsumedContextError { + pub term: TerminalSymbol, + pub location: Location, + pub state: usize, } /// Error type for feed() #[derive(Clone, Debug)] -pub enum ParseError { +pub enum ParseError { /// No action defined for the given terminal in the parser table. + /// + /// A deterministic parser has one active branch. This error is produced before any reduce + /// action runs on that branch, so the parser context remains usable after a failed `feed` or + /// `accept` and the user data stays in the context. NoAction(NoActionError), /// Error from reduce action. - ReduceAction(ReduceActionError), + /// + /// A reduce action may have mutated semantic stacks or user data before returning `Err`. Since + /// the deterministic context has no sibling branch to fall back to, the whole context is + /// consumed and the user data is moved into this error. + ReduceAction(ReduceActionError), + + /// A feed or accept was attempted after the context had already been consumed. + /// + /// This happens after successful `accept`, or after a previous `ReduceAction` error moved the + /// user data out of the context. This error only reports the attempted terminal, location, and + /// current parser state. + ConsumedContext(ConsumedContextError), } -impl ParseError { +impl ParseError { pub fn location(&self) -> &Location { match self { ParseError::NoAction(err) => &err.location, ParseError::ReduceAction(err) => &err.location, + ParseError::ConsumedContext(err) => &err.location, } } @@ -39,6 +62,7 @@ impl ParseError { match self { ParseError::NoAction(err) => &err.term, ParseError::ReduceAction(err) => &err.term, + ParseError::ConsumedContext(err) => &err.term, } } @@ -46,11 +70,41 @@ impl ParseError { match self { ParseError::NoAction(err) => err.state, ParseError::ReduceAction(err) => err.state, + ParseError::ConsumedContext(err) => err.state, } } + + pub fn userdata(&self) -> Option<&UserData> { + match self { + ParseError::NoAction(_) => None, + ParseError::ReduceAction(err) => Some(&err.userdata), + ParseError::ConsumedContext(_) => None, + } + } + + pub fn into_userdata(self) -> Option { + match self { + ParseError::NoAction(_) => None, + ParseError::ReduceAction(err) => Some(err.userdata), + ParseError::ConsumedContext(_) => None, + } + } + + pub fn reduce_action_error(&self) -> Option<&ReduceAction> { + match self { + ParseError::NoAction(_) => None, + ParseError::ReduceAction(err) => Some(&err.source), + ParseError::ConsumedContext(_) => None, + } + } + + pub fn is_consumed_context(&self) -> bool { + matches!(self, ParseError::ConsumedContext(_)) + } } -impl Display for ParseError +impl Display + for ParseError where Term: Display, ReduceAction: Display, @@ -67,6 +121,9 @@ where err.term, err.state, err.source ) } + ParseError::ConsumedContext(err) => { + write!(f, "ConsumedContext: {}, State: {}", err.term, err.state) + } } } } diff --git a/rusty_lr_core/src/parser/nondeterministic/context.rs b/rusty_lr_core/src/parser/nondeterministic/context.rs index c9d89fe0..2f5d0b13 100644 --- a/rusty_lr_core/src/parser/nondeterministic/context.rs +++ b/rusty_lr_core/src/parser/nondeterministic/context.rs @@ -5,6 +5,7 @@ use super::Node; use super::NodeId; use super::ParseError; use super::error::FeedSuccess; +use super::error::ParseErrorBranch; use crate::Location; use crate::TerminalSymbol; @@ -107,6 +108,11 @@ enum FailedFeedPlan { FailedAndAlreadyRemoved, } +struct ReduceFailure { + state: usize, + source: Source, +} + #[derive(Clone, Copy)] struct GlrFeedPlan { shift: Option>, @@ -470,9 +476,14 @@ pub struct Context< /// For recovery from error fallback_branches: Vec>, - /// For temporary use. store reduce errors returned from `reduce_action`. - /// But we don't want to reallocate every `feed` call - pub(crate) reduce_errors: Vec, + /// Reusable branch-scoped errors returned while processing one lookahead. + /// + /// Keeping this buffer on the context avoids reallocating on every `feed` call. + pub(crate) branch_errors: Vec>, + + /// Whether this context has been consumed by successful accept or by a failure that moved or + /// mutated every active branch. + consumed: bool, /// Reusable container for CFG-only GLR feed plans. feed_plan_container: FeedPlanContainer, @@ -507,7 +518,8 @@ impl< empty_node_indices: Default::default(), current_branches: Default::default(), next_branches: Default::default(), - reduce_errors: Default::default(), + branch_errors: Default::default(), + consumed: false, fallback_branches: Default::default(), feed_plan_container: Default::default(), can_feed_plan_container: Default::default(), @@ -984,7 +996,7 @@ impl< shift: &mut bool, can_reuse_node_for_reduce: bool, userdata: &mut Data::UserData, - ) -> Result + ) -> Result> where Data: Clone, P::Term: Clone, @@ -1020,8 +1032,9 @@ impl< Ok(node_to_shift) } Err(err) => { + let state = self.state(node_to_shift); self.try_remove_node_recursive(node_to_shift); - Err(err) + Err(ReduceFailure { state, source: err }) } } } @@ -1036,12 +1049,34 @@ impl< self.current_branches.is_empty() } + fn is_consumed(&self) -> bool { + self.consumed + } + + fn context_consumed_error( + &self, + term: TerminalSymbol, + location: Data::Location, + ) -> ParseError { + ParseError { + term, + location, + branch_errors: Vec::new(), + context_consumed: true, + } + } + /// End this context and return the first successful start symbol and user data pair. + /// + /// `ParseError` from EOF leaves branches that only saw `NoAction` reusable. Successful + /// acceptance moves accepted branch values out and consumes the context. If every branch is + /// consumed by reduce-action failure or failed recovery, later feeds or accepts return a + /// consumed-context error. pub fn accept( - self, + &mut self, ) -> Result< (Start::StartType, Data::UserData), - ParseError, + ParseError, > where Data: Clone, @@ -1053,11 +1088,15 @@ impl< } /// End this context and return iterator of the start value and user data from each successful path. + /// + /// EOF is handled with the same branch lifecycle as a normal feed: branches that only see + /// `NoAction` remain reusable, branches that fail during reduce-action replay are consumed, and + /// successful acceptance moves all accepted branch values out and consumes the context. pub fn accept_all( - mut self, + &mut self, ) -> Result< impl Iterator, - ParseError, + ParseError, > where Data: Clone, @@ -1065,15 +1104,14 @@ impl< P::Term: Clone, P::NonTerm: std::fmt::Debug, { - self.feed_eof()?; + if let Err(err) = self.feed_eof() { + return Err(err); + } // since `eof` is feeded, every node graph should be like this: // Root <- Start <- EOF // ^^^ here, current_node - let Context { - mut nodes_pool, - current_branches, - .. - } = self; + let mut nodes_pool = std::mem::take(&mut self.nodes_pool); + let current_branches = std::mem::take(&mut self.current_branches); let mut accepted = Vec::with_capacity(current_branches.len()); for Branch { node, userdata } in current_branches { @@ -1094,6 +1132,12 @@ impl< accepted.push((start_value, userdata)); } + self.empty_node_indices.clear(); + self.next_branches.clear(); + self.fallback_branches.clear(); + self.branch_errors.clear(); + self.consumed = true; + Ok(accepted.into_iter()) } @@ -1258,8 +1302,8 @@ impl< &mut self, term: Data::Term, ) -> Result< - FeedSuccess, - ParseError, + FeedSuccess, + ParseError, > where P::Term: Clone, @@ -1398,7 +1442,11 @@ impl< if can_reuse_node_for_reduce { node_removed = true; } - self.reduce_errors.push(err); + self.branch_errors.push(ParseErrorBranch::reduce_action( + err.state, + err.source, + branch_userdata, + )); } } @@ -1459,6 +1507,39 @@ impl< self.try_remove_node_subtree_recursive(node); } + fn cloned_no_action_branch_errors_from<'a>( + &'a self, + branches: impl Iterator>, + ) -> Vec> + where + Data::UserData: Clone, + { + branches + .map(|branch| { + ParseErrorBranch::no_action(self.state(branch.node), branch.userdata.clone()) + }) + .collect() + } + + fn push_no_action_branch_error(&mut self, branch: Branch) { + let state = self.state(branch.node); + self.discard_branch_node(branch.node); + self.branch_errors + .push(ParseErrorBranch::no_action(state, branch.userdata)); + } + + fn push_fallback_no_action_branch_errors(&mut self) { + while let Some(branch) = self.fallback_branches.pop() { + self.push_no_action_branch_error(branch); + } + } + + fn take_branch_errors( + &mut self, + ) -> Vec> { + std::mem::take(&mut self.branch_errors) + } + fn discard_fallback_branches(&mut self) { while let Some(branch) = self.fallback_branches.pop() { self.discard_branch_node(branch.node); @@ -1570,13 +1651,24 @@ impl< /// eligible for panic-mode recovery. If at least one branch can grammatically shift the /// terminal, recovery is not entered; failed sibling branches are returned in /// `FeedSuccess::errors`. + /// + /// Branch lifecycle for one lookahead: + /// + /// - A branch that only reaches `NoAction` has not mutated its stack or user data. If no branch + /// succeeds, it is restored into `current_branches` and the context remains reusable. + /// - A branch that fails in a reduce action is consumed; its user data moves into + /// `ParseErrorBranch::ReduceAction`. + /// - If any branch succeeds, only successful branches remain active. Failed sibling branches + /// are reported in `FeedSuccess::errors`. + /// - If no branch succeeds and no reusable `NoAction` branch remains, the whole context is + /// consumed and later feeds or accepts return a consumed-context error. pub fn feed_location( &mut self, term: P::Term, location: Data::Location, ) -> Result< - FeedSuccess, - ParseError, + FeedSuccess, + ParseError, > where P::Term: Clone, @@ -1586,7 +1678,11 @@ impl< { use crate::Location; - self.reduce_errors.clear(); + if self.is_consumed() { + return Err(self.context_consumed_error(TerminalSymbol::Terminal(term), location)); + } + + self.branch_errors.clear(); self.fallback_branches.clear(); self.next_branches.clear(); @@ -1637,20 +1733,28 @@ impl< debug_assert!(self.next_branches.is_empty()); if !P::ERROR_USED || feedable_branch_found { - // Recovery is reserved for pure syntax failure. If an admitted branch died while - // replaying its plan, report that semantic/runtime failure instead. - + // Recovery is reserved for pure syntax failure. Branches that only saw `NoAction` + // have not mutated their stacks, so they are restored even when sibling branches + // were consumed by reduce-action failures. + let no_action_branch_errors = + self.cloned_no_action_branch_errors_from(self.fallback_branches.iter()); std::mem::swap(&mut self.current_branches, &mut self.fallback_branches); + self.branch_errors.extend(no_action_branch_errors); + if self.current_branches.is_empty() { + self.consumed = true; + } self.feed_plan_container = container; return Err(ParseError { term: TerminalSymbol::Terminal(term), location, - reduce_action_errors: std::mem::take(&mut self.reduce_errors), - states: self.states().collect(), + branch_errors: self.take_branch_errors(), + context_consumed: false, }); } + let no_action_branch_errors = + self.cloned_no_action_branch_errors_from(self.fallback_branches.iter()); let mut extra_state_stack = container.take_extra_state_stack(); while let Some(Branch { node, userdata }) = self.fallback_branches.pop() { // Recovery searches only concrete branch stacks; the implicit root state precedes @@ -1660,12 +1764,14 @@ impl< container.put_extra_state_stack(extra_state_stack); if self.next_branches.is_empty() { + self.branch_errors.extend(no_action_branch_errors); self.feed_plan_container = container; + self.consumed = true; Err(ParseError { term: TerminalSymbol::Terminal(term), location, - reduce_action_errors: std::mem::take(&mut self.reduce_errors), - states: self.states().collect(), + branch_errors: self.take_branch_errors(), + context_consumed: false, }) } else { debug_assert!(self.current_branches.is_empty()); @@ -1717,23 +1823,24 @@ impl< if self.next_branches.is_empty() { self.feed_plan_container = container; + self.consumed = true; Err(ParseError { term: TerminalSymbol::Terminal(term), location, - reduce_action_errors: std::mem::take(&mut self.reduce_errors), - states: self.states().collect(), + branch_errors: self.take_branch_errors(), + context_consumed: false, }) } else { // A recovered branch keeps the feed successful. Semantic failures from sibling // recovery branches are still surfaced, matching ordinary GLR branch pruning. - let errors = if self.reduce_errors.is_empty() { + let errors = if self.branch_errors.is_empty() { None } else { Some(ParseError { term: TerminalSymbol::Terminal(term), location: location.clone(), - reduce_action_errors: std::mem::take(&mut self.reduce_errors), - states: Vec::new(), + branch_errors: self.take_branch_errors(), + context_consumed: false, }) }; @@ -1748,20 +1855,16 @@ impl< debug_assert!(!self.next_branches.is_empty()); // Surviving original branches define feed success; sibling failures are diagnostics, // not a fatal parse result. - let errors = if self.fallback_branches.is_empty() && self.reduce_errors.is_empty() { + let errors = if self.fallback_branches.is_empty() && self.branch_errors.is_empty() { None } else { - let states = self - .fallback_branches - .iter() - .map(|branch| self.state(branch.node)) - .collect(); + self.push_fallback_no_action_branch_errors(); self.discard_fallback_branches(); Some(ParseError { term: TerminalSymbol::Terminal(term), location, - reduce_action_errors: std::mem::take(&mut self.reduce_errors), - states, + branch_errors: self.take_branch_errors(), + context_consumed: false, }) }; std::mem::swap(&mut self.current_branches, &mut self.next_branches); @@ -1774,6 +1877,9 @@ impl< /// So this function will return `false` even if term can be shifted as `error` token, /// and will return `true` if `Err` variant is returned by `reduce_action`. pub fn can_feed(&self, term: &P::Term) -> bool { + if self.is_consumed() { + return false; + } let class = P::TermClass::from_term(term); let mut container = self.can_feed_plan_container.borrow_mut(); let mut extra_state_stack = container.take_extra_state_stack(); @@ -1792,6 +1898,9 @@ impl< /// Check if current context can enter panic mode. pub fn can_panic(&self) -> bool { + if self.is_consumed() { + return false; + } // if `error` token was not used in the grammar, early return here if !P::ERROR_USED { return false; @@ -1840,9 +1949,14 @@ impl< } /// Feed eof symbol with default zero-length location from the end of stream. + /// + /// EOF uses the same branch lifecycle as `feed_location`: successful branches move to + /// `next_branches`; `NoAction` branches are restored when no branch accepts; reduce-action + /// failures consume only their branch. The context is consumed only when no reusable branch + /// remains, or when acceptance succeeds and accepted values are moved out. fn feed_eof( &mut self, - ) -> Result<(), ParseError> + ) -> Result<(), ParseError> where P::Term: Clone, P::NonTerm: std::fmt::Debug, @@ -1851,17 +1965,22 @@ impl< { use crate::location::Location; - self.reduce_errors.clear(); - self.fallback_branches.clear(); - self.next_branches.clear(); - let eof_location = if let Some(branch) = self.current_branches.first() { Data::Location::new(self.location_iter(branch.node), 0) } else { Data::Location::new(std::iter::empty(), 0) }; + if self.is_consumed() { + return Err(self.context_consumed_error(TerminalSymbol::Eof, eof_location)); + } + + self.branch_errors.clear(); + self.fallback_branches.clear(); + self.next_branches.clear(); + let mut container = std::mem::take(&mut self.feed_plan_container); + let mut feedable_branch_found = false; let mut extra_state_stack = container.take_extra_state_stack(); while let Some(Branch { node, userdata }) = self.current_branches.pop() { let node_eof_location = Data::Location::new(self.location_iter(node), 0); @@ -1872,6 +1991,7 @@ impl< if let Some(plan_id) = container.plan_feed(self, &mut extra_state_stack, node_range, P::TermClass::EOF) { + feedable_branch_found = true; match self.apply_feed_plan( &container, plan_id, @@ -1896,14 +2016,21 @@ impl< // check for panic mode // and restore nodes to original state from fallback_branches if self.next_branches.is_empty() { + let no_action_branch_errors = + self.cloned_no_action_branch_errors_from(self.fallback_branches.iter()); std::mem::swap(&mut self.current_branches, &mut self.fallback_branches); + self.branch_errors.extend(no_action_branch_errors); + if self.current_branches.is_empty() { + debug_assert!(feedable_branch_found); + self.consumed = true; + } self.feed_plan_container = container; Err(ParseError { term: TerminalSymbol::Eof, location: eof_location, - reduce_action_errors: std::mem::take(&mut self.reduce_errors), - states: self.states().collect(), + branch_errors: self.take_branch_errors(), + context_consumed: false, }) } else { self.discard_fallback_branches(); @@ -1914,6 +2041,9 @@ impl< } /// Check if current context can be terminated and get the start value. pub fn can_accept(&self) -> bool { + if self.is_consumed() { + return false; + } let mut container = self.can_feed_plan_container.borrow_mut(); let mut extra_state_stack = container.take_extra_state_stack(); let can_accept = self.current_branches.iter().any(|branch| { @@ -1962,7 +2092,8 @@ where empty_node_indices: self.empty_node_indices.clone(), current_branches: self.current_branches.clone(), next_branches: Default::default(), - reduce_errors: Default::default(), + branch_errors: Default::default(), + consumed: self.consumed, fallback_branches: Default::default(), feed_plan_container: Default::default(), can_feed_plan_container: Default::default(), diff --git a/rusty_lr_core/src/parser/nondeterministic/error.rs b/rusty_lr_core/src/parser/nondeterministic/error.rs index bd752b2a..8e3ef219 100644 --- a/rusty_lr_core/src/parser/nondeterministic/error.rs +++ b/rusty_lr_core/src/parser/nondeterministic/error.rs @@ -3,16 +3,48 @@ use std::fmt::Display; /// Error type for feed() #[derive(Clone, Debug)] -pub struct ParseError { +pub struct ParseError { /// The terminal symbol that caused the error. pub term: crate::TerminalSymbol, /// Location of the terminal symbol. pub location: Location, - /// Error from reduce action (from every diverged paths) - pub reduce_action_errors: Vec, + /// Errors grouped by the GLR branch that produced them. + pub branch_errors: Vec>, + /// Whether this error was produced by using a context after it had already been consumed. + pub context_consumed: bool, +} - /// States when the error occurred (from all diverged paths) - pub(crate) states: Vec, +/// Errors observed while processing one GLR branch for a single lookahead. +/// +/// A branch failure is either a grammatical `NoAction` at one parser state or one reduce-action +/// error raised at one parser state. Keeping that pair in a single value makes it explicit that +/// the state and semantic error came from the same nondeterministic branch. Reusing a context after +/// every branch has already been consumed is reported by `ParseError::context_consumed`, because no +/// reusable branch remains to attach branch-local state to. +#[derive(Clone, Debug)] +pub enum ParseErrorBranch { + /// No parser-table action was available for this branch. + /// + /// This branch did not run a reduce action for the lookahead. Its stack and user data remain + /// reusable when no branch succeeds and the context restores `NoAction` branches. + NoAction { + /// Parser state associated with this branch's failure. + state: usize, + /// User data owned by this branch when it failed. + userdata: UserData, + }, + /// A reduce action rejected this branch. + /// + /// The branch may have partially mutated semantic state before returning `Err`, so this branch + /// is consumed and its user data is moved into the error. + ReduceAction { + /// Parser state associated with this branch's failure. + state: usize, + /// Error returned from the reduce action while replaying this branch. + source: ReduceActionError, + /// User data owned by this branch when it failed. + userdata: UserData, + }, } /// Successful GLR feed result. @@ -21,32 +53,99 @@ pub struct ParseError { /// success, but `errors` records the branches that were pruned by `NoAction` or reduce-action /// failure. #[derive(Clone, Debug)] -pub struct FeedSuccess { +pub struct FeedSuccess { /// Branch errors observed while at least one GLR branch survived this feed. /// /// This is intentionally optional so callers that only care about fatal parse failure can keep /// using `context.feed(token)?` or `context.feed(token).unwrap()`. - pub errors: Option>, + pub errors: Option>, } -impl ParseError { +impl + ParseError +{ pub fn location(&self) -> &Location { &self.location } pub fn term(&self) -> &crate::TerminalSymbol { &self.term } - /// States when the error occurred (from all diverged paths) + /// States when the error occurred, flattened across all failed branches. + pub fn states(&self) -> impl Iterator + '_ { + self.branch_errors.iter().flat_map(|branch| branch.states()) + } + + pub fn is_consumed_context(&self) -> bool { + self.context_consumed + } +} + +impl ParseErrorBranch { + pub(crate) fn no_action(state: usize, userdata: UserData) -> Self { + ParseErrorBranch::NoAction { state, userdata } + } + + pub(crate) fn reduce_action( + state: usize, + source: ReduceActionError, + userdata: UserData, + ) -> Self { + ParseErrorBranch::ReduceAction { + state, + source, + userdata, + } + } + + /// State associated with this branch's failure. + pub fn state(&self) -> usize { + match self { + ParseErrorBranch::NoAction { state, .. } => *state, + ParseErrorBranch::ReduceAction { state, .. } => *state, + } + } + + /// State associated with this branch's failure, returned as an iterator for API symmetry with + /// `ParseError::states()`. pub fn states(&self) -> impl Iterator + '_ { - self.states.iter().copied() + std::iter::once(self.state()) + } + + /// Reduce-action error for this branch, if this branch failed semantically. + pub fn reduce_action_error(&self) -> Option<&ReduceActionError> { + match self { + ParseErrorBranch::NoAction { .. } => None, + ParseErrorBranch::ReduceAction { source, .. } => Some(source), + } + } + + /// User data owned by this branch when it failed. + pub fn userdata(&self) -> &UserData { + match self { + ParseErrorBranch::NoAction { userdata, .. } => userdata, + ParseErrorBranch::ReduceAction { userdata, .. } => userdata, + } + } + + /// Consume this branch error and return its user data. + pub fn into_userdata(self) -> UserData { + match self { + ParseErrorBranch::NoAction { userdata, .. } => userdata, + ParseErrorBranch::ReduceAction { userdata, .. } => userdata, + } } } -impl Display for ParseError +impl Display + for ParseError where Term: Display, { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(f, "ParseError: {}, States: {:?}", self.term, self.states) + if self.context_consumed { + return write!(f, "ConsumedContext: {}", self.term); + } + let states: Vec<_> = self.states().collect(); + write!(f, "ParseError: {}, States: {:?}", self.term, states) } } diff --git a/rusty_lr_core/src/parser/nondeterministic/mod.rs b/rusty_lr_core/src/parser/nondeterministic/mod.rs index 358cff50..58a4ac53 100644 --- a/rusty_lr_core/src/parser/nondeterministic/mod.rs +++ b/rusty_lr_core/src/parser/nondeterministic/mod.rs @@ -5,5 +5,6 @@ mod node; pub use context::Context; pub use error::FeedSuccess; pub use error::ParseError; +pub use error::ParseErrorBranch; pub use node::Node; pub(crate) use node::NodeId; diff --git a/rusty_lr_parser/src/emit.rs b/rusty_lr_parser/src/emit.rs index 399bd906..8a078414 100644 --- a/rusty_lr_parser/src/emit.rs +++ b/rusty_lr_parser/src/emit.rs @@ -23,6 +23,7 @@ impl Grammar { let termclass_typename = Ident::new("TerminalClasses", Span::call_site()); let location_typename = &self.location_typename; let reduce_error_typename = &self.error_typename; + let user_data_typename = &self.userdata_typename; let table_structname = if self.emit_dense { format_ident!("DenseFlatTables") @@ -88,7 +89,7 @@ impl Grammar { #[allow(non_camel_case_types,dead_code)] pub type #tables_typename = #module_prefix::parser::table::#table_structname<#termclass_typename, #nonterm_typename, #rule_container_type, #state_index_typename>; #[allow(non_camel_case_types,dead_code)] - pub type #parse_error_typename = #module_prefix::parser::nondeterministic::ParseError<#token_typename, #location_typename, #reduce_error_typename>; + pub type #parse_error_typename = #module_prefix::parser::nondeterministic::ParseError<#token_typename, #location_typename, #reduce_error_typename, #user_data_typename>; }); } else { stream.extend(quote! { @@ -97,7 +98,7 @@ impl Grammar { #[allow(non_camel_case_types,dead_code)] pub type #tables_typename = #module_prefix::parser::table::#table_structname<#termclass_typename, #nonterm_typename, #rule_container_type, #state_index_typename>; #[allow(non_camel_case_types,dead_code)] - pub type #parse_error_typename = #module_prefix::parser::deterministic::ParseError<#token_typename, #location_typename, #reduce_error_typename>; + pub type #parse_error_typename = #module_prefix::parser::deterministic::ParseError<#token_typename, #location_typename, #reduce_error_typename, #user_data_typename>; }); } } diff --git a/rusty_lr_parser/src/parser/parser_expanded.rs b/rusty_lr_parser/src/parser/parser_expanded.rs index 4eb2cc18..81d26f8b 100644 --- a/rusty_lr_parser/src/parser/parser_expanded.rs +++ b/rusty_lr_parser/src/parser/parser_expanded.rs @@ -204,6 +204,7 @@ pub type ParseError = ::rusty_lr_core::parser::deterministic::ParseError< Lexed, Location, ::rusty_lr_core::DefaultReduceActionError, + GrammarArgs, >; /// A enum that represents terminal classes #[allow(non_camel_case_types, dead_code)] diff --git a/scripts/diff/calculator.rs b/scripts/diff/calculator.rs index f2ffcebd..5d2f1ae8 100644 --- a/scripts/diff/calculator.rs +++ b/scripts/diff/calculator.rs @@ -48,6 +48,7 @@ pub type ParseError = ::rusty_lr::parser::deterministic::ParseError< Token, ::rusty_lr::DefaultLocation, String, + i32, >; /// A enum that represents terminal classes #[allow(non_camel_case_types, dead_code)] diff --git a/scripts/diff/calculator_u8.rs b/scripts/diff/calculator_u8.rs index 7e7c4de4..7419ddfc 100644 --- a/scripts/diff/calculator_u8.rs +++ b/scripts/diff/calculator_u8.rs @@ -48,6 +48,7 @@ pub type ParseError = ::rusty_lr::parser::deterministic::ParseError< char, ::rusty_lr::DefaultLocation, ::rusty_lr::DefaultReduceActionError, + i32, >; /// A enum that represents terminal classes #[allow(non_camel_case_types, dead_code)] diff --git a/scripts/diff/json.rs b/scripts/diff/json.rs index e7e9744b..d787d53a 100644 --- a/scripts/diff/json.rs +++ b/scripts/diff/json.rs @@ -131,6 +131,7 @@ pub type ParseError = ::rusty_lr::parser::deterministic::ParseError< char, std::ops::Range, ::rusty_lr::DefaultReduceActionError, + Vec>, >; /// A enum that represents terminal classes #[allow(non_camel_case_types, dead_code)]