Add `--debug-tokens` Option to Print Token List

[LunaStev]

Background:

This feature aims to add a debug option to the Wave CLI that prints the list of tokens generated by the lexer before parsing. It’s designed to help new contributors understand how the lexer works and get familiar with the CLI argument parsing structure. This is a perfect starting point for first-time contributors who want to explore the internals of lexer.rs.

Expected Behavior:

When running a Wave program with the --debug-tokens flag, e.g.,

wavec run file.wave --debug-tokens

the CLI should output the list of tokens generated from the input file. The normal execution flow should continue afterward. The token list should be printed in a readable format, one per line.

User Scenarios:

1. A developer wants to see how the lexer tokenizes their Wave code.
2. A new contributor wants to understand the lexer structure by implementing a small but useful feature.
3. A contributor wants to learn how CLI arguments are parsed in Wave.

Additional Information:

• Difficulty: Easy

• Relevant files: lexer.rs, CLI parsing module

• Example output:

Token { token_type: Fun, lexeme: "fun", line: 1 }
Token { token_type: Identifier("main"), lexeme: "main", line: 1 }
Token { token_type: Lparen, lexeme: "(", line: 1 }
Token { token_type: Rparen, lexeme: ")", line: 1 }
Token { token_type: Lbrace, lexeme: "{", line: 1 }
Token { token_type: Println, lexeme: "println", line: 2 }
Token { token_type: Lparen, lexeme: "(", line: 2 }
Token { token_type: String("Hello World"), lexeme: "\"Hello World\"", line: 2 }
Token { token_type: Rparen, lexeme: ")", line: 2 }
Token { token_type: SemiColon, lexeme: ";", line: 2 }
Token { token_type: Rbrace, lexeme: "}", line: 3 }
Token { token_type: Eof, lexeme: "", line: 3 }

• This flag is purely for debugging and has no effect on actual program behavior.

• Goal: Lower the entry barrier and improve understanding of Wave's internal structure.