Fixing The E1001 Illegal Character Error In Lexer

The Mystery of the Undefined Error: E1001

Have you ever encountered a bug that feels like a ghost in the machine? You see a definition, a clear intention, but it’s never actually called upon. That’s precisely the situation with the E1001 'illegal-character' error code in our system. It’s defined in pkg/errors/codes.go, sitting there, ready to be used, but the lexer – the component responsible for scanning the raw input and breaking it down into meaningful tokens – never actually triggers it. Instead, when the lexer finds a character it doesn't recognize, it merrily creates an ILLEGAL token. This ILLEGAL token then gets passed along, only to be caught later by the parser, which, in its turn, throws a more general E2001 error, typically indicating an 'unexpected token'. This means that even when we have a genuinely illegal character, the error message we receive is less specific than it could be, leading to potential confusion and a less-than-ideal debugging experience for our users. The core of the issue lies in pkg/lexer/lexer.go at line 259, where the ILLEGAL token is generated without the crucial step of associating it with the specific E1001 error code. This oversight means that a perfectly good error code, designed to give precise feedback about invalid characters, remains dormant, unused, and untested.

The Current Predicament: An Example in Action

Let's walk through a scenario to really understand the current behavior. Imagine a simple piece of code in our EZ language, something like this:

do main() {
    temp x int = 5 ` 3  // backtick is not a valid character
}

Here, the backtick character () is an invalid character within the context of this EZ code. We would expect our system to identify this as a clear violation – an *illegal character*. However, due to the current implementation, here’s what actually happens. The lexer encounters the backtick, recognizes it as something it doesn’t know how to handle, and generates an ILLEGALtoken. This token, carrying the 'illegality' but not the specific error code, is then passed to the parser. The parser, seeing this unexpected token, flags it with the genericE2001` error. The output you’d see in your terminal would be:

error[E2001]: unexpected token '`'

While this error does tell you something is wrong, it doesn’t pinpoint the exact nature of the problem. It doesn't explicitly state that the character itself is the issue, just that it’s unexpected in that position. This is a critical distinction. An 'unexpected token' could arise from a variety of syntax errors, but an 'illegal character' is a more direct and specific diagnosis. This lack of specificity can make debugging more challenging, especially for newcomers to the language or for complex codebases where distinguishing between different types of syntax errors is crucial for quick resolution. The path from lexer to parser, in this case, bypasses the opportunity for more granular error reporting, leaving a gap in our diagnostic capabilities.

The Desired Outcome: Precise Error Reporting

Our goal is to achieve a much more precise and informative error reporting mechanism. When the lexer encounters a character that is fundamentally not allowed in the source code – like the backtick in our previous example – it should immediately emit the E1001 error code. This isn't just about having a code; it's about leveraging that code to provide meaningful feedback to the developer. The expected behavior, therefore, is for the system to recognize the backtick as an illegal character and report it as such. Ideally, the error message would look something like this:

error[E1001]: illegal character '`'
  --> file.ez:2:20
   |
 2 |     temp x int = 5 ` 3
   |                    ^ illegal character in source

Notice the key differences here. The error code E1001 is clearly stated, immediately telling us the type of problem. The message explicitly says