Case-insensitive lexing with Alex

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Case-insensitive lexing with Alex

Joel Reymont
Any suggestions on how to implement case-insensitive lexing with Alex?

        Thanks, Joel

--
http://wagerlabs.com





_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Case-insensitive lexing with Alex

Joel Reymont

On Jan 28, 2008, at 2:44 PM, Geoffrey Mainland wrote:

> map toLower onto your input before you pass it to your lexer? Or do  
> you
> only want keywords to be case-insensitive?


Just keywords. You can have "Array" or "array" or "aRrAy".

--
http://wagerlabs.com





_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Case-insensitive lexing with Alex

Geoffrey Mainland-2
Joel Reymont wrote:
>
> On Jan 28, 2008, at 2:44 PM, Geoffrey Mainland wrote:
>
>> map toLower onto your input before you pass it to your lexer? Or do you
>> only want keywords to be case-insensitive?
>
>
> Just keywords. You can have "Array" or "array" or "aRrAy".

One old trick for reducing the size of lexers is to not put keyword
recognition into the lexer rules (that is, don't write rules to match
keywords explicitly), but to look up identifiers in a map from strings
to keyword tokens. In your case, apply toLower to the identifier before
you look it up in the map from keywords to tokens.

The C parsing library I gave you a link to off-list does its lexing
using this technique if you want a concrete example.

Geoff
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe