Identifiers

This definition reflects the assumption discussed earlier that the input text would be ASCII rather than Unicode. A change to Unicode requires modification of the sets given here.

Identifier[11]:

Identifier:     $[A-Za-z$_][A-Za-z0-9$_]*       [mkidn]

This macro is invoked in definition 6.

The token processor mkidn encodes the string matched by the regular expression with a unique integer value. Every occurrence of a given string will be encoded with the same integer. A pointer to the characters of the string can be obtained by applying StringTable to the integer encoding.