This definition reflects the assumption discussed earlier that the input text would be ASCII rather than Unicode. A change to Unicode requires modification of the sets given here.
The token processor mkidn encodes the string matched by the regular expression with a unique integer value. Every occurrence of a given string will be encoded with the same integer. A pointer to the characters of the string can be obtained by applying StringTable to the integer encoding.