|
|||||||||||
| PREV PACKAGE NEXT PACKAGE | FRAMES NO FRAMES | ||||||||||
| Class Summary | |
| DefaultTokeniser | A composed tokeniser containing a SimpleTokeniser and a
Transducer.
|
| DFSMState | Implements a state of the deterministic finite state machine of the tokeniser. |
| FSMState | A state of the finite state machine that is the kernel tokeniser |
| SimpleTokeniser | Implementation of a Unicode rule based tokeniser. |
| TestTokeniser | |
| UnicodeType | Used as an object wrapper that holds an Unicode type (the byte value of the static member of java.lang.Character). |
| Exception Summary | |
| InvalidRuleException | Fired when an invalid tokeniser rule is found |
| TokeniserException | The top level exception for all the exceptions fired by the tokeniser |
|
|||||||||||
| PREV PACKAGE NEXT PACKAGE | FRAMES NO FRAMES | ||||||||||