C# Класс Lucene.Net.Analysis.Standard.StandardTokenizerImpl

Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.

Fills CharTermAttribute with the current token text.

Creates a new scanner

Enters a new lexical state

Returns the character at position pos from the matched text. It is equivalent to YyText().charAt(pos), but faster

Closes the input stream.

Pushes the specified amount of characters back into the input stream. They will be read again by then next call of the scanning method

Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL. Internal scan buffer is resized down to its initial length, if it has grown.

Refills the input buffer.

Reports an error that occured while scanning. In a wellformed scanner (no or only correct usage of YyPushBack(int) and a match-all fallback rule) this method will only be called with things that "Can't Possibly Happen". If this method is called, something is seriously wrong (e.g. a JFlex bug producing a faulty scanner etc.). Usual syntax/scanner level error handling should be done in error fallback rules.

Unpacks the compressed character translation table.