C# Class Lucene.Net.Analysis.Standard.StandardTokenizerImpl

This class is a scanner generated by JFlex 1.4.1 on 12/18/07 9:22 PM from the specification file /Volumes/User/grantingersoll/projects/lucene/java/lucene-clean/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex
Mostrar archivo Open project: apache/lucenenet Class Usage Examples

Public Properties

Property Type Description
HANGUL_TYPE int
HIRAGANA_TYPE int
IDEOGRAPHIC_TYPE int
KATAKANA_TYPE int
NUMERIC_TYPE int
SOUTH_EAST_ASIAN_TYPE int
WORD_TYPE int
YYEOF int

Public Methods

Method Description
GetNextToken ( ) : int

Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.

GetText ( ICharTermAttribute t ) : void

Fills CharTermAttribute with the current token text.

StandardTokenizerImpl ( TextReader @in ) : Lucene.Net.Analysis.Tokenattributes

Creates a new scanner

YyBegin ( int newState ) : void

Enters a new lexical state

YyCharAt ( int pos ) : char

Returns the character at position pos from the matched text. It is equivalent to YyText().charAt(pos), but faster

YyClose ( ) : void

Closes the input stream.

YyPushBack ( int number ) : void

Pushes the specified amount of characters back into the input stream. They will be read again by then next call of the scanning method

YyReset ( TextReader reader ) : void

Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL. Internal scan buffer is resized down to its initial length, if it has grown.

Private Methods

Method Description
ZzRefill ( ) : bool

Refills the input buffer.

ZzScanError ( int errorCode ) : void

Reports an error that occured while scanning. In a wellformed scanner (no or only correct usage of YyPushBack(int) and a match-all fallback rule) this method will only be called with things that "Can't Possibly Happen". If this method is called, something is seriously wrong (e.g. a JFlex bug producing a faulty scanner etc.). Usual syntax/scanner level error handling should be done in error fallback rules.

ZzUnpackAction ( string packed, int offset, int result ) : int
ZzUnpackAction ( ) : int[]
ZzUnpackAttribute ( string packed, int offset, int result ) : int
ZzUnpackAttribute ( ) : int[]
ZzUnpackCMap ( string packed ) : char[]

Unpacks the compressed character translation table.

ZzUnpackRowMap ( string packed, int offset, int result ) : int
ZzUnpackRowMap ( ) : int[]
ZzUnpackTrans ( string packed, int offset, int result ) : int
ZzUnpackTrans ( ) : int[]

Method Details

GetNextToken() public method

Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
if any I/O-Error occurs
public GetNextToken ( ) : int
return int

GetText() public method

Fills CharTermAttribute with the current token text.
public GetText ( ICharTermAttribute t ) : void
t ICharTermAttribute
return void

StandardTokenizerImpl() public method

Creates a new scanner
public StandardTokenizerImpl ( TextReader @in ) : Lucene.Net.Analysis.Tokenattributes
@in TextReader
return Lucene.Net.Analysis.Tokenattributes

YyBegin() public method

Enters a new lexical state
public YyBegin ( int newState ) : void
newState int the new lexical state
return void

YyCharAt() public method

Returns the character at position pos from the matched text. It is equivalent to YyText().charAt(pos), but faster
public YyCharAt ( int pos ) : char
pos int the position of the character to fetch. /// A value from 0 to YyLength()-1. ///
return char

YyClose() public method

Closes the input stream.
public YyClose ( ) : void
return void

YyPushBack() public method

Pushes the specified amount of characters back into the input stream. They will be read again by then next call of the scanning method
public YyPushBack ( int number ) : void
number int the number of characters to be read again. /// This number must not be greater than YyLength()!
return void

YyReset() public method

Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL. Internal scan buffer is resized down to its initial length, if it has grown.
public YyReset ( TextReader reader ) : void
reader TextReader the new input stream
return void

Property Details

HANGUL_TYPE public_oe static_oe property

public static int HANGUL_TYPE
return int

HIRAGANA_TYPE public_oe static_oe property

public static int HIRAGANA_TYPE
return int

IDEOGRAPHIC_TYPE public_oe static_oe property

public static int IDEOGRAPHIC_TYPE
return int

KATAKANA_TYPE public_oe static_oe property

public static int KATAKANA_TYPE
return int

NUMERIC_TYPE public_oe static_oe property

Numbers
public static int NUMERIC_TYPE
return int

SOUTH_EAST_ASIAN_TYPE public_oe static_oe property

Chars in class \p{Line_Break = Complex_Context} are from South East Asian scripts (Thai, Lao, Myanmar, Khmer, etc.). Sequences of these are kept together as as a single token rather than broken up, because the logic required to break them at word boundaries is too complex for UAX#29.

See Unicode Line Breaking Algorithm: http://www.unicode.org/reports/tr14/#SA

public static int SOUTH_EAST_ASIAN_TYPE
return int

WORD_TYPE public_oe static_oe property

Alphanumeric sequences
public static int WORD_TYPE
return int

YYEOF public_oe static_oe property

This character denotes the end of file
public static int YYEOF
return int