C# Class Lucene.Net.Analysis.Standard.ClassicTokenizerImpl

This class implements the classic lucene StandardTokenizer up until 3.0

Inheritance: StandardTokenizerInterface

Datei anzeigen Open project: apache/lucenenet

Public Properties

Property	Type	Description
ACRONYM	int
ACRONYM_DEP	int
ALPHANUM	int
APOSTROPHE	int
CJ	int
COMPANY	int
EMAIL	int
HOST	int
NUM	int
TOKEN_TYPES	string[]
YYEOF	int

Public Methods

Method	Description
GetNextToken ( ) : int	Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
GetText ( ICharTermAttribute t ) : void	Fills CharTermAttribute with the current token text.
YyBegin ( int newState ) : void	Enters a new lexical state
YyCharAt ( int pos ) : char	Returns the character at position `pos` from the matched text. It is equivalent to YyText().charAt(pos), but faster
YyClose ( ) : void	Closes the input stream.
YyPushBack ( int number ) : void	Pushes the specified amount of characters back into the input stream. They will be read again by then next call of the scanning method
YyReset ( TextReader.TextReader reader ) : void	Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to `ZZ_INITIAL`. Internal scan buffer is resized down to its initial length, if it has grown.

Private Methods

Method	Description
ClassicTokenizerImpl ( TextReader.TextReader @in ) : System	Creates a new scanner
ZzRefill ( ) : bool	Refills the input buffer.
ZzScanError ( int errorCode ) : void	Reports an error that occured while scanning. In a wellformed scanner (no or only correct usage of YyPushBack(int) and a match-all fallback rule) this method will only be called with things that "Can't Possibly Happen". If this method is called, something is seriously wrong (e.g. a JFlex bug producing a faulty scanner etc.). Usual syntax/scanner level error handling should be done in error fallback rules.
ZzUnpackAction ( string packed, int offset, int result ) : int
ZzUnpackAction ( ) : int[]
ZzUnpackAttribute ( string packed, int offset, int result ) : int
ZzUnpackAttribute ( ) : int[]
ZzUnpackCMap ( string packed ) : char[]	Unpacks the compressed character translation table.
ZzUnpackRowMap ( string packed, int offset, int result ) : int
ZzUnpackRowMap ( ) : int[]
ZzUnpackTrans ( string packed, int offset, int result ) : int
ZzUnpackTrans ( ) : int[]

Method Details

GetNextToken() public method

Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.

if any I/O-Error occurs

public GetNextToken ( ) : int
return	int

GetText() public method

Fills CharTermAttribute with the current token text.

public GetText ( ICharTermAttribute t ) : void
t	ICharTermAttribute
return	void

YyBegin() public method

Enters a new lexical state

public YyBegin ( int newState ) : void
newState	int	the new lexical state
return	void

YyCharAt() public method

Returns the character at position pos from the matched text. It is equivalent to YyText().charAt(pos), but faster

public YyCharAt ( int pos ) : char
pos	int	the position of the character to fetch. /// A value from 0 to YyLength()-1. ///
return	char

YyClose() public method

Closes the input stream.

public YyClose ( ) : void
return	void

YyPushBack() public method

Pushes the specified amount of characters back into the input stream. They will be read again by then next call of the scanning method

public YyPushBack ( int number ) : void
number	int	the number of characters to be read again. /// This number must not be greater than YyLength()!
return	void

YyReset() public method

Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL. Internal scan buffer is resized down to its initial length, if it has grown.

public YyReset ( TextReader.TextReader reader ) : void
reader	System.IO.TextReader.TextReader	the new input stream
return	void

Property Details

ACRONYM public_oe static_oe property

public static int ACRONYM
return	int

ACRONYM_DEP public_oe static_oe property

public static int ACRONYM_DEP
return	int

ALPHANUM public_oe static_oe property

public static int ALPHANUM
return	int

APOSTROPHE public_oe static_oe property

public static int APOSTROPHE
return	int

CJ public_oe static_oe property

public static int CJ
return	int

COMPANY public_oe static_oe property

public static int COMPANY
return	int

EMAIL public_oe static_oe property

public static int EMAIL
return	int

HOST public_oe static_oe property

public static int HOST
return	int

NUM public_oe static_oe property

public static int NUM
return	int

TOKEN_TYPES public_oe static_oe property

public static string[] TOKEN_TYPES
return	string[]

YYEOF public_oe static_oe property

This character denotes the end of file

public static int YYEOF
return	int