C# Class Lucene.Net.Analysis.Tokenizer

A Tokenizer is a TokenStream whose input is a Reader.

this is an abstract class; subclasses must override #IncrementToken()

NOTE: Subclasses overriding #IncrementToken() must call AttributeSource#ClearAttributes() before setting attributes.

Inheritance: TokenStream

Protected Properties

Property	Type	Description
input	System.IO.TextReader

Method	Description
Dispose ( ) : void	{@inheritDoc} NOTE: The default implementation closes the input Reader, so be sure to call `super.close()` when overriding this method.
Reset ( ) : void

Method	Description
CorrectOffset ( int currentOff ) : int	Return the corrected offset. If #input is a CharFilter subclass this method calls CharFilter#correctOffset, else returns `currentOff`.
Tokenizer ( AttributeFactory factory, TextReader input ) : System	Construct a token stream processing the given input using the given AttributeFactory.
Tokenizer ( TextReader input ) : System	Construct a token stream processing the given input.

Method	Description
SetReaderTestPoint ( ) : bool

Return the corrected offset. If #input is a CharFilter subclass this method calls CharFilter#correctOffset, else returns currentOff.

protected CorrectOffset ( int currentOff ) : int
currentOff	int	offset as seen in the output
return	int

{@inheritDoc}

NOTE: The default implementation closes the input Reader, so be sure to call super.close() when overriding this method.

public Dispose ( ) : void
return	void

public Reset ( ) : void
return	void

Construct a token stream processing the given input using the given AttributeFactory.

protected Tokenizer ( AttributeFactory factory, TextReader input ) : System
factory	AttributeFactory
input	System.IO.TextReader
return	System

Construct a token stream processing the given input.

protected Tokenizer ( TextReader input ) : System
input	System.IO.TextReader
return	System

The text source for this Tokenizer.

protected TextReader,System.IO input
return	System.IO.TextReader