C# Class Lucene.Net.Analysis.Core.LowerCaseTokenizer

LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.

Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

You must specify the required Version compatibility when creating LowerCaseTokenizer:

As of 3.1, CharTokenizer uses an int based API to normalize and detect token characters. See CharTokenizer#isTokenChar(int) and CharTokenizer#normalize(int) for details.

Inheritance: LetterTokenizer

Show file Open project: paulirwin/lucene.net Class Usage Examples

Public Methods

Method	Description
LowerCaseTokenizer ( Lucene.Net.Util.Version matchVersion, AttributeFactory factory, TextReader @in ) : System.IO	Construct a new LowerCaseTokenizer using a given org.apache.lucene.util.AttributeSource.AttributeFactory.
LowerCaseTokenizer ( Lucene.Net.Util.Version matchVersion, TextReader @in ) : System.IO	Construct a new LowerCaseTokenizer.

Protected Methods

Method	Description
Normalize ( int c ) : int	Converts char to lower case Character#toLowerCase(int).

Method Details

LowerCaseTokenizer() public method

Construct a new LowerCaseTokenizer using a given org.apache.lucene.util.AttributeSource.AttributeFactory.

public LowerCaseTokenizer ( Lucene.Net.Util.Version matchVersion, AttributeFactory factory, TextReader @in ) : System.IO
matchVersion	Lucene.Net.Util.Version	/// Lucene version to match See above"/>
factory	AttributeFactory	/// the attribute factory to use for this
@in	System.IO.TextReader
return	System.IO

LowerCaseTokenizer() public method

Construct a new LowerCaseTokenizer.

public LowerCaseTokenizer ( Lucene.Net.Util.Version matchVersion, TextReader @in ) : System.IO
matchVersion	Lucene.Net.Util.Version	/// Lucene version to match See above"/> ///
@in	System.IO.TextReader
return	System.IO

Normalize() protected method

Converts char to lower case Character#toLowerCase(int).

protected Normalize ( int c ) : int
c	int
return	int