C# Class Lucene.Net.Analysis.CJK.CJKTokenizer

CJKTokenizer was modified from StopTokenizer which does a decent job for most European languages. and it perferm other token method for double-byte chars: the token will return at each two charactors with overlap match.
Example: "java C1C2C3C4" will be segment to: "java" "C1C2" "C2C3" "C3C4" it also need filter filter zero length token ""
for Digit: digit, '+', '#' will token as letter
for more info on Asia language(Chinese Japanese Korean) text segmentation: please search google

@author Che, Dong @version $Id: CJKTokenizer.java,v 1.3 2003/01/22 20:54:47 otis Exp $
Inheritance: Lucene.Net.Analysis.Tokenizer
显示文件 Open project: synhershko/lucene.net

Public Methods

Method Description
CJKTokenizer ( AttributeFactory factory, TextReader _in ) : System
CJKTokenizer ( Lucene.Net.Util.AttributeSource source, TextReader _in ) : System
CJKTokenizer ( TextReader _in ) : System

Construct a token stream processing the given input.
End ( ) : void
IncrementToken ( ) : bool
Reset ( ) : void
Reset ( TextReader reader ) : void

Private Methods

Method Description
Init ( ) : void

Method Details

CJKTokenizer() public method

public CJKTokenizer ( AttributeFactory factory, TextReader _in ) : System
factory AttributeFactory
_in TextReader
return System

CJKTokenizer() public method

public CJKTokenizer ( Lucene.Net.Util.AttributeSource source, TextReader _in ) : System
source Lucene.Net.Util.AttributeSource
_in TextReader
return System

CJKTokenizer() public method

Construct a token stream processing the given input.
public CJKTokenizer ( TextReader _in ) : System
_in TextReader I/O reader
return System

End() public method

public End ( ) : void
return void

IncrementToken() public method

public IncrementToken ( ) : bool
return bool

Reset() public method

public Reset ( ) : void
return void

Reset() public method

public Reset ( TextReader reader ) : void
reader TextReader
return void