C# Class Lucene.Net.Analysis.CJK.CJKTokenizer

CJKTokenizer was modified from StopTokenizer which does a decent job for most European languages. and it perferm other token method for double-byte chars: the token will return at each two charactors with overlap match.
Example: "java C1C2C3C4" will be segment to: "java" "C1C2" "C2C3" "C3C4" it also need filter filter zero length token ""
for Digit: digit, '+', '#' will token as letter
for more info on Asia language(Chinese Japanese Korean) text segmentation: please search google

@author Che, Dong @version $Id: CJKTokenizer.java,v 1.3 2003/01/22 20:54:47 otis Exp $
Inheritance: Lucene.Net.Analysis.Tokenizer
Afficher le fichier Open project: synhershko/lucene.net

Méthodes publiques

Méthode Description
CJKTokenizer ( AttributeFactory factory, TextReader _in ) : System
CJKTokenizer ( Lucene.Net.Util.AttributeSource source, TextReader _in ) : System
CJKTokenizer ( TextReader _in ) : System

Construct a token stream processing the given input.
End ( ) : void
IncrementToken ( ) : bool
Reset ( ) : void
Reset ( TextReader reader ) : void

Private Methods

Méthode Description
Init ( ) : void

Method Details

CJKTokenizer() public méthode

public CJKTokenizer ( AttributeFactory factory, TextReader _in ) : System
factory AttributeFactory
_in TextReader
Résultat System

CJKTokenizer() public méthode

public CJKTokenizer ( Lucene.Net.Util.AttributeSource source, TextReader _in ) : System
source Lucene.Net.Util.AttributeSource
_in TextReader
Résultat System

CJKTokenizer() public méthode

Construct a token stream processing the given input.
public CJKTokenizer ( TextReader _in ) : System
_in TextReader I/O reader
Résultat System

End() public méthode

public End ( ) : void
Résultat void

IncrementToken() public méthode

public IncrementToken ( ) : bool
Résultat bool

Reset() public méthode

public Reset ( ) : void
Résultat void

Reset() public méthode

public Reset ( TextReader reader ) : void
reader TextReader
Résultat void