C# Class org.apache.lucene.analysis.standard.ClassicTokenizer

A grammar-based tokenizer constructed with JFlex

This should be a good tokenizer for most European-language documents:

  • Splits words at punctuation characters, removing punctuation. However, a dot that's not followed by whitespace is considered part of a token.
  • Splits words at hyphens, unless there's a number in the token, in which case the whole token is interpreted as a product number and is not split.
  • Recognizes email addresses and internet hostnames as one token.

Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer. ClassicTokenizer was named StandardTokenizer in Lucene versions prior to 3.1. As of 3.1, StandardTokenizer implements Unicode text segmentation, as specified by UAX#29.

Inheritance: Tokenizer
Afficher le fichier Open project: paulirwin/lucene.net Class Usage Examples

Méthodes publiques

Свойство Type Description
TOKEN_TYPES string[]

Méthodes publiques

Méthode Description
ClassicTokenizer ( Version matchVersion, AttributeFactory factory, Reader input ) : Lucene.Net.Analysis.Standard

Creates a new ClassicTokenizer with a given org.apache.lucene.util.AttributeSource.AttributeFactory
ClassicTokenizer ( Version matchVersion, Reader input ) : Lucene.Net.Analysis.Standard

Creates a new instance of the ClassicTokenizer. Attaches the input to the newly created JFlex scanner.
close ( ) : void
end ( ) : void
incrementToken ( ) : bool
reset ( ) : void

Private Methods

Méthode Description
init ( Version matchVersion ) : void

Method Details

ClassicTokenizer() public méthode

Creates a new ClassicTokenizer with a given org.apache.lucene.util.AttributeSource.AttributeFactory
public ClassicTokenizer ( Version matchVersion, AttributeFactory factory, Reader input ) : Lucene.Net.Analysis.Standard
matchVersion Version
factory AttributeFactory
input Reader
Résultat Lucene.Net.Analysis.Standard

ClassicTokenizer() public méthode

Creates a new instance of the ClassicTokenizer. Attaches the input to the newly created JFlex scanner.
public ClassicTokenizer ( Version matchVersion, Reader input ) : Lucene.Net.Analysis.Standard
matchVersion Version
input Reader The input reader /// /// See http://issues.apache.org/jira/browse/LUCENE-1068
Résultat Lucene.Net.Analysis.Standard

close() public méthode

public close ( ) : void
Résultat void

end() public méthode

public end ( ) : void
Résultat void

incrementToken() public méthode

public incrementToken ( ) : bool
Résultat bool

reset() public méthode

public reset ( ) : void
Résultat void

Property Details

TOKEN_TYPES public_oe static_oe property

String token types that correspond to token type int constants
public static string[] TOKEN_TYPES
Résultat string[]