C# Класс Lucene.Net.Analysis.Pattern.PatternTokenizer

This tokenizer uses regex pattern matching to construct distinct tokens for the input stream. It takes two arguments: "pattern" and "group".

"pattern" is the regular expression.
"group" says which group to extract into tokens.

group=-1 (the default) is equivalent to "split". In this case, the tokens will be equivalent to the output from (without empty tokens): String#split(java.lang.String)

Using group >= 0 selects the matching group as the token. For example, if you have:

 pattern = \'([^\']+)\' group = 0 input = aaa 'bbb' 'ccc'

the output will be two tokens: 'bbb' and 'ccc' (including the ' marks). With the same input but using group=1, the output would be: bbb and ccc (no ' marks)

NOTE: This Tokenizer does not output tokens that are of zero length.

Наследование: Tokenizer

Показать файл Открыть проект Примеры использования класса

Открытые методы

Метод	Описание
End ( ) : void
IncrementToken ( ) : bool
PatternTokenizer ( AttributeFactory factory, TextReader input, Regex pattern, int group ) : Lucene.Net.Analysis.Tokenattributes	creates a new PatternTokenizer returning tokens from group (-1 for split functionality)
PatternTokenizer ( TextReader input, Regex pattern, int group ) : Lucene.Net.Analysis.Tokenattributes	creates a new PatternTokenizer returning tokens from group (-1 for split functionality)
Reset ( ) : void

Приватные методы

Метод	Описание
FillBuffer ( StringBuilder sb, TextReader input ) : void

Описание методов

End() публичный Метод

public End ( ) : void
Результат	void

IncrementToken() публичный Метод

public IncrementToken ( ) : bool
Результат	bool

PatternTokenizer() публичный Метод

creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

public PatternTokenizer ( AttributeFactory factory, TextReader input, Regex pattern, int group ) : Lucene.Net.Analysis.Tokenattributes
factory	AttributeFactory
input	TextReader
pattern	System.Text.RegularExpressions.Regex
group	int
Результат	Lucene.Net.Analysis.Tokenattributes

PatternTokenizer() публичный Метод

creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

public PatternTokenizer ( TextReader input, Regex pattern, int group ) : Lucene.Net.Analysis.Tokenattributes
input	TextReader
pattern	System.Text.RegularExpressions.Regex
group	int
Результат	Lucene.Net.Analysis.Tokenattributes

Reset() публичный Метод

public Reset ( ) : void
Результат	void