Method | Description | |
---|---|---|
CharTokenizer ( Lucene.Net.Util.Version matchVersion, AttributeFactory factory, |
Creates a new CharTokenizer instance
|
|
End ( ) : void | ||
IncrementToken ( ) : bool | ||
Reset ( ) : void |
Method | Description | |
---|---|---|
CharTokenizer ( Lucene.Net.Util.Version matchVersion, |
Creates a new CharTokenizer instance
|
|
IsTokenChar ( char c ) : bool |
Returns true iff a codepoint should be included in a token. This tokenizer generates as tokens adjacent sequences of codepoints which satisfy this predicate. Codepoints for which this is false are used to define token boundaries and are not included in tokens.
|
|
Normalize ( int c ) : int |
Called on each token character to normalize it before it is added to the token. The default implementation does nothing. Subclasses may use this to, e.g., lowercase tokens.
|
public CharTokenizer ( Lucene.Net.Util.Version matchVersion, AttributeFactory factory, |
||
matchVersion | Lucene.Net.Util.Version | /// Lucene version to match |
factory | AttributeFactory |
/// the attribute factory to use for this |
input | /// the input to split up into tokens | |
return | System.Diagnostics |
protected CharTokenizer ( Lucene.Net.Util.Version matchVersion, |
||
matchVersion | Lucene.Net.Util.Version | /// Lucene version to match |
input | /// the input to split up into tokens | |
return | System.Diagnostics |
protected abstract IsTokenChar ( char c ) : bool | ||
c | char | |
return | bool |