C# (CSharp) Lucene.Net.Analysis Namespace

This class runs all tests twice, one time with {@link TokenStream#setOnlyUseNewAPI} false and after that one time with true.

this class is abstract: at a minimum you must implement #read(char[], int, int), transforming the input in some way from #input, and #correct(int) to adjust the offsets to match the originals.

You can optionally provide more efficient implementations of additional methods like #read(), #read(char[]), #read(java.nio.CharBuffer), but this is not required.

For examples and integration with Analyzer, see the Lucene.Net.Analysis Analysis package documentation.

this analyzer is a replacement for Whitespace/Simple/KeywordAnalyzers for unit tests. If you are testing a custom component such as a queryparser or analyzer-wrapper that consumes analysis streams, its a great idea to test it with this analyzer instead. MockAnalyzer has the following behavior:

• By default, the assertions in MockTokenizer are turned on for extra checks that the consumer is consuming properly. These checks can be disabled with #setEnableChecks(boolean).
• Payload data is randomly injected into the stream for more thorough testing of payloads.

this tokenizer is a replacement for #WHITESPACE, #SIMPLE, and #KEYWORD tokenizers. If you are writing a component such as a TokenFilter, its a great idea to test it wrapping this tokenizer instead for extra checks. this tokenizer has the following behavior:

• An internal state-machine is used for checking consumer consistency. These checks can be disabled with #setEnableChecks(boolean).
• For convenience, optionally lowercases terms that it outputs.

this is an abstract class; concrete subclasses are:

• Tokenizer, a TokenStream whose input is a Reader; and
• TokenFilter, a TokenStream whose input is another TokenStream.

TokenStream now extends AttributeSource, which provides access to all of the token Attributes for the TokenStream. Note that only one instance per AttributeImpl is created and reused for every token. this approach reduces object creation and allows local caching of references to the AttributeImpls. See #IncrementToken() for further details.

The workflow of the new TokenStream API is as follows:

1. Instantiation of TokenStream/TokenFilters which add/get attributes to/from the AttributeSource.
2. The consumer calls TokenStream#reset().
3. The consumer retrieves attributes from the stream and stores local references to all attributes it wants to access.
4. The consumer calls #IncrementToken() until it returns false consuming the attributes after each call.
5. The consumer calls #end() so that any end-of-stream operations can be performed.
6. The consumer calls #close() to release any resource when finished using the TokenStream.

You can find some example code for the new API in the analysis package level Javadoc.

Sometimes it is desirable to capture a current state of a TokenStream, e.g., for buffering purposes (see CachingTokenFilter, TeeSinkTokenFilter). For this usecase AttributeSource#captureState and AttributeSource#restoreState can be used.

The {@code TokenStream}-API in Lucene is based on the decorator pattern. Therefore all non-abstract subclasses must be final or have at least a final implementation of #incrementToken! this is checked when Java assertions are enabled.

this is an abstract class; subclasses must override #IncrementToken()

NOTE: Subclasses overriding #IncrementToken() must call AttributeSource#ClearAttributes() before setting attributes.