C# (CSharp) Lucene.Net.Index Namespace

For efficiency, in this API documents are often referred to via document numbers, non-negative integers which each name a unique document in the index. These document numbers are ephemeral -- they may change as documents are added to and deleted from an index. Clients should thus not rely on a given document having the same number between sessions.

NOTE: {@link IndexReader} instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently. If your application requires external synchronization, you should not synchronize on the IndexReader instance; use your own (non-Lucene) objects instead.

As this tool checks every byte in the index, on a large index it can take quite a long time to run.

WARNING: this tool and API is new and experimental and is subject to suddenly change in the next release. Please make a complete backup of your index before using this to fix your index!

WARNING: this API is new and experimental and is subject to suddenly change in the next release.

WARNING: this API is new and experimental and is subject to suddenly change in the next release.

Since each DeleteSlice maintains its own head and the list is only single linked the garbage collector takes care of pruning the list for us. All nodes in the list that are still relevant should be either directly or indirectly referenced by one of the DWPT's private DeleteSlice or by the global BufferedUpdates slice.

Each DWPT as well as the global delete pool maintain their private DeleteSlice instance. In the DWPT case updating a slice is equivalent to atomically finishing the document. The slice update guarantees a "happens before" relationship to all other updates in the same indexing session. When a DWPT updates a document it:

1. consumes a document and finishes its processing
2. updates its private DeleteSlice either by calling #updateSlice(DeleteSlice) or #add(Term, DeleteSlice) (if the document has a delTerm)
3. applies all deletes in the slice to its private BufferedUpdates and resets it
4. increments its internal document id

To prevent OOM Errors and ensure IndexWriter's stability this class blocks incoming threads from indexing once 2 x number of available ThreadStates in DocumentsWriterPerThreadPool is exceeded. Once flushing catches up and the number of flushing DWPT is equal or lower than the number of active ThreadStates threads are released and can continue indexing.

NOTE: If you override #getLiveDocs(), you will likely need to override #numDocs() as well and vice-versa.

NOTE: If this FilterAtomicReader does not change the content the contained reader, you could consider overriding #getCoreCacheKey() so that IFieldCache and CachingWrapperFilter share the same entries for this atomic reader and the wrapped one. #getCombinedCoreAndDeletesKey() could be overridden as well if the #getLiveDocs() live docs are not changed either.

NOTE: If the order of terms and documents is not changed, and if these terms are going to be intersected with automata, you could consider overriding #intersect for better performance.

Expert: represents a single commit into an index as seen by the {@link IndexDeletionPolicy} or {@link IndexReader}.

Changes to the content of an index are made visible only after the writer who made that change commits by writing a new segments file (segments_N). This point in time, when the action of writing of a new segments file to the directory is completed, is an index commit.

Each index commit point has a unique segments file associated with it. The segments file associated with a later index commit point would have a larger N.

WARNING: This API is a new and experimental and may suddenly change.

NOTE: extensions used by codecs are not listed here. You must interact with the Codec directly. @lucene.internal

There are two different types of IndexReaders:

• AtomicReader: These indexes do not consist of several sub-readers, they are atomic. They support retrieval of stored fields, doc values, terms, and postings.
• CompositeReader: Instances (like DirectoryReader) of this reader can only be used to get stored fields from the underlying AtomicReaders, but it is not possible to directly retrieve postings. To do that, get the sub-readers via CompositeReader#getSequentialSubReaders. Alternatively, you can mimic an AtomicReader (with a serious slowdown), by wrapping composite readers with SlowCompositeReaderWrapper.

IndexReader instances for indexes on disk are usually constructed with a call to one of the static DirectoryReader.open() methods, e.g. DirectoryReader#open(Lucene.Net.Store.Directory). DirectoryReader implements the CompositeReader interface, it is not possible to directly get postings.

For efficiency, in this API documents are often referred to via document numbers, non-negative integers which each name a unique document in the index. These document numbers are ephemeral -- they may change as documents are added to and deleted from an index. Clients should thus not rely on a given document having the same number between sessions.

NOTE: {@link IndexReader} instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently. If your application requires external synchronization, you should not synchronize on the IndexReader instance; use your own (non-Lucene) objects instead.

This tool does file-level copying of segments files. This means it's unable to split apart a single segment into multiple segments. For example if your index is a single segment, this tool won't help. Also, it does basic file-level copying (using simple FileInfo) so it will not work with non FSDirectory Directory impls.

 java -cp lucene-core.jar Lucene.Net.Index.IndexUpgrader [-delete-prior-commits] [-verbose] indexDir 

this tool keeps only the last commit in an index; for this reason, if the incoming index has more than one commit, the tool refuses to run by default. Specify {@code -delete-prior-commits} to override this, allowing the tool to delete all but the last commit. From Java code this can be enabled by passing {@code true} to #IndexUpgrader(Directory,Version,StreamWriter,boolean).

Warning: this tool may reorder documents if the index was partially upgraded before execution (e.g., documents were added). If your application relies on "monotonicity" of doc IDs (which means that the order in which the documents were added to the index is preserved), do a full forceMerge instead. The MergePolicy set by IndexWriterConfig may also reorder documents.

If an index will not have more documents added for a while and optimal search performance is desired, then the optimize method should be called before the index is closed.

NOTE: warm is called before any deletes have been carried over to the merged segment.

All setter methods return IndexWriterConfig to allow chaining settings conveniently, for example:

 IndexWriterConfig conf = new IndexWriterConfig(analyzer); conf.setter1().setter2(); 

This class implements a {@link MergePolicy} that tries to merge segments into levels of exponentially increasing size, where each level has < mergeFactor segments in it. Whenever a given levle has mergeFactor segments or more in it, they will be merged.

This class is abstract and requires a subclass to define the {@link #size} method which specifies how a segment's size is determined. {@link LogDocMergePolicy} is one subclass that measures size by document count in the segment. {@link LogByteSizeMergePolicy} is another subclass that measures size as the total byte size of the file(s) for the segment.

Expert: a MergePolicy determines the sequence of primitive merge operations to be used for overall merge and optimize operations.

Whenever the segments in an index have been altered by {@link IndexWriter}, either the addition of a newly flushed segment, addition of many segments from addIndexes* calls, or a previous merge that may now need to cascade, {@link IndexWriter} invokes {@link #findMerges} to give the MergePolicy a chance to pick merges that are now required. This method returns a {@link MergeSpecification} instance describing the set of merges that should be done, or null if no merges are necessary. When IndexWriter.optimize is called, it calls {@link #findMergesForOptimize} and the MergePolicy should then return the necessary merges.

Note that the policy can return more than one merge at a time. In this case, if the writer is using {@link SerialMergeScheduler}, the merges will be run sequentially but if it is using {@link ConcurrentMergeScheduler} they will be run concurrently.

The default MergePolicy is {@link LogByteSizeMergePolicy}.

NOTE: This API is new and still experimental (subject to change suddenly in the next release)

This is useful, e.g., with collections that have large fields which change rarely and small fields that change more frequently. The smaller fields may be re-indexed in a new index and both indexes may be searched together.

Warning: It is up to you to make sure all indexes are created and modified the same way. For example, if you add documents to one index, you need to add the same documents in the same order to the other indexes. Failure to do so will result in undefined behavior.

NOTE: This API is new and still experimental (subject to change suddenly in the next release)