public class CommitMitigatingTieredMergePolicy
extends org.apache.lucene.index.MergePolicy
MergePolicy
extends Lucene's TieredMergePolicy
by providing mitigation
to the aggressiveness of merges in case the index is under high commit load.
That's because in the case of Oak we currently have that we store Lucene indexes in storage systems which require
themselves some garbage collection task to be executed to get rid of deleted / unused files, similarly to what Lucene's
merge does.
So the bottom line is that with this MergePolicy
we should have less but bigger merges, only after commit rate
is under a certain threshold (in terms of added docs per sec and MBs per sec).
Auto tuning params:
In this merge policy we would like to avoid having to adjust parameters by hand, but rather have them "auto tune".
This means that the no. of merges should be mitigated with respect to a max commit rate (docs, mb), but also adapt to
the average commit rate and anyway do not let the no. of segments explode.Modifier and Type | Class and Description |
---|---|
protected static class |
CommitMitigatingTieredMergePolicy.MergeScore
Holds score and explanation for a single candidate
merge.
|
org.apache.lucene.index.MergePolicy.DocMap, org.apache.lucene.index.MergePolicy.MergeAbortedException, org.apache.lucene.index.MergePolicy.MergeException, org.apache.lucene.index.MergePolicy.MergeSpecification, org.apache.lucene.index.MergePolicy.MergeTrigger, org.apache.lucene.index.MergePolicy.OneMerge
Modifier and Type | Field and Description |
---|---|
static double |
DEFAULT_NO_CFS_RATIO
Default noCFSRatio.
|
Constructor and Description |
---|
CommitMitigatingTieredMergePolicy()
Sole constructor, setting all settings to their
defaults.
|
Modifier and Type | Method and Description |
---|---|
void |
close() |
org.apache.lucene.index.MergePolicy.MergeSpecification |
findForcedDeletesMerges(org.apache.lucene.index.SegmentInfos infos) |
org.apache.lucene.index.MergePolicy.MergeSpecification |
findForcedMerges(org.apache.lucene.index.SegmentInfos infos,
int maxSegmentCount,
Map<org.apache.lucene.index.SegmentCommitInfo,Boolean> segmentsToMerge) |
org.apache.lucene.index.MergePolicy.MergeSpecification |
findMerges(org.apache.lucene.index.MergePolicy.MergeTrigger mergeTrigger,
org.apache.lucene.index.SegmentInfos infos) |
double |
getFloorSegmentMB()
Returns the current floorSegmentMB.
|
double |
getForceMergeDeletesPctAllowed()
Returns the current forceMergeDeletesPctAllowed setting.
|
int |
getMaxMergeAtOnce()
Returns the current maxMergeAtOnce setting.
|
int |
getMaxMergeAtOnceExplicit()
Returns the current maxMergeAtOnceExplicit setting.
|
double |
getMaxMergedSegmentMB()
Returns the current maxMergedSegmentMB setting.
|
double |
getReclaimDeletesWeight()
|
double |
getSegmentsPerTier()
Returns the current segmentsPerTier setting.
|
protected CommitMitigatingTieredMergePolicy.MergeScore |
score(List<org.apache.lucene.index.SegmentCommitInfo> candidate,
boolean hitTooLarge,
long mergingBytes)
Expert: scores one merge; subclasses can override.
|
CommitMitigatingTieredMergePolicy |
setFloorSegmentMB(double v)
Segments smaller than this are "rounded up" to this
size, ie treated as equal (floor) size for merge
selection.
|
CommitMitigatingTieredMergePolicy |
setForceMergeDeletesPctAllowed(double v)
When forceMergeDeletes is called, we only merge away a
segment if its delete percentage is over this
threshold.
|
CommitMitigatingTieredMergePolicy |
setMaxCommitRateDocs(double maxCommitRate)
set the maximum no.
|
CommitMitigatingTieredMergePolicy |
setMaxCommitRateMB(int maxCommitRateMB)
set the max no.
|
CommitMitigatingTieredMergePolicy |
setMaxMergeAtOnce(int v)
Maximum number of segments to be merged at a time
during "normal" merging.
|
CommitMitigatingTieredMergePolicy |
setMaxMergeAtOnceExplicit(int v)
Maximum number of segments to be merged at a time,
during forceMerge or forceMergeDeletes.
|
CommitMitigatingTieredMergePolicy |
setMaxMergedSegmentMB(double v)
Maximum sized segment to produce during
normal merging.
|
CommitMitigatingTieredMergePolicy |
setMaxNoOfSegsForMitigation(int maxNoOfSegsForMitigation)
Maximum number of segments allowed for mitigation to happen.
|
CommitMitigatingTieredMergePolicy |
setReclaimDeletesWeight(double v)
Controls how aggressively merges that reclaim more
deletions are favored.
|
CommitMitigatingTieredMergePolicy |
setSegmentsPerTier(double v)
Sets the allowed number of segments per tier.
|
String |
toString() |
public static final double DEFAULT_NO_CFS_RATIO
MergePolicy.setNoCFSRatio(double)
,
Constant Field Valuespublic CommitMitigatingTieredMergePolicy()
public CommitMitigatingTieredMergePolicy setMaxMergeAtOnce(int v)
setMaxMergeAtOnceExplicit(int)
. Default is 10.v
- the max no. of mergespublic CommitMitigatingTieredMergePolicy setMaxNoOfSegsForMitigation(int maxNoOfSegsForMitigation)
maxNoOfSegsForMitigation
- max no. of segments per mitigationpublic CommitMitigatingTieredMergePolicy setMaxCommitRateDocs(double maxCommitRate)
maxCommitRate
- maxCommitRate maximum commit rate (docs/sec)public int getMaxMergeAtOnce()
setMaxMergeAtOnce(int)
public CommitMitigatingTieredMergePolicy setMaxMergeAtOnceExplicit(int v)
v
- the no. of max mergespublic int getMaxMergeAtOnceExplicit()
setMaxMergeAtOnceExplicit(int)
public CommitMitigatingTieredMergePolicy setMaxMergedSegmentMB(double v)
v
- the maximum segment size in MBpublic double getMaxMergedSegmentMB()
getMaxMergedSegmentMB()
public CommitMitigatingTieredMergePolicy setReclaimDeletesWeight(double v)
v
- the reclaim deletes weightpublic double getReclaimDeletesWeight()
public CommitMitigatingTieredMergePolicy setFloorSegmentMB(double v)
v
- floor segment size in MBpublic double getFloorSegmentMB()
setFloorSegmentMB(double)
public CommitMitigatingTieredMergePolicy setForceMergeDeletesPctAllowed(double v)
v
- the force merge deletes allowed percentagepublic double getForceMergeDeletesPctAllowed()
setForceMergeDeletesPctAllowed(double)
public CommitMitigatingTieredMergePolicy setSegmentsPerTier(double v)
NOTE: this value should be >= the setMaxMergeAtOnce(int)
otherwise you'll force too much
merging to occur.
Default is 10.0.
v
- segments per tierpublic double getSegmentsPerTier()
setSegmentsPerTier(double)
public CommitMitigatingTieredMergePolicy setMaxCommitRateMB(int maxCommitRateMB)
maxCommitRateMB
- max commit rate in MB (mb/sec)public org.apache.lucene.index.MergePolicy.MergeSpecification findMerges(org.apache.lucene.index.MergePolicy.MergeTrigger mergeTrigger, org.apache.lucene.index.SegmentInfos infos) throws IOException
findMerges
in class org.apache.lucene.index.MergePolicy
IOException
protected CommitMitigatingTieredMergePolicy.MergeScore score(List<org.apache.lucene.index.SegmentCommitInfo> candidate, boolean hitTooLarge, long mergingBytes) throws IOException
candidate
- a list of candidate segmentshitTooLarge
- hit too large settingmergingBytes
- the bytes to mergeIOException
public org.apache.lucene.index.MergePolicy.MergeSpecification findForcedMerges(org.apache.lucene.index.SegmentInfos infos, int maxSegmentCount, Map<org.apache.lucene.index.SegmentCommitInfo,Boolean> segmentsToMerge) throws IOException
findForcedMerges
in class org.apache.lucene.index.MergePolicy
IOException
public org.apache.lucene.index.MergePolicy.MergeSpecification findForcedDeletesMerges(org.apache.lucene.index.SegmentInfos infos) throws IOException
findForcedDeletesMerges
in class org.apache.lucene.index.MergePolicy
IOException
public void close()
close
in interface Closeable
close
in interface AutoCloseable
close
in class org.apache.lucene.index.MergePolicy
Copyright © 2012–2017 The Apache Software Foundation. All rights reserved.