Skip to content

Commit ebe321a

Browse files
HurSungYunvirajjasani
authored andcommitted
HBASE-14847 Add FIFO compaction section to HBase book
Closes #2298 Signed-off-by: Viraj Jasani <vjasani@apache.org>
1 parent 227084c commit ebe321a

1 file changed

Lines changed: 54 additions & 0 deletions

File tree

src/main/asciidoc/_chapters/architecture.adoc

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2542,6 +2542,60 @@ All the settings that apply to normal compactions (see <<compaction.parameters>>
25422542
The exceptions are the minimum and maximum number of files, which are set to higher values by default because the files in stripes are smaller.
25432543
To control these for stripe compactions, use `hbase.store.stripe.compaction.minFiles` and `hbase.store.stripe.compaction.maxFiles`, rather than `hbase.hstore.compaction.min` and `hbase.hstore.compaction.max`.
25442544
2545+
[[ops.fifo]]
2546+
===== FIFO Compaction
2547+
2548+
FIFO compaction policy selects only files which have all cells expired. The column family *MUST* have non-default TTL.
2549+
Essentially, FIFO compactor only collects expired store files.
2550+
2551+
Because we don't do any real compaction, we do not use CPU and IO (disk and network) and evict hot data from a block cache.
2552+
As a result, both RW throughput and latency can be improved.
2553+
2554+
[[ops.fifo.when]]
2555+
===== When To Use FIFO Compaction
2556+
2557+
Consider using FIFO Compaction when your use case is
2558+
2559+
* Very high volume raw data which has low TTL and which is the source of another data (after additional processing).
2560+
* Data which can be kept entirely in a a block cache (RAM/SSD). No need for compaction of a raw data at all.
2561+
2562+
Do not use FIFO compaction when
2563+
2564+
* Table/ColumnFamily has MIN_VERSION > 0
2565+
* Table/ColumnFamily has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
2566+
2567+
[[ops.fifo.enable]]
2568+
====== Enabling FIFO Compaction
2569+
2570+
For Table:
2571+
2572+
[source,java]
2573+
----
2574+
HTableDescriptor desc = new HTableDescriptor(tableName);
2575+
desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
2576+
FIFOCompactionPolicy.class.getName());
2577+
----
2578+
2579+
For Column Family:
2580+
2581+
[source,java]
2582+
----
2583+
HColumnDescriptor desc = new HColumnDescriptor(family);
2584+
desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
2585+
FIFOCompactionPolicy.class.getName());
2586+
----
2587+
2588+
From HBase Shell:
2589+
2590+
[source,bash]
2591+
----
2592+
create 'x',{NAME=>'y', TTL=>'30'}, {CONFIGURATION => {'hbase.hstore.defaultengine.compactionpolicy.class' => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy', 'hbase.hstore.blockingStoreFiles' => 1000}}
2593+
----
2594+
2595+
Although region splitting is still supported, for optimal performance it should be disabled, either by setting explicitly `DisabledRegionSplitPolicy` or by setting `ConstantSizeRegionSplitPolicy` and very large max region size.
2596+
You will have to increase to a very large number store's blocking file (`hbase.hstore.blockingStoreFiles`) as well.
2597+
There is a sanity check on table/column family configuration in case of FIFO compaction and minimum value for number of blocking file is 1000.
2598+
25452599
[[arch.bulk.load]]
25462600
== Bulk Loading
25472601

0 commit comments

Comments
 (0)