HIVE-28755: Statistics Management Task by DanielZhu58 · Pull Request #6438 · apache/hive

DanielZhu58 · 2026-04-17T10:29:19Z

What changes were proposed in this pull request?

Added StatisticsManagementTask, a new MetastoreTaskThread that periodically scans TAB_COL_STATS and PART_COL_STATS and deletes column statistics whose lastAnalyzed timestamp is older than the configured retention period. Three new configs are introduced: metastore.column.statistics.auto.deletion (default: false), metastore.column.statistics.retention.period (default: 365 days), and metastore.column.statistics.management.task.frequency (default: 7 days). Individual tables can opt out by setting the table property statistics.auto.deletion.exclude.

Why are the changes needed?

Column statistics can become stale over time and consume unnecessary storage. There was no existing mechanism to automatically clean them up based on age, requiring manual intervention.

Does this PR introduce any user-facing change?

Yes. Three new metastore configuration properties are introduced. The feature is opt-in and disabled by default (metastore.column.statistics.auto.deletion=false), so existing deployments are unaffected after upgrade.

How was this patch tested?

Added TestStatisticsManagement with a few unit tests. There is one verifying that expired table-level column statistics are deleted when the task runs, and one verifying that tables marked with the exclude property are left untouched. And there are a few more for corner cases.

Copilot

Pull request overview

Adds a new Metastore background task intended to automatically delete expired column statistics, and wires it into the metastore task thread list. The PR also introduces a benchmark hook and a new unit test for the statistics-management behavior.

Changes:

Introduce StatisticsManagementTask (a MetastoreTaskThread) that deletes column stats older than a configured retention window.
Add new metastore configuration knobs for stats-management frequency/retention/enablement and register the task in the task thread list.
Add a micro-benchmark entry and a new unit test class for statistics management.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java	Adds a benchmark for `StatisticsManagementTask` (currently simulates/validates the wrong thing).
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java	Wires the new benchmark into the benchmark suite (one suite name is inconsistent).
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestStatisticsManagement.java	New unit tests for stats auto-deletion (currently missing required `statsDesc` on the stats object).
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatisticsManagementTask.java	New background task implementation (contains multiple correctness/compilation/runtime issues).
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java	Adds new conf vars and registers the task; docstrings currently don’t match how the feature is enabled.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

soumyakanti3578

I haven't reviewed StatisticsManagementTask.java yet because there are too many sonar/checkstyle warning.

Please follow the hive coding conventions to get rid of those warnings: https://hive.apache.org/community/resources/howtocontribute/#coding-conventions

You can also see the checkstyle file for standalone-metastore at standalone-metastore/checkstyle/checkstyle.xml which has the default values that sonar checks against.

I think you can also use mvn checkstyle from the console or download the checkstyle plugin for IntelliJ IDEA to catch these warnings before submitting the PR.

In general, if you're creating a new file, it's a good idea to follow the official coding conventions, and if you're updating small parts of a file then it's okay to follow the existing style to maintain readability. :)

soumyakanti3578 · 2026-05-06T22:14:49Z

      "Automatic partition management will look for tables using the specified table pattern"),

+    STATISTICS_MANAGEMENT_TASK_FREQUENCY("metastore.statistics.management.task.frequency",
+            "metastore.statistics.management.task.frequency",


The first two arguments are the same. Should the second one (hiveName) be something else, like, hive.column.statistics.management.task.frequency? Not sure if it even matters tbh.

Similar issue for the other two props too.

The second argument (hiveName) exists for backward compatibility — when a config previously lived under a hive. prefix in older Hive versions, HMS needs to recognize both names. Since these three configs are brand new and have no prior hive.-prefixed equivalent, it's ok to keep them same.

dengzhhu653 · 2026-05-11T11:55:05Z

+              + "table.database.name, "
+              + "table.tableName, "
+              + "colName, "
+              + "table.parameters.get(\"" + STATISTICS_AUTO_DELETION_EXCLUDE_TBLPROPERTY + "\")");


Could add a filter for the tblQuery: "table.parameters.get(\"" + STATISTICS_AUTO_DELETION_EXCLUDE_TBLPROPERTY + "\")"

Acknowledged. Added a filter to filter out the excluded tables.

dengzhhu653 · 2026-05-11T11:59:41Z

+    if (!MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.COLUMN_STATISTICS_AUTO_DELETION)
+        || MetastoreConf.getTimeVar(conf, MetastoreConf.ConfVars.COLUMN_STATISTICS_RETENTION_PERIOD,
+        TimeUnit.MILLISECONDS) <= 0) {
+      return Long.MAX_VALUE;


Acknowledged and changed.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 14 comments.

DanielZhu58 · 2026-05-12T06:46:47Z

+    try (Query tblQuery = pm.newQuery(MTableColumnStatistics.class)) {
+      tblQuery.setFilter(
+          "lastAnalyzed < threshold "
+              + "&& table.parameters.get(\""
+              + STATISTICS_AUTO_DELETION_EXCLUDE_TBLPROPERTY + "\") != \"true\"");
+      tblQuery.declareParameters("long threshold");
+      tblQuery.setRange(0, 1000);
+      tblQuery.setResult(
+          "table.database.catalogName, "
+              + "table.database.name, "
+              + "table.tableName, "
+              + "colName");
+      @SuppressWarnings("unchecked")
+      List<Object[]> rows = (List<Object[]>) tblQuery.execute(lastAnalyzedThreshold);
+      return new ArrayList<>(rows);
+    }


Acknowledged. I added a filter which only collect the hive table and hive partitions. We do not want to touch any impala/spark tables and partitions.

DanielZhu58 · 2026-05-12T06:48:20Z

+      boolean committed = false;
+      openTransaction();
+      try {
+        deleteTableColumnStatistics(coords[0], coords[1], coords[2], entry.getValue(), "hive");


Same issue as the previous one. Resolved.

DanielZhu58 · 2026-05-12T06:48:45Z

+    try (Query partQuery = pm.newQuery(MPartitionColumnStatistics.class)) {
+      partQuery.setFilter(
+          "lastAnalyzed < threshold "
+              + "&& partition.table.parameters.get(\""
+              + STATISTICS_AUTO_DELETION_EXCLUDE_TBLPROPERTY + "\") != \"true\"");
+      partQuery.declareParameters("long threshold");
+      partQuery.setRange(0, 1000);
+      partQuery.setResult(
+          "partition.table.database.catalogName, "
+              + "partition.table.database.name, "
+              + "partition.table.tableName, "
+              + "partition.partitionName, "
+              + "colName");
+      @SuppressWarnings("unchecked")
+      List<Object[]> rows = (List<Object[]>) partQuery.execute(lastAnalyzedThreshold);
+      return new ArrayList<>(rows);
+    }


Same issue as the previous one. Resolved.

DanielZhu58 · 2026-05-12T06:49:00Z

+        deletePartitionColumnStatistics(coords[0], coords[1], coords[2],
+            Collections.singletonList(coords[3]), entry.getValue(), "hive");


Same issue as the previous one. Resolved.

DanielZhu58 · 2026-05-12T06:49:44Z

+   *
+   * @param pm                    the JDO persistence manager to use for the query
+   * @param lastAnalyzedThreshold epoch seconds; rows with lastAnalyzed below this value are expired
+   * @return list of projected rows: [catName, dbName, tblName, colName, excludeVal]


Acknowledged and changed.

DanielZhu58 · 2026-05-12T06:55:21Z

+  private void assertHasPartitionColStats(String db, String tbl, String partName,
+                                          String col) throws TException {
+    Map<String, List<ColumnStatisticsObj>> statsMap = client.getPartitionColumnStatistics(
+        db, tbl, Collections.singletonList(partName), Collections.singletonList(col), "hive");


Acknowledged and changed. Will leave the engine as "hive" for now.

+      q.setFilter("table.tableName == t && table.database.name == d");
+      q.declareParameters("java.lang.String t, java.lang.String d");
+      @SuppressWarnings("unchecked")
+      List<MTableColumnStatistics> rows = (List<MTableColumnStatistics>) q.execute(tbl, db);


+      q.setFilter("partition.table.tableName == t && partition.table.database.name == d");
+      q.declareParameters("java.lang.String t, java.lang.String d");


DanielZhu58 · 2026-05-12T07:02:56Z

+            7, TimeUnit.DAYS, "Frequency at which timer task runs to do automatic statistics \n" +
+            "management for tables. Statistics management include 2 configs. \n" +
+            "One is 'metastore.column.statistics.auto.deletion', and the other is 'metastore.column.statistics.retention.period'. \n" +
+            "When 'metastore.column.statistics.auto.deletion'='true' is set, statistics management will look for tables which their\n" +
+            "column statistics are over the retention period, and then delete the column stats. \n"),
+
+    COLUMN_STATISTICS_RETENTION_PERIOD("metastore.column.statistics.retention.period",
+            "metastore.column.statistics.retention.period", 365, TimeUnit.DAYS, "The retention period " +
+            "that we want to keep the stats for each table, which means if the stats are older than this period\n" +
+            "of time, the stats will be automatically deleted. \n"),
+
+    COLUMN_STATISTICS_AUTO_DELETION("metastore.column.statistics.auto.deletion", "metastore.column.statistics.auto.deletion", false,
+            "Whether table/partition column statistics will be auto deleted after retention period"),


The description is very clear, no need to change.

DanielZhu58 · 2026-05-12T07:04:24Z

+  @Test
+  public void testExpiredTableColStatsAreDeleted() throws Exception {


Acknowledged and changed.

dengzhhu653 · 2026-05-12T07:31:05Z

+            "that we want to keep the stats for each table, which means if the stats are older than this period\n" +
+            "of time, the stats will be automatically deleted. \n"),
+
+    COLUMN_STATISTICS_AUTO_DELETION("metastore.column.statistics.auto.deletion", "metastore.column.statistics.auto.deletion", false,


since we can use metastore.column.statistics.management.task.frequency to turn on/off the management task, so I think we don't need extra COLUMN_STATISTICS_AUTO_DELETION to configure the task

I think it's ok to keep COLUMN_STATISTICS_AUTO_DELETION. The two configs have distinct responsibilities: the boolean controls whether the feature is active at all, while the frequency controls how often it runs. Conflating them makes the configuration less readable. It can gives users a simple toggle to temporarily disable deletion without having to remember and restore their frequency setting.

dengzhhu653 · 2026-05-12T07:32:41Z

+   * statistics deletion regardless of the global retention setting.
+   */
+  public static final String STATISTICS_AUTO_DELETION_EXCLUDE_TBLPROPERTY =
+      "statistics.auto.deletion.exclude";


metastore.statistics.auto.deletion.exclude ?
can we add this parameter to the database or table, so we don't need to set it explicitly per table/partition?

That's a good point and I agree that database-level exclusion would be more convenient when users want to exclude all tables in a database. However, that's a very rare case. The Jira is created to delete the stale column stats, the exclusion is just for the users to prevent. Since this feature usually delete the column stats created one year ago, it is rare that the users want to exclude a certain table out of auto deletion. We can defer that to a follow-up Jira later.

dengzhhu653 · 2026-05-12T07:41:00Z

+    createDbAndTable(dbName, tableName, false);
+    writeTableLevelColStats(dbName, tableName, "c1");
+    assertHasTableColStats(dbName, tableName, "c1");
+    makeAllTableColStatsOlderThanRetention(dbName, tableName);


Create some fresh stats so we can check those stats are not deleted after runStatisticsManagementTask

Acknowledged. I added 2 new tests to check both table and partitions so not delete fresh column stats.

sonarqubecloud · 2026-05-12T14:19:00Z

Quality Gate passed

Issues
15 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

asf-ci-hive added tests pending tests failed and removed tests pending labels Apr 17, 2026

DanielZhu58 force-pushed the HIVE-28755 branch from ed8c62d to 443fddb Compare April 28, 2026 00:22

asf-ci-hive added tests pending and removed tests failed labels Apr 28, 2026

DanielZhu58 marked this pull request as ready for review April 28, 2026 00:22

dengzhhu653 requested a review from Copilot April 28, 2026 02:41

Copilot started reviewing on behalf of dengzhhu653 April 28, 2026 02:42 View session

Copilot AI reviewed Apr 28, 2026

View reviewed changes

dengzhhu653 requested a review from Copilot April 28, 2026 02:52

asf-ci-hive added tests unstable and removed tests pending labels Apr 28, 2026

Copilot started reviewing on behalf of dengzhhu653 April 28, 2026 03:00 View session

Copilot AI reviewed Apr 28, 2026

View reviewed changes

dengzhhu653 reviewed Apr 28, 2026

View reviewed changes

Comment thread ...etastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatisticsManagementTask.java Outdated

Hongdan Zhu added 5 commits May 6, 2026 02:41

HIVE-28755: Statistics Management Task

e1841cf

HIVE-28755: commit 2

94be1ff

HIVE-28755: commit 3 after review comments

1193c11

HIVE-28755: general tests added

0d05096

HIVE-28755: final changes

ddcf54b

DanielZhu58 force-pushed the HIVE-28755 branch from 443fddb to ddcf54b Compare May 6, 2026 09:41

asf-ci-hive added tests pending tests unstable and removed tests unstable tests pending labels May 6, 2026

soumyakanti3578 reviewed May 6, 2026

View reviewed changes

HIVE-28755: new changes based on reviews

73d7606

DanielZhu58 requested a review from dengzhhu653 May 7, 2026 08:27

asf-ci-hive removed the tests unstable label May 7, 2026

asf-ci-hive added the tests pending label May 11, 2026

dengzhhu653 reviewed May 11, 2026

View reviewed changes

Comment thread ...etastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatisticsManagementTask.java

dengzhhu653 reviewed May 11, 2026

View reviewed changes

asf-ci-hive added tests unstable and removed tests pending labels May 11, 2026

HIVE-28755: new changes

86f2af4

asf-ci-hive added tests pending and removed tests unstable labels May 12, 2026

dengzhhu653 requested review from Copilot and dengzhhu653 May 12, 2026 04:06

Copilot AI reviewed May 12, 2026

View reviewed changes

Copilot started reviewing on behalf of dengzhhu653 May 12, 2026 04:22 View session

asf-ci-hive added tests unstable and removed tests pending labels May 12, 2026

dengzhhu653 reviewed May 12, 2026

View reviewed changes

Comment thread ...etastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestStatisticsManagement.java

dengzhhu653 reviewed May 12, 2026

View reviewed changes

HIVE-28755: latest changes, fixing test failures and review comments

85868ce

asf-ci-hive added tests pending tests unstable and removed tests unstable tests pending labels May 12, 2026

HIVE-28755: new changes and new tests

272005c

asf-ci-hive added tests pending and removed tests unstable labels May 12, 2026

asf-ci-hive added tests passed and removed tests pending labels May 12, 2026

		deletePartitionColumnStatistics(coords[0], coords[1], coords[2],
		Collections.singletonList(coords[3]), entry.getValue(), "hive");

		q.setFilter("partition.table.tableName == t && partition.table.database.name == d");
		q.declareParameters("java.lang.String t, java.lang.String d");

		@Test
		public void testExpiredTableColStatsAreDeleted() throws Exception {

Conversation

DanielZhu58 commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soumyakanti3578 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

DanielZhu58 commented Apr 17, 2026 •

edited

Loading