Flaky test: DeleteAndVerifyFileRemovalsIT.testDeletingClonedTablePersistsFiles

[dlmarion]

Test name(s)
DeleteAndVerifyFileRemovalsIT.testDeletingClonedTablePersistsFiles

Describe the failure observed

java.lang.IllegalStateException: . Timeout exceeded
	at org.apache.accumulo.test.util.Wait.waitFor(Wait.java:125)
	at org.apache.accumulo.test.util.Wait.waitFor(Wait.java:89)
	at org.apache.accumulo.test.functional.DeleteAndVerifyFileRemovalsIT.testDeletingClonedTablePersistsFiles(DeleteAndVerifyFileRemovalsIT.java:261)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Testing Environment:

• Version of this project: 2.1 branch

Additional context
The table delete operation on line 238 will not remove the files if MANAGER_TABLE_DELETE_OPTIMIZATION is true (which it is by default). Instead, it creates gc candidates and relies on the garbage collector to remove the files. The check on line 261 could fail if the garbage collector runs and deletes the files between the time the table is deleted and the check is performed (lines 238 and 261).

accumulo/test/src/main/java/org/apache/accumulo/test/functional/DeleteAndVerifyFileRemovalsIT.java

Lines 238 to 261 in da5c8d1

	client.tableOperations().delete(sourceTable);
	// The source directory and its files must still be present because the GC will not
	// delete files that are still referenced by the clone.
	assertTrue(fs.exists(sourceDir),
	"Source HDFS directory must survive after source table is deleted: " + sourceDir);
	assertTrue(hasRFiles(fs, sourceDir, 4),
	"Source RFiles must survive after source table is deleted: " + sourceDir);
	// Verify that the files have been removed from accumulo metadata.
	try (var ample = getCluster().getServerContext().getAmple().readTablets()
	.forTable(sourceTableId).fetch(TabletMetadata.ColumnType.FILES).build()) {
	assertEquals(0, ample.stream().count());
	}
	// Verify that the files have been added to the clone table's accumulo metadata.
	try (var ample = getCluster().getServerContext().getAmple().readTablets()
	.forTable(cloneTableId).fetch(TabletMetadata.ColumnType.FILES).build()) {
	assertEquals(4, ample.stream().count());
	}
	// A GcCandidate for each tablet directory should exist until the shared references are
	// compacted.
	Wait.waitFor(() -> countGcCandidates(sourceTableId, 4), GC_MAX_WAIT, POLLING_WAIT);

[Amemeda]

Looking into this