HBASE-30320 TestMemStoreLAB#testLABChunkQueue OOM on fast hardware due to unbounded off-pool chunk allocation - #8533
Open
Umeshkumar9414 wants to merge 1 commit into
Open
Conversation
…e to unbounded off-pool chunk allocation Add a memory guard in testLABChunkQueue's allocation threads: before each copyCellInto call, check heap usage via ManagementFactory and stop allocating if used memory exceeds 80% of max heap. This prevents OOM on fast hardware (e.g. Apple Silicon) where threads can allocate ~12.5GB of 256KB chunks in the 1-second test window, while preserving the time-based test semantics.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TestMemStoreLAB#testLABChunkQueueOOMs on machines with fast memory subsystems (e.g. Apple Silicon) even with large heap sizes (12GB+).Root Cause: The test spawns 10 threads allocating 256KB chunks in a tight loop for a fixed 1-second window. Once the ChunkCreator pool reaches
maxCount, subsequent allocations use unbounded off-pool chunk creation. All chunks remain live untilmslab.close()is called after the threads stop. On fast hardware (~5,000 iterations/thread/sec), this creates ~12.5GB in 1 second, exceeding heap. On slower x86 CI machines (~700 iter/thread/sec), only ~1.7GB is created, fitting within the default 2.2GB surefire heap.Fix: Add a memory guard — before each
copyCellInto, check heap usage viaManagementFactory.getMemoryMXBean().getHeapMemoryUsage().getUsed()and stop allocating if used memory exceeds 80% of max heap. The 1-second time window is preserved; threads exit early only on memory-constrained JVMs.Test plan
TestMemStoreLAB(all 5 tests) passes with-Dsurefire.Xmx=4gon Apple Silicon