fix(metrics): export late-created thread reject and error code metrics to Prometheus#16167
Open
Wubabalala wants to merge 1 commit intoapache:3.3from
Open
fix(metrics): export late-created thread reject and error code metrics to Prometheus#16167Wubabalala wants to merge 1 commit intoapache:3.3from
Wubabalala wants to merge 1 commit intoapache:3.3from
Conversation
…s to Prometheus MetricsNameCountSampler.samplesChanged is consumed by the first reporter poll before any actual metric series exist. When the first real event arrives later, SimpleMetricsCountSampler.inc() creates a new counter entry but samplesChanged is never set back to true, so the reporter never re-registers the new series to Prometheus. Refactor getAtomicCounter() into incrementAndGetCreated() that returns whether a new Metric->AtomicLong entry was created. Override inc() in MetricsNameCountSampler to set samplesChanged only on new series creation, avoiding unnecessary re-registration for existing series. Fixes apache#16148
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change?
Fixes #16148
Dynamically added metric series in
MetricsNameCountSamplersubclasses (e.g.,ThreadRejectMetricsCountSampler,ErrorCodeSampler) are not exported to Prometheus when the first event arrives after the initial reporter sync cycle.Root Cause Analysis
MetricsNameCountSampler.samplesChangedis initialized totrueand consumed by the reporter's firstcalSamplesChanged()poll (CAStrue → false). At that point,sample()returns nothing because no actual metric series exist yet.When the first real event arrives later (e.g., a thread pool rejection),
SimpleMetricsCountSampler.inc()creates a new metric series entry inmetricCounter. However,MetricsNameCountSampler.samplesChangedis only updated during metric name registration — it is never set back totruewhen a new metric series is first created at runtime. The reporter seesfalseon subsequent polls and never re-registers the new metric series to the Prometheus registry.Brief changelog
SimpleMetricsCountSampler: RefactoredgetAtomicCounter()intoincrementAndGetCreated()which returnstruewhen a new metric series is created for the first time (detected via reference equality against the candidateAtomicLong).MetricsNameCountSampler: Overrideinc()to callincrementAndGetCreated()and setsamplesChanged = trueonly when a new metric series is created. This avoids unnecessary re-registration for updates to already-registered series. Also madeaddMetricName()idempotent.How to verify
Thread pool reject path:
dubbo.metrics.enable-threadpool: trueanddubbo.protocol.threads: 2.samplesChangedflag is consumed)./actuator/prometheusfordubbo_thread_pool_reject_thread_count— it should now appear.Error code path is covered by the unit test
ErrorCodeSampleTest.testErrorCodeMetricChangesAfterFirstLateEvent, which verifies the same timing sequence with error code events.New tests
ErrorCodeSampleTest.testErrorCodeMetricChangesAfterFirstLateEvent: Verifies the exact timing sequence — initial flag consumed, then first error event sets flag, repeat of same error does not, new error code sets flag again.PrometheusMetricsThreadPoolTest.testThreadPoolRejectMetricsExportedAfterLateFirstEvent: End-to-end test — simulates the full startup → first poll → late event timeline and asserts that Prometheus scrape output contains the late-arriving reject metric series.