feat: metricsV2 + oTel + prometheus sample and Grafana dashboard by csviri · Pull Request #3154 · operator-framework/java-operator-sdk

csviri · 2026-02-04T16:40:41Z

Goal of this PR is to provide a OTel + Prometheus + Grafana setup. So we:

verify integration with OTel
Provide a default Grafana dashboard for metrics.
Adds a new operator and E2E test for h OTel + Prometheus + Grafana handling verification. This should be easy to done by a users, so dey can check and validate the dashboard.
new metrics implementations

Notes on new metrics implementation:

only static gauges (lives until the operator lives) - that makes much easir to manage them
no manual counter removal (should not be done)
sanitized names and labels according to best practices
added some metrics counter for last attempt retries.

csviri · 2026-02-09T17:35:19Z

JVM metrics:

csviri · 2026-02-09T17:40:23Z

JODSK metrics:

Added TODOs to improve those, like we should not have controller name as suffix, rather as a tag:

operator_sdk_reconciliations_executions_webpagestandalonedependentsreconciler

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

…torsdk/operator/sample/metrics/MetricsHandlingSampleOperator.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

…tor/api/monitoring/Metrics.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

…operatorsdk/operator/sample/deployment.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

xstefank · 2026-03-03T11:48:53Z

docs/content/en/docs/documentation/observability.md

+| Meter name (Micrometer)                  | Type    | Tags                              | Description                                                          |
+|------------------------------------------|---------|-----------------------------------|----------------------------------------------------------------------|
+| `reconciliations.executions`             | gauge   | `controller.name`                 | Number of reconciler executions currently in progress                |
+| `reconciliations.active`                 | gauge   | `controller.name`                 | Number of resources currently queued for reconciliation              |


is "active" really a good name for this?

xstefank · 2026-03-03T12:43:18Z

...ort/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java

+  private static final String RECONCILIATIONS_STARTED = RECONCILIATIONS + "started" + TOTAL_SUFFIX;
+
+  private static final String RECONCILIATIONS_EXECUTIONS_GAUGE = RECONCILIATIONS + "executions";
+  private static final String RECONCILIATIONS_QUEUE_SIZE_GAUGE = RECONCILIATIONS + "active";


queue would be better name

The thing is that queue is a bit misleading, because this is actually the queue + ongoing reconiliations, that is the reason I added active. (ExecutorService has a queue). Can expand the docs on this

Actually made an adjustment so we have now active and queue where queue is that are submitted to executor service but not started yet, the active is the number of actually running threads with reconiliation logic. This seems to be more intuitive and useful.

- deleted redundant delete events - adjusted metrics now we have queue and active reconciliations (with the intuitive seantics) Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Signed-off-by: Chris Laprun <metacosm@gmail.com>

metacosm · 2026-03-04T08:39:16Z

docs/content/en/docs/migration/v5-3-migration.md

-```
+```
+
+## Metrics interface changes


This would technically be an API break and would require a new major version.

Strinctly sepaking yes, but such minor API changes we do some times, see the migration document. As other frameworks sometimes. It is basically I think a better choice in terms of tradeoff, since because we don't really want to increase the major verion that often and we on the other hand we have quite an amount of APIs, that sometimes better to evolve this way IMO.

I also was trying to do backwards compatible, we still could. But at the end it looked like that it would be more confusing, that just having a table to be able to easily migrate from current impl. If that makes sense.

metacosm · 2026-03-04T08:41:00Z

observability/install-observability.sh

+
+# Check if helm is installed, download locally if not
+echo -e "\n${YELLOW}Checking helm installation...${NC}"
+if ! command -v helm &> /dev/null; then


Installing helm should be made optional. People might not want to have helm automatically installed or want to install it themselves some other way.

What this does, it checks if helm command works (so already installed), if not, downloads it as a binary, but just locally in the project dir (so does not install it), and uses that binary directly. So this is not very invasive at the end IMO, usually it is done this way in such scripts.

metacosm · 2026-03-04T08:45:10Z

...ort/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java

+import java.util.function.Consumer;
+import java.util.function.Function;
+
+import org.jspecify.annotations.NonNull;


This should not have been introduced here. Nullability should be reviewed at the SDK level, not simply here for 2 methods.

removed this, not sure how it got here, good catch!

metacosm · 2026-03-04T08:53:29Z

docs/content/en/docs/documentation/observability.md

+
+\* `namespace` tag is only included when `withNamespaceAsTag()` is enabled.
+
+The execution timer uses explicit SLO boundaries (10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s, 10s, 30s) to ensure


SLO acronym should be used expanded first before.

SLO I consider a well known term, but actually we can just remote that from here. done.

metacosm · 2026-03-04T08:53:55Z

docs/content/en/docs/documentation/observability.md

+\* `namespace` tag is only included when `withNamespaceAsTag()` is enabled.
+
+The execution timer uses explicit SLO boundaries (10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s, 10s, 30s) to ensure
+compatibility with `histogram_quantile()` queries in Prometheus. This is important when using the OTLP registry, where


OTLP acronym should be used expanded first.

metacosm · 2026-03-04T09:08:45Z

...ort/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java

+      if (resourceEvent.getAction() == ResourceAction.ADDED) {
+        gauges.get(numberOfResourcesRefName(getControllerName(metadata))).incrementAndGet();
+      }
+      var namespace = resourceEvent.getRelatedCustomResourceID().getNamespace().orElse(null);


Shouldn't that be in sync with what's done for MDC, i.e. use a default value instead of null for clustered resources?

This way at the end we don't add the namespace tag (if null), that is according the guidelines.

metacosm · 2026-03-04T09:12:09Z

...ort/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java

+
+  private void incrementCounter(
+      String counterName, String namespace, Map<String, Object> metadata, Tag... additionalTags) {
+    final var tags = new ArrayList<Tag>(2 + additionalTags.length);


This shouldn't be a List but a Set.

That is really an implementation detail IMO

metacosm · 2026-03-04T09:14:34Z

...ort/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java

+          null,
+          metadata,
+          Tag.of(EVENT, event.getClass().getSimpleName()),
+          Tag.of(ACTION, UNKNOWN_ACTION));


Why unknown action here?

Removed this, supposedly if we cannot add meaningful value it should added, thank you pointing out

metacosm · 2026-03-04T09:17:42Z

...ort/src/main/java/io/javaoperatorsdk/operator/monitoring/micrometer/MicrometerMetricsV2.java

+
+    int retryNumber = retryInfo.map(RetryInfo::getAttemptCount).orElse(0);
+    if (retryNumber > 0) {
+      incrementCounter(RECONCILIATIONS_RETRIES_NUMBER, namespace, metadata);


How does the retry number get propagated? Seems like the counter that is incremented is completely decorrelated from the actual retry number?

The idea is that we measure the number of retries, so we increase this number if the actual reconiliation is a retry. What is interesting at the end in the metrics is the rate of the retries.

Signed-off-by: Chris Laprun <metacosm@gmail.com>

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 4, 2026

This was linked to issues Feb 4, 2026

Provide a default Grafana dashboard for the default metrics #2975

Open

Investigate integration possibilities with Opentelemetry #2135

Open

csviri changed the title ~~OTel+Prometheus sample and Graphana dashboard~~ [WIP] OTel+Prometheus sample and Graphana dashboard Feb 4, 2026

csviri changed the title ~~[WIP] OTel+Prometheus sample and Graphana dashboard~~ [WIP] OTel+Prometheus sample and Grfana dashboard Feb 8, 2026

csviri changed the title ~~[WIP] OTel+Prometheus sample and Grfana dashboard~~ [WIP] OTel+Prometheus sample and Grafana dashboard Feb 8, 2026

csviri force-pushed the otel-metrics-grafana branch from c7e6ca2 to ece63e8 Compare February 8, 2026 15:25

csviri mentioned this pull request Feb 9, 2026

Micrometer metrics enhancement requests #2974

Open

csviri linked an issue Feb 10, 2026 that may be closed by this pull request

Micrometer metrics enhancement requests #2974

Open

csviri mentioned this pull request Feb 10, 2026

[WIP] improve: micrometer metrics improvements #3164

Closed

csviri requested review from metacosm and xstefank February 10, 2026 11:50

csviri changed the title ~~[WIP] OTel+Prometheus sample and Grafana dashboard~~ [WIP] MetricsV2 + OTel+Prometheus sample and Grafana dashboard Feb 11, 2026

csviri force-pushed the next branch from 280b162 to 429c2c5 Compare February 16, 2026 09:11

csviri force-pushed the otel-metrics-grafana branch 2 times, most recently from 88c118c to cf9eb57 Compare February 21, 2026 12:06

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 23, 2026

csviri force-pushed the otel-metrics-grafana branch from 24f992f to 0438611 Compare February 27, 2026 09:52

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 27, 2026

csviri force-pushed the next branch from bb01a06 to 4868613 Compare February 28, 2026 09:55

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 1, 2026

csviri force-pushed the otel-metrics-grafana branch from 9014b2b to fe4ab6a Compare March 1, 2026 09:51

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 1, 2026

csviri changed the title ~~[WIP] MetricsV2 + OTel+Prometheus sample and Grafana dashboard~~ MetricsV2 + OTel+Prometheus sample and Grafana dashboard Mar 1, 2026

csviri self-assigned this Mar 1, 2026

csviri marked this pull request as ready for review March 1, 2026 10:45

Copilot AI review requested due to automatic review settings March 1, 2026 10:45

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 1, 2026

csviri and others added 16 commits March 3, 2026 08:15

documentation update

a751867

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

4fc3069

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

logging

a3d935d

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Update sample-operators/metrics-processing/src/main/java/io/javaopera…

e6ad757

…torsdk/operator/sample/metrics/MetricsHandlingSampleOperator.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Update sample-operators/metrics-processing/pom.xml

5f46c5c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Update sample-operators/metrics-processing/pom.xml

23416f1

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Update operator-framework-core/src/main/java/io/javaoperatorsdk/opera…

08f33db

…tor/api/monitoring/Metrics.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Update observability/install-observability.sh

e5acc60

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

4eda27b

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

Update sample-operators/metrics-processing/src/main/resources/io/java…

6282ca4

…operatorsdk/operator/sample/deployment.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

0074f84

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

e2c4751

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

0d1a23c

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

6ceb74d

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

45fc814

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

cf114e3

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

csviri force-pushed the otel-metrics-grafana branch from d60193a to cf114e3 Compare March 3, 2026 07:15

xstefank approved these changes Mar 3, 2026

View reviewed changes

csviri and others added 7 commits March 3, 2026 15:28

Refinements on metrics

a804406

- deleted redundant delete events - adjusted metrics now we have queue and active reconciliations (with the intuitive seantics) Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

fbb67b6

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

docs improvement

42e2649

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

fix: add deprecation information

ebac5e2

Signed-off-by: Chris Laprun <metacosm@gmail.com>

refactor: consistent constant definition

810ba90

Signed-off-by: Chris Laprun <metacosm@gmail.com>

refactor: reuse available methods to help inlining

995ace7

Signed-off-by: Chris Laprun <metacosm@gmail.com>

refactor: avoid creating intermediate collection

4a2ec96

Signed-off-by: Chris Laprun <metacosm@gmail.com>

metacosm requested changes Mar 4, 2026

View reviewed changes

metacosm and others added 3 commits March 4, 2026 11:40

refactor: remove unused constant

05de31a

Signed-off-by: Chris Laprun <metacosm@gmail.com>

fixed from code review

392f40e

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

wip

2b86f00

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>

csviri requested a review from metacosm March 4, 2026 20:25


		\* `namespace` tag is only included when `withNamespaceAsTag()` is enabled.

		The execution timer uses explicit SLO boundaries (10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s, 10s, 30s) to ensure

Conversation

csviri commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

csviri commented Feb 9, 2026

Uh oh!

csviri commented Feb 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

csviri Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

csviri Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

csviri commented Feb 4, 2026 •

edited

Loading

csviri Mar 3, 2026 •

edited

Loading

csviri Mar 4, 2026 •

edited

Loading