otelcol_exporter_send_failed_spans metric is missing in jaeger V2.16.0

[RishithReddyRR]

When I deployed jaeger V2.16.0 failed to find the metric "otelcol_exporter_send_failed_spans" , so to confirm from which version does it missing, I tried to deploy jaeger V2.14.1,V2.15.0 and V2.15.1 in all of them I found "otelcol_exporter_send_failed_spans" metric. Does this metric is removed in V2.16.0 or by mistake does it been removed? can someone please help on this

[dosubot[bot]]

The metric wasn't removed — it was renamed to include the _total suffix to comply with OpenTelemetry/Prometheus naming conventions.

New metric name: otelcol_exporter_send_failed_spans_total

This change was introduced in PR #7903 and affects several counter metrics:

• otelcol_exporter_send_failed_spans → otelcol_exporter_send_failed_spans_total
• otelcol_exporter_sent_spans → otelcol_exporter_sent_spans_total
• otelcol_receiver_accepted_spans → otelcol_receiver_accepted_spans_total
• otelcol_receiver_refused_spans → otelcol_receiver_refused_spans_total

Update your Prometheus queries and Grafana dashboards to use the new metric name with the _total suffix.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[dosubot[bot]]

That's interesting — both otelcol_exporter_sent_spans_total and otelcol_exporter_send_failed_spans_total should be registered at startup by the exporterhelper package and appear together, even with zero values. They're not created dynamically on first failure.

A few things to check:

1. Confirm you're querying with _total suffix: Make sure you're searching for otelcol_exporter_send_failed_spans_total (not the version without _total)

2. Check the raw metrics endpoint: Query the collector's /metrics endpoint directly (typically on port 8888) and search for send_failed to see if the metric exists there

3. Check your Prometheus scrape config: Ensure Prometheus isn't filtering out metrics with zero values through metric relabeling rules

If otelcol_exporter_sent_spans appears but the failed variant doesn't (even with _total), that would be unexpected behavior. Can you share the output from curl localhost:8888/metrics | grep send_failed on one of your collector pods?

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[RishithReddyRR]

@dosu
curl localhost:8888/metrics | grep otelcol_exporter
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100# HELP otelcol_exporter_sent_spans Number of spans successfully sent to destination. [Alpha]

TYPE otelcol_exporter_sent_spans counter

otelcol_exporter_sent_spans{exporter="jaeger_storage_exporter",service_instance_id="f2851bb4-6aa2-406f-b6dc-a0acf38bd552",service_name="jaeger",service_version="v2.16.0"} 2
14037 0 14037 0 0 7047k 0 --:--:-- --:--:-- --:--:-- 13.3M

curl localhost:8888/metrics | grep send_failed
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 14048 0 14048 0 0 7533k 0 --:--:-- --:--:-- --:--:-- 13.3M

[dosubot[bot]]

Thanks for confirming — the grep for send_failed returning empty confirms the metric truly doesn't exist in the /metrics output at all.

This appears to be lazy metric registration behavior. The otelcol_exporter_send_failed_spans_total metric is only created after the first actual export failure occurs. Since your collector hasn't encountered any export failures, the metric simply doesn't exist yet.

This differs from earlier versions where the metric may have been pre-registered with a zero value at startup. The OTel Collector dependency update in v2.16.0 (to v0.146.x) likely changed this registration behavior.

For monitoring/alerting purposes, you have a few options:

1. Use absent() in Prometheus to detect when the metric doesn't exist: absent(otelcol_exporter_send_failed_spans_total{...})
2. Use or vector(0) to default to zero: otelcol_exporter_send_failed_spans_total{...} or vector(0)
3. Accept that the metric will only appear when failures occur

This is technically expected behavior for lazily-registered counters, though I understand it's a change from what you observed in earlier versions. Would you prefer to have this metric pre-registered at startup even with zero value? If so, that could be raised as a feature request.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[RishithReddyRR]

@dosu
curl localhost:8888/metrics | grep otelcol_receiver_failed_spans
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 14039 0 14039 0 0 9061k 0 --:--:-- --:--:-- --:--:-- 13.3M

HELP otelcol_receiver_failed_spans The number of spans that failed to be processed by the receiver due to internal errors. [Alpha]

TYPE otelcol_receiver_failed_spans counter

otelcol_receiver_failed_spans{receiver="jaeger",service_instance_id="f2851bb4-6aa2-406f-b6dc-a0acf38bd552",service_name="jaeger",service_version="v2.16.0",transport="collector_http"} 0

here above "otelcol_receiver_failed_spans" metric value is also zero still it is not lazy loading?

[dosubot[bot]]

That's a great observation and you're right to question the inconsistency. The behavior difference is real — receiver metrics are pre-registered with zero values while exporter failure metrics are lazily registered.

This inconsistency comes from the upstream OpenTelemetry Collector implementation, not Jaeger itself. Jaeger v2 uses standard OTel Collector receiver and exporter components, and the metric registration behavior is inherited from those libraries. The receiverhelper and exporterhelper packages in OTel Collector handle metric registration differently.

This does seem like inconsistent behavior that could be improved. If you'd like this to be addressed, the appropriate place to report it would be the upstream opentelemetry-collector repository, since that's where the registration logic lives.

In the meantime, the workarounds I mentioned earlier (absent() or or vector(0) in your Prometheus queries) would help ensure your alerting works correctly despite the missing metric.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}