fix: Observability stack now works

[NickLarsenNZ]

Sibling PR to #398

Tip

Once stackabletech/stackable-cockpit#440 is released, testing of this will be far more automated.

See test procedure further below for now.

• Tempo uses port 3200 for querying (eg: from Grafana) after a chart update.
• Jaeger changed the name of the service.
• My local testing (mess) deploys to a different namespace, and relies on OpenTelemetryCollector having FQDNs in their configs.

Test procedure

Drop the URL prefix for the stack in stacks-v2.yaml (in case of local changes), then run:

stackablectl stack --stack-file=./stacks/stacks-v2.yaml install observability

Then, with values.yaml:

yes overkill with anchors, but I used it with other operators too - those are removed for brevity, but the anchors seemed useful in future.

# See: https://github.com/open-telemetry/opentelemetry-operator
otelSidecarAnnotations: &otel-sidecar-annotations
  sidecar.opentelemetry.io/inject: default/otel-collector-grpc

# See: https://docs.stackable.tech/home/stable/concepts/observability/telemetry/
telemetry: &operator-telemetry
  consoleLog:
    level: warn
  otelLogExporter:
    enabled: true
    level: info
  otelTraceExporter:
    enabled: true
    level: info

instrumented-operator: &instrumented-operator
  podAnnotations:
    <<: *otel-sidecar-annotations
  telemetry:
    <<: *operator-telemetry

trino-operator:
  <<: *instrumented-operator

Install an operator with OTLP logs/traces:

stackablectl operator install --operator-values=values.yaml trino

Then create a Trino Cluster to generate some logs/traces.

kubectl apply -f -
# Press CTRL-D after pasting the below

---
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
metadata:
  name: simple-trino
spec:
  image:
    productVersion: "479"
  clusterConfig:
    catalogLabelSelector:
      matchLabels:
        trino: simple-trino
  coordinators:
    roleConfig:
      listenerClass: external-unstable
    roleGroups:
      default:
        replicas: 1
  workers:
    roleGroups:
      default:
        replicas: 1

Then check that there are trino-operator logs and traces in Grafana.