Dynatrace Integration as observability vendors

[oguzhan78]

feat(sre): observability vendor framework + Dynatrace integration & Grail recorder

Summary

This PR adds a pluggable observability-vendor framework to the SRE scenarios and uses it to onboard Dynatrace (and Datadog) as first-class, toggleable vendors alongside the existing in-cluster stack. On top of agent deployment of Dynatrace, it introduces a Dynatrace Grail recorder that exports telemetry (metrics, spans, logs, events, and Kubernetes metrics) back out at scenario teardown for offline analysis.

Every vendor is disabled by default, so existing scenarios are unaffected unless a vendor is explicitly enabled.

What's included

Vendor framework (tools role)

• Generic vendor install/uninstall wiring in manage_tools.yaml and the tools role, with per-vendor tasks, argument specs, and namespace ownership guards (ensure_vendor_namespace, assert_vendor_resource_ownership, delete_vendor_namespace_if_safe).
• Molecule test suites: deployment_vendors and vendor_contract (with Datadog valid / mismatched-secret / with-secret fixtures).

Dynatrace (agent)

• install_dynatrace.yaml / uninstall_dynatrace.yaml: deploys the DynaKube CR from an operator-provided dynakube.yaml, pinned via dynakube_api_version for consistent readiness/cleanup.
• OpenTelemetry demo instrumentation hooks in install_opentelemetry_demo.yaml and values.j2 so app + Prometheus telemetry flow to Dynatrace.

Datadog (agent)

• install_datadog.yaml / uninstall_datadog.yaml, manifest validation, and credential handling via env vars (DATADOG_API_KEY, DATADOG_APP_KEY).

Dynatrace Grail recorder (export — new)

• Because Grail retains history queryable by timeframe, the recorder runs once at teardown and back-queries the full incident window [scenario start → now].
• stamp_dynatrace_start_time.yaml (runs in the pre-fault-injection phase) writes the scenario start into a ConfigMap in the data-recorders namespace, so the timestamp survives across separate install/uninstall playbook runs.
• install_dynatrace_recorder.yaml reads a platform token (prefix dt0s16., Grail read scope) from a gitignored file (platform_token_path, under secrets/) and injects it into the recorder Job as an env value — mirroring the ClickHouse recorder's CLICKHOUSE_PASSWORD approach. No env export or Kubernetes Secret required.
• gather.py runs DQL queries and writes JSON/CSV datasets into observability_information_from_dynatrace; wired into uninstall.yaml's export step.
• run_dynatrace_recorder.yaml: a standalone playbook to deploy the recorder Job manually against an already-running scenario (export only — no cronjob suspend, no namespace delete).
• Uninstall ordering: Dynatrace is torn down last, after Prometheus and application pods, because it injects into them for OTel data collection.

AWX

• Vendor credential types (datadog_injectors, dynatrace_injectors), credential/job/workflow configuration, and passing the vendors var into AWX (with no_log).

Config, docs, and secrets hygiene

• New scenarios/sre/docs/observability-vendors.md (543 lines) documenting the framework and both vendors.
• observability_vendors.yaml.example and runner/vendors.yaml.example templates.
• .gitignore: ignore exports/; new scenarios/sre/.gitignore ignores secrets/ and the AWX runner vendors.yaml (must be Ansible-Vault encrypted, never committed in plaintext). .secrets.baseline updated.

Configuration

Enable and configure per vendor in inventory/group_vars/environment/observability_vendors.yaml (see the example file). For the recorder:

observability_vendors:
  dynatrace:
    recorder:
      enabled: true
      platform_url: "https://<env-id>.apps.dynatrace.com"   # NOT the ingest URL
      platform_token_path: "secrets/dynatrace/platform_token"  # dt0s16.<...>, Grail read
      namespace: otel-demo
      format: both        # json | csv | both