Add apiserver, cache-sync, and db-manager checks to katib-controller /readyz

[bharathjyothi]

What this PR does / why we need it:

Follows up on the readyz TODO left in #1934:

need to more detailed check whether is it possible to communicate with k8s-apiserver or db-manager at '/readyz'

Today /readyz only checks hookServer.StartedChecker(), basically checking that the webhook's HTTPS listener has started. It says nothing about whether the controller can actually do its job. This PR adds three checks, each addressing a different way "ready" and "actually working" can diverge:

1. readyz (cache-sync) : /readyz now also waits on mgr.GetCache().WaitForCacheSync. Previously a pod could report ready before its Experiment/Trial/Suggestion informers had done their initial List/Watch, so both the admission webhooks (which read cluster state through the cache-backed client) and the reconcilers could serve stale/incomplete results for a window right after startup, or indefinitely if the caches never synced (e.g. an RBAC misconfiguration).
2. apiserver : a direct, uncached Get through mgr.GetAPIReader(), proving live apiserver connectivity independent of the cache-sync check above (a cache that synced once and went stale wouldn't catch a since-severed connection).
3. db-manager (diagnostic, non-gating): periodically polls katib-db-manager's gRPC health service (which itself only reports SERVING after a SELECT 1 against its DB succeeds) and logs state transitions. Deliberately not wired into AddReadyzCheck, see below.

Why db-manager doesn't gate readiness:

I initially registered it as a readyz check too, since trial-controller does depend on db-manager synchronously (pkg/controller.v1beta1/trial/managerclient) for observation logs. Verified live against a kind/minikube cluster, this reproduces a startup deadlock: katib-controller's pod-mutating webhook (failurePolicy: Fail) gates creation of every pod in the kubeflow namespace, including katib-db-manager's own pod. Kubernetes only routes Service traffic to Ready pods. So: db-manager's pod can't be created until the webhook is reachable → which needs katib-controller Ready → which needed db-manager reachable. A from-scratch deploy of controller+db-manager+mysql together hung indefinitely (dial tcp ...:443: connect: connection refused from the replicaset controller) until this check was taken out of the readiness gate. Re-verified after the fix: the same fresh deploy comes up cleanly in ~16s, and the controller stays Ready through a live db-manager outage while the diagnostic correctly logs it.

Not addressed here: the second half of the original ask, tuning probe frequency/startupProbe parameters (referenced in this discussion), is left as a separate follow-up; happy to open a new issue for it if that's preferred over leaving #1973 open.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1973

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign johnugeorge for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🎉 Welcome to the Kubeflow Katib repo! 🎉

Thanks for opening your first PR! We're excited to have you onboard 🚀

Next steps:

• Our team will review your PR soon! cc @kubeflow/wg-automl-leads
• Check out the Contributing Guide and the Kubeflow Contributor Guide
• Join the Kubeflow Slack channels: https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels
• Join the AutoML & Training WG meetings: https://bit.ly/2PWVCkV

Feel free to ask questions in the comments. Thanks again for contributing! 🙏