Skip to content

perf: fix GenericObjectController bypassing controller-runtime cache on every reconcile#211

Open
GnatorX wants to merge 1 commit into
awslabs:mainfrom
GnatorX:garvinp-fix-generic-object-controller
Open

perf: fix GenericObjectController bypassing controller-runtime cache on every reconcile#211
GnatorX wants to merge 1 commit into
awslabs:mainfrom
GnatorX:garvinp-fix-generic-object-controller

Conversation

@GnatorX

@GnatorX GnatorX commented Jun 9, 2026

Copy link
Copy Markdown

Issue #, if available:

GenericObjectController.Reconcile was calling kubeClient.Get with a *UnstructuredAdapter[T] object. Because UnstructuredAdapter embeds
unstructured.Unstructured, it implements the runtime.Unstructured interface. controller-runtime's cached client checks for this
interface in shouldBypassCache() and, when cacheUnstructured is not explicitly enabled, skips the cache and sends a direct HTTP GET to
the API server.

Since GenericObjectController requeues every 10 seconds, this produced tens of thousands of uncached node (or other resource) GET
requests per minute under normal idle operation — visible in audit logs and a meaningful load on the API server.

This matches the issue reported and fixed in karpenter upstream: kubernetes-sigs/karpenter#3069

Description of changes:
Refactored Controller.reconcile into two internal helpers:

  • cleanupNotFound — handles the not-found cleanup path (metric deletion, finalizer events)
  • reconcileFound — handles all post-fetch processing (conditions, metrics, transitions)

GenericObjectController.Reconcile now:

  1. Calls kubeClient.Get with a typed T object, which goes through the cache normally
  2. Converts the result to proper unstructured form via runtime.DefaultUnstructuredConverter.ToUnstructured before wrapping in
    UnstructuredAdapter

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants