perf: Initial configuration takes too long

[martin-schulze-e2m]

Issue Faced

When starting up the ingress controller with several route CRDs (8 routes, 6 upstreams), it takes about 5 minutes to send the first sync request to the apisix instance. This means that there is a downtime of 5m if no apisix instance has the current configuration, e.g. when migrating from another chart (bitnami).

Logs

Note the timestamps

{"level":"info","message":"ADC server is running on: /sockets/adc.sock","timestamp":"2026-02-27T09:26:11.662Z"
{"level":"info","message":"PUT /sync","requestId":"ef75a39f-035f-4d18-8d2e-5dc3c10d9870","timestamp":"2026-02-27T09:31:43.141Z"} 

Steps to Reproduce

1. Install apisix helm chart
2. observe apisix pod not ready for ~5 minutes

Environment

• APISIX Ingress controller version: image apache/apisix-ingress-controller:2.0.1

Version: 2.0.1
Git SHA: 
Go Version: go1.24.11
Building OS/Arch: /
Running OS/Arch: linux/amd64

• Kubernetes cluster version (run kubectl version): v1.29.15

[Baoyuantop]

Hi @martin-schulze-e2m, the 5-minute delay you're observing is caused by a hardcoded readiness check timeout. When the ingress controller starts, it enters a WaitReady state to ensure all Kubernetes resources (like ApisixRoutes, Ingresses, etc.) have been processed at least once before it performs the first full configuration sync to APISIX. This is a safety mechanism to prevent pushing incomplete configurations.

This waiting period has a timeout of exactly 5 minutes, which is defined in internal/provider/apisix/provider.go:

func (d *apisixProvider) Start(ctx context.Context) error {
	d.log.Info("starting provider, waiting for readiness")
	d.readier.WaitReady(ctx, 5*time.Minute) // This line causes the 5-minute block
	d.log.Info("Ready detected, starting sync loop")
    // ...
}

In your case, it seems that at least one of the watched resources is not being marked as "ready" by its reconciler within this 5-minute window, causing the WaitReady function to time out. After the timeout, the controller proceeds with the sync, which is why you see the PUT /sync log entry exactly 5 minutes after startup.

While simply changing the timeout might seem like a solution, the core issue is understanding why the initial reconciliation is taking so long for a small number of resources. To move forward, we need to identify which resource is causing the delay.

Could you please help by providing more detailed debug logs?

[martin-schulze-e2m]

I restarted to get the logs from ingress controller:

# the three following lines repeat many times for different values
manager 2026-03-02T12:59:05.999Z    INFO    status.updater    status/updater.go:143    received a status update    {"namespace": "<redacted>", "name": "<redacted>", "kind": "ApisixUpstream"}
manager 2026-03-02T12:59:06.000Z    INFO    controller-runtime    controller/controller.go:239    Starting Controller    {"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway"}
manager 2026-03-02T12:59:06.000Z    INFO    controller-runtime    controller/controller.go:248    Starting workers    {"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "worker count": 1}      

# the next line repeat 9x more times for different namespace/name pairs and state_counts within 4ms
manager 2026-03-02T12:59:06.002Z    INFO    readiness    readiness/manager.go:163    marking resource as done    {"gvk": "apisix.apache.org/v2, Kind=ApisixRoute", "name": {"name":"<redacted>","namespace":<redacted>"}, "state_count": 8}

manager 2026-03-02T12:59:06.002Z    INFO    controller-runtime    controller/controller.go:239    Starting Controller    {"controller": "ingress", "controllerGroup": "networking.k8s.io", "controllerKind": "Ingress"}
manager 2026-03-02T12:59:06.003Z    INFO    controller-runtime    controller/controller.go:248    Starting workers    {"controller": "ingress", "controllerGroup": "networking.k8s.io", "controllerKind": "Ingress", "worker count": 1}
manager 2026-03-02T12:59:06.003Z    INFO    controllers.Ingress    controller/ingress_controller.go:153    reconciling ingress    {"ingress": "apisix-data-plane"}
manager 2026-03-02T12:59:06.003Z    INFO    controllers.Ingress    controller/ingress_controller.go:558    service not found    {"namespace": "apisix", "name": "apisix-data-plane"}
manager 2026-03-02T12:59:06.003Z    INFO    readiness    readiness/manager.go:163    marking resource as done    {"gvk": "apisix.apache.org/v2, Kind=ApisixGlobalRule", "name": {"name":"<redacted>","namespace":"<redacted>"}, "state_count": 1}
manager 2026-03-02T12:59:06.003Z    INFO    readiness    readiness/manager.go:163    marking resource as done    {"gvk": "networking.k8s.io/v1, Kind=Ingress", "name": {"name":"apisix-data-plane","namespace":"apisix"}, "state_count": 2}
manager 2026-03-02T12:59:06.003Z    INFO    controllers.Ingress    controller/ingress_controller.go:153    reconciling ingress    {"ingress": "<redacted2>"}
manager 2026-03-02T12:59:06.003Z    INFO    readiness    readiness/manager.go:163    marking resource as done    {"gvk": "networking.k8s.io/v1, Kind=Ingress", "name": {"name":"<redacted2>","namespace":"<redacted2>"}, "state_count": 1}
manager 2026-03-02T12:59:06.005Z    INFO    controller-runtime.KubeAPIWarningLogger    log/warning_handler.go:65    unknown field "status"
manager 2026-03-02T12:59:06.005Z    INFO    status.updater    status/updater.go:143    received a status update    {"namespace": "<redacted>", "name": "<redacted>", "kind": "ApisixUpstream"}
manager 2026-03-02T12:59:06.006Z    INFO    readiness    readiness/manager.go:163    marking resource as done    {"gvk": "apisix.apache.org/v2, Kind=ApisixRoute", "name": {"name":"<redacted>","namespace":"<redacted>"}, "state_count": 0}
manager 2026-03-02T12:59:06.006Z    INFO    readiness    readiness/manager.go:163    marking resource as done    {"gvk": "apisix.apache.org/v2, Kind=ApisixRoute", "name": {"name":"<redacted>","namespace":"<redacted>"}, "state_count": 0}
manager 2026-03-02T12:59:06.010Z    INFO    controller-runtime.KubeAPIWarningLogger    log/warning_handler.go:65    unknown field "status"

# next two lines repeat 3x more times for different namespace/name pairs within 16ms
manager 2026-03-02T12:59:06.010Z    INFO    status.updater    status/updater.go:143    received a status update    {"namespace": "<redacted>", "name": "<redacted>", "kind": "ApisixUpstream"}
manager 2026-03-02T12:59:06.014Z    INFO    controller-runtime.KubeAPIWarningLogger    log/warning_handler.go:65    unknown field "status"

# the next line repeats 13x with different namespaces/name pairs within one millisecond (one of them is kind ApisixGlobalRule)
manager 2026-03-02T12:59:06.027Z    INFO    status.updater    status/updater.go:143    received a status update    {"namespace": "<redacted>", "name": "<redacted>", "kind": "ApisixRoute"}
manager 2026-03-02T13:04:05.882Z    INFO    provider    apisix/provider.go:256    Ready detected, starting sync loop
manager 2026-03-02T13:04:05.882Z    INFO    provider.client    client/client.go:177    syncing all resources
manager 2026-03-02T13:04:05.882Z    INFO    provider.client    client/client.go:201    syncing resources for config    {"service_number": 13}
adc-server {"level":"info","message":"PUT /sync","requestId":"b0223828-cf1c-4bae-83e2-099e26055f84","timestamp":"2026-03-02T13:04:05.893Z"}

Of these I find most notable:

1. the missing service for ingress apisix-data-plane, which seems to come from the previous installation of the bitnami apisix chart. I have no idea why this was not removed. I deleted it manually and retried without seeing a difference in timing, but the log line is gone. (I would have wished for that to be level warn so it had more visibility.)
2. the unknown field "status" messages (no warning either!) that seems to be connected to upstreams which had already been installed with the bitnami chart. Deleting all upstream CRDs removed the status error messages but did not change the timing.

The upstreams are more or less the same with:

apiVersion: apisix.apache.org/v2                                                                                                                                                                                                                                                                                                                                                        
kind: ApisixUpstream                                                                                                                                                                                                                                                                                                                                                                    
metadata: # the usual fields
spec:
  timeout:
    connect: 60s
    read: 60s
    send: 60s

[martin-schulze-e2m]

I just found that there were a globalpluginconfig and globalroute that defined configuration for opentelemetry. After removing these, the startup time reduced to 30s. These were their contents:

apiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule # or ApisixPluginConfig, the rest is the same
metadata:
  name: opentelemetry
  namespace: apisix
spec:
  plugins:
  - name: opentelemetry
    enable: true
    config:
      sampler:
        name: always_on

[pbaranow]

I observed the same issue on our cluster. In our case 5 minutes startup was observed on the cluster where ApisixUpstream objects are configured.

I reproduced this issue on a dev cluster, where I had only ApisixRoute and ApisixConsumer objects configured. After adding ApisixUpstream object startup time went up from a few seconds to 5 minutes.

I passed the logs to an AI Agent and it pointed out that ApisixUpstream controller is initialized without a Readier:

        // internal/manager/controllers.go
        &apiv2.ApisixUpstream{}: &controller.ApisixUpstreamReconciler{
            Client:  mgr.GetClient(),
            Scheme:  mgr.GetScheme(),
            Log:     ctrl.LoggerFrom(ctx).WithName("controllers").WithName(types.KindApisixUpstream),
            Updater: updater,
            // MISSING: Readier: readier,
        },

and that later Done() function is not called

        // internal/controller/apisixupstream_controller.go
        func (r *ApisixUpstreamReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
            // MISSING: defer r.Readier.Done(&apiv2.ApisixUpstream{}, req.NamespacedName)
            // ...
        }

It also found that similar case is for ApisixPluginConfig objects.

I suppose an Expert from this area needs to take a look at this, to confirm.