bug: VLCluster deploys in single mode

[L1ghtman2k]

Sample VLCluster:

apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: cluster-1
  namespace: vmlog
spec:
  vlinsert:
    hpa:
      enabled: true
      minReplicas: 1
      maxReplicas: 3
      targetCPUUtilizationPercentage: 80
  vlselect:
    hpa:
      enabled: true
      minReplicas: 1
      maxReplicas: 3
      targetCPUUtilizationPercentage: 80
  vlstorage:
    retentionPeriod: "30d"
    storage:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 50Gi
    hpa:
      enabled: true
      minReplicas: 1
      maxReplicas: 3
      targetCPUUtilizationPercentage: 80

Then run:

kubectl logs -n vmlog vlinsert-cluster-1-...

2026-04-29T00:23:30.179Z        info    VictoriaLogs/app/victoria-logs/main.go:43       starting VictoriaLogs at "[:9481]"...
2026-04-29T00:23:30.179Z        info    VictoriaLogs/app/vlstorage/main.go:145  opening storage at -storageDataPath=victoria-logs-data

You can also query the vlselect pods, and those will have no logs, and if you query the insert logs, they will contain the logs

[L1ghtman2k]

I was thinking that it could be caused by HPA in vlstorage, but removing that also seems to make no difference

[AndrewChubatiuk]

right, this case caused by HPA, have you tried to disable it and set storage replicaCount to at least 1. Also you can try this image, which contains a fix from a linked PR
docker.io/victoriametrics/operator:heads-vl-hpa-fix-0-g8f1c9378-dirty-a8b6cb81

[L1ghtman2k]

apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
  name: cluster-1
  namespace: vmlog
spec:
  vlinsert:
    hpa:
      enabled: true
      minReplicas: 1
      maxReplicas: 3
      targetCPUUtilizationPercentage: 80
  vlselect:
    hpa:
      enabled: true
      minReplicas: 1
      maxReplicas: 3
      targetCPUUtilizationPercentage: 80
  vlstorage:
    retentionPeriod: "30d"
    storage:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 50Gi
    replicas: 3

Still causes the issue, vlinsert nodes point at storageDataPath:

2026-04-29T19:29:21.488Z        info    VictoriaMetrics@v1.140.1-0.20260414051809-8a20ccf21db7/lib/logger/flag.go:12    build version: victoria-logs-20260414-184331-tags-v1.50.0-0-g80d223f95f
2026-04-29T19:29:21.488Z        info    VictoriaMetrics@v1.140.1-0.20260414051809-8a20ccf21db7/lib/logger/flag.go:13    command-line flags
2026-04-29T19:29:21.488Z        info    VictoriaMetrics@v1.140.1-0.20260414051809-8a20ccf21db7/lib/logger/flag.go:20      -httpListenAddr=":9481"
2026-04-29T19:29:21.488Z        info    VictoriaMetrics@v1.140.1-0.20260414051809-8a20ccf21db7/lib/logger/flag.go:20      -internalselect.disable="true"
2026-04-29T19:29:21.488Z        info    VictoriaLogs/app/victoria-logs/main.go:43       starting VictoriaLogs at "[:9481]"...
2026-04-29T19:29:21.488Z        info    VictoriaLogs/app/vlstorage/main.go:145  opening storage at -storageDataPath=victoria-logs-data

Also, for my own understanding, what is the behavior of when a new storage node is added under the HPA? Is there a disdovery of some sort, or does the operator restart vlselect, and vlinsert nodes? What happens when a storage node is decommissioned, is HPA a correct approach here even if it operates as expected?

[AndrewChubatiuk]

there's no replicas property in vmstorage, use replicaCount instead

what is the behavior of when a new storage node is added under the HPA?

your understanding is correct, vmselect and vminsert will be restarted and discovery is not expected to be supported

[L1ghtman2k]

Ah, thank you... I don't know where i got replicas from, this worked.

If I understand the architecture correctly, we may not want to scale down vlstorage without significant precautions because scaling down vlstorage will remove the data, and it wouldn't be replicated across remaining vlstorage nodes? Is there a recommended pattern for using HPAs with vlstorage?

[AndrewChubatiuk]

right, scaling nodes down leads to data loss. in general HPA for storage nodes is not recommended and usually users calculate cluster capacity taking in account ingestion rate, churn rate, retention period. but if storage is constantly growing HPA can be useful to scale it up. In case if it's needed to scale down VL/VM/VTStorage it have maintenanceInsertNodeIDs parameter, which excludes storage nodes from insert and then pods can be removed after retention period, but it should be done manually