vector: poll for worker.env inside populator instead of systemd retry

[motatoes]

Summary

#254 (just merged) made the populator `exit 1` when `/etc/opensandbox/{worker,server}.env` was missing, expecting systemd's `Restart=on-failure` to retry until cloud-init lands the file. It hit a follow-up bug on prod that the in-place dev test in #254 didn't catch.

What broke on prod

Observed on `osb-worker-c0741893` (rotated to the post-#254 AMI):

```
vector.service: failed
populate-vector-env.service: failed
worker.env mtime: 00:44:00 (cloud-init wrote it)
populator journal:
00:43:08 populator: Start request repeated too quickly
00:43:08 populator: Failed with result 'exit-code'
00:43:09 populator: Start request repeated too quickly
00:43:09 populator: Failed
... 5 more in <2 seconds
```

Mechanism:

1. `vector.service` has `Restart=always`.
2. Each Vector restart re-requests `populate-vector-env.service` (Wants= dep).
3. systemd counts those re-requests against the populator's `StartLimitBurst=5` / `IntervalSec=120`.
4. Vector restarts faster than `RestartSec=10s` can pace the populator's own retries — burst exhausts in 2 seconds.
5. Populator enters `failed`. Vector also `failed`. Both stuck until manual intervention.
6. ~50 seconds later, cloud-init writes worker.env. Too late — both units are dead.

Fix

Poll inside the script instead of relying on systemd retry. Single invocation, internal wait up to 90s. No restart-budget interaction:

```bash
DEADLINE=$(($(date +%s) + 90))
while [ $(date +%s) -lt $DEADLINE ]; do
[ -f /etc/opensandbox/worker.env ] || [ -f /etc/opensandbox/server.env ] && break
log "waiting for cloud-init to write env file..."
sleep 5
done
[ -f /etc/opensandbox/worker.env ] && . /etc/opensandbox/worker.env
[ -f /etc/opensandbox/server.env ] && . /etc/opensandbox/server.env
VAULT_NAME="${OPENSANDBOX_AZURE_KEY_VAULT_NAME:-}"
```

Behaves identically on dev (env file already exists → first iteration breaks out, no wait).

Why #254 dev test didn't catch this

The dev test in #254 confirmed that the systemd ordering cycle was gone. It did NOT exercise the cloud-init-delay path because:

• Dev cluster's `bootstrap.sh` writes `/etc/opensandbox/worker.env` BEFORE Vector's install step.
• So at boot on dev, `worker.env` always exists when the populator runs — the script's "missing env file" branch is never hit.
• The cycle test happened to pass because it didn't depend on retry timing; the retry-burst test would have needed an artificial cloud-init delay (e.g. `systemd-run --on-active=60s touch /etc/opensandbox/worker.env` before reboot) to reproduce.

Adding a runbook for that simulation would close this gap going forward.

Test plan

• Apply on dev — `worker.env` exists, populator should run once and succeed immediately (no wait).
• Simulate cloud-init delay on dev: `rm /etc/opensandbox/worker.env` + `systemd-run --on-active=45 sh -c 'echo OPENSANDBOX_AZURE_KEY_VAULT_NAME=opencomputer-dev-kv > /etc/opensandbox/worker.env'` + reboot. Confirm populator waits, succeeds at ~45s, vector starts.
• Roll to prod via AMI rebake + worker rotation. Confirm `vector.service: active` on freshly-booted prod workers.

🤖 Generated with Claude Code

[breardon2011]

Approve