feat: Add Azure Container Apps Job GPU Infrastructure (#14)

* feat: add CUDA/gsplat environment check script

Add scripts/gsplat_check — a lightweight Python tool (managed by uv) that
verifies whether the current device can run the gsplat 3DGS training backend.

Checks performed:
- CUDA GPU detection via nvidia-smi + PyTorch tensor smoke-test
- gsplat library import and rasterization kernel validation (8 Gaussians)
- External tool availability (nvidia-smi, python3, ffmpeg, colmap)

Reports a structured pass/fail verdict similar to the Rust preflight binary.

Usage: cd scripts/gsplat_check && uv run main.py

Also adds a reference to the new tool in the root README documentation section.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* feat: add Azure Container App Job infrastructure for GPU 3DGS processing

Add complete azd-based infrastructure for running the 3DGS video processor
as a GPU Container App Job on Azure:

Infrastructure (infra/):
- Bicep modules for ACR, Storage, Container Apps Environment (GPU T4 profile),
  Container Apps Job, Managed Identity, Log Analytics, and RBAC
- Standalone RBAC deployment (infra/rbac/) for privileged user separation
- Parameter bindings for azd environment variables

Scripts (infra/scripts/):
- assign-rbac.sh / verify-rbac.sh / cleanup-rbac.sh for RBAC management
- hooks/acr-build.sh: builds GPU image via ACR Tasks with minimal staging dir
- hooks/preprovision.sh: captures deployer identity, RBAC preflight
- hooks/postprovision.sh: builds image and updates job post-provision
- run-job.sh: start job with --wait/--logs support
- deploy-job.sh: rebuild and redeploy image
- upload-testdata.sh: upload South Building test videos to blob storage

Bug fixes required for GPU batch mode:
- src/azure/sdk.rs: pass AZURE_CLIENT_ID to ManagedIdentityCredential for
  user-assigned managed identity support in Container Apps
- src/backends/gsplat.rs: fix COLMAP sparse dir and images dir resolution
  for batch mode (TEMP_PATH-based layout vs workspace-relative)
- src/backends/gsplat.rs: add inline PLY-to-SPLAT converter fallback when
  external converter tools are unavailable
- scripts/gsplat_train.py: fix cameras.bin parser to read correct number of
  parameters per camera model instead of reading to EOF
- .dockerignore: exclude .venv dirs, output/, infra/ from build context;
  keep Dockerfile for ACR Tasks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: disable ANSI color codes in container log output

Use std::io::IsTerminal to detect non-TTY environments (containers,
log aggregators) and disable ANSI escape sequences. This ensures
clean text output in Azure Container Apps Log Analytics and other
log collection systems.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: add Azure Container Apps Job GPU deployment guide

Add comprehensive section to DEPLOYMENT.md covering:
- Quick start steps for azd-based GPU deployment
- What resources get provisioned (table of Bicep modules)
- Detailed RBAC requirements with specific permissions, role IDs,
  and clear guidance on what fails without each role
- Deployer vs Managed Identity permission separation
- Test data upload and job execution instructions
- Configuration variables and GPU region availability
- Scripts reference with privilege requirements
- Troubleshooting guide for common failures

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: add changelog for Azure Container Apps GPU infrastructure

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: enable BuildKit inline cache for ACR builds

Add BUILDKIT_INLINE_CACHE=1 build arg to embed cache metadata in
pushed images. While ACR Tasks don't persist Docker layer cache
between runs (each gets a fresh VM), the inline cache metadata
enables faster rebuilds when using BuildKit-aware builders.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: address PR #14 review feedback (7 items)

1. gsplat_train.py: Fix COLMAP camera model parameter counts
   (OPENCV_FISHEYE=8, FULL_OPENCV=12, FOV=5, THIN_PRISM_FISHEYE=12).
   Fail fast on unknown model_id instead of defaulting to 4.

2. gsplat.rs: Fix PLY-to-SPLAT converter fallback order — try
   configured converter binary before inline Python fallback.

3. gsplat.rs: Remove unused numpy import from inline Python script;
   keep it stdlib-only (struct + sys) for maximum portability.

4. run-job.sh: Apply BATCH_INPUT_PREFIX to job env vars via
   az containerapp job update before starting execution.

5. storage.bicep: Default allowSharedKeyAccess to false; only enable
   when useStorageKeys=true (reduces blast radius of leaked keys).

6. main.bicep + storage.bicep: Wire storageConnectionString from
   storage module to job module when useStorageKeys=true, preventing
   empty-secret deployment failures.

7. sdk.rs: Filter empty AZURE_CLIENT_ID strings before constructing
   ManagedIdentityCredentialOptions to prevent confusing auth errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: add local Docker build+push for fast incremental rebuilds

Add local-build.sh as alternative to acr-build.sh for development:
- Uses Docker BuildKit with persistent layer cache
- Incremental rebuilds (src/ change only) take ~2.5 min vs ~35 min on ACR
- Push uses delta layers — only changed layers uploaded (~12s vs full push)
- deploy-job.sh supports --local flag to select build method

Build comparison (src/ change only):
  ACR Tasks:  ~35 min (no cache between runs, fresh VM each time)
  Local+push: ~2.5 min build + ~12s push = ~3 min total

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Author: 3 people

.dockerignore

@@ -50,7 +50,6 @@ flycheck*/
 
 # Docker files (don't need these in the build context)
 .dockerignore
-Dockerfile
 docker-compose*.yml
 
 # Scripts not needed in container
@@ -59,6 +58,8 @@ scripts/download-*.sh
 scripts/generate-*.sh
 scripts/e2e-*.sh
 scripts/run-*.sh
+scripts/gsplat_check/
+scripts/e2e/
 
 # Python cache
 __pycache__/
@@ -67,6 +68,19 @@ __pycache__/
 *.pyd
 .Python
 
+# Python virtual environments
+.venv/
+venv/
+**/.venv/
+**/venv/
+.e2e-venv/
+
+# Output and infra directories
+output/
+infra/
+.azure/
+container-test/
+
 # IDE and workspace files
 .copilot/
 *.workspace

.github/instructions/docker-batch-e2e.instructions.md

@@ -194,9 +194,6 @@ Mount the project's `container-test/config.yaml` (a **file**, not a directory):
 -v /absolute/path/to/container-test/config.yaml:/config/config.yaml:ro
 ```
 
-⚠️ **Pitfall:** `container-test/config.1.yaml` is a directory (created by a prior bad mount),
-not a file. Always use `container-test/config.yaml`.
-
 ## Batch mode vs Watch mode (in Docker)
 
 | Aspect | Batch Mode | Watch Mode |

.gitignore

@@ -36,3 +36,6 @@ output
 # better test files from COLMAP
 testdata/south**
 .e2e-venv
+
+# Azure Developer CLI
+.azure/

README.md

@@ -436,6 +436,7 @@ See [docs/CONFIGURATION.md](docs/CONFIGURATION.md) for the complete configuratio
 * [Deployment](docs/DEPLOYMENT.md) - Production deployment patterns and best practices
 * [Troubleshooting](docs/TROUBLESHOOTING.md) - Common issues and solutions
 * [PRD](docs/3dgs-video-processor-prd.md) - Product requirements specification
+* [gsplat Environment Check](scripts/gsplat_check/) - Python script to verify CUDA + gsplat functionality on a device
 
 ## Requirements
 

azure.yaml

@@ -0,0 +1,11 @@
+name: 3dgs-processor
+metadata:
+  template: 3dgs-processor@0.0.1
+hooks:
+  preprovision:
+    shell: sh
+    run: ./infra/scripts/hooks/preprovision.sh
+    continueOnError: true
+  postprovision:
+    shell: sh
+    run: ./infra/scripts/hooks/postprovision.sh

compose.yaml

@@ -17,7 +17,7 @@ services:
     restart: unless-stopped
     volumes:
       - ./output/data:/data
-      - ./container-test/config.1.yaml:/app/config.yaml:ro
+      - ./container-test/config.yaml:/app/config.yaml:ro
       - ./output/tmp:/tmp/3dgs-work
     environment:
       CONFIG_PATH: /app/config.yaml
@@ -56,7 +56,7 @@ services:
               capabilities: [gpu]
     volumes:
       - ./output/data:/data
-      - ./container-test/config.1.yaml:/app/config.yaml:ro
+      - ./container-test/config.yaml:/app/config.yaml:ro
       - ./output/tmp:/tmp/3dgs-work
     environment:
       CONFIG_PATH: /app/config.yaml

docs/DEPLOYMENT.md

@@ -5,6 +5,7 @@ Production deployment patterns and operational best practices.
 ## Table of Contents
 
 * [Container Deployment](#container-deployment)
+* [Azure Container Apps Job (GPU) — azd](#azure-container-apps-job-gpu--azd)
 * [Batch Mode (Azure SDK)](#batch-mode-azure-sdk)
 * [Resource Requirements](#resource-requirements)
 * [Storage Configuration](#storage-configuration)
@@ -308,6 +309,296 @@ az container delete --resource-group $RESOURCE_GROUP --name $CONTAINER_NAME
 - Set `--restart-policy OnFailure` for one-time jobs
 - Use spot instances for non-critical workloads
 
+---
+
+## Azure Container Apps Job (GPU) — azd
+
+Deploy the 3DGS processor as a **serverless GPU job** on Azure Container Apps using the
+Azure Developer CLI (`azd`). The job runs in batch mode: download videos from Azure Blob
+Storage → extract frames → COLMAP reconstruction → gsplat GPU training → export PLY/SPLAT
+→ upload results → exit.
+
+**Key characteristics:**
+- **Serverless GPU** — NVIDIA T4 (16 GB VRAM) via `Consumption-GPU-NC8as-T4` workload profile
+- **No local Docker required** — images are built remotely via ACR Tasks
+- **Batch mode** — single job execution, no long-running container
+- **Managed Identity** — user-assigned MI for RBAC-based access to ACR and Storage
+- **RBAC separation** — infrastructure provisioning and role assignments are separate steps
+
+### Prerequisites
+
+| Requirement | Purpose |
+|-------------|---------|
+| [Azure CLI](https://aka.ms/installazurecli) (`az`) | Azure resource management |
+| [Azure Developer CLI](https://aka.ms/install-azd) (`azd`) | Infrastructure-as-code orchestration |
+| Azure subscription | With GPU quota in your target region |
+| Test data | Download via `./scripts/e2e/01-download-testdata.sh` |
+
+**No local Docker daemon is needed.** The GPU image is built remotely on Azure Container
+Registry Tasks.
+
+### Quick Start
+
+```bash
+# 1. Initialize the azd environment
+azd init
+
+# 2. Configure environment
+azd env set AZURE_LOCATION swedencentral
+azd env set USE_GPU true
+
+# 3. Provision infrastructure (also builds the GPU image on ACR — ~40 min)
+azd provision
+
+# 4. (Privileged user) Assign RBAC roles to the Managed Identity
+./infra/scripts/assign-rbac.sh
+
+# 5. Verify RBAC assignments
+./infra/scripts/verify-rbac.sh
+
+# 6. Upload test data to Azure Blob Storage
+./infra/scripts/upload-testdata.sh
+
+# 7. Run the GPU job
+./infra/scripts/run-job.sh --logs
+```
+
+### What Gets Provisioned
+
+`azd provision` creates the following Azure resources (all in a single resource group):
+
+| Resource | Bicep Module | Purpose |
+|----------|-------------|---------|
+| Resource Group | `main.bicep` | `rg-<env-name>` |
+| User-Assigned Managed Identity | `modules/managed-identity.bicep` | Authenticate to ACR and Storage |
+| Azure Container Registry (Basic) | `modules/acr.bicep` | Store GPU Docker images |
+| Storage Account + 4 containers | `modules/storage.bicep` | `input`, `output`, `processed`, `error` |
+| Log Analytics Workspace | `modules/monitoring.bicep` | Container log aggregation |
+| Container Apps Environment | `modules/container-apps-env.bicep` | GPU workload profile (T4) |
+| Container Apps Job (Manual trigger) | `modules/container-apps-job.bicep` | The processor job itself |
+
+After provisioning, the `postprovision` hook automatically:
+1. Builds the GPU Docker image via ACR Tasks (`infra/scripts/hooks/acr-build.sh`)
+2. Updates the Container Apps Job with the new image
+
+### RBAC Requirements — Read This Carefully
+
+RBAC role assignments are **intentionally separated** from infrastructure provisioning.
+This is because in many organizations, the person deploying infrastructure does not have
+permission to assign IAM roles. The two operations may require different privilege levels.
+
+#### Permissions Required by the Deployer (the person running `azd provision`)
+
+The deployer needs these permissions **on the Azure subscription or resource group**:
+
+| Permission | Why |
+|------------|-----|
+| `Microsoft.Resources/subscriptions/resourceGroups/write` | Create the resource group |
+| `Microsoft.ContainerRegistry/registries/*` | Create ACR and push images |
+| `Microsoft.Storage/storageAccounts/*` | Create storage account and containers |
+| `Microsoft.App/managedEnvironments/*` | Create Container Apps Environment |
+| `Microsoft.App/jobs/*` | Create Container Apps Job |
+| `Microsoft.ManagedIdentity/userAssignedIdentities/*` | Create managed identity |
+| `Microsoft.OperationalInsights/workspaces/*` | Create Log Analytics workspace |
+
+Typically the **Contributor** built-in role on the subscription is sufficient.
+
+The `preprovision` hook (`infra/scripts/hooks/preprovision.sh`) also attempts to assign
+deployer-level RBAC (AcrPush, Storage Blob Data Contributor) so the deployer can push
+images and upload test data. This requires the deployer to have **User Access Administrator**
+or **Owner** on the resource group. If the deployer lacks this permission, the hook
+continues (it's non-blocking) — a privileged user can assign these roles later.
+
+#### Permissions Required for the Managed Identity (assigned after provisioning)
+
+The Managed Identity needs two role assignments so the container job can pull images
+and read/write blobs at runtime:
+
+| Role | Scope | Role Definition ID | Purpose |
+|------|-------|-------------------|---------|
+| **AcrPull** | Container Registry | `7f951dda-4ed3-4680-a7ca-43fe172d538d` | Pull GPU image from ACR |
+| **Storage Blob Data Contributor** | Storage Account | `ba92f5b4-2d11-453d-a403-e96b0029c9fe` | Download input videos, upload outputs, move blobs |
+
+#### Who Can Assign These Roles?
+
+A user with **one** of these roles on the target scope:
+- **Owner** (full control, including RBAC)
+- **User Access Administrator** (RBAC management only)
+- A custom role with `Microsoft.Authorization/roleAssignments/write` permission
+
+> **If the deployer does not have these permissions**, the `azd provision` step will
+> succeed but the job will fail at runtime with authentication errors (HTTP 403 on
+> Storage or image pull failures on ACR). Have a privileged user run the RBAC scripts.
+
+#### Assigning RBAC Roles
+
+```bash
+# Assign roles via Azure CLI (reads values from azd env automatically)
+./infra/scripts/assign-rbac.sh
+
+# Or assign via Bicep deployment (alternative)
+./infra/scripts/assign-rbac.sh --use-bicep
+```
+
+#### Verifying RBAC Roles
+
+```bash
+# Check that both roles are assigned
+./infra/scripts/verify-rbac.sh
+```
+
+Expected output when roles are correctly assigned:
+```
+🔍 Verifying RBAC role assignments for Managed Identity...
+   Principal ID: <managed-identity-principal-id>
+
+  ✅ AcrPull on Container Registry
+  ✅ Storage Blob Data Contributor on Storage Account
+
+✅ All RBAC role assignments are in place.
+```
+
+If any roles are missing:
+```
+  ❌ AcrPull on Container Registry — MISSING
+
+⚠️  1 RBAC role assignment(s) missing.
+   Run './infra/scripts/assign-rbac.sh' as a privileged user to fix.
+```
+
+#### What Fails Without RBAC
+
+| Missing Role | Symptom |
+|--------------|---------|
+| **AcrPull** | Job execution fails immediately — Container Apps cannot pull the image. The execution shows `Failed` status with no container logs (image never starts). |
+| **Storage Blob Data Contributor** | Container starts but fails with `Failed to download input blobs from Azure Blob Storage` — the managed identity token is rejected with HTTP 403. |
+
+### Uploading Test Data
+
+The South Building dataset (128 multi-view images from UNC Chapel Hill) is used for testing.
+The upload script downloads it if needed, creates 3 test videos, and uploads them:
+
+```bash
+# Download test data + upload to blob storage (default prefix: south_building/)
+./infra/scripts/upload-testdata.sh
+
+# Custom blob prefix
+./infra/scripts/upload-testdata.sh --prefix my_scene/
+
+# Download only (no upload)
+./infra/scripts/upload-testdata.sh --download-only
+```
+
+### Running the Job
+
+```bash
+# Start job and return immediately
+./infra/scripts/run-job.sh
+
+# Start job, wait for completion, show status
+./infra/scripts/run-job.sh --wait
+
+# Start job, wait, then show container logs
+./infra/scripts/run-job.sh --logs
+```
+
+A successful run produces these outputs in the `output` blob container:
+
+| File | Description | Typical Size |
+|------|-------------|-------------|
+| `south_building/south_building.ply` | 3D Gaussian point cloud (real GPU-trained geometry) | ~65 KB |
+| `south_building/south_building.splat` | Web-optimized format for real-time rendering | ~53 KB |
+| `south_building/manifest.json` | Video metadata (resolution, duration, codec, frame count) | ~7 KB |
+| `south_building/.checkpoint.json` | Pipeline progress tracking | ~11 KB |
+
+Input videos are moved from `input/` to `processed/` on success, or `error/` on failure.
+
+### Redeploying Code Changes
+
+After modifying the processor code, rebuild and redeploy without re-provisioning:
+
+```bash
+# Build new image on ACR + update the job
+./infra/scripts/deploy-job.sh
+
+# Or skip the build and just redeploy the existing image
+./infra/scripts/deploy-job.sh --skip-build
+```
+
+### Configuration Variables
+
+Set these via `azd env set <NAME> <VALUE>` before provisioning:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `AZURE_LOCATION` | *(required)* | Azure region. Must support GPU T4 (e.g., `swedencentral`, `eastus`, `westus`) |
+| `USE_GPU` | `true` | Enable GPU workload profile |
+| `GPU_PROFILE_TYPE` | `Consumption-GPU-NC8as-T4` | GPU type (`Consumption-GPU-NC8as-T4` or `Consumption-GPU-NC24-A100`) |
+| `PROCESSOR_BACKEND` | `gsplat` | 3DGS backend (`gsplat`, `gaussian-splatting`, `mock`) |
+| `INCLUDE_RBAC` | `true` | Include RBAC assignments in Bicep deployment |
+| `USE_STORAGE_KEYS` | `false` | Use storage account keys instead of RBAC (fallback) |
+
+### GPU Region Availability
+
+Serverless GPU (T4) Container Apps is available in these regions:
+
+`swedencentral`, `eastus`, `westus`, `canadacentral`, `brazilsouth`, `australiaeast`,
+`italynorth`, `francecentral`, `centralindia`, `japaneast`, `northcentralus`,
+`southcentralus`, `southeastasia`, `southindia`, `westeurope`, `westus2`, `westus3`
+
+If you get a deployment error about workload profiles, ensure your subscription has
+GPU quota in the selected region. Check quota at:
+[Azure Portal → Quotas](https://portal.azure.com/#view/Microsoft_Azure_Capacity/QuotaMenuBlade)
+
+### Cleaning Up RBAC Before Teardown
+
+Before running `azd down`, remove the RBAC assignments:
+
+```bash
+./infra/scripts/cleanup-rbac.sh
+azd down --purge --force
+```
+
+### Scripts Reference
+
+All infrastructure scripts are in `infra/scripts/`:
+
+| Script | Purpose | Requires Privilege? |
+|--------|---------|-------------------|
+| `hooks/preprovision.sh` | Captures deployer identity, runs RBAC preflight check | No (auto-run by azd) |
+| `hooks/postprovision.sh` | Builds GPU image on ACR, updates job | No (auto-run by azd) |
+| `hooks/acr-build.sh` | Creates minimal staging dir, runs `az acr build` for GPU target | No |
+| `assign-rbac.sh` | Assigns AcrPull + Storage Blob Data Contributor to MI | **Yes** — Owner or User Access Admin |
+| `verify-rbac.sh` | Checks if required RBAC roles are assigned | No |
+| `cleanup-rbac.sh` | Removes RBAC role assignments | **Yes** — Owner or User Access Admin |
+| `run-job.sh` | Starts a job execution with `--wait`/`--logs` options | No |
+| `deploy-job.sh` | Rebuilds image on ACR + updates job | No |
+| `upload-testdata.sh` | Downloads South Building dataset + uploads videos to blob storage | No (needs Storage Blob Data Contributor on deployer) |
+
+### Troubleshooting
+
+**"MANAGED_IDENTITY_PRINCIPAL_ID is not set"**
+Run `azd provision` first — this creates the managed identity and saves its principal ID.
+
+**Image pull failure (no container logs)**
+The AcrPull role is missing. Run `./infra/scripts/assign-rbac.sh`.
+
+**"Failed to download input blobs from Azure Blob Storage"**
+Either: (a) Storage Blob Data Contributor is missing — run `./infra/scripts/assign-rbac.sh`,
+or (b) no blobs exist at the `BATCH_INPUT_PREFIX` — run `./infra/scripts/upload-testdata.sh`.
+
+**"Reconstruction quality too low"**
+Increase `FRAME_RATE` (e.g., from 2 to 3) in the job env vars to extract more frames.
+
+**ACR build timeout**
+The GPU image build compiles COLMAP from source (~30 min). The default timeout is 3600s.
+If it still times out, check ACR Tasks quotas.
+
+**COLMAP matching timeout**
+Ensure `COLMAP_MATCHER=sequential` (not `exhaustive`) in the job env vars.
+
+---
+
 ## Batch Mode (Azure SDK)
 
 Batch mode processes a single job using the Azure Blob Storage SDK directly — no BlobFuse2