ENG-3517: Upload and promotion for user uploaded files - schema variant, storage util, hooks, config

[mikeGarifullin]

Ticket ENG-3517

Stack: depends on #8110 (data-layer foundation). Merge that first.

Description Of Changes

Schema + storage + service-extension half of the upload foundation. Pairs with the data-layer PR (#8110); the upload service and endpoint live in fidesplus and consume both.

Adds the Privacy Center config variant for file fields, a magic-byte sniff catalog so the upload route can verify a payload's claimed type at byte-zero (client-supplied Content-Type is not trusted), two config knobs for the global upload ceilings, and two PrivacyRequestService hook points so fidesplus can plug attachment-resolve and attachment-promote into the existing submission flow without overriding create_privacy_request wholesale.

Code Changes

• schemas/privacy_center_config.py: FileUploadCustomPrivacyRequestField discriminated-union variant (field_type="file", max_size_bytes > 0, allowed_file_types non-empty + must be a subset of AllowedFileType).
• schemas/attachment.py: AttachmentUploadResponse (the {id: "att_..."} payload returned by the fidesplus upload route).
• service/storage/util.py:
  • FilesMagicBytes — known signatures (pdf, jpg/jpeg, png, gif, ...) and from_bytes(...) for byte-zero detection.
  • AllowedFileType.mime_types(), MIME_TO_EXTENSION, extension_for_mime(...) helpers.
  • FileUploadConstraints — runtime-validated config bag (per-field cap + allowed types) reused by the upload route and the schema validator.
• service/privacy_request/privacy_request_service.py:
  • _resolve_attachment_state(submitted_data, *, db) and _promote_attachment_state(privacy_request, state) — overridable no-op hooks on the OSS service. Default impls do nothing so OSS behavior is unchanged.
  • create_privacy_request calls _resolve_attachment_state before persisting and _promote_attachment_state after caching/masking-secrets. On promotion failure the request row is deleted (clear_cached_values() runs as part of delete()) and the original error is rewrapped as PrivacyRequestError.
• common/urn_registry.py: route constant for the future fidesplus upload endpoint.
• config/security_settings.py: request_attachment_max_bytes global ceiling (caps the per-field max_size_bytes).
• config/execution_settings.py: attachment_orphan_ttl_seconds for the orphan-cleanup sweep.
• Tests for the schema variant, magic-byte catalog, and FileUploadConstraints.

Steps to Confirm

1. Stack rebased onto ENG-3517: Foundation - attachment_user_provided model + repository #8110. From this branch run nox -s "pytest(ops-unit-non-api)" -- tests/ops/schemas/test_attachment.py tests/ops/schemas/test_privacy_center_config.py tests/ops/service/storage/test_util.py — all green.
2. Construct a FileUploadCustomPrivacyRequestField with max_size_bytes=0 or allowed_file_types=[] — pydantic should reject. Construct one with an extension outside AllowedFileType — pydantic should reject.
3. FilesMagicBytes.from_bytes(open("sample.pdf","rb").read(8)) returns the pdf member. The same call against random bytes returns None.
4. From a Python shell: instantiate PrivacyRequestService and call create_privacy_request(...) with a non-file payload — flow runs unchanged (hooks are no-ops).
5. (Cross-PR) The fidesplus upload route consumes FileUploadConstraints, FilesMagicBytes, and the new hook points — verify in the companion fidesplus PR.

Pre-Merge Checklist

• Issue requirements met
• All CI pipelines succeeded
• CHANGELOG.md updated
  • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
  • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
  • Updates unreleased work already in Changelog, no new entry necessary
• UX feedback:
  • All UX related changes have been reviewed by a designer
  • No UX review needed
• Followup issues:
  • Followup issues created
  • No followup issues
• Database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!
  • No migrations
• Documentation:
  • Documentation complete, PR opened in fidesdocs
  • Documentation issue created in fidesdocs
  • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • No documentation updates required

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
fides-plus-nightly	Ignored	Preview	May 7, 2026 2:52pm
fides-privacy-center	Ignored		May 7, 2026 2:52pm

[codecov]

Codecov Report

❌ Patch coverage is 92.91339% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.25%. Comparing base (7655326) to head (ab007a5).

Files with missing lines	Patch %	Lines
...service/privacy_request/privacy_request_service.py	50.00%	9 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff               @@
##           ENG-3517-2    #8113      +/-   ##
==============================================
- Coverage       85.26%   85.25%   -0.01%     
==============================================
  Files             641      642       +1     
  Lines           42091    42213     +122     
  Branches         4941     4948       +7     
==============================================
+ Hits            35888    35990     +102     
- Misses           5096     5114      +18     
- Partials         1107     1109       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.