Add support for RAUC file parsing by b-rowan · Pull Request #132 · UpstreamDataInc/goosebit

b-rowan · 2024-09-10T18:17:11Z

Allow parsing of RAUC files using PySquashFsImage.

Closes #131

+
+async def parse_remote(url: str):
+    async with httpx.AsyncClient() as c:
+        file = await c.get(url)


To fix this SSRF vulnerability, you should restrict the set of URLs that can be accessed via the url parameter. The best way is to maintain a whitelist of allowed domains or base URLs, and only allow requests to those. If the set of allowed domains is small and known (e.g., trusted update servers), you can check that the parsed domain of the user-provided URL matches one of the allowed domains before making the request. If you must allow arbitrary domains, you should at least block requests to private IP ranges and sensitive endpoints (such as localhost, 127.0.0.1, 169.254.169.254, etc.).

The changes should be made in goosebit/updates/swdesc/__init__.py, specifically in the parse_remote function. You will need to:

Parse the URL and check that its hostname is in a whitelist (e.g., a list of allowed domains).

If not, raise an exception and do not make the request.

Add any necessary imports for URL parsing (e.g., urllib.parse).

You should also add a whitelist definition at the top of the file, or as a constant, and use it in the validation.

tsagadar

Not sure if it is a bit too aggressive to already add RAUC support. It is not only about the file format but also about being able to document / test its usage. I'd expect minor differences in comparison with SWUpdate.

Some more open things

Adding db migrations (as discussed in #133)
Adjusting documentation
Exposing image type in UI? Currently the type is stored but never used
Squash the "[pre-commit.ci]" commit? Or is the idea not to run the code formatting tools and have a lot of those commits on main?

easybe · 2024-09-13T06:29:33Z

I think it is a good idea to support RAUC as long as we don’t have to completely refactor the application at this point. IMHO we should focus on making what we have production-ready and then go from there.

b-rowan · 2024-09-13T15:42:40Z

I think it is a good idea to support RAUC as long as we don’t have to completely refactor the application at this point. IMHO we should focus on making what we have production-ready and then go from there.

I agree here. This was relatively easy to throw together, so if we don't want to add this right now I will leave as a draft for reference.

tsagadar · 2024-10-26T14:30:23Z

Another thing I realized today: goosebit does log message parsing (to detect download progress / update status) - this also would require adjustments. However, the parsing ideally gets reduced to only extract the progress.

Tristoris · 2025-08-17T13:21:03Z

Hello!
I have read through the changes and issues and also used today's master version and cherry-picked the two commits and rauc seemed to work with goosebit (I did have to resolve a few conflicts though). Are there any issues with this pull request so that the changes are not on master yet? Is something missing to apply the changes/ has something else to be done before closing this?

b-rowan · 2025-08-19T15:14:38Z

Hello! I have read through the changes and issues and also used today's master version and cherry-picked the two commits and rauc seemed to work with goosebit (I did have to resolve a few conflicts though). Are there any issues with this pull request so that the changes are not on master yet? Is something missing to apply the changes/ has something else to be done before closing this?

Not entirely sure. Our main goal was to ensure that gooseBit as a whole is production ready before we attempt to add complexity with something like this. If you are able to provide log examples that I can work on parsing with this we likely could add it, but I think there also needs to be consideration made as to how we track which files are which types, and which devices want what type of updates.

Tristoris · 2025-08-19T16:41:28Z

Hello! I have read through the changes and issues and also used today's master version and cherry-picked the two commits and rauc seemed to work with goosebit (I did have to resolve a few conflicts though). Are there any issues with this pull request so that the changes are not on master yet? Is something missing to apply the changes/ has something else to be done before closing this?

Not entirely sure. Our main goal was to ensure that gooseBit as a whole is production ready before we attempt to add complexity with something like this. If you are able to provide log examples that I can work on parsing with this we likely could add it, but I think there also needs to be consideration made as to how we track which files are which types, and which devices want what type of updates.

I understand. However isn't it the job of the update manager to know what files are supported by what devices? Since goosebit is really just a way to bring that file to the device / decide what devices will receive what updates. The only time it has to do something with the file type is when it tries to read the header for information about the uploaded update, nothing more; which was accounted for in those two commits here.

b-rowan · 2025-08-19T18:11:49Z

However isn't it the job of the update manager to know what files are supported by what devices?

I think partially, yes. I worry about edge cases, like what if a user has 2 devices, one which supports RAUC and one which supports SWUpdate, and the user uploads 2 files with the same versioning? It's a good idea to be at least partially aware of this information, if we can save the user from having the device download invalid updates I think that is a good idea.

Curious what others think here though, I have no opposition to merging this as-is, but we should deliberate.

easybe · 2025-08-19T18:27:33Z

I think it is fine to push this forward. I have started to whip up a device Docker image for RAUC, so we could even have E2E tests.

easybe · 2025-09-04T08:09:34Z

Another thing I realized today: goosebit does log message parsing (to detect download progress / update status) - this also would require adjustments. However, the parsing ideally gets reduced to only extract the progress.

With my Docker-based demo setup this seems to work out of the box. I see the following:

Download complete. 0.40 MB/s
File checksum OK.
installing
  0% Installing
  0% Determining slot states
 10% Determining slot states done.
 10% Checking bundle
 10% Verifying signature
 20% Verifying signature done.
 20% Checking bundle done.
 20% Checking manifest contents
 30% Checking manifest contents done.
 30% Determining target install group
 40% Determining target install group done.
 40% Updating slots
 40% Checking slot rootfs.1 (rootfs1)
 46% Checking slot rootfs.1 (rootfs1) done.
 46% Copying image to rootfs.1
 99% Copying image to rootfs.1 done.
 99% Updating slots done.
100% Installing done.
idle
Software bundle installed successfully.

Here the container's log output during an update:

rauc-update.log

easybe · 2025-09-04T08:13:19Z

If you are happy with my changes, feel free to squash all 3 commits into 1.

easybe · 2025-09-04T08:15:57Z

The RAUC docs use version=2016.08-1 as an example. Perhaps we need to add support for that format as well. However, that can be addressed later.

b-rowan · 2025-09-04T13:19:28Z

This all looks good to me, we can add log parsing at a later time. I won't be able to do anything on this until next week, as I am away this week, but this seems good to merge.

easybe · 2025-09-04T14:04:04Z

This all looks good to me, we can add log parsing at a later time. I won't be able to do anything on this until next week, as I am away this week, but this seems good to merge.

OK, I will finish the PR then.

As mentioned here, log parsing seems to work just fine.

issue addressed

easybe · 2025-09-04T14:36:18Z

@b-rowan needs your approval to merge

easybe

Tests need to be fixed

github-actions · 2025-09-04T14:51:16Z

filepath	function	failed	SUBTOTAL
tests/unit/updates/rauc/test_swdesc.py	test_parse_software_header	1	1
TOTAL		1	54

Allow parsing of RAUC files using `PySquashFsImage`.

easybe

Moved the test to the correct location and adapted it to my latest changes.

easybe · 2025-09-06T08:18:37Z

@Tristoris, with the latest release we now have RAUC support. Let us know if that works for you.

Tristoris · 2025-10-07T15:31:45Z

@Tristoris, with the latest release we now have RAUC support. Let us know if that works for you.

Hello. I am sorry, I did not see the notification and missed the message, I will get onto testing rightaway!

Tristoris · 2025-10-07T21:30:51Z

Hi! I wanted to ask whether it is intended that the compatibility text is split like that "r - aspberrypi4" (My set compatibility was raspberrypi4 and raspberrypi4 during build). I also noticed that the devices are not getting the updates i uploaded, which could be explained by the compatibility section there.

github-advanced-security AI found potential problems Sep 10, 2024

View reviewed changes

b-rowan mentioned this pull request Sep 10, 2024

aerich cannot generate migrations #133

Closed

tsagadar previously requested changes Sep 11, 2024

View reviewed changes

Comment thread goosebit/updates/swdesc/rauc.py Outdated

Comment thread goosebit/updates/swdesc/rauc.py

Comment thread goosebit/updates/__init__.py Outdated

Comment thread goosebit/updates/swdesc/__init__.py Outdated

UpstreamData force-pushed the dev_rauc_support branch 3 times, most recently from 4a48a30 to ff3f48e Compare September 11, 2024 14:55

b-rowan mentioned this pull request Sep 11, 2024

File moves may fail with OSError #130

Closed

b-rowan force-pushed the dev_rauc_support branch from ff3f48e to 57a5ab0 Compare August 19, 2025 19:21

github-advanced-security AI found potential problems Aug 19, 2025

View reviewed changes

Comment thread goosebit/updates/swdesc/__init__.py Dismissed

b-rowan force-pushed the dev_rauc_support branch from 57a5ab0 to 3d60c98 Compare August 19, 2025 19:34

b-rowan marked this pull request as ready for review August 19, 2025 19:38

easybe self-requested a review September 3, 2025 20:16

easybe force-pushed the dev_rauc_support branch from 3d60c98 to c3407de Compare September 3, 2025 20:20

easybe reviewed Sep 4, 2025

View reviewed changes

Comment thread goosebit/updates/swdesc/rauc.py Outdated

easybe requested a review from tsagadar September 4, 2025 08:34

easybe force-pushed the dev_rauc_support branch 2 times, most recently from 4da243e to 2d1875b Compare September 4, 2025 12:00

easybe requested changes Sep 4, 2025

View reviewed changes

Comment thread goosebit/db/migrations/models/1_20240911081506_add_image_format.py

easybe force-pushed the dev_rauc_support branch from 2d1875b to 338d694 Compare September 4, 2025 14:31

easybe approved these changes Sep 4, 2025

View reviewed changes

easybe reviewed Sep 4, 2025

View reviewed changes

Comment thread tests/updates/swu/test_swdesc.py Outdated

UpstreamData previously approved these changes Sep 4, 2025

View reviewed changes

easybe requested changes Sep 4, 2025

View reviewed changes

easybe dismissed UpstreamData’s stale review via 01f3f2e September 4, 2025 14:50

easybe force-pushed the dev_rauc_support branch from 338d694 to 01f3f2e Compare September 4, 2025 14:50

Add support for RAUC file parsing

3dbc788

Allow parsing of RAUC files using `PySquashFsImage`.

easybe force-pushed the dev_rauc_support branch from 01f3f2e to 3dbc788 Compare September 4, 2025 18:24

easybe approved these changes Sep 4, 2025

View reviewed changes

UpstreamData approved these changes Sep 4, 2025

View reviewed changes

b-rowan merged commit dac8ce8 into master Sep 4, 2025
9 checks passed

easybe mentioned this pull request Oct 8, 2025

RAUC compatibility “raspberrypi4” not parsed correctly #382

Closed

@@ -3,21 +3,35 @@
             import aiofiles
             import httpx
             from anyio import Path, open_file
+            from urllib.parse import urlparse
             from ...db.models import SoftwareImageFormat
             from . import rauc, swu
+            # Whitelist of allowed domains for remote software updates
+            ALLOWED_DOMAINS = {
+                "trusted-updates.example.com",
+                "updates.example.org",
+                # Add other trusted domains here
+            }
             logger = logging.getLogger(__name__)
             async def parse_remote(url: str):
+                parsed = urlparse(url)
+                if parsed.scheme not in ("http", "https"):
+                    logger.warning(f"Blocked non-http(s) scheme in remote URL: {url}")
+                    raise ValueError("Only http(s) URLs are allowed for remote software updates")
+                if parsed.hostname not in ALLOWED_DOMAINS:
+                    logger.warning(f"Blocked remote URL with disallowed domain: {parsed.hostname}")
+                    raise ValueError("Remote software update URL domain is not allowed")
                 async with httpx.AsyncClient() as c:
                     file = await c.get(url)
                     async with aiofiles.tempfile.NamedTemporaryFile("w+b") as f:
                         await f.write(file.content)
                         return await parse_file(Path(str(f.name)))
             async def parse_file(file: Path):
                 async with await open_file(file, "r+b") as f:
                     magic = await f.read(4)

Conversation

b-rowan commented Sep 10, 2024

Uh oh!

Check failure

Copilot Autofix

tsagadar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

easybe commented Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

b-rowan commented Sep 13, 2024

Uh oh!

tsagadar commented Oct 26, 2024

Uh oh!

Tristoris commented Aug 17, 2025

Uh oh!

b-rowan commented Aug 19, 2025

Uh oh!

Tristoris commented Aug 19, 2025

Uh oh!

b-rowan commented Aug 19, 2025

Uh oh!

easybe commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

easybe commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

easybe commented Sep 4, 2025

Uh oh!

easybe commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

b-rowan commented Sep 4, 2025

Uh oh!

easybe commented Sep 4, 2025

Uh oh!

easybe commented Sep 4, 2025

Uh oh!

Uh oh!

easybe left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 4, 2025

Uh oh!

easybe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

easybe commented Sep 6, 2025

Uh oh!

Tristoris commented Oct 7, 2025

Uh oh!

Tristoris commented Oct 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

easybe commented Sep 13, 2024 •

edited

Loading

easybe commented Aug 19, 2025 •

edited

Loading

easybe commented Sep 4, 2025 •

edited

Loading

easybe commented Sep 4, 2025 •

edited

Loading