Future maintainability & scaling – ideas for load distribution and structure

[Paddy0174]

Hello everybody!

I’ve been using waste_collection_schedule for quite a while now and still think it’s one of the most useful HA integrations out there. For me, it's one of the absolute must-have integrations.

Recently I ran into some provider-side issues (blocking / rate limiting), which made me look deeper into timing, fetch patterns and overall structure.

As the project has grown a lot over the years (number of sources, users, providers), I realized that some of these challenges are probably no longer trivial to solve — especially for a project that started as, and still largely is, a one-person effort.

I’m not looking to take over or fork the project in any hostile way – quite the opposite. I’d be happy to experiment with structural improvements (e.g. load distribution, deterministic jitter, cleaner separation of fetch logic, user agents) in a separate, optional refactoring workspace, if that helps in the long run.

Just to make this very clear: my intention is to improve the great work that has already been done within this integration. There is no need or intention for a new integration, replacement, or competing solution. Any work would remain strictly tied to this project and be designed for upstream integration.

One open question for me is how larger structural work could be done in a way that does not add pressure or additional workload to the main maintainer. One idea could be to prepare changes in an isolated working repository or branch and only bring things back here once they are well-tested and ready — but only if that approach is welcome.

Before doing anything bigger, I wanted to ask:

• Are others running into similar issues?
• Would there be interest in discussing a more modular internal structure over time?

No rush, no pressure – just opening the conversation.

Thanks again for the great work over the years!

[5ila5]

I'm currently the only one active here with maintainer permissions. I don't have that much time anymore. I don't really think that requests / User-Agent are that big of a problem. Lately a lot of pages enforced a User-Agent (probably to block AI scrapers?) but once they are transitioned that's not that big of a problem. The much more pressing issue is that I don't have time to look at all the issues and all the sources that break because the service provider changed something on their end. I don't think more modulation does help that much with that.

Especially a lot of UK and Australia sources, which often cover a small area (one municipality) and may require HTML parsing and complicated web requests, seem to cause a majority of issues and broken sources.

Nevertheless, if you have good ideas to improve the development / scalability workflow, they are very welcome. I think they should follow normal development workflow on some branch and if they are finished you can open a PR. If you want some discussion or early comment, you can open a PR as Draft.