fix: Prevent accidental request dropping with maxRequestsPerCrawl

[janbuchar]

• closes maxRequestsPerCrawl with RQ optimizations drops requests #3153

• core issue: maxRequestsPerCrawl and enqueueLinks({ limit }) pre-truncated request lists before sending them to the request queue. Duplicate URLs consumed budget slots, starving actual new URLs.

• Fix: Instead of guessing upfront which requests will be new, let the request queue tell us.

  • A new maxNewRequests option on addRequestsBatched feeds requests to the queue in budget-capped chunks, counts how many were actually new from the wasAlreadyPresent results, and stops pulling once the budget is exhausted.
  • chunkedAsyncIterable was extended to accept a dynamic size callback
  • Leftovers from the source iterator are returned as requestsOverLimit for callers to report.

[barjin]

Thank you @janbuchar !

Here are some late-night ideas I got (otherwise it looks pretty solid):

[barjin]

Let's maybe mention that this effectively forces waitForAllRequestsToBeAdded as per this line?

[barjin]

I feel like this generateRequests() call will create a new generator instance (and therefore push all the requests to requestsOverLimit, regardless of which ones were processed).

const result = await rq.addRequestsBatched([...e.g. 5 urls...], {
        maxNewRequests: 1,
});

// result.addedRequests.length === 1
// result.requestsOverLimit.length === 5 (should be 4?)

Can we have a test for this?

[barjin]

On a side note, the intent of sharing the state through a common closure variable (the iterator) took me some time to understand. Can we pass the half-eaten iterator explicitly as a param?