Zoomeye bulletproof multipage results - #21770
Open
e2002e wants to merge 5 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the gather/zoomeye_search auxiliary module’s ZoomEye pagination to reduce infinite-loop risk on empty pages by forcing a 10-result page size and adding retry/skip handling when a page returns no matches.
Changes:
- Set ZoomEye API
pagesizeto 10 and update total-page computation accordingly. - Add retry-and-skip logic when a page returns
matches: nilto avoid getting stuck on empty pages.
Impact Analysis:
- Blast radius: medium — affects all users of
auxiliary/gather/zoomeye_searchand the completeness/ordering of collected results. - Data and contract effects: output row count and which pages are queried can change (10-per-page assumption is now baked into both request and page-count math).
- Rollback and test focus: rollback is straightforward (single module); validate by running the module against queries known to produce intermittent empty pages and confirming (1) no infinite loop, (2) skip messages show the correct page number, and (3) retries apply per page.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
172
to
188
| while page < maxpage | ||
| page_result = dork_search(resource, dork, page + 1, facets, api_key) | ||
| page_result = dork_search(resource, dork, page + skipped + 1, facets, api_key) | ||
| if page_result['matches'].nil? | ||
| next | ||
| retrying += 1 | ||
| if retrying < 3 | ||
| next | ||
| else | ||
| print_error("Skipping page #{page + skipped}") | ||
| if page + skipped >= maxpage #stop the strafe once we reach the theorical maxpage, but let it try it first. | ||
| break | ||
| end | ||
| skipped += 1 | ||
| next | ||
| end | ||
| else | ||
| retrying = 0 | ||
| end |
Comment on lines
+166
to
+167
| # If search results greater than 20, loop & get all results | ||
| if results[first_page]['total'] > 20 | ||
| if results[first_page]['total'] > 10 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hello,
This pull request modifies the loop on result pages so that it can handle empty ones without looping forever or truncating the list.
It uses a retry count (set to 3) for earch page, if it exeeds it, it goes to the next page.
Keeping the amount of results per page to the minimum of 10 is necessary to limit the loss in case some return nil.
wc outfile > 606 (5 are header) so we lost 17 results but it took 2 minutes and the list is quite exhaustive. I don't know if there is really that total of results but if so I don't see any way of getting it.