Skip to content

[Fix][Connector-V2][Http] Stop cursor pagination when cursor does not advance#10944

Open
officialasishkumar wants to merge 1 commit into
apache:devfrom
officialasishkumar:fix/http-cursor-no-progress
Open

[Fix][Connector-V2][Http] Stop cursor pagination when cursor does not advance#10944
officialasishkumar wants to merge 1 commit into
apache:devfrom
officialasishkumar:fix/http-cursor-no-progress

Conversation

@officialasishkumar
Copy link
Copy Markdown

Purpose of this pull request

Fixes #10929.

This PR stops HTTP cursor pagination when a response returns the same non-empty cursor as the previous request. The existing cursor path only stopped when the next cursor was empty, so APIs that signal the terminal page by repeating the final cursor could keep polling without progress.

Does this PR introduce any user-facing change?

Yes. HTTP cursor pagination now stops when the cursor does not advance, avoiding repeated requests for cursor APIs that return the final cursor again at end-of-data.

How was this patch tested?

  • JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 MAVEN_OPTS=-Xmx1536m ./mvnw -B -pl seatunnel-connectors-v2/connector-http/connector-http-base -DskipIT=true -D"license.skipAddThirdParty"=true -D"skip.ui"=true spotless:apply
  • JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 MAVEN_OPTS=-Xmx1536m ./mvnw -B -U -pl seatunnel-connectors-v2/connector-http/connector-http-base -Dtest=org.apache.seatunnel.connectors.seatunnel.http.HttpSourceReaderInternalPollNextTest -DfailIfNoTests=false -DskipIT=true -D"license.skipAddThirdParty"=true -D"skip.ui"=true test

Check list

Copy link
Copy Markdown
Contributor

@DanielLeens DanielLeens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. I reviewed the latest head from scratch and retraced the bounded HTTP cursor-pagination path. The change is focused, and I did not find a new blocker on the current revision.

What This PR Fixes

  • User pain: some cursor-based HTTP APIs signal the terminal page by repeating the last cursor instead of returning an empty cursor. The old reader only stopped on an empty cursor, so it could keep polling forever without progress.
  • Fix approach: the latest head treats a non-empty but unchanged cursor as "not advanced" and stops the bounded reader.
  • One-line summary: this is a precise fix for an actual no-progress polling loop in the cursor path.

Full Runtime Chain

internalPollNext()
  -> pollAndCollectData(...)
  -> collect(output, data) [384-433]
     -> cursor pagination branch [397-417]
        -> parse next cursor from JSON
        -> compare with previous cursor
        -> if empty or unchanged => noMoreElementFlag = true
  -> finally block
     -> bounded source + noMoreElementFlag => signalNoMoreElement()

Key Findings

  1. The normal bounded cursor-pagination path definitely hits this logic.
  2. The new stop condition matches the real no-progress scenario from the PR description.
  3. The regression test verifies both the bounded stop and the fact that the reader does not keep issuing requests forever.
  4. I did not find a new source-level blocker on the current head.

Findings

No new blocking issues found on the current head.

Merge Conclusion

Conclusion: can merge

  1. Blocking items
  • None from Daniel on the current revision.
  1. Suggested non-blocking follow-up
  • The top-level Build signal still looks freshly unsettled, so I would let the latest run finish before merging.

Overall, this looks like a clean fix for a real bounded-reader edge case.

@davidzollo davidzollo added the First-time contributor First-time contributor label May 25, 2026
@davidzollo
Copy link
Copy Markdown
Contributor

Please enable CI check following this instruction
https://github.com/apache/seatunnel/pull/10944/checks?check_run_id=77651231864

@DanielLeens
Copy link
Copy Markdown
Contributor

Thanks for the CI follow-up. From the code review side, I do not have any additional source-level blockers on the current head. The next step here is to enable / rerun the workflow and let the Build result settle. Once the contributor pushes a new commit or there is a new technical update, I am happy to take another look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][connector] http connector cursor type pageing not check the number of returned items is less than batchSize?

3 participants