Skip to content

fix: allow opting out of anti-bot blocking#2088

Open
nightcityblade wants to merge 1 commit into
unclecode:developfrom
nightcityblade:fix/issue-2058
Open

fix: allow opting out of anti-bot blocking#2088
nightcityblade wants to merge 1 commit into
unclecode:developfrom
nightcityblade:fix/issue-2058

Conversation

@nightcityblade

Copy link
Copy Markdown
Contributor

Summary

Fixes #2058.

Adds a CrawlerRunConfig.check_blocked option, defaulting to True, so callers can opt out of anti-bot classification when they want to apply their own content-quality policy. When disabled, detector matches no longer trigger proxy retries, fallback fetches, or a failed CrawlResult.

List of files changed and why

  • crawl4ai/async_configs.py - defines and serializes the safe check_blocked configuration option.
  • crawl4ai/async_webcrawler.py - skips anti-bot classification when the option is disabled.
  • tests/proxy/test_antibot_opt_out.py - verifies the detector is not called and a small page remains successful.

How Has This Been Tested?

  • .venv/bin/pytest -q tests/proxy/test_antibot_opt_out.py — 1 passed.
  • .venv/bin/python tests/proxy/test_antibot_detector.py — 47 passed.
  • .venv/bin/ruff check --ignore F401,F405 crawl4ai/async_configs.py crawl4ai/async_webcrawler.py tests/proxy/test_antibot_opt_out.py — passed (F401/F405 are pre-existing in the touched modules).
  • .venv/bin/black --check --target-version py312 tests/proxy/test_antibot_opt_out.py — passed.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant