Skip to content

feat: add Tavily as search engine option in research_web#1052

Closed
ParakhJaggi wants to merge 1 commit intoScrapeGraphAI:mainfrom
tavily-ai:feat/tavily-migration/tests-research-web
Closed

feat: add Tavily as search engine option in research_web#1052
ParakhJaggi wants to merge 1 commit intoScrapeGraphAI:mainfrom
tavily-ai:feat/tavily-migration/tests-research-web

Conversation

@ParakhJaggi
Copy link
Copy Markdown

Summary

  • Added Tavily as a configurable search engine alongside existing providers (duckduckgo, bing, searxng, serper)
  • Implemented _search_tavily() helper using TavilyClient from tavily-python, with proper error handling for missing API keys, import errors, and API failures
  • Added tavily_api_key parameter to SearchConfig model and search_on_web() function
  • Added 7 new test cases covering: config validation, successful search, missing/empty API key, API errors, empty results, and end-to-end dispatch

Files changed

  • scrapegraphai/utils/research_web.py — Added tavily to valid engines, tavily_api_key field, _search_tavily() function, and tavily branch in search_on_web()
  • tests/utils/research_web_test.py — Added 7 tavily test cases using mocked TavilyClient
  • pyproject.toml — Added tavily-python as optional dependency under [project.optional-dependencies]

Dependency changes

  • Added tavily-python>=0.5.0 as optional dependency (pip install scrapegraphai[tavily])

Environment variable changes

  • TAVILY_API_KEY — Required when using search_engine="tavily", passed via tavily_api_key parameter

Notes for reviewers

  • Tavily is added as an additive option — existing search engines are unchanged
  • tavily-python is imported lazily inside _search_tavily() to avoid hard dependency
  • All 7 new tests pass with mocked TavilyClient (no API key needed for tests)

🤖 Generated with Claude Code

Automated Review

  • Passed after 1 attempt(s)
  • Final review: The Tavily migration is correct and complete. The implementation follows existing codebase patterns precisely: lazy import, consistent error handling with SearchConfigError/SearchRequestError, URL extraction from response["results"], and optional dependency in pyproject.toml. The 7 new tests are well-structured with proper sys.modules mocking. Three minor issues were found (type annotation, missing docstring entry, weak integration test assertion) but none block approval.

@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. dependencies Pull requests that update a dependency file enhancement New feature or request tests Improvements or additions to test labels Mar 24, 2026
@tavily-ai tavily-ai closed this by deleting the head repository Mar 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file enhancement New feature or request size:L This PR changes 100-499 lines, ignoring generated files. tests Improvements or additions to test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants