|
| 1 | +--- |
| 2 | +layout: integration |
| 3 | +name: ddgs |
| 4 | +description: Keyless, multi-engine web search for Haystack via ddgs (Dux Distributed Global Search) |
| 5 | +authors: |
| 6 | + - name: deepset |
| 7 | + socials: |
| 8 | + github: deepset-ai |
| 9 | + twitter: deepset_ai |
| 10 | + linkedin: https://www.linkedin.com/company/deepset-ai/ |
| 11 | +pypi: https://pypi.org/project/ddgs-haystack |
| 12 | +repo: https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/ddgs |
| 13 | +type: Search & Extraction |
| 14 | +report_issue: https://github.com/deepset-ai/haystack-core-integrations/issues |
| 15 | +version: Haystack 2.0 |
| 16 | +toc: true |
| 17 | +--- |
| 18 | + |
| 19 | +### Table of Contents |
| 20 | + |
| 21 | +- [Overview](#overview) |
| 22 | +- [Installation](#installation) |
| 23 | +- [Usage](#usage) |
| 24 | + - [DDGSWebSearch](#ddgswebsearch) |
| 25 | +- [License](#license) |
| 26 | + |
| 27 | +## Overview |
| 28 | + |
| 29 | +[ddgs](https://github.com/deedy5/ddgs) ("Dux Distributed Global Search") is a free metasearch library that aggregates results from multiple backends — DuckDuckGo, Google, Bing, Brave, Yahoo, Yandex, and more. |
| 30 | + |
| 31 | +This integration provides: |
| 32 | +- [`DDGSWebSearch`](https://docs.haystack.deepset.ai/docs/ddgswebsearch): Searches the web through ddgs and returns results as Haystack `Document` objects along with source URLs. |
| 33 | + |
| 34 | +No API key is required. |
| 35 | + |
| 36 | +## Installation |
| 37 | + |
| 38 | +```bash |
| 39 | +pip install ddgs-haystack |
| 40 | +``` |
| 41 | + |
| 42 | +## Usage |
| 43 | + |
| 44 | +### DDGSWebSearch |
| 45 | + |
| 46 | +`DDGSWebSearch` queries multiple search backends through ddgs and returns results as Haystack `Document` objects containing the content snippets and metadata (title, URL). Source URLs are also returned separately. |
| 47 | + |
| 48 | +#### Basic Example |
| 49 | + |
| 50 | +```python |
| 51 | +from haystack_integrations.components.websearch.ddgs import DDGSWebSearch |
| 52 | + |
| 53 | +web_search = DDGSWebSearch(top_k=5) |
| 54 | + |
| 55 | +result = web_search.run(query="What is Haystack by deepset?") |
| 56 | +documents = result["documents"] |
| 57 | +links = result["links"] |
| 58 | +``` |
| 59 | + |
| 60 | +#### In a Pipeline |
| 61 | + |
| 62 | +Here is an example of a RAG pipeline that uses `DDGSWebSearch` to retrieve web content and answer a question: |
| 63 | + |
| 64 | +```python |
| 65 | +from haystack import Pipeline |
| 66 | +from haystack.utils import Secret |
| 67 | +from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder |
| 68 | +from haystack.components.generators.chat import OpenAIChatGenerator |
| 69 | +from haystack.dataclasses import ChatMessage |
| 70 | +from haystack_integrations.components.websearch.ddgs import DDGSWebSearch |
| 71 | + |
| 72 | +web_search = DDGSWebSearch(top_k=3) |
| 73 | + |
| 74 | +prompt_template = [ |
| 75 | + ChatMessage.from_system("You are a helpful assistant."), |
| 76 | + ChatMessage.from_user( |
| 77 | + "Given the information below:\n" |
| 78 | + "{% for document in documents %}{{ document.content }}\n{% endfor %}\n" |
| 79 | + "Answer the following question: {{ query }}.\nAnswer:", |
| 80 | + ), |
| 81 | +] |
| 82 | + |
| 83 | +prompt_builder = ChatPromptBuilder( |
| 84 | + template=prompt_template, |
| 85 | + required_variables={"query", "documents"}, |
| 86 | +) |
| 87 | + |
| 88 | +llm = OpenAIChatGenerator( |
| 89 | + api_key=Secret.from_env_var("OPENAI_API_KEY"), |
| 90 | + model="gpt-4o-mini", |
| 91 | +) |
| 92 | + |
| 93 | +pipe = Pipeline() |
| 94 | +pipe.add_component("search", web_search) |
| 95 | +pipe.add_component("prompt_builder", prompt_builder) |
| 96 | +pipe.add_component("llm", llm) |
| 97 | + |
| 98 | +pipe.connect("search.documents", "prompt_builder.documents") |
| 99 | +pipe.connect("prompt_builder.prompt", "llm.messages") |
| 100 | + |
| 101 | +query = "What is Haystack by deepset?" |
| 102 | +result = pipe.run(data={"search": {"query": query}, "prompt_builder": {"query": query}}) |
| 103 | +print(result["llm"]["replies"][0].text) |
| 104 | +``` |
| 105 | + |
| 106 | +#### Async Support |
| 107 | + |
| 108 | +The component supports asynchronous execution via `run_async`: |
| 109 | + |
| 110 | +```python |
| 111 | +import asyncio |
| 112 | +from haystack_integrations.components.websearch.ddgs import DDGSWebSearch |
| 113 | + |
| 114 | +async def main(): |
| 115 | + web_search = DDGSWebSearch(top_k=3) |
| 116 | + result = await web_search.run_async(query="What is Haystack by deepset?") |
| 117 | + print(f"Found {len(result['documents'])} documents") |
| 118 | + |
| 119 | +asyncio.run(main()) |
| 120 | +``` |
| 121 | + |
| 122 | +#### Parameters |
| 123 | + |
| 124 | +- **`top_k`**: Maximum number of results to return. Defaults to 10. |
| 125 | +- **`backend`**: Comma-separated ddgs backends to query, or `"auto"` to let ddgs choose (e.g. `"duckduckgo, google, brave"`). Defaults to `"auto"`. |
| 126 | +- **`region`**: Region/locale for the search, e.g. `"us-en"`, `"de-de"`, or `"wt-wt"` (no region). Defaults to `"us-en"`. |
| 127 | +- **`safesearch`**: Safe-search level: `"on"`, `"moderate"`, or `"off"`. Defaults to `"moderate"`. |
| 128 | +- **`search_params`**: Additional keyword arguments forwarded to `DDGS().text()`, for example `page` or `timelimit`. |
| 129 | + |
| 130 | +### License |
| 131 | + |
| 132 | +`ddgs-haystack` is distributed under the terms of the [Apache-2.0](https://spdx.org/licenses/Apache-2.0.html) license. |
0 commit comments