Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Copy this file to .env for local runs. Do not commit real API keys.

# Main backporting workflow with regular OpenAI.
OPENAI_API_KEY=

# Main backporting workflow when llm_provider: deepseek.
DEEPSEEK_API_KEY=

# LLM-backed prejudge workflow.
OPENROUTER_API_KEY=
14 changes: 13 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ Dockerfile Containerized runtime
- [Architecture](docs/architecture.md)
- [Runtime Safety](docs/runtime-safety.md)
- [Configuration Reference](docs/config-reference.md)
- [Provider Configuration](docs/provider-configuration.md)
- [Tool Contracts](docs/tool-contracts.md)
- [Evaluation](docs/evaluation.md)
- [Prompt Design](docs/prompt-design.md)
Expand All @@ -60,7 +61,7 @@ Dockerfile Containerized runtime
- Universal Ctags (`ctags`) for symbol indexing
- A disposable clone of the target project
- Optional build dependencies required by the target project
- An OpenAI API key or Azure OpenAI configuration for the main workflow
- OpenAI, DeepSeek, or Azure OpenAI provider credentials for the main workflow
- `OPENROUTER_API_KEY` if you run the prejudge LLM tools

Install Python dependencies:
Expand Down Expand Up @@ -147,6 +148,13 @@ openai_key: sk-...
project_dir: dataset/libsdl-org/libtiff
patch_dataset_dir: ~/backports/patch_dataset/libtiff/CVE-2023-3576/

llm_provider: openai
llm_model: gpt-4-turbo
llm_base_url: https://api.openai.com/v1
llm_temperature: 0.5
llm_max_tokens:
llm_thinking:

use_azure: false
# azure_endpoint: "https://your-resource.openai.azure.com/"
# azure_deployment: "gpt-5"
Expand All @@ -165,6 +173,10 @@ Field reference:
- `error_message`: Text expected in PoC output when the bug is still triggered.
- `tag`: Case identifier, commonly a CVE or bug ID.
- `openai_key`: API key for the main backporting agent.
- `llm_provider`, `llm_model`, `llm_base_url`, `llm_temperature`,
`llm_max_tokens`, `llm_thinking`: Optional OpenAI-compatible model settings.
For DeepSeek, set `llm_provider: deepseek` and provide the key with
`DEEPSEEK_API_KEY` or `deepseek_key`.
- `project_dir`: Local target-project Git repository.
- `patch_dataset_dir`: Case directory containing validation scripts and copied
logs.
Expand Down
14 changes: 13 additions & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ Dockerfile 容器化运行环境
- [架构](docs/architecture.md)
- [运行安全](docs/runtime-safety.md)
- [配置参考](docs/config-reference.md)
- [Provider 配置](docs/provider-configuration.md)
- [工具契约](docs/tool-contracts.md)
- [评估方法](docs/evaluation.md)
- [Prompt 设计](docs/prompt-design.md)
Expand All @@ -50,7 +51,7 @@ Dockerfile 容器化运行环境
- Universal Ctags(命令为 `ctags`),用于符号索引
- 目标项目的一份可丢弃 clone
- 目标项目自身需要的可选构建依赖
- 主回移流程使用的 OpenAI API key 或 Azure OpenAI 配置
- 主回移流程使用的 OpenAI、DeepSeek 或 Azure OpenAI provider 凭据
- 如果运行预判 LLM 工具,需要设置 `OPENROUTER_API_KEY`

安装 Python 依赖:
Expand Down Expand Up @@ -131,6 +132,13 @@ openai_key: sk-...
project_dir: dataset/libsdl-org/libtiff
patch_dataset_dir: ~/backports/patch_dataset/libtiff/CVE-2023-3576/

llm_provider: openai
llm_model: gpt-4-turbo
llm_base_url: https://api.openai.com/v1
llm_temperature: 0.5
llm_max_tokens:
llm_thinking:

use_azure: false
# azure_endpoint: "https://your-resource.openai.azure.com/"
# azure_deployment: "gpt-5"
Expand All @@ -148,6 +156,10 @@ use_azure: false
- `error_message`:PoC 输出中代表 bug 仍可触发的关键文本。
- `tag`:案例标识,通常是 CVE 或 bug ID。
- `openai_key`:主回移 agent 使用的 API key。
- `llm_provider`、`llm_model`、`llm_base_url`、`llm_temperature`、
`llm_max_tokens`、`llm_thinking`:可选的 OpenAI-compatible 模型配置。使用
DeepSeek 时设置 `llm_provider: deepseek`,并通过 `DEEPSEEK_API_KEY` 或
`deepseek_key` 提供 key。
- `project_dir`:本地目标项目 Git 仓库路径。
- `patch_dataset_dir`:案例目录,包含验证脚本并接收日志副本。
- `use_azure`、`azure_endpoint`、`azure_deployment`、`azure_api_version`:可选 Azure OpenAI 配置。
Expand Down
34 changes: 31 additions & 3 deletions docs/config-reference.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# Configuration Reference

The main backporting CLI reads a YAML file through `load_yml()` in `src/backporting.py`.
See [Provider Configuration](provider-configuration.md) for provider-specific
environment variables and model examples.

Run:

Expand Down Expand Up @@ -29,6 +31,13 @@ openai_key: sk-...
project_dir: dataset/libsdl-org/libtiff
patch_dataset_dir: ~/backports/patch_dataset/libtiff/CVE-2023-3576/

llm_provider: openai
llm_model: gpt-4-turbo
llm_base_url: https://api.openai.com/v1
llm_temperature: 0.5
llm_max_tokens:
llm_thinking:

use_azure: false
# azure_endpoint: "https://your-resource.openai.azure.com/"
# azure_deployment: "gpt-5"
Expand All @@ -44,13 +53,19 @@ use_azure: false
- `new_patch`: Upstream fixed commit. RetroPatch reads this commit with `git show` and splits the resulting patch into hunks.
- `new_patch_parent`: Parent or vulnerable-side commit for `new_patch`. This gives the agent a newer vulnerable reference for history tracing.
- `target_release`: Older revision that should receive the backported fix.
- `openai_key`: API key used by the main backporting LLM path. Azure mode also reads this field as the Azure OpenAI key.
- `openai_key`: API key used by the main backporting LLM path. Azure mode also reads this field as the Azure OpenAI key. For regular OpenAI, this can also be supplied with `OPENAI_API_KEY`.

## Optional Fields

- `tag`: Case identifier used in log filenames. Common values are CVE IDs or bug IDs.
- `sanitizer`: Metadata describing the sanitizer used by the case. The current main workflow does not make decisions from this field.
- `error_message`: Text expected in `poc.sh` output when the bug is still triggered. If this field is empty, `Project` uses a placeholder value and the run logs a warning.
- `llm_provider`: OpenAI-compatible provider name. Defaults to `openai`. Use `deepseek` for DeepSeek.
- `llm_model`: Chat model name. Defaults to `gpt-4-turbo`, or `deepseek-v4-pro` when `llm_provider: deepseek`.
- `llm_base_url`: OpenAI-compatible API base URL. Defaults to `https://api.openai.com/v1`, or `https://api.deepseek.com` when `llm_provider: deepseek`.
- `llm_temperature`: Chat model temperature. Defaults to `0.5`.
- `llm_max_tokens`: Optional maximum completion token budget. Defaults to `4096` for DeepSeek because reasoning models may consume reasoning tokens before producing visible text.
- `llm_thinking`: Optional DeepSeek thinking mode setting. Defaults to `disabled` for DeepSeek so tool-calling conversations do not need to replay provider-specific reasoning content.
- `use_azure`: Boolean selecting the Azure OpenAI client path. Defaults to `false`.
- `azure_endpoint`: Azure OpenAI endpoint. Required when `use_azure: true`.
- `azure_deployment`: Azure OpenAI deployment name. Defaults to `gpt-4` in the loader if omitted.
Expand All @@ -76,12 +91,25 @@ Each value must resolve to a valid commit in the target repository. After valida

## Main LLM Selection

The current main workflow has two model paths:
The current main workflow has three model paths:

- regular OpenAI, configured in `src/agent/invoke_llm.py`
- DeepSeek or another OpenAI-compatible endpoint, selected with `llm_provider`
- Azure OpenAI, selected with `use_azure: true`

Regular OpenAI currently uses a hard-coded model and base URL in code. Azure mode reads deployment and endpoint fields from YAML.
DeepSeek example:

```yaml
llm_provider: deepseek
llm_base_url: https://api.deepseek.com
llm_model: deepseek-v4-pro
llm_max_tokens: 4096
llm_thinking: disabled
```

Provide the key with `DEEPSEEK_API_KEY` or `deepseek_key`. Prefer the environment variable for local runs.

Azure mode reads deployment and endpoint fields from YAML and takes precedence when `use_azure: true`.

## Prejudge Configuration

Expand Down
82 changes: 82 additions & 0 deletions docs/provider-configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Provider Configuration

RetroPatch can run the main backporting workflow with regular OpenAI, Azure
OpenAI, or an OpenAI-compatible provider such as DeepSeek.

Keep API keys in environment variables or local-only files. The repository
includes `.env.example` as a template, while `.env` is ignored by Git.

## Environment Variables

```shell
cp .env.example .env
```
Comment on lines +9 to +13

Fill only the keys required for the provider you use:

- `OPENAI_API_KEY`: regular OpenAI for the main workflow.
- `DEEPSEEK_API_KEY`: DeepSeek for the main workflow.
- `OPENROUTER_API_KEY`: LLM-backed prejudge workflow.

If you export variables manually, do it in the same shell that runs
`src/backporting.py`.

## Regular OpenAI

Use the default provider settings:

```yaml
use_azure: false
llm_provider: openai
llm_model: gpt-4-turbo
llm_base_url: https://api.openai.com/v1
llm_temperature: 0.5
```

Provide the key with `OPENAI_API_KEY`. The legacy `openai_key` YAML field still
works, but environment variables are safer for shared configs.

## DeepSeek

Select DeepSeek with the OpenAI-compatible provider fields:

```yaml
use_azure: false
llm_provider: deepseek
llm_base_url: https://api.deepseek.com
llm_model: deepseek-v4-pro
llm_max_tokens: 4096
llm_thinking: disabled
```

Provide the key with `DEEPSEEK_API_KEY`. `llm_thinking: disabled` is the
recommended setting for tool-calling runs because it avoids provider-specific
reasoning content replay requirements.

## Azure OpenAI

Azure mode takes precedence when `use_azure: true`:

```yaml
use_azure: true
azure_endpoint: "https://your-resource.openai.azure.com/"
azure_deployment: "gpt-5"
azure_api_version: "2024-12-01-preview"
```

The current loader reads the Azure key from `openai_key`. Keep Azure configs in
local-only YAML files unless your deployment has a separate secret injection
path.

## Prejudge Provider Keys

The prejudge LLM path is configured separately from the main backporting YAML.
It expects `OPENROUTER_API_KEY` in the environment and accepts provider names
through the prejudge CLI path.

## Common Checks

- Confirm the selected key is present in the shell environment.
- Confirm `llm_base_url` matches the provider endpoint.
- Confirm the configured model supports tool calling.
- Use local config files for secrets and commit only redacted examples.
33 changes: 27 additions & 6 deletions src/agent/invoke_llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,13 @@
from tools.utils import split_patch


class NonStreamingChatOpenAI(ChatOpenAI):
"""ChatOpenAI variant for providers that reject streamed reasoning turns."""

def stream(self, input, config=None, *, stop=None, **kwargs):
yield self.invoke(input, config=config, stop=stop, **kwargs)


def initial_agent(project: Project, data, debug_mode: bool):
use_azure = data.use_azure

Expand All @@ -38,15 +45,29 @@ def initial_agent(project: Project, data, debug_mode: bool):
verbose=True,
)
else:
# Regular OpenAI configuration
logger.info("Using OpenAI API")
base_url = "https://api.openai.com/v1"
llm = ChatOpenAI(
temperature=0.5,
model="gpt-4-turbo",
provider = getattr(data, "llm_provider", "openai")
model = getattr(data, "llm_model", "gpt-4-turbo")
base_url = getattr(data, "llm_base_url", "https://api.openai.com/v1")
temperature = getattr(data, "llm_temperature", 0.5)
max_tokens = getattr(data, "llm_max_tokens", None)
thinking = getattr(data, "llm_thinking", None)
logger.info(f"Using {provider} API: {base_url} (model: {model})")
llm_kwargs = {}
if max_tokens is not None:
llm_kwargs["max_tokens"] = max_tokens
model_kwargs = {}
if provider == "deepseek" and thinking:
model_kwargs["extra_body"] = {"thinking": {"type": thinking}}
if model_kwargs:
llm_kwargs["model_kwargs"] = model_kwargs
llm_class = NonStreamingChatOpenAI if provider == "deepseek" else ChatOpenAI
llm = llm_class(
temperature=temperature,
model=model,
api_key=data.openai_key,
openai_api_base=base_url,
verbose=True,
**llm_kwargs,
)

prompt = ChatPromptTemplate.from_messages(
Expand Down
56 changes: 39 additions & 17 deletions src/backporting.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,28 @@ def load_yml(file_path: str):
data.project_url = config.get("project_url")
data.project_dir = config.get("project_dir")
data.patch_dataset_dir = config.get("patch_dataset_dir")
data.openai_key = config.get("openai_key")
data.openai_key = config.get("openai_key") or os.getenv("OPENAI_API_KEY", "")
data.tag = config.get("tag")

# OpenAI-compatible LLM configuration
data.llm_provider = config.get("llm_provider", "openai").lower()
data.llm_model = config.get("llm_model", "gpt-4-turbo")
Comment on lines +58 to +60
data.llm_base_url = config.get("llm_base_url", "https://api.openai.com/v1")
data.llm_temperature = config.get("llm_temperature", 0.5)
data.llm_max_tokens = config.get("llm_max_tokens")
data.llm_thinking = config.get("llm_thinking")

if data.llm_provider == "deepseek":
data.openai_key = (
config.get("deepseek_key")
or os.getenv("DEEPSEEK_API_KEY", "")
or config.get("openai_key")
)
data.llm_base_url = config.get("llm_base_url", "https://api.deepseek.com")
data.llm_model = config.get("llm_model", "deepseek-v4-pro")
data.llm_max_tokens = config.get("llm_max_tokens", 4096)
data.llm_thinking = config.get("llm_thinking", "disabled")

# Azure OpenAI configuration (optional)
data.use_azure = config.get("use_azure", False)
data.azure_endpoint = config.get("azure_endpoint", "")
Expand Down Expand Up @@ -150,33 +169,36 @@ def main():
project = Project(data)
project.repo.git.clean("-fdx")
start_time = time.time()
before_usage = get_usage(data.openai_key)
track_openai_usage = not data.use_azure and data.llm_provider == "openai"
before_usage = get_usage(data.openai_key) if track_openai_usage else None
agent_executor, llm = initial_agent(project, data, debug_mode)
try:
do_backport(agent_executor, project, data, llm, logfile)
Comment on lines +172 to 176
end_time = time.time()
time.sleep(10)
after_usage = get_usage(data.openai_key)
logger.debug(
f"This patch total cost: ${(after_usage['total_cost'] - before_usage['total_cost']):.2f}"
)
logger.debug(
f"This patch total consume tokens: {(after_usage['total_consume_tokens'] - before_usage['total_consume_tokens'])/1000}(k)"
)
if track_openai_usage:
time.sleep(10)
after_usage = get_usage(data.openai_key)
logger.debug(
f"This patch total cost: ${(after_usage['total_cost'] - before_usage['total_cost']):.2f}"
)
logger.debug(
f"This patch total consume tokens: {(after_usage['total_consume_tokens'] - before_usage['total_consume_tokens'])/1000}(k)"
)
logger.debug(
f"This patch total cost time: {int(end_time - start_time)} Seconds."
)
except KeyboardInterrupt:
logger.debug("Start to calculate cost!")
end_time = time.time()

after_usage = get_usage(data.openai_key)
logger.debug(
f"This patch total cost: ${(after_usage['total_cost'] - before_usage['total_cost']):.2f}"
)
logger.debug(
f"This patch total consume tokens: {(after_usage['total_consume_tokens'] - before_usage['total_consume_tokens'])/1000}(k)"
)
if track_openai_usage:
after_usage = get_usage(data.openai_key)
logger.debug(
f"This patch total cost: ${(after_usage['total_cost'] - before_usage['total_cost']):.2f}"
)
logger.debug(
f"This patch total consume tokens: {(after_usage['total_consume_tokens'] - before_usage['total_consume_tokens'])/1000}(k)"
)
logger.debug(
f"This patch total cost time: {int(end_time - start_time)} Seconds."
)
Expand Down
9 changes: 9 additions & 0 deletions src/example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,15 @@ patch_dataset_dir: ~/backports/patch_dataset/libtiff/CVE-2023-3576/
# Set use_azure: true for Azure OpenAI
use_azure: false

# OpenAI-compatible provider configuration
# For DeepSeek, set:
# llm_provider: deepseek
# llm_base_url: "https://api.deepseek.com"
# llm_model: "deepseek-v4-pro"
# llm_max_tokens: 4096
# llm_thinking: disabled
# Then provide the key via DEEPSEEK_API_KEY.

# Azure OpenAI Configuration (only needed if use_azure: true)
# azure_endpoint: "https://your-resource.openai.azure.com/"
# azure_deployment: "gpt-5"
Expand Down
Loading