Repository navigation
feat(git): add background polling engine and native gitlab and gitee platform adapters - #61
Conversation
|
Thank you for your dedicated work on this PR! Introducing proactive polling and multi-platform adapters is a crucial capability for developers working behind corporate firewalls and NATs. Roadmap & Development PrioritiesAs this Git automation feature was just officially introduced today, our current architecture intentionally maintains a clean, flexible, and pluggable adapter shell. Our current roadmap priority is to first polish and perfect the GitHub implementation end-to-end (ensuring webhook handling, polling mechanisms, comment lifecycles, and rate-limiting are fully matured), before systematically expanding into deep adapters for GitLab, Gitee, and other platforms. We definitely plan to push the local polling engine forward in upcoming releases. If you are interested in continuing to refine this PR, we warmly welcome you to iterate on it based on the review feedback below. Once this PR reaches a milestone-grade completion, please feel free to ping/tag me directly, and I will review and handle the merge promptly! Key Review Feedback & Milestone Checklist1. Full Event Lifecycle (PR + Issue + Comment Closed Loop)
2. Network & Performance Optimizations (Prevent API Rate Limit Exhaustion)
3. Prevent "Thundering Herd" on Initial Setup & Cap Concurrency
4. Disk I/O Optimization
5. Adapter Architecture Purity
6. Branch Rebase
Looking forward to seeing this reach a milestone state! 非常感谢您在此 PR 上的辛勤付出!引入主动轮询引擎并适配多平台,对于在企业内网、VPN 和 NAT 防火墙后无法接收公网 Webhook 的开发者而言是一项非常核心的能力。 路线图与研发优先级说明由于 Git 自动化相关功能今天刚正式上线,我们目前的架构特意保留了一个轻量、灵活、易于插拔的适配器壳。 我们当前的研发重心是先将 GitHub 的核心能力打磨至成熟完备(确保 Webhook 处理、轮询机制、评论生命周期以及 API 限流控制在 GitHub 上完全闭环),在此之后再系统性、逐步地推进 GitLab、Gitee 等其他平台的深度适配与开发。 本地轮询模式我们后续必定会持续大力推进。如果您有兴趣继续完善该 PR,非常欢迎您参考下方的审查建议进行迭代优化。待该 PR 达到一个里程碑式的完结状态时,欢迎您随时在 PR 中 @ 我,我会第一时间跟进复核并处理合并! 关键审查意见与里程碑整改清单1. 实现完整的事件生命周期(PR + Issue + Comment 闭环)
2. 网络与性能防爆炸优化(避免 API 限流耗尽)
3. 避免初次监听的“海啸审查”(Baseline Seeding)与并发限流
4. 优化磁盘 I/O 磨损
5. 适配器模式规范化
6. 分支 Rebase
期待看到该 PR 迈向里程碑式的完善! |
46cb631 to
e34ef12
Compare
…te limiting, and adapter purity Address review feedback on PR jeikl#61: implement baseline seeding, concurrency rate limiting, in-memory heartbeats, GitLab Note ID purity, and dynamic Gitee routing. 1. Prevent Thundering Herd via Baseline Seeding: - Added initialized flag to RepoPollState and implemented seed_repo_baseline and is_repo_initialized. - When polling an existing repository for the first time, all existing open PRs and their head SHAs are recorded purely as historical baselines without triggering reviews or emitting ACK comments. Only newly opened PRs or commit pushes in subsequent cycles trigger review workflows. 2. Concurrency Rate Limiting: - Introduced a tokio::sync::Semaphore capping concurrent automated review workflows to 2 parallel tasks, preventing unconstrained spawns from exhausting daemon resources or triggering provider 429 errors. 3. Disk I/O Optimization via In-Memory Heartbeats: - Maintained LAST_POLLED_CACHE in memory for high-frequency last_polled_at timestamps. Eliminated periodic 15-second disk writes to git-poll-state.json, persisting to disk only upon genuine state mutations (baseline seeding or new PR/SHA updates). 4. Adapter Architecture Purity (GitLab & Gitee): - Formally updated GitPlatformAdapter::update_comment signature with target_number: Option<u64>. - Eliminated the 32-bit bitmasking hack in GitLabAdapter, returning raw Note IDs and cleanly updating MR notes using mr_iid and comment_id. - Implemented dynamic routing in GiteeAdapter to gracefully route between /pulls/comments/ and /issues/comments/. 5. Enhanced GitHub Polling Query: - Added sort=updated&direction=desc&per_page=50 to fetch_active_pull_requests to prevent pagination starvation on active repositories. 6. Rebase & Verification: - Rebased cleanly onto upstream/beta, resolving merge conflicts. - Added unit tests test_baseline_seeding_prevents_thundering_herd and test_in_memory_heartbeat_updates. 17/17 git_adapter tests and 427/427 jeikcode-daemon tests pass. --- 落实 PR jeikl#61 审查意见:实现基线快照初始化、并发限流、内存心跳、GitLab Note ID 规范化以及 Gitee 动态路由。 1. 基线快照初始化(防止海啸审查): - 在 RepoPollState 中增加 initialized 标记,并实现 seed_repo_baseline 与 is_repo_initialized。 - 当仓库首次接入主动轮询时,仅对存量 open PR 及其 Head SHA 录入基线快照,严禁触发审查与发送 ACK 评论。只有后续周期中新产生的提交或新打开的 PR 才触发自动化工作流。 2. 并发限流控制: - 引入 tokio::sync::Semaphore,将并发自动审查任务数严格限制为最多 2 个,防止多任务并发导致守护进程资源暴增或触发模型 429 报错。 3. 磁盘 I/O 优化(心跳内存化): - 在内存缓存 LAST_POLLED_CACHE 中维护 last_polled_at,消除每 15 秒向 git-poll-state.json 的无用写盘,仅在发生真实业务状态变动(基线录入或新提交 SHA)时持久化写盘。 4. 适配器接口规范化(GitLab 与 Gitee): - 在 GitPlatformAdapter::update_comment 签名中正式追加 target_number: Option<u64>。 - 移除 GitLabAdapter 中 32 位的位拼接打包 hack,直接返回原生 Note ID 并安全更新对应 MR 的 Note,彻底消除 ID 截断损坏。 - 在 Gitee 评论更新中实现动态路由,支持在 /pulls/comments/ 与 /issues/comments/ 间智能回退。 5. GitHub 轮询查询优化: - 在 fetch_active_pull_requests 中追加 sort=updated&direction=desc&per_page=50,避免活跃仓库产生分页饥饿。 6. 分支 Rebase 与验证: - 基于最新 upstream/beta 完成 rebase 并消除冲突。 - 增加 test_baseline_seeding_prevents_thundering_herd 与 test_in_memory_heartbeat_updates 单元测试。17/17 git_adapter 测试与 427/427 jeikcode-daemon 测试全部通过。 Co-Authored-By: JeikCode <code@jeikcode.top>
|
Hi @jeikl, Thank you for the detailed and constructive review roadmap! We have resolved all merge conflicts against
Ready for re-review! 嗨 @jeikl, 非常感谢您详尽而富有建设性的路线图指导!我们已基于最新的
欢迎随时复核! |
🔍 JeikCode PR 代码深度审查报告 (Code Review Report)评审对象:PR #61: feat(git): add background polling engine and native gitlab and gitee platform adapters 一、 目标分支演进比对与前期审查落实核验1. 前期审查改进点落实核验 (Commit
|
…platform adapters Implement proactive background pull request polling and native platform adapters for GitLab and Gitee to support automated code reviews in corporate intranets and developer environments behind NAT/firewalls. Key changes: 1. Extend `GitPlatformAdapter` trait with `fetch_active_pull_requests` to enable automated PR/MR discovery in polling mode across platforms. 2. Implement `GitLabAdapter` (`gitlab.rs`) handling `X-Gitlab-Token` webhook verification, `Merge Request Hook` and `Note Hook` payload parsing, URL-encoded project routing, and GitLab v4 REST APIs (notes, MRs, and connectivity test). 3. Implement `GiteeAdapter` (`gitee.rs`) handling `X-Gitee-Token` verification, PR and note payload parsing (prioritizing comment note parsing over PR switch to prevent comment payloads with PR objects from being misclassified), and Gitee v5 REST APIs. 4. Implement `fetch_active_pull_requests` for `GitHubAdapter` (`github.rs`). 5. Build the background Git Polling Engine in `polling.rs` with persistent state tracking (`git-poll-state.json`), head commit SHA deduplication, and automated review dispatch via `execute_git_workflow`. 6. Export `execute_git_workflow` in `api_git_webhook.rs` and wire the polling engine into the daemon startup and graceful shutdown lifecycle in `lib.rs`. 7. Add 15 unit tests across `git_adapter` verifying webhook token verification, MR/PR payload parsing, and deduplication logic. --- 实现后台 Pull Request 主动轮询引擎以及 GitLab 与 Gitee 的原生平台驱动, 为企业内网、VPN 与 NAT 防火墙后无法接收公网 Webhook 回调的本地开发环境 提供完整的自动化代码审查与讨论功能。 主要变更: 1. 在 `GitPlatformAdapter` trait 中扩展 `fetch_active_pull_requests`, 统一各平台在轮询模式下的活跃 PR/MR 查询规范。 2. 实现 `GitLabAdapter` (`gitlab.rs`),支持 `X-Gitlab-Token` 验签、`Merge Request Hook` 与 `Note Hook` 归一化事件解析、URL 编码项目路由 (`path_with_namespace`),以及 GitLab v4 REST API(发表/更新评论、查询活跃 MR 与连通性测试)。 3. 实现 `GiteeAdapter` (`gitee.rs`),支持 `X-Gitee-Token` 验签、PR 与 Note 事件解析 (优先解析 Note 评论,防止包含 `pull_request` 上下文对象的评论误判为 PR 事件), 以及 Gitee v5 REST API。 4. 在 `GitHubAdapter` (`github.rs`) 中实现 `fetch_active_pull_requests`。 5. 在 `polling.rs` 中构建后台 Git 轮询引擎,基于 `~/.jeikcode/git-poll-state.json` 持久化追踪各仓库 PR 的 `head_sha`,实现精准防重复审查去重与新提交检测。 6. 在 `api_git_webhook.rs` 中将 `execute_git_workflow` 暴露为 crate 可见, 并在 `lib.rs` 的服务启动与优雅停机生命周期中挂载 `start_git_polling_engine`。 7. 增加 15 组完备的单元测试,覆盖各平台 Token 验签、事件解析与轮询去重逻辑。 Co-Authored-By: JeikCode <code@jeikcode.top>
…te limiting, and adapter purity Address review feedback on PR jeikl#61: implement baseline seeding, concurrency rate limiting, in-memory heartbeats, GitLab Note ID purity, and dynamic Gitee routing. 1. Prevent Thundering Herd via Baseline Seeding: - Added initialized flag to RepoPollState and implemented seed_repo_baseline and is_repo_initialized. - When polling an existing repository for the first time, all existing open PRs and their head SHAs are recorded purely as historical baselines without triggering reviews or emitting ACK comments. Only newly opened PRs or commit pushes in subsequent cycles trigger review workflows. 2. Concurrency Rate Limiting: - Introduced a tokio::sync::Semaphore capping concurrent automated review workflows to 2 parallel tasks, preventing unconstrained spawns from exhausting daemon resources or triggering provider 429 errors. 3. Disk I/O Optimization via In-Memory Heartbeats: - Maintained LAST_POLLED_CACHE in memory for high-frequency last_polled_at timestamps. Eliminated periodic 15-second disk writes to git-poll-state.json, persisting to disk only upon genuine state mutations (baseline seeding or new PR/SHA updates). 4. Adapter Architecture Purity (GitLab & Gitee): - Formally updated GitPlatformAdapter::update_comment signature with target_number: Option<u64>. - Eliminated the 32-bit bitmasking hack in GitLabAdapter, returning raw Note IDs and cleanly updating MR notes using mr_iid and comment_id. - Implemented dynamic routing in GiteeAdapter to gracefully route between /pulls/comments/ and /issues/comments/. 5. Enhanced GitHub Polling Query: - Added sort=updated&direction=desc&per_page=50 to fetch_active_pull_requests to prevent pagination starvation on active repositories. 6. Rebase & Verification: - Rebased cleanly onto upstream/beta, resolving merge conflicts. - Added unit tests test_baseline_seeding_prevents_thundering_herd and test_in_memory_heartbeat_updates. 17/17 git_adapter tests and 427/427 jeikcode-daemon tests pass. --- 落实 PR jeikl#61 审查意见:实现基线快照初始化、并发限流、内存心跳、GitLab Note ID 规范化以及 Gitee 动态路由。 1. 基线快照初始化(防止海啸审查): - 在 RepoPollState 中增加 initialized 标记,并实现 seed_repo_baseline 与 is_repo_initialized。 - 当仓库首次接入主动轮询时,仅对存量 open PR 及其 Head SHA 录入基线快照,严禁触发审查与发送 ACK 评论。只有后续周期中新产生的提交或新打开的 PR 才触发自动化工作流。 2. 并发限流控制: - 引入 tokio::sync::Semaphore,将并发自动审查任务数严格限制为最多 2 个,防止多任务并发导致守护进程资源暴增或触发模型 429 报错。 3. 磁盘 I/O 优化(心跳内存化): - 在内存缓存 LAST_POLLED_CACHE 中维护 last_polled_at,消除每 15 秒向 git-poll-state.json 的无用写盘,仅在发生真实业务状态变动(基线录入或新提交 SHA)时持久化写盘。 4. 适配器接口规范化(GitLab 与 Gitee): - 在 GitPlatformAdapter::update_comment 签名中正式追加 target_number: Option<u64>。 - 移除 GitLabAdapter 中 32 位的位拼接打包 hack,直接返回原生 Note ID 并安全更新对应 MR 的 Note,彻底消除 ID 截断损坏。 - 在 Gitee 评论更新中实现动态路由,支持在 /pulls/comments/ 与 /issues/comments/ 间智能回退。 5. GitHub 轮询查询优化: - 在 fetch_active_pull_requests 中追加 sort=updated&direction=desc&per_page=50,避免活跃仓库产生分页饥饿。 6. 分支 Rebase 与验证: - 基于最新 upstream/beta 完成 rebase 并消除冲突。 - 增加 test_baseline_seeding_prevents_thundering_herd 与 test_in_memory_heartbeat_updates 单元测试。17/17 git_adapter 测试与 427/427 jeikcode-daemon 测试全部通过。 Co-Authored-By: JeikCode <code@jeikcode.top>
…lign polling parameters Add dynamic fallback routing between pull requests and issues for comment creation and retrieval in GitLab and Gitee adapters, align polling sort and pagination query parameters, and enforce the enabled check in the polling engine loop. - Check !config.enabled at the head of poll_cycle_tick in polling.rs to ensure disabled repositories are skipped. - Support dynamic endpoint fallback in GitLabAdapter::post_comment, update_comment, and find_bot_primary_comment between /merge_requests/ and /issues/ paths. - Support dynamic endpoint fallback in GiteeAdapter::post_comment and find_bot_primary_comment between /pulls/ and /issues/ paths. - Align polling query parameters across all platforms with sort=updated&direction=desc&per_page=50 (or order_by=updated_at&sort=desc&per_page=50 for GitLab). --- 在 GitLab 与 Gitee 适配器中为评论创建与检索增加 Pull Request 与 Issue 间的动态备用路由,对齐轮询排序与分页查询参数,并在轮询引擎循环中强化启用状态校验。 - 在 polling.rs 的 poll_cycle_tick 循环头部增加 !config.enabled 校验,确保已禁用的仓库配置不会触发远程轮询。 - 在 GitLabAdapter::post_comment、update_comment 与 find_bot_primary_comment 中支持 /merge_requests/ 与 /issues/ 之间的动态端点回退。 - 在 GiteeAdapter::post_comment 与 find_bot_primary_comment 中支持 /pulls/ 与 /issues/ 之间的动态端点回退。 - 将所有平台的轮询查询参数统一对齐为 sort=updated&direction=desc&per_page=50(GitLab 为 order_by=updated_at&sort=desc&per_page=50),防止长列表分页饥饿。 Co-Authored-By: JeikCode <code@jeikcode.top>
e34ef12 to
f3e5635
Compare
|
Hi @jeikl, All review items have been resolved and the branch has been cleanly rebased onto Summary of Completed Milestones & Fixes
The branch is ready for final review and merge into 嗨 @jeikl, 所有审查建议已全部落实闭环,分支已基于最新的 完成项与整改总结
分支已准备就绪,欢迎进行最终复核并合入 Co-Authored-By: JeikCode code@jeikcode.top |
|
@bot review the latest commit |
🔍 JeikCode PR Deep Review Report: Latest Commit AnalysisReview Target: PR #61: feat(git): add background polling engine and native gitlab and gitee platform adapters 1. Verification of Latest Commit (
|
🏁 JeikCode PR 状态与合流终审报告 (PR Status & Final Review Report)评审对象:PR #61: feat(git): add background polling engine and native gitlab and gitee platform adapters 一、 现有方案与功能落实核验结论经对目标分支最新代码提交
二、 历史审查反馈收敛全貌
三、 最终问题评级与跟进状态
🏁 JeikCode PR Status & Final Review ReportReview Target: PR #61: feat(git): add background polling engine and native gitlab and gitee platform adapters 1. Verification of Implemented Features & Technical SpecificationsIn-depth inspection of merge commit
2. Review Feedback Resolution Matrix
3. Final Issue Prioritization & Project Status
🌟 Generated by JeikCode |
变更类型
基于的分支与版本号
基于
upstream/beta(v7.2.5-beta.4)当前问题、对用户使用体验会产生什么问题或影响
在 PR #60 中,我们完成了 GitHub Webhook 的真实模型审查驱动串联与基础验签支持。然而在实际企业级内网研发与私有化部署中:
polling)」的选项,但后台缺少常驻轮询驱动引擎,导致该模式形同虚设。X-Gitlab-Token、X-Gitee-Token)、事件解析(GitLabMerge Request Hook/Note Hook、GiteePull Request Hook/Note Hook)以及相应的 REST API 操作(URL 编码项目路径、评论发布与更新)。解决方案
GitPlatformAdapter规范:增加fetch_active_pull_requests异步方法,各平台适配器提供原生开放 PR/MR 列表查询与标准事件归一化。GitLabAdapter(gitlab.rs):X-Gitlab-Token请求头安全验签与放行控制。Merge Request Hook和Note Hook,自适应提取 MR 标题、分支、Head SHA、评论内容与上下文。group/subgroup/project的 URL 编码、私有令牌鉴权、评论发布与看板更新、连通性探测)。GiteeAdapter(gitee.rs):X-Gitee-Token请求头验签。pull_request上下文对象的评论误判为 PR 开关事件。GitHubAdapter中实现fetch_active_pull_requests:完善 GitHub 原生活跃 PR 列表提取。polling.rs):sync_mode: GitSyncMode::Polling的绑定。~/.jeikcode/git-poll-state.json中持久化记录每个仓库 PR 的最后处理head_sha。api_git_webhook::execute_git_workflow,保持与 Webhook 完全一致的审查流程、动态提示词注入与看板式评论更新。lib.rs启动时通过start_git_polling_engine启动后台守护任务,并在停机信号到来时优雅中止 (abort())。提交清单(组内独立、可单独回退)
46cb63128feat(git): add background polling engine and native gitlab and gitee platform adapters每项提交引起的变化和影响面
crates/jeikcode-daemon/src/git_adapter/mod.rs:扩展fetch_active_pull_requests、get_all_bindings,并将平台工厂分发至GitLabAdapter与GiteeAdapter。crates/jeikcode-daemon/src/git_adapter/github.rs:新增fetch_active_pull_requests实现。crates/jeikcode-daemon/src/git_adapter/gitlab.rs:新增完整的 GitLab 平台适配器实现与单测。crates/jeikcode-daemon/src/git_adapter/gitee.rs:新增完整的 Gitee 平台适配器实现与单测。crates/jeikcode-daemon/src/git_adapter/polling.rs:新增后台轮询引擎与去重状态管理。crates/jeikcode-daemon/src/api_git_webhook.rs:将execute_git_workflow暴露为pub(crate)供轮询引擎复用。crates/jeikcode-daemon/src/lib.rs:挂载后台轮询任务与优雅退出。自检(与 CI 相同的命令)
cargo fmt -- --checkcargo test -p jeikcode-daemon --lib425/425 测试全量通过(含 15 项新增及现有git_adapter专属单测)。署名
Co-Authored-By: JeikCode code@jeikcode.top