添加geo监测功能，以及部分ai运维能力。#43
Closed
majin72 wants to merge 61 commits into
Closed
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
…al-time updates - Introduced a new job chaining mechanism for URL imports, allowing for sequential processing of tasks. - Added real-time updates via Echo for better user feedback during URL import operations. - Updated environment configuration to support new job timeout settings and queue management. - Modified entrypoint behavior to disable automatic migrations by default, enabling explicit control during container initialization. - Enhanced the UI with retry functionality for failed URL imports, improving user experience. This commit significantly improves the URL import workflow and user interaction with the system.
…tion - Updated the production Dockerfile to use Aliyun mirrors for faster package installations. - Added necessary tools for building PHP extensions and improved the apt repository configuration. - Introduced new environment variables for Reverb configuration to synchronize with frontend settings. These changes optimize the build process and ensure consistent configuration across environments.
…iguration - Added Vite build arguments to the production Dockerfiles and docker-compose configuration to ensure frontend environment variables are correctly injected during the build process. - Updated the docker-compose.prod.yml to reference the new build arguments for all relevant services. These changes enhance the integration between the frontend and backend, ensuring that necessary environment variables are available at build time.
…ndling - Modified the Reverb service command in docker-compose.prod.yml to remove hardcoded port, allowing the port to be defined via the REVERB_SERVER_PORT environment variable. - Updated the port mapping to reflect the dynamic port configuration. - Enhanced the Nginx configuration to proxy WebSocket connections to the Reverb service, ensuring compatibility with the new port settings. These changes improve flexibility in service configuration and maintain alignment with the environment settings.
…and integration - Added a new external fetch feature to allow the use of a local opencli browser for scraping content from anti-bot sites. - Introduced configuration options for enabling the external fetch, setting the bridge endpoint, and managing fallback HTTP status codes. - Updated the URL import processing service to utilize the external fetch service, enhancing the robustness of the URL import process. - Added new language entries for both English and Chinese to support the external fetch feature in the admin interface. - Updated the admin site settings view to include a link to the new external fetch configuration page. These changes significantly enhance the URL import capabilities, providing a more flexible and reliable method for fetching content.
- Updated docker-compose configurations to include `extra_hosts` for resolving `host.docker.internal` to the host gateway IP. - This change facilitates access to the external-fetch-bridge running on the host machine from within the Docker containers, enhancing the integration of local and remote services. - Added documentation in the external-fetch-bridge README to explain the setup and usage of SSH reverse tunneling with autossh for seamless connectivity. These updates improve the functionality and usability of the external-fetch feature in a Dockerized environment.
- Updated docker-compose.prod.yml to include additional network configurations for services, allowing simultaneous access to both default and host_access networks. - Improved comments for clarity on the purpose of the host.docker.internal resolution and its integration with the external-fetch-bridge. - This change enhances container communication and bypasses Docker's isolation rules, facilitating better connectivity between services.
- Removed unnecessary network configurations from docker-compose.prod.yml, streamlining the setup for container communication. - Updated comments to clarify the usage of host.docker.internal and recommended practices for accessing host services in Linux environments. - These changes enhance the readability and maintainability of the Docker configuration while ensuring effective service connectivity.
- Introduced a comprehensive deployment guide for the external-fetch-bridge, detailing the setup process for connecting a local macOS browser scraping service to a remote Linux server's GEOFlow queue container. - Included step-by-step instructions covering pre-deployment requirements, deployment procedures, and troubleshooting tips for common issues related to network configurations and firewall settings. - Enhanced documentation with clear explanations of the underlying network architecture and security considerations, ensuring users can effectively bypass access restrictions on various platforms.
- Changed REVERB_PORT from 18081 to 18080 for alignment with service configuration. - Updated REVERB_BROADCAST_PORT from 18080 to 8080 to reflect new port settings. - Modified REVERB_SERVER_PORT from 18080 to 8080 to ensure uniformity across environment variables.
- Added REVERB_SERVER_PATH to .env.example and .env.prod.example for specifying the server path. - Updated VITE_REVERB_PATH in frontend configurations to utilize the new server path variable. - Modified Docker configurations to include VITE_REVERB_PATH for consistent environment variable injection. - Adjusted Nginx configuration to route WebSocket connections through the specified server path, improving compatibility with existing routes. - Updated Echo.js to dynamically set the WebSocket path based on the new environment variable.
- Introduced new fields for ICP filing number, police filing number, and police filing record code in the site settings. - Updated validation rules and error messages for the new fields. - Enhanced the admin site settings view to include input fields for the new information. - Added corresponding translations for English and Chinese to support the new fields in the admin interface. - Updated footer templates to conditionally display the filing information based on the new settings.
- Added a new AI Ops section in the admin interface for managing AI-driven tasks and sessions. - Implemented routes and controllers for handling AI Ops sessions and chat interactions. - Integrated Tavily article search functionality, allowing AI-generated content to utilize web search for up-to-date information. - Updated site settings to include configuration options for enabling and managing the article search feature. - Enhanced language files for both English and Chinese to support new AI Ops and article search terminology. - Modified the admin header and site settings views to include links to the new AI Ops and article search configurations.
- Introduced a new approval mechanism for high-risk write operations in the AI Ops feature, requiring admin confirmation before execution. - Enhanced the AdminOpsAdminActionTool and AdminOpsSitePatchBasicsTool to evaluate risks and create pending approvals. - Updated AdminAiOpsController to handle tool approval and rejection processes, including new routes for approving and rejecting tool actions. - Added UI components for displaying pending approvals and capturing admin responses in the admin interface. - Enhanced language files to support new approval-related terminology in both English and Chinese. - Updated the AdminAiOpsRun model to track tool approval records and their statuses.
- Created a directory and an empty SQLite database file during the Docker image build process to ensure the Laravel application has a default database connection. - Updated comments to clarify the purpose of the changes related to the database setup.
- Changed base image to composer:2-php8.4-bookworm to meet project dependencies in composer.json and lock file. - Updated Debian package sources to use Aliyun mirrors for improved download speeds. - Configured Composer to use Aliyun Packagist for faster package retrieval. - Ensured ext-pcntl is installed to align with runtime requirements for Horizon and lock file checks.
…oser integration - Updated base image to php:8.4-cli-bookworm for consistency with runtime PHP version. - Improved package installation process by adding necessary dependencies and removing unnecessary files. - Copied Composer binary from official image to streamline Composer usage. - Configured Composer to use Aliyun Packagist for enhanced performance in package retrieval.
- Changed default output path for the static sitemap to storage/app/public/sitemap.xml for better compatibility with Docker. - Added methods in SiteStaticSitemapBuilder for handling chunked sitemap files and validating segment names. - Updated console command and related tests to reflect the new output path. - Adjusted Nginx configuration to serve sitemap files from the storage volume. - Enhanced documentation and comments for clarity on sitemap generation process.
…dencies - Updated .env.prod.example to include permissions guidance for the ./storage directory to ensure proper write access for the PHP-FPM user in Docker. - Modified docker-compose.prod.yml to reflect the updated storage permissions requirements for Linux hosts. - Added new dependencies for dompurify and marked in package.json and package-lock.json to enhance content sanitization and markdown parsing capabilities. - Updated Vite configuration to include a new JavaScript file for article preview functionality. - Enhanced AdminAiOpsChatAgent.php with detailed instructions for creating article generation tasks. - Improved error handling in ImageLibraryController for image uploads, now supporting additional formats (AVIF, BMP). - Updated language files to reflect new upload format support and improved user guidance. - Refactored AdminOpsMirrorTasksHandler to streamline boolean flag parsing for task creation options.
…n and session management - Added a new JavaScript file for AI Ops in Vite configuration to support additional features. - Updated AdminAiOpsChatAgent with detailed instructions for article generation tasks and integrated web search capabilities. - Implemented session deletion functionality in AdminAiOpsController, allowing admins to remove specific chat sessions. - Enhanced TavilyWebSearchTool to support AI Ops context and improved error handling for API key configuration. - Updated language files to include new terms related to web search and session management. - Improved the admin interface to display web search configuration options and hints when the API key is missing. - Added tests for new session deletion and web search functionalities to ensure proper behavior.
…ment - Implemented a new approval mechanism for various write operations, requiring admin confirmation before execution. - Updated AdminAiOpsChatAgent with detailed instructions for site settings modifications, including the use of site_name instead of site_title. - Enhanced AdminOps tools to utilize a write guard for pending approvals, ensuring actions are only executed upon admin consent. - Added functionality to normalize site_title to site_name in patch requests, improving consistency in site settings updates. - Updated language files to reflect new approval phases and error messages related to tool execution. - Improved the admin interface to display tool approval statuses and provide clearer feedback on actions taken. - Added tests to verify the correct handling of site title aliases and tool approval processes.
… response - Removed the normalizePatchKeys method from AdminOpsSitePatchBasicsTool, centralizing the normalization logic in AdminOpsSiteWriteService. - Updated AdminAiOpsController to include executed_ok in the approval response, indicating the success of the executed tool. - Added tool_call_id to AdminAiOpsToolApproval model for better tracking of tool execution. - Enhanced patchBasics method in AdminOpsSiteWriteService to utilize the new normalization method. - Cleaned up unused methods and improved overall code clarity. - Added tests to verify the correct behavior of the approval process and site patch handling.
…nt functionality - Introduced AdminOpsFetchUrlTool to enable fetching external URLs for reference by agents. - Updated AdminAiOpsChatAgent with detailed instructions for using the new tool, including guidelines for handling responses and potential errors. - Enhanced AdminAiOpsChatService to conditionally include the URL fetch tool based on configuration settings. - Added configuration options for external URL fetching in geoflow.php, including timeout and response size limits. - Improved overall documentation and comments for clarity on the new functionality and its intended use cases.
- Integrated AdminAiOpsUtf8::sanitizeString for improved string sanitization across multiple services and controllers. - Updated AdminAiOpsRun model to utilize Utf8SafeArrayCast for better handling of array attributes. - Implemented normalization of response body encoding in AdminAiOpsUrlFetchService to ensure UTF-8 compliance. - Enhanced various methods in AdminAiOpsAssistantTimelineRecorder to sanitize tool call arguments and error messages. - Improved overall data integrity and security by sanitizing outputs before processing or returning them.
- Changed the base image in the production Dockerfile from nginx:1.27-alpine to nginx:1.30.1-alpine for improved performance and security features.
…actions - Updated AdminAiOpsChatAgent with revised instructions for site management tools, improving clarity on usage. - Removed the deprecated AdminOpsAdminActionTool, centralizing admin actions within specific tools for better organization. - Introduced new methods in AdminAiOpsAssistantTimelineRecorder to manage tool approval states, ensuring only relevant tools are marked for approval. - Enhanced the AdminAiOpsToolApprovalService to utilize a more flexible execution mechanism for approved tools. - Updated various services and controllers to reflect changes in tool handling and approval workflows. - Improved language files to include new messages related to tool approval and rejection scenarios. - Added tests to verify the correct behavior of the new approval mechanisms and tool state management.
…terface - Updated AdminAiOpsChatAgent with clearer instructions for task management tools, emphasizing the use of specific fields for task creation. - Enhanced AdminAiOpsToolApprovalService to allow for multiple pending approvals without interrupting the agent flow, returning detailed JSON responses for pending actions. - Improved AdminAiOpsRunService to include the count of remaining approvals in the response payload, enhancing the admin interface's feedback on approval status. - Refined the handling of tool phases in the JavaScript frontend, ensuring proper transitions and visual feedback for tool approval states. - Updated language files to include new messages related to the approval queue and processing status. - Added tests to verify the correct behavior of the new approval mechanisms and tool state management.
…feedback - Replaced AdminAiOpsToolRiskEvaluator and AdminAiOpsToolApprovalService with AdminAiOpsPendingWriteGuard to manage write operations requiring admin approval. - Updated AdminOpsSitePatchBasicsTool to utilize the new write guard, simplifying the approval flow and ensuring actions are executed only upon admin consent. - Enhanced AdminAiOpsController with improved documentation and clearer instructions for session management and tool approval processes. - Refined the JavaScript frontend to handle tool approval states more effectively, providing better user feedback during approval actions. - Added tests to verify the correct behavior of the new approval mechanisms and ensure proper handling of pending write operations.
…ation - Eliminated NGINX_IMAGE variable from .env.prod.example and docker-compose.prod.yml for cleaner configuration. - Updated Dockerfile.prod to directly use the nginx:1.30.1-alpine image, ensuring consistency across the setup.
Resolve conflicts by keeping dev URL import queue settings alongside main outbound HTTP proxy configuration and knowledge retrieval worker wiring. Co-authored-by: Cursor <cursoragent@cursor.com>
…rker wiring to align with dev URL import queue settings.
…cker profile Introduce admin-managed projects, accounts, probes, and maintenance flows backed by a Python sidecar. Unify sidecar deployment via compose profile geo-monitor for optional containerized Chromium and noVNC. Co-authored-by: Cursor <cursoragent@cursor.com>
…ium integration Add support for using Aliyun mirrors for pip installations in the geo-monitor sidecar. Update Dockerfiles and environment configurations to streamline the build process and improve performance by skipping unnecessary downloads. Introduce new environment variables for pip index URL and trusted host, and update documentation to reflect these changes.
…process Remove BuildKit-specific commands from the sidecar Dockerfile to ensure compatibility with ECS classic builders. Update documentation to clarify build commands and provide guidance for enabling BuildKit for faster builds. This enhances the overall build experience and streamlines the setup process.
…p process Update the Dockerfile to include the installation of `scrapling[fetchers]` for improved functionality. Refactor the main script to dynamically import CLI commands, enhancing modularity. Introduce a cleanup function in the common script to manage stale X display states after container restarts, preventing potential conflicts. Update the start script to call this cleanup function, ensuring a smoother operation of the noVNC service.
Change the GEO_MONITOR_NOVNC_BIND environment variable from 127.0.0.1 to 0.0.0.0 across multiple Docker configurations to enable external access for noVNC. This adjustment ensures proper Docker port mapping while maintaining host restrictions. Additionally, update related documentation to reflect these changes.
…tartup Change the execution command for starting noVNC in the entrypoint-sidecar.sh script from a direct call to './scripts/novnc/start-novnc.sh' to 'sh ./scripts/novnc/start-novnc.sh'. This ensures proper execution permissions and compatibility in the Docker environment.
…bility Update the shebang in entrypoint-sidecar.sh from '#!/bin/sh' to '#!/usr/bin/env bash' for improved compatibility with bash-specific features. Additionally, change the command in the Dockerfile to use 'bash' instead of 'sh' for executing the entrypoint script, ensuring consistent behavior in the Docker environment.
…eam configuration Add a resolver directive to handle dynamic IP changes in Docker containers, preventing 502 errors. Refactor upstream definitions for reverb and PHP services to enhance clarity and maintainability.
…tion Add new environment variables for Docker paths and SSH host configuration in the .env.prod.example file. Update GeoMonitoringAccountController to catch all throwable exceptions. Refactor maintenance steps in GeoMonitorMaintenanceService to improve clarity and support for interactive browser sessions. Enhance the maintenance page to display SSH tunnel commands and noVNC URLs based on runtime conditions. Update language files for improved descriptions and instructions related to headless Linux and interactive sessions. Add tests to verify the new functionality and ensure proper rendering of maintenance steps.
Change the GEO_MONITOR_EVIDENCE_ROOT environment variable in multiple Docker configurations to point to "/app/evidence/sidecar" for improved organization. Introduce a new method in GeoMonitorProbePersister to normalize evidence paths, ensuring compatibility with the updated structure. Enhance GeoMonitorEvidencePathResolver to handle legacy paths and add tests for path normalization functionality.
…ecar integration Refactor the evidence directory paths in multiple configuration files to unify them under 'storage/app/geo-monitor/evidence', ensuring compatibility with Docker mounts. Introduce a method in AppServiceProvider to ensure the existence of the evidence directory. Update the GeoMonitorEvidencePathResolver to handle new paths and legacy structures, and enhance permissions management in the sidecar entrypoint script. Add tests to verify the new path resolution logic.
…ermissions Update the Docker entrypoint scripts to automatically create and set permissions for the evidence directory used by the geo-monitoring feature. Ensure that the directory is created with appropriate ownership and permissions for the www-data user. Modify the AppServiceProvider to verify the existence and permissions of the evidence directory. Adjust documentation to reflect that the evidence directory is now automatically managed by the Docker entrypoint, eliminating the need for manual creation.
…or sidecar debugging Revise the docker-compose.novnc.yml file to clarify its purpose for sidecar debugging without the need for profiles. Update comments to reflect the current usage and provide clearer instructions for running the service. This change enhances the documentation and usability of the sidecar configuration.
Add a new relationship for project scheduling in the GeoMonitorProject model, allowing for a single schedule per project. Update the GeoMonitoringController to include schedule data when displaying project details. Introduce a method in the GeoMonitorRunService to evaluate alerts after a run completes, enhancing the monitoring capabilities. Additionally, update the .env.prod.example file with new configuration instructions for alerts and notifications.
Resolve WorkerExecutionService conflict: keep Tavily article search tools from dev and max_tokens truncation detection from main. Co-authored-by: Cursor <cursoragent@cursor.com>
Introduce new environment variables for public noVNC access in .env.example and .env.prod.example, allowing users to enable public access without SSH tunneling. Update GeoMonitorMaintenanceService to handle public noVNC configurations and adjust the maintenance page to display relevant URLs and authentication modes. Enhance documentation to reflect these changes and provide clearer instructions for public access setup. Additionally, update Docker configurations to support the new environment variables.
…stallation Modify the Dockerfile to use a configurable APK mirror for package installations, ensuring compatibility with both domestic and overseas builds. This change enhances the build process by allowing users to specify a mirror, improving reliability and speed during image creation.
…guration Add support for geo-monitor noVNC public access by creating a new snippet configuration file. Update the Dockerfile to ensure the snippet directory is created and initialized. Modify the entrypoint script to generate the location configuration based on environment variables, enhancing flexibility for public access management.
Update the Nginx configuration to include the geo-monitor noVNC snippet before static resource locations, ensuring proper handling of CSS and JS files. Modify the Dockerfile to remove the obsolete geo-monitor configuration file and streamline the snippet creation process. Adjust the entrypoint script to use a precise location directive for the noVNC path, enhancing routing accuracy.
…outing Add a new environment variable for the noVNC public path in the Docker Compose file, improving flexibility for public access. Update the GeoMonitorNovncConfig class to generate the public VNC URL with WebSocket parameters for better connectivity. Modify the Nginx entrypoint script to dynamically set location directives based on the public path, ensuring accurate routing. Update tests to reflect changes in the public URL structure.
…esolution Implement improvements to the noVNC session management by adding a method to resolve the public noVNC URL in the GeoMonitorMaintenanceService. Update the GeoMonitoringAccountController and GeoMonitoringMaintenanceController to include the public noVNC URL in the session data upon account creation and browser launch. Modify the maintenance page to automatically open the noVNC URL in a new tab if available. Additionally, update relevant tests to ensure proper functionality of the new features.
…nfiguration Refactor the GeoMonitorMaintenanceService to return the correct public noVNC URL. Update the Nginx entrypoint script to encode the WebSocket path for improved connectivity and add specific proxy directives for handling noVNC HTML requests. Adjust tests to verify the new URL structure and ensure proper session handling during browser launches.
…functionality Add destroy methods to GeoMonitoringAccountController, GeoMonitoringBrowserProfileController, and GeoMonitoringProxyController to handle the deletion of accounts, profiles, and proxies. Implement checks to ensure that deletions are only performed when there are no active observations or ongoing maintenance. Update routes and views to support deletion actions with confirmation prompts. Enhance language files for user feedback on deletion actions.
Temporarily disable the 'ai_ops' navigation entry in the admin header by commenting it out, while keeping other navigation items intact. This change is part of ongoing adjustments to the admin interface.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
geo监测功能
增加豆包，DeepSeek，元宝等平台的自动监测，在后台登录账号即可使用，详见tools/geo-monitor-poc
AI运维
ai运维的导航暂时隐藏了。仅内部体验使用。/resources/views/admin/partials/header.blade.php。
$menu = [
'dashboard' => ['route' => 'admin.dashboard', 'name' => __('admin.nav.dashboard')],
'analytics' => ['route' => 'admin.analytics', 'name' => __('admin.nav.analytics')],
'geo_monitoring' => ['route' => 'admin.geo-monitoring.index', 'name' => __('admin.nav.geo_monitoring')],
'tasks' => ['route' => 'admin.tasks.index', 'name' => __('admin.nav.tasks')],
'distribution' => ['route' => 'admin.distribution.index', 'name' => __('admin.nav.distribution')],
'articles' => ['route' => 'admin.articles.index', 'name' => __('admin.nav.articles')],
'materials' => ['route' => 'admin.materials.index', 'name' => __('admin.nav.materials')],
#'ai_ops' => ['route' => 'admin.ai-ops.index', 'name' => __('admin.nav.ai_ops')],
'ai_config' => ['route' => 'admin.ai.configurator', 'name' => __('admin.nav.ai_config')],
'site_settings' => ['route' => 'admin.site-settings.index', 'name' => __('admin.nav.site_settings')],
];
删除'ai_ops' => ['route' => 'admin.ai-ops.index', 'name' => __('admin.nav.ai_ops')],前的注释打开