feat: add toolset plugin

[AlinsRan]

Motivation

Diagnosing performance issues or memory leaks in a production APISIX deployment is difficult today:

• Enabling verbose logging restarts APISIX and affects all traffic.
• Adding observability requires route-level plugin changes that persist in etcd.
• There is no lightweight way to temporarily instrument request phases or monitor internal Lua state without side effects.

This PR introduces the toolset plugin — a low-overhead diagnostics framework that can be toggled at runtime without restarting APISIX or touching any route configuration.

Design

The toolset plugin acts as a container for lightweight sub-plugins. Sub-plugins are not configured via the APISIX Admin API; instead they are configured by editing a single Lua file (apisix/plugins/toolset/config.lua) on disk. The plugin polls this file every second and hot-reloads sub-plugins when a change is detected.

Key design choices:

• No per-route schema — the plugin always operates at global scope, so it instruments every request without touching routes or services.
• File-based configuration — avoids etcd round-trips and lets operators apply changes by editing a single file (or with a configuration management tool), with sub-second propagation.
• Dynamic load/unload — sub-plugins are loaded or unloaded at runtime; only sub-plugins with non-empty configuration are active.

Sub-plugins

trace

Instruments all APISIX request phases (access, balancer, upstream, header_filter, body_filter, log) and emits a formatted timing table to the error log for sampled requests that match the configured filters.

Features:

• Configurable sampling rate (N out of 100 requests)
• Host and path allowlist filtering with glob pattern support
• Recognises common trace headers (x-request-id, traceparent, sw8, x-b3-traceid) and attaches their values to the log line
• Optional UUID generation when no trace header is present (gen_uid)
• Minimum total-duration threshold to suppress fast requests from the log
• Capture of additional nginx/APISIX variables alongside the timing table

Attributes (trace):

Name	Type	Default	Description
rate	integer	1	Sampling rate as N-out-of-100. 1 = 1%; 100 = every request.
hosts	array	[]	Allowlist of Host header values (glob). Empty = all hosts.
paths	array	[]	Allowlist of request URI patterns (glob). Empty = all paths.
gen_uid	boolean	false	Generate a UUID trace ID when no standard trace header is found.
vars	array	[]	Extra nginx/APISIX variables to prepend to the trace output.
timespan_threshold	number	0	Minimum total request duration (seconds) before emitting the log.

table_count

Periodically measures and logs the entry count of specified Lua module tables. Useful for detecting table leaks or monitoring memory growth (e.g. router or plugin state that is expected to be stable).

Attributes (table_count):

Name	Type	Default	Description
lua_modules	array	—	Lua module paths to measure (e.g. ["apisix.router"]).
interval	integer	5	Seconds between measurements.
depth	integer	10	Recursion depth when counting sub-tables.
scopes	array	["worker","privileged agent"]	APISIX process types in which this sub-plugin runs.

Enable Plugin

Add toolset to the plugins list in config.yaml:

plugins:
  - toolset

Then edit apisix/plugins/toolset/config.lua to activate the desired sub-plugins. The default file ships with all sub-plugins disabled. Example full configuration:

return {
    trace = {
        rate = 10,              -- sample 10% of requests
        hosts = { "*.example.com" },
        paths = { "/api/*" },
        gen_uid = true,
        vars = { "remote_addr", "upstream_addr" },
        timespan_threshold = 0.5  -- only log requests slower than 500ms
    },
    table_count = {
        lua_modules = { "apisix.router" },
        interval = 30,
        depth = 5,
        scopes = { "worker" }
    }
}

Changes take effect within one second — no APISIX restart required.

To disable a sub-plugin at runtime, remove its key or set lua_modules to {} and save the file.

Example Usage

1. Tracing slow requests across a specific host

Edit apisix/plugins/toolset/config.lua:

return {
    trace = {
        rate   = 100,
        hosts  = { "api.example.com" },
        timespan_threshold = 1.0   -- only requests slower than 1s
    }
}

When a matching request exceeds 1 second, APISIX writes a timing table to the error log at WARN level:

[toolset/trace] uid=7f3a1c2d-... remote_addr=10.0.0.1
+----------+---------------------------+----------+-------------------------+
| Role     | Phase                     | Timespan | Start time              |
+----------+---------------------------+----------+-------------------------+
| APISIX   | access                    | 3ms      | 2024-01-01 12:00:00.123 |
| APISIX   | _match_route              | 1ms      | 2024-01-01 12:00:00.124 |
| APISIX   | balancer                  | 1ms      | 2024-01-01 12:00:00.125 |
| Upstream | upstream (req + response) | 1200ms   | 2024-01-01 12:00:00.126 |
| APISIX   | header_filter             | 0ms      | 2024-01-01 12:00:00.646 |
| APISIX   | body_filter               | 0ms      | 2024-01-01 12:00:00.646 |
| Client   | response                  | 1ms      | 2024-01-01 12:00:00.647 |
| APISIX   | log                       | 0ms      | 2024-01-01 12:00:00.648 |
+----------+---------------------------+----------+-------------------------+

2. Sampling traffic with path and trace-ID correlation

return {
    trace = {
        rate    = 5,               -- 5% of requests
        paths   = { "/api/v1/*" },
        gen_uid = true,            -- attach generated UID for tracing
        vars    = { "remote_addr" }
    }
}

The gen_uid option ensures every sampled request has a traceable ID even when the client does not send a trace header, making it easy to correlate the log entry with downstream access logs.

3. Monitoring Lua table growth

return {
    table_count = {
        lua_modules = { "apisix.router", "apisix.core.config_etcd" },
        interval    = 60,
        depth       = 5,
        scopes      = { "worker" }
    }
}

Every 60 seconds, each worker logs:

package apisix.router table count is: 1482 for loaded: 1
package apisix.core.config_etcd table count is: 347 for loaded: 1

A growing count between intervals indicates a potential table leak in that module.

Changes

File	Description
apisix/plugins/toolset/init.lua	Plugin entry point; 1-second timer for config hot-reload
apisix/plugins/toolset/config.lua	Default (disabled) sub-plugin configuration
apisix/plugins/toolset/src/trace.lua	trace sub-plugin implementation
apisix/plugins/toolset/src/table-count/init.lua	table_count sub-plugin implementation
conf/config.yaml.example	Added toolset entry (commented out, disabled by default)
docs/en/latest/plugins/toolset.md	English documentation
docs/zh/latest/plugins/toolset.md	Chinese documentation
t/plugin/toolset.t / t/plugin/trace*.t / t/plugin/table-count.t	Test suite

[Copilot]

Pull request overview

This PR introduces a new global toolset plugin intended as a diagnostics/observability host that can dynamically load/unload lightweight sub-plugins (notably trace and table_count), along with tests, docs, and packaging/config updates.

Changes:

• Added toolset plugin framework with sub-plugins for request phase timing logs (trace) and periodic Lua table size counting (table_count).
• Added t::APISIX tests for trace, table_count, and toolset reload/sync behavior.
• Updated defaults/docs/navigation and installation packaging to include the new plugin and its configuration examples.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
t/table-count-example.lua	Test helper module used by table_count tests (tables, circular refs, deep nesting).
t/plugin/trace.t	End-to-end tests for trace table output, sampling, and reload behavior.
t/plugin/trace.path.t	Tests glob path matching behavior for trace allowlist.
t/plugin/trace.host.t	Tests host matching behavior for trace allowlist.
t/plugin/trace.headers.t	Tests trace header detection, var capture, and UUID generation.
t/plugin/trace.dns.t	Tests DNS resolve phase instrumentation logging.
t/plugin/toolset.t	Tests toolset sync loop and reload behavior with config changes.
t/plugin/table-count.t	Tests periodic module table counting, circular detection, and depth enforcement.
Makefile	Installs toolset plugin Lua files into the runtime Lua dir.
docs/zh/latest/plugins/toolset.md	Chinese documentation for toolset plugin configuration and examples.
docs/zh/latest/config.json	Adds toolset doc entry to the Chinese docs sidebar/nav.
docs/en/latest/plugins/toolset.md	English documentation for toolset plugin configuration and examples.
docs/en/latest/config.json	Adds toolset doc entry to the English docs sidebar/nav.
conf/config.yaml.example	Adds toolset to plugin list (commented) and documents plugin_attr.toolset examples.
apisix/plugins/toolset/src/trace.lua	Implements trace sub-plugin by instrumenting APISIX phases and logging a timing table.
apisix/plugins/toolset/src/table-count/init.lua	Implements periodic Lua module table counting with depth/cycle handling and scope filtering.
apisix/plugins/toolset/init.lua	Toolset entry point: periodic sync loop and dynamic load/unload of sub-plugins.
apisix/plugins/toolset/config.lua	Default sub-plugin configuration values (currently loaded via require).
apisix/cli/config.lua	Adds default plugin_attr.toolset configuration in CLI defaults.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.