Repository navigation
Commit 91f4263
feat: Implement enum-based output formatters (Table, JSON, YARA) (#127)
* feat(output): add JSON, table, and YARA formatters
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* Implement table output formatter with TTY and plain modes
Adds a human-readable table output formatter for extracted strings, with automatic TTY detection. In TTY mode, outputs an aligned table with headers and truncated columns; in non-TTY mode, outputs plain string text, one per line. Includes dynamic column width calculation, tag formatting, Unicode-safe truncation, and comprehensive integration and unit tests with snapshot verification.
* Implement JSON and YARA output formatters
Added full implementations for `format_json` and `format_yara` functions, including serialization logic, escaping, and tag/group handling. Updated documentation and examples to reflect new UTF-16 extraction API and output formatting. Added comprehensive integration and snapshot tests for both output formats to ensure correctness and stability.
* feat(output): add generated_at timestamp to output metadata
- Introduced an optional `generated_at` field in `OutputMetadata` for deterministic outputs.
- Updated YARA formatter to utilize the `generated_at` timestamp instead of runtime timestamps.
- Adjusted integration tests and snapshots to reflect changes in generated timestamps.
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* Enable superpowers plugin in Claude settings
Added 'superpowers@claude-plugins-official' to the enabledPlugins list in .claude/settings.json.
* refactor(output): improve YARA formatter code quality and test coverage
Address issues identified during PR review:
- Rename misleading underscore-prefixed parameters (_strings, _metadata)
to strings, metadata since they are actively used
- Return "CLOCK_ERROR" instead of "0" on timestamp failure to make
errors clearly distinguishable from valid timestamps
- Remove dead code: get_yara_modifier() function and its tests after
inlining "ascii" constant for ASCII/UTF-8 encodings
- Simplify utf16be_hex_string() using iterator chains with flat_map()
- Add Tag import and simplify tag_name() function
Add comprehensive unit tests for UTF-16 encoding functions:
- escape_yara_unicode_literal: basic escapes, control chars, unicode
passthrough, empty string
- utf16be_hex_string: basic ASCII, empty string, non-ASCII BMP chars,
surrogate pairs
- escape_yara_string: additional control character coverage
- with_generated_at builder method validation
- Default timestamp fallback behavior
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(docs): clarify ASCII rule for Unicode handling
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* fix(reviews): clarify ASCII rule for Unicode punctuation
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* chore(settings): remove enabled plugins from configuration
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* chore(contributing): add contributing guidelines document
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* refactor: address code review findings and add project documentation
Immediate fixes:
- Fix failing doctests in extraction/mod.rs by wrapping in fn main()
- Fix rustdoc warning in patterns/ip.rs by escaping [::1]:8080
- Fix O(n^2) algorithms in dedup.rs using HashSet for unique detection
New features:
- Add OutputFormatter trait for extensible output formatting
- Add #[non_exhaustive] to OutputFormat enum for API stability
- Add Hash derive to Encoding and StringSource enums
Documentation:
- Create CHANGELOG.md following Keep a Changelog format
- Create ROADMAP.md documenting medium/long-term improvements
- Create examples/ directory with basic_extraction, network_indicators,
and output_formats examples
The O(n^2) fixes include:
- unique_sections detection now uses HashSet instead of Vec::contains
- unique_sources detection now uses HashSet instead of Vec::contains
- merge_tags now uses HashSet for deduplication
- Grouping key simplified from format!("{:?}", encoding) to Encoding
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* chore(devcontainer): add Rust devcontainer configuration
chore(dependabot): update schedules to weekly for various ecosystems
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* refactor(output): split table.rs into module directory
Split the oversized table.rs (708 lines) into a module directory
with four focused files, all under the 500-line project limit:
- table/mod.rs (120 lines): Public API, constants, entry points
- table/tty.rs (220 lines): TTY table rendering with column alignment
- table/plain.rs (96 lines): Plain text output for piping
- table/formatting.rs (326 lines): Shared utilities (truncate, pad, format_tags)
This addresses the ticket acceptance criterion requiring all files
to be under 500 lines.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(yara): correct UTF-16LE encoding and prevent injection attacks
Fix two issues in YARA output formatter:
1. UTF-16LE non-ASCII handling: The `wide` modifier only works for ASCII
characters. Non-ASCII UTF-16LE strings now use hex encoding instead
of the incorrect `wide` modifier approach.
- Add `utf16le_hex_string` helper (mirrors `utf16be_hex_string`)
- Branch on `is_ascii()` to choose appropriate encoding
2. Injection prevention: Escape `binary_name` and `timestamp` in comment
headers and meta fields to prevent newline/quote injection attacks.
Add 8 new tests covering:
- UTF-16LE hex string encoding (basic, empty, non-ASCII, surrogate pairs)
- ASCII UTF-16LE still uses wide modifier
- Non-ASCII UTF-16LE uses hex encoding
- Binary name injection escaping
- Timestamp injection escaping
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* chore(tests): add comprehensive testing strategy analysis
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* chore(devcontainer): update Docker features and remove unused ones
- Added 'moby' configuration to Docker feature
- Removed unused features from the devcontainer configuration
- Added mise.toml for tool dependencies
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* chore(setup): update setup commands and add mise installation
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* chore(cleanup): remove megalinter configurations and references
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* chore(mise): add node version to tools configuration
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* refactor(yara): split module to stay under 500-line limit
Extract YARA string escaping utilities to separate escaping.rs module:
- escape_yara_string, escape_yara_unicode_literal
- utf16be_hex_string, utf16le_hex_string
- All associated tests
The original yara.rs was 536 lines, now split into:
- yara/mod.rs (357 lines) - main formatting logic
- yara/escaping.rs (204 lines) - escaping utilities
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* chore(agents): fix formatting in critical rules section
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* docs(AGENTS): improve AI agent guidelines with fixes and additions
Fix formatting bug where rules 3 and 4 were merged on one line, correct
typos, add Rust version requirements, expand development commands, clarify
module structure, and document key dependencies.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: address PR review comments for output formatters
- Fix test metadata count mismatches in output_table_integration.rs
- Remove underscore prefix from used parameters in json.rs
- Use crate re-export for RankingConfig import
- Fix truncate_string edge case for wide multibyte characters
- Update table docs to reflect actual tag selection behavior
- Add control character sanitization for TTY output
- Use UTF-8 encoding for Unicode test content
- Add Claude Code AI assistance section to CONTRIBUTING.md
- Update snapshots for encoding and sanitization changes
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* chore: cleanup devcontainer, justfile, and AGENTS.md
- Remove duplicate rust-bundle extension, keep official rust-analyzer
- Refactor setup recipes to reuse mise-install recipe
- Fix markdown lint: convert emphasis-as-heading to list items
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor(justfile): use mise exec for all tool commands
Add mise_exec variable and prefix all tool commands (cargo, prettier,
pre-commit, actionlint, cspell, markdownlint, lychee, dist, mdbook,
goreleaser) with mise exec to ensure correct tool versions are used
regardless of shell activation state.
Also adds dotenv-load and ignore-comments settings for consistency
with other projects.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>1 parent 8fcbb47 commit 91f4263
91 files changed
Lines changed: 4900 additions & 253 deletions
File tree
- .claude
- .devcontainer
- .github
- .vscode
- examples
- src
- classification
- patterns
- extraction
- output
- table
- yara
- tests
- snapshots
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
3 | | - | |
4 | | - | |
| 2 | + | |
5 | 3 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
351 | 351 | | |
352 | 352 | | |
353 | 353 | | |
354 | | - | |
| 354 | + | |
355 | 355 | | |
356 | 356 | | |
357 | 357 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
| 16 | + | |
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | | - | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
124 | | - | |
125 | | - | |
126 | 124 | | |
127 | 125 | | |
128 | 126 | | |
129 | 127 | | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
11 | 10 | | |
12 | 11 | | |
13 | 12 | | |
| |||
26 | 25 | | |
27 | 26 | | |
28 | 27 | | |
29 | | - | |
30 | | - | |
| 28 | + | |
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
| 18 | + | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
25 | | - | |
26 | | - | |
| 26 | + | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
48 | 49 | | |
49 | 50 | | |
50 | 51 | | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
51 | 56 | | |
52 | 57 | | |
53 | 58 | | |
| |||
60 | 65 | | |
61 | 66 | | |
62 | 67 | | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
63 | 76 | | |
64 | 77 | | |
65 | 78 | | |
| |||
0 commit comments