Skip to content

Commit 91f4263

Browse files
unclesp1d3rclaude
andauthored
feat: Implement enum-based output formatters (Table, JSON, YARA) (#127)
* feat(output): add JSON, table, and YARA formatters Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * Implement table output formatter with TTY and plain modes Adds a human-readable table output formatter for extracted strings, with automatic TTY detection. In TTY mode, outputs an aligned table with headers and truncated columns; in non-TTY mode, outputs plain string text, one per line. Includes dynamic column width calculation, tag formatting, Unicode-safe truncation, and comprehensive integration and unit tests with snapshot verification. * Implement JSON and YARA output formatters Added full implementations for `format_json` and `format_yara` functions, including serialization logic, escaping, and tag/group handling. Updated documentation and examples to reflect new UTF-16 extraction API and output formatting. Added comprehensive integration and snapshot tests for both output formats to ensure correctness and stability. * feat(output): add generated_at timestamp to output metadata - Introduced an optional `generated_at` field in `OutputMetadata` for deterministic outputs. - Updated YARA formatter to utilize the `generated_at` timestamp instead of runtime timestamps. - Adjusted integration tests and snapshots to reflect changes in generated timestamps. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * Enable superpowers plugin in Claude settings Added 'superpowers@claude-plugins-official' to the enabledPlugins list in .claude/settings.json. * refactor(output): improve YARA formatter code quality and test coverage Address issues identified during PR review: - Rename misleading underscore-prefixed parameters (_strings, _metadata) to strings, metadata since they are actively used - Return "CLOCK_ERROR" instead of "0" on timestamp failure to make errors clearly distinguishable from valid timestamps - Remove dead code: get_yara_modifier() function and its tests after inlining "ascii" constant for ASCII/UTF-8 encodings - Simplify utf16be_hex_string() using iterator chains with flat_map() - Add Tag import and simplify tag_name() function Add comprehensive unit tests for UTF-16 encoding functions: - escape_yara_unicode_literal: basic escapes, control chars, unicode passthrough, empty string - utf16be_hex_string: basic ASCII, empty string, non-ASCII BMP chars, surrogate pairs - escape_yara_string: additional control character coverage - with_generated_at builder method validation - Default timestamp fallback behavior Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(docs): clarify ASCII rule for Unicode handling Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * fix(reviews): clarify ASCII rule for Unicode punctuation Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore(settings): remove enabled plugins from configuration Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore(contributing): add contributing guidelines document Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * refactor: address code review findings and add project documentation Immediate fixes: - Fix failing doctests in extraction/mod.rs by wrapping in fn main() - Fix rustdoc warning in patterns/ip.rs by escaping [::1]:8080 - Fix O(n^2) algorithms in dedup.rs using HashSet for unique detection New features: - Add OutputFormatter trait for extensible output formatting - Add #[non_exhaustive] to OutputFormat enum for API stability - Add Hash derive to Encoding and StringSource enums Documentation: - Create CHANGELOG.md following Keep a Changelog format - Create ROADMAP.md documenting medium/long-term improvements - Create examples/ directory with basic_extraction, network_indicators, and output_formats examples The O(n^2) fixes include: - unique_sections detection now uses HashSet instead of Vec::contains - unique_sources detection now uses HashSet instead of Vec::contains - merge_tags now uses HashSet for deduplication - Grouping key simplified from format!("{:?}", encoding) to Encoding Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore(devcontainer): add Rust devcontainer configuration chore(dependabot): update schedules to weekly for various ecosystems Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * refactor(output): split table.rs into module directory Split the oversized table.rs (708 lines) into a module directory with four focused files, all under the 500-line project limit: - table/mod.rs (120 lines): Public API, constants, entry points - table/tty.rs (220 lines): TTY table rendering with column alignment - table/plain.rs (96 lines): Plain text output for piping - table/formatting.rs (326 lines): Shared utilities (truncate, pad, format_tags) This addresses the ticket acceptance criterion requiring all files to be under 500 lines. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(yara): correct UTF-16LE encoding and prevent injection attacks Fix two issues in YARA output formatter: 1. UTF-16LE non-ASCII handling: The `wide` modifier only works for ASCII characters. Non-ASCII UTF-16LE strings now use hex encoding instead of the incorrect `wide` modifier approach. - Add `utf16le_hex_string` helper (mirrors `utf16be_hex_string`) - Branch on `is_ascii()` to choose appropriate encoding 2. Injection prevention: Escape `binary_name` and `timestamp` in comment headers and meta fields to prevent newline/quote injection attacks. Add 8 new tests covering: - UTF-16LE hex string encoding (basic, empty, non-ASCII, surrogate pairs) - ASCII UTF-16LE still uses wide modifier - Non-ASCII UTF-16LE uses hex encoding - Binary name injection escaping - Timestamp injection escaping Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore(tests): add comprehensive testing strategy analysis Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore(devcontainer): update Docker features and remove unused ones - Added 'moby' configuration to Docker feature - Removed unused features from the devcontainer configuration - Added mise.toml for tool dependencies Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore(setup): update setup commands and add mise installation Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore(cleanup): remove megalinter configurations and references Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore(mise): add node version to tools configuration Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * refactor(yara): split module to stay under 500-line limit Extract YARA string escaping utilities to separate escaping.rs module: - escape_yara_string, escape_yara_unicode_literal - utf16be_hex_string, utf16le_hex_string - All associated tests The original yara.rs was 536 lines, now split into: - yara/mod.rs (357 lines) - main formatting logic - yara/escaping.rs (204 lines) - escaping utilities Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore(agents): fix formatting in critical rules section Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * docs(AGENTS): improve AI agent guidelines with fixes and additions Fix formatting bug where rules 3 and 4 were merged on one line, correct typos, add Rust version requirements, expand development commands, clarify module structure, and document key dependencies. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: address PR review comments for output formatters - Fix test metadata count mismatches in output_table_integration.rs - Remove underscore prefix from used parameters in json.rs - Use crate re-export for RankingConfig import - Fix truncate_string edge case for wide multibyte characters - Update table docs to reflect actual tag selection behavior - Add control character sanitization for TTY output - Use UTF-8 encoding for Unicode test content - Add Claude Code AI assistance section to CONTRIBUTING.md - Update snapshots for encoding and sanitization changes Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: cleanup devcontainer, justfile, and AGENTS.md - Remove duplicate rust-bundle extension, keep official rust-analyzer - Refactor setup recipes to reuse mise-install recipe - Fix markdown lint: convert emphasis-as-heading to list items Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(justfile): use mise exec for all tool commands Add mise_exec variable and prefix all tool commands (cargo, prettier, pre-commit, actionlint, cspell, markdownlint, lychee, dist, mdbook, goreleaser) with mise exec to ensure correct tool versions are used regardless of shell activation state. Also adds dotenv-load and ignore-comments settings for consistency with other projects. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 8fcbb47 commit 91f4263

91 files changed

Lines changed: 4900 additions & 253 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/settings.json‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
11
{
2-
"enabledPlugins": {
3-
"commit@cc-marketplace": true
4-
}
2+
"enabledPlugins": {}
53
}

‎.coderabbit.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -351,7 +351,7 @@ reviews:
351351
- mode: "warning"
352352
name: "ASCII Only"
353353
instructions: |
354-
Verify that no Unicode punctuation is introduced:
354+
Verify that no Unicode punctuation is introduced unless explicitly required:
355355
1. No emojis in code or documentation
356356
2. No em-dashes - use regular hyphens
357357
3. No smart quotes - use straight quotes

‎.devcontainer/devcontainer.json‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
{
2+
"name": "Rust",
3+
"image": "mcr.microsoft.com/devcontainers/rust:2-1-trixie",
4+
"features": {
5+
"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {
6+
"installDockerBuildx": true,
7+
"version": "latest",
8+
"dockerDashComposeVersion": "v2",
9+
"moby": false
10+
},
11+
"ghcr.io/devcontainers/features/github-cli:1": {
12+
"installDirectlyFromGitHubRelease": true,
13+
"version": "latest"
14+
},
15+
"ghcr.io/eitsupi/devcontainer-features/mdbook:1": {
16+
"version": "latest"
17+
},
18+
"ghcr.io/devcontainers-extra/features/claude-code:1": {
19+
"version": "latest"
20+
},
21+
"ghcr.io/devcontainers-extra/features/mise:1": {
22+
"version": "latest"
23+
}
24+
},
25+
"customizations": {
26+
"vscode": {
27+
"extensions": [
28+
"mikestead.dotenv",
29+
"EditorConfig.EditorConfig",
30+
"tamasfe.even-better-toml",
31+
"github.vscode-github-actions",
32+
"GitHub.vscode-pull-request-github",
33+
"skellock.just",
34+
"yzhang.markdown-all-in-one",
35+
"bierner.markdown-checkbox",
36+
"bierner.markdown-footnotes",
37+
"bierner.markdown-mermaid",
38+
"bierner.markdown-yaml-preamble",
39+
"DavidAnson.vscode-markdownlint",
40+
"rust-lang.rust-analyzer",
41+
"foxundermoon.shell-format",
42+
"redhat.vscode-yaml",
43+
"ms-vscode-remote.remote-containers"
44+
]
45+
}
46+
}
47+
}

‎.github/dependabot.yml‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,12 @@ updates:
1313
- package-ecosystem: "github-actions"
1414
directory: "/"
1515
schedule:
16-
interval: "daily"
16+
interval: "weekly"
1717
- package-ecosystem: "rust-toolchain"
1818
directory: "/"
1919
schedule:
20-
interval: "daily"
20+
interval: "weekly"
21+
- package-ecosystem: "devcontainers"
22+
directory: "/"
23+
schedule:
24+
interval: "weekly"

‎.gitignore‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -121,9 +121,13 @@ docs/book/
121121
.envrc
122122
.direnv/
123123

124-
megalinter-reports/
125-
126124
# Override global gitignore
127125
!bin/
128126
# Added by goreleaser init:
129127
.intentionally-empty-file.o
128+
129+
130+
megalinter-reports/*
131+
target/*
132+
stringy-output/*
133+
tests/fixtures/*

‎.mdformat.toml‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@ exclude = [
77
"**/*.tpl.md",
88
"**/CHANGELOG.md",
99
"target/**",
10-
"megalinter-reports/**",
1110
]
1211
validate = true
1312
number = true
@@ -26,5 +25,4 @@ extensions = [
2625

2726
[plugin.mkdocs]
2827
align_semantic_breaks_in_lists = true
29-
ignore_missing_references = true
30-
28+
ignore_missing_references = true

‎.mega-linter.yml‎

Lines changed: 0 additions & 48 deletions
This file was deleted.

‎.repomixignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
megalinter-reports/*
2+
target/*
3+
stringy-output/*
4+
tests/fixtures/*

‎.vscode/settings.json‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"ruff.path": [
3+
"${workspaceFolder}/.vscode/mise-tools/ruff"
4+
],
5+
"ruff.interpreter": [
6+
"${workspaceFolder}/.vscode/mise-tools/python"
7+
],
8+
"python.defaultInterpreterPath": "${workspaceFolder}/.vscode/mise-tools/python",
9+
"debug.javascript.defaultRuntimeExecutable": {
10+
"pwa-node": "${workspaceFolder}/.vscode/mise-tools/node"
11+
}
12+
}

‎AGENTS.md‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,24 +6,25 @@
66

77
1. **No `unsafe` code** - `#![forbid(unsafe_code)]` enforced
88
2. **Zero warnings** - `cargo clippy -- -D warnings` must pass
9-
3. **ASCII only** - No emojis, em-dashes, smart quotes, or Unicode punctuation
9+
3. **ASCII only** - No emojis, em-dashes, smart quotes, or Unicode punctuation (except when explicitly testing or working with Unicode strings or emojis)
1010
4. **File size limit** - Keep files under 500 lines; split larger files
1111
5. **No blanket `#[allow]`** - Any `allow` requires inline justification
1212

1313
## Project Summary
1414

1515
Stringy extracts meaningful strings from ELF, PE, and Mach-O binaries using format-specific knowledge and semantic classification. Unlike standard `strings`, it is section-aware and semantically intelligent.
1616

17-
**Data flow**: Binary -> Format Detection -> Container Parsing -> String Extraction -> Deduplication -> Classification -> Ranking -> Output
17+
- **Rust**: Edition 2024, MSRV 1.91
18+
- **Data flow**: Binary -> Format Detection -> Container Parsing -> String Extraction -> Deduplication -> Classification -> Ranking -> Output
1819

1920
## Module Structure
2021

2122
| Module | Purpose |
2223
| ----------------- | ---------------------------------------------------------------- |
2324
| `container/` | Format detection, section analysis, imports/exports via `goblin` |
2425
| `extraction/` | ASCII/UTF-8/UTF-16 extraction, deduplication, PE resources |
25-
| `classification/` | Semantic tagging (URLs, IPs, domains, paths, GUIDs) |
26-
| `output/` | Formatters (JSON, human-readable, YARA-friendly) |
26+
| `classification/` | Semantic tagging (URLs, IPs, domains, paths, GUIDs), ranking |
27+
| `output/` | Formatters: `json/`, `table/` (tty/plain), `yara/` |
2728
| `types/` | Core data structures, error handling with `thiserror` |
2829

2930
## Key Patterns
@@ -48,6 +49,10 @@ just test # Run tests with nextest
4849
just lint # Full lint suite
4950
just fix # Auto-fix clippy warnings
5051
just ci-check # Full CI suite locally
52+
just build # Debug build
53+
just run <args> # Run stringy with arguments
54+
just bench # Run benchmarks
55+
just format # Format all (Rust, JSON, YAML, Markdown, Justfile)
5156
```
5257

5358
## Testing
@@ -60,6 +65,14 @@ just ci-check # Full CI suite locally
6065

6166
Import from `stringy::extraction` or `stringy::types`, not deeply nested paths. Re-exports are in `lib.rs`.
6267

68+
## Key Dependencies
69+
70+
- `goblin` - Binary format parsing (ELF, PE, Mach-O)
71+
- `pelite` - PE resource extraction
72+
- `thiserror` - Error type definitions
73+
- `insta` - Snapshot testing (dev)
74+
- `criterion` - Benchmarking (dev)
75+
6376
## Adding Features
6477

6578
**New semantic tag**: Add variant to `Tag` enum in `types.rs`, implement pattern in `classification/semantic.rs`

0 commit comments

Comments
 (0)