Skip to content

Commit 1bca004

Browse files
unclesp1d3rCopilot
andauthored
feat(classification): implement file path classification for POSIX, Windows, and registry paths (#121)
* feat(docs): add AI agent guidelines and character usage policy Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore(docs): revise AI agent guidelines for clarity and rules Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore(docs): update module structure formatting in documentation Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * feat(classification): implement file path classification for POSIX and Windows - Added POSIX and Windows file path pattern matching - Included registry path detection - Comprehensive unit and integration tests added Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * feat(classification): enhance path and registry detection - Implement POSIX, Windows, and UNC path classification - Add support for Windows registry path detection - Update documentation and tests for new capabilities Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore: minor docs and test adjustments - Fix MSRV reference from 1.91+ to 1.85+ in copilot instructions - Reorganize classification docs for clarity - Increase classification test timeout for CI reliability * fix(classification): address code review feedback on path classification performance and consistency (#122) * Initial plan * fix: address code review feedback on path classification - Clarify Windows path separator validation in docs - Optimize registry path matching with zero-allocation case-insensitive search - Prevent duplicate Tag::FilePath entries - Add comment explaining 500ms performance test timeout Co-authored-by: unclesp1d3r <251112+unclesp1d3r@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: unclesp1d3r <251112+unclesp1d3r@users.noreply.github.com> * chore: add comprehensive codebase analysis documentation Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore: add CodeRabbit configuration file for project setup This configuration file includes general settings, review guidelines, path filters, and instructions for various modules to streamline code reviews and maintain project standards. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore: improve formatting and readability in codebase analysis Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore: update formatting in copilot instructions Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore: update Cargo.toml and codebase_analysis.md formatting Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore: refresh task list to reflect current implementation state * chore: add documentation for core flows and technical plan - Introduced detailed documentation for core user interactions in Stringy v1.0. - Added technical plan outlining architecture, design decisions, and integration strategies. - Included performance optimization ticket and end-to-end testing strategy. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore: add MSRV check to CI workflow Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore: update character restrictions in copilot instructions Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore: update documentation and improve formatting - Adjusted numbering format in user interaction flows for consistency. - Enhanced clarity in the Epic Brief by refining problem statements. - Improved formatting in the Technical Plan for better readability. - Updated performance optimization ticket to reflect new dependencies. - Enhanced clarity in the pipeline orchestration ticket scope. - Refined semantic classification ticket to include additional patterns. - Improved integration testing ticket to cover more binary types. - Updated FoundString data model ticket for clarity on new fields. - Enhanced CLI argument parsing ticket to include all flags. - Improved output formatters ticket to clarify scope and dependencies. - Refined ranking system ticket to ensure clarity on scoring logic. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore: update directory structure path in analysis Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore: update Cargo.toml and rust-toolchain for Rust 1.91 - Bump rust-version in Cargo.toml to 1.91 - Update rust-toolchain.toml to use channel 1.91.0 - Enhance semantic classification for Windows paths with case-insensitivity and additional validation checks - Add new test file for let chains example Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> --------- Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
1 parent 8039ec8 commit 1bca004

27 files changed

Lines changed: 4052 additions & 266 deletions

‎.claude/settings.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"enabledPlugins": {
3+
"commit@cc-marketplace": true
4+
}
5+
}

‎.coderabbit.yml‎

Lines changed: 583 additions & 0 deletions
Large diffs are not rendered by default.

‎.github/copilot-instructions.md‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
# Copilot Instructions for Stringy
2+
3+
## Project Overview
4+
5+
Stringy is a **smarter strings tool** for extracting meaningful strings from ELF, PE, and Mach-O binaries using format-specific knowledge and semantic classification. Unlike the standard `strings` command, Stringy is data-structure aware, section-aware, and semantically intelligent.
6+
7+
## Architecture & Data Flow
8+
9+
```text
10+
Binary -> Format Detection (goblin) -> Container Parsing -> String Extraction -> Deduplication -> Classification -> Ranking -> Output
11+
```
12+
13+
### Module Organization
14+
15+
- **`container/`** \[COMPLETE\]: Format detection (ELF/PE/Mach-O), section analysis, imports/exports via `goblin`
16+
- **`extraction/`** \[COMPLETE\]: ASCII/UTF-8/UTF-16 string extraction, deduplication, PE resources
17+
- **`classification/`** \[PARTIAL\]: Semantic tagging (URLs, IPs, domains, paths, GUIDs, etc.)
18+
- **`output/`** \[PLANNED\]: JSON/human-readable/YARA-friendly formatting
19+
- **`types/`** \[COMPLETE\]: Core data structures (`FoundString`, `ContainerInfo`, etc.), error handling
20+
21+
## Critical Coding Standards
22+
23+
### Zero Tolerance Policies
24+
25+
- **No `unsafe` code**: `#![forbid(unsafe_code)]` enforced at package level
26+
- **Zero warnings**: `cargo clippy -- -D warnings` must pass (`#![deny(warnings)]` enforced)
27+
- **Rust 2024 Edition**: MSRV 1.85+, always use latest edition features
28+
- **File size limit**: Keep files \<=500-600 lines; split larger files into focused modules
29+
- **No blanket `#[allow]`**: Any `allow` attribute requires inline justification and cannot apply to entire files/modules
30+
- **Character restrictions**: Never use emojis, em-dashes, or other non-Latin characters in code or documentation. Use standard ASCII punctuation (hyphens, quotes, etc.)
31+
32+
### Error Handling with `thiserror`
33+
34+
Use structured errors with detailed context (see `src/types.rs`):
35+
36+
```rust
37+
#[derive(Debug, Error)]
38+
pub enum StringyError {
39+
#[error("Binary parsing error: {0}")]
40+
ParseError(String),
41+
42+
#[error("Invalid encoding at offset {offset}")]
43+
EncodingError { offset: u64 },
44+
}
45+
```
46+
47+
Convert external errors with `From` implementations. Provide offsets, section names, and file paths in error messages.
48+
49+
## Key Implementation Patterns
50+
51+
### Section Weight System
52+
53+
Container parsers assign weights (1.0-10.0) to sections based on string likelihood:
54+
55+
```rust
56+
// ELF example from container/elf.rs
57+
".rodata" | ".rodata.str1.*" => 10.0 // Highest priority
58+
".comment" | ".note.*" => 9.0 // Build info
59+
".data.rel.ro" => 7.0 // Read-only data
60+
".data" => 5.0 // Writable data (lower priority)
61+
```
62+
63+
**Pattern**: Use match expressions with fallthrough to assign weights; higher = more likely to contain meaningful strings.
64+
65+
### String Deduplication (`extraction/dedup.rs`)
66+
67+
Strings are grouped by `(text, encoding)` tuple in a `HashMap<(String, Encoding), Vec<StringOccurrence>>`:
68+
69+
- **Preserve all occurrences**: Each occurrence captures offset, RVA, section, source, tags, score, confidence
70+
- **Tag merging**: Union all tags via `HashSet`, then sort
71+
- **Combined scoring formula**:
72+
```text
73+
base_score = max(occurrence.original_score)
74+
occurrence_bonus = 5 * (count - 1)
75+
cross_section_bonus = 10 (if >1 unique section)
76+
multi_source_bonus = 15 (if >1 unique StringSource)
77+
confidence_boost = (max_confidence * 10.0) as i32
78+
```
79+
80+
### Non-Exhaustive Structs
81+
82+
Use `#[non_exhaustive]` for public API structs like `ContainerInfo` and provide explicit constructors (see `types.rs`):
83+
84+
```rust
85+
#[non_exhaustive]
86+
pub struct ContainerInfo { /* fields */ }
87+
88+
impl ContainerInfo {
89+
pub fn new(format: BinaryFormat, sections: Vec<SectionInfo>, ...) -> Self { ... }
90+
}
91+
```
92+
93+
## Testing Standards
94+
95+
- **Snapshot testing**: Use `insta` for output verification (`tests/integration_*.rs`)
96+
- **Fixtures**: Binary test fixtures in `tests/fixtures/` (see `fixtures/README.md`)
97+
- **Integration tests**: Named `test_*.rs` or `integration_*.rs` in `tests/`
98+
- **Run tests**: `just test` (uses `cargo nextest`)
99+
100+
Example pattern from `tests/integration_elf.rs`:
101+
102+
```rust
103+
fn get_fixture_path(name: &str) -> PathBuf {
104+
Path::new(env!("CARGO_MANIFEST_DIR"))
105+
.join("tests/fixtures")
106+
.join(name)
107+
}
108+
109+
#[test]
110+
fn test_elf_import_export_extraction() {
111+
let data = fs::read(&get_fixture_path("test_binary_elf")).expect("...");
112+
let parser = ElfParser::new();
113+
let info = parser.parse(&data).expect("...");
114+
// Verify imports/exports with specific assertions
115+
}
116+
```
117+
118+
## Development Workflow
119+
120+
### Common Commands (`justfile`)
121+
122+
**Setup**: `just setup` (installs rustfmt, clippy, llvm-tools-preview, mdformat)
123+
124+
**Development**:
125+
126+
- `just build` - Debug build
127+
- `just test` - Run tests with nextest
128+
- `just lint` - Full lint suite (rustfmt, clippy, actionlint, cspell, markdown)
129+
- `just check` - Pre-commit checks + lint
130+
- `just run <file>` - Run binary against test file
131+
132+
**Code Quality**:
133+
134+
- `just fmt` - Format Rust/markdown/YAML/JSON
135+
- `just fix` - Auto-fix clippy warnings with `--fix`
136+
- `just coverage` - Generate LCOV coverage report
137+
138+
**CI Parity**: `just ci-check` (runs full CI suite locally)
139+
140+
### Windows vs Unix
141+
142+
The `justfile` uses OS annotations (`[windows]`/`[unix]`) for cross-platform compatibility. PowerShell on Windows, bash on Unix.
143+
144+
## Dependencies & Crates
145+
146+
**Core parsing**: `goblin` (ELF/PE/Mach-O), `pelite` (PE resources)\
147+
**CLI**: `clap` with derive macros\
148+
**Error handling**: `thiserror`\
149+
**Serialization**: `serde`, `serde_json`\
150+
**Regex**: `regex` for classification\
151+
**Testing**: `insta` (snapshots), `criterion` (benchmarks), `tempfile`
152+
153+
## Import Conventions
154+
155+
- Re-export commonly used types in `lib.rs` for ergonomic imports
156+
- Import from `stringy::extraction` or `stringy::types`, not deeply nested paths
157+
- Within `extraction/mod.rs`, do NOT import locally-defined types; downstream code imports from `stringy::extraction`
158+
159+
## What NOT to Do
160+
161+
- Don't use `async` (this is a synchronous CLI tool)
162+
- Don't add `unsafe` blocks (forbidden)
163+
- Don't ignore clippy warnings (they're errors)
164+
- Don't create files >600 lines without splitting
165+
- Don't use blanket `#[allow]` on modules/files
166+
- Don't guess at section weights (refer to existing parsers in `container/`)
167+
168+
## Current Implementation Status
169+
170+
**Complete**:
171+
172+
- ELF/PE/Mach-O format detection and parsing
173+
- ASCII, UTF-8, UTF-16LE/BE string extraction
174+
- PE resource string extraction (VERSIONINFO, STRINGTABLE, MANIFEST)
175+
- String deduplication with occurrence tracking
176+
- IPv4/IPv6, URL, domain classification
177+
178+
**In Progress**:
179+
180+
- Full semantic classification suite (GUIDs, paths, format strings, Base64)
181+
- Ranking/scoring algorithm implementation
182+
- CLI (`main.rs` is placeholder)
183+
- Output formatters (JSON, YARA-friendly, human-readable)
184+
185+
## Quick Reference Examples
186+
187+
**Adding a new section weight** (in `container/elf.rs`, `pe.rs`, or `macho.rs`):
188+
189+
```rust
190+
let weight = match section_name {
191+
".mydata" => 8.0, // New section type
192+
_ => existing_match_arms
193+
};
194+
```
195+
196+
**Extracting strings from a section**:
197+
198+
```rust
199+
use stringy::extraction::{extract_ascii_strings, AsciiExtractionConfig};
200+
let config = AsciiExtractionConfig { min_length: 4, max_length: 1024 };
201+
let strings = extract_ascii_strings(&section_data, &config);
202+
```
203+
204+
**Adding a semantic tag**:
205+
206+
1. Add variant to `Tag` enum in `types.rs`
207+
2. Implement pattern matching in `classification/semantic.rs`
208+
3. Update deduplication tag merging if needed

‎.github/workflows/ci.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,21 @@ jobs:
6868
- name: Run clippy (all features)
6969
run: cargo clippy --all-targets --all-features -- -D warnings
7070

71+
# MSRV (Minimum Supported Rust Version) check
72+
msrv:
73+
runs-on: ubuntu-latest
74+
needs: changes
75+
if: needs.changes.outputs.rust == 'true'
76+
steps:
77+
- uses: actions/checkout@v6
78+
- uses: dtolnay/rust-toolchain@1.91.0
79+
80+
- name: Cache Rust dependencies
81+
uses: Swatinem/rust-cache@v2
82+
83+
- name: Check MSRV compliance
84+
run: cargo check --all-features
85+
7186
test:
7287
runs-on: ubuntu-latest
7388
needs: changes

0 commit comments

Comments
 (0)