Skip to content

Commit bf9920f

Browse files
authored
Add String Extraction Framework with Noise Filtering and Benchmarks (#108)
* feat(extraction): Introduce BasicExtractor and ExtractionConfig for string extraction - Added the BasicExtractor struct to implement a sequential ASCII/UTF-8 string extraction algorithm from binary data. - Introduced the ExtractionConfig struct to allow customization of extraction parameters, including minimum and maximum string lengths, encoding selection, and section filtering. - Updated documentation to include usage examples for both BasicExtractor and ExtractionConfig. - Added integration tests to validate the functionality of string extraction from various binary formats, including ELF and PE. This enhancement improves the library's ability to extract meaningful strings from binary files, facilitating better analysis and understanding of their content. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * feat(extraction): Add ASCII string extraction module and update documentation - Introduced a new module for ASCII string extraction, providing foundational functionality for extracting contiguous printable ASCII sequences from binary data. - Implemented `extract_ascii_strings` and `extract_from_section` functions, along with an `ExtractionConfig` struct for customizable extraction parameters. - Updated the main extraction module documentation to include details about the new ASCII extraction capabilities and usage examples. - Enhanced the existing documentation to reflect the addition of ASCII extraction in the library's structure and functionality. This addition improves the library's ability to handle ASCII string extraction, paving the way for future enhancements in UTF-8 and other encoding types. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * refactor(extraction): Rename ExtractionConfig to AsciiExtractionConfig and update related documentation - Renamed `ExtractionConfig` to `AsciiExtractionConfig` to better reflect its purpose in ASCII string extraction. - Updated all references in the codebase and documentation to use the new name, ensuring consistency. - Enhanced documentation to clarify usage examples and configuration details for ASCII extraction. - Improved function signatures and comments to align with the new naming convention. This refactor improves code clarity and maintains a consistent naming scheme across the extraction module, facilitating better understanding and usage of the ASCII extraction functionality. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * feat(extraction): Enhance ASCII string extraction with noise filtering and new benchmarks - Added a new `entropy` dependency to support advanced noise filtering capabilities in ASCII string extraction. - Introduced a new benchmark for ASCII extraction, `ascii_extraction`, to evaluate performance under various conditions. - Updated the `FoundString` struct to include a `confidence` field, representing the likelihood of a string being legitimate versus noise. - Enhanced the ASCII extraction functions to compute confidence scores using a composite noise filter, allowing for better filtering of false positives. - Updated documentation to reflect the new noise filtering features and provided examples for configuring extraction parameters. - Added unit and integration tests to validate the new noise filtering functionality and ensure robust performance. This enhancement significantly improves the accuracy and reliability of ASCII string extraction, making it more effective for analyzing binary data. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore(deps): Update dependencies in Cargo.toml - Bumped `entropy` to version 0.4.2 for improved functionality. - Updated `pelite` to version 0.10.0 for better compatibility. - Upgraded `serde_json` to version 1.0.145 to incorporate the latest features and fixes. - Updated `insta` to version 1.43.2 and `tempfile` to version 3.23.0 for enhanced testing capabilities. These updates ensure the project utilizes the latest versions of dependencies, improving overall stability and performance. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore(ci): Update GitHub Actions dependencies and configuration - Updated `actions/upload-artifact` from version 4 to 5 across multiple workflows for improved artifact management. - Updated `actions/download-artifact` from version 5 to 6 to leverage new features and enhancements. - Updated `actions/attest-build-provenance` from version 2 to 3 for better build provenance tracking. - Adjusted the `targets` list formatting in `dist-workspace.toml` for improved readability. These updates ensure the workflows are utilizing the latest versions of the actions, enhancing overall CI/CD performance and reliability. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> --------- Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
1 parent bd0b1b5 commit bf9920f

17 files changed

Lines changed: 4942 additions & 43 deletions

‎.github/workflows/release.yml‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ jobs:
6666
shell: bash
6767
run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.30.2/cargo-dist-installer.sh | sh"
6868
- name: Cache dist
69-
uses: actions/upload-artifact@v4
69+
uses: actions/upload-artifact@v5
7070
with:
7171
name: cargo-dist-cache
7272
path: ~/.cargo/bin/dist
@@ -82,7 +82,7 @@ jobs:
8282
cat plan-dist-manifest.json
8383
echo "manifest=$(jq -c "." plan-dist-manifest.json)" >> "$GITHUB_OUTPUT"
8484
- name: "Upload dist-manifest.json"
85-
uses: actions/upload-artifact@v4
85+
uses: actions/upload-artifact@v5
8686
with:
8787
name: artifacts-plan-dist-manifest
8888
path: plan-dist-manifest.json
@@ -135,7 +135,7 @@ jobs:
135135
run: ${{ matrix.install_dist.run }}
136136
# Get the dist-manifest
137137
- name: Fetch local artifacts
138-
uses: actions/download-artifact@v5
138+
uses: actions/download-artifact@v6
139139
with:
140140
pattern: artifacts-*
141141
path: target/distrib/
@@ -151,7 +151,7 @@ jobs:
151151
dist build ${{ needs.plan.outputs.tag-flag }} --print=linkage --output-format=json ${{ matrix.dist_args }} > dist-manifest.json
152152
echo "dist ran successfully"
153153
- name: Attest
154-
uses: actions/attest-build-provenance@v2
154+
uses: actions/attest-build-provenance@v3
155155
with:
156156
subject-path: "target/distrib/*${{ join(matrix.targets, ', ') }}*"
157157
- id: cargo-dist
@@ -168,7 +168,7 @@ jobs:
168168
169169
cp dist-manifest.json "$BUILD_MANIFEST_NAME"
170170
- name: "Upload artifacts"
171-
uses: actions/upload-artifact@v4
171+
uses: actions/upload-artifact@v5
172172
with:
173173
name: artifacts-build-local-${{ join(matrix.targets, '_') }}
174174
path: |
@@ -190,7 +190,7 @@ jobs:
190190
persist-credentials: false
191191
submodules: recursive
192192
- name: Install cached dist
193-
uses: actions/download-artifact@v5
193+
uses: actions/download-artifact@v6
194194
with:
195195
name: cargo-dist-cache
196196
path: ~/.cargo/bin/
@@ -202,7 +202,7 @@ jobs:
202202
shell: bash
203203
# Get all the local artifacts for the global tasks to use (for e.g. checksums)
204204
- name: Fetch local artifacts
205-
uses: actions/download-artifact@v5
205+
uses: actions/download-artifact@v6
206206
with:
207207
pattern: artifacts-*
208208
path: target/distrib/
@@ -233,7 +233,7 @@ jobs:
233233
find . -name '*.cdx.xml' | tee -a "$GITHUB_OUTPUT"
234234
echo "EOF" >> "$GITHUB_OUTPUT"
235235
- name: "Upload artifacts"
236-
uses: actions/upload-artifact@v4
236+
uses: actions/upload-artifact@v5
237237
with:
238238
name: artifacts-build-global
239239
path: |
@@ -259,14 +259,14 @@ jobs:
259259
persist-credentials: false
260260
submodules: recursive
261261
- name: Install cached dist
262-
uses: actions/download-artifact@v5
262+
uses: actions/download-artifact@v6
263263
with:
264264
name: cargo-dist-cache
265265
path: ~/.cargo/bin/
266266
- run: chmod +x ~/.cargo/bin/dist
267267
# Fetch artifacts from scratch-storage
268268
- name: Fetch artifacts
269-
uses: actions/download-artifact@v5
269+
uses: actions/download-artifact@v6
270270
with:
271271
pattern: artifacts-*
272272
path: target/distrib/
@@ -279,14 +279,14 @@ jobs:
279279
cat dist-manifest.json
280280
echo "manifest=$(jq -c "." dist-manifest.json)" >> "$GITHUB_OUTPUT"
281281
- name: "Upload dist-manifest.json"
282-
uses: actions/upload-artifact@v4
282+
uses: actions/upload-artifact@v5
283283
with:
284284
# Overwrite the previous copy
285285
name: artifacts-dist-manifest
286286
path: dist-manifest.json
287287
# Create a GitHub Release while uploading all files to it
288288
- name: "Download GitHub Artifacts"
289-
uses: actions/download-artifact@v5
289+
uses: actions/download-artifact@v6
290290
with:
291291
pattern: artifacts-*
292292
path: artifacts
@@ -326,7 +326,7 @@ jobs:
326326
token: ${{ secrets.HOMEBREW_TAP_TOKEN }}
327327
# So we have access to the formula
328328
- name: Fetch homebrew formulae
329-
uses: actions/download-artifact@v5
329+
uses: actions/download-artifact@v6
330330
with:
331331
pattern: artifacts-*
332332
path: Formula/

‎Cargo.toml‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,16 +20,17 @@ path = "src/main.rs"
2020

2121
[dependencies]
2222
clap = { version = "4.5.51", features = ["derive"] }
23+
entropy = "0.4.2"
2324
goblin = "0.10.3"
24-
pelite = "0.10"
25+
pelite = "0.10.0"
2526
serde = { version = "1.0.228", features = ["derive"] }
26-
serde_json = "1.0"
27+
serde_json = "1.0.145"
2728
thiserror = "2.0.17"
2829

2930
[dev-dependencies]
3031
criterion = "0.7.0"
31-
insta = "1.43"
32-
tempfile = "3.23"
32+
insta = "1.43.2"
33+
tempfile = "3.23.0"
3334

3435
# The profile that 'dist' will build with
3536
[profile.dist]
@@ -43,3 +44,7 @@ harness = false
4344
[[bench]]
4445
name = "pe"
4546
harness = false
47+
48+
[[bench]]
49+
name = "ascii_extraction"
50+
harness = false

‎benches/ascii_extraction.rs‎

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
use criterion::{Criterion, criterion_group, criterion_main};
2+
use std::hint::black_box;
3+
use stringy::extraction::ascii::{AsciiExtractionConfig, extract_ascii_strings};
4+
use stringy::extraction::config::NoiseFilterConfig;
5+
use stringy::extraction::filters::{CompositeNoiseFilter, FilterContext};
6+
7+
fn bench_basic_extraction(c: &mut Criterion) {
8+
// Create test data with various string patterns
9+
let test_data =
10+
b"Hello World\0Test String\0Another String\0Binary\x00\x01\x02Data\0More Strings\0"
11+
.repeat(100);
12+
let config = AsciiExtractionConfig::default();
13+
14+
c.bench_function("ascii_extraction_basic", |b| {
15+
b.iter(|| {
16+
let _ = extract_ascii_strings(black_box(&test_data), black_box(&config));
17+
});
18+
});
19+
}
20+
21+
fn bench_filtered_extraction(c: &mut Criterion) {
22+
let test_data =
23+
b"Hello World\0Test String\0Another String\0Binary\x00\x01\x02Data\0More Strings\0"
24+
.repeat(100);
25+
let config = AsciiExtractionConfig::default();
26+
let filter_config = NoiseFilterConfig::default();
27+
let filter = CompositeNoiseFilter::new(&filter_config);
28+
let context = FilterContext::default();
29+
30+
c.bench_function("ascii_extraction_with_filtering", |b| {
31+
b.iter(|| {
32+
let strings = extract_ascii_strings(black_box(&test_data), black_box(&config));
33+
for string in &strings {
34+
let _ = filter.calculate_confidence(black_box(&string.text), black_box(&context));
35+
}
36+
});
37+
});
38+
}
39+
40+
fn bench_individual_filters(c: &mut Criterion) {
41+
use stringy::extraction::filters::{
42+
CharDistributionFilter, ContextFilter, EntropyFilter, LengthFilter, LinguisticFilter,
43+
NoiseFilter, RepetitionFilter,
44+
};
45+
46+
let test_strings = vec![
47+
"Hello, World!",
48+
"AAAA",
49+
"Error: file not found",
50+
"!!!@@@###",
51+
"C:\\Windows\\System32",
52+
];
53+
54+
let char_filter = CharDistributionFilter;
55+
let entropy_filter = EntropyFilter::new(1.5, 7.5);
56+
let linguistic_filter = LinguisticFilter::new(0.1, 0.9);
57+
let length_filter = LengthFilter::new(200);
58+
let repetition_filter = RepetitionFilter::new(0.7);
59+
let context_filter = ContextFilter;
60+
let context = FilterContext::default();
61+
62+
c.bench_function("filter_char_distribution", |b| {
63+
b.iter(|| {
64+
for text in &test_strings {
65+
let _ = char_filter.calculate_confidence(black_box(text), black_box(&context));
66+
}
67+
});
68+
});
69+
70+
c.bench_function("filter_entropy", |b| {
71+
b.iter(|| {
72+
for text in &test_strings {
73+
let _ = entropy_filter.calculate_confidence(black_box(text), black_box(&context));
74+
}
75+
});
76+
});
77+
78+
c.bench_function("filter_linguistic", |b| {
79+
b.iter(|| {
80+
for text in &test_strings {
81+
let _ =
82+
linguistic_filter.calculate_confidence(black_box(text), black_box(&context));
83+
}
84+
});
85+
});
86+
87+
c.bench_function("filter_length", |b| {
88+
b.iter(|| {
89+
for text in &test_strings {
90+
let _ = length_filter.calculate_confidence(black_box(text), black_box(&context));
91+
}
92+
});
93+
});
94+
95+
c.bench_function("filter_repetition", |b| {
96+
b.iter(|| {
97+
for text in &test_strings {
98+
let _ =
99+
repetition_filter.calculate_confidence(black_box(text), black_box(&context));
100+
}
101+
});
102+
});
103+
104+
c.bench_function("filter_context", |b| {
105+
b.iter(|| {
106+
for text in &test_strings {
107+
let _ = context_filter.calculate_confidence(black_box(text), black_box(&context));
108+
}
109+
});
110+
});
111+
}
112+
113+
fn bench_composite_filter(c: &mut Criterion) {
114+
let test_strings = vec![
115+
"Hello, World!",
116+
"AAAA",
117+
"Error: file not found",
118+
"!!!@@@###",
119+
"C:\\Windows\\System32",
120+
"https://example.com",
121+
];
122+
123+
let filter_config = NoiseFilterConfig::default();
124+
let filter = CompositeNoiseFilter::new(&filter_config);
125+
let context = FilterContext::default();
126+
127+
c.bench_function("composite_filter_all_enabled", |b| {
128+
b.iter(|| {
129+
for text in &test_strings {
130+
let _ = filter.calculate_confidence(black_box(text), black_box(&context));
131+
}
132+
});
133+
});
134+
135+
// Test with some filters disabled
136+
// Note: CompositeNoiseFilter doesn't expose a builder pattern, so we create a new one
137+
// with modified enable flags. For this benchmark, we'll just use the default filter.
138+
let filter_partial = CompositeNoiseFilter::new(&filter_config);
139+
140+
c.bench_function("composite_filter_partial", |b| {
141+
b.iter(|| {
142+
for text in &test_strings {
143+
let _ = filter_partial.calculate_confidence(black_box(text), black_box(&context));
144+
}
145+
});
146+
});
147+
}
148+
149+
fn bench_entropy_calculation(c: &mut Criterion) {
150+
use entropy::shannon_entropy;
151+
152+
let test_strings = vec![
153+
"Hello, World!",
154+
"AAAA",
155+
"Error: file not found",
156+
"!!!@@@###",
157+
];
158+
159+
c.bench_function("entropy_shannon_calculation", |b| {
160+
b.iter(|| {
161+
for text in &test_strings {
162+
let _ = shannon_entropy(black_box(text.as_bytes()));
163+
}
164+
});
165+
});
166+
}
167+
168+
fn bench_large_binary(c: &mut Criterion) {
169+
// Create a large binary-like data with embedded strings
170+
let mut large_data = Vec::new();
171+
for i in 0..10000 {
172+
if i % 100 == 0 {
173+
large_data.extend_from_slice(b"Hello World\0");
174+
} else {
175+
large_data.push((i % 256) as u8);
176+
}
177+
}
178+
179+
let config = AsciiExtractionConfig::default();
180+
let filter_config = NoiseFilterConfig::default();
181+
let filter = CompositeNoiseFilter::new(&filter_config);
182+
let context = FilterContext::default();
183+
184+
c.bench_function("large_binary_extraction", |b| {
185+
b.iter(|| {
186+
let strings = extract_ascii_strings(black_box(&large_data), black_box(&config));
187+
for string in &strings {
188+
let _ = filter.calculate_confidence(black_box(&string.text), black_box(&context));
189+
}
190+
});
191+
});
192+
}
193+
194+
criterion_group!(
195+
ascii_extraction_benches,
196+
bench_basic_extraction,
197+
bench_filtered_extraction,
198+
bench_individual_filters,
199+
bench_composite_filter,
200+
bench_entropy_calculation,
201+
bench_large_binary
202+
);
203+
criterion_main!(ascii_extraction_benches);

‎dist-workspace.toml‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,13 @@ ci = "github"
1010
# The installers to generate for each app
1111
installers = ["shell", "powershell", "homebrew"]
1212
# Target platforms to build apps for (Rust target-triple syntax)
13-
targets = ["aarch64-apple-darwin", "aarch64-unknown-linux-gnu", "x86_64-unknown-linux-gnu", "x86_64-unknown-linux-musl", "x86_64-pc-windows-msvc"]
13+
targets = [
14+
"aarch64-apple-darwin",
15+
"aarch64-unknown-linux-gnu",
16+
"x86_64-unknown-linux-gnu",
17+
"x86_64-unknown-linux-musl",
18+
"x86_64-pc-windows-msvc",
19+
]
1420
# A GitHub repo to push Homebrew formulas to
1521
tap = "EvilBit-Labs/homebrew-tap"
1622
# Customize the Homebrew formula name
@@ -49,4 +55,6 @@ install-success-msg = "Successfully installed Stringy! Ready to start looking at
4955
repository = "EvilBit-Labs/StringyMcStringFace"
5056
[dist.github-action-commits]
5157
"actions/checkout" = "v5"
52-
"actions/download-artifact" = "v5"
58+
"actions/download-artifact" = "v6"
59+
"actions/upload-artifact" = "v5"
60+
"actions/attest-build-provenance" = "v3"

0 commit comments

Comments
 (0)