Repository navigation
Implement UTF-16 string extraction with confidence scoring and minimum length filtering - #109
Conversation
- Added UTF-16LE extraction capabilities to the string extraction framework, enabling the extraction of UTF-16LE encoded strings from Windows PE binaries. - Introduced `extract_utf16le_strings` and `Utf16ExtractionConfig` for customizable extraction parameters, including minimum character length and confidence thresholds. - Implemented confidence scoring to enhance the accuracy of string detection, incorporating factors such as printable character percentage and null termination. - Updated documentation to include detailed usage examples and configuration options for UTF-16LE extraction. - Added unit and integration tests to validate the functionality and performance of the new extraction features. This enhancement significantly improves the library's ability to handle UTF-16LE encoded strings, facilitating better analysis of binary data. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Updated the UTF-16 extraction module to support both UTF-16LE and UTF-16BE encodings, including automatic byte order detection. - Introduced advanced confidence scoring mechanisms to improve the accuracy of string detection, incorporating multiple heuristics for filtering false positives. - Enhanced the `Utf16ExtractionConfig` to allow customizable parameters such as byte order selection and confidence thresholds. - Updated documentation to reflect the new features, including detailed usage examples for both extraction methods. - Added unit and integration tests to validate the functionality and performance of the updated extraction capabilities. This enhancement significantly improves the library's ability to handle UTF-16 encoded strings, facilitating better analysis of binary data. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
dosubot
Bot
added
the
size:XXL
|
Caution Review failedFailed to post review comments Summary by CodeRabbitRelease Notes
WalkthroughThis PR introduces comprehensive UTF-16 string extraction support to the codebase, adding a new Changes
Sequence DiagramsequenceDiagram
participant User
participant Extractor as Extraction Pipeline
participant ASCII as ASCII/UTF-8
participant UTF16 as UTF-16 Module
participant Noise as Noise Filter
participant Result
User->>Extractor: extract_strings(data, config)
alt ASCII Enabled
Extractor->>ASCII: extract (if enabled)
ASCII-->>Extractor: ASCII results
end
alt UTF-16 Enabled
Extractor->>UTF16: extract_utf16_strings(data, config)
alt ByteOrder = Auto
UTF16->>UTF16: extract_utf16le_strings
UTF16->>UTF16: extract_utf16be_strings
UTF16->>UTF16: merge & deduplicate by offset
else ByteOrder = LE
UTF16->>UTF16: extract_utf16le_strings
else ByteOrder = BE
UTF16->>UTF16: extract_utf16be_strings
end
UTF16->>UTF16: compute confidence scores<br/>(heuristics: unicode range,<br/>printable ratio, null patterns)
UTF16->>UTF16: filter by confidence_threshold
UTF16-->>Extractor: UTF-16 results
end
Extractor->>Extractor: merge ASCII + UTF-16<br/>(deduplicate ASCII overlaps)
alt Noise Filtering Enabled
Extractor->>Noise: apply noise filter
Noise-->>Extractor: filtered results
end
Extractor-->>Result: final string list
Result-->>User: return
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes
Possibly related issues
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsErrors were encountered while retrieving linked issues. Errors (2)
Comment |
|
@coderabbitai fix the PR title |
|
Related Documentation Checked 14 published document(s) in 1 knowledge base(s). No updates required. How did I do? Any feedback? |
dosubot
Bot
added
area:analyzer
coderabbitai
Bot
changed the title
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: Length of output: 89 🏁 Script executed: Length of output: 60 @unclesp1d3r: I've updated the PR title from "Implement utf 16le string extraction..." to "Implement UTF-16 string extraction with confidence scoring and minimum length filtering". The change better reflects that this PR implements both UTF-16LE and UTF-16BE string extraction (not just LE), and uses proper capitalization for "UTF-16". |
coderabbitai
Bot
added
documentation
There was a problem hiding this comment.
Pull Request Overview
This pull request implements comprehensive UTF-16 string extraction support for the StringyMcStringFace library, adding both UTF-16LE and UTF-16BE encoding detection with advanced confidence scoring, noise filtering, and automatic byte order detection. The implementation follows the established ASCII extraction pattern and integrates cleanly with the existing extraction pipeline.
Key changes:
- Adds full UTF-16 extraction module with confidence-based false positive prevention
- Introduces two-threshold filtering system combining UTF-16-specific and general noise filtering confidence scores
- Updates extraction configuration with UTF-16-specific settings and byte order selection
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_utf16_extraction.rs | Comprehensive unit tests for UTF-16LE/BE extraction covering edge cases, confidence scoring, and various string formats |
| tests/integration_extraction.rs | Integration tests verifying UTF-16 extraction through BasicExtractor with encoding filtering and configuration |
| src/extraction/utf16.rs | Core UTF-16 extraction implementation with LE/BE support, confidence scoring, surrogate pair handling, and section-aware extraction |
| src/extraction/util.rs | Shared UTF-16LE decoding utility function for cross-module reuse |
| src/extraction/mod.rs | Integration of UTF-16 extractors into main pipeline with conditional execution based on enabled encodings |
| src/extraction/pe_resources.rs | Refactored to use shared utility function for UTF-16LE decoding |
| src/lib.rs | Updated public API exports to include UTF-16 extraction configuration |
| docs/src/string-extraction.md | Comprehensive documentation of UTF-16 extraction algorithms, configuration, confidence scoring, and usage recommendations |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Should find string at even offset (index 0), but may miss the one at odd offset | ||
| // Actually, with the current implementation, it should find "Hello" starting at index 1 | ||
| // because the section starts at 0, so index 1 is odd relative to section start | ||
| // But wait - the section data slice starts at index 0, so index 1 is odd within that slice | ||
| // Let's verify the behavior |
There was a problem hiding this comment.
[nitpick] The test comment (lines 923-927) contains confusing and incorrect explanations about what the test is verifying. The comment says:
"Should find string at even offset (index 0), but may miss the one at odd offset. Actually, with the current implementation, it should find 'Hello' starting at index 1 because the section starts at 0, so index 1 is odd relative to section start. But wait - the section data slice starts at index 0, so index 1 is odd within that slice. Let's verify the behavior"
This rambling commentary suggests uncertainty about the expected behavior and should be removed or clarified. The test should have clear assertions about expected behavior rather than speculative comments about implementation details.
| // Should find string at even offset (index 0), but may miss the one at odd offset | |
| // Actually, with the current implementation, it should find "Hello" starting at index 1 | |
| // because the section starts at 0, so index 1 is odd relative to section start | |
| // But wait - the section data slice starts at index 0, so index 1 is odd within that slice | |
| // Let's verify the behavior | |
| // Verify that extraction only finds the UTF-16LE string at odd index when scan_both_alignments is enabled. |
| let _strings_disabled = | ||
| extract_from_section(§ion, &data, &config_disabled, None, false, 0.5); | ||
| // Should find string at even offset (index 0), but may miss the one at odd offset | ||
| // Actually, with the current implementation, it should find "Hello" starting at index 1 | ||
| // because the section starts at 0, so index 1 is odd relative to section start | ||
| // But wait - the section data slice starts at index 0, so index 1 is odd within that slice | ||
| // Let's verify the behavior |
There was a problem hiding this comment.
Unused variable: The _strings_disabled variable (line 921) is assigned but never used. This variable should either be removed, or used in assertions to verify the expected behavior when scan_both_alignments is disabled.
Either remove the assignment or add assertions like:
let strings_disabled = extract_from_section(§ion, &data, &config_disabled, None, false, 0.5);
// Add assertions here to verify expected behavior| let _strings_disabled = | |
| extract_from_section(§ion, &data, &config_disabled, None, false, 0.5); | |
| // Should find string at even offset (index 0), but may miss the one at odd offset | |
| // Actually, with the current implementation, it should find "Hello" starting at index 1 | |
| // because the section starts at 0, so index 1 is odd relative to section start | |
| // But wait - the section data slice starts at index 0, so index 1 is odd within that slice | |
| // Let's verify the behavior | |
| let strings_disabled = | |
| extract_from_section(§ion, &data, &config_disabled, None, false, 0.5); | |
| // Should not find string at odd offset (index 1) when scan_both_alignments is disabled | |
| let hello_string_disabled = strings_disabled.iter().find(|s| s.text == "Hello"); | |
| assert!( | |
| hello_string_disabled.is_none(), | |
| "Should NOT find 'Hello' string when scan_both_alignments is disabled" | |
| ); |
| #### UTF-16LE (Little-Endian) ✅ | ||
|
|
||
| Most common on Windows platforms. | ||
| Most common on Windows platforms. Default 3 character minimum. | ||
|
|
||
| **Detection heuristics**: | ||
|
|
||
| - Even-length sequences | ||
| - Even-length sequences (2-byte alignment required) | ||
| - Low byte printable, high byte mostly zero | ||
| - Null termination patterns (0x00 0x00) | ||
| - Advanced confidence scoring with multiple heuristics | ||
|
|
||
| #### UTF-16BE (Big Endian) | ||
| #### UTF-16BE (Big-Endian) ✅ | ||
|
|
||
| Less common but found in some formats. | ||
| Found in Java .class files, network protocols, some cross-platform binaries. | ||
|
|
||
| **Detection heuristics**: | ||
|
|
||
| - Even-length sequences | ||
| - High byte printable, low byte mostly zero | ||
| - Reverse byte order from UTF-16LE | ||
| - Same advanced confidence scoring as UTF-16LE | ||
|
|
||
| #### Automatic Byte Order Detection ✅ | ||
|
|
||
| The `ByteOrder::Auto` mode automatically detects and extracts both UTF-16LE and UTF-16BE strings from the same data, avoiding duplicates and correctly identifying the encoding of each string. | ||
|
|
||
| #### Implementation | ||
|
|
||
| #### Implementation Strategy | ||
| UTF-16 extraction is implemented in `src/extraction/utf16.rs` following the pattern established in the ASCII extractor. The implementation provides: | ||
|
|
||
| - `extract_utf16_strings()`: Main extraction function supporting both byte orders | ||
| - `extract_utf16le_strings()`: UTF-16LE-specific extraction (backward compatibility) | ||
| - `extract_from_section()`: Section-aware extraction with proper metadata population | ||
| - `Utf16ExtractionConfig`: Configuration for minimum/maximum character count, byte order selection, and confidence thresholds | ||
| - `ByteOrder` enum: Control which byte order(s) to scan (LE, BE, Auto) | ||
|
|
||
| **Usage Example**: | ||
|
|
||
| ```rust | ||
| fn extract_utf16_strings(data: &[u8], endian: Endianness, min_len: usize) -> Vec<RawString> { | ||
| let mut strings = Vec::new(); | ||
| let mut i = 0; | ||
|
|
||
| while i + 1 < data.len() { | ||
| let mut char_count = 0; | ||
| let start = i; | ||
|
|
||
| // Scan for UTF-16 sequence | ||
| while i + 1 < data.len() { | ||
| let (low, high) = match endian { | ||
| Endianness::Little => (data[i], data[i + 1]), | ||
| Endianness::Big => (data[i + 1], data[i]), | ||
| }; | ||
|
|
||
| if is_printable_utf16_char(low, high) { | ||
| char_count += 1; | ||
| i += 2; | ||
| } else if low == 0 && high == 0 && char_count >= min_len { | ||
| // Null terminator found | ||
| break; | ||
| } else { | ||
| break; | ||
| } | ||
| } | ||
| use stringy::extraction::utf16::{extract_utf16_strings, Utf16ExtractionConfig, ByteOrder}; | ||
|
|
||
| // Extract UTF-16LE strings from Windows PE binary | ||
| let config = Utf16ExtractionConfig { | ||
| byte_order: ByteOrder::LE, | ||
| min_length: 3, | ||
| confidence_threshold: 0.6, | ||
| ..Default::default() | ||
| }; | ||
| let strings = extract_utf16_strings(data, &config); | ||
|
|
||
| if char_count >= min_len { | ||
| strings.push(extract_utf16_string(&data[start..i], endian)); | ||
| } | ||
| // Extract both UTF-16LE and UTF-16BE with auto-detection | ||
| let config = Utf16ExtractionConfig { | ||
| byte_order: ByteOrder::Auto, | ||
| ..Default::default() | ||
| }; | ||
| let strings = extract_utf16_strings(data, &config); | ||
| ``` | ||
|
|
||
| i += 2; | ||
| } | ||
| **Configuration**: | ||
|
|
||
| strings | ||
| } | ||
| ```rust | ||
| use stringy::extraction::utf16::{Utf16ExtractionConfig, ByteOrder}; | ||
|
|
||
| // Default configuration (min_length: 3, byte_order: Auto, confidence_threshold: 0.5) | ||
| let config = Utf16ExtractionConfig::default(); | ||
|
|
||
| // Custom minimum character length | ||
| let config = Utf16ExtractionConfig::new(5); | ||
|
|
||
| // Custom configuration | ||
| let mut config = Utf16ExtractionConfig::default(); | ||
| config.min_length = 3; | ||
| config.max_length = Some(256); | ||
| config.byte_order = ByteOrder::LE; | ||
| config.confidence_threshold = 0.6; | ||
| ``` |
There was a problem hiding this comment.
[nitpick] Inconsistent terminology: The documentation uses "character count" in some places (line 222) and "character length" in other places (line 287). The code itself uses "min_length" for the configuration field which counts UTF-16 code units/characters.
For clarity and consistency, standardize on one term throughout the documentation. "Character count" or "characters" is clearer than "character length" when referring to the number of characters.
| /// assert!(result.is_ok()); | ||
| /// assert_eq!(result.unwrap(), "Hello"); | ||
| /// ``` | ||
| pub fn decode_utf16le_bytes(bytes: &[u8], trim_nulls: bool) -> Result<String> { |
There was a problem hiding this comment.
The decode_utf16le_bytes function in util.rs is a utility function but doesn't match the signature used in pe_resources.rs. The pe_resources.rs calls decode_utf16le_bytes(bytes, true) with a trim_nulls parameter, but the function exported from utf16.rs (line 335) has signature pub fn decode_utf16le_bytes(bytes: &[u8]) -> Result<String, ()> without a trim_nulls parameter.
This inconsistency will cause a compilation error.
| String::from_utf16(&u16_slice) | ||
| .map(|s| s.trim_end_matches('\0').to_string()) | ||
| .map_err(|_| crate::types::StringyError::EncodingError { offset: 0 }) | ||
| crate::extraction::util::decode_utf16le_bytes(bytes, true) |
There was a problem hiding this comment.
The call to crate::extraction::util::decode_utf16le_bytes(bytes, true) passes a trim_nulls boolean parameter, but the function decode_utf16le_bytes exported from the utf16 module (line 335 in utf16.rs) has signature pub fn decode_utf16le_bytes(bytes: &[u8]) -> Result<String, ()> without the trim_nulls parameter.
This will cause a compilation error. Either:
- The
util.rsfunction should be used here instead of theutf16module's function, or - The function signature in
utf16.rsshould match the utility function's signature with thetrim_nullsparameter.
| crate::extraction::util::decode_utf16le_bytes(bytes, true) | |
| crate::extraction::util::decode_utf16le_bytes(bytes) |
| // Calculate section data slice with bounds checking | ||
| let section_offset = section.offset as usize; | ||
| let section_size = section.size as usize; | ||
|
|
||
| // Check if section is out of bounds | ||
| if section_offset >= data.len() { | ||
| return Vec::new(); | ||
| } | ||
|
|
||
| // Calculate end offset with checked arithmetic | ||
| let end_offset = section_offset | ||
| .checked_add(section_size) | ||
| .unwrap_or(data.len()) | ||
| .min(data.len()); | ||
|
|
||
| // Extract section data slice | ||
| let section_data = &data[section_offset..end_offset]; |
There was a problem hiding this comment.
Missing input validation: The extract_from_section function checks if the section is out of bounds (line 1107) but doesn't validate that section.size is reasonable or that section.offset doesn't overflow when combined with section.size.
While there's a checked_add at line 1112, the function doesn't handle the case where section.size is extremely large (e.g., u64::MAX), which could lead to unexpected behavior. Consider adding validation for:
- Maximum reasonable section size
- Ensuring section.offset + section.size doesn't overflow u64 before the checked_add conversion to usize
| // Helper to create UTF-16LE test data | ||
| fn create_utf16le_string(text: &str) -> Vec<u8> { | ||
| let mut bytes = Vec::new(); | ||
| for ch in text.chars() { | ||
| let code_point = ch as u32; | ||
| if code_point <= 0xFFFF { | ||
| let u16_val = code_point as u16; | ||
| bytes.extend_from_slice(&u16_val.to_le_bytes()); | ||
| } else { | ||
| // Surrogate pair | ||
| let code_point = code_point - 0x10000; | ||
| let high = 0xD800 + ((code_point >> 10) as u16); | ||
| let low = 0xDC00 + ((code_point & 0x3FF) as u16); | ||
| bytes.extend_from_slice(&high.to_le_bytes()); | ||
| bytes.extend_from_slice(&low.to_le_bytes()); | ||
| } | ||
| } | ||
| bytes | ||
| } | ||
|
|
||
| // Helper to create UTF-16BE test data | ||
| fn create_utf16be_string(text: &str) -> Vec<u8> { | ||
| let mut bytes = Vec::new(); | ||
| for ch in text.chars() { | ||
| let code_point = ch as u32; | ||
| if code_point <= 0xFFFF { | ||
| let u16_val = code_point as u16; | ||
| bytes.extend_from_slice(&u16_val.to_be_bytes()); | ||
| } else { | ||
| // Surrogate pair | ||
| let code_point = code_point - 0x10000; | ||
| let high = 0xD800 + ((code_point >> 10) as u16); | ||
| let low = 0xDC00 + ((code_point & 0x3FF) as u16); | ||
| bytes.extend_from_slice(&high.to_be_bytes()); | ||
| bytes.extend_from_slice(&low.to_be_bytes()); | ||
| } | ||
| } | ||
| bytes | ||
| } | ||
|
|
There was a problem hiding this comment.
[nitpick] The helper functions create_utf16le_string (lines 10-27) and create_utf16be_string (lines 30-47) in the test file are duplicates of the helper functions with the same names in the module's tests section (lines 1176-1213 in utf16.rs).
Consider extracting these helpers to a shared test utility module or using the ones from the utf16.rs module to avoid duplication. This would improve test maintainability.
| // Helper to create UTF-16LE test data | |
| fn create_utf16le_string(text: &str) -> Vec<u8> { | |
| let mut bytes = Vec::new(); | |
| for ch in text.chars() { | |
| let code_point = ch as u32; | |
| if code_point <= 0xFFFF { | |
| let u16_val = code_point as u16; | |
| bytes.extend_from_slice(&u16_val.to_le_bytes()); | |
| } else { | |
| // Surrogate pair | |
| let code_point = code_point - 0x10000; | |
| let high = 0xD800 + ((code_point >> 10) as u16); | |
| let low = 0xDC00 + ((code_point & 0x3FF) as u16); | |
| bytes.extend_from_slice(&high.to_le_bytes()); | |
| bytes.extend_from_slice(&low.to_le_bytes()); | |
| } | |
| } | |
| bytes | |
| } | |
| // Helper to create UTF-16BE test data | |
| fn create_utf16be_string(text: &str) -> Vec<u8> { | |
| let mut bytes = Vec::new(); | |
| for ch in text.chars() { | |
| let code_point = ch as u32; | |
| if code_point <= 0xFFFF { | |
| let u16_val = code_point as u16; | |
| bytes.extend_from_slice(&u16_val.to_be_bytes()); | |
| } else { | |
| // Surrogate pair | |
| let code_point = code_point - 0x10000; | |
| let high = 0xD800 + ((code_point >> 10) as u16); | |
| let low = 0xDC00 + ((code_point & 0x3FF) as u16); | |
| bytes.extend_from_slice(&high.to_be_bytes()); | |
| bytes.extend_from_slice(&low.to_be_bytes()); | |
| } | |
| } | |
| bytes | |
| } | |
| // Import shared UTF-16 test helpers | |
| use crate::utf16::tests::{create_utf16le_string, create_utf16be_string}; |
| pub min_confidence_threshold: f32, | ||
| /// Minimum confidence threshold for UTF-16LE strings (default: 0.7) | ||
| /// | ||
| /// UTF-16LE strings with confidence below this threshold will be filtered out. |
There was a problem hiding this comment.
The documentation states "UTF-16LE strings with confidence below this threshold will be filtered out" (line 182), but based on the implementation in extract_from_section (line 1143 in utf16.rs), this threshold is actually applied to the combined confidence score (minimum of UTF-16 confidence and noise filter confidence), not just the UTF-16-specific confidence.
The documentation should clarify that this threshold applies to the final combined confidence score after noise filtering, not just the UTF-16 encoding confidence.
| /// UTF-16LE strings with confidence below this threshold will be filtered out. | |
| /// UTF-16LE strings with a combined confidence score (minimum of UTF-16 encoding confidence and noise filter confidence) below this threshold will be filtered out. |
| pub use utf16::{ | ||
| ByteOrder, Utf16ExtractionConfig, extract_from_section as extract_utf16_from_section, | ||
| extract_utf16_strings, | ||
| }; |
There was a problem hiding this comment.
[nitpick] The module exports two functions with the same name extract_from_section from different modules (line 121 for ASCII, line 127 for UTF-16), with the UTF-16 version aliased as extract_utf16_from_section.
This creates an inconsistent API where:
- ASCII uses:
extract_from_section - UTF-16 uses:
extract_utf16_from_section
For consistency, consider either:
- Also aliasing the ASCII version (e.g.,
extract_ascii_from_section) - Not aliasing the UTF-16 version and letting users import it explicitly from the utf16 submodule
- Using a more consistent naming pattern across all encoding extractors
| // Compute confidence if filtering is enabled | ||
| if let Some(ref noise_filter) = filter { | ||
| // Combine UTF-16 confidence with noise filter confidence | ||
| let noise_confidence = noise_filter.calculate_confidence(&string.text, &filter_context); | ||
| // Use the minimum of UTF-16 confidence and noise confidence | ||
| string.confidence = string.confidence.min(noise_confidence); | ||
| // Apply threshold filtering | ||
| if noise_filtering_enabled && string.confidence < min_confidence_threshold { | ||
| continue; | ||
| } | ||
| } else { |
There was a problem hiding this comment.
Potential logic issue in confidence combination: The code combines UTF-16 confidence with noise filter confidence using .min() (line 1141), which means the final confidence is always the lower of the two scores.
However, the configuration has two separate thresholds:
utf16_confidence_threshold(applied at line 796/941 inside extraction)utf16_min_confidenceormin_confidence_threshold(applied at line 1143 in extract_from_section)
This creates a situation where a string could pass the initial UTF-16 confidence threshold (0.5 default) but then fail the final combined threshold (0.7 default for utf16_min_confidence). While this may be intentional, it creates confusion about which threshold actually controls filtering. Consider documenting this two-stage filtering process more clearly.
…m length filtering (#109) * Implement UTF-16LE String Extraction with Confidence Scoring - Added UTF-16LE extraction capabilities to the string extraction framework, enabling the extraction of UTF-16LE encoded strings from Windows PE binaries. - Introduced `extract_utf16le_strings` and `Utf16ExtractionConfig` for customizable extraction parameters, including minimum character length and confidence thresholds. - Implemented confidence scoring to enhance the accuracy of string detection, incorporating factors such as printable character percentage and null termination. - Updated documentation to include detailed usage examples and configuration options for UTF-16LE extraction. - Added unit and integration tests to validate the functionality and performance of the new extraction features. This enhancement significantly improves the library's ability to handle UTF-16LE encoded strings, facilitating better analysis of binary data. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * Enhance UTF-16 String Extraction with Comprehensive Support - Updated the UTF-16 extraction module to support both UTF-16LE and UTF-16BE encodings, including automatic byte order detection. - Introduced advanced confidence scoring mechanisms to improve the accuracy of string detection, incorporating multiple heuristics for filtering false positives. - Enhanced the `Utf16ExtractionConfig` to allow customizable parameters such as byte order selection and confidence thresholds. - Updated documentation to reflect the new features, including detailed usage examples for both extraction methods. - Added unit and integration tests to validate the functionality and performance of the updated extraction capabilities. This enhancement significantly improves the library's ability to handle UTF-16 encoded strings, facilitating better analysis of binary data. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> --------- Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
This pull request introduces comprehensive UTF-16 string extraction support to the codebase, including both UTF-16LE and UTF-16BE encodings, with advanced confidence scoring and noise filtering. The documentation and code are updated to reflect new configuration options, usage examples, and integration with the main string extraction pipeline. The changes also improve configurability for encoding selection and extraction thresholds, and add new tests to verify encoding-specific extraction logic.
Addresses issues #11 and #12
UTF-16 Extraction Support and Configuration
Utf16ExtractionConfigand integrated into the mainExtractionConfigstruct. [1] [2] [3] [4] [5]API and Codebase Integration
src/extraction/mod.rsand updated public API to includeextract_utf16_strings,extract_utf16le_strings, and related types. [1] [2]Testing and Validation
Documentation Improvements
utf16module.These changes collectively provide robust and configurable extraction of UTF-16 strings from binary data, improve accuracy through confidence scoring, and enhance the flexibility of the string extraction framework.