diff --git a/README.md b/README.md index 2d1d1959d..e63b6e4ec 100644 --- a/README.md +++ b/README.md @@ -1,27 +1,32 @@ -# nom, eating data byte by byte +
+ nom logo in CC0 license, by Ange Albertini -[![LICENSE](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) -[![Join the chat at https://gitter.im/Geal/nom](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Geal/nom?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) -[![Build Status](https://github.com/rust-bakery/nom/actions/workflows/ci.yml/badge.svg)](https://github.com/rust-bakery/nom/actions/workflows/ci.yml) -[![Coverage Status](https://coveralls.io/repos/github/rust-bakery/nom/badge.svg?branch=main)](https://coveralls.io/github/rust-bakery/nom?branch=main) +# nom, eating data byte by byte [![crates.io Version](https://img.shields.io/crates/v/nom.svg)](https://crates.io/crates/nom) [![Minimum rustc version](https://img.shields.io/badge/rustc-1.65.0+-lightgray.svg)](#rust-version-requirements-msrv) +[![Build Status](https://github.com/rust-bakery/nom/actions/workflows/ci.yml/badge.svg)](https://github.com/rust-bakery/nom/actions/workflows/ci.yml) +[![Coverage Status](https://coveralls.io/repos/github/rust-bakery/nom/badge.svg?branch=main)](https://coveralls.io/github/rust-bakery/nom?branch=main) +[![LICENSE](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) +[![Join the chat at https://gitter.im/Geal/nom](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Geal/nom?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) +
-nom is a parser combinators library written in Rust. Its goal is to provide tools +`nom` is a parser combinators library written in Rust. Its goal is to provide tools to build safe parsers without compromising the speed or memory consumption. To that end, it uses extensively Rust's *strong typing* and *memory safety* to produce fast and correct parsers, and provides functions, macros and traits to abstract most of the error prone plumbing. -![nom logo in CC0 license, by Ange Albertini](https://raw.githubusercontent.com/Geal/nom/main/assets/nom.png) - -*nom will happily take a byte out of your files :)* - - +
+ Table of Contents +- [Installation](#installation) + - [Compilation features](#compilation-features) - [Example](#example) - [Documentation](#documentation) -- [Why use nom?](#why-use-nom) +- [Why use `nom`?](#why-use-nom) - [Binary format parsers](#binary-format-parsers) - [Text format parsers](#text-format-parsers) - [Programming language parsers](#programming-language-parsers) @@ -29,22 +34,31 @@ error prone plumbing. - [Parser combinators](#parser-combinators) - [Technical features](#technical-features) - [Rust version requirements](#rust-version-requirements-msrv) -- [Installation](#installation) - [Related projects](#related-projects) -- [Parsers written with nom](#parsers-written-with-nom) +- [Parsers written with `nom`](#parsers-written-with-nom) - [Contributors](#contributors) +
+ +## Installation +`nom` is available on [crates.io](https://crates.io/crates/nom) and can be included in your Cargo enabled project like this: + +```toml +[dependencies] +nom = "8" +``` - +### Compilation features +This crate supports standard `no_std` environments by configuring its default features: -## Example +- `alloc`: (Activated by default) Enables combinators requiring an allocator (like `many0`). Disable this for pure allocator-free environments. +- `std`: (Activated by default, implies `alloc`) Enables standard library support. Disable this for `no_std` builds. +## Example [Hexadecimal color](https://developer.mozilla.org/en-US/docs/Web/CSS/color) parser: ```rust use nom::{ - bytes::complete::{tag, take_while_m_n}, - combinator::map_res, - sequence::Tuple, + bytes::{tag, take_while_m_n}, IResult, Parser, }; @@ -56,64 +70,41 @@ pub struct Color { pub blue: u8, } -fn from_hex(input: &str) -> Result { - u8::from_str_radix(input, 16) -} - -fn is_hex_digit(c: char) -> bool { - c.is_digit(16) -} - -fn hex_primary(input: &str) -> IResult<&str, u8> { - map_res( - take_while_m_n(2, 2, is_hex_digit), - from_hex - ).parse(input) +fn hex_channel(input: &str) -> IResult<&str, u8> { + take_while_m_n(2, 2, |c: char| c.is_ascii_hexdigit()) + .map_res(|s| u8::from_str_radix(s, 16)) + .parse_complete(input) } fn hex_color(input: &str) -> IResult<&str, Color> { - let (input, _) = tag("#")(input)?; - let (input, (red, green, blue)) = (hex_primary, hex_primary, hex_primary).parse(input)?; + let (input, _) = tag("#").parse_complete(input)?; + let (input, (red, green, blue)) = (hex_channel, hex_channel, hex_channel).parse_complete(input)?; Ok((input, Color { red, green, blue })) } fn main() { - println!("{:?}", hex_color("#2F14DF")) -} - -#[test] -fn parse_color() { - assert_eq!( - hex_color("#2F14DF"), - Ok(( - "", - Color { - red: 47, - green: 20, - blue: 223, - } - )) - ); + match hex_color("#2F14DF") { + Ok((_, color)) => println!("Successfully parsed color: {color:?}"), + Err(_) => eprintln!("Failed to parse color") + } } ``` ## Documentation - - [Reference documentation](https://docs.rs/nom) - [The Nominomicon: A Guide To Using Nom](https://tfpk.github.io/nominomicon/) - [Various design documents and tutorials](https://github.com/rust-bakery/nom/tree/main/doc) - [List of combinators and their behaviour](https://github.com/rust-bakery/nom/blob/main/doc/choosing_a_combinator.md) -If you need any help developing your parsers, please ping `geal` on IRC (Libera, Geeknode, OFTC), go to `#nom-parsers` on Libera IRC, or on the [Gitter chat room](https://gitter.im/Geal/nom). - -## Why use nom +> [!TIP] +> If you need any help developing your parsers, please ping `geal` on IRC (Libera, Geeknode, OFTC), go to `#nom-parsers` on Libera IRC, or on the [Gitter chat room](https://gitter.im/Geal/nom). +## Why use `nom` If you want to write: ### Binary format parsers - -nom was designed to properly parse binary formats from the beginning. Compared -to the usual handwritten C parsers, nom parsers are just as fast, free from +`nom` was designed to properly parse binary formats from the beginning. Compared +to the usual handwritten C parsers, `nom` parsers are just as fast, free from buffer overflow vulnerabilities, and handle common patterns for you: - [TLV](https://en.wikipedia.org/wiki/Type-length-value) @@ -123,50 +114,49 @@ buffer overflow vulnerabilities, and handle common patterns for you: Example projects: -- [FLV parser](https://github.com/rust-av/flavors) -- [Matroska parser](https://github.com/rust-av/matroska) -- [tar parser](https://github.com/Keruspe/tar-parser.rs) +- [Game Boy ROM](https://github.com/MarkMcCaskey/gameboy-rom-parser) +- [X.509 public key certificate standard](https://github.com/rusticata/x509-parser) +- [GIF](https://github.com/Geal/gif.rs) ### Text format parsers - -While nom was made for binary format at first, it soon grew to work just as +While `nom` was made for binary format at first, it soon grew to work just as well with text formats. From line based formats like CSV, to more complex, nested -formats such as JSON, nom can manage it, and provides you with useful tools: +formats such as JSON, `nom` can manage it, and provides you with useful tools: - Fast case insensitive comparison - Recognizers for escaped strings -- Regular expressions can be embedded in nom parsers to represent complex character patterns succinctly +- Regular expressions can be embedded in `nom` parsers to represent complex character patterns succinctly - Special care has been given to managing non ASCII characters properly Example projects: -- [HTTP proxy](https://github.com/sozu-proxy/sozu/blob/main/lib/src/protocol/h2/parser.rs) - [TOML parser](https://github.com/joelself/tomllib) +- [Fountain screenplay markup](https://github.com/adamchalmers/fountain-rs) +- [Distinguished Encoding Rules for certificates](https://github.com/rusticata/der-parser) ### Programming language parsers - While programming language parsers are usually written manually for more -flexibility and performance, nom can be (and has been successfully) used +flexibility and performance, `nom` can be (and has been successfully) used as a prototyping parser for a language. -nom will get you started quickly with powerful custom error types, that you +`nom` will get you started quickly with powerful custom error types, that you can leverage with [nom_locate](https://github.com/fflorent/nom_locate) to pinpoint the exact line and column of the error. No need for separate -tokenizing, lexing and parsing phases: nom can automatically handle whitespace +tokenizing, lexing and parsing phases: `nom` can automatically handle whitespace parsing, and construct an AST in place. Example projects: -- [PHP VM](https://github.com/tagua-vm/parser) -- [xshade shading language](https://github.com/xshade-lang/xshade) +- [SystemVerilog](https://github.com/dalance/sv-parser) +- [Askama templates](https://crates.io/crates/askama_parser/) +- [Filter for MeiliSearch](https://github.com/meilisearch/meilisearch/tree/main/crates/filter-parser) ### Streaming formats - While a lot of formats (and the code handling them) assume that they can fit the complete data in memory, there are formats for which we only get a part of the data at once, like network formats, or huge files. -nom has been designed for a correct behaviour with partial data: If there is -not enough data to decide, nom will tell you it needs more instead of silently +`nom` has been designed for a correct behaviour with partial data: If there is +not enough data to decide, `nom` will tell you it needs more instead of silently returning a wrong result. Whether your data comes entirely or in chunks, the result should be the same. @@ -174,11 +164,11 @@ It allows you to build powerful, deterministic state machines for your protocols Example projects: -- [HTTP proxy](https://github.com/sozu-proxy/sozu/blob/main/lib/src/protocol/h2/parser.rs) -- [Using nom with generators](https://github.com/rust-bakery/generator_nom) +- [HTTP proxy](https://github.com/sozu-proxy/sozu/blob/main/lib/src/protocol/proxy_protocol/parser.rs) +- [Matroska parser](https://github.com/rust-av/matroska) +- [Prometheus protocol](https://github.com/vectordotdev/vector/blob/master/lib/prometheus-parser/src/line.rs) ## Parser combinators - Parser combinators are an approach to parsers that is very different from software like [lex](https://en.wikipedia.org/wiki/Lex_(software)) and [yacc](https://en.wikipedia.org/wiki/Yacc). Instead of writing the grammar @@ -192,65 +182,41 @@ written with other parser approaches. This has a few advantages: - The parsers are small and easy to write -- The parsers components are easy to reuse (if they're general enough, please add them to nom!) +- The parsers components are easy to reuse (if they're general enough, please add them to `nom`!) - The parsers components are easy to test separately (unit tests and property-based tests) - The parser combination code looks close to the grammar you would have written - You can build partial parsers, specific to the data you need at the moment, and ignore the rest ## Technical features - -nom parsers are for: -- [x] **byte-oriented**: The basic type is `&[u8]` and parsers will work as much as possible on byte array slices (but are not limited to them) -- [x] **bit-oriented**: nom can address a byte slice as a bit stream -- [x] **string-oriented**: The same kind of combinators can apply on UTF-8 strings as well -- [x] **zero-copy**: If a parser returns a subset of its input data, it will return a slice of that input, without copying -- [x] **streaming**: nom can work on partial data and detect when it needs more data to produce a correct result -- [x] **descriptive errors**: The parsers can aggregate a list of error codes with pointers to the incriminated input slice. Those error lists can be pattern matched to provide useful messages. -- [x] **custom error types**: You can provide a specific type to improve errors returned by parsers -- [x] **safe parsing**: nom leverages Rust's safe memory handling and powerful types, and parsers are routinely fuzzed and tested with real world data. So far, the only flaws found by fuzzing were in code written outside of nom -- [x] **speed**: Benchmarks have shown that nom parsers often outperform many parser combinators library like Parsec and attoparsec, some regular expression engines and even handwritten C parsers - -Some benchmarks are available on [GitHub](https://github.com/rust-bakery/parser_benchmarks). +`nom` parsers are for: +- **byte-oriented**: The basic type is `&[u8]` and parsers will work as much as possible on byte array slices (but are not limited to them) +- **bit-oriented**: `nom` can address a byte slice as a bit stream +- **string-oriented**: The same kind of combinators can apply on UTF-8 strings as well +- **zero-copy**: If a parser returns a subset of its input data, it will return a slice of that input, without copying +- **streaming**: `nom` can work on partial data and detect when it needs more data to produce a correct result +- **descriptive errors**: The parsers can aggregate a list of error codes with pointers to the incriminated input slice. Those error lists can be pattern matched to provide useful messages. +- **custom error types**: You can provide a specific type to improve errors returned by parsers +- **safe parsing**: `nom` leverages Rust's safe memory handling and powerful types, and parsers are routinely fuzzed and tested with real world data. So far, the only flaws found by fuzzing were in code written outside of `nom` +- **speed**: Benchmarks have shown that `nom` parsers often outperform many parser combinators library like Parsec and attoparsec, some regular expression engines and even handwritten C parsers + +> [!NOTE] +> Some benchmarks are available on [GitHub](https://github.com/rust-bakery/parser_benchmarks). ## Rust version requirements (MSRV) +The 8.0 series of `nom` supports **Rustc version 1.65 or greater**. -The 8.0 series of nom supports **Rustc version 1.65 or greater**. - -The current policy is that this will only be updated in the next major nom release. - -## Installation - -nom is available on [crates.io](https://crates.io/crates/nom) and can be included in your Cargo enabled project like this: - -```toml -[dependencies] -nom = "8" -``` - -There are a few compilation features: - -* `alloc`: (activated by default) if disabled, nom can work in `no_std` builds without memory allocators. If enabled, combinators that allocate (like `many0`) will be available -* `std`: (activated by default, activates `alloc` too) if disabled, nom can work in `no_std` builds - -You can configure those features like this: - -```toml -[dependencies.nom] -version = "8" -default-features = false -features = ["alloc"] -``` +The current policy is that this will only be updated in the next major `nom` release. # Related projects +- [Get line and column info in `nom`'s input type](https://github.com/fflorent/nom_locate) +- [Using `nom` as lexer and parser](https://github.com/Rydgel/monkey-rust) +- [Using `nom` with generators](https://github.com/rust-bakery/generator_nom) -- [Get line and column info in nom's input type](https://github.com/fflorent/nom_locate) -- [Using nom as lexer and parser](https://github.com/Rydgel/monkey-rust) - -# Parsers written with nom +# Parsers written with `nom` +Here is a (non exhaustive) list of known projects using `nom`: -Here is a (non exhaustive) list of known projects using nom: - -- Text file formats: [Ceph Crush](https://github.com/cholcombe973/crushtool), +- Text file formats: +[Ceph Crush](https://github.com/cholcombe973/crushtool), [Cronenberg](https://github.com/ayrat555/cronenberg), [Email](https://github.com/deuxfleurs-org/eml-codec), [XFS Runtime Stats](https://github.com/ChrisMacNaughton/xfs-rs), @@ -264,12 +230,16 @@ Here is a (non exhaustive) list of known projects using nom: [PDB](https://github.com/TianyiShi2001/nom-pdb), [proto files](https://github.com/tafia/protobuf-parser), [Fountain screenplay markup](https://github.com/adamchalmers/fountain-rs), -[vimwiki](https://github.com/chipsenkbeil/vimwiki-rs/tree/master/vimwiki), [vimwiki_macros](https://github.com/chipsenkbeil/vimwiki-rs/tree/master/vimwiki_macros), -[Kconfig language](https://github.com/Mcdostone/nom-kconfig), [Askama templates](https://crates.io/crates/askama_parser/), [LP files](https://github.com/dandxy89/lp_parser_rs) +[vimwiki](https://github.com/chipsenkbeil/vimwiki-rs/tree/master/vimwiki), +[vimwiki_macros](https://github.com/chipsenkbeil/vimwiki-rs/tree/master/vimwiki_macros), +[Kconfig language](https://github.com/Mcdostone/nom-kconfig), +[Askama templates](https://crates.io/crates/askama_parser/), +[LP files](https://github.com/dandxy89/lp_parser_rs), +[TOML parser](https://github.com/joelself/tomllib) - Programming languages: [PHP](https://github.com/tagua-vm/parser), [Basic Calculator](https://github.com/balajisivaraman/basic_calculator_rs), -[GLSL](https://sr.ht/~hadronized/glsl) +[GLSL](https://sr.ht/~hadronized/glsl), [Lua](https://github.com/rozbb/nom-lua53), [Python](https://github.com/ProgVal/rust-python-parser), [SQL](https://github.com/ms705/nom-sql), @@ -279,29 +249,31 @@ Here is a (non exhaustive) list of known projects using nom: [CSML](https://github.com/CSML-by-Clevy/csml-engine/tree/dev/csml_interpreter), [Wasm](https://github.com/fabrizio-m/wasm-nom), [Pseudocode](https://github.com/Gungy2/pseudocod), -[Filter for MeiliSearch](https://github.com/meilisearch/meilisearch), +[Filter for MeiliSearch](https://github.com/meilisearch/meilisearch/tree/main/crates/filter-parser), [PotterScript](https://github.com/fmiras/potterscript), -[R](https://github.com/kpagacz/tergo) -- Interface definition formats: [Thrift](https://github.com/thehydroimpulse/thrust) +[R](https://github.com/kpagacz/tergo/tree/main/spongia), +[xshade shading language](https://github.com/xshade-lang/xshade) - Audio, video and image formats: [GIF](https://github.com/Geal/gif.rs), [MagicaVoxel .vox](https://github.com/dust-engine/dot_vox), [MIDI](https://github.com/derekdreery/nom-midi-rs), [SWF](https://github.com/open-flash/swf-parser), [WAVE](https://github.com/Noise-Labs/wave), +[FLV parser](https://github.com/rust-av/flavors), [Matroska (MKV)](https://github.com/rust-av/matroska), [Exif/Metadata parser for JPEG/HEIF/HEIC/MOV/MP4](https://github.com/mindeng/nom-exif) - Document formats: [TAR](https://github.com/Keruspe/tar-parser.rs), [GZ](https://github.com/nharward/nom-gzip), -[GDSII](https://github.com/erihsu/gds2-io) +[GDSII](https://github.com/erihsu/gds2-io), +[PDF](https://github.com/J-F-Liu/lopdf) - Cryptographic formats: [X.509](https://github.com/rusticata/x509-parser) - Network protocol formats: [Bencode](https://github.com/jbaum98/bencode.rs), [D-Bus](https://github.com/toshokan/misato), [DHCP](https://github.com/rusticata/dhcp-parser), -[HTTP](https://github.com/sozu-proxy/sozu/tree/main/lib/src/protocol/http), +[HTTP](https://github.com/sozu-proxy/sozu/blob/main/lib/src/protocol/proxy_protocol/parser.rs), [URI](https://github.com/santifa/rrp/blob/master/src/uri.rs), [IMAP](https://github.com/djc/tokio-imap) ([alt](https://github.com/duesee/imap-codec)), [IRC](https://github.com/Detegr/RBot-parser), @@ -327,20 +299,19 @@ Here is a (non exhaustive) list of known projects using nom: [ANT FIT](https://github.com/stadelmanma/fitparse-rs), [Version Numbers](https://github.com/fosskers/rs-versions), [Telcordia/Bellcore SR-4731 SOR OTDR files](https://github.com/JamesHarrison/otdrs), -[MySQL binary log](https://github.com/PrivateRookie/boxercrab), [URI](https://github.com/Skasselbard/nom-uri), [Furigana](https://github.com/sachaarbonel/furigana.rs), [Wordle Result](https://github.com/Fyko/wordle-stats/tree/main/parser), [NBT](https://github.com/phoenixr-codes/mcnbt) -Want to create a new parser using `nom`? A list of not yet implemented formats is available [here](https://github.com/rust-bakery/nom/issues/14). - -Want to add your parser here? Create a pull request for it! +> [!NOTE] +> Want to create a new parser using `nom`? A list of not yet implemented formats is available [here](https://github.com/rust-bakery/nom/issues/14). +> +> Want to add your parser here? Create a [pull request](https://github.com/rust-bakery/nom/compare) for it! # Contributors - -nom is the fruit of the work of many contributors over the years, many thanks for your help! +`nom` is the fruit of the work of many contributors over the years, many thanks for your help! - +