Skip to content

Add IOBuf<->std::iostream adapters for zero-copy I/O#3341

Merged
chenBright merged 1 commit into
apache:masterfrom
chenBright:iobuf_iostream
Jun 14, 2026
Merged

Add IOBuf<->std::iostream adapters for zero-copy I/O#3341
chenBright merged 1 commit into
apache:masterfrom
chenBright:iobuf_iostream

Conversation

@chenBright

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: resolve

Problem Summary:

IOBuf integrates with protobuf via IOBufAsZeroCopy{Input,Output}Stream, but
there is no direct bridge to the std::iostream hierarchy. As a result, any
library that consumes/produces std::istream& / std::ostream& — most notably
nlohmann::json — cannot read from or write to an IOBuf without an
intermediate std::string:

auto j = nlohmann::json::parse(req->request_attachment().to_string());

This is an obstacle to adopting nlohmann::json in bRPC.

What is changed and the side effects?

Changed:

Introduce four classes in src/butil/iobuf.{h,cpp} that bridge IOBuf and
the standard iostream hierarchy:

  • IOBufAsInputStreamBuf — read-only std::streambuf over an IOBuf,
    walking the backing blocks via the public backing_block() API. Overrides
    underflow, xsgetn(bulk memcpy across block boundaries to avoid the
    default per-byte sbumpc loop), and showmanyc (saturating sum of
    remaining block sizes).
  • IOBufInputStreamstd::istream view that wires the streambuf above.
  • IOBufAsOutputStreamBuf — append-only std::streambuf that delegates to
    the existing IOBufAsZeroCopyOutputStream for block allocation and the
    three-branch BackUp (including the shared-block degradation that protects
    IOBufs sharing blocks from being corrupted by an in-flight write). Overrides
    overflow, xsputn (block-sized memcpy instead of per-byte sputc),
    and sync (forwards to shrink()).
  • IOBufOutputStreamstd::ostream view that wires the streambuf above;
    accepts an optional block_size to allocate dedicated blocks instead of
    drawing from the per-thread TLS pool.

Usage with nlohmann::json:

  // Parse: zero string copy
  butil::IOBufInputStream in(request_body);
  auto j = nlohmann::json::parse(in);

  // Serialize: bytes flow straight into IOBuf blocks
  butil::IOBufOutputStream os(response_body);
  os << j;

Side effects:

  • Performance effects:

  • Breaking backward compatibility:


Check List:

Introduce four classes that bridge IOBuf and the standard iostream
hierarchy, enabling parsers and serializers that take std::istream& /
std::ostream& (e.g. nlohmann::json) to read from and write to IOBuf
directly without an intermediate string copy:

  - IOBufAsInputStreamBuf  : read-only streambuf over IOBuf blocks
  - IOBufInputStream       : std::istream view over IOBuf
  - IOBufAsOutputStreamBuf : append-only streambuf, reusing
                             IOBufAsZeroCopyOutputStream's TLS block pool
  - IOBufOutputStream      : std::ostream view over IOBuf
@wwbmmm

wwbmmm commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

LGTM

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds zero-copy adapters between butil::IOBuf and the standard std::istream/std::ostream ecosystem by introducing std::streambuf-based wrappers, enabling direct integration with libraries like nlohmann::json without converting through std::string.

Changes:

  • Add IOBufAs{Input,Output}StreamBuf plus IOBuf{Input,Output}Stream wrappers in src/butil/iobuf.{h,cpp}.
  • Add unit tests validating correctness across multi-block buffers and (optionally) round-tripping with nlohmann::json.
  • Add Bazel dependencies for nlohmann_json via both WORKSPACE (http_archive) and MODULE.bazel (bzlmod), and wire it into the test target.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
WORKSPACE Adds http_archive for nlohmann_json to support new adapter tests.
MODULE.bazel Adds nlohmann_json as a Bazel module dev dependency for tests.
test/BUILD.bazel Enables JSON-backed tests and links @nlohmann_json//:json into butil_unittests.
test/iobuf_unittest.cpp Adds extensive tests for the new iostream adapters and JSON round-trip coverage.
src/butil/iobuf.h Declares the new streambuf and iostream adapter classes with usage documentation.
src/butil/iobuf.cpp Implements the new streambuf adapters (underflow/xsgetn/showmanyc and overflow/xsputn/sync).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/butil/iobuf.cpp
Comment thread src/butil/iobuf.cpp
Comment thread src/butil/iobuf.cpp
Comment thread test/iobuf_unittest.cpp
Comment thread src/butil/iobuf.h
@chenBright chenBright merged commit 4cd0a72 into apache:master Jun 14, 2026
25 of 26 checks passed
@chenBright chenBright deleted the iobuf_iostream branch June 14, 2026 04:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants