Skip to content

Support streaming replication #418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
cthulhu-rider wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Support streaming replication #418

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a935b31
WIP
cthulhu-rider Sep 1, 2026
74bea57
PUT-like
cthulhu-rider Sep 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions object/service.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,9 @@ service ObjectService {
// - **CONTAINER_NOT_FOUND** (3072, SECTION_CONTAINER): \
// the container to which the replicated object is associated was not found.
rpc Replicate(ReplicateRequest) returns (ReplicateResponse);

// TODO: docs.
rpc ReplicateV2(stream ReplicateRequest) returns (ReplicateResponse);

@cthulhu-rider cthulhu-rider Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my initial thought how to make new op as close to Replicate as possible is to stream its requests. Require additional messages to have object.payload (i.e. chunk) field only

having this, for new client to go back to old API, the only things to be done are

  1. change method name to Replicate
  2. place full object into the single message

2 will be needed anyway to implement a size-dependent logic transmitting small objects in one message

@roman-khimov roman-khimov Sep 2, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Object size is known before the request is made, so I'd opt for simpler v2 protocol (two specific request-level fields for header and payload). Magic payload/non-payload handling is harder to implement to me. ReplicateRequest can be extended with payload field as well leaving object for complete object or header only depending on version. Needs to be checked against real code anyway.

@cthulhu-rider cthulhu-rider Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as in #333, the main thing to decide is whether 1st message carries payload prefix (full payload in particular) or not

if no, we'll have PutRequest-like stream w/o meta/verification headers. Otherwise, prefix and extra chunk fields can be transmitted in the same field (currently proposed) or in the diff ones (PUT)

i'll make diff commits for all opts to make it clear

}

// GET object request.
Expand Down
Loading