Skip to content
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
fa94e6b
PBM-1715 Parallel file processing for backups and restores
rasika-chivate Jun 11, 2026
e022790
Added restore flag
rasika-chivate Jun 11, 2026
b36ea88
Update backup-physical.md
rasika-chivate Jun 11, 2026
c39a56d
Update restore-physical.md
rasika-chivate Jun 11, 2026
99db9e3
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
c21aad3
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
7ae0a39
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
aad2bf0
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
132ad22
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
f6fbad7
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
4181d3c
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
4b3fe64
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
53ab1e1
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
ffc30b4
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
303c5a3
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
94232ca
self review
rasika-chivate Jun 11, 2026
5ce1aa2
Remove extra trailing blank lines in backup options docs
Copilot Jun 11, 2026
b73439f
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
fdaa87d
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
4f9fc29
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
66f72da
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
b2d07f7
Add numParallelFiles to backup YAML example
Copilot Jun 11, 2026
d08cf9a
Remove trailing blank lines in restore options doc
Copilot Jun 11, 2026
f5c0f06
Add restore numParallelFiles to YAML example
Copilot Jun 11, 2026
aef469c
Add restore numParallelFiles to YAML example
Copilot Jun 11, 2026
2973877
Remove generated cache artifact and ignore .cache
Copilot Jun 11, 2026
c43bd8d
docs: add num-parallel-files to pbm backup flags
Copilot Jun 11, 2026
badea9e
Document --num-parallel-files for pbm restore
Copilot Jun 11, 2026
22d720f
Update backup-physical.md
rasika-chivate Jun 11, 2026
231d400
Potential fix for pull request finding
rasika-chivate Jun 11, 2026
d4d074d
Update backup-physical.md
rasika-chivate Jun 11, 2026
24216b9
Merge branch 'PBM-1715-Parallel-file-processing-for-backups-and-resto…
rasika-chivate Jun 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/reference/backup-options.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,21 @@ The duration (in minutes) of oplog slices saved with the logical backup snapshot
The number of parallel collections to process during a logical backup. By default, the number of parallel collections is half of the number of CPU cores. By setting the value for this option you define the new default.
Available starting with version 2.7.0.


## backup.numParallelFiles

*Type*: int <br>
*Default*: 1 <br>
*Storage*: Filesystem only <br>
Comment thread
Copilot marked this conversation as resolved.
Outdated
*Backup type:* Physical only
Comment thread
rasika-chivate marked this conversation as resolved.

The number of files to copy in parallel during a physical backup
to filesystem or NFS storage.

The default value of `1` copies files sequentially. If parameter is not specified, files will be uploaded/downloaded sequentially.
Comment thread
Copilot marked this conversation as resolved.
Outdated






Comment thread
rasika-chivate marked this conversation as resolved.
Outdated
19 changes: 18 additions & 1 deletion docs/reference/restore-options.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,4 +99,21 @@ This is useful when you want to:

- Prevent restore operations from waiting indefinitely
- Enforce time limits in automated workflows
- Fail fast if the balancer cannot be stopped
- Fail fast if the balancer cannot be stopped


## restore.numParallelFiles
Comment thread
Copilot marked this conversation as resolved.
Outdated

*Type*: int <br>
*Default*: 1 <br>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be precise, there is no default actually. If it is not specified, it's zero and that's the same as 1, which means sequentially.

*Storage*: Filesystem only <br>
Comment thread
Copilot marked this conversation as resolved.
Outdated
*Restore type:* Physical only
Comment thread
rasika-chivate marked this conversation as resolved.

The number of files to copy in parallel during a physical restore
from filesystem or NFS storage.

The default value of `1` copies files sequentially. If parameter is not specified, files will be uploaded/downloaded sequentially.
Comment thread
Copilot marked this conversation as resolved.
Outdated




24 changes: 24 additions & 0 deletions docs/usage/backup-physical.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,30 @@ pbm backup --type=physical

Starting with [2.4.0](../release-notes/2.4.0.md), PBM doesn't stop [point-in-time recovery oplog slicing](../features/point-in-time-recovery.md#oplog-slicing), if it's enabled, but runs it in parallel. This ensures [point-in-time recovery](pitr-tutorial.md) to any timestamp if it takes too long (e.g. hours) to make a backup snapshot.

## Improve backup performance on filesystem storage

For physical backups stored on a filesystem, you can control how many files PBM processes in parallel during backup and restore operations. Increasing parallelism can improve performance by allowing multiple files to be transferred or copied simultaneously.
Comment thread
Copilot marked this conversation as resolved.
Outdated


To copy files in parallel, set `backup.numParallelFiles` in the
PBM configuration:
Comment thread
Copilot marked this conversation as resolved.
Outdated

```yaml
backup:
numParallelFiles: 4
```

Or pass `--num-parallel-files`
Comment thread
Copilot marked this conversation as resolved.
Outdated

```sh
pbm backup --num-parallel-files=4
Comment thread
Copilot marked this conversation as resolved.
Outdated
```
Comment thread
rasika-chivate marked this conversation as resolved.
Comment thread
rasika-chivate marked this conversation as resolved.

!!! note
Parallel file copy applies to **physical backups to filesystem
or NFS storage only**. It has no effect on logical backups or
on S3-compatible storage.
Comment thread
Copilot marked this conversation as resolved.
Outdated

## Next steps

[List backups](../usage/list-backup.md){.md-button}
Expand Down
22 changes: 22 additions & 0 deletions docs/usage/restore-physical.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,28 @@ restore:
* `maxDownloadBufferMb` - the maximum size of memory buffer to store the downloaded data chunks for decompression and ordering. It is calculated as `numDownloadWorkers * downloadChunkMb * 16`
* `downloadChunkMb` is the size of the data chunk to download (by default, 32 MB)

## Improve restore performance on filesystem storage

For physical restores from filesystem storage, you can control how many files PBM copies in parallel during the restore operation. Increasing the number of parallel file copies can reduce restore time by allowing multiple files to be processed simultaneously, depending on the available system and storage resources.
Comment thread
rasika-chivate marked this conversation as resolved.
Outdated

To copy files in parallel, set `restore.numParallelFiles` in the
PBM configuration:
Comment thread
Copilot marked this conversation as resolved.
Outdated

```yaml
restore:
numParallelFiles: 4
```

Or pass `--num-parallel-files`:

```sh
pbm restore <backup_name> --num-parallel-files=4
```
Comment thread
rasika-chivate marked this conversation as resolved.

!!! note
Parallel file copy applies to **physical restores from
filesystem or NFS storage only**. It has no effect on logical
restores or on S3-compatible storage.
Comment thread
Copilot marked this conversation as resolved.
Outdated

## Next steps

Expand Down