diff --git a/.gitignore b/.gitignore index 4da6fc20..a0962e49 100644 --- a/.gitignore +++ b/.gitignore @@ -9,6 +9,7 @@ venv/ # Local Netlify folder .netlify site +.cache/ styles/ diff --git a/docs/reference/backup-options.md b/docs/reference/backup-options.md index 5f5c052e..c05696a3 100644 --- a/docs/reference/backup-options.md +++ b/docs/reference/backup-options.md @@ -13,6 +13,7 @@ backup: startingStatus: 60 oplogSpanMin: numParallelCollections: + numParallelFiles: ``` ## priority @@ -121,3 +122,14 @@ The duration (in minutes) of oplog slices saved with the logical backup snapshot The number of parallel collections to process during a logical backup. By default, the number of parallel collections is half of the number of CPU cores. By setting the value for this option you define the new default. Available starting with version 2.7.0. + +## backup.numParallelFiles + +*Type*: int
+*Default*: 1
+*Storage*: Filesystem / NFS only
+*Backup type:* Physical only + +The number of files to copy in parallel during a physical backup to filesystem or NFS storage. + +The default value is `1`, which copies files sequentially. diff --git a/docs/reference/pbm-commands.md b/docs/reference/pbm-commands.md index f1c54b23..3820ab99 100644 --- a/docs/reference/pbm-commands.md +++ b/docs/reference/pbm-commands.md @@ -25,6 +25,7 @@ The command accepts the following flags: | `--compression`| Create a backup with compression.
Supported compression methods: `gzip`, `snappy`, `lz4`, `s2`, `pgzip`, `zstd`. Default: `s2`
The `none` value means no compression is done during backup. | | `--compression-level` | Configure the compression level from 0 to 10. The default value depends on the compression method used. | | `--num-parallel-collections`| Sets the number of collections to process in parallel during a specific logical backup. When undefined, `pbm-agent` processes the number of parallel collections defined for the `backup.numParallelCollections` configuration parameter. If that is undefined, the default number of collections to process in parallel is the half of the number of logical CPUs. Available starting with version 2.7.0.| +| `--num-parallel-files`| Sets the number of files to process in parallel during a specific physical backup. When undefined, `pbm-agent` uses the value defined for the `backup.numParallelFiles` configuration parameter. If that is undefined, the default number of files to process in parallel is `1`. | | `-o`, `--out=text` | Shows the output format as either plain text or a JSON object. Supported values: `text`, `json` | | `--wait` | Wait for the backup to finish. The flag blocks the shell session.| | `--wait-time` | The time to wait for PBM to report the status of the command execution. Use this flag together with the `--wait` flag. You can specify the duration in minutes or hours (for example 5m, 1h).

When not set, PBM waits till the command executes.

If it takes longer than the defined waiting time to execute the command, PBM prints the `Operation is in progress. Check pbm status and logs` error message and unblocks the shell session. The `pbm-agent` continues to execute the command enabling you to track its progress via the `pbm status` command. Available starting with version 2.6.0.| @@ -940,6 +941,7 @@ The command accepts the following flags: | `--with-users-and-roles` | Restores users and roles created in custom databases during selective restore. Use this flag with the `--ns` flag. Available starting with version 2.5.0.| | `-c`, `--config` | The path to the `mongod.conf` file | | `--num-parallel-collections`| Sets the number of collections to process in parallel during a specific logical restore. When undefined, `pbm-agent` processes the number of parallel collections defined for the `restore.numParallelCollections` configuration parameter. If that is undefined, the default number of collections is the half of the number of logical CPUs. Available starting with version 2.7.0.| +| `--num-parallel-files` | Sets the number of files to copy in parallel during a specific physical restore from filesystem or NFS storage. When undefined, PBM uses the value configured for the `restore.numParallelFiles` option (default: `1`). | | `--num-insertion-workers-per-collection`| Specifies the number of insertion workers to run concurrently per collection. Increasing the number for large import may increase the speed of the import. Available starting with version 2.8.0.| | `--ns-from`="database.collection" |Specifies the name of the collection you want to restore under a new name. It indicates the collection present in the backup that you are targeting for restoration. Available starting with version 2.8.0.| | `--ns-to`="database.collection" | Specifies the new name for the collection you are restoring from the original one. Available starting with version 2.8.0.| diff --git a/docs/reference/restore-options.md b/docs/reference/restore-options.md index 6747acbe..02490c64 100644 --- a/docs/reference/restore-options.md +++ b/docs/reference/restore-options.md @@ -6,6 +6,7 @@ restore: numInsertionWorkers: numParallelCollections: numDownloadWorkers: + numParallelFiles: maxDownloadBufferMb: downloadChunkMb: mongodLocation: @@ -99,4 +100,16 @@ This is useful when you want to: - Prevent restore operations from waiting indefinitely - Enforce time limits in automated workflows -- Fail fast if the balancer cannot be stopped \ No newline at end of file +- Fail fast if the balancer cannot be stopped + + +### restore.numParallelFiles + +*Type*: int
+*Default*: 1
+*Storage*: Filesystem / NFS only
+*Restore type:* Physical only + +The number of files to copy in parallel during a physical restore from filesystem or NFS storage. + +The default value is `1`, which copies files sequentially. diff --git a/docs/usage/backup-physical.md b/docs/usage/backup-physical.md index 62f21db5..b2cd81dc 100644 --- a/docs/usage/backup-physical.md +++ b/docs/usage/backup-physical.md @@ -18,6 +18,26 @@ pbm backup --type=physical Starting with [2.4.0](../release-notes/2.4.0.md), PBM doesn't stop [point-in-time recovery oplog slicing](../features/point-in-time-recovery.md#oplog-slicing), if it's enabled, but runs it in parallel. This ensures [point-in-time recovery](pitr-tutorial.md) to any timestamp if it takes too long (e.g. hours) to make a backup snapshot. +## Parallel file copy for filesystem storage + +For physical backups stored on a filesystem, you can control how many files PBM processes in parallel during the backup operation. Increasing parallelism can improve performance by allowing multiple files to be copied simultaneously. + +To copy files in parallel, set `backup.numParallelFiles` in the PBM configuration: + +```yaml +backup: + numParallelFiles: 4 +``` + +Or pass `--num-parallel-files`: + +```sh +pbm backup --type=physical --num-parallel-files=4 +``` + +!!! note + Parallel file copy applies to **physical backups and incremental physical backups stored on filesystem only**. It has no effect on logical backups or on any S3-compatible storage, regardless of backup type. + ## Next steps [List backups](../usage/list-backup.md){.md-button} diff --git a/docs/usage/restore-physical.md b/docs/usage/restore-physical.md index 9b8c7569..534cad7b 100644 --- a/docs/usage/restore-physical.md +++ b/docs/usage/restore-physical.md @@ -151,6 +151,25 @@ restore: * `maxDownloadBufferMb` - the maximum size of memory buffer to store the downloaded data chunks for decompression and ordering. It is calculated as `numDownloadWorkers * downloadChunkMb * 16` * `downloadChunkMb` is the size of the data chunk to download (by default, 32 MB) +### Parallel file copy for filesystem or NFS storage + +By default, PBM copies backup files sequentially during a physical restore from filesystem or NFS storage. On fast NFS mounts, this limits restore throughput regardless of available network bandwidth. + +To copy files in parallel, set `restore.numParallelFiles` in the PBM configuration: + +```yaml +restore: + numParallelFiles: 4 +``` + +Or pass `--num-parallel-files`: + +```sh +pbm restore --num-parallel-files=4 +``` + +!!! note + Parallel file copy applies to **physical restores from filesystem or NFS storage only**. It has no effect on logical restores or on S3-compatible storage. ## Next steps