PBM-1715 Parallel file processing for backups and restores

.gitignore

@@ -9,6 +9,7 @@ venv/
 # Local Netlify folder
 .netlify
 site
+.cache/
 
 styles/
 

docs/reference/backup-options.md

@@ -13,6 +13,7 @@ backup:
     startingStatus: 60
   oplogSpanMin: <float64>
   numParallelCollections: <int>
+  numParallelFiles: <int>
 ```
 
 ## priority
@@ -121,3 +122,14 @@ The duration (in minutes) of oplog slices saved with the logical backup snapshot
 The number of parallel collections to process during a logical backup. By default, the number of parallel collections is half of the number of CPU cores. By setting the value for this option you define the new default.
 Available starting with version 2.7.0.
 
+
+## backup.numParallelFiles
+
+*Type*: int <br>
+*Default*: 1 <br>
+*Storage*: Filesystem / NFS only <br>
+*Backup type:* Physical only
+
+The number of files to copy in parallel during a physical backup to filesystem or NFS storage.
+
+The default value is `1`, which copies files sequentially.

docs/reference/pbm-commands.md

@@ -25,6 +25,7 @@ The command accepts the following flags:
 | `--compression`| Create a backup with compression. <br> Supported compression methods: `gzip`, `snappy`, `lz4`, `s2`, `pgzip`, `zstd`. Default: `s2` <br> The `none` value means no compression is done during backup. |
 | `--compression-level` | Configure the compression level from 0 to 10. The default value depends on the compression method used.  |
 | `--num-parallel-collections`| Sets the number of collections to process in parallel during a specific logical backup. When undefined, `pbm-agent` processes the number of parallel collections defined for the `backup.numParallelCollections` configuration parameter. If that is undefined, the default number of collections to process in parallel is the half of the number of logical CPUs. Available starting with version 2.7.0.|
+| `--num-parallel-files`| Sets the number of files to process in parallel during a specific physical backup. When undefined, `pbm-agent` uses the value defined for the `backup.numParallelFiles` configuration parameter. If that is undefined, the default number of files to process in parallel is `1`. |
 | `-o`, `--out=text`    | Shows the output format as either plain text or a JSON object. Supported values: `text`, `json` |
 | `--wait`       | Wait for the backup to finish. The flag blocks the shell session.|
 | `--wait-time`  | The time to wait for PBM to report the status of the command execution. Use this flag together with the `--wait` flag. You can specify the duration in minutes or hours (for example 5m, 1h). <br><br>When not set, PBM waits till the command executes. <br><br>If it takes longer than the defined waiting time to execute the command, PBM prints the `Operation is in progress. Check pbm status and logs` error message and unblocks the shell session. The `pbm-agent` continues to execute the command enabling you to track its progress via the `pbm status` command. Available starting with version 2.6.0.| 
@@ -940,6 +941,7 @@ The command accepts the following flags:
 | `--with-users-and-roles` | Restores users and roles created in custom databases during selective restore. Use this flag with the `--ns` flag. Available starting with version 2.5.0.| 
 | `-c`, `--config`     | The path to the `mongod.conf` file |
 | `--num-parallel-collections`| Sets the number of collections to process in parallel during a specific logical restore. When undefined, `pbm-agent` processes the number of parallel collections defined for the `restore.numParallelCollections` configuration parameter. If that is undefined,  the default number of collections is the half of the number of logical CPUs. Available starting with version 2.7.0.|
+| `--num-parallel-files` | Sets the number of files to copy in parallel during a specific physical restore. When undefined, PBM uses the value configured for the `restore.numParallelFiles` option. |
 | `--num-insertion-workers-per-collection`| Specifies the number of insertion workers to run concurrently per collection. Increasing the number for large import may increase the speed of the import. Available starting with version 2.8.0.|
 | `--ns-from`="database.collection" |Specifies the name of the collection you want to restore under a new name. It indicates the collection present in the backup that you are targeting for restoration. Available starting with version 2.8.0.|
 | `--ns-to`="database.collection" | Specifies the new name for the collection you are restoring from the original one. Available starting with version 2.8.0.|

docs/reference/restore-options.md

@@ -6,6 +6,7 @@ restore:
   numInsertionWorkers: <int>
   numParallelCollections: <int>
   numDownloadWorkers: <int>
+  numParallelFiles: <int>
   maxDownloadBufferMb: <int>
   downloadChunkMb: <int>
   mongodLocation: <string>
@@ -99,4 +100,16 @@ This is useful when you want to:
 
 - Prevent restore operations from waiting indefinitely
 - Enforce time limits in automated workflows
-- Fail fast if the balancer cannot be stopped
+- Fail fast if the balancer cannot be stopped
+
+
+### restore.numParallelFiles
+
+*Type*: int <br>
+*Default*: 1 <br>
+*Storage*: Filesystem / NFS only <br>
+*Restore type:* Physical only
+
+The number of files to copy in parallel during a physical restore from filesystem or NFS storage.
+
+The default value is `1`, which copies files sequentially.

docs/usage/backup-physical.md

@@ -18,6 +18,26 @@ pbm backup --type=physical
 
 Starting with [2.4.0](../release-notes/2.4.0.md), PBM doesn't stop [point-in-time recovery oplog slicing](../features/point-in-time-recovery.md#oplog-slicing), if it's enabled, but runs it in parallel. This ensures [point-in-time recovery](pitr-tutorial.md) to any timestamp if it takes too long (e.g. hours) to make a backup snapshot.
 
+## Parallel file copy for filesystem storage
+
+For physical backups stored on a filesystem, you can control how many files PBM processes in parallel during the backup operation. Increasing parallelism can improve performance by allowing multiple files to be copied simultaneously.
+
+To copy files in parallel, set `backup.numParallelFiles` in the PBM configuration:
+
+```yaml
+backup:
+  numParallelFiles: 4
+```
+
+Or pass `--num-parallel-files`:
+
+```sh
+pbm backup --type=physical --num-parallel-files=4
+```
+
+!!! note
+    Parallel file copy applies to **physical backups or incremental backups to filesystem only**. It has no effect on logical backups or on S3-compatible storage.
+
 ## Next steps
 
 [List backups](../usage/list-backup.md){.md-button}

docs/usage/restore-physical.md

@@ -151,6 +151,25 @@ restore:
 * `maxDownloadBufferMb` - the maximum size of memory buffer to store the downloaded data chunks for decompression and ordering. It is calculated as `numDownloadWorkers * downloadChunkMb * 16`
 * `downloadChunkMb` is the size of the data chunk to download (by default, 32 MB)
 
+### Parallel file copy for filesystem or NFS storage
+
+By default, PBM copies backup files sequentially during a physical restore from filesystem or NFS storage. On fast NFS mounts, this limits restore throughput regardless of available network bandwidth.
+
+To copy files in parallel, set `restore.numParallelFiles` in the PBM configuration:
+
+```yaml
+restore:
+  numParallelFiles: 4
+```
+
+Or pass `--num-parallel-files`:
+
+```sh
+pbm restore <backup_name> --num-parallel-files=4
+```
+
+!!! note
+    Parallel file copy applies to **physical restores from filesystem or NFS storage only**. It has no effect on logical restores or on S3-compatible storage.
 
 ## Next steps