DOC-13858 Server File-Based Rebalance for Data Service

modules/ROOT/nav.adoc

@@ -358,6 +358,7 @@ include::cli:partial$cbcli/nav.adoc[]
     **** xref:rest-api:rest-cluster-removenode.adoc[Removing Nodes from Clusters]
 
    *** xref:rest-api:rest-rebalance-overview.adoc[Rebalance]
+    **** xref:rest-api:file-based-data-rebalance.adoc[]
     **** xref:rest-api:rest-retrieve-cluster-rebalance-reason-codes.adoc[Getting Rebalance Reason Codes]
     **** xref:rest-api:rest-cluster-rebalance.adoc[Rebalancing the Cluster]
     **** xref:rest-api:rest-get-rebalance-progress.adoc[Getting Rebalance Progress]

modules/introduction/partials/new-features-81.adoc

@@ -15,7 +15,29 @@ TBD
 
 Couchbase Server 8.1 introduces several new features for the Data Service.
 
-TBD
+==== File-Based Rebalance (FBR)
+
+Couchbase Server 8.1 introduces File-Based Rebalance (FBR) for the Data Service.
+FBR accelerates cluster rebalance by copying vBucket storage files directly between nodes rather than streaming data through the DCP (Database Change Protocol) replication pipeline.
+This eliminates the serialization and pipeline overhead of DCP backfill for large, disk-resident datasets.
+
+The following changes apply to the Data Service rebalance behavior in 8.1:
+
+* *File-Based Rebalance*: FBR transfers vBucket data files directly from the source node to the destination node during the backfill phase of a vBucket move, bypassing the full DCP backfill mechanism used in prior releases.
+
+* *Enabled by default*: FBR is enabled by default for Enterprise Edition, both for self-managed deployments and Couchbase Capella.
+No configuration is required to activate it.
+
+* *Automatic rebalance type selection*: The server automatically determines whether FBR or DCP is more efficient for each vBucket move.
+When FBR is not applicable or not expected to be faster, the server falls back to DCP automatically.
+
+* *New bucket-level rebalance type setting*: A new per-bucket setting, `dataServiceRebalanceType`, allows operators to control rebalance behavior at the bucket level, overriding the cluster-level FBR setting.
+
+* *Separate vBucket move concurrency for FBR*: A new setting, `dataServiceFileBasedRebalanceMovesPerNode`, controls the maximum number of concurrent file-based vBucket moves per node.
+This is independent of the existing `rebalanceMovesPerNode` setting, which applies to DCP rebalance.
+
+NOTE: FBR is an Enterprise Edition feature.
+Community Edition continues to use DCP-based rebalance for all vBucket moves.
 
 
 === Non-Data Services

modules/manage/pages/manage-nodes/add-node-and-rebalance.adoc

@@ -186,6 +186,10 @@ Note that the figure in the *Items* column for node `10.142.181.101` is `31.5 K/
 The figure for `10.142.181.102` indicates the converse.
 Therefore, replication has successfully distributed the contents of `travel-sample` across both nodes, providing a single replica vBucket for each active vBucket.
 
+NOTE: By default, Couchbase Server Enterprise Edition automatically uses File-Based Rebalance (FBR) to move data for eligible vBuckets during node addition.
+The server selects the optimal rebalance method for each vBucket move transparently.
+For information, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)].
+
 [#node-information-within-the-ui]
 ==== Node Information Within the UI
 

modules/manage/pages/manage-settings/general-settings.adoc

@@ -193,6 +193,16 @@ The *Max moves per node during rebalance* option establishes the maximum number
 The minimum value for the parameter is `1`, the maximum `64`, the default `4`.
 For information, see xref:learn:clusters-and-availability/rebalance.adoc#limiting-concurrent-vbucket-moves[Limiting Concurrent vBucket Moves].
 
+Couchbase Server Enterprise Edition clusters have an additional control named *Maximum Concurrent vBucket Moves (File-Based)*.
+It controls how many vBucket files a node can copy concurrently when performing a File-Based Rebalance (FBR).
+The range for this setting is from `1` to `1024`.
+This setting only applies to FBR-based vBucket moves and is independent of the DCP concurrent-moves setting above.
+
+The *Retry rebalance* option applies to both DCP and FBR rebalance types.
+No separate configuration is required; retry behavior is the same regardless of which rebalance method was used for the failed attempt.
+
+For information on FBR, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)].
+
 [#data-settings]
 === Data Settings
 The fields that appear when you expand the *Advanced Data Settings* section let you control filesystem use limits and I/O thread allocation.

modules/rest-api/pages/file-based-data-rebalance.adoc

@@ -0,0 +1,217 @@
+= Configure File-Based Data Rebalance
+:description: pass:q[You control Data Service File-Based Rebalance (FBR) using the `/internalSettings` REST API endpoint.]
+:page-edition: Enterprise Edition
+:page-topic-type: reference
+:page-toclevels: 3
+
+[abstract]
+{description}
+
+== Description
+
+The file-based rebalance option for the Data Service directly copies vBucket data files between nodes during rebalance. 
+This usually improves rebalance speed compared to DCP-based backfill method.
+See xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance] for more information about file-based rebalance and how it compares to DCP-based rebalance. 
+
+== HTTP Methods 
+
+This API endpoint supports the following methods:
+
+* <<#get-settings>>
+* <<#set-settings>>
+
+[#get-settings]
+== Get FBR Settings
+
+Get the current values of the FBR settings.
+The FBR settings are part of the internal settings, so this API returns all internal settings.
+
+.List All Internal Settings
+----
+GET /internalSettings
+----
+
+=== curl Syntax
+
+[source,bash]
+----
+curl -u ${USER}:${PASSWORD} -X GET \
+        http[s]://${HOST}:${PORT}/internalSettings \
+        | jq 'with_entries(select(.key | startswith("dataService")))' 
+----
+
+.Path Parameters
+:priv-link: get-privs
+include::partial$user-pw-host-port-params.adoc[]
+
+
+[#get-privs]
+=== Required Privileges
+
+You must have at least 1 of the following roles to get the FBR settings:
+
+* xref:learn:security/roles.adoc#full-admin[Full Admin]
+* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin]
+* xref:learn:security/roles.adoc#ro-security-admin[Read-Only Security Admin]
+* xref:learn:security/roles.adoc#security-admin[Security Admin]
+* xref:learn:security/roles.adoc#local-user-security-admin[Local User Admin]
+
+=== Responses
+
+`200 OK`::
+Returns the internal settings. 
+See examples for an example of the settings.
+
+`401 Unauthorized`::
+Returned when authentication fails, such as when the password is incorrect.
+
+`403 Forbidden`::
+Returned if you do not have 1 of the roles listed in <<get-privs>>.
+
+[#gey-settings-example]
+=== Examples
+
+The following example gets the current values of the FBR settings.
+It filters the internal settings using the `jq` command to show only the FBR settings.
+
+[source,bash]
+----
+curl -s -u Administrator:password \
+     -X GET http://node1.example.com:8091/internalSettings \
+     | jq 'with_entries(select(.key | startswith("dataService")))'
+----
+
+Running the example returns a JSON object containing the current FBR settings, such as the following:
+
+[source,json]
+----
+{
+  "dataServiceFileBasedRebalanceEnabled": true,
+  "dataServiceFileBasedRebalanceMovesPerNode": 4
+}
+----
+
+The keys returned by the previous example are the FBR settings you can use to control FBR behavior:
+
+dataServiceFileBasedRebalanceEnabled::
+When `true`, eligible vBucket moves use FBR; when `false`, all vBucket moves use DCP.
+
+dataServiceFileBasedRebalanceMovesPerNode::
+The maximum number of concurrent file-based vBucket moves per node during rebalance.
+
+
+[#set-settings]
+== Configure FBR
+
+You can change the FBR settings using the REST API.
+
+.Change FBR Settings
+----
+POST /internalSettings
+----
+
+=== curl Syntax
+
+[source,bash]
+----
+curl -sS -u ${USER}:${PASSWORD} \
+  -X POST http[s]://${HOST}:${PORT}//internalSettings \
+  [-d dataServiceFileBasedRebalanceEnabled={true|false}] \
+  [-d dataServiceFileBasedRebalanceMovesPerNode=<integer>]
+----
+
+=== Path Parameters
+:priv-link: settings-privs
+include::partial$user-pw-host-port-params.adoc[]
+
+=== Parameters
+
+`dataServiceFileBasedRebalanceEnabled`::
+Set to `true` (the default) to allow the Data Service to use FBR when rebalancing data in eligible vBucket.
+Set to `false` to force the Data Service to use DCP when rebalancing data.
+
++
+NOTE: This setting can be overridden on a per-bucket basis using the bucket's  `dataServiceRebalanceType` setting.
+See xref:manage:manage-buckets/edit-bucket.adoc#bucket-rebalance-type[Bucket-Level Rebalance Type] for more information.
+
+`dataServiceFileBasedRebalanceMovesPerNode`::
+Integer value that sets the maximum number of concurrent file-based vBucket moves per node during rebalance.
+Valid range of values is from `1` to `1024`.
+The default value is `4`.
+Increase this value when you perform a rebalance when your cluster has low loads.
+For example, consider increasing this value during scheduled maintenance windows. 
+Decrease this value to reduce the overhead of rebalance on running workloads.
+
++
+NOTE: This setting is independent of `rebalanceMovesPerNode`, which applies to DCP rebalance.
+
+[#settings-privs]
+=== Required Privileges
+
+You must have 1 of the following roles to change the FBR settings:
+
+* xref:learn:security/roles.adoc#full-admin[Full Admin]
+* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin]
+
+=== Responses
+
+`200 OK`::
+Updating the value or values succeeded.
+
+`400 Bad Request`::
+Returned if you specify an invalid value for either setting or an out-of-range value for `dataServiceFileBasedRebalanceMovesPerNode`.
+
++ 
+Also returns a JSON object that describes the error, such as the following:
+
++
+[source,json]
+----
+{
+  "errors": [
+    "dataServiceFileBasedRebalanceMovesPerNode - The value must be between 1 and 1024."
+  ]
+}
+----
+
+'401 Unauthorized`::
+Returned when authentication fails, such as when the password is incorrect.
+
+`403 Forbidden`::
+Returned if you do not have the proper roles to call this API.
+See <<#settings-privs>>.
+
+
+[#settings-examples]
+=== Examples
+
+[#disable-fbr-example]
+.Disable FBR
+
+The following example prevents the Data Service from using FBR cluster-wide:
+
+[source,bash]
+----
+curl -s -u Administrator:password \
+     -X POST http://node1.example.com:8091/internalSettings \
+     -d dataServiceFileBasedRebalanceEnabled=false 
+----
+
+.Set the Maximum Concurrent FBR Moves per Node
+
+The following example sets the maximum number of concurrent file-based vBucket moves per node to `8`:
+
+
+
+[source,bash]
+----
+curl -s -u Administrator:password \
+     -X POST http://node1.example.com:8091/internalSettings \
+     -d dataServiceFileBasedRebalanceMovesPerNode=8 
+----
+
+## See Also
+
+* For conceptual information, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)].
+* For UI configuration, see xref:manage:manage-settings/general-settings.adoc#rebalance-settings[Rebalance Settings].
+* For per-bucket FBR configuration, see xref:manage:manage-buckets/edit-bucket.adoc#bucket-rebalance-type[Bucket-Level Rebalance Type].

modules/rest-api/pages/rest-bucket-create.adoc

@@ -94,9 +94,10 @@ curl -X POST -u <administrator>:<password>
   -d warmupBehavior=[ background | blocking | none ]
   -d memoryLowWatermark=<integer>
   -d memoryHighWatermark=<integer>
+  -d dataServiceRebalanceType=[ auto | preferFileBased | preferDcp ]
 ----
 
-All parameters are described in the following subsections.
+The following subsections describe these paremeters.
 
 NOTE: The `threadsNumber` parameter, which sets the number of threads for the bucket, has not had any effect since version Couchbase Server 7.0.0. 
 It's deprecated and is no longer listed in the syntax.
@@ -139,6 +140,7 @@ All other parameters are optional and have a default value.
 ** <<warmupbehavior,warmupBehavior>>
 ** <<memorylowwatermark,memoryLowWatermark>>
 ** <<memoryhighwatermark,memoryHighWatermark>>
+** <<dataservicerebalancetype,dataServiceRebalanceType>>
 +
 NOTE: When migrating a bucket between storage backends, you can edit only the bucket's xref:rest-api:rest-bucket-create.adoc#ramQuota[ramQuota], xref:rest-api:rest-bucket-create.adoc#evictionpolicy[evictionPolicy], and xref:rest-api:rest-bucket-create.adoc#storagebackend[storageBackend] parameters.
 For more information, see xref:manage:manage-buckets/migrate-bucket.adoc[].
@@ -600,6 +602,60 @@ curl -v -X POST http://localhost:8091/pools/default/buckets/testBucket /
 
 Success returns `200 OK`, and changes the `rank` of `testBucket` to `200`.
 
+[#dataservicerebalancetype]
+=== dataServiceRebalanceType
+[.edition]#{enterprise}#
+
+Controls the rebalance method used for this bucket's vBucket moves during Data Service rebalance.
+This setting overrides the cluster-level `dataServiceFileBasedRebalanceEnabled` setting for the bucket you set it on.
+
+The valid values are:
+
+* `auto` (default): The server automatically selects File-Based Rebalance (FBR) or DCP for each vBucket move based on which is estimated to be at least 10% faster.
+This is the recommended setting for most workloads.
+
+* `preferFileBased`: FBR is used for all eligible vBucket moves.
+DCP is used only when required, for example during storage engine migration or eviction policy changes.
+
+* `preferDcp`: All vBucket moves for this bucket use DCP, regardless of the cluster-level FBR setting.
+Couchbase Server never uses FBR for the bucket when you select this setting.
+
+You can set this parameter at bucket creation and modify it after bucket creation.
+
+See xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)] for more information about FBR and how the `dataServiceRebalanceType` bucket-level setting interacts with the cluster-level `dataServiceFileBasedRebalanceEnabled` setting.
+
+[#example-dataservicerebalancetype-create]
+==== Example: Specifying a Rebalance Type During Bucket Creation
+
+The following creates a bucket named `testBucket` with `preferFileBased` rebalance.
+
+[source,bash]
+----
+curl -v -X POST http://localhost:8091/pools/default/buckets \
+  -u Administrator:password \
+  -d name=testBucket \
+  -d ramQuota=125 \
+  -d dataServiceRebalanceType=preferFileBased
+----
+
+If the call is successful, `202 Accepted` is returned.
+
+[#example-dataservicerebalancetype-edit]
+==== Example: Updating the Rebalance Type
+
+The following updates the rebalance type for `testBucket` to `auto`:
+
+[source,bash]
+----
+curl -v -X POST http://localhost:8091/pools/default/buckets/testBucket \
+  -u Administrator:password \
+  -d dataServiceRebalanceType=auto
+----
+
+Success returns `200 OK`.
+
+For conceptual information on FBR and the bucket-level setting, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)] and xref:manage:manage-buckets/edit-bucket.adoc#bucket-rebalance-type[Bucket-Level Rebalance Type].
+
 [#replicanumber]
 === replicaNumber
 

modules/rest-api/pages/rest-rebalance-overview.adoc

@@ -1,5 +1,5 @@
 = Rebalance
-:description: pass:q[When one or more nodes have been brought into or taken out of a cluster, _rebalance_ redistributes data, indexes, event processing, and query processing among available nodes.]
+:description: pass:q[When one or more nodes have been brought into or taken out of a cluster, rebalance redistributes data, indexes, event processing, and query processing among available nodes.]
 :page-topic-type: reference
 
 [abstract]
@@ -8,7 +8,7 @@ Rebalance can be performed and configured by means of the REST API.
 
 == APIs in this Section
 
-_Rebalance_ must be performed whenever the number of nodes in a cluster have changed, and whenever buckets have been added or removed.
+Rebalance must be performed whenever the number of nodes in a cluster have changed, and whenever buckets have been added or removed.
 A complete overview is provided in xref:learn:clusters-and-availability/rebalance.adoc[Rebalance].
 
 The REST API for rebalance is as follows:

modules/rest-api/partials/rest-rebalance-table.adoc

@@ -42,4 +42,12 @@
 | `/internalSettings`
 | xref:rest-api:rest-cluster-disable-query.adoc[Disabling Consistent View Query Results on Rebalance]
 
+| `GET`
+| `/internalSettings`
+| xref:rest-api:file-based-data-rebalance.adoc#get-settings[Get File-Based Rebalance (FBR) Settings]
+
+| `POST`
+| `/internalSettings`
+| xref:rest-api:file-based-data-rebalance.adoc#set-settings[Configure FBR]
+
 |===