[DOC] warn against explain ALL in production

docs/configs.md

@@ -45,7 +45,7 @@ Name | Description | Default Value | Applicable at
 <a name="memory.pinnedPool.size"></a>spark.rapids.memory.pinnedPool.size|The size of the pinned memory pool in bytes unless otherwise specified. Use 0 to disable the pool.|0|Startup
 <a name="sql.batchSizeBytes"></a>spark.rapids.sql.batchSizeBytes|Set the target number of bytes for a GPU batch. Splits sizes for input data is covered by separate configs.|1073741824|Runtime
 <a name="sql.enabled"></a>spark.rapids.sql.enabled|Enable (true) or disable (false) sql operations on the GPU|true|Runtime
-<a name="sql.explain"></a>spark.rapids.sql.explain|Explain why some parts of a query were not placed on a GPU or not. Possible values are ALL: print everything, NONE: print nothing, NOT_ON_GPU: print only parts of a query that did not go on the GPU|NOT_ON_GPU|Runtime
+<a name="sql.explain"></a>spark.rapids.sql.explain|Explain why some parts of a query were not placed on a GPU or not. Possible values are ALL: print everything, NONE: print nothing, NOT_ON_GPU: print only parts of a query that did not go on the GPU. ALL is intended only for debugging and can generate a large amount of driver log output for complex or high-volume workloads, potentially degrading driver performance or making it unresponsive. Do not use ALL in production; use NOT_ON_GPU (the default) or NONE instead.|NOT_ON_GPU|Runtime
 <a name="sql.metrics.level"></a>spark.rapids.sql.metrics.level|GPU plans can produce a lot more metrics than CPU plans do. In very large queries this can sometimes result in going over the max result size limit for the driver. Supported values include DEBUG which will enable all metrics supported and typically only needs to be enabled when debugging the plugin. MODERATE which should output enough metrics to understand how long each part of the query is taking and how much data is going to each part of the query. ESSENTIAL which disables most metrics except those Apache Spark CPU plans will also report or their equivalents.|MODERATE|Runtime
 <a name="sql.multiThreadedRead.numThreads"></a>spark.rapids.sql.multiThreadedRead.numThreads|The maximum number of threads on each executor to use for reading small files in parallel. This can not be changed at runtime after the executor has started. Used with COALESCING and MULTITHREADED readers, see spark.rapids.sql.format.parquet.reader.type, spark.rapids.sql.format.orc.reader.type, or spark.rapids.sql.format.avro.reader.type for a discussion of reader types. If it is not set explicitly and spark.executor.cores is set, it will be tried to assign value of `max(MULTITHREAD_READ_NUM_THREADS_DEFAULT, spark.executor.cores)`, where MULTITHREAD_READ_NUM_THREADS_DEFAULT = 20.|20|Startup
 <a name="sql.reader.batchSizeBytes"></a>spark.rapids.sql.reader.batchSizeBytes|Soft limit on the maximum number of bytes the reader reads per batch. The readers will read chunks of data until this limit is met or exceeded. Note that the reader may estimate the number of bytes that will be used on the GPU in some cases based on the schema and number of rows in each batch.|2147483647|Runtime

docs/supported_ops.md

@@ -65,9 +65,12 @@ the level of compatibility with Apache Spark.  Those are covered [here](configs.
 Others are a part of Apache Spark itself and those are a bit harder to document.
 The work of updating this to cover that support is still ongoing.
 
-In general though if you ever have any question about why an operation is not running
-on the GPU you may set `spark.rapids.sql.explain` to ALL and it will try to give all of
-the reasons why this particular operator or expression is on the CPU or GPU.
+For debugging, set `spark.rapids.sql.explain` to ALL to report why
+operators or expressions run on the CPU or GPU. ALL can generate
+substantial driver log output for complex or high-volume workloads,
+potentially degrading driver performance or making the driver
+unresponsive. Do not enable ALL in production; use NOT_ON_GPU
+(the default) or NONE.
 
 # Key
 ## Types

sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala

@@ -2576,7 +2576,10 @@ val SHUFFLE_COMPRESSION_LZ4_CHUNK_SIZE = conf("spark.rapids.shuffle.compression.
   val EXPLAIN = conf("spark.rapids.sql.explain")
     .doc("Explain why some parts of a query were not placed on a GPU or not. Possible " +
       "values are ALL: print everything, NONE: print nothing, NOT_ON_GPU: print only parts of " +
-      "a query that did not go on the GPU")
+      "a query that did not go on the GPU. ALL is intended only for debugging and can generate " +
+      "a large amount of driver log output for complex or high-volume workloads, potentially " +
+      "degrading driver performance or making it unresponsive. Do not use ALL in production; " +
+      "use NOT_ON_GPU (the default) or NONE instead.")
     .commonlyUsed()
     .stringConf
     .createWithDefault("NOT_ON_GPU")

sql-plugin/src/main/scala/com/nvidia/spark/rapids/TypeChecks.scala

@@ -1898,9 +1898,12 @@ object SupportedOpsDocs {
     ConsoleOutput.writeLine("Others are a part of Apache Spark itself and those are a bit harder to document.")
     ConsoleOutput.writeLine("The work of updating this to cover that support is still ongoing.")
     ConsoleOutput.writeLine()
-    ConsoleOutput.writeLine("In general though if you ever have any question about why an operation is not running")
-    ConsoleOutput.writeLine("on the GPU you may set `spark.rapids.sql.explain` to ALL and it will try to give all of")
-    ConsoleOutput.writeLine("the reasons why this particular operator or expression is on the CPU or GPU.")
+    ConsoleOutput.writeLine("For debugging, set `spark.rapids.sql.explain` to ALL to report why")
+    ConsoleOutput.writeLine("operators or expressions run on the CPU or GPU. ALL can generate")
+    ConsoleOutput.writeLine("substantial driver log output for complex or high-volume workloads,")
+    ConsoleOutput.writeLine("potentially degrading driver performance or making the driver")
+    ConsoleOutput.writeLine("unresponsive. Do not enable ALL in production; use NOT_ON_GPU")
+    ConsoleOutput.writeLine("(the default) or NONE.")
     ConsoleOutput.writeLine()
     ConsoleOutput.writeLine("# Key")
     ConsoleOutput.writeLine("## Types")