Add SCCACHE_ALLOW_INCREMENTAL to keep caching the non-incremental parts of a build

README.md

@@ -374,7 +374,7 @@ Known Caveats
 ### Rust
 
 * Crates that invoke the system linker cannot be cached. This includes `bin`, `dylib`, `cdylib`, and `proc-macro` crates. You may be able to improve compilation time of large `bin` crates by converting them to a `lib` crate with a thin `bin` wrapper.
-* Incrementally compiled crates cannot be cached. By default, in the debug profile Cargo will use incremental compilation for workspace members and path dependencies. [You can disable incremental compilation.](https://doc.rust-lang.org/cargo/reference/profiles.html#incremental)
+* Incrementally compiled crates cannot be cached. By default, in the debug profile Cargo will use incremental compilation for workspace members and path dependencies. [You can disable incremental compilation.](https://doc.rust-lang.org/cargo/reference/profiles.html#incremental) Because of this, sccache refuses to run when `CARGO_INCREMENTAL`/`CARGO_BUILD_INCREMENTAL` is enabled; `SCCACHE_ALLOW_INCREMENTAL=1` lifts that refusal and leaves only those crates uncached, still caching the build's registry dependencies.
 
 [More details on Rust caveats](/docs/Rust.md)
 

docs/Configuration.md

@@ -178,6 +178,7 @@ Note that some env variables may need sccache server restart to take effect.
 ### misc
 
 * `SCCACHE_ALLOW_CORE_DUMPS` to enable core dumps by the server
+* `SCCACHE_ALLOW_INCREMENTAL` set to `1` to let Rust builds with incremental compilation proceed instead of exiting with an error; only the invocations carrying `-C incremental=` go uncached. Any other value keeps the default refusal. See [Rust caveats](Rust.md)
 * `SCCACHE_CONF` configuration file path
 * `SCCACHE_BASEDIRS` base directory (or directories) to strip from paths for cache key computation. This is similar to ccache's `CCACHE_BASEDIR` and enables cache hits across different absolute paths when compiling the same source code. Multiple directories can be separated by `;` on Windows hosts and by `:` on any other operating system. When multiple directories are specified, the longest matching prefix is used. Path matching is **case-insensitive** on Windows and **case-sensitive** on other operating systems. Environment variable takes precedence over file configuration. Only absolute paths are supported; relative paths will cause an error and prevent the server from start.
 * `SCCACHE_CACHED_CONF`

docs/Rust.md

@@ -7,7 +7,7 @@ sccache includes support for caching Rust compilation. This includes many caveat
 * Compilation from stdin is not supported, a source file must be provided.
 * Values from `env!` require Rust >= 1.46 to be tracked in caching.
 * Procedural macros that read files from the filesystem may not be cached properly.
-* `rustc`'s incremental compilation needs to be disabled. See [The Cargo Book](https://doc.rust-lang.org/cargo/reference/profiles.html#incremental)
+* `rustc`'s incremental compilation needs to be disabled. See [The Cargo Book](https://doc.rust-lang.org/cargo/reference/profiles.html#incremental). sccache exits with an error when `CARGO_INCREMENTAL`/`CARGO_BUILD_INCREMENTAL` is set. `SCCACHE_ALLOW_INCREMENTAL=1` lifts that: cargo passes `-C incremental=` only to workspace members and path dependencies, so only those crates go uncached and the registry dependencies are cached as usual.
 * Crates that invoke the system linker cannot be cached. Examples are `bin`, `dylib`, `cdylib`, and `proc-macro` crates.
 
 If you are using Rust 1.18 or later, you can ask cargo to wrap all compilation with sccache by setting `RUSTC_WRAPPER=sccache` in your build environment.

src/commands.rs

@@ -891,19 +891,28 @@ pub fn run_command(cmd: Command) -> Result<i32> {
         } => {
             trace!("Command::Compile {{ {:?}, {:?}, {:?} }}", exe, cmdline, cwd);
 
-            let incr_env_strs = ["CARGO_BUILD_INCREMENTAL", "CARGO_INCREMENTAL"];
-            incr_env_strs
-                .iter()
-                .for_each(|incr_str| match env::var(incr_str) {
-                    Ok(incr_val) if incr_val == "1" => {
-                        println!(
-                            "sccache: incremental compilation is prohibited: Unset {} to continue.",
-                            incr_str
-                        );
-                        std::process::exit(1);
-                    }
-                    _ => (),
-                });
+            // Incremental crates cannot be cached, hence the refusal below.
+            // Lifting it leaves only the crates cargo hands `-C incremental=`
+            // uncached (compiler/rust.rs already treats those as CannotCache).
+            // Compared against "1" so a global opt-in can be overridden per build.
+            let allow_incremental = env::var("SCCACHE_ALLOW_INCREMENTAL").as_deref() == Ok("1");
+            if !allow_incremental {
+                let incr_env_strs = ["CARGO_BUILD_INCREMENTAL", "CARGO_INCREMENTAL"];
+                incr_env_strs
+                    .iter()
+                    .for_each(|incr_str| match env::var(incr_str) {
+                        Ok(incr_val) if incr_val == "1" => {
+                            println!(
+                                "sccache: incremental compilation is prohibited: Unset {} to \
+                                 continue, or set SCCACHE_ALLOW_INCREMENTAL=1 to keep caching \
+                                 the non-incremental parts of the build.",
+                                incr_str
+                            );
+                            std::process::exit(1);
+                        }
+                        _ => (),
+                    });
+            }
 
             let jobserver = Client::new();
             let conn = connect_or_start_server(&get_addr(), startup_timeout)?;

src/compiler/rust.rs

@@ -1579,10 +1579,14 @@ where
             // already uniquely identify the relevant registries.
             // CARGO_BUILD_JOBS only affects Cargo's parallelism, not rustc output.
             // CARGO_ENCODED_RUSTFLAGS is already cached in argument list
+            // CARGO_INCREMENTAL/CARGO_BUILD_INCREMENTAL only reach rustc as
+            // `-C incremental=`, already cached in argument list.
             if var == "CARGO_MAKEFLAGS"
                 || var.starts_with("CARGO_REGISTRIES_")
                 || var == "CARGO_BUILD_JOBS"
                 || var == "CARGO_ENCODED_RUSTFLAGS"
+                || var == "CARGO_INCREMENTAL"
+                || var == "CARGO_BUILD_INCREMENTAL"
             {
                 continue;
             }

src/dist/mod.rs

@@ -157,7 +157,7 @@ mod path_transform {
         pub fn disk_mappings(&self) -> impl Iterator<Item = (PathBuf, String)> {
             let mut normal_mappings = HashMap::new();
             let mut verbatim_mappings = HashMap::new();
-            for (_dist_path, local_path) in self.dist_to_local_path.iter() {
+            for local_path in self.dist_to_local_path.values() {
                 if !local_path.is_absolute() {
                     continue;
                 }

tests/sccache_cargo.rs

@@ -409,3 +409,86 @@ fn test_rust_cargo_cmd_readonly_preemtive_block() -> Result<()> {
         .try_success()?;
     Ok(())
 }
+
+/// Test that without `SCCACHE_ALLOW_INCREMENTAL`, enabling cargo's incremental
+/// compilation still makes sccache refuse to run.
+#[test]
+#[serial]
+fn test_rust_cargo_incremental_refused_by_default() -> Result<()> {
+    let test_info = SccacheTest::new(None)?;
+    // Clean before setting the env var below: the refusal fires on any rustc
+    // invocation, including the `rustc -vV` probe `cargo clean` makes.
+    cargo_clean(&test_info)?;
+
+    Command::new(CARGO.as_os_str())
+        .args(["build", "--color=never"])
+        .envs(test_info.env.iter().cloned())
+        .env("CARGO_INCREMENTAL", "1")
+        // Explicitly off, so the default is still tested on a machine that opted
+        // in globally (`.env_remove` cannot undo cargo's `[env]` table).
+        .env("SCCACHE_ALLOW_INCREMENTAL", "0")
+        .current_dir(CRATE_DIR.as_os_str())
+        .assert()
+        .try_stderr(predicates::str::contains("incremental compilation is prohibited").from_utf8())?
+        .try_failure()?;
+
+    Ok(())
+}
+
+/// Test that with `SCCACHE_ALLOW_INCREMENTAL` an incremental build succeeds and
+/// splits per invocation: the workspace crates cargo hands `-C incremental=`
+/// bypass the cache, while its registry dependencies keep hitting it.
+#[test]
+#[serial]
+fn test_rust_cargo_incremental_allowed_deps_cached() -> Result<()> {
+    let test_info = SccacheTest::new(None)?;
+    cargo_clean(&test_info)?;
+
+    // First build in the default, non-incremental mode: everything misses and
+    // populates the cache.
+    Command::new(CARGO.as_os_str())
+        .args(["build", "--color=never"])
+        .envs(test_info.env.iter().cloned())
+        .current_dir(CRATE_DIR.as_os_str())
+        .assert()
+        .try_success()?;
+
+    // Second build from a clean target, now incremental: the registry dep must
+    // still hit the entry the non-incremental build wrote.
+    cargo_clean(&test_info)?;
+    Command::new(CARGO.as_os_str())
+        .args(["build", "--color=never"])
+        .envs(test_info.env.iter().cloned())
+        .env("CARGO_INCREMENTAL", "1")
+        .env("SCCACHE_ALLOW_INCREMENTAL", "1")
+        .current_dir(CRATE_DIR.as_os_str())
+        .assert()
+        .try_success()?;
+
+    // The workspace lib really went through rustc's incremental machinery.
+    let incremental_dir = test_info.tempdir.path().join("cargo/debug/incremental");
+    let has_mylib_session = std::fs::read_dir(&incremental_dir)?
+        .filter_map(|e| e.ok())
+        .any(|e| e.file_name().to_string_lossy().starts_with("mylib"));
+    assert!(
+        has_mylib_session,
+        "expected an incremental session dir for `mylib` in {incremental_dir:?}"
+    );
+
+    test_info
+        .show_stats()?
+        // itoa: one miss (first build), one hit (second build, across modes).
+        .try_stdout(predicates::str::contains(r#""cache_hits":{"counts":{"Rust":1}"#).from_utf8())?
+        .try_stdout(
+            predicates::str::contains(
+                r#""cache_misses":{"counts":{"Rust":2},"adv_counts":{"rust":2}}"#,
+            )
+            .from_utf8(),
+        )?
+        // The workspace lib bypassed the cache. One, not two: `mybin` is rejected
+        // on `--crate-type bin` first, which cargo passes ahead of the flag.
+        .try_stdout(predicates::str::contains(r#""incremental":1"#).from_utf8())?
+        .try_success()?;
+
+    Ok(())
+}