diff --git a/README.md b/README.md index 1e5c99df9..9e0e0f57b 100644 --- a/README.md +++ b/README.md @@ -327,7 +327,7 @@ This is most useful when using sccache for Rust compilation, as rustc supports u --- -Normalizing Paths with `SCCACHE_BASEDIRS` +Normalizing paths with `SCCACHE_BASEDIRS` ----------------------------------------- By default, sccache requires absolute paths to match for cache hits. To enable cache sharing across different build directories, you can set `SCCACHE_BASEDIRS` to strip a base directory from paths before hashing: @@ -344,13 +344,26 @@ export SCCACHE_BASEDIRS="/home/user/project:/home/user/workspace" Path matching is **case-insensitive** on Windows and **case-sensitive** on other operating systems. +For Rust compilations, sccache normalizes matching absolute source arguments, +the source side of `--remap-path-prefix`, Cargo path variables, tracked +environment dependency values that are absolute paths, and the current working +directory before computing the cache key. + This is similar to ccache's `CCACHE_BASEDIR` and helps when: * Building the same project from different directories * Sharing cache between CI jobs with different checkout paths * Multiple developers working with different username paths * Working with multiple project checkouts simultaneously -**Note:** Only absolute paths are supported. Relative paths will prevent server from starting. +**Note:** Only absolute paths are supported. A relative request value fails that +compiler invocation. A relative config-file value prevents the server from +starting. + +**Rust note:** This setting normalizes cache-key inputs; it does not rewrite +paths embedded in compiled artifacts. If a crate deliberately embeds an +absolute path, for example with `env!("CARGO_MANIFEST_DIR")`, a cache hit from +another checkout can contain the path from the compilation that populated the +cache. Use this opt-in setting only when that behavior is acceptable. You can also configure this in the sccache config file: @@ -362,6 +375,17 @@ basedirs = ["/home/user/project"] basedirs = ["/home/user/project", "/home/user/workspace"] ``` +The environment variable applies to each compiler invocation. Its value +overrides the config-file value for that request, so a persistent sccache +daemon can serve builds from different checkout roots without a restart. Set +`SCCACHE_BASEDIRS=""` on an invocation to disable the config-file fallback for +that request. + +Only the config file defines the daemon's fallback. A `SCCACHE_BASEDIRS` value +in the environment that starts the daemon does not become global daemon state. +Statistics report the config-file fallback and do not accumulate paths from +individual requests. + --- Known Caveats diff --git a/docs/Configuration.md b/docs/Configuration.md index fe21e309e..eba773195 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -179,7 +179,7 @@ Note that some env variables may need sccache server restart to take effect. * `SCCACHE_ALLOW_CORE_DUMPS` to enable core dumps by the server * `SCCACHE_CONF` configuration file path -* `SCCACHE_BASEDIRS` base directory (or directories) to strip from paths for cache key computation. This is similar to ccache's `CCACHE_BASEDIR` and enables cache hits across different absolute paths when compiling the same source code. Multiple directories can be separated by `;` on Windows hosts and by `:` on any other operating system. When multiple directories are specified, the longest matching prefix is used. Path matching is **case-insensitive** on Windows and **case-sensitive** on other operating systems. Environment variable takes precedence over file configuration. Only absolute paths are supported; relative paths will cause an error and prevent the server from start. +* `SCCACHE_BASEDIRS` base directory (or directories) to strip from paths for cache key computation. This is similar to ccache's `CCACHE_BASEDIR` and enables cache hits across different absolute paths when compiling the same source code. Multiple directories can be separated by `;` on Windows hosts and by `:` on any other operating system. When multiple directories are specified, the longest matching prefix is used. Path matching is **case-insensitive** on Windows and **case-sensitive** on other operating systems. For Rust, sccache normalizes matching absolute source arguments, the source side of `--remap-path-prefix`, Cargo path variables, tracked environment dependency values that are absolute paths, and the current working directory. The environment variable applies to each compiler invocation and overrides the config-file fallback for that request. An explicitly empty value disables the fallback for that request. Only the config file defines the daemon fallback; the daemon ignores `SCCACHE_BASEDIRS` in its startup environment. Statistics report only the configured fallback. Only absolute paths are supported; a relative path causes an error for the affected invocation or during config-file loading. This setting changes cache keys but does not rewrite paths embedded in artifacts; a Rust artifact can retain an absolute path from the compilation that populated the cache. * `SCCACHE_CACHED_CONF` * `SCCACHE_IDLE_TIMEOUT` how long the local daemon process waits for more client requests before exiting, in seconds. Set to `0` to run sccache permanently * `SCCACHE_STARTUP_NOTIFY` specify a path to a socket which will be used for server completion notification diff --git a/docs/Rust.md b/docs/Rust.md index 5d6f98c3c..a43b3997c 100644 --- a/docs/Rust.md +++ b/docs/Rust.md @@ -9,5 +9,6 @@ sccache includes support for caching Rust compilation. This includes many caveat * Procedural macros that read files from the filesystem may not be cached properly. * `rustc`'s incremental compilation needs to be disabled. See [The Cargo Book](https://doc.rust-lang.org/cargo/reference/profiles.html#incremental) * Crates that invoke the system linker cannot be cached. Examples are `bin`, `dylib`, `cdylib`, and `proc-macro` crates. +* `SCCACHE_BASEDIRS` normalizes matching paths in cache-key inputs, but it does not rewrite paths embedded in artifacts. For example, a crate that uses `env!("CARGO_MANIFEST_DIR")` can retain the path from the compilation that populated a shared cache entry. If you are using Rust 1.18 or later, you can ask cargo to wrap all compilation with sccache by setting `RUSTC_WRAPPER=sccache` in your build environment. diff --git a/src/cache/basedirs.rs b/src/cache/basedirs.rs new file mode 100644 index 000000000..2e0c1047e --- /dev/null +++ b/src/cache/basedirs.rs @@ -0,0 +1,176 @@ +// Copyright 2026 Mozilla Foundation +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::sync::Arc; +use std::time::Duration; + +use async_trait::async_trait; +use bytes::Bytes; + +use crate::cache::multilevel::MultiLevelStats; +use crate::cache::{Cache, CacheMode, CacheWrite, GetPathResult, Storage}; +use crate::compiler::PreprocessorCacheEntry; +use crate::config::PreprocessorCacheModeConfig; +use crate::errors::*; + +struct BasedirsStorage { + inner: Arc, + basedirs: Vec>, +} + +pub(crate) fn with_basedirs(storage: Arc, basedirs: Vec>) -> Arc { + if storage.basedirs() == basedirs { + storage + } else { + Arc::new(BasedirsStorage { + inner: storage, + basedirs, + }) + } +} + +#[async_trait] +impl Storage for BasedirsStorage { + async fn get(&self, key: &str) -> Result { + self.inner.get(key).await + } + + async fn put(&self, key: &str, entry: CacheWrite) -> Result { + self.inner.put(key, entry).await + } + + async fn get_raw(&self, key: &str) -> Result> { + self.inner.get_raw(key).await + } + + async fn put_raw(&self, key: &str, data: Bytes) -> Result { + self.inner.put_raw(key, data).await + } + + async fn check(&self) -> Result { + self.inner.check().await + } + + fn location(&self) -> String { + self.inner.location() + } + + fn cache_type_name(&self) -> &'static str { + self.inner.cache_type_name() + } + + async fn current_size(&self) -> Result> { + self.inner.current_size().await + } + + async fn max_size(&self) -> Result> { + self.inner.max_size().await + } + + fn multilevel_stats(&self) -> Option { + self.inner.multilevel_stats() + } + + fn preprocessor_cache_mode_config(&self) -> PreprocessorCacheModeConfig { + self.inner.preprocessor_cache_mode_config() + } + + fn basedirs(&self) -> &[Vec] { + &self.basedirs + } + + async fn get_path(&self, key: &str) -> GetPathResult { + self.inner.get_path(key).await + } + + async fn get_preprocessor_cache_entry( + &self, + key: &str, + ) -> Result>> { + self.inner.get_preprocessor_cache_entry(key).await + } + + async fn put_preprocessor_cache_entry( + &self, + key: &str, + entry: PreprocessorCacheEntry, + ) -> Result<()> { + self.inner.put_preprocessor_cache_entry(key, entry).await + } +} + +#[cfg(test)] +mod test { + use super::*; + use crate::cache::disk::DiskCache; + use crate::cache::readonly::ReadOnlyStorage; + + #[tokio::test] + async fn request_view_preserves_storage_capabilities() -> Result<()> { + let tempdir = tempfile::tempdir()?; + + let fallback = vec![b"/configured/".to_vec()]; + let storage: Arc = Arc::new(DiskCache::new( + tempdir.path(), + 1024 * 1024, + &tokio::runtime::Handle::current(), + PreprocessorCacheModeConfig::default(), + CacheMode::ReadWrite, + fallback.clone(), + )); + + let unchanged = with_basedirs(Arc::clone(&storage), fallback.clone()); + assert!(Arc::ptr_eq(&unchanged, &storage)); + + let request_basedirs = vec![b"/request/".to_vec()]; + let view = with_basedirs(Arc::clone(&storage), request_basedirs.clone()); + assert!(!Arc::ptr_eq(&view, &storage)); + assert_eq!(view.basedirs(), request_basedirs); + assert_eq!(storage.basedirs(), fallback); + assert_eq!(view.location(), storage.location()); + assert_eq!(view.cache_type_name(), storage.cache_type_name()); + + let key = "0123456789abcdef"; + let raw: Bytes = CacheWrite::default().finish()?.into(); + + view.put_raw(key, raw.clone()).await?; + assert_eq!(view.get_raw(key).await?.as_deref(), Some(raw.as_ref())); + assert!(matches!(view.get_path(key).await, GetPathResult::Found(_))); + + let entry_key = "fedcba9876543210"; + view.put(entry_key, CacheWrite::default()).await?; + assert!(matches!(view.get(entry_key).await?, Cache::Hit(_))); + + let preprocessor_key = "preprocessor"; + view.put_preprocessor_cache_entry(preprocessor_key, PreprocessorCacheEntry::default()) + .await?; + assert!( + view.get_preprocessor_cache_entry(preprocessor_key) + .await? + .is_some() + ); + assert_eq!( + view.preprocessor_cache_mode_config(), + storage.preprocessor_cache_mode_config() + ); + assert_eq!(view.current_size().await?, storage.current_size().await?); + assert_eq!(view.max_size().await?, storage.max_size().await?); + + let read_only: Arc = Arc::new(ReadOnlyStorage(Arc::clone(&storage))); + let read_only_view = with_basedirs(read_only, vec![b"/read-only-request/".to_vec()]); + assert_eq!(read_only_view.check().await?, CacheMode::ReadOnly); + + Ok(()) + } +} diff --git a/src/cache/mod.rs b/src/cache/mod.rs index 9959d2173..40bfb6244 100644 --- a/src/cache/mod.rs +++ b/src/cache/mod.rs @@ -14,6 +14,7 @@ #[cfg(feature = "azure")] pub mod azure; +pub(crate) mod basedirs; #[allow(clippy::module_inception)] pub mod cache; pub mod cache_io; @@ -52,6 +53,7 @@ pub mod webdav; ))] pub(crate) mod http_client; +pub(crate) use crate::cache::basedirs::with_basedirs; pub use crate::cache::cache::*; pub use crate::cache::cache_io::*; pub use crate::cache::ipc_storage::IpcStorage; diff --git a/src/commands.rs b/src/commands.rs index 5ea8b949c..ac3128ffe 100644 --- a/src/commands.rs +++ b/src/commands.rs @@ -715,7 +715,15 @@ where pub fn run_command(cmd: Command) -> Result { // Config isn't required for all commands, but if it's broken then we should flag // it early and loudly. - let config = &Config::load()?; + let config = if matches!( + &cmd, + Command::StartServer | Command::InternalStartServer | Command::ShowStats(_, _) + ) { + Config::load_with_file_basedirs()? + } else { + Config::load()? + }; + let config = &config; let startup_timeout = config.server_startup_timeout; match cmd { diff --git a/src/compiler/nvcc.rs b/src/compiler/nvcc.rs index ccfb5408a..62ffb1cbc 100644 --- a/src/compiler/nvcc.rs +++ b/src/compiler/nvcc.rs @@ -1314,16 +1314,19 @@ where }, )), CompilerArguments::Ok(hasher) => { + let env_vars = env_vars + .iter() + .chain([("SCCACHE_DIRECT".into(), "false".into())].iter()) + .cloned() + .collect::>(); + let storage = srvc.storage_for_request(&env_vars)?; srvc.start_compile_task( compiler, hasher, args, cwd.to_owned(), - env_vars - .iter() - .chain([("SCCACHE_DIRECT".into(), "false".into())].iter()) - .cloned() - .collect::>(), + env_vars, + storage, ) .await } diff --git a/src/compiler/rust.rs b/src/compiler/rust.rs index 779e9dd79..1acb0583b 100644 --- a/src/compiler/rust.rs +++ b/src/compiler/rust.rs @@ -26,7 +26,10 @@ use crate::dist::pkg; #[cfg(feature = "dist-client")] use crate::lru_disk_cache::{LruCache, Meter}; use crate::mock_command::{CommandCreatorSync, RunCommand}; -use crate::util::{Digest, fmt_duration_as_secs, hash_all, hash_all_archives, run_input_output}; +use crate::util::{ + Digest, fmt_duration_as_secs, hash_all, hash_all_archives, run_input_output, + strip_path_basedirs, +}; use crate::util::{HashToDigest, OsStrExt}; use crate::{counted_array, dist}; use async_trait::async_trait; @@ -44,7 +47,7 @@ use std::collections::{HashMap, HashSet}; use std::env::consts::DLL_EXTENSION; #[cfg(feature = "dist-client")] use std::env::consts::{DLL_PREFIX, EXE_EXTENSION}; -use std::ffi::OsString; +use std::ffi::{OsStr, OsString}; use std::fmt; use std::future::Future; use std::hash::Hash; @@ -235,7 +238,90 @@ static ALLOWED_EMIT: LazyLock> = LazyLock::new(|| ["link", "metadata", "dep-info"].iter().copied().collect()); /// Version number for cache key. -const CACHE_VERSION: &[u8] = b"6"; +const CACHE_VERSION: &[u8] = b"7"; + +fn hash_rust_value(digest: &mut Digest, value: &OsStr, basedirs: Option<&[Vec]>) { + let encoded = encode_rust_value(value); + let normalized = basedirs.map_or(encoded, |basedirs| strip_path_basedirs(encoded, basedirs)); + digest.update(&(normalized.len() as u64).to_le_bytes()); + digest.update(normalized); +} + +fn hash_rust_argument( + digest: &mut Digest, + arg: &OsStr, + value: Option<&OsStr>, + basedirs: Option<&[Vec]>, +) { + let arg_normalizer = if value.is_none() && Path::new(arg).is_absolute() { + basedirs + } else { + None + }; + hash_rust_value(digest, arg, arg_normalizer); + + let Some(value) = value else { + return; + }; + if arg == "--remap-path-prefix" { + hash_rust_remap(digest, value, basedirs); + return; + } + hash_rust_value(digest, value, None); +} + +fn hash_rust_remap(digest: &mut Digest, value: &OsStr, basedirs: Option<&[Vec]>) { + let Some(basedirs) = basedirs else { + hash_rust_value(digest, value, None); + return; + }; + let encoded = encode_rust_value(value); + let Some(separator) = encoded.iter().rposition(|byte| *byte == b'=') else { + hash_rust_value(digest, value, None); + return; + }; + let source = strip_path_basedirs(&encoded[..separator], basedirs); + let replacement = &encoded[separator..]; + digest.update(&((source.len() + replacement.len()) as u64).to_le_bytes()); + digest.update(source); + digest.update(replacement); +} + +fn encode_rust_value(value: &OsStr) -> &[u8] { + value.as_encoded_bytes() +} + +fn sort_paths_for_hash(paths: &mut [PathBuf], basedirs: Option<&[Vec]>) { + let Some(basedirs) = basedirs else { + return; + }; + if !paths.iter().any(|path| { + let path = encode_rust_value(path.as_os_str()); + strip_path_basedirs(path, basedirs).len() != path.len() + }) { + return; + } + + paths.sort_by(|left, right| { + let left = strip_path_basedirs(encode_rust_value(left.as_os_str()), basedirs); + let right = strip_path_basedirs(encode_rust_value(right.as_os_str()), basedirs); + left.cmp(right) + }); +} + +fn is_path_cargo_env(var: &OsStr) -> bool { + matches!( + var.to_str(), + Some( + "CARGO_HOME" + | "CARGO_INSTALL_ROOT" + | "CARGO_MANIFEST_DIR" + | "CARGO_MANIFEST_PATH" + | "CARGO_TARGET_DIR" + | "CARGO_TARGET_TMPDIR" + ) + ) || var.as_encoded_bytes().starts_with(b"CARGO_BIN_EXE_") +} /// Get absolute paths for all source files and env-deps listed in rustc's dep-info output. async fn get_source_files_and_env_deps( @@ -1388,10 +1474,12 @@ where _may_dist: bool, pool: &tokio::runtime::Handle, _rewrite_includes_only: bool, - _storage: Arc, + storage: Arc, _cache_control: CacheControl, ) -> Result> { trace!("[{}]: generate_hash_key", self.parsed_args.crate_name); + let basedirs = storage.basedirs(); + let basedirs = (!basedirs.is_empty()).then_some(basedirs); // TODO: this doesn't produce correct arguments if they should be concatenated - should use iter_os_strings let os_string_arguments: Vec<(OsString, Option)> = self .parsed_args @@ -1422,7 +1510,7 @@ where // Find all the source files and hash them let source_hashes_pool = pool.clone(); let source_files_and_hashes_and_env_deps = async { - let (source_files, env_deps) = get_source_files_and_env_deps( + let (mut source_files, env_deps) = get_source_files_and_env_deps( creator, &self.parsed_args.crate_name, &self.executable, @@ -1432,6 +1520,7 @@ where pool, ) .await?; + sort_paths_for_hash(&mut source_files, basedirs); let source_hashes = hash_all(&source_files, &source_hashes_pool).await?; Ok((source_files, source_hashes, env_deps)) }; @@ -1442,12 +1531,13 @@ where self.parsed_args.crate_name, self.parsed_args.externs.len() ); - let abs_externs = self + let mut abs_externs = self .parsed_args .externs .iter() .map(|e| cwd.join(e)) .collect::>(); + sort_paths_for_hash(&mut abs_externs, basedirs); let extern_hashes = hash_all(&abs_externs, pool); // Hash the contents of the staticlibs listed on the commandline. trace!( @@ -1455,12 +1545,13 @@ where self.parsed_args.crate_name, self.parsed_args.staticlibs.len() ); - let abs_staticlibs = self + let mut abs_staticlibs = self .parsed_args .staticlibs .iter() .map(|s| cwd.join(s)) .collect::>(); + sort_paths_for_hash(&mut abs_staticlibs, basedirs); let staticlib_hashes = hash_all_archives(&abs_staticlibs, pool); // Hash the content of the specified target json file, if any. @@ -1501,64 +1592,63 @@ where } let weak_toolchain_key = m.clone().finish(); // 3. The full commandline (self.arguments) - // TODO: there will be full paths here, it would be nice to - // normalize them so we can get cross-machine cache hits. // A few argument types are not passed in a deterministic order // by cargo: --extern, -L, --cfg. We'll filter those out, sort them, // and append them to the rest of the arguments. - let args = { - let (mut sortables, rest): (Vec<_>, Vec<_>) = os_string_arguments - .iter() - // We exclude a few arguments from the hash: - // -L, --extern, --out-dir, --diagnostic-width - // These contain paths which aren't relevant to the output, and the compiler inputs - // in those paths (rlibs and static libs used in the compilation) are used as hash - // inputs below. - .filter(|&(arg, _)| { - !(arg == "--extern" - || arg == "-L" - || arg == "--check-cfg" - || arg == "--out-dir" - || arg == "--diagnostic-width") - }) - // We also exclude `--target` if it specifies a path to a .json file. The file content - // is used as hash input below. - // If `--target` specifies a string, it continues to be hashed as part of the arguments. - .filter(|&(arg, _)| self.parsed_args.target_json.is_none() || arg != "--target") - // A few argument types were not passed in a deterministic order - // by older versions of cargo: --extern, -L, --cfg. We'll filter the rest of those - // out, sort them, and append them to the rest of the arguments. - .partition(|&(arg, _)| arg == "--cfg"); - sortables.sort(); - rest.into_iter() - .chain(sortables) - .flat_map(|(arg, val)| iter::once(arg).chain(val.as_ref())) - .fold(OsString::new(), |mut a, b| { - a.push(b); - a - }) - }; - args.hash(&mut HashToDigest { digest: &mut m }); + let (mut sortable_args, remaining_args): (Vec<_>, Vec<_>) = os_string_arguments + .iter() + // We exclude a few arguments from the hash: + // -L, --extern, --out-dir, --diagnostic-width + // These contain paths which aren't relevant to the output, and the compiler inputs + // in those paths (rlibs and static libs used in the compilation) are used as hash + // inputs below. + .filter(|&(arg, _)| { + !(arg == "--extern" + || arg == "-L" + || arg == "--check-cfg" + || arg == "--out-dir" + || arg == "--diagnostic-width") + }) + // We also exclude `--target` if it specifies a path to a .json file. The file content + // is used as hash input below. + // If `--target` specifies a string, it continues to be hashed as part of the arguments. + .filter(|&(arg, _)| self.parsed_args.target_json.is_none() || arg != "--target") + // A few argument types were not passed in a deterministic order + // by older versions of cargo: --extern, -L, --cfg. We'll filter the rest of those + // out, sort them, and append them to the rest of the arguments. + .partition(|&(arg, _)| arg == "--cfg"); + sortable_args.sort(); + let argument_count = remaining_args + .iter() + .chain(&sortable_args) + .map(|(_, value)| 1 + usize::from(value.is_some())) + .sum::(); + m.delimiter(b"rust-arguments"); + m.update(&(argument_count as u64).to_le_bytes()); + for (arg, value) in remaining_args.into_iter().chain(sortable_args) { + hash_rust_argument(&mut m, arg, value.as_deref(), basedirs); + } // 4. The digest of all source files (this includes src file from cmdline). // 5. The digest of all files listed on the commandline (self.externs). // 6. The digest of all static libraries listed on the commandline (self.staticlibs). // 7. The digest of the content of the target json file specified via `--target` (if any). - for h in source_hashes + for hash in source_hashes .into_iter() .chain(extern_hashes) .chain(staticlib_hashes) .chain(target_json_hash) { - m.update(h.as_bytes()); + m.update(hash.as_bytes()); } // 8. Environment variables: Hash all environment variables listed in the rustc dep-info // output. Additionally also has all environment variables starting with `CARGO_`, // since those are not listed in dep-info but affect cacheability. env_deps.sort(); + m.delimiter(b"rust-env-deps"); + m.update(&(env_deps.len() as u64).to_le_bytes()); for (var, val) in env_deps.iter() { - var.hash(&mut HashToDigest { digest: &mut m }); - m.update(b"="); - val.hash(&mut HashToDigest { digest: &mut m }); + hash_rust_value(&mut m, var, None); + hash_rust_value(&mut m, val, basedirs); } let mut env_vars: Vec<_> = env_vars .iter() @@ -1568,31 +1658,34 @@ where .cloned() .collect(); env_vars.sort(); - for (var, val) in env_vars.iter() { - if !var.starts_with("CARGO_") { - continue; - } - - // CARGO_MAKEFLAGS will have jobserver info which is extremely non-cacheable. - // CARGO_REGISTRIES_*_TOKEN contains non-cacheable secrets. - // Registry override config doesn't need to be hashed, because deps' package IDs - // already uniquely identify the relevant registries. - // CARGO_BUILD_JOBS only affects Cargo's parallelism, not rustc output. - // CARGO_ENCODED_RUSTFLAGS is already cached in argument list - if var == "CARGO_MAKEFLAGS" - || var.starts_with("CARGO_REGISTRIES_") - || var == "CARGO_BUILD_JOBS" - || var == "CARGO_ENCODED_RUSTFLAGS" - { - continue; - } - - var.hash(&mut HashToDigest { digest: &mut m }); - m.update(b"="); - val.hash(&mut HashToDigest { digest: &mut m }); + let cargo_env_vars = env_vars.iter().filter(|(var, _)| { + var.starts_with("CARGO_") + // CARGO_MAKEFLAGS will have jobserver info which is extremely non-cacheable. + // CARGO_REGISTRIES_*_TOKEN contains non-cacheable secrets. + // Registry override config doesn't need to be hashed, because deps' package IDs + // already uniquely identify the relevant registries. + // CARGO_BUILD_JOBS only affects Cargo's parallelism, not rustc output. + // CARGO_ENCODED_RUSTFLAGS is already cached in argument list. + && var != "CARGO_MAKEFLAGS" + && !var.starts_with("CARGO_REGISTRIES_") + && var != "CARGO_BUILD_JOBS" + && var != "CARGO_ENCODED_RUSTFLAGS" + }); + let cargo_env_count = cargo_env_vars.clone().count(); + m.delimiter(b"rust-cargo-env"); + m.update(&(cargo_env_count as u64).to_le_bytes()); + for (var, val) in cargo_env_vars { + hash_rust_value(&mut m, var, None); + let normalizer = if is_path_cargo_env(var) { + basedirs + } else { + None + }; + hash_rust_value(&mut m, val, normalizer); } // 9. The cwd of the compile. This will wind up in the rlib. - cwd.hash(&mut HashToDigest { digest: &mut m }); + m.delimiter(b"rust-cwd"); + hash_rust_value(&mut m, cwd.as_os_str(), basedirs); // 10. The version of the compiler. self.version.hash(&mut HashToDigest { digest: &mut m }); @@ -2721,7 +2814,6 @@ mod test { use crate::test::utils::*; use fs::File; use itertools::Itertools; - use std::ffi::OsStr; use std::io::{self, Write}; use std::sync::{Arc, Mutex}; use test_case::test_case; @@ -3584,24 +3676,30 @@ proc_macro false // sysroot shlibs digests. m.update(FAKE_DIGEST.as_bytes()); // Arguments, with cfgs sorted at the end. - OsStr::new("ab--cfgabc--cfgxyz").hash(&mut HashToDigest { digest: &mut m }); - // bar.rs (source file, from dep-info) - m.update(empty_digest.as_bytes()); - // foo.rs (source file, from dep-info) - m.update(empty_digest.as_bytes()); - // bar.rlib (extern crate, from externs) - m.update(empty_digest.as_bytes()); - // libbaz.a (static library, from staticlibs), containing a single - // file, baz.o, consisting of 1024 bytes of zeroes. + let args = ["a", "b", "--cfg", "abc", "--cfg", "xyz"]; + m.delimiter(b"rust-arguments"); + m.update(&(args.len() as u64).to_le_bytes()); + for arg in args { + hash_rust_value(&mut m, OsStr::new(arg), None); + } + // bar.rs and foo.rs (source files), then bar.rlib (extern crate). + for _ in 0..3 { + m.update(empty_digest.as_bytes()); + } + // libbaz.a contains baz.o, consisting of 1024 bytes of zeroes. m.update(libbaz_a_digest.as_bytes()); - // Env vars - OsStr::new("CARGO_BLAH").hash(&mut HashToDigest { digest: &mut m }); - m.update(b"="); - OsStr::new("abc").hash(&mut HashToDigest { digest: &mut m }); - OsStr::new("CARGO_PKG_NAME").hash(&mut HashToDigest { digest: &mut m }); - m.update(b"="); - OsStr::new("foo").hash(&mut HashToDigest { digest: &mut m }); - f.tempdir.path().hash(&mut HashToDigest { digest: &mut m }); + // rustc environment dependencies. + m.delimiter(b"rust-env-deps"); + m.update(&0_u64.to_le_bytes()); + // Cargo environment variables. + m.delimiter(b"rust-cargo-env"); + m.update(&2_u64.to_le_bytes()); + hash_rust_value(&mut m, OsStr::new("CARGO_BLAH"), None); + hash_rust_value(&mut m, OsStr::new("abc"), None); + hash_rust_value(&mut m, OsStr::new("CARGO_PKG_NAME"), None); + hash_rust_value(&mut m, OsStr::new("foo"), None); + m.delimiter(b"rust-cwd"); + hash_rust_value(&mut m, f.tempdir.path().as_os_str(), None); TEST_RUSTC_VERSION.hash(&mut HashToDigest { digest: &mut m }); let digest = m.finish(); assert_eq!(res.key, digest); @@ -3675,6 +3773,175 @@ proc_macro false Ok(()) } + fn rust_argument_key(arg: &str, value: Option<&str>, basedirs: Option<&[Vec]>) -> String { + let mut digest = Digest::new(); + hash_rust_argument( + &mut digest, + OsStr::new(arg), + value.map(OsStr::new), + basedirs, + ); + digest.finish() + } + + #[test] + fn test_strip_rust_basedirs_paths() { + let basedirs = [b"/work/project/".to_vec(), b"/work/project/crate/".to_vec()]; + + for (value, expected) in [ + ( + b"/work/project/crate/src/lib.rs".as_slice(), + b"src/lib.rs".as_slice(), + ), + (b"/work/project", b""), + (b"/work/project/", b""), + ( + b"prefix=/work/project/src/lib.rs", + b"prefix=/work/project/src/lib.rs", + ), + ( + b"/work/project copy/src/lib.rs", + b"/work/project copy/src/lib.rs", + ), + (b"/work/project=copy", b"/work/project=copy"), + (b"/other/project/src/lib.rs", b"/other/project/src/lib.rs"), + ] { + assert_eq!(strip_path_basedirs(value, &basedirs), expected); + } + } + + #[cfg(unix)] + #[test] + fn test_strip_rust_basedirs_non_utf8() { + use std::os::unix::ffi::OsStrExt as _; + let key = |value: &[u8], basedir: &[u8]| { + let mut digest = Digest::new(); + hash_rust_value( + &mut digest, + OsStr::from_bytes(value), + Some(&[basedir.to_vec()]), + ); + digest.finish() + }; + + let first = key(b"/work/one/src/non-utf8-\xff.rs", b"/work/one/"); + let second = key(b"/work/two/src/non-utf8-\xff.rs", b"/work/two/"); + assert_eq!(first, second); + assert_ne!(first, key(b"/work/two/src/non-utf8-\xfe.rs", b"/work/two/")); + } + + #[cfg(unix)] + #[test] + fn test_sort_paths_for_hash_uses_normalized_paths() { + let first_basedirs = vec![b"/a/work/".to_vec()]; + let second_basedirs = vec![b"/z/work/".to_vec()]; + let mut first = vec![ + PathBuf::from("/a/work/src/lib.rs"), + PathBuf::from("/m/shared.rs"), + ]; + let mut second = vec![ + PathBuf::from("/m/shared.rs"), + PathBuf::from("/z/work/src/lib.rs"), + ]; + let keys = |paths: &[PathBuf], basedirs: &[Vec]| { + paths + .iter() + .map(|path| { + strip_path_basedirs(encode_rust_value(path.as_os_str()), basedirs).to_vec() + }) + .collect::>() + }; + + sort_paths_for_hash(&mut first, Some(&first_basedirs)); + sort_paths_for_hash(&mut second, Some(&second_basedirs)); + assert_eq!( + keys(&first, &first_basedirs), + keys(&second, &second_basedirs) + ); + } + + #[cfg(target_os = "windows")] + #[test] + fn test_strip_rust_basedirs_windows_paths() { + let basedirs = [b"c:/work/project/".to_vec()]; + assert_eq!( + strip_path_basedirs(b"C:\\Work\\Project\\src\\lib.rs", &basedirs), + b"src\\lib.rs".as_slice() + ); + assert_eq!( + strip_path_basedirs(b"C:\\Work\\Project", &basedirs), + b"".as_slice() + ); + assert_eq!( + strip_path_basedirs(b"C:\\Work\\Project\\\xc4\xb0", &basedirs), + b"C:\\Work\\Project\\\xc4\xb0".as_slice() + ); + + assert_ne!( + rust_argument_key("package 😀", None, None), + rust_argument_key("package 😁", None, None) + ); + assert_eq!( + rust_argument_key( + r"C:\Work\one\src\lib.rs", + None, + Some(&[b"c:/work/one/".to_vec()]) + ), + rust_argument_key( + r"D:\Work\two\src\lib.rs", + None, + Some(&[b"d:/work/two/".to_vec()]) + ) + ); + } + + #[test] + fn test_hash_rust_inputs_normalize_only_paths() { + let first = [b"/work/one/".to_vec()]; + let second = [b"/work/two/".to_vec()]; + let value_key = |value: &str, basedirs: &[Vec]| { + let mut digest = Digest::new(); + hash_rust_value(&mut digest, OsStr::new(value), Some(basedirs)); + digest.finish() + }; + + #[cfg(not(target_os = "windows"))] + { + assert_eq!( + rust_argument_key("/work/one/src/lib.rs", None, Some(&first)), + rust_argument_key("/work/two/src/lib.rs", None, Some(&second)) + ); + assert_ne!( + rust_argument_key("/work/one/src/lib.rs", None, Some(&first)), + rust_argument_key("/work/two/other/lib.rs", None, Some(&second)) + ); + } + assert_ne!( + rust_argument_key("ab", Some("c"), None), + rust_argument_key("a", Some("bc"), None) + ); + assert_eq!( + rust_argument_key("--remap-path-prefix", Some("/work/one=/src"), Some(&first)), + rust_argument_key("--remap-path-prefix", Some("/work/two=/src"), Some(&second)) + ); + assert_ne!( + rust_argument_key("--remap-path-prefix", Some("/work/one=/src"), None), + rust_argument_key("--remap-path-prefix", Some("/work/two=/src"), None) + ); + assert_eq!( + value_key("/work/one/Cargo.toml", &first), + value_key("/work/two/Cargo.toml", &second) + ); + assert_ne!( + rust_argument_key("--cfg", Some("root=/work/one"), Some(&first)), + rust_argument_key("--cfg", Some("root=/work/two"), Some(&second)) + ); + assert_eq!( + rust_argument_key("/other/src/lib.rs", None, None), + rust_argument_key("/other/src/lib.rs", None, Some(&first)) + ); + } + #[test_case(true ; "with preprocessor cache")] #[test_case(false ; "without preprocessor cache")] fn test_equal_hashes_externs(preprocessor_cache_mode: bool) { diff --git a/src/config.rs b/src/config.rs index 00f3e8c92..57bd64c8c 100644 --- a/src/config.rs +++ b/src/config.rs @@ -14,7 +14,7 @@ use crate::cache::CacheMode; #[cfg(target_os = "windows")] -use crate::util::normalize_win_path; +use crate::util::{normalize_win_path, strip_windows_verbatim_prefix}; use directories::ProjectDirs; use fs::File; use fs_err as fs; @@ -27,13 +27,17 @@ use serde::{ #[cfg(test)] use serial_test::serial; use std::env; +use std::ffi::OsStr; use std::io::{Read, Write}; use std::path::{Path, PathBuf}; use std::result::Result as StdResult; use std::str::FromStr; use std::sync::{LazyLock, Mutex}; -use std::{collections::HashMap, fmt}; -use typed_path::Utf8TypedPathBuf; +use std::{ + collections::{HashMap, HashSet}, + fmt, +}; +use typed_path::TypedPathBuf; use crate::errors::*; @@ -1324,9 +1328,19 @@ pub struct Config { impl Config { pub fn load() -> Result { - let env_conf = config_from_env()?; - let skip_cache_check = skip_cache_check_from_env()?; + Self::load_with_env_config(config_from_env()?) + } + pub(crate) fn load_with_file_basedirs() -> Result { + let mut env_conf = config_from_env()?; + // A daemon can outlive the shell that started it. Per-invocation basedirs + // arrive with compile requests, so only the file can define the fallback. + env_conf.basedirs = None; + Self::load_with_env_config(env_conf) + } + + fn load_with_env_config(env_conf: EnvConfig) -> Result { + let skip_cache_check = skip_cache_check_from_env()?; let file_conf_path = config_file("SCCACHE_CONF", "config"); let file_conf = try_read_config_file(&file_conf_path) .context("Failed to load config file")? @@ -1366,48 +1380,8 @@ impl Config { file_basedirs }; - // Validate that all basedirs are absolute paths - // basedirs_raw is Vec - let mut basedirs = Vec::with_capacity(basedirs_raw.len()); - for d in basedirs_raw { - let p = Utf8TypedPathBuf::from(d); - if !p.is_absolute() { - bail!("Basedir path must be absolute: {:?}", p); - } - // Normalize basedir: - // remove double separators, cur_dirs, parent_dirs, trailing slashes - let p_norm = p.normalize(); - let mut bytes = p_norm.to_string().into_bytes(); - - // Always add a trailing `/` to basedirs to ensure we only match complete path - // components - bytes.push(b'/'); - - // normalize windows paths: use slashes and lowercase - let normalized = { - #[cfg(target_os = "windows")] - { - normalize_win_path(&bytes) - } - - #[cfg(not(target_os = "windows"))] - { - bytes - } - }; - // push only if not already present - if !basedirs.contains(&normalized) { - basedirs.push(normalized); - } - } - - if !basedirs.is_empty() && log::log_enabled!(log::Level::Debug) { - let basedirs_str: Vec = basedirs - .iter() - .map(|b| String::from_utf8_lossy(b).into_owned()) - .collect(); - debug!("Using basedirs for path normalization: {:?}", basedirs_str); - } + let basedirs = normalize_basedirs(basedirs_raw.into_iter().map(String::into_bytes))?; + log_basedirs(&basedirs); let client_side_mode = env_client_side_mode.unwrap_or(file_client_side_mode) // Logging always writes to stderr in the client process, disregarding @@ -1432,6 +1406,64 @@ impl Config { } } +fn normalize_basedirs(basedirs_raw: impl IntoIterator>) -> Result>> { + let mut basedirs = Vec::new(); + let mut seen = HashSet::new(); + for directory in basedirs_raw { + let path = TypedPathBuf::from(directory); + if !path.is_absolute() { + bail!("Basedir path must be absolute: {:?}", path); + } + + // Normalize duplicate separators, current and parent components, and + // trailing separators before adding one stable component boundary. + let normalized = path.normalize().as_bytes().to_vec(); + #[cfg(target_os = "windows")] + let mut normalized = { + let mut normalized = normalize_win_path(&normalized); + strip_windows_verbatim_prefix(&mut normalized); + normalized + }; + #[cfg(not(target_os = "windows"))] + let mut normalized = normalized; + + if !normalized.ends_with(b"/") { + normalized.push(b'/'); + } + + if seen.insert(normalized.clone()) { + basedirs.push(normalized); + } + } + + Ok(basedirs) +} + +fn log_basedirs(basedirs: &[Vec]) { + if !basedirs.is_empty() && log::log_enabled!(log::Level::Debug) { + let basedirs_str: Vec = basedirs + .iter() + .map(|basedir| String::from_utf8_lossy(basedir).into_owned()) + .collect(); + debug!("Using basedirs for path normalization: {:?}", basedirs_str); + } +} + +pub(crate) fn parse_basedirs_env(value: &OsStr) -> Result>> { + #[cfg(target_os = "windows")] + let separator = b';'; + #[cfg(not(target_os = "windows"))] + let separator = b':'; + + normalize_basedirs( + value + .as_encoded_bytes() + .split(|byte| *byte == separator) + .filter(|path| !path.is_empty()) + .map(<[u8]>::to_vec), + ) +} + #[derive(Clone, Debug, Default, Serialize, Deserialize)] #[serde(default)] #[serde(deny_unknown_fields)] @@ -2026,6 +2058,38 @@ fn config_basedirs_overrides() { assert!(config.basedirs.is_empty()); } +#[test] +fn request_basedirs_parse_multiple_paths_and_reject_relative_paths() { + #[cfg(target_os = "windows")] + let (value, expected) = ( + OsStr::new(r"C:\first\root;D:/second/root"), + vec![b"c:/first/root/".to_vec(), b"d:/second/root/".to_vec()], + ); + #[cfg(not(target_os = "windows"))] + let (value, expected) = ( + OsStr::new("/first/root:/second/root"), + vec![b"/first/root/".to_vec(), b"/second/root/".to_vec()], + ); + + assert_eq!(parse_basedirs_env(value).unwrap(), expected); + assert!(parse_basedirs_env(OsStr::new("relative/root")).is_err()); +} + +#[test] +#[cfg(unix)] +fn request_basedirs_preserve_non_utf8_paths() { + use std::os::unix::ffi::OsStrExt as _; + + let raw = b"/tmp/non-\xff-utf8"; + let mut expected = raw.to_vec(); + expected.push(b'/'); + + assert_eq!( + parse_basedirs_env(OsStr::from_bytes(raw)).unwrap(), + vec![expected] + ); +} + #[test] #[cfg(not(target_os = "windows"))] fn test_deserialize_basedirs() { @@ -3122,14 +3186,25 @@ fn test_integration_normalized_path_with_double_slashes() { cache: Default::default(), dist: Default::default(), server_startup_timeout_ms: None, - basedirs: vec!["/home//user///project/".to_string()], + basedirs: vec![ + "/home//user///project/".to_string(), + "/".to_string(), + "/home/user/project\\".to_string(), + ], client_side_mode: false, }; let config = Config::from_env_and_file_configs(env_conf, file_conf).unwrap(); // Config should normalize to single slashes with one trailing slash - assert_eq!(config.basedirs, vec![b"/home/user/project/"]); + assert_eq!( + config.basedirs, + vec![ + b"/home/user/project/".to_vec(), + b"/".to_vec(), + b"/home/user/project\\/".to_vec(), + ] + ); // Verify it works with strip_basedirs let input = b"# 1 \"/home/user/project/src/main.c\""; @@ -3153,14 +3228,17 @@ fn test_integration_windows_path_normalization() { cache: Default::default(), dist: Default::default(), server_startup_timeout_ms: None, - basedirs: vec!["C:\\Users\\Test\\Project".to_string()], + basedirs: vec!["C:\\Users\\Test\\Project".to_string(), "C:\\".to_string()], client_side_mode: false, }; let config = Config::from_env_and_file_configs(env_conf, file_conf).unwrap(); // Should be normalized to lowercase with forward slashes - assert_eq!(config.basedirs, vec![b"c:/users/test/project/"]); + assert_eq!( + config.basedirs, + vec![b"c:/users/test/project/".to_vec(), b"c:/".to_vec()] + ); // Test with mixed case preprocessor output let input = b"# 1 \"C:\\Users\\Test\\Project\\src\\main.c\""; diff --git a/src/server.rs b/src/server.rs index 2f142684a..3c04c5dfc 100644 --- a/src/server.rs +++ b/src/server.rs @@ -13,15 +13,13 @@ // limitations under the License.SCCACHE_MAX_FRAME_LENGTH use crate::cache::readonly::ReadOnlyStorage; -use crate::cache::{CacheMode, Storage, storage_from_config}; +use crate::cache::{CacheMode, Storage, storage_from_config, with_basedirs}; use crate::compiler::PreprocessorCacheEntry; use crate::compiler::{ CacheControl, CompileResult, Compiler, CompilerArguments, CompilerHasher, CompilerKind, CompilerProxy, DistType, Language, MissType, get_compiler_info, }; -#[cfg(feature = "dist-client")] -use crate::config; -use crate::config::Config; +use crate::config::{self, Config}; use crate::dist; use crate::jobserver::Client; use crate::mock_command::{CommandCreatorSync, ProcessCommandCreator}; @@ -43,7 +41,7 @@ use serde::{Deserialize, Serialize}; use std::cell::Cell; use std::collections::{HashMap, HashSet}; use std::env; -use std::ffi::OsString; +use std::ffi::{OsStr, OsString}; use std::future::Future; use std::io::{self, Write}; use std::marker::Unpin; @@ -82,6 +80,18 @@ const DEFAULT_IDLE_TIMEOUT: u64 = 600; #[cfg(feature = "dist-client")] const DIST_CLIENT_RECREATE_TIMEOUT: Duration = Duration::from_secs(30); +fn is_basedirs_env(name: &OsStr) -> bool { + #[cfg(target_os = "windows")] + { + name.as_encoded_bytes() + .eq_ignore_ascii_case(b"SCCACHE_BASEDIRS") + } + #[cfg(not(target_os = "windows"))] + { + name == "SCCACHE_BASEDIRS" + } +} + /// Result of background server startup. #[derive(Debug, Serialize, Deserialize)] pub enum ServerStartup { @@ -1169,12 +1179,29 @@ where let cmd = compile.args; let cwd: PathBuf = compile.cwd.into(); let env_vars = compile.env_vars; + let storage = self.storage_for_request(&env_vars)?; let me = self.clone(); let info = self .compiler_info(exe.into(), cwd.clone(), &cmd, &env_vars) .await; - Ok(me.check_compiler(info, cmd, cwd, env_vars).await) + Ok(me.check_compiler(info, cmd, cwd, env_vars, storage).await) + } + + pub(crate) fn storage_for_request( + &self, + env_vars: &[(OsString, OsString)], + ) -> Result> { + let Some((_, value)) = env_vars + .iter() + .rev() + .find(|(name, _)| is_basedirs_env(name)) + else { + return Ok(self.storage.clone()); + }; + + let basedirs = config::parse_basedirs_env(value)?; + Ok(with_basedirs(self.storage.clone(), basedirs)) } /// Run a compile entirely in the current process (used in client-side mode). @@ -1363,6 +1390,7 @@ where cmd: Vec, cwd: PathBuf, env_vars: Vec<(OsString, OsString)>, + storage: Arc, ) -> SccacheResponse { match compiler { Err(e) => { @@ -1382,7 +1410,7 @@ where let body = self .clone() - .start_compile_task(c, hasher, cmd, cwd, env_vars) + .start_compile_task(c, hasher, cmd, cwd, env_vars, storage) .and_then(|res| async { Ok(Response::CompileFinished(res)) }) .boxed(); @@ -1426,6 +1454,7 @@ where arguments: Vec, cwd: PathBuf, env_vars: Vec<(OsString, OsString)>, + storage: Arc, ) -> Result { self.stats.lock().await.requests_executed += 1; @@ -1466,7 +1495,7 @@ where &me, client, me.creator.clone(), - me.storage.clone(), + storage, arguments, cwd, env_vars, diff --git a/src/util.rs b/src/util.rs index 408ae8a55..92d4dd0c7 100644 --- a/src/util.rs +++ b/src/util.rs @@ -1209,6 +1209,153 @@ pub fn strip_basedirs<'a>(preprocessor_output: &'a [u8], basedirs: &[Vec]) - Cow::Owned(result) } +#[cfg(any(target_os = "windows", test))] +enum WindowsVerbatimPrefix { + Drive, + Unc, +} + +#[cfg(any(target_os = "windows", test))] +fn windows_path_starts_with(value: &[u8], prefix: &[u8]) -> bool { + value.len() >= prefix.len() + && value.iter().zip(prefix).all(|(&actual, &expected)| { + let actual = match actual { + b'A'..=b'Z' => actual + (b'a' - b'A'), + b'\\' => b'/', + _ => actual, + }; + actual == expected + }) +} + +#[cfg(any(target_os = "windows", test))] +/// Match a normalized Windows prefix while treating separator runs as one. +/// Rust dep-info escapes backslashes, so a separator can occupy multiple bytes. +/// The returned offset still indexes the original value, avoiding an allocation. +fn windows_path_prefix_len(value: &[u8], prefix: &[u8]) -> Option { + let mut value_index = 0; + let mut prefix_index = 0; + + while prefix_index < prefix.len() { + if matches!(prefix[prefix_index], b'/' | b'\\') { + let prefix_start = prefix_index; + while prefix_index < prefix.len() && matches!(prefix[prefix_index], b'/' | b'\\') { + prefix_index += 1; + } + + let value_start = value_index; + while value_index < value.len() && matches!(value[value_index], b'/' | b'\\') { + value_index += 1; + } + if value_index == value_start + || (prefix_start == 0 + && prefix_index - prefix_start >= 2 + && value_index - value_start < 2) + { + return None; + } + } else { + let actual = *value.get(value_index)?; + let actual = match actual { + b'A'..=b'Z' => actual + (b'a' - b'A'), + _ => actual, + }; + if actual != prefix[prefix_index] { + return None; + } + value_index += 1; + prefix_index += 1; + } + } + + Some(value_index) +} + +#[cfg(any(target_os = "windows", test))] +fn windows_verbatim_prefix(value: &[u8]) -> Option { + if value.len() >= 7 + && windows_path_starts_with(value, b"//?/") + && value[4].is_ascii_alphabetic() + && value[5] == b':' + && matches!(value[6], b'/' | b'\\') + { + Some(WindowsVerbatimPrefix::Drive) + } else if windows_path_starts_with(value, b"//?/unc/") { + Some(WindowsVerbatimPrefix::Unc) + } else { + None + } +} + +/// Strips the longest configured base directory from a complete path. +/// Configured base directories must be normalized and end with a slash. +#[doc(hidden)] +pub fn strip_path_basedirs<'a>(value: &'a [u8], basedirs: &[Vec]) -> &'a [u8] { + if basedirs.is_empty() || value.is_empty() { + return value; + } + + #[cfg(target_os = "windows")] + { + strip_windows_path_basedirs(value, basedirs) + } + + #[cfg(not(target_os = "windows"))] + { + let mut longest = None; + for basedir in basedirs.iter().filter(|basedir| basedir.ends_with(b"/")) { + let root = &basedir[..basedir.len() - 1]; + let match_len = if value == root { + Some(root.len()) + } else if value.starts_with(basedir) { + Some(basedir.len()) + } else { + None + }; + if match_len > longest { + longest = match_len; + } + } + + longest.map_or(value, |length| &value[length..]) + } +} + +#[cfg(any(target_os = "windows", test))] +fn strip_windows_path_basedirs<'a>(value: &'a [u8], basedirs: &[Vec]) -> &'a [u8] { + // Unicode case folding can change byte lengths. Keep non-ASCII values + // unchanged so offsets into the original value remain valid. + if !value.is_ascii() { + return value; + } + + let (basedir_prefix, tail, removed_len): (&[u8], &[u8], usize) = + match windows_verbatim_prefix(value) { + Some(WindowsVerbatimPrefix::Drive) => (b"", &value[4..], 4), + Some(WindowsVerbatimPrefix::Unc) => (b"//", &value[8..], 8), + None => (b"", value, 0), + }; + + let mut longest = None; + for basedir in basedirs.iter().filter(|basedir| basedir.ends_with(b"/")) { + let configured_len = basedir.len(); + let root = basedir[..configured_len - 1].strip_prefix(basedir_prefix); + let prefix = basedir.strip_prefix(basedir_prefix); + let exact_match = root.and_then(|root| { + windows_path_prefix_len(tail, root).filter(|length| *length == tail.len()) + }); + let match_end = + exact_match.or_else(|| prefix.and_then(|prefix| windows_path_prefix_len(tail, prefix))); + if let Some(match_end) = match_end + && longest.is_none_or(|(length, _)| configured_len > length) + { + longest = Some((configured_len, match_end)); + } + } + + longest.map_or(value, |(_, match_end)| &value[removed_len + match_end..]) +} + /// Double every `/` in a normalized path. /// /// Paths inside preprocessor output are C string literals, so on Windows @@ -1292,6 +1439,23 @@ pub fn normalize_win_path(path: &[u8]) -> Vec { result } +/// Remove the Win32 verbatim prefix from normalized drive and UNC paths. +/// +/// `std::fs::canonicalize` adds this prefix on Windows, while compiler +/// arguments commonly use the equivalent path without it. +#[cfg(any(target_os = "windows", test))] +pub(crate) fn strip_windows_verbatim_prefix(path: &mut Vec) { + match windows_verbatim_prefix(path) { + Some(WindowsVerbatimPrefix::Drive) => { + path.drain(..4); + } + Some(WindowsVerbatimPrefix::Unc) => { + path.drain(2..8); + } + None => {} + } +} + /// Resolve the compiler executable, avoiding ccache/sccache wrappers. /// /// This function handles scenarios where ccache/sccache might interfere: @@ -1833,6 +1997,70 @@ mod tests { assert_eq!(normalized, b"c:/users/test/project"); } + #[test] + fn test_strip_windows_path_basedirs_verbatim_paths() { + let basedirs = vec![ + b"c:/users/test/".to_vec(), + b"c:/users/test/project/".to_vec(), + b"//server/share/project/".to_vec(), + ]; + + for (input, expected) in [ + ( + b"\\\\?\\C:\\Users\\Test\\Project\\src\\lib.rs".as_slice(), + b"src\\lib.rs".as_slice(), + ), + (b"//?/c:/users/test/project".as_slice(), b"".as_slice()), + ( + b"\\\\?\\UNC\\Server\\Share\\Project\\src\\lib.rs".as_slice(), + b"src\\lib.rs".as_slice(), + ), + ( + b"C:/Users/Test/Project/src/lib.rs".as_slice(), + b"src/lib.rs".as_slice(), + ), + ( + b"C:\\\\Users\\\\Test\\\\Project\\\\src\\\\lib.rs".as_slice(), + b"src\\\\lib.rs".as_slice(), + ), + ( + b"\\\\?\\C:\\\\Users\\\\Test\\\\Project".as_slice(), + b"".as_slice(), + ), + ( + b"\\\\?\\Volume{1234}\\Project\\src\\lib.rs".as_slice(), + b"\\\\?\\Volume{1234}\\Project\\src\\lib.rs".as_slice(), + ), + ] { + assert_eq!( + super::strip_windows_path_basedirs(input, &basedirs), + expected + ); + } + } + + #[test] + fn test_strip_windows_verbatim_prefix() { + for (input, expected) in [ + ( + b"\\\\?\\C:\\Users\\Test\\Project".as_slice(), + b"c:/users/test/project".as_slice(), + ), + ( + b"\\\\?\\UNC\\Server\\Share\\Project".as_slice(), + b"//server/share/project".as_slice(), + ), + ( + b"\\\\?\\Volume{1234}\\Project".as_slice(), + b"//?/volume{1234}/project".as_slice(), + ), + ] { + let mut normalized = super::normalize_win_path(input); + super::strip_windows_verbatim_prefix(&mut normalized); + assert_eq!(normalized, expected); + } + } + #[test] fn test_normalize_win_path_utf8() { // Test with UTF-8 characters (e.g., German umlauts) diff --git a/tests/sccache_cargo.rs b/tests/sccache_cargo.rs index 6734a33f5..076cf1aca 100644 --- a/tests/sccache_cargo.rs +++ b/tests/sccache_cargo.rs @@ -13,7 +13,8 @@ use fs_err as fs; use helpers::{SCCACHE_BIN, SccacheTest}; use predicates::prelude::*; use serial_test::serial; -use std::path::Path; +use std::ffi::{OsStr, OsString}; +use std::path::{Path, PathBuf}; use std::process::Command; #[macro_use] @@ -43,6 +44,280 @@ fn test_rust_cargo_build_readonly() -> Result<()> { test_rust_cargo_cmd_readonly("build", SccacheTest::new(None)?) } +#[test] +#[serial] +fn test_rust_cargo_build_across_basedirs() -> Result<()> { + let test_info = SccacheTest::new(None)?; + let first = test_info.tempdir.path().join("first"); + let second = test_info.tempdir.path().join("second"); + + for root in [&first, &second] { + write_basedirs_crate(root, "basedirs-test", 42)?; + fs::write( + root.join("src/main.rs"), + "fn main() { println!(\"{}\", basedirs_test::MANIFEST); }\n", + )?; + } + let first = fs::canonicalize(first)?; + let second = fs::canonicalize(second)?; + + stop_sccache()?; + let config_path = test_info.tempdir.path().join("config"); + write_basedirs_config(&config_path, &[first.clone(), second.clone()])?; + restart_sccache( + &test_info, + Some(vec![ + ("SCCACHE_CONF".into(), config_path.as_os_str().to_owned()), + // The public launcher and daemon must both ignore this stale value. + ("SCCACHE_BASEDIRS".into(), "relative/startup-value".into()), + ]), + )?; + + for root in [&first, &second] { + cargo_build(&test_info, root, None, false)?; + } + + let stdout = Command::new(CARGO.as_os_str()) + .args(["run", "--quiet", "--color=never"]) + .envs(test_info.env.iter().cloned()) + .env("CARGO_TARGET_DIR", second.join("target")) + .current_dir(&second) + .assert() + .try_success()? + .get_output() + .stdout + .clone(); + let stdout = + std::str::from_utf8(&stdout).context("cached Cargo binary output was not UTF-8")?; + let reported = PathBuf::from(stdout.trim_end_matches(&['\r', '\n'][..])); + let reported = fs::canonicalize(&reported) + .with_context(|| format!("failed to canonicalize reported path {reported:?}"))?; + assert_eq!(reported, first); + + write_basedirs_crate(&second, "basedirs-test", 43)?; + cargo_build(&test_info, &second, None, false)?; + + test_info + .show_stats()? + .try_stdout(predicates::str::contains(r#""cache_hits":{"counts":{"Rust":1}"#).from_utf8())? + .try_stdout( + predicates::str::contains(r#""cache_misses":{"counts":{"Rust":2}"#).from_utf8(), + )? + .try_success()?; + + Ok(()) +} + +#[test] +#[serial] +fn test_request_basedirs_reuse_one_daemon() -> Result<()> { + let test_info = SccacheTest::new(None)?; + let roots = ["first", "second", "empty", "override"] + .into_iter() + .map(|name| test_info.tempdir.path().join(name)) + .collect::>(); + + for root in &roots { + write_basedirs_crate(root, "request-basedirs-test", 42)?; + } + let roots = roots + .into_iter() + .map(fs::canonicalize) + .collect::>>()?; + let config_path = test_info.tempdir.path().join("config"); + write_basedirs_config(&config_path, &roots)?; + restart_sccache( + &test_info, + Some(vec![( + "SCCACHE_CONF".into(), + config_path.as_os_str().to_owned(), + )]), + )?; + + let unrelated = fs::canonicalize(CRATE_DIR.as_os_str())?; + for (client_side, answer) in [(false, 42), (true, 7)] { + if client_side { + // Force Cargo to invoke rustc again without restarting the daemon. + for root in &roots { + write_basedirs_crate(root, "request-basedirs-test", answer)?; + } + } + zero_sccache_stats()?; + + // Normal mode sends each root with its request. Client-side mode omits + // the override here and gets the file fallback through the daemon handshake. + let first_basedir = (!client_side).then_some(roots[0].as_os_str()); + let second_basedir = (!client_side).then_some(roots[1].as_os_str()); + cargo_build(&test_info, &roots[0], first_basedir, client_side)?; + cargo_build(&test_info, &roots[1], second_basedir, client_side)?; + + // Both roots are in the file fallback. An empty or nonmatching request + // override must replace that fallback, so these are misses rather than hits. + cargo_build(&test_info, &roots[2], Some(OsStr::new("")), client_side)?; + cargo_build( + &test_info, + &roots[3], + Some(unrelated.as_os_str()), + client_side, + )?; + + test_info + .show_stats()? + .try_stdout( + predicates::str::contains(r#""cache_hits":{"counts":{"Rust":1}"#).from_utf8(), + )? + .try_stdout( + predicates::str::contains(r#""cache_misses":{"counts":{"Rust":3}"#).from_utf8(), + )? + .try_stdout(predicate::function(|output: &[u8]| { + serde_json::from_slice::(output) + .ok() + .and_then(|stats| stats["basedirs"].as_array().map(Vec::len)) + == Some(4) + }))? + .try_success()?; + } + + Ok(()) +} + +#[test] +#[serial] +fn test_show_stats_without_daemon_uses_file_basedirs() -> Result<()> { + #[cfg(target_os = "windows")] + let (configured, expected) = ("C:/configured/fallback", "c:/configured/fallback/"); + #[cfg(not(target_os = "windows"))] + let (configured, expected) = ("/configured/fallback", "/configured/fallback/"); + + let tempdir = tempfile::Builder::new() + .prefix("sccache_test_show_stats_basedirs") + .tempdir()?; + let cache_dir = tempdir.path().join("cache"); + fs::create_dir(&cache_dir)?; + let config_path = tempdir.path().join("config"); + fs::write(&config_path, format!("basedirs = [{configured:?}]\n"))?; + + stop_sccache()?; + Command::new(SCCACHE_BIN.as_os_str()) + .args(["--show-stats", "--stats-format=json"]) + .env("SCCACHE_DIR", &cache_dir) + .env("SCCACHE_CONF", &config_path) + .env("SCCACHE_BASEDIRS", "relative/request-value") + .assert() + .try_stdout(predicates::str::contains(format!(r#""basedirs":["{expected}"]"#)).from_utf8())? + .try_success()?; + + Ok(()) +} + +#[test] +#[serial] +fn test_concurrent_request_basedirs_do_not_leak() -> Result<()> { + let test_info = SccacheTest::new(None)?; + let first_a = test_info.tempdir.path().join("first-a"); + let second_a = test_info.tempdir.path().join("second-a"); + let first_b = test_info.tempdir.path().join("first-b"); + let second_b = test_info.tempdir.path().join("second-b"); + + for root in [&first_a, &second_a] { + write_basedirs_crate(root, "concurrent-basedirs-a", 42)?; + } + for root in [&first_b, &second_b] { + write_basedirs_crate(root, "concurrent-basedirs-b", 7)?; + } + let first_a = fs::canonicalize(first_a)?; + let second_a = fs::canonicalize(second_a)?; + let first_b = fs::canonicalize(first_b)?; + let second_b = fs::canonicalize(second_b)?; + + cargo_build(&test_info, &first_a, Some(first_a.as_os_str()), false)?; + cargo_build(&test_info, &first_b, Some(first_b.as_os_str()), false)?; + zero_sccache_stats()?; + + let mut build_a = + cargo_build_command(&test_info, &second_a, Some(second_a.as_os_str()), false).spawn()?; + let mut build_b = + cargo_build_command(&test_info, &second_b, Some(second_b.as_os_str()), false).spawn()?; + assert!(build_a.wait()?.success()); + assert!(build_b.wait()?.success()); + + test_info + .show_stats()? + .try_stdout(predicates::str::contains(r#""cache_hits":{"counts":{"Rust":2}"#).from_utf8())? + .try_stdout(predicates::str::contains(r#""cache_misses":{"counts":{}"#).from_utf8())? + .try_success()?; + + Ok(()) +} + +fn write_basedirs_crate(root: &Path, name: &str, answer: u32) -> Result<()> { + fs::create_dir_all(root.join("src"))?; + fs::write( + root.join("Cargo.toml"), + format!("[package]\nname = {name:?}\nversion = \"0.1.0\"\nedition = \"2024\"\n"), + )?; + fs::write( + root.join("src/lib.rs"), + format!( + "pub const MANIFEST: &str = env!(\"CARGO_MANIFEST_DIR\");\npub fn answer() -> u32 {{ {answer} }}\n" + ), + )?; + Ok(()) +} + +fn write_basedirs_config(config_path: &Path, basedirs: &[PathBuf]) -> Result<()> { + let basedirs = basedirs + .iter() + .map(|path| format!("{:?}", path.to_string_lossy())) + .collect::>() + .join(", "); + fs::write(config_path, format!("basedirs = [{basedirs}]\n"))?; + Ok(()) +} + +fn zero_sccache_stats() -> Result<()> { + Command::new(SCCACHE_BIN.as_os_str()) + .arg("--zero-stats") + .assert() + .try_success()?; + Ok(()) +} + +fn cargo_build_command( + test_info: &SccacheTest, + root: &Path, + basedirs: Option<&OsStr>, + client_side: bool, +) -> Command { + let mut command = Command::new(CARGO.as_os_str()); + command + .args(["build", "--color=never"]) + .envs(test_info.env.iter().cloned()) + .env("CARGO_TARGET_DIR", root.join("target")) + .env_remove("SCCACHE_BASEDIRS") + .env_remove("SCCACHE_CLIENT_SIDE") + .current_dir(root); + if let Some(basedirs) = basedirs { + command.env("SCCACHE_BASEDIRS", basedirs); + } + if client_side { + command.env("SCCACHE_CLIENT_SIDE", "1"); + } + command +} + +fn cargo_build( + test_info: &SccacheTest, + root: &Path, + basedirs: Option<&OsStr>, + client_side: bool, +) -> Result<()> { + cargo_build_command(test_info, root, basedirs, client_side) + .assert() + .try_success()?; + Ok(()) +} + #[test] #[serial] #[cfg(unix)] @@ -190,7 +465,7 @@ fn test_rust_cargo_cmd(cmd: &str, test_info: SccacheTest) -> Result<()> { fn restart_sccache( test_info: &SccacheTest, - additional_envs: Option>, + additional_envs: Option>, ) -> Result<()> { let cache_dir = test_info.tempdir.path().join("cache");