Gazelle CI Ubuntu failures: missing files in repository rules

[jayconrod]

Gazelle has been experiencing a very high flake rate: 10 of the last 20 builds on master failed. Most of these are on Ubuntu, like this one.

(00:16:12) ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/011561ce453ad491f744b6cf5241f06f/external/gazelle+/internal/go_repository.bzl:336:13: An error occurred during the fetch of repository 'gazelle++_go_deps+++go_tools+com_github_google_uuid':
--
  | Traceback (most recent call last):
  | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/011561ce453ad491f744b6cf5241f06f/external/gazelle+/internal/go_repository.bzl", line 336, column 13, in _go_repository_impl
  | fail("%s: %s" % (ctx.name, result.stderr))
  | Error in fail: gazelle++_go_deps+++go_tools+com_github_google_uuid: fetch_repo: failed copying repo: lstat /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/011561ce453ad491f744b6cf5241f06f/external/gazelle++non_module_deps+bazel_gazelle_go_repository_cache/pkg/mod/github.com/google/uuid@v1.3.0/time.go: no such file or directory
  | (00:16:12) ERROR: no such package '@@gazelle++_go_deps+++go_tools+com_github_google_uuid//': gazelle++_go_deps+++go_tools+com_github_google_uuid: fetch_repo: failed copying repo: lstat /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/011561ce453ad491f744b6cf5241f06f/external/gazelle++non_module_deps+bazel_gazelle_go_repository_cache/pkg/mod/github.com/google/uuid@v1.3.0/time.go: no such file or directory
  |  

I think this is a CI infrastructure problem: the repository rule directory is being deleted while the repository rule is running. This error comes from a part of the code using filepath.Walk, which reads each directory and lstat's each file. We'd get no such file or directory if something were racing to delete the directory while fetch_repo was running.

I've only seen this happen on Ubuntu jobs. Retrying the job usually doesn't fixing the problem, but pushing an empty commit often does.

Are any other projects seeing errors like this? Is there anything happening in the infrastructure that might cause this?

[meteorcloudy]

Maybe this is a bug in Bazel? I'm not aware of any CI infra changes that could cause this.

[ted-xie]

1. I see a similar error signature on MacOS here and here. It's not just Ubuntu that exhibits this behavior.
2. I've launched 5 runs of the Gazelle build at v0.50.1 (181c8b6675c21ea85b5c1d6aad47aad9adf0b216) from April this year. 5/10 of your last scheduled builds have failed with similar errors, so 5 builds at an old known-good commit should be enough to determine isolate CI environment problems.

• Run 1 https://buildkite.com/bazel/bazel-gazelle/builds/6134
• Run 2 https://buildkite.com/bazel/bazel-gazelle/builds/6135
• Run 3 https://buildkite.com/bazel/bazel-gazelle/builds/6136
• Run 4 https://buildkite.com/bazel/bazel-gazelle/builds/6137
• Run 5 https://buildkite.com/bazel/bazel-gazelle/builds/6138

These have all passed, at least for Linux and Mac (the Windows build queue is currently woefully behind...). I'm not saying this absolves BazelCI, but I think it does lower the suspicion. Keep in mind I haven't really checked differences between the "old" build and your new (failing builds). Different Bazel versions? Different Linux executors? Subtle differences in the code? Different golang runtimes?