Fix/rsync publishable - #2
Merged
Merged
Conversation
Signed-off-by: Polina Simonenko <rabarbrablad@gmail.com>
Signed-off-by: Polina Simonenko <rabarbrablad@gmail.com>
Signed-off-by: Polina Simonenko <rabarbrablad@gmail.com>
Signed-off-by: Polina Simonenko <rabarbrablad@gmail.com>
Signed-off-by: Polina Simonenko <rabarbrablad@gmail.com>
There was a problem hiding this comment.
Pull request overview
This PR broadens host/format plumbing to make publishing and verification more robust across drivers (rsync/S3/Pages/local), while tightening validation and improving developer ergonomics (linting + faster -short test loop) and operational clarity (status reporting, better “unknown repo” errors).
Changes:
- Refactors host repository configuration into per-driver structs and adds a single
Repository.Driver()shape check used at the composition root. - Splits “tree-bytes verification” into
internal/buildgraph.VerifyTreeand moves publishing/materialization intointernal/release, updating host adapters and tests accordingly. - Centralizes endpoint validation (
internal/endpoint) and S3 error classification/client setup (internal/awss3), adds staticcheck linting, and introduces-shortskips for slow integration-style tests.
Reviewed changes
Copilot reviewed 111 out of 113 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| staticcheck.conf | Disables ST1005 to avoid degrading proper-noun/acronym-leading error strings. |
| PLAN.md | Updates planning notes to reflect implemented import and broader S3 support. |
| Makefile | Adds lint (staticcheck) and test-short targets; includes lint in check. |
| internal/wire/hosts.go | Validates repository driver shape before resolving; improves unsupported-host error message. |
| internal/testutil/wheel.go | Updates zip header timestamp setting to the newer field. |
| internal/status/render.go | Compares blob bindings as sets via state.BlobDigests. |
| internal/state/store.go | Uses formats.For + unified endpoint validation; improves unsupported host error details. |
| internal/state/model.go | Bumps plan schema and adds ReportsManifestDigest to plan repositories. |
| internal/state/lockshard.go | Removes unused bytesEqual helper. |
| internal/state/lockshard_test.go | Adds skipInShortMode and skips scale-heavy sharding tests under -short. |
| internal/state/ledger.go | Makes blob bindings a de-duplicated sorted set (BlobDigests + compactSorted). |
| internal/state/git.go | Adds “workspace git state” reader; improves uncommitted-change error guidance; adds revision-unchanged assertion. |
| internal/state/git_layout_cache.go | Removes unused forgetGitLayout helper. |
| internal/state/cas.go | Avoids memoizing format facts when identity isn’t derived from bytes; adds helper. |
| internal/release/materialize.go | Renames package, uses buildgraph-only verification for tree matching; factors manifest reads via buildgraph. |
| internal/release/materialize_hardening_test.go | Updates package name and fixtures to match new release/buildgraph split. |
| internal/release/commit.go | Renames package to release. |
| internal/release/commit_other.go | Renames package to release (non-linux/darwin). |
| internal/release/commit_linux.go | Renames package to release (linux). |
| internal/release/commit_exchange_test.go | Renames package to release for exchange tests. |
| internal/release/commit_darwin.go | Renames package to release (darwin). |
| internal/knowledge/signing_test.go | Makes signing digest stability test clearer and non-tautological. |
| internal/endpoint/endpoint.go | New unified endpoint validation + HTTPS requirement with loopback exception. |
| internal/endpoint/endpoint_test.go | Tests endpoint shape rules and HTTPS/loopback behavior. |
| internal/buildgraph/verify.go | New format-neutral manifest+tree verifier and manifest reader. |
| internal/awss3/awss3.go | New shared AWS S3 client builder + error classification. |
| internal/awss3/awss3_test.go | Tests shared S3 error classification and client option application. |
| internal/app/snapshot_test.go | Updates snapshot fixture to use buildgraph/release materialization. |
| internal/app/pypi_endpoint_test.go | Skips pip-based endpoint tests under -short; uses release.Materialize. |
| internal/app/platform_image_test.go | Skips networked image-index resolution under -short. |
| internal/app/deb_endpoint.go | Uses unified loopback detection from internal/endpoint. |
| internal/app/deb_endpoint_test.go | Uses release.Materialize; skips container verification under -short. |
| host/host.go | Restructures host.Repository driver config and adds Driver() invariant check; extends host capabilities. |
| host/driver_test.go | Adds coverage for Repository.Driver() shape enforcement. |
| go.mod | Updates dependency versions (notably AWS SDK and x/net). |
| gate/gate.go | Removes unused JSON decode helper. |
| formats/helm/inspect.go | Tightens tar entry handling for Helm chart inspection. |
| formats/format.go | Extends Format interface (install steps, structural verification, endpoint probe); adds ForID; improves unsupported-format errors. |
| formats/deb/inspect.go | Tightens tar entry handling for Debian data archive inspection. |
| formats/conformance_test.go | Adds conformance tests for install steps, endpoint probes, and format identity resolution. |
| engine/two_runners_test.go | Ensures tests provide explicit host resolver (no implicit local fallback). |
| engine/status.go | Status no longer requires clean git; reports uncommitted paths; uses revision-unchanged check. |
| engine/status_uncommitted_test.go | New tests for status-with-uncommitted behavior and plan still requiring committed state. |
| engine/staging_test.go | Updates tests to pass explicit host resolver. |
| engine/site.go | Adds JSON tags to site index result fields for stable machine output. |
| engine/shared_blob_test.go | New regression tests for identical-bytes artifacts and binding stability. |
| engine/rollback.go | Requires a host resolver; improves unknown-repo errors. |
| engine/rollback_test.go | Minor cleanup to updated request construction. |
| engine/raw_workspace_test.go | Updates tests to pass explicit host resolver. |
| engine/prune.go | Improves unknown-repo error reporting. |
| engine/prune_test.go | Updates tests to pass explicit host resolver. |
| engine/progress_test.go | Updates tests to pass explicit host resolver. |
| engine/plan_bench_test.go | Updates benchmark to pass explicit host resolver. |
| engine/placements.go | Improves unknown-repo error reporting. |
| engine/placements_test.go | Updates tests to pass explicit host resolver. |
| engine/origin_restore_test.go | Updates tests to pass explicit host resolver. |
| engine/notconfigured.go | New helper for unknown repository errors with suggestions. |
| engine/localhosts_test.go | Adds local-only resolver for tests; asserts resolvers are required at entry points. |
| engine/keys.go | Improves unknown-repo error reporting for key attachment. |
| engine/gitattributes_test.go | Updates tests to pass explicit host resolver. |
| engine/forge_identity_test.go | Updates tests to pass explicit host resolver. |
| engine/engine.go | Uses release.Materialize in build commands. |
| engine/engine_test.go | Refactors python/container availability helpers; skips slow checks under -short. |
| engine/endpoint_support_test.go | Ensures endpoint verification tests pass resolver into apply request. |
| engine/collect.go | Requires a host resolver; improves unknown-repo errors. |
| engine/collect_test.go | Updates tests to pass explicit host resolver. |
| engine/clientverify.go | New registry-based client verification dispatch by ecosystem. |
| engine/clientverify_test.go | Adds registry consistency tests and semantic-invalid-manifest test. |
| engine/ci.go | Replaces TODO placeholders with clearer “FILL IN” prompts in templates. |
| engine/ci_gitlab.go | Replaces TODO placeholders with clearer “FILL IN” prompts in templates. |
| engine/applydecision.go | Extracts and documents apply decision logic for testability/readability. |
| engine/apply_concurrent_test.go | Updates concurrency tests to pass explicit host resolver. |
| engine/adoptsession.go | Improves unknown-repo error reporting in adopt session loading. |
| engine/adopt_test.go | Updates tests to pass explicit host resolver. |
| docs/signing.md | New documentation on signing keys, rotation, and review/approval gates. |
| docs/operating.md | New operational documentation (status, CI, exit codes, scaling guidance). |
| docs/hosts.md | New documentation of host semantics and requirements (rsync/S3/Pages). |
| docs/deploying.md | New documentation for shipping/running snailmail itself. |
| docs/curating.md | New documentation for promotions/yanks/prune/collect/rollback behavior. |
| docs/adopting.md | New documentation for import/adopt/doctor and provenance semantics. |
| Dockerfile | Updates pinned build image digest and clarifies repinning rationale. |
| cmd/snailmail/unknowncommand_test.go | Tests command suggestion list stays in sync with dispatcher. |
| cmd/snailmail/setupflags.go | Adds grouped/scoped setup help and rejects inapplicable flags with better messages. |
| cmd/snailmail/setupflags_test.go | Pins flag-group scopes and validates help/rejection behavior. |
| cmd/snailmail/main_test.go | Updates status human output expectations to the new observation-scope phrasing. |
| cmd/snailmail/exitcode_test.go | Pins exit-code mapping behavior and ensures notes exist where needed. |
| adapters/host/s3/two_runners_test.go | Updates fixtures to use new host.Repository S3 config struct. |
| adapters/host/s3/s3.go | Uses buildgraph verification; adopts endpoint validator; reads S3 config from nested struct; reports manifest digest capability. |
| adapters/host/s3/helm_test.go | Updates S3 host repo fixtures to nested S3 config. |
| adapters/host/s3/collect_test.go | Updates S3 host repo fixtures to nested S3 config. |
| adapters/host/s3/aws.go | Uses shared awss3 client builder and shared error classification. |
| adapters/host/s3/aws_test.go | Adds positive translation tests for shared classification into host adapter sentinels. |
| adapters/host/rsync/rsync.go | Uses buildgraph verification; reads rsync config from nested struct. |
| adapters/host/rsync/rsync_test.go | Updates fixtures to release.Materialize and nested rsync config; adds capability pin test. |
| adapters/host/rsync/collect_test.go | Updates path usage to nested rsync config. |
| adapters/host/local/local.go | Uses buildgraph verification and release.PublishVerifiedDirectory; reads local config from nested struct. |
| adapters/host/githubpages/githubpages.go | Uses buildgraph verification; reads pages config from nested struct; reports manifest digest capability. |
| adapters/host/githubpages/githubpages_test.go | Updates fixtures to release.Materialize and nested pages config. |
| adapters/blob/s3/aws.go | Uses unified endpoint validation, shared awss3 client builder, and shared error classification. |
| adapters/blob/s3/aws_test.go | Adds tests for bare-status normalization and endpoint HTTPS requirements. |
| .gitignore | Fixes over-broad ignore of cmd/snailmail path; adds .snailmail/ runtime state ignore. |
| .github/workflows/ci.yml | Runs make lint in CI. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| switch header.Typeflag { | ||
| case tar.TypeDir: | ||
| case tar.TypeReg, tar.TypeRegA: | ||
| case tar.TypeReg: |
Comment on lines
+213
to
+214
| case tar.TypeReg, tar.TypeDir, tar.TypeSymlink, tar.TypeLink: | ||
| if header.Typeflag == tar.TypeReg { |
Comment on lines
+133
to
+151
| func validateManifestFile(file ManifestFile, index int, files []ManifestFile) error { | ||
| if file.Path == "" || path.IsAbs(file.Path) || path.Clean(file.Path) != file.Path || file.Path == "." || strings.HasPrefix(file.Path, "../") || strings.ContainsRune(file.Path, '\\') { | ||
| return fmt.Errorf("repository manifest contains unsafe path %q", file.Path) | ||
| } | ||
| if file.Path == ManifestFilename { | ||
| return fmt.Errorf("repository manifest includes itself") | ||
| } | ||
| if index > 0 && files[index-1].Path >= file.Path { | ||
| return errors.New("repository manifest file list is not uniquely sorted") | ||
| } | ||
| if file.Size < 0 { | ||
| return fmt.Errorf("repository file %q has a negative size", file.Path) | ||
| } | ||
| decoded, err := hex.DecodeString(file.SHA256) | ||
| if err != nil || len(decoded) != sha256.Size { | ||
| return fmt.Errorf("repository file %q has an invalid SHA-256", file.Path) | ||
| } | ||
| return nil | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.