Skip to content

feat: Scope golden failure artifacts to environment directories - #179

Open
RyanHolanda wants to merge 1 commit into
Betterment:mainfrom
RyanHolanda:feat/env-aware-failure-artifacts
Open

feat: Scope golden failure artifacts to environment directories#179
RyanHolanda wants to merge 1 commit into
Betterment:mainfrom
RyanHolanda:feat/env-aware-failure-artifacts

Conversation

@RyanHolanda

Copy link
Copy Markdown

Description

When golden tests run with both platform and CI variants (via Alchemist's TestVariant mechanism), Flutter's default getFailureFile() uses only the filename portion of the golden URI to name failure diff images. Since both variants produce the same filename (e.g., my_widget.png), the macOS and CI failure artifacts overwrite each other in the flat failures/ directory.

This change scopes failure artifact output to environment-specific subdirectories (failures/{env}/), so each variant's diff images coexist without collision.

Before: macOS and CI failures both write to the same files:

failures/my_widget_masterImage.png   ← last one wins

After: Each variant writes to its own directory:

failures/macos/my_widget_masterImage.png  ← macOS variant
failures/ci/my_widget_masterImage.png     ← CI variant

Implementation

  • AlchemistFileComparator — accepts an optional environmentName parameter. Overrides getFailureFile() to write to failures/{env}/ when the name is provided.
  • FlutterGoldenTestRunner.run() — always wraps the comparator when environmentName is provided (previously only when diffThreshold > 0).
  • goldenTest() — passes variantConfig.environmentName (e.g., "macOS", "CI") through to the runner.
  • Backward compatible: when environmentName is null, behavior is unchanged.

Type of Change

  • ✨ New feature (non-breaking change which adds functionality)
  • 🛠️ Bug fix (non-breaking change which fixes an issue)
  • ❌ Breaking change (fix or feature that would cause existing functionality to change)
  • 🧹 Code refactor
  • ✅ Build configuration change
  • 📝 Documentation
  • 🗑️ Chore

When golden tests run with both platform and CI variants (via Alchemist's
TestVariant mechanism), both variants write failure diff images to the
same flat failures/ directory, overwriting each other.

This change:
- Adds environmentName parameter to AlchemistFileComparator
- Overrides getFailureFile() to write to failures/{env}/ instead of failures/
- Always wraps the comparator when environmentName is provided, not only
  when diffThreshold > 0
- Passes variantConfig.environmentName from goldenTest through to the
  runner

Now when both [macOS] and [CI] variants fail, the failure artifacts go to:
  failures/macos/foo_masterImage.png
  failures/ci/foo_masterImage.png

Closes: #N/A
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant