Spike: Importer deduplication rework

[SiteRelEnby]

What problem does this solve?

Problem: Member name is not a good deduplication surface (and really, kind of an oversight on our part as a lot of the related importer behaviour is outside any intentional design parameters we had at the time we build the conflict handling)

• Members may have the same name
  • Workaround is clunky:
    • Initial import pass, make note of skipped members due to name conflict
    • Rename imported members with conflicting names to a new name (e.g. Alice_1) and set display name to desired name (Alice)
    • Rerun import, selecting the correct conflict member of the pair (or set)
    • Repeat if there's 3+ of a conflicting name
• Sheaf supports members with the same name if created directly, keying off name from the importer is generally lazy/naive

(todo: Check/verify current behaviour for same-name imports in an import file is coherent)

Proposed solution

Concept:

• At import time: Skip duplicates / overwrite duplicates / resolve duplicates manually
• Resolve manually presents an interface for each member conflict
  • Member selection happens before the importer on the server parses the file, this would need new endpoints and the UI to check for conflicts. Probably a conflict status, an endpoint for conflicts, list of all (browseable UI makes sense more than just sequentially presenting), details of specific conflict
  • How to handle multiway conflicts? set of pairs of each or all-at-once? 2 conflict sources, existing members, and conflicts in same import
  • DB storage for data for conflicts pre-resolution? Probably a set lifespan to resolve conflicts in - pg vs redis? image handling?
• Track source of a member from an import? Useful in the case of reimports, especially where there exist multiple with same name.
  • If the import contains a stable ID, we should add this as metadata (already there with openplural imports, but we could track ID for other platform-specific imports too
    • by doing this we identify duplicates that are the same import
      • Considerations about a member being in more than one different import, (e.g. [A,B,C] where import 1 has [A,B], import 2 has [B,C] - can we detect B is the same? then present a merge UI if the data for B is different.
      • specific considerations for merging data between the two?
    • when we hit a duplicate while importing with no obvious source of truth for uniqueness, we present the two profiles to the user: select which one to keep, or to keep both (manual merge? a lot more UI)

Alternatives considered

No response

Additional context

No response

[SiteRelEnby]

• "overwrite duplicates" should not actually clobber the entry, but instead behind the scenes replace (create new imported and leave the overwritten member as a pending-delete entry)? short timeout (1 day if system safety does not have a delay configured, equal to the standard member-delete delay if configured to >1d) and direct the user to make sure everything looks ok.
  • Check if system safety code will support a pending delete even if system safety is off? should afaik, but worth checking and also documenting as explicitly supported. Other potential cases it's useful too.
• add importer provenance data to member's profile scratchpad field as a user-visible identifier of import? (datetime, (file id? file sha256?)
• in terms of whether we get a unique ID from imports, that's a case to be tested per-app, but most should support something usable as long as we have a good way to track what's from where and retrieve it when needed for dedupe