RFC: Phase 0 - XBRL Mapping Coverage Expansion (7% → 95%)

[dgunning]

Overview

This discussion seeks feedback on Phase 0 of the Context-Aware XBRL Standardization initiative. Phase 0 is the critical foundation that must be completed before we can implement context-aware disambiguation.

The Problem

Analysis revealed a massive coverage gap in EdgarTools XBRL standardization:

Metric	EdgarTools	@mpreiss9 Production	Gap
GAAP tags mapped	157	2,343	7% coverage
Ambiguous tags handled	0	214	0%
"DropThisItem" exclusions	0	276	Missing

Before implementing context-aware disambiguation for the 214 ambiguous tags, we must first expand our base mapping coverage.

---

Proposed Approach: Hybrid Taxonomy

We recommend Option C (Hybrid) - use @mpreiss9's proven taxonomy internally, with user-friendly display names for backwards compatibility.

New Data Structures

1. gaap_mappings.json (2,343 mappings)

{
  "AccountsPayableCurrentAndNoncurrent": {
    "standard_tags": ["TradePayables", "OtherOperatingNonCurrentLiabilities"],
    "ambiguous": true,
    "comment": "Curr/NonCurr ambiguity"
  },
  "AccountsPayableCurrent": {
    "standard_tags": ["TradePayables"],
    "ambiguous": false
  }
}

2. exclusions.py (276 tags)

EXCLUDED_TAGS = {
    "AcceleratedShareRepurchasesFinalPricePaidPerShare",
    "AcceleratedShareRepurchasesInitialPricePaidPerShare",
    # ... 276 "DropThisItem" tags that confuse analysis
}

3. display_names.json (backwards compatibility)

{
  "TradePayables": "Accounts Payable",
  "TradeReceivables": "Accounts Receivable",
  "PlantPropertyEquipmentNet": "Property, Plant and Equipment",
  "CostOfGoodsAndServicesSold": "Cost of Revenue"
}

4. Reverse Index (O(1) lookup)

Built at runtime from gaap_mappings.json for fast company concept → standard concept resolution.

---

Deliverables

• gaap_mappings.json - 2,343 GAAP tag mappings
• exclusions.py - 276 "DropThisItem" tags
• display_names.json - User-friendly display names
• reverse_index.py - O(1) lookup structure
• scripts/import_mpreiss9_mappings.py - Migration script
• Updated MappingStore to use new structures

---

Questions for Discussion

For @mpreiss9:

1. Taxonomy Naming: Is the hybrid approach (your taxonomy internally + display names) acceptable? Or would you recommend full adoption of your naming conventions?

2. Multi-mapping Format: Your CSV uses colon separator (e.g., TagA:TagB) for ambiguous tags. Should we preserve this format or convert to a list structure in JSON?

3. DropThisItem Tags: Are there any tags in the 276 exclusions that might actually be useful in certain contexts? Should exclusions be configurable?

4. Priority Order: When a tag maps to multiple standard concepts (like AccountsPayableCurrentAndNoncurrent → TradePayables:OtherOperatingNonCurrentLiabilities), does the order indicate priority/preference?

5. Deprecated Tags: Are any of the 2,343 GAAP mappings for deprecated/obsolete tags we should flag?

For the Community:

6. Display Names: Are there any standard concept names in EdgarTools today that you'd prefer we keep vs. adopting mpreiss9's naming?

7. Breaking Changes: Would a minor breaking change in output names be acceptable for v6.0.0 if it means better standardization?

---

Timeline

Phase	Goal	Effort
0 (this)	Coverage expansion (7% → 95%)	2-3 weeks
1	Balance sheet validation	1-2 weeks
2	Section membership dictionary	1 week
3-4	Context-aware disambiguation	5-7 weeks
5-6	Logging + granularity profiles	3-5 weeks

---

Related

• GitHub Issue: Create documentation on how to customize standardization tagging #494
• Full Epic: docs-internal/planning/future-enhancements/context-aware-standardization.md
• mpreiss9 CSV Analysis: 6,177 mappings from 390 companies

cc: @mpreiss9 - Your input on this approach would be invaluable given your production experience with 390+ companies. Thank you for sharing your mapping data!

[mpreiss9]

**Replies to your questions **

1. I would NOT argue for adopting my names for everyone. They work for me but I make no claim for other use cases. What I DO argue for is that whatever solution is built allows for people to use their own mapping and labels as needed. In fact, the labels I provided you are not even exactly the ones I use, since I didn't want to run afoul of licensing issues with the names used on the spreadsheet component I purchased.
2. I don't have a strong opinion on colon vs. lists for handling muli-name mapping. I suppose a list is more pythonic and json friendly but not strongly so.
3. Absolutely, this relates to item 1. Items I need to drop are not necessarily items other people want to drop. In fact, I wouldn't be surprised if other users want to drop MORE items than I do, or perhaps collapse items that I distinguish. For instance, The various tax and retirement related Balance Sheet items I have might just get collopsed into more generic non-operating items for other users. Flexibililty is the watchword.
4. In my case no priority or preference is implied by the order. Resolution is strictly based on examining the surrounding context in the statement.
5. I only tracked deprecation on xbrl tags so I could clean out superfluous tags after a period of time. I only use 10 years of data for my purposes so I didn't want the burden of carrying obsolete tags. There was a lot of churn in the standard xbrl in the first few years. Over time I expected that to settle down, and it seems that's the case.

Please let me know if you need more input on this. I'm really looking forward to seeing where this leads.

[dgunning]

Thanks,

I think in the code I did so far I used my labels not yours but I can check.

Also can you review this PR #573 .. this also does standardization.

[mpreiss9]

I don't think I was as clear as I should have been on item 1. What I would love to see is the ability to map xbrl tags to whatever label the user wants (based on the json mapping file). What I would hate to see is a mapping from xbrl to user-defined to your mapping with user-defined than invisible in the output. That would be terrible.

[mpreiss9]

For what it's worth here's the code I use to assign sections to the balance sheet. The mapping of section to tags should be pretty obvious. My data sits in a class that's basically a list of dataclass rows - so that's self.tbl in this code.

TAG_TO_BS_SECTION: dict[Tag, Tag] = {
    tag: section
    for section, tags in BS_SECTIONS.items()
    for tag in tags
}
...

from boltons.iterutils import windowed_iter
...
    def fill_subsection(self):
        # Pass 1: Trivial mapping and Section Headers
        for row in self.tbl:
            if row.std_tag in TAG_TO_BS_SECTION:
                row.section = TAG_TO_BS_SECTION[row.std_tag]
            elif row.std_tag in BS_SECTIONS:
                row.section = row.std_tag

        # Pass 2: Contextual fill using windowed_iter
        # We use a window of 3 (prev, curr, next)
        for prev, curr, nxt in windowed_iter(self.tbl, size=3, fill=None):
            if not curr or curr.section:
                continue

            # 1. Same context: Row is sandwiched between identical sections
            if prev and nxt and prev.section == nxt.section:
                curr.section = prev.section

            # 2. Boundary: Previous row was a total
            elif prev and prev.std_tag in BS_SECTIONS:
                # If prev was a total, we must be in the next section (defined by next)
                curr.section = nxt.section if nxt else prev.section

            # 3. Coloned Tag Ambiguity
            elif ":" in (curr.std_tag or ""):
                candidates = [p.strip() for p in curr.std_tag.split(":") if p.strip()]
                candidate_sections = {TAG_TO_BS_SECTION.get(c) for c in candidates if TAG_TO_BS_SECTION.get(c)}

                if prev and prev.section in candidate_sections:
                    curr.section = prev.section
                elif nxt and nxt.section in candidate_sections:
                    curr.section = nxt.section

            # 4. Fallback to forward fill or initial default
            if not curr.section:
                if prev and prev.section:
                    curr.section = prev.section
                else:
                    # Default for the very first rows if they aren't tagged
                    curr.section = Tag.ASSETS_CURR_BAL

        return self