Skip to content

Commit d02db68

Browse files
ksfikasclaude
andcommitted
Transform Box: offer the richest repeating element first as the record, and explain the choice
The field chooser defaulted to the most frequent repeating element, which for a SOAP feed is a single reading (no id to pick). It now orders candidates by how much each holds (a whole detector first), drops plain-text leaves, and says in words what to pick. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
1 parent e730da8 commit d02db68

3 files changed

Lines changed: 24 additions & 3 deletions

File tree

‎providerharvest_service/templates/transform/step.html‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,8 @@ <h1>{{ "Edit" if number else "Add" }}: {{ title }}</h1>
5050
{% else %}<a href="?format={{ r.kind }}&record={{ c.path|urlencode }}">{{ c.path }} &times;{{ c.count }}</a>{% endif %}
5151
{% endfor %}
5252
</div>
53-
<p class="muted" style="margin:0 0 0.6rem;">The repeating parts of the file, with how many times each occurs. Usually the one with the most records you care about.</p>
53+
<p class="muted" style="margin:0 0 0.6rem;">The repeating parts of the file, with how many times each occurs. Pick the part that holds <strong>one whole record</strong> -- everything you want to
54+
keep about one item (its id <em>and</em> its details). The first one is our best guess. A part that holds only some details (for example one reading) has no id to choose.</p>
5455
{% else %}
5556
<p class="muted">Nothing repeating was found in this file.</p>
5657
{% endif %}

‎providerharvest_service/transform_engine.py‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -393,8 +393,18 @@ def detect_format(content: bytes, filename: str = "") -> str | None:
393393
def candidates(content: bytes, reader: str) -> list[dict]:
394394
"""The places in a sample that look like "one record, repeated": [{"path", "count"}], biggest first."""
395395
if reader == "xml_records":
396-
counts = Counter(_local(e.tag) for e in _parse_xml(content).iter())
397-
return [{"path": n, "count": c} for n, c in counts.most_common() if c > 1][:20]
396+
counts: Counter = Counter()
397+
size: Counter = Counter() # elements inside, summed over every occurrence
398+
structured: set[str] = set() # has child elements or attributes: could be a record (a plain text leaf never is)
399+
for element in _parse_xml(content).iter():
400+
name = _local(element.tag)
401+
counts[name] += 1
402+
size[name] += sum(1 for _ in element.iter()) - 1
403+
if len(element) or element.attrib:
404+
structured.add(name)
405+
found = [{"path": n, "count": c, "inside": size[n] / c} for n, c in counts.items() if c > 1 and n in structured]
406+
# The best guess first: the repeating element that holds the most (a whole detector, not one reading of it).
407+
return sorted(found, key=lambda c: (-c["inside"], -c["count"]))[:20]
398408
if reader == "json_records":
399409
found: list[dict] = []
400410

‎tests/test_transform_engine.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -385,3 +385,13 @@ def test_add_field_needs_a_value():
385385
for value in (None, ""):
386386
with pytest.raises(TransformError, match="Give a value"):
387387
te.step_add_field([{"a": 1}], {"name": "z", "kind": "constant", "value": value}, te.Context(lookups={}))
388+
389+
390+
def test_the_record_that_holds_the_most_is_offered_first_and_plain_text_leaves_never_are():
391+
found = te.candidates(SOAP.encode("utf-8"), "xml_records")
392+
paths = [c["path"] for c in found]
393+
assert paths[0] == "ds" # a whole detector, before the readings or the single-value elements inside it
394+
assert "ident" not in paths and "tstore" not in paths and "detectorState" not in paths # text leaves: never a record
395+
assert "value" in paths # a leaf with attributes can be a record (as in a flat list of items)
396+
flat = "<l><i code='A'/><i code='B'/><i code='C'/></l>"
397+
assert [c["path"] for c in te.candidates(flat.encode(), "xml_records")] == ["i"]

0 commit comments

Comments
 (0)