Skip to content

Commit e8b44db

Browse files
ksfikasclaude
andcommitted
CSV reader: skip lines, name the columns from a list or a header file, empty-value pattern; compact date-times are read as UTC
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
1 parent 49e1838 commit e8b44db

6 files changed

Lines changed: 191 additions & 11 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -827,3 +827,5 @@ The Overview is the **default screen**: `/` and a login that was not sent anywhe
827827
A new dataset **and a new data service** are created with the visibility chosen on the form (*Who can see it in CKAN*: private, the default, or public); an entry that is already in CKAN keeps its own visibility, which is changed with the *Make public* / *Make private* button, and a later push never resets it (a data service used to be created public and re-sent as public with every update).
828828

829829
**API documentation of each public address** (`api_docs.py`). Every published pipeline has a Swagger page at `/api-docs/<source>[/<pipeline>]` and its OpenAPI 3.1 description at `.../openapi.json`, public like the address itself (exempt from the login) and a 404 for anything that is not published of an active source. It is made from what the loader knows: the path and method (for an API source, the one callers use, and the request template's parameters as required query parameters or a JSON body), the output format's content type with an example and, for a source that stores data, the column names and types and example rows of the newest output (a pass-through API source says "as the provider returns it" instead of inventing a shape), the ETag / 304 behaviour, the 404 / 429 / 502 answers and the per-address rate limit. The Swagger UI script and style are loaded from jsdelivr (`swagger-ui-dist`, a pinned version), so nothing is installed on the VM. The Load Box shows an *API docs* link on the public-address line, and a data service's `documentation` always includes the link.
830+
831+
**CSV files that are not a plain table** (`transform_engine.csv_options`; the reader step in the Transform Box, under "The file has lines before the table, no header row, or marks for 'no value'"). A CSV reader can skip a number of lines at the top, name the columns from a list when the file has no header row (typed, or filled from a header file such as EMY's `header_info.txt` in the browser: the first token of each line, a titles line skipped, a repeated name numbered), and turn values that match a pattern (`/+` for the `////` a weather station writes where it has no sensor) into empty ones. A date and time with no separators (`202604010150`, also 8 and 14 digits) is read as UTC by the *datetime* conversion. These are the pieces for the two-line files the EMY stations send.

‎providerharvest_service/templates/transform/step.html‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,41 @@ <h1>{% if kind in ("input", "output") %}{{ title }}{% else %}{{ "Edit" if number
7373
{% endif %}
7474
{% endif %}
7575

76+
{% if r.kind == "csv_records" %}
77+
<form method="get" class="csv-shape" style="margin:0 0 0.8rem;">
78+
<input type="hidden" name="format" value="csv_records"><input type="hidden" name="delimiter" value="{{ r.delimiter or ',' }}">
79+
<details{% if r.csv.skip_lines or r.csv.column_names or r.csv.empty_regex %} open{% endif %}>
80+
<summary>The file has lines before the table, no header row, or marks for "no value"</summary>
81+
<div class="row-pair"><div><label for="skip_lines">Skip this many lines at the top</label><input id="skip_lines" name="skip_lines" value="{{ r.csv.skip_lines }}" size="3" maxlength="3"></div>
82+
<div><label for="empty_regex">Values that mean "nothing" (a pattern, e.g. <code>/+</code> for ////)</label><input id="empty_regex" name="empty_regex" value="{{ r.csv.empty_regex }}" maxlength="100"></div></div>
83+
<label for="column_names">The file has no header row: name its columns (one per line, or separated by commas)</label>
84+
<textarea id="column_names" name="column_names" rows="5" placeholder="Leave empty when the first line of the table holds the column names">{{ r.csv.column_names }}</textarea>
85+
<p class="muted" style="margin:0.2rem 0 0.4rem;">Or fill the list from a header file (a text file whose lines start with the column name, such as <code>header_info.txt</code>; a first line of titles is skipped and a name used twice gets a number):
86+
<input type="file" id="header-file" accept=".txt,.csv,text/plain" style="display:inline-block; width:auto;"></p>
87+
<button type="submit" class="btn-secondary btn-sm">Apply and look at the columns</button>
88+
</details>
89+
</form>
90+
<script>
91+
(function () {
92+
var input = document.getElementById('header-file'); if (!input) return;
93+
input.addEventListener('change', function () {
94+
var file = input.files && input.files[0]; if (!file) return;
95+
var reader = new FileReader();
96+
reader.onload = function () {
97+
var lines = String(reader.result || '').split(/\r?\n/).filter(function (l) { return l.trim(); });
98+
if (lines.length && /^name$/i.test(lines[0].trim().split(/\s+/)[0])) lines.shift();
99+
var seen = {}, names = lines.map(function (l) {
100+
var n = l.trim().split(/[\s,;]+/)[0].replace(/\W+/g, '_'); if (/^[^A-Za-z_]/.test(n)) n = 'f_' + n;
101+
seen[n] = (seen[n] || 0) + 1; return seen[n] > 1 ? n + '_' + seen[n] : n;
102+
});
103+
document.getElementById('column_names').value = names.join('\n');
104+
};
105+
reader.readAsText(file);
106+
});
107+
})();
108+
</script>
109+
{% endif %}
110+
76111
{% if r.info %}
77112
<form method="post" action="{{ '/transform/sources/%s/steps/%d/edit' % (source.id, number) if number else '/transform/sources/%s/steps/new' % source.id }}">
78113
<input type="hidden" name="kind" value="read">
@@ -87,6 +122,7 @@ <h1>{% if kind in ("input", "output") %}{{ title }}{% else %}{{ "Edit" if number
87122
{% endif %}
88123
{% elif r.kind == "csv_records" %}
89124
<div class="row-pair"><div><label for="delim">Delimiter</label><input id="delim" name="delimiter" value="{{ r.delimiter or ',' }}" size="2" maxlength="1"></div></div>
125+
<input type="hidden" name="skip_lines" value="{{ r.csv.skip_lines }}"><input type="hidden" name="column_names" value="{{ r.csv.column_names }}"><input type="hidden" name="empty_regex" value="{{ r.csv.empty_regex }}">
90126
{% elif r.info.repeating %}
91127
<label for="explode">A repeated part inside each record gets one row per repeat</label>
92128
<select id="explode" name="explode" style="max-width:26rem;">

‎providerharvest_service/transform_engine.py‎

Lines changed: 63 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -226,23 +226,70 @@ def read_json(content: bytes, options: dict) -> list[Record]:
226226

227227
# -- CSV -------------------------------------------------------------------------------------------------------
228228

229+
MAX_SKIP_LINES = 100
230+
231+
232+
def csv_options(options: dict) -> tuple[int, list[str] | None, "re.Pattern | None"]:
233+
"""The options of a CSV reader that are about the file's shape: how many lines to skip before the table starts, the column names to use when the
234+
file has no header row (a list, or text with one name per line or separated by commas), and a pattern for values that mean "nothing" (``/+`` for a
235+
station that writes ``////`` where it has no reading). Raises ``TransformError`` with a sentence."""
236+
try:
237+
skip = int(options.get("skip_lines") or 0)
238+
except (TypeError, ValueError):
239+
raise TransformError("The number of lines to skip must be a whole number.") from None
240+
if not 0 <= skip <= MAX_SKIP_LINES:
241+
raise TransformError("Skip between 0 and %d lines." % MAX_SKIP_LINES)
242+
names = options.get("column_names")
243+
if isinstance(names, str):
244+
names = re.split(r"[\s,;]+", names.strip())
245+
names = [str(n).strip() for n in (names or []) if str(n).strip()]
246+
if names:
247+
if len(names) > MAX_FIELDS:
248+
raise TransformError("At most %d columns." % MAX_FIELDS)
249+
for name in names:
250+
_check_name(name, "A column name")
251+
duplicated = sorted({n for n in names if names.count(n) > 1})
252+
if duplicated:
253+
raise TransformError("The column name %s is used twice: every column needs its own." % ", ".join(repr(d) for d in duplicated))
254+
pattern = None
255+
raw = str(options.get("empty_regex") or "").strip()
256+
if raw:
257+
if len(raw) > 100:
258+
raise TransformError("The pattern for empty values is longer than 100 characters.")
259+
try:
260+
pattern = re.compile(raw)
261+
except re.error as exc:
262+
raise TransformError("The pattern for empty values is not valid: %s" % exc) from None
263+
return skip, names or None, pattern
264+
265+
266+
def _csv_text(content: bytes, encoding: str, skip: int) -> str:
267+
try:
268+
text = content.decode(encoding)
269+
except (LookupError, UnicodeDecodeError) as exc:
270+
raise TransformError("Cannot read the file as %s: %s" % (encoding, exc)) from None
271+
return "".join(text.splitlines(keepends=True)[skip:]) if skip else text
272+
273+
229274
def read_csv(content: bytes, options: dict) -> list[Record]:
230275
encoding = str(options.get("encoding") or "utf-8-sig")
231276
delimiter = str(options.get("delimiter") or ",")
232277
if len(delimiter) != 1:
233278
raise TransformError("The delimiter must be one character.")
234-
try:
235-
text = content.decode(encoding)
236-
except (LookupError, UnicodeDecodeError) as exc:
237-
raise TransformError("Cannot read the file as %s: %s" % (encoding, exc)) from None
238-
reader = csv.DictReader(io.StringIO(text, newline=""), delimiter=delimiter)
279+
skip, names, empty = csv_options(options)
280+
reader = csv.DictReader(io.StringIO(_csv_text(content, encoding, skip), newline=""), delimiter=delimiter, fieldnames=names)
239281
if not reader.fieldnames:
240282
raise TransformError("The CSV has no header row.")
241283
fields = _fields(options, required=False, plain=True)
242284
columns = fields or [(re.sub(r"\W", "_", c).strip("_") or "col", c) for c in reader.fieldnames]
285+
286+
def cell(value: object) -> str | None:
287+
text = _text(value)
288+
return None if text is not None and empty is not None and empty.fullmatch(text) else text
289+
243290
records: list[Record] = []
244291
for row in reader:
245-
records.append({name: _text(row.get(column)) for name, column in columns})
292+
records.append({name: cell(row.get(column)) for name, column in columns})
246293
if len(records) > MAX_RECORDS:
247294
raise TransformError("More than %d records: refusing to continue." % MAX_RECORDS)
248295
return records
@@ -430,7 +477,7 @@ def sniff_delimiter(content: bytes) -> str:
430477
return best if first.count(best) else ","
431478

432479

433-
def describe_record(content: bytes, reader: str, record: str, sample: int = 50, delimiter: str | None = None, sheet: str | None = None) -> dict:
480+
def describe_record(content: bytes, reader: str, record: str, sample: int = 50, delimiter: str | None = None, sheet: str | None = None, csv_shape: dict | None = None) -> dict:
434481
"""The fields available inside a record: {"fields": [{"path", "example", "repeats"}], "repeating": [path...]}.
435482
``repeats`` marks a path that occurs more than once in a single record (a candidate to "explode")."""
436483
seen: dict[str, dict] = {}
@@ -496,8 +543,11 @@ def walk_json(node: Any, prefix: str) -> None:
496543
if isinstance(item, dict):
497544
walk_json(item, "")
498545
elif reader == "csv_records":
499-
rows = csv.reader(io.StringIO(content.decode("utf-8-sig", "replace"), newline=""), delimiter=delimiter or sniff_delimiter(content))
500-
header = [c.strip() for c in next(rows, [])]
546+
skip, names, _ = csv_options(csv_shape or {})
547+
text = content.decode("utf-8-sig", "replace")
548+
text = "".join(text.splitlines(keepends=True)[skip:]) if skip else text
549+
rows = csv.reader(io.StringIO(text, newline=""), delimiter=delimiter or sniff_delimiter(content))
550+
header = names or [c.strip() for c in next(rows, [])]
501551
first_rows = [r for _, r in zip(range(sample), rows)]
502552
for index, column in enumerate(header):
503553
if column:
@@ -553,7 +603,8 @@ def _convert(value: Any, kind: str) -> Any:
553603
raise ValueError
554604
if kind == "datetime":
555605
text = str(value).strip().replace("Z", "+00:00").replace("z", "+00:00")
556-
moment = datetime.datetime.fromisoformat(text)
606+
compact = {8: "%Y%m%d", 12: "%Y%m%d%H%M", 14: "%Y%m%d%H%M%S"}.get(len(text)) if text.isdigit() else None # (202604010150: a date and time with no separators, UTC)
607+
moment = datetime.datetime.strptime(text, compact) if compact else datetime.datetime.fromisoformat(text)
557608
if moment.tzinfo is None:
558609
moment = moment.replace(tzinfo=datetime.timezone.utc)
559610
return moment.astimezone(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
@@ -829,6 +880,8 @@ def validate(steps: object) -> list[str]:
829880
if kind == "xml_records" and not str(step.get("record") or "").strip():
830881
raise TransformError("Choose the repeating element that holds one record.")
831882
_fields(step, required=kind == "xml_records", plain=kind in ("csv_records", "xlsx_records"))
883+
if kind == "csv_records":
884+
csv_options(step)
832885
if step.get("explode"):
833886
_check_path(step["explode"].get("path"), "The path to the repeated element")
834887
_fields(step["explode"], required=False)

‎providerharvest_service/transform_forms.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,14 @@ def build_reader(kind: str, form: Mapping) -> dict:
165165
if kind == "csv_records":
166166
step["delimiter"] = _one(form, "delimiter", ",") or ","
167167
step["encoding"] = _one(form, "encoding", "utf-8-sig") or "utf-8-sig"
168+
shape = {key: _one(form, key) for key in ("skip_lines", "column_names", "empty_regex") if _one(form, key)}
169+
skip, names, _ = te.csv_options(shape) # (checks them; a bad one is told in words)
170+
if skip:
171+
step["skip_lines"] = skip
172+
if names:
173+
step["column_names"] = names
174+
if shape.get("empty_regex"):
175+
step["empty_regex"] = shape["empty_regex"]
168176
elif kind == "xlsx_records":
169177
if _one(form, "sheet"):
170178
step["sheet"] = _one(form, "sheet")

‎providerharvest_service/ui_transform.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -373,7 +373,7 @@ async def set_output(source_id: str, request: Request, session: Session = Depend
373373

374374
def _reader_context(session: Session, hsession: Session, source_id: str, request: Request, existing: dict | None) -> dict:
375375
sample = transform_runner.sample_for(session, hsession, source_id)
376-
ctx: dict = {"sample": sample, "kind": None, "candidates": [], "record": "", "info": None, "selected": {}, "explode": "", "readers": te.READERS, "error": None,
376+
ctx: dict = {"csv": {"skip_lines": "", "column_names": "", "empty_regex": ""}, "sample": sample, "kind": None, "candidates": [], "record": "", "info": None, "selected": {}, "explode": "", "readers": te.READERS, "error": None,
377377
"sheets": [], "sheet": ""}
378378
if sample is None:
379379
src = _source_or_none(session, source_id)
@@ -399,8 +399,13 @@ def _reader_context(session: Session, hsession: Session, source_id: str, request
399399
ctx["sheets"] = te.xlsx_sheets(sample.content)
400400
ctx["sheet"] = params.get("sheet") or (existing or {}).get("sheet") or ""
401401
if kind in tf.TABULAR or record or kind == "json_records":
402+
shape = {key: (params.get(key) if key in params else (existing or {}).get(key)) for key in ("skip_lines", "column_names", "empty_regex")}
403+
if isinstance(shape["column_names"], list):
404+
shape["column_names"] = "\n".join(shape["column_names"])
405+
ctx["csv"] = {key: ("" if value is None else str(value)) for key, value in shape.items()}
402406
ctx["info"] = te.describe_record(
403407
sample.content, kind, record, delimiter=params.get("delimiter") or (existing or {}).get("delimiter"), sheet=ctx.get("sheet") or None,
408+
csv_shape=ctx["csv"] if kind == "csv_records" else None,
404409
)
405410
except te.TransformError as exc:
406411
ctx["error"] = str(exc)

0 commit comments

Comments
 (0)