You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2ccdcda
Browse filesBrowse the repository at this point in the historyBrowse files
The service is an ETL: sources come in (Extract), may be transformed (Transform), and go out
(Load). Only the tab label changes; addresses stay, so existing links keep working. The README
describes the planned Transform and Load boxes.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: README.md
+16-1Lines changed: 16 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,6 +21,21 @@ pipeline. Do not add a dependency from here back onto
21
21
`ckanext.providerharvest` (or vice versa) -- the whole point is these
22
22
are independently deployable.
23
23
24
+
## The shape: Extract, Transform, Load
25
+
26
+
The service follows an ETL flow, and its tabs follow it: sources come **in**, may be **transformed**, and go **out**
27
+
(to data.gov.gr, or by other means).
28
+
29
+
| Tab | Role | State |
30
+
|---|---|---|
31
+
|**Extract Box**| the sources: registering them, how they deliver (pull over HTTP/SFTP/FTP, or push by API key or FTP/SFTP login), their runs and files | built (this was "Provider portal") |
32
+
|**Transform Box**| turning what was extracted into what is wanted (field mapping, cleaning) | planned |
33
+
|**Load Box**| sending the result out to data.gov.gr or elsewhere | planned |
34
+
| Admin dashboard, **Data boxes**, Security box | operating it: approvals and retention; the HTTP and FTP machines; access control | built |
35
+
36
+
Only the tab is renamed for now; the addresses (`/provider/...`) are unchanged so existing links and the providers' own
37
+
bookmarks keep working.
38
+
24
39
## Status: engine + CKAN adapter + working CI/CD + a live FTP/SFTP receiving server + token provisioning + full provider/admin source lifecycle (register/edit/delete) + connection health checks + a live API proxy + a real, manually-triggered harvest that stores data in PostgreSQL
25
40
26
41
`providerharvest_service/engine/` -- the transport implementations
@@ -96,7 +111,7 @@ the admin API can't be pointed at an arbitrary path. Generated
96
111
passwords are bcrypt-hashed and shown exactly once, at creation (or
97
112
rotation) -- never returned again.
98
113
99
-
The whole site -- the provider portal, the admin pages, the API docs --
114
+
The whole site -- the Extract Box (the provider-facing pages), the admin pages, the API docs --
100
115
sits behind a real login (`admin_session.py`: a signed session cookie,
101
116
one shared username/password, enforced by a single ASGI middleware). It
102
117
is **deny by default**: a path needs the session unless it is on a short
0 commit comments