Skip to content

Commit c555285

Browse files
committed
Correct the migration guide against the API
open_dataset replaces create_dataset_reader throughout, read takes a register name as well as a class or address, and the epoch example and the parameter table match the current reader: epoch is set once on open_dataset, keep_type keeps its harp-python spelling, and time_index is new, since harp-python inferred the index from the first frame. read_all is gone, so whole-session loading becomes a comprehension over contents, which gains a section of its own. A new section warns that harp-data returns a bitmask register as a single integer column unless demux_bit_masks is set, so a script selecting a flag by name needs it. The static package example imports from the harp.device namespace and names no distribution. Em-dashes, second person and two mid-sentence enumerations are rephrased, and the horizontal rules are removed.
1 parent fee81da commit c555285

1 file changed

Lines changed: 82 additions & 60 deletions

File tree

Lines changed: 82 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,12 @@
11
# Migrating from harp-python
22

33
`harp-data` is the successor to `harp-python` for reading Harp binary data files into
4-
pandas DataFrames. The core concepts are unchanged — device schemas, register maps,
5-
binary files — but the API has been reorganized to separate data reading from device
4+
pandas DataFrames. The core concepts of device schemas, register maps and binary files
5+
are unchanged, but the API has been reorganized to separate data reading from device
66
communication.
77

88
This guide covers the three workflows most users relied on in `harp-python`.
99

10-
---
11-
1210
## Swap the package
1311

1412
Replace the old dependency:
@@ -21,20 +19,18 @@ pip install harp-python
2119
pip install harp-data
2220
```
2321

24-
If you want the full toolkit serial transport, device client, and data readinginstall
25-
the umbrella package instead:
22+
For the full toolkit of serial transport, device client and data reading, install the
23+
umbrella package instead:
2624

2725
```sh
2826
pip install harp
2927
```
3028

31-
---
32-
3329
## Loading a device schema at runtime
3430

3531
In `harp-python`, `harp.create_reader()` accepted a dataset folder and handled
36-
schema loading internally. `create_dataset_reader` is the direct replacement it
37-
finds the `device.yml` inside the folder automatically:
32+
schema loading internally. `open_dataset` is the direct replacement, and it finds the
33+
`device.yml` inside the folder automatically:
3834

3935
```python title="Before"
4036
import harp
@@ -43,41 +39,44 @@ reader = harp.create_reader("session.harp")
4339
```
4440

4541
```python title="After"
46-
from harp.data import create_dataset_reader
42+
from harp.data import open_dataset
4743

48-
reader = create_dataset_reader("session.harp")
44+
reader = open_dataset("session.harp")
4945
```
5046

5147
If the schema lives outside the data folder, pass it explicitly:
5248

5349
```python
54-
reader = create_dataset_reader("session.harp", schema="/path/to/device.yml")
50+
reader = open_dataset("session.harp", schema="/path/to/device.yml")
5551
```
5652

57-
### Accessing the device module
53+
### Finding out what a session holds
5854

59-
The reader holds a reference to the compiled device module at `reader.device_module`.
60-
Use it to look up register classes by name — no need to keep a separate variable:
55+
`reader.contents` maps the name of every register with data in the folder to its address,
56+
in address order, which is the quickest way to see what was recorded:
6157

6258
```python
63-
reader = create_dataset_reader("session.harp")
59+
reader = open_dataset("session.harp")
60+
print(reader.contents) # {'WhoAmI': 0, 'DigitalInputState': 32, ...}
61+
```
6462

65-
# access any register class through the reader
63+
The reader also holds the compiled device module at `reader.device_module`, so a register
64+
class can be reached without keeping a separate variable:
65+
66+
```python
6667
df = reader.read(reader.device_module.AnalogData)
6768
```
6869

6970
!!! note
70-
The old `harp.read_schema()` had no direct equivalent you needed to call separately.
71-
`create_dataset_reader` handles schema loading in one step, matching the convenience
71+
The old `harp.read_schema()` had no direct equivalent that needed calling separately.
72+
`open_dataset` handles schema loading in one step, matching the convenience
7273
of the original API.
7374

74-
---
75-
7675
## Reading a single register
7776

78-
The old API gave you attribute access on the reader`reader.AnalogData.read()`. The
79-
new API inverts this: you call `reader.read()` and pass the register class or its
80-
address as the argument.
77+
The old API allowed reads through attribute access on the reader. The new
78+
API inverts this, so `reader.read()` takes the register class, its name or
79+
its address as the argument.
8180

8281
```python title="Before"
8382
# by attribute name
@@ -94,13 +93,22 @@ df = reader.registers[44].read()
9493
# by register class (accessed through the reader)
9594
df = reader.read(reader.device_module.AnalogData)
9695

96+
# by name, the direct analogue of the old string lookup
97+
df = reader.read("AnalogData")
98+
9799
# by address
98100
df = reader.read(44)
99101
```
100102

103+
Names resolve against the device address space rather than the module namespace, so a
104+
common register such as `reader.read("WhoAmI")` works even though a device module does
105+
not name it.
106+
101107
### Absolute timestamps
102108

103-
The `epoch` parameter moves from the reader constructor into the `read()` call:
109+
The `epoch` parameter keeps its name and moves from `create_reader` to `open_dataset`.
110+
`harp-python` also allowed it per read. `harp-data` sets it once for the dataset, so
111+
every register is read on the same clock.
104112

105113
```python title="Before"
106114
reader = harp.create_reader("session.harp", epoch=harp.REFERENCE_EPOCH)
@@ -110,35 +118,53 @@ df = reader.AnalogData.read()
110118
```python title="After"
111119
from harp.data import REFERENCE_EPOCH
112120

113-
df = reader.read(reader.device_module.AnalogData, epoch=REFERENCE_EPOCH)
121+
reader = open_dataset("session.harp", epoch=REFERENCE_EPOCH)
122+
df = reader.read(reader.device_module.AnalogData)
114123
```
115124

116125
### Reading the whole session at once
117126

118-
`read_all()` returns a dictionary of DataFrames keyed by register name. Registers
119-
with no corresponding `.bin` file are skipped:
127+
There is no `read_all()`. Whole-session loading is a comprehension over `contents`,
128+
which keeps the choice of what to load with the caller:
120129

121130
```python
122-
everything: dict[str, pd.DataFrame] = reader.read_all()
131+
everything = {name: reader.read(name) for name in reader.contents}
123132
```
124133

134+
### Bitmask registers lose their per-flag columns by default
135+
136+
This is the change most likely to break working code. `harp-python` always expanded a
137+
bitmask register into one boolean column per flag, so a script could select a flag by
138+
name. `harp-data` returns a single integer column instead, and expands the flags only
139+
when asked:
140+
141+
```python title="Before"
142+
led = reader.DigitalOutputSet.read()["GP15"]
143+
```
144+
145+
```python title="After"
146+
led = reader.read("DigitalOutputSet", demux_bit_masks=True)["GP15"]
147+
```
148+
149+
Group masks need no such flag. `harp-python` mapped each value to its member name, and
150+
`harp-data` decodes them by default, as a `pd.Categorical` rather than plain strings, so
151+
a comparison against a string still reads naturally.
152+
125153
### Parameter reference
126154

127155
| harp-python | harp-data | Notes |
128156
|---|---|---|
129-
| `keep_type=True` | `message_type=True` | Renamed |
130-
| `epoch=REFERENCE_EPOCH` | `epoch=REFERENCE_EPOCH` | Same |
131-
| `epoch=None` | `epoch=None` (default) | Float seconds; same |
132-
|| `decode_enums=True` | New: enum fields as `pd.Categorical` |
133-
|| `demux_bit_masks=False` | New: expand bitmask flags into one column per flag |
134-
135-
---
157+
| `keep_type=True` | `keep_type=True` | Same, but the column is named `message_type` rather than `MessageType` |
158+
| `epoch=REFERENCE_EPOCH` | `epoch=REFERENCE_EPOCH` | Same, but set on `open_dataset` rather than per read |
159+
| `epoch=None` | `epoch=None` (default) | Float seconds, same |
160+
| inferred from the first frame | `time_index=False` | Needed for data carrying no timestamp |
161+
| always on | `demux_bit_masks=True` | Needed to keep per-flag columns |
162+
| always on | `decode_enums=True` (default) | Group mask values, now `pd.Categorical` |
136163

137164
## Schemaless read
138165

139-
If you have a raw `.bin` file and no schema — or you just want to inspect the data
140-
quickly — the `read()` function works the same as before. Only the import path and
141-
one parameter name change:
166+
For a raw `.bin` file with no schema, or a quick look at the data, the `read()`
167+
function works the same as before. Only the import path changes:
142168

143169
```python title="Before"
144170
import harp
@@ -151,38 +177,34 @@ df = harp.read("Behavior_44.bin", keep_type=True)
151177
from harp.data import read
152178

153179
df = read("Behavior_44.bin")
154-
df = read("Behavior_44.bin", message_type=True)
180+
df = read("Behavior_44.bin", keep_type=True)
155181
```
156182

157-
Both functions infer the payload type and element count automatically from the first
158-
frame — no register metadata needed.
159-
160-
---
183+
Both functions infer the payload type and element count from the frame, so no register
184+
metadata is needed. The new one assumes timestamped data, which is what a device sends,
185+
and takes `time_index=False` for the rare buffer that is not. It also takes `epoch` per
186+
call, since a single file has no dataset to set one on.
161187

162188
## Going further: static device packages
163189

164-
Loading a YAML at runtime is convenient, but for production workflows — or when you
165-
want IDE autocompletion and type safety — Harp device packages are pre-compiled Python
166-
modules that give you the same interface without any schema parsing at startup.
167-
168-
A static device package installs its register map as a proper Python module. You
169-
import it, pass it directly to `DatasetReader`, and the rest of the API is identical:
190+
Loading a YAML at runtime is convenient, but for production workflows, or where IDE
191+
autocompletion and type checking matter, a generated device package gives the same
192+
interface without parsing a schema at startup.
170193

171-
```python
172-
pip install harp-device-behavior
173-
```
194+
Such a package is an ordinary Python module under the `harp.device` namespace. Import
195+
it, pass it to `open_dataset`, and the rest of the API is identical:
174196

175197
```python
176-
from harp.device.behavior import device as behavior
177-
from harp.data import DatasetReader
198+
from harp.device import behavior
199+
from harp.data import open_dataset
178200

179-
reader = DatasetReader(behavior, "session.harp")
201+
reader = open_dataset("session.harp", behavior)
180202

181-
# everything works the same
203+
# a register class now resolves statically
182204
df = reader.read(behavior.AnalogData)
183-
everything = reader.read_all()
184205
```
185206

186-
The static module is faster to start up and ships with stubs for autocompletion. See
207+
A generated module starts up faster and resolves under a type checker, which a module
208+
built from a schema at runtime cannot. See
187209
[Generating Registers from a Schema](../examples/create_device_module/create_device_module.md)
188210
for how device modules are structured.

0 commit comments

Comments
 (0)