Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/api/data.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,5 @@
::: harp.data.to_file
::: harp.data.to_buffer
::: harp.data.REFERENCE_EPOCH
::: harp.data.timecode.decode_timecode_from_samples
::: harp.data.timecode.decode_timecode_from_transitions
14 changes: 14 additions & 0 deletions docs/examples/time-code.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Align to the Harp Time Code

Devices that are not part of the Harp bus run on their own clock, so their timestamps drift with respect to Harp time. The [Harp Time Code](https://harp-tech.org/protocol/TimeCode.html) exists so they can be aligned afterwards: a Harp device emits the current whole Harp second as a slow serial signal (typically 1 kbps) on a digital output, the non-Harp system records it on a spare digital or analog input, and `harp.data.timecode` decodes that recording back into Harp seconds.

Each packet starts exactly on the whole second it carries, so every decoded packet is an anchor: the sample (or local time) of its first falling edge, against the Harp second that begins there.

!!! note
Every packet that frames cleanly is returned as decoded, including the rare one carrying a corrupt value, and a reset of the Harp clock mid-recording shows up as a jump in the anchors. The example below drops anchors that do not line up with the rest before fitting the offset and drift between the two clocks.

<!--codeinclude-->
```python
[](./time_code.py)
```
<!--/codeinclude-->
33 changes: 33 additions & 0 deletions docs/examples/time_code.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import numpy as np
from harp.data.timecode import decode_timecode_from_samples, decode_timecode_from_transitions

# A non-Harp acquisition system recorded the Harp Time Code on one of its digital
# lines. `samples` is that line, sampled at the system's own rate.
samples = np.load("timecode_line.npy") # digital or analog; the threshold is detected
sample_rate = 30_000.0

# Decode it: one row per whole Harp second, keyed on the sample where that second
# starts, the axis this system timestamps the rest of its data on too.
anchors = decode_timecode_from_samples(samples, sample_rate, baud_rate=1000.0)
print(anchors.head())
# Time
# Sample
# 36000 3806874.0
# 66003 3806875.0

# Every cleanly framed packet is returned as decoded. Harp time minus local time stays
# nearly constant, so the rare corrupt value, or a reset of the Harp clock, stands out.
offset = anchors["Time"] - anchors.index / sample_rate
anchors = anchors[(offset - offset.median()).abs() < 1.0]

# A line through the anchors gives the offset and drift between the two clocks, and
# extends past the first and last anchors. Any of the system's timestamps (spikes,
# video frames, stimulus onsets) can then be placed on the Harp axis.
slope, intercept = np.polyfit(anchors.index, anchors["Time"], 1)
spike_samples = np.load("spike_samples.npy")
harp_times = intercept + slope * spike_samples

# Event-based systems report line transitions instead of a sampled waveform: a local
# time and the level the line took. Anchors then carry local seconds.
transitions = np.load("line_transitions.npy")
anchors = decode_timecode_from_transitions(transitions[:, 0], transitions[:, 1])
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ nav:
- Read a dataset folder: examples/dataset.md
- Read a single register file: examples/register-file.md
- Registers from a schema: examples/registers-from-schema.md
- Align to the Harp Time Code: examples/time-code.md
- Guides:
- Choose a device module: articles/device-modules.md
- Migrate from harp-python: articles/harp-python-migration.md
Expand Down
29 changes: 29 additions & 0 deletions src/packages/harp-data/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,35 @@ _data, timestamps, _msg, payload = AnalogData.parse_bulk(raw)
df = data.payload_to_dataframe(payload)
```

## Align a non-Harp device with the Harp Time Code

Devices outside the Harp bus keep their own clock. A Harp device emitting the
[Harp Time Code](https://harp-tech.org/protocol/TimeCode.html) sends the current
whole Harp second as a slow serial signal (typically 1 kbps) on a digital output,
so those devices can record it and be aligned post-hoc. `harp.data.timecode`
turns such a recording back into a table of anchors, keyed on whichever axis the
device timestamps its own data on (the sample, or the local time in seconds, of
each packet's first falling edge) against the Harp second (`"Time"`) that starts
there:

```python
import numpy as np
from harp.data import timecode

anchors = timecode.decode_timecode_from_samples(line, sample_rate=30_000.0) # Sample -> Time
slope, intercept = np.polyfit(anchors.index, anchors["Time"], 1)
harp_times = intercept + slope * spike_samples

# event-based systems report transitions instead, so anchors carry local seconds
anchors = timecode.decode_timecode_from_transitions(edge_times, edge_states) # LocalTime -> Time
```

Packets that fail their start/stop bit check are dropped. Every other packet is
returned as decoded, so a rare corrupt value or a reset of the Harp clock shows up
as an anchor that does not line up with the rest; the
[example](https://harp-tech.org/python/examples/time-code/) filters those out
before fitting.

## Write data back out

`to_file` and `to_buffer` are the inverse of the readers, encoding values as Harp frames. Useful for round-tripping data or generating test corpora:
Expand Down
9 changes: 9 additions & 0 deletions src/packages/harp-data/src/harp/data/timecode/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
from ._decode import (
decode_timecode_from_samples,
decode_timecode_from_transitions,
)

__all__ = [
"decode_timecode_from_samples",
"decode_timecode_from_transitions",
]
205 changes: 205 additions & 0 deletions src/packages/harp-data/src/harp/data/timecode/_decode.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
import warnings
from typing import Any

import numpy as np
import pandas as pd
from numpy.typing import ArrayLike, NDArray

_BITS_PER_BYTE = 10 # 8N1: start bit, eight data bits, stop bit
_PACKET_BYTES = 4 # whole second as a little-endian U32
_RESYNC_WINDOW = 0.3 # max start-bit offset, in bits, to re-synchronize a frame on
_THRESHOLD_SUBSAMPLE = 1_000_000 # samples used to detect the threshold


def decode_timecode_from_transitions(
timestamps: ArrayLike,
states: ArrayLike,
*,
baud_rate: float = 1000.0,
) -> pd.DataFrame:
"""Decode a Harp Time Code from line transitions.

See the [Harp Time Code specification](https://harp-tech.org/protocol/TimeCode.html).
``timestamps`` are the local times, in seconds, of each transition and ``states`` the
level the line took (0 low, 1 high). Returns the whole Harp second of each packet
(``"Time"``) indexed by the local time of its first falling edge (``"LocalTime"``).

Packets with bad framing are dropped; every other packet is returned as decoded, so
a clock reset or a corrupt value shows up as an anchor that does not line up.
"""
times, levels = _as_transitions(timestamps, states)
local, harp = _decode(times, levels, baud_rate=baud_rate)
return pd.DataFrame({"Time": harp}, index=pd.Index(local, name="LocalTime"))


def decode_timecode_from_samples(
samples: ArrayLike,
sample_rate: float,
*,
threshold: float | None = None,
baud_rate: float = 1000.0,
) -> pd.DataFrame:
"""Decode a Harp Time Code from a sampled line.

See the [Harp Time Code specification](https://harp-tech.org/protocol/TimeCode.html).
Sample ``i`` is taken at local time ``i / sample_rate``. Samples are binarized as
``samples >= threshold``; by default the threshold is detected from the data as the
midpoint between the high and low levels. Returns the whole Harp second of each packet
(``"Time"``) indexed by the sample of its first falling edge (``"Sample"``), filtered
as in :func:`decode_timecode_from_transitions`.
"""
if sample_rate <= 0:
raise ValueError(f"sample_rate must be positive, got {sample_rate}.")
if sample_rate < 2 * baud_rate:
raise ValueError(
f"A {baud_rate} bps signal cannot be decoded from samples taken at "
f"{sample_rate} Hz; at least 2 samples per bit are required (5 recommended)."
)

level = _as_levels(samples, threshold)
edges = np.flatnonzero(np.diff(level)) + 1
times = edges / sample_rate
local, harp = _decode(times, level[edges], baud_rate=baud_rate)
sample = np.rint(local * sample_rate).astype(np.int64)
return pd.DataFrame({"Time": harp}, index=pd.Index(sample, name="Sample"))


def _as_transitions(
timestamps: ArrayLike, states: ArrayLike
) -> tuple[NDArray[np.float64], NDArray[np.bool_]]:
"""Validate transitions into sorted times and boolean levels."""
times = np.asarray(timestamps, dtype=np.float64)
levels = np.asarray(states)
if times.ndim != 1 or levels.ndim != 1:
raise ValueError("timestamps and states must be one-dimensional.")
if times.size != levels.size:
raise ValueError(
f"timestamps and states must have the same length, got {times.size} and {levels.size}."
)
if np.any(np.diff(times) < 0):
raise ValueError("timestamps must be sorted in ascending order.")
return times, levels.astype(np.bool_)


def _as_levels(samples: ArrayLike, threshold: float | None) -> NDArray[np.bool_]:
"""Binarize a sampled waveform into digital line levels."""
values = np.asarray(samples)
if values.ndim != 1:
raise ValueError("samples must be one-dimensional.")
if threshold is None:
if values.dtype == np.bool_:
return values
threshold = _detect_threshold(values)
return values >= threshold


def _detect_threshold(values: NDArray[Any]) -> float:
"""Midpoint between the low and high levels of the line."""
if values.size == 0:
return 0.0
if values.size > _THRESHOLD_SUBSAMPLE:
# a fixed-seed random subsample is plenty, and a strided one could alias with the bits
values = values[np.random.default_rng(0).integers(0, values.size, _THRESHOLD_SUBSAMPLE)]
# extreme percentiles rather than min/max, so that rare glitches are ignored
return float(np.percentile(values, [0.01, 99.99]).mean())


def _level_at(
times: NDArray[np.float64], levels: NDArray[np.bool_], at: NDArray[np.floating[Any]]
) -> NDArray[np.bool_]:
"""Line level at each time in ``at``."""
index = np.searchsorted(times, at, side="right") - 1
# transitions alternate, so before the first one the line held the opposite level
return np.where(index < 0, ~levels[0], levels[np.maximum(index, 0)])


def _resync(falling: NDArray[np.float64], expected: float, bit: float) -> float:
"""Snap a predicted start-bit time to a nearby falling edge."""
best, window = expected, _RESYNC_WINDOW * bit
index = int(np.searchsorted(falling, expected))
for candidate in (index - 1, index):
if 0 <= candidate < falling.size:
distance = abs(falling[candidate] - expected)
if distance <= window:
best, window = float(falling[candidate]), distance
return best


def _decode_packet(
times: NDArray[np.float64],
levels: NDArray[np.bool_],
falling: NDArray[np.float64],
origin: float,
bit: float,
) -> tuple[int, float] | None:
"""Decode the packet starting at ``origin`` into (seconds, end time), or ``None``."""
payload = bytearray(_PACKET_BYTES)
offsets = (np.arange(_BITS_PER_BYTE) + 0.5) * bit
start = origin
for frame in range(_PACKET_BYTES):
if frame:
# predict from the previous start bit, not from the packet, so that a small
# baud rate error stays a fraction of a bit instead of accumulating
start = _resync(falling, start + _BITS_PER_BYTE * bit, bit)
bits = _level_at(times, levels, start + offsets)
if bits[0] or not bits[-1]: # start bit must be low, stop bit high
return None
payload[frame] = np.packbits(bits[1:-1], bitorder="little")[0]
return int.from_bytes(payload, "little"), start + _BITS_PER_BYTE * bit


def _decode(
times: NDArray[np.float64],
levels: NDArray[np.bool_],
*,
baud_rate: float,
) -> tuple[NDArray[np.float64], NDArray[np.float64]]:
"""Decode every packet into (first falling edge, Harp second) arrays."""
if baud_rate <= 0:
raise ValueError(f"baud_rate must be positive, got {baud_rate}.")
local, harp = _decode_packets(times, levels, 1.0 / baud_rate)
if local.size == 0 and np.any(~levels):
warnings.warn(_explain_no_packets(times, levels, baud_rate), stacklevel=3)
return local, harp


def _decode_packets(
times: NDArray[np.float64], levels: NDArray[np.bool_], bit: float
) -> tuple[NDArray[np.float64], NDArray[np.float64]]:
falling = times[~levels] # a packet can only start on a falling edge
local: list[float] = []
harp: list[float] = []
decoded_until = -np.inf
for origin in falling:
if origin < decoded_until:
continue # inside a packet we already decoded
packet = _decode_packet(times, levels, falling, float(origin), bit)
if packet is None:
continue
seconds, decoded_until = packet
local.append(float(origin))
harp.append(float(seconds))
return np.array(local, dtype=np.float64), np.array(harp, dtype=np.float64)


def _explain_no_packets(
times: NDArray[np.float64], levels: NDArray[np.bool_], baud_rate: float
) -> str:
"""Guess why a line with falling edges yielded no packet."""
message = f"No Harp Time Code packet decoded from {np.count_nonzero(~levels)} falling edges."
if _decode_packets(times, ~levels, 1.0 / baud_rate)[0].size:
return f"{message} The line looks inverted (idle low); invert it before decoding."
pulses = np.diff(times)
pulses = pulses[pulses > 0]
if pulses.size == 0:
return message
# most pulses inside a packet are one bit long, so the short end of the
# distribution estimates the bit period actually on the line
apparent = 1.0 / float(np.percentile(pulses, 10))
if abs(apparent / baud_rate - 1) <= 0.2:
return f"{message} Its pulses match {baud_rate:g} bps, but no packet frames cleanly."
rate = f"{apparent:,.0f}" if apparent >= 10 else f"{apparent:.2g}"
message += f" Its pulses look like ~{rate} bps, not the {baud_rate:g} bps of baud_rate."
if apparent < baud_rate / 100:
message += " If the timestamps are sample numbers, divide them by the sample rate."
return message
Loading
Loading