Skip to content

Add harp.data.timecode for decoding the Harp Time Code - #22

Open
bruno-f-cruz wants to merge 7 commits into
mainfrom
feat-external-timestamp-alignment
Open

bruno-f-cruz wants to merge 7 commits into
mainfrom
feat-external-timestamp-alignment

Conversation

@bruno-f-cruz

@bruno-f-cruz bruno-f-cruz commented Aug 12, 2026 •

Copy link
Copy Markdown
Member

Add harp.data.timecode for decoding the Harp Time Code

Closes #14. Supersedes harp-tech/harp-python#38. Implements the analysis side of the Harp Time Code specification (harp-tech/protocol#241, #258).

Why

Devices outside the Harp bus timestamp their data on their own clock. A Harp device emitting the Harp Time Code sends the current whole Harp second as a slow serial signal (typically 1 kbps) on a digital output, so acquisition systems sampling at 5–30 kHz can record it on a spare digital or analog input and be aligned afterwards. Decoding that recording is currently done with ad-hoc scripts, one per lab. This PR adds it to harp.data.

What

from harp.data.timecode import decode_timecode_from_samples, decode_timecode_from_transitions

decode_timecode_from_samples(samples, sample_rate, *, threshold=None, baud_rate=1000.0)
decode_timecode_from_transitions(timestamps, states, *, baud_rate=1000.0)

Two entry points, one per shape of recording, each keyed on the axis that system already timestamps its own data on:

input index column
uniformly sampled waveform (digital or analog) Sample (int64) Time
line transitions (time, 0/1), as event-based systems report LocalTime (seconds) Time

Time holds the decoded whole Harp second, the same name the readers give the Harp time axis. Each row is an anchor: the first falling edge of a packet, which the spec fixes as the start of the second it encodes. Placing the device's own timestamps on the Harp axis is left to the caller; the docs fit a line through the anchors (np.polyfit), which absorbs the clock offset and drift and extends past the first and last anchors.

The wire format decoded is the one in the spec: idle high, 8N1, LSB first, four back-to-back bytes holding the whole second as a little-endian U32, with no header or checksum.

Decoding decisions worth reviewing

  • Per-frame re-synchronization. Bits are read at bit-period centers, and each frame re-syncs on its own start bit like a real UART, rather than predicting all 40 bit positions from the packet start. Without it a ~3% baud error walks off the bits by the fourth byte; with it that error stays a fraction of a bit.
  • Framing is the only filter. Packets whose start/stop bits do not check out are dropped, so a glitched or half-recorded packet costs one anchor, not the packets around it. Every other packet is returned as decoded: a rare corrupt value, or a reset of the Harp clock mid-recording, shows up as an anchor that does not line up, and the example filters those on Harp minus local time before fitting.
  • Threshold detection for sampled input. With threshold=None (the default) the cut is the midpoint of the 0.01st and 99.99th percentiles of a fixed-seed random subsample of up to 1 M samples, so rare glitches are ignored and a 1-hour recording costs ~20 ms. This handles analog channels, including integer ADC data such as Open Ephys int16, and a line that is low only ~0.5% of the time. Pass threshold to override.
  • An empty result explains itself. When a line has falling edges but no packet frames, a warning says why: the line decodes when inverted, the shortest pulses show a different baud rate than baud_rate, or they look thousands of times too long, which points to sample numbers passed as seconds. It only runs when nothing decoded, so a working recording pays nothing for it.
  • Sub-Nyquist input raises rather than silently returning garbage: fewer than two samples per bit is a ValueError naming both rates (five per bit is the practical recommendation, in line with the spec's "multiple samples per bit").

Tests

39 tests in tests/data/test_timecode.py, driven by a synthetic emitter that renders packets to either transitions or a sampled waveform:

  • framing: byte patterns whose runs of all-zeros/all-ones leave long stretches without a transition, 5 kHz and 30 kHz sampling, analog input with an explicit or detected threshold (noisy, mostly idle, integer ADC data, rare glitches), empty and transition-free recordings;
  • robustness: idle-line glitches, packets truncated at either end of the recording, ±0.2-bit jitter on every edge, 3% baud error, corrupt and jumping values returned as decoded, dropped packets, wrong baud rate, sample numbers passed as seconds, an inverted line, invalid arguments;
  • a minute-long 30 kHz recording combining 0.1% clock drift, jitter and one lost packet;
  • regression against real data: two Open Ephys traces of a Behavior board emitting at 1 kbps, lifted from harp-python#38, decoding to the Harp times and start times that PR expects (3806874-5 and 2600957-60), including its fifth packet, which frames cleanly but carries a corrupt value.

Docs

harp.data.timecode entries on the Data API page, an "Align a non-Harp device with the Harp Time Code" section in the harp-data README, and a new example page, Align to the Harp Time Code, in the nav.

ruff format/ruff check, codespell and mkdocs build --strict are clean, and the full suite passes (487 tests). pyright reports no issues in the new code.

@bruno-f-cruz

Copy link
Copy Markdown
Member Author

@jsiegle Leaving this as a placeholder and describing what the current blockers are that must be resolved at the specification level before this can be merged.

Decode a recorded Harp Time Code, as specified in harp-tech/protocol
TimeCode.md, into anchors mapping a non-Harp system's sample or local
time to the whole Harp second that starts there. Packets are four
headerless 8N1 bytes, and anchors sit on each packet's first falling
edge, which the specification fixes as the start of the encoded second.
@bruno-f-cruz
bruno-f-cruz force-pushed the feat-external-timestamp-alignment branch from 53f6a91 to f4e3576 Compare November 20, 2026 21:37
@bruno-f-cruz bruno-f-cruz changed the title Add harp.data.synchronization for aligning non-Harp devices to the Harp clock Add harp.data.timecode for decoding the Harp Time Code Nov 20, 2026
@bruno-f-cruz
bruno-f-cruz marked this pull request as ready for review November 20, 2026 23:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add module for aligning local timestamps to the Harp clock

1 participant