Skip to content

Fix/broadcastify call skew - #1158

Merged
robotastic merged 1 commit into
TrunkRecorder:masterfrom
jfgreco:fix/broadcastify-call-skew
Sep 1, 2026
Merged

robotastic merged 1 commit into
TrunkRecorder:masterfrom
jfgreco:fix/broadcastify-call-skew

Conversation

@jfgreco

@jfgreco jfgreco commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes Broadcastify Calls uploads being rejected with
REJECTED-CALL-SKEW-TOO-LONG when a Trunk Recorder call remains active
after its final retained RF transmission.

Trunk Recorder's normal start_time / stop_time metadata describes the
retained over-the-air transmissions. A call may remain open after its last
transmission while waiting for the inactivity timeout. Broadcastify evaluates
node skew from the call end timestamp, so promptly uploaded calls could appear
15+ seconds stale and be rejected.

This change:

  • captures a fixed call conclusion timestamp when Call_Data_t is created;
  • builds Broadcastify-specific metadata whose stop_time is the conclusion time;
  • derives start_time from conclusion time minus playable audio duration;
  • aligns freqList[].time and srcList[].time with playable-audio positions;
  • leaves the original call JSON unchanged for archives and other uploaders;
  • preserves the conclusion timestamp across retries so stale retries are not
    made artificially fresh.

Reproduction

On a live P25 Phase II system, the host clock was NTP synchronized and normal
upload latency was typically sub-second, but Broadcastify still rejected calls
with skew commonly in the 15–30 second range.

A 24-hour baseline produced 131 skew rejects.

Instrumentation showed successful calls could have a start_time more than
15 seconds old, while rejected calls tracked the age of the last retained
transmission / stop_time.

Calls containing a short retained transmission followed by a long inactivity
period could therefore be uploaded immediately but rejected with 16–18 seconds
of skew.

Validation

Tested against the live feed in stages.

Final timeline behavior:

  • 225 successful Broadcastify uploads
  • 92 duplicate/already-received skips
  • 0 skew rejects
  • 0 metadata errors

Final implementation with the fixed conclusion timestamp preserved across
retries:

  • 107 successful Broadcastify uploads
  • 18 duplicate/already-received skips
  • 0 skew rejects
  • 0 metadata errors

The final implementation was built and tested from v5.2.1.

@jfgreco
jfgreco force-pushed the fix/broadcastify-call-skew branch from 6f20cf5 to da0b5bd Compare August 28, 2026 22:51
@robotastic

Copy link
Copy Markdown
Collaborator

@blantonl - Could someone on your team give this a look over?

@blantonl

blantonl commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Looks good to me, ship it!

@jfgreco

jfgreco commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Additional production validation

We've been running this fix continuously since Friday and did some additional analysis of the remaining Broadcastify skew rejections.

The original inactivity-gap skew behavior appears resolved. We are no longer seeing the prior pattern where promptly uploaded calls were rejected simply because the call remained open after its last retained RF transmission.

We did observe a small number of remaining REJECTED-CALL-SKEW-TOO-LONG responses, but tracing them showed they were caused by real processing/upload latency after the call concluded rather than the timestamp calculation addressed by this PR.

The correlation was effectively 1:1. Examples from Aug 31:

  • ~15.5s between call conclusion and Broadcastify attempt → 15.452s skew
  • ~22.2s delay → 22.205s skew
  • ~35.3s delay → 35.227s skew
  • ~40.0s delay → 39.971s skew

We also saw several ~90s skew rejects. In each case, the OpenMHz uploader blocked until an HTTP 504 before the Broadcastify uploader was invoked. plugman_call_end() currently runs the plugins synchronously, with OpenMHz ahead of Broadcastify, so that delay propagated directly into Broadcastify skew.

One 122.078s rejection was a Broadcastify retry after the initial metadata request failed; the retry occurred ~121s later. This also confirms that preserving the original conclusion timestamp across retries is behaving as intended rather than artificially making stale retries appear fresh.

So the remaining rejects we've identified appear to be a separate plugin/upload-latency issue and not a regression in this change.

Based on several days of production use, the fix in this PR is doing what it was intended to do.

@robotastic

Copy link
Copy Markdown
Collaborator

GREAT! Nice work @jfgreco - thanks for the patch

@robotastic
robotastic merged commit 7b27f8b into TrunkRecorder:master Sep 1, 2026
@jfgreco
jfgreco deleted the fix/broadcastify-call-skew branch September 1, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants