Add optional parallel plugin call_end execution

[jfgreco]

Summary

Adds optional parallel execution of plugin call_end() callbacks to prevent one slow uploader from delaying other upload plugins.

Trunk Recorder currently invokes call_end() plugins serially. In production, slow OpenMHz uploads and HTTP 504 responses were observed blocking later Broadcastify uploads for 15–90+ seconds. Because Broadcastify validates call timing, that head-of-line blocking caused otherwise healthy Broadcastify uploads to be rejected for excessive skew.

This change adds:

"parallelPluginCallEnd": true

The option defaults to false so existing installations retain the current serial behavior.

When enabled:

• active call_end() plugins are dispatched concurrently;
• the call worker still waits for all plugin callbacks to complete before continuing;
• existing file lifecycle behavior is preserved;
• failed plugin indexes are still collected into plugin_retry_list;
• the existing selective retry behavior remains unchanged.

Production validation

Tested on a live P25 Phase II system using the OpenMHz and Broadcastify uploaders.

Before this change, slow OpenMHz requests frequently delayed Broadcastify by the same amount. Examples included OpenMHz delays of ~15, ~22, ~35, ~40, and ~90 seconds, with corresponding Broadcastify skew rejections.

With parallelPluginCallEnd enabled:

• 5,518 successful Broadcastify uploads
• 564 duplicate/already-received Broadcastify responses
• 6,083 successful OpenMHz uploads
• 18 OpenMHz upload failures
• 0 Broadcastify skew rejections attributable to OpenMHz latency

A representative failure case:

• Broadcastify upload completed successfully
• OpenMHz remained blocked for approximately 90 seconds
• OpenMHz eventually returned HTTP 504
• Broadcastify was unaffected

The single Broadcastify skew rejection observed during the test followed a separate Broadcastify SSL connection failure and delayed Broadcastify retry, not cross-plugin blocking.

The service remained stable throughout the test period.

Compatibility

parallelPluginCallEnd defaults to false, so this does not change plugin execution semantics unless explicitly enabled.

[taclane]

Sounds like a valid fix, and honestly could be a default behavior in many cases. Prior to this, the best resolution was to not use the "simple" configuration for openmhz or broadcastify, and manually set their execution order in the plugins section.

Serial execution seemed like it worked most of the time, but as noted above, did lead to some issues when either OpenMhz or Broadcastify had issues with their upload API/

[cschmittiey]

this would be awesome - currently upload using several plugins and would love to get the latency down even further