Skip to content

pvagw stability issues under high load #204

Description

@SourabhHalli

I get the following errors when running pvagw:

  • double free or corruption
  • malloc(): unaligned tcache chunk
  • Bus error

Logs also show:

  • stale GET_FIELD, bad request id, connection resets

My current setup:

  • ~100 IOCs (~15k PVs each → ~1500k total)
  • Many heavy AD PVs (NDArray, image, ArrayData)
  • Single VM gateway, ~5 client machines
  • Initially one process, now split across multiple processes

What I tried:

  • Splitting load (15-20 IOCs per process) → improves but still crashes
  • Blocking NDArray PVs → more stable
  • Tuning:
    • channel_cache (lower helps)
    • getholdoff (0.05–0.1 helps, large values crash faster)

The issue becomes reproducible when scaling clients:

  • 3 clients → ~2.7 GB/s → stable
  • 4 clients → ~3.4–4 GB/s → gateway crashes
    At this point, these requests are almost equal to 400k*3 PVs at a time to gateway server.

Is gateway able to handle this amount of traffic?

Observation: Crashes correlate with sustained load, rising threads, and heavy GET/monitor traffic.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions