#1958 [Ideas] Add instrumentation and latency metrics to the Anser subsystem New discussion created by leborchuk Description Description Anser (

gpcontrib/anser/

) is a runtime pub/sub facility: producer nodes on the segments publish a bloom filter over a join-build key, the coordinator unions the parts, and consumer nodes on the segments receive it and prune probe rows. See

gpcontrib/anser/README.md

for the architecture before starting. Today we can see what the filter did (

Rows Removed by Bloom Filter

in

EXPLAIN ANALYZE

) but nothing about what it cost. A producer blocks until the coordinator acknowledges its part; a consumer blocks until the payload is delivered. If those waits are expensive, the filter can be a net loss and we currently have no way to tell. Goal Make every wait and every queueing delay in Anser measurable, both per query (in

EXPLAIN

) and cluster-wide (in shared memory). Definition of done •

EXPLAIN (ANALYZE, VERBOSE)

shows producer and consumer wait count and wait time, including for segment-executed nodes. •

SELECT * FROM anser.stats()

shows queue time, service-cycle latency, and the histograms;

anser.stats_reset()

works. •

SELECT * FROM anser.session_stats()

reflects the current session only. • Regression tests pass and contain no timing-dependent expected values. •

README.md

gains a "Metrics" section: every field, its unit, and where it is measured. • No new LWLock acquisitions on the data path; no per-tuple timing calls. • Core (

src/

) is untouched. What to measure There are two distinct vantage points. Do not mix them up — this is the single most important thing to understand before writing code. | Vantage point | Who waits | Where the wait actually happens | | ---------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Executor node (segment or coordinator) | the query | segment: the libpq round trip in AnserClientPublish / AnserClientConsumeWait (src/anserclient.c); coordinator: the direct calls AnserPublish / AnserWaitProducersRegistered + AnserWaitReady + AnserConsumeReady (src/anser.c) | | Coordinator internals | the QD backend serving a segment, and the two background workers | AnserWaitSubmissionAck (src/anser.c:906), AnserWaitSlotResult (src/anser.c:1110), AnserGatherServiceCycle (src/anser.c:1235), AnserSendServiceCycle (src/anser.c:1433) | The first belongs in

EXPLAIN

. The second belongs in shared-memory counters, because no single backend sees it. Implementation Suggest three changes in this order. Each is independently reviewable and testable. Change 1 — per-node wait time in EXPLAIN • Time the publish call in the producer node and the receive call in the consumer node. Record, per node: number of waits, total wait time, and longest single wait. • Producer:

AnserProducePublishPart()

in

src/anserbloomproduce.c

— it picks the transport by role, so wrapping it covers both segment and coordinator. • Consumer:

ExecAnserBloomFilterConsume()

in

src/anserbloomconsume.c

. • Store the accumulators in

AnserBloomProduceScanState

/

AnserBloomConsumeScanState

(

src/anserplanexec.c:81

and

:95

). • Print them from

anser_produce_explain()

(

src/anserplanexec.c:422

) and

anser_consume_explain()

(

:636

), gated on

es->analyze

(there is nothing to print without execution) and shown only at

EXPLAIN (ANALYZE, VERBOSE)

— i.e.

es->analyze && es->verbose

. Use

ExplainPropertyFloat("Anser Wait Time", "ms", ...)

and

ExplainPropertyInteger("Anser Waits", NULL, ...)

. The trap you must handle. A field you add to the node state on a segment does not reach the QD. Read the comment at `src/anserplanexec.c:583-588`: the existing code deliberately uses

InstrCountFiltered1/2

because only the fixed fields of

CdbExplain_StatInst

(

src/backend/commands/explain_gp.c:44

) travel back. Your new timers are not in that struct. The supported escape hatch is the per-node extra text channel, which is how

Hash

reports `Extra Text: (seg2) Hash chain length ...`: •

PlanState.cdbexplainbuf

and

PlanState.cdbexplainfun

(

src/include/nodes/execnodes.h:1152-1153

) • collected on the segment by

cdbexplain_collectExtraText()

(

src/backend/commands/explain_gp.c:1308

) and shipped to the QD • copy the pattern from

src/backend/executor/nodeRuntimeFilter.c:174-177

— it is the closest existing analogue (a runtime-filter node doing exactly this) So: allocate

cdbexplainbuf

in

*_begin

when

estate->es_instrument

is set, install a

cdbexplainfun

that appends your numbers, and the QD will show them as

Extra Text

per segment. No core change is needed — if you find yourself editing anything under

src/

, stop and re-read this paragraph. Change 2 — subsystem-wide counters and latency histograms in shared memory • Add a stats block to Anser's shared memory. Either extend

AnserControl

(

include/anser.h

) or add a separate struct; account for it in

AnserShmemSize()

and initialize it in

AnserShmemInit()

(

src/anser.c

), exactly as the existing structures do. • Use

pg_atomic_uint64

(

src/include/storage/atomics.h

) for every counter and bucket, and update them without taking

AnserChannelLock

/

AnserRingLock

. These are on the data path; adding lock traffic for metrics would change what we are trying to measure. • Measure, at minimum: | Metric | Where | | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | time a part waits in the submission queue (enqueue → gather picks it up) | stamp the time in AnserEnqueueSubmission() (src/anser.c:860), read it in AnserGatherServiceCycle() (:1235) — add a field to AnserSubmissionEntry | | time a consumer slot waits for delivery (register → send stamps payload) | stamp in AnserRegisterWaitSlot() (:1077), read in AnserSendServiceCycle() (:1433) — add a field to AnserWaitSlot | | producer ack wait, on the QD side | AnserWaitSubmissionAck() (:906) | | consumer delivery wait, on the QD side | AnserWaitSlotResult() (:1110) | | background worker cycle duration, per service | around the AnserGatherServiceCycle() / AnserSendServiceCycle() calls in AnserServiceLoop() (src/anserservice.c) | | counts: parts submitted / accepted / rejected, channels cancelled by timeout, consumers delivered / cancelled | the same call sites … apache/cloudberry