Revert "pad: Don't drop LATENCY queries with default implementation"

This reverts commit 794944f779f954375fc74a3fffcc2067bba6a3e5.

Accumulating non-live latency values generally makes no sense and often
gives invalid results with min>max