The Observer in the Loop: How Async Runtimes Distort Network Telemetry
Across 173 runs with kernel timestamps, I measured how much Python asyncio, Go and Rust tokio delay the ICMP replies a monitoring engine reports.
On the Linux hosts where I run Ephemres, a plain ping from the terminal to a monitored target came back steady and low. Ephemres, running on the same machine and probing the same targets, did not agree. Its RTTs carried a slight, regular offset above the terminal figures, its jitter was higher, and every so often its p95 and p99 jumped in a way the terminal never showed.
Same host, same targets, same wire. Either the network got slower only when Ephemres was looking at it, or Ephemres was adding its own delay to what it reported. I was fairly sure it was the second, and I had a suspect in mind. What I did not have was data to prove it. So I built a lab, ephemres-observer-lab, to measure it before changing anything.
Some context on how Ephemres (formerly LNMP) was developed. It grew up on single-vCPU Debian and Ubuntu VMs inside GNS3, so running well on one core has been a design consideration from the start. In current versions, each monitored endpoint gets a 60 s slot. Within it, the endpoint sends 5 pings 8 s apart (about 32 s of probing), then uses the remaining time to write a cycle summary to the database. Each endpoint’s start second within the minute is derived from its UUID, so 100 endpoints are spread across the full 60 s rather than firing together.
Back to the discrepancy. The suspect sat at the end of each cycle. When an endpoint’s fifth ping comes back (icmp_seq % 5 == 4 in the lab’s numbering), the endpoint acquires db_write_semaphore and does around 3 ms of synchronous CPU work: Pydantic validation, cycle statistics, state-machine evaluation and SQLAlchemy ORM serialisation. I had already raised that semaphore from 4 to 15 in production, and I was sure it would improve ICMP accuracy by letting end-of-cycle writes clear faster, so that fewer endpoints backed up behind the queue. Measurements did look better after that change, and I took it as confirmation that end-of-cycle writes were interfering with measurement.
The mechanism I had in mind was simple. All endpoints share one asyncio event loop on one thread. At 100 endpoints, about 53 are mid-probe at any moment. If endpoint A is busy for 3 ms building ORM objects, the loop cannot run the socket reader, so endpoint B’s echo reply waits in the kernel’s receive buffer. Ephemres stamps arrival time in user space, after reading the socket, so B’s RTT is charged for A’s CPU time.
If that was right, the usual advice would be to rewrite the engine in Go or Rust. Before considering anything of the kind, I wanted to know how large the effect was, and whether it was there at all.
Two Clocks on Every Packet
The lab is built around a self-check that needs no capture hardware and no cooperation from the target. Every echo reply gets two timestamps from the same clock (CLOCK_REALTIME, in nanoseconds):
- Clock A is the kernel’s arrival stamp. With
SO_TIMESTAMPING(orSO_TIMESTAMPNSas a fallback), Linux records when the network stack first handles the packet and passes that stamp to the program as ancillary data onrecvmsg(). It is fixed before any user-space scheduling happens, so nothing the program does can move it. - Clock B is read by the program the moment
recvmsg()returns.
The difference between them is the observer lag:
observer lag (intrusion) = Clock B - Clock A
It is how long the reply sat in the socket queue waiting for the program. The packet’s journey looks like this:
program kernel network target
─────── ────── ─────── ──────
tx = clock() ── sendto() ───▶ send path ───────────────────▶ wire ─▶ echo request ─▶ target OS
│
▼
Clock B = clock() ◀─ recvmsg ─ socket queue ◀─ Clock A ◀──── wire ◀── echo reply ◀─────┘
(program reads the packet) (packet waits) (kernel stamps
arrival here)
Clock A - tx is the RTT as the kernel saw it, and Clock B - tx is what a user-space monitor reports. The gap between them is the program’s own contribution. In IETF terms, this is the difference between wire time and host time, which RFC 2330 §10.2 already treats as a measurement error, and the kernel’s timestamping documentation describes the interface. The lab records both clocks on every packet. For the byte-level detail, the scm_timestamping layout and how each engine unpacks it is in the engines document, and the harness architecture and NDJSON event protocol are in the architecture document.
The idea is not new. More than twenty years ago, Pásztor and Veitch showed that ordinary user-space probers suffer from system noise at the moment the clock is read. More recently, Willem de Bruijn’s Netdev 0x17 talk, SO_TIMESTAMPING: Powering Fleetwide RPC Monitoring, described how kernel timestamps separate host delay from network delay in fleetwide monitoring at Google. What I wanted was narrower: a per-packet measure of how much one specific monitoring architecture distorts its own readings, compared fairly across runtimes.
Comparing fairly is the difficult part. The lab runs three engines with the same mechanics: the same raw socket, packet bytes, schedule and semaphore, and a 3 ms CPU spin standing in for the ORM work. Each engine runs in two CPU modes, which gives six configurations:
| Runtime | Single-core (pinned to core 7) | Multi-core |
|---|---|---|
Python asyncio | ORM spin inline on the event loop | ORM spin offloaded to a ProcessPoolExecutor |
| Go goroutines | GOMAXPROCS=1 | GOMAXPROCS = all CPUs |
Rust tokio | current_thread runtime | multi-thread work-stealing runtime |
Each engine also runs a canary, a background task that asks to sleep for 10 ms and records how late it actually wakes. The canary measures how long the runtime kept other work waiting, independent of any packet, so it shows starvation even when no reply happens to be waiting. All of this ran on one laptop, an Intel Core i5-8250U, probing a target on the local network, 192.168.70.2, whose kernel-measured RTT sat at about 370 to 480 µs under the staggered load. The methodology document defines every quantity.
The Staggered Schedule: What the ORM Work Cost
The first result was not the one I expected. Under Ephemres’s own configuration (100 endpoints, 60 s stagger, semaphore 15, 3 ms of ORM work), the ORM work added no measurable intrusion above the idle floor, which is the same run with the CPU spin set to zero. With the spin on and with it off, the medians were effectively the same.
The queueing arithmetic explains why. Endpoints finish a cycle about every 600 ms. Each write holds the CPU for 3 ms, plus 1 ms of simulated database I/O, so the average number of writers active at any moment is:
On average, there is almost never a second write in progress, let alone a reply waiting behind one. It also explains the 54-run semaphore sweep, where semaphores 5, 30 and 50 produced overlapping results for every engine. A limit that is never reached cannot change anything.
That result also undid something I had believed for a while. If fewer than one write is active on average, a cap of 4 was never the bottleneck either, so raising it to 15 is unlikely to be what improved Ephemres’s readings. (The lab’s smallest setting was 5, not 4, but under this load the difference between them should not matter.) Whatever improvement I saw after that release probably came from other changes to the code. It is uncomfortable to learn this about a fix I was sure of, and it is a good argument for measuring before believing.
What the stagger does leave is each runtime’s floor, and those differ:
Table 1: Staggered baselines and idle floors (100 endpoints, 60 s stagger, sem 15)
| Configuration | Idle floor median | Loaded median | Net mean intrusion | Loaded p99 | Error % of kernel RTT |
|---|---|---|---|---|---|
| Python single-core | 98.3 µs | 100.8 µs | +9.8 µs | 580.1 µs | 27.3% |
| Go single-core | 72.2 µs | 74.7 µs | +2.2 µs | 539.1 µs | 20.2% |
| Rust single-core | 71.8 µs | 70.0 µs | +10.3 µs | 921.1 µs | 22.7% |
| Python multi-core | 100.3 µs | 98.5 µs | -6.4 µs | 207.1 µs | 24.4% |
| Go multi-core | 69.7 µs | 71.2 µs | -0.9 µs | 135.7 µs | 15.5% |
| Rust multi-core | 69.0 µs | 68.1 µs | -2.8 µs | 120.2 µs | 15.9% |
Floors are single runs from Session 1. Net mean intrusion is the loaded mean minus the floor mean, and every value sits inside the repeat-to-repeat noise. Full matrices: Tables 1.1 to 1.4 and the semaphore sweep.
Python pays about 95 to 101 µs at the median across both sessions. On a LAN round trip of about 445 µs, that adds roughly 27% on top of the true round trip. A fixed cost of this kind, paid on every packet, is consistent with a steady offset against the terminal ping. Go and Rust pay about 68 to 75 µs. The gap is real but modest, around 30 µs, and it is present with or without the ORM work. It is the cost of Python’s path from socket readiness to a user-space timestamp, not a cost of the workload.
The next question was whether Python could narrow that gap by moving the CPU work off the event loop. Python’s multi-core mode does exactly that: the 3 ms spin goes to a pool of worker processes. It helps the event loop, with the canary average falling by 20 to 73 µs, but it does nothing for packets. The median stays at 98.5 µs against 100.8 µs inline, and the offload itself costs 0.741 ms on average per write, with a worst case of 12.11 ms, spent pickling the payload and passing it through IPC pipes. The multi-core idle floor settles the matter. It has no ORM work and no pool traffic at all, yet its median is 100.3 µs. Whatever the 30 µs is, it is not GIL contention or pipe handling. It appears to be the baseline cost of reading a packet in CPython.
What the First Session Got Wrong
After the first session of 90 runs, I had a tidy story. Parts of it were wrong, and I found out only because I distrusted three results enough to check them.
The first was that Python seemed to degrade with scale. At 250 endpoints, its median rose to 113.7 µs, outside the ranges at 25 and 100 endpoints, and the canary rose with it. That fitted my original suspicion well, which in hindsight should have made me more sceptical of it.
The second was a burst-mode comparison (Table 2.6 in the results document, now withdrawn). It suggested that about 43% of Python’s burst intrusion was present even with no ORM work, which implied that the other 57% came from the ORM work.
The third was the single-core p99 tail. Single-core runs had a p99 of 335 to 921 µs, while multi-core Go and Rust sat at 112 to 140 µs. Every host report the lab generated warned that the NIC’s interrupts might be handled on probe core 7, which made them an obvious candidate.
I checked the interrupts first, because that was quick. /proc/irq/159/effective_affinity_list for enp0s31f6 showed CPU 5. /proc/interrupts showed all 192,205 interrupts on that IRQ since boot on CPU 5, with none on core 7 or on CPU 3, its hyper-threading sibling. The warning had come from the metadata recording only the allowed mask (CPUs 0 to 7), not where interrupts actually ran. The NIC was ruled out, and the tail remains unexplained.
The other two needed a different session design. In Session 1, I had recorded idle floors once per configuration, in some cases hours apart from the tests they were compared against. This turned out to be the main methodological lesson of the project: the machine’s state drifted during the day by more than any staggered intrusion I was trying to measure. A morning test compared against an afternoon floor measures the afternoon as much as the ORM work.
Session 2 (81 runs, with engine order rotated each round) recorded idle floors minutes before their tests for the burst and 100-endpoint batches. The 25- and 250-endpoint floors followed their tests about an hour later. With that design, the 250-endpoint degradation disappeared: Python’s 250-endpoint test came in at 87.9 µs against a floor of 98.1 µs. It was below its floor, but given the hour between them, I would not read anything into the direction. At 100 endpoints, the interleaved floors and tests agreed within ±2 µs at the median for Python and Rust.
The burst comparison came apart in a more interesting way. When I sorted Python’s burst runs by the time they were recorded, they split into two regimes, and floors and tests switched regime together:
Table 2: Python’s bimodal burst regimes (100 endpoints, 0 s stagger, single-core unless noted)
| Recorded (2026-10-25) | Run | Runs | Median | Kernel RTT median | Send lag mean | Regime |
|---|---|---|---|---|---|---|
| 10:35 to 10:50 | test | 2 | 2.30 ms | 149.7 µs | 3.63 ms | Slow |
| 14:33 to 14:50 | floor | 2 | 1.09 ms | 235.2 µs | 2.25 ms | Fast |
| 18:56 to 18:59 | floor | 1 | 2.48 ms | 134.9 µs | 3.77 ms | Slow |
| 18:56 to 18:59 | test | 1 | 2.25 ms | 141.5 µs | 3.62 ms | Slow |
| 19:11 to 21:04 | floor | 3 | 1.11 ms | 235.4 µs | 2.29 ms | Fast |
| 19:11 to 21:04 | test | 12 | 1.19 ms | 224.6 µs | 2.27 ms | Fast |
The 14:33 floors include both CPU modes, and the 19:11 tests include semaphores 5, 15 and 50 plus multi-core runs. Source: §R3.5 and the interleaved pairs in §R3.2.
The withdrawn 43% had compared a slow-regime test from 10:35 with fast-regime floors from 14:33. Compared like for like, Python’s burst floor and test agree on every packet measure: a median of 1.11 ms against 1.22 ms with overlapping ranges, and ping 5 at 1.02 ms against 1.01 ms. Even in a burst, the ORM work added no measurable packet lag to Python. Only the canary changed, rising from 66 ms to 108 ms.
I do not know what sets the regime. The median tracks the send lag closely, which suggests that in a burst, Python’s read cost is really the cost of its own send loop. One switch coincided with a brief link drop, but the slow morning runs had no link event, so that cannot be the whole explanation.
The Burst Schedule: Three Runtimes, Three Failure Modes
A 0 s stagger, with all 100 endpoints starting at the same instant, is not how Ephemres runs. It is the stress test, and it is where the runtimes separate. It also showed me how easily run-level averages mislead.
In Session 1, Go’s burst mean was 13.02 ms and Python’s 2.65 ms, which made Go look five times worse. The per-packet CSVs told a different story: Go was reading most packets in tens of microseconds, and its delay sat almost entirely on one packet per cycle, ping 5, whose reply releases the endpoint into its ORM write. Splitting pings 1 to 4 from ping 5 (icmp_seq % 5 == 4) made the results readable.
Table 3: Burst mode, pings 1 to 4 vs ping 5 (Session 2, 100 endpoints, sem 15, db 3 ms)
| Engine | CPU mode | Run median | Pings 1 to 4 median | Ping 5 median | Worst packet | Canary max |
|---|---|---|---|---|---|---|
| Python | single-core | 1.22 ms | 1.28 ms | 1.01 ms | 3.41 ms | 107.87 ms |
| Python | multi-core | 1.13 ms | 1.20 ms | 1.01 ms | 3.39 ms | 74.34 ms |
| Go | single-core | 65.9 µs | 40.6 µs | 48.84 ms | 53.05 ms | 318.94 ms |
| Go | multi-core | 33.3 µs | 31.3 µs | 46.2 µs | 869.2 µs | 2.98 ms |
| Rust | single-core | 70.3 µs | 55.0 µs | 24.23 ms | 27.85 ms | 42.81 ms |
| Rust | multi-core | 32.8 µs | 26.4 µs | 3.17 ms | 9.04 ms | 10.41 ms |
Medians of three repeats. Ranges, the semaphore sweep and the Session 1 figures are in §R3.1, §R3.3 and §R2.3.
Python was slow on every packet but never severely so. Each packet waited about 1.1 to 2.3 ms depending on the regime, and the reader drained about 100 packets per wakeup, reading each round of replies in a single pass. The same happened with the ORM spin set to zero. Its worst packet stayed in single-digit milliseconds (3.41 to 8.54 ms across the summary tables), even while the canary was starved for 79 to 219 ms as the semaphore went from 5 to 50. The data seems to suggest why: the loop sends all 100 requests and reads all 100 replies before any endpoint reaches its ORM spin, so when the loop finally blocks, no reply is waiting. That ordering is my reading of the numbers, not something I traced directly.
Go behaved the opposite way. Pings 1 to 4 came back in roughly 30 to 60 µs, about 20 to 30 times faster than Python. On a single core, ping 5 stalled for 36.8 to 62.0 ms at the median across semaphores 5, 15 and 50, with a worst case of 208.22 ms in Session 1, and the canary was starved for 238 to 329 ms, roughly the full 100 × 3 ms of spinning. Go does preempt long-running goroutines, but only after about 10 ms (Clements’s non-cooperative preemption design describes the mechanism). Each 3 ms spin finishes inside that window, so preemption does not fire. Why the reader waits behind more than one semaphore’s worth of spins is less clear. It is possible that, with GOMAXPROCS=1, the goroutines woken by the network poller and by timers queue behind everything the burst made runnable. That is a hypothesis; the size of the stall is measured. With more cores, the stall disappeared in these runs: ping 5 at 46.2 µs, a run median of 33.3 µs and a canary maximum under 3 ms. It was the only configuration in the study that absorbed a burst without an end-of-cycle stall.
Rust sat in between, with a pattern I still find curious. On a single core, the canary froze for almost exactly semaphore × 3 ms: 15.83, 42.81 and 148.51 ms at semaphores 5, 15 and 50. The socket reader followed that rule only at semaphore 5, reading ping 5 after 15.2 ms. At 15 and 50, it read ping 5 after about 24.2 to 24.5 ms in both cases, as if it gets a turn after roughly eight spins regardless of the queue’s length. I am inclined to think this is tokio’s scheduler polling its I/O driver after a fixed amount of task work, not only when the run queue empties, but I have not isolated it. Multi-core Rust brought ping 5 down to 3.17 ms (9.04 ms worst), a large improvement but short of Go’s multi-core result. Possibly the reader task sometimes sits on a worker that is busy spinning until another worker steals it.
In no engine did a larger semaphore shorten a burst stall. For Rust’s runtime and Python’s event loop, it lengthened it.
The burst also showed something easy to miss: the kernel-measured RTT moved as well. Under burst, Clock A - tx fell to about 150 to 250 µs for Python and rose to about 528 to 889 µs for Go and Rust, with the same shift in the idle floors. This is the external observer effect, where the probe’s own send pattern changes what it measures. The likeliest reading is that Go and Rust fire their 100 requests in a tight volley, which queues them on the local send path and at the target, while Python’s slower, serial sends queue less. It is the kind of schedule-induced distortion that Sommers and colleagues studied in active SLA monitoring, in their work on loss measurement with Badabing and SLA compliance monitoring with SLAM. My own setup has a caveat here. tx is read in user space just before sendto(), so my “kernel RTT” still includes the host’s send path. Observer lag is unaffected, because tx cancels out of it, but the external-effect figures carry a little host time.
What the Questions Came Back With
The lab was built to answer the questions the discrepancy raised, not to decide what Ephemres should become. This is where each one landed.
How large is the distortion for a single-threaded asyncio monitor under Ephemres’s real load? About 100 µs at the median (100.8 µs, roughly 27% of a 445 µs LAN round trip), with a single-core p99 in the hundreds of microseconds. Almost all of it is Python’s own read path: the idle floor has the same median, with or without the 3 ms of ORM work.
How does it change with schedule, semaphore and endpoint count? The schedule dominates. Under the 60 s stagger, neither the semaphore (5 to 50) nor the endpoint count (25 to 250) moved the median beyond noise once floors were interleaved. Under a burst, every Python packet slowed to about 1 to 2.5 ms, and Go and Rust stalled the last ping of each cycle by tens of milliseconds. A larger semaphore never shortened a burst stall.
How do Go and Rust compare with Python? Under the stagger, they were about 25 to 30 µs faster at the median on one core or many, and on multi-core their p99 was much lower. Under a burst, the number of cores decided the ranking: single-core Go had the longest stalls in the study, while multi-core Go was the only configuration without an end-of-cycle stall.
What does Python gain, and pay, by offloading to worker processes? A calmer event loop, at a cost of 0.741 ms per write, with no change to packet lag on either schedule.
How do schedule, hardware and runtime each relate to the error? In these measurements, the schedule decided whether the ORM work mattered at all. Extra cores mostly helped Go and Rust, and mostly in the tail. The runtime set the read floor, at roughly 70 µs for the compiled engines and 100 µs for Python, and no configuration brought the staggered median below about 68 µs. The one thing that sidesteps that floor is the timestamp the lab already used as its reference: Clock A, the kernel’s arrival stamp from SO_TIMESTAMPING, which by construction excludes the user-space read delay in any language. The lab never used it as a monitor’s own timestamp, though, so how it would behave inside Ephemres is a question this study leaves open. It would also cover only the receive side. With tx still read in user space, send-side delay stays in the RTT, which is why a kernel transmit stamp (SOF_TIMESTAMPING_TX_SOFTWARE) is on the lab’s list.
What Remains Open
Several questions remain, each with a specific run that would settle it:
- What switches Python between its slow and fast burst regimes? Floors and tests switch together, and the median tracks send lag, but I have no cause. The next run is a day of Python burst floors logging CPU frequency and load, then one with the
performancegovernor. - Where does the single-core p99 tail come from? The NIC interrupts have been ruled out. The remaining candidates are the hyper-threading sibling, CPU 3, which shares core 7’s physical core, and the
powersavegovernor withC10deep sleep enabled (about 890 µs exit latency on this host). Neither has been isolated. The test is a single-core floor with cores 3 and 7 isolated, then again with C-states limited. - Why does Tokio’s reader get a turn after about eight spins? A burst at semaphores 8, 10 and 12 on single-core Rust would show where the cut-off sits.
- Why does multi-core Rust still read ping 5 late? Moving the reader to a dedicated thread would test the work-stealing explanation.
There are also limits to what the study can say. The 3 ms spin allocates no memory, so none of this says anything about garbage collection pauses in CPython or Go. Most settings had two or three repeats, which is enough to separate milliseconds but not single microseconds; five repeats is the standard I would hold a follow-up to. The replication guide describes how to rerun the matrix that way on your own host, and the open questions table is where I will record what changes.
What stays with me is less the 30 µs than how many of my own beliefs the data took apart along the way: the ORM work I suspected, the semaphore fix I was sure of, and three first-session findings that a better-designed session overturned. The instrument I trust most now is the floor recorded just before each test.
Kenneth Nnorom Website: kennethnnorom.com | LinkedIn: linkedin.com/in/kennethnnorom