Skip to content

fix(libudx): keep receiving after an undelivered datagram on Windows - #34

Closed
ColbySerpa wants to merge 1 commit into
Rightbracket:mainfrom
HORNET-Storage:fix/windows-udp-receive-loop
Closed

ColbySerpa wants to merge 1 commit into
Rightbracket:mainfrom
HORNET-Storage:fix/windows-udp-receive-loop

Conversation

@ColbySerpa

Copy link
Copy Markdown

Summary

  • On Windows, a UdxSocket stopped receiving after one of its datagrams went undelivered. Its receive loop ended at the first receive error, and Windows reports the ICMP error that such a datagram draws on the next receive of that socket.
  • The loop now carries on after those errors (WSAECONNRESET, WSAENETRESET), as libuv does. Other systems are unaffected.

Motivation

Windows hands the ICMP error for a datagram that a UDP socket sent earlier to the next receive on that socket (recvfrom):

  • WSAECONNRESET: "a previous send operation resulted in an ICMP Port Unreachable message", so a datagram to a peer that has gone;
  • WSAENETRESET: "the time to live has expired", so a low-TTL holepunch probe (SocketRef::send_holepunch).

Neither says anything about the socket, which still sends and receives. But the receive loop in UdxSocketInner::ensure_recv_loop was while let Ok(..) = udp.recv_from(..).await, so the first such error ended it. After that the socket delivered no more datagrams, and no more packets to the streams registered on it, until another stream was registered and started the loop again.

A DHT node shares a socket between its RPC and its streams, and routinely sends to peers that have gone. So on Windows a node stopped hearing replies, and its open streams stalled.

libuv, which the Node side runs on, ignores exactly these two errors for UDP and keeps receiving (uv__process_udp_recv_req in src/win/udp.c: "A previous sendto operation failed; ignore this error").

Changes

Crate Change
libudx The receive loop carries on after a receive error that only reports an undelivered datagram: ConnectionReset, and on Windows OS error 10052 (WSAENETRESET, which std gives no ErrorKind). It still ends on any other error.
libudx Tests: a socket, and a stream, still receive after a datagram from their socket was refused by a closed local port. Unit tests cover which errors the loop carries on after.
libudx CHANGELOG entry.

No public API changes.

Testing

  • cargo test --workspace (Linux: 1086 passed, 0 failed, 15 ignored)
  • cargo test -p peeroxide-cli --test live_commands -- --ignored (Linux: 6 passed)
  • cargo clippy --workspace --all-targets -- -D warnings (Linux, and -p libudx on Windows)

On Windows 11, cargo test -p libudx --test socket:

  • without the change, socket_receives_after_a_refused_datagram and stream_receives_after_a_refused_datagram fail in 3 runs of 3: the receive times out;
  • with it, both pass in 5 runs of 5.

On Linux they pass either way, since no error is reported there.

Notes

  • WSAENETRESET is handled from the documentation, not from a reproduction. A TTL-1 probe from my network drew no ICMP answer, so I could not observe it. WSAECONNRESET is reproduced by the tests.
  • Windows is not in CI, so windows_icmp_errors_report_an_undelivered_datagram, and the Windows behaviour of the two integration tests, only run locally.
  • Unrelated, seen while testing on Windows: stream_write_read_10mb and throughput_sustained time out there with and without this change. The rest of the libudx, peeroxide-dht and peeroxide suites pass, Node interop included. That is a pacing matter, which I will raise separately.

On Windows, a UDP socket reports on its next receive that a datagram it
sent earlier was not delivered: WSAECONNRESET after an ICMP "port
unreachable", and WSAENETRESET after an ICMP "time to live exceeded".
The socket itself still works.

The receive loop of `UdxSocket` ended at the first receive error. So one
datagram to a peer that had gone, or one low-TTL holepunch probe, left the
socket deaf: it delivered no more datagrams, and no more packets to the
streams registered on it, until another stream was registered. A DHT node
shares one socket between its RPC and its streams, so on Windows a node
stopped hearing replies and its streams stalled.

The loop now carries on after those two errors, as libuv does
(`uv__process_udp_recv_req`), and still ends on any other error. Other
systems report no such error on an unconnected socket, so nothing changes
for them.

Tests: a socket, and a stream, still receive after a datagram from their
socket was refused by a closed local port. On Windows both fail without
this change, where the receive times out, and pass with it. Elsewhere they
pass either way. Unit tests cover which errors the loop carries on after.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant