Skip to main content

Posts

Featured

Weird network behavior

Recently, we were asked to debug connectivity problems with an internal service, where about 1% of requests were resulting in "Connection reset by peer" errors.  There were no reported errors from the server itself, so we decided to take a closer look at the traffic on the wire.

First, we wanted to verify the ratio of resets occurring on the client side, so I ngrepped for all the TCP flags on the service port.
ngrep -d any port $SERVER_PORT | grep "\[" | grep -e "^T" | tr -s ' ' | cut -d ' ' -f5 | head -100000 | sort | uniq -c 42188 [A] 14041 [AF] 29212 [AP] 261 [AR] 14298 [AS]
Indeed, the number of resets on the client is higher than we'd expect.  The same thing on the server side.

ngrep port $SERVER_PORT | grep "\[" | grep -e "^T" | tr -s ' ' | cut -d ' ' -f5 | head -100000 | sort | uniq -c 37912 [A] 15269 [AF] 31238 [AP] 76 [AR] 34 [R] 15471 [S]
The weird thing here in this sample…

Latest Posts

Call me maybe

Do Androids Dream of Electric Sheep?

Applying Interfaces to external dependencies: Golang

RabbitMQ