It's been an interesting night. We get ping flooded about 8pm, then at
about 11:15 PM, our router starts turning its head. When I telneted into
the router to check the status of our link, it began acting really wierd.
For starters, it was constantly dropping packets to me. A "show s1" would
only return about half the statistics, and pressing enter again would
return me to the prompt. The link status on our T1 was toggling between
ESTABLISHED and DISCONNECTING, and we had TX, and other errors. The router
refused to complete many commands. I rebooted it, and reset S1 to see if
that would help. No luck.
At this point, the router was acting so strangley, that I was concerned
that it was a denial of service attack directed at my router, so I shut
down my S1 port. Viewing the stats on S1 showed much more outbound traffic
than inbound, so I did a ptrace to see what was coming in via the ethernet
interface. I noticed a strange SMTP connection continuing to flood in that
was directed at an external host. Another thing I noticed (it was very
choppy), was that all the Sequence and Ack numbers were identical on
thousands of packets.
Just to see if that was the cause, I shut down the switch port that was
attached to that machine, re-enabled my S1 port and everything worked
smoothly. When I re-enabled that port, I was quickly turned off from the
external net and the router began acting strangly again. In this state, it
was hard to get any commands completed (a "sh mem" would either only print
out the first line, then return me to the prompt, or print nothing at all
and return me to the prompt). After several attempts, I managed to get a
'sh mem' across, and it appeared that my 'nbufs' were down to 21.
So I shut down the offending port on my switch and did a traceroute to the
remote machine that the machine was sending haywire packets to: it wasn't
up.
I surmised that the unpredictable behavior of the router was due to its
nbufs being depleated, but what caused it? Was it the fact that a machine
on my ethernet was pounding out IP packets to a down machine, thus filling
up the router's buffers (I could see how this would flood my T1, but not
how it would kill my router) or was it a bug? Could it have had something
to do with the fact that the sequence numbers on all of these TCP headers
were the same (doubt it, that's level 3)?
Placing a filter for that IP address on the ethernet allowed me to
re-enable that switch port (it had some 8 other machines on it), but I am
a bit confused at how that could have killed my router to the point it
couldn't even communicate with me, I mean, it wouldn't even print a login
prompt for me).
Does anyone have any ideas?
-Jason
-
To unsubscribe, email 'majordomo@livingston.com' with
'unsubscribe portmaster-users' in the body of the message.