Recovering Ethernet Deadlock (Shutdown)

NOTE: The deadlock state has been observed in only one network where seven PortMasters (TM) shared the same Ethernet segment with one Cisco AGS+. Gerenal guidelines for Ethernet network design suggest that a maximum of four PortMasters share the same Ethernet segment. This condition could exist in a properly designed Ethernet network, thus, you should read this technote thoroughly.

RELEASE:

This enhancement became available in ComOS Release pm2_3.3.3c2 and will be rolled into subsequent releases. For a copy of pm2_3.3.3c2, please contact Lucent's Technical Support department.

FUNCTION OF ENHANCEMENT:

ComOS checks the error recovery flag in the Ethernet interface driver at a one-second interval. The enhancement ensures that the interface remains active in an unusually poor environment with network interface statistics like those described below under "ETHERNET NETWORK CONDITIONS INDICATING DEADLOCK."

MESSAGE INDICATING DEADLOCK:

  "net_check_ether (ether0): DEADLOCK - restarting"
  • The message is generated to the command line console. Whether you login via Telnet or via the s0 console port, a "set console" is required to receive this system message.
  • The message is generated every time the Ethernet interface is re-enabled.
  • The message is not sent to SYSLOG. 

Note: The message is not generated by any previous ComOS releases.
ETHERNET NETWORK CONDITIONS INDICATING DEADLOCK:

Before you upgrade your ComOS, ensure that the deadlock conditions exists! If rebooting is the only way to recover the use of the Ethernet interface, some other problem might be occurring.

The following conditions exist during a deadlock state:

  • The Ethernet interface is no longer reachable.
  • When the Ethernet interface shuts down, all IP traffic through and to the interface stops.
  • The Ethernet interface IP address is not pingable from:
  • A network device on the Ethernet (i.e. workstation or router)
  • A Dial-up network device
  • PortMaster console
  • The "attach s<port #> command fails.
  • In order to recover the Ethernet interface:
  • A "reset nic" command can be issued (the best way to determine a deadlock).
  • A "reboot" command or hard Power Cycle (not desirable in determining a deadlock).

A deadlock (shutdown) condition clearly indicates that the interface is being subjected to an unusually poor network environment. An example of an unusually abused Ethernet interface had the following statistics:

 
Command> show netstat

Name   Ipkts    Ierrs  Opkts    Oerrs Collis  Resets Queue
ether0 29812179 146128 29989920 1279  9379640 4397   0

Name = interface name
Ipkts = number of packets received since reboot
Ierrs = number of input errors since reboot
Opkts = number of packets sent since reboot
Oerrs = number of output errors since reboot
Collis = number of output collisions since reboot
Resets = number of resets on the interface since reboot
Queue = number of packets waiting to be sent from the interface

ETHERNET TROUBLESHOOTING BENCHMARKS:

Some useful benchmarks in troubleshooting an Ethernet interface:

Collision rate (%) = <number of Collis>/<number of Opkts>.

Excellent Ethernet conditions:
Zero collisions, Ierrs, Oerrs, and Resets.

Good Ethernet conditions:
Collision rate less than 1-5% and zero Ierrs, Oerrs, and Resets.

Marginal Ethernet conditions:
Collision rate 5-10% and zero Ierrs, Oerrs, and Resets.

Poor Ethernet conditions:
Collision rate above 10% and single-digit Ierrs, Oerrs, or Resets in one day.

Very poor Ethernet conditions:
Collision rate above 10% and/or double-digit Ierrs, Oerrs, or Resets in one day.

Unusually poor Ethernet conditions (meltdown):
Collision rate above 30% and/or more than 1 Reset an hour, and a high rate of Ierrs and Oerrs.

Anytime you troubleshoot an evasive Ethernet segment/interface issue, be sure to consult the IEEE specifications for the respective Ethernet network types and assure your network design follows proper guidelines.

  • 10Base-2 - Thinnet - Coax with T-connectors.
  • 10Base-5 - Thicknet - Monster Coax with Vampire tap connectors.
  • 10Base-T - Twisted Pair - RJ45 connectors.