Performance Benchmark Comparison of Lucent Technologies PortMaster 3 Integrated Access Server vs.
Ascend's MAX 4048 InterNetworking Systems Concentrator vs.
3Com's Total Control InterNetworking Systems Concentrator
December, 1997
In December 1997, LANQuest Labs conducted a InterNetworking Systems Server performance comparison which included products from three vendors. The units tested were: The Lucent Technologies PortMaster 3 Integrated Access Server utilizing the Lucent based K56flex modems, the Ascend MAX 4048 InterNetworking Systems Concentrator utilizing the Rockwell based K56flex modems, and the 3Com Total Control InterNetworking Systems Concentrator utilizing the US Robotics based 56Kx2 technology modems.
The goal was to determine the maximum throughput (in bytes per second) of each InterNetworking Systems server (RAS) while handling various numbers of calls/connections. All connections were across analog modem lines. The number of calls established across each Unit Under Test (UUT) were 1, 2, 4, 8, 12, 16, 20, and 23. LANQuest's Net/WRx 4.0 traffic generator/analyzer was used to generate download traffic into the Ethernet interfaces of each UUT, out the WAN interface of each UUT, into a call generator/switch, into external analog modems, and finally into PC workstations. Ethernet Packet sizes of 256 bytes were used to test the limitations of each UUT. The test results determined that the Lucent Technologies' PortMaster 3 Integrated Access Server outperformed the other InterNetworking Systems servers in both total average throughput and total average aggregate throughput.
CONCLUSION
The results of this performance test determined a definite variance between the three InterNetworking Systems servers tested. As displayed below, the PortMaster 3 Integrated Access Server outperformed the other InterNetworking Systems servers in both total average throughput (per connection) and total aggregate throughput (for all connections). Lucent Technologies' PortMaster 3 Integrated Access Server delivered a superior throughput performance for all packet rates and number of connections compared to the Ascend MAX 4048 and the 3Com Total Control InterNetworking Systems Concentrator. The PortMaster 3 was as much as 232% faster than Ascend's MAX 4048 and as much as 29% faster than 3Com's Total Control. Furthermore only the PortMaster 3 scaled linearly up to and provided maximum throughput for the 23 maximum number of connections tested.
Performance Test #1: TOTAL AVERAGE THROUGHPUT PER CONNECTION
The goal of this test was to determine the total average throughput for each analog connection. To obtain this value, first the average throughput per connection was measured, then all the calls were averaged to get the total average of all calls for that test run. The throughput was measured in bytes per second (Bps). When measuring the total average throughput per call), Lucent Technologies' PortMaster 3 Integrated Access Server performed up to 232% better than Ascend's MAX 4048, and 29% better than 3Com's Total Control. The total average throughput performance (in bytes per second) is shown in the following table and graph for each of the InterNetworking Systems servers tested.
| Average Bps/call | 1 Call | 2 Calls | 4 Calls | 8 Call | 12 Calls | 16 Calls | 20 Calls | 23 Calls |
|---|---|---|---|---|---|---|---|---|
| Lucent PortMaster 3 | 11,528 | 11,507 | 11,518 | 11,521 | 11,518 | 11,503 | 11,353 | 11,511 |
| Ascend MAX 4048 | 11,523 | 11,528 | 11,516 | 10,595 | 7,047 | 5,172 | 4,038 | 3,466 |
| 3Com Total Control | 11,399 | 11,333 | 11,143 | 11,298 | 11,300 | 11,278 | 10,371 | 8,909 |
Note: The upper limit per call is the maximum speed of each PCs serial port (measured as 115,200 bits per sec or 11,520 data bytes per second with start & stop bits). Up through 4 calls the minor differences of Bps/call shown in the table above reflect the data compression differences of the Digital Signal Processors (DSP) in the Modems and UUTs.
Performance Test #2: TOTAL AVERAGE AGGREGATE THROUGHPUT (ALL CONNECTIONS)
The goal of this second test was to determine the total average aggregate throughput for all analog connection for that specific test run. To obtain this value, first the average throughput per connection was measured, then added together for a total aggregate of all calls for that test run. The throughput was measured in bytes per second. When measuring the total average aggregate throughput (for all connections), Lucent Technologies' PortMaster 3 Integrated Access Server provided up to 232% faster throughput than Ascend's MAX 4048, and up to 29% faster than 3Com's Total Control. The total average aggregate throughput performance (in bytes per second) is shown in the following table and graph for each of the InterNetworking Systems servers tested.
| Aggregate Ave (Bps) | 1 Call | 2 Calls | 4 Calls | 8 Call | 12 Calls | 16 Calls | 20 Calls | 23 Calls |
|---|---|---|---|---|---|---|---|---|
| Lucent PortMaster 3 | 11,528 | 23,015 | 46,072 | 92,170 | 138,216 | 184,046 | 227,050 | 264,742 |
| Ascend MAX 4048 | 11,523 | 23,056 | 46,065 | 84,759 | 84,568 | 82,758 | 80,766 | 79,726 |
| 3Com Total Control | 11,399 | 22,666 | 44,472 | 90,381 | 135,597 | 180,441 | 207,414 | 204,907 |
TEST METHODOLOGY AND TEST BED SETUP
LANQuest Labs used the latest software images and related software release for all three InterNetworking Systems servers. The latest images were downloaded from the respective websites and installed per instructions. For the Lucent Technologies PortMaster, boot image CmOS 3.7.2 was used. For the Ascend MAX 4048, image file 5.0Ap36 was used. For the 3Com Total Control, the following was used: Total Control NetServer Card V 3.5.34, data dump software rev 5.0.0, supervisor software rev 5.6.6, and hardware rev 3.0.0. All InterNetworking Systems servers were configured in accordance with their manufacturer's instructions. Compression was enabled on all analog modems.
The Lucent Technologies PortMaster 3 Integrated Access Server utilized Lucent based K56flex modems, the Ascend MAX 4048 utilized Rockwell based K56flex modems, and the 3Com Total Control InterNetworking Systems Concentrator used US Robotics based 56Kx2 technology modems.
The test bed comprised of 24 (23 used as clients and 1 used as controller for call generator/switch) P200 PCs with 32 MB RAM running Microsoft Windows 95. The 23 client PCs used either a USRobotics Sportster FAXModem (model # 00178602) when testing the 3Com UUT or a Hayes ACCURA 56k + FAX (model # 5674US) when testing the Lucent and Ascend UUTs. The modems were connected to the serial ports on each of the PCs. A Cybex AV-8 Autoview Commander was used to switch between each of the 24 PCs for monitoring and control through a single monitor and keyboard/mouse. Each PC was defined by its IP address. LANQuest's Net/WRx 4.0 traffic generator/analyzer was used to simulate traffic loads across the test bed. The Ethernet Packet sizes were fixed at 256 bytes for all test runs, and packet rate varied according to the number of calls being setup. A Pentium 200MHz PC with 32 MB RAM and running Microsoft Windows NT 4.0 (service pack 1) was used as the Net/WRX 4.0 platform. A 3Com 3C905 TX Fast EtherLink XL adapter card was used. A Zarak Systems Abacus bulk call generator and call switch was used to provide dial tone, call setup and call switching between the clients and the InterNetworking Systems server under test. The Abacus was set to emulate a 5ESS switch. A Bay Networks Baystack model 50 10 Base-T 4 port hub was used to connect NICs from traffic generating PC to the LAN interface on the InterNetworking Systems server. The drawing below illustrates the equipment setup.
All tests on the InterNetworking Systems servers were conducted with identical procedures, test equipment, equipment configurations, and test software. The performance analysis of the total average throughput (per connection/call) for each UUT comprised of sending 256 byte packets across simultaneous analog modem connections from 1 to 23. The number of calls were incremented from 1 up to 23. Throughput was measured for 1 , 2, 4, 8, 12, 16, 20, and 23 calls respectively. Throughput was measured in bytes per second. The packet rates varied according to the number of calls. The Ethernet packet rates were as follows: 150 packets per second (pps) for 1 call, 300 pps for 2 calls, 400 pps for 4 calls, 550 pps for 8 calls, 700 pps for 12 calls, 850 pps for 16 calls, 950 pps for 20 calls, and 1200 pps for 23 calls. A total of 4 samples (at 3 second intervals each) were recorded per configuration.
The desired number of calls or connections were set up between the Abacus and each of the external analog modems. After the Abacus provided dial tone and connections were up, Microsoft's Windows 95 dial up networking (on each of the client PCs) was used to dial in to the Abacus. Once the connections were up, traffic was generated across the test bed by the LANQuest Net/WRx 4.0 traffic generator. Traffic was sent through the hub, into the LAN port of the InterNetworking Systems server under test, out the WAN port, through the Abacus where it was directed to each targeted modem. From each modem the generated traffic reached the individual client PCs. Microsoft's System Monitor was used to monitor the bytes received per second by each modem. All data readings were sampled for 3 seconds and a total of 4 readings were taken for each test run. Averages for each connection were calculated.
LANQuest LABS
LANQuest Labs is the nation's leading independent network product testing laboratory. Since 1987, LANQuest has tested thousands of combinations of network operating systems, servers, routers, bridges, adapters, workstations, protocol analyzers and network applications. LANQuest's testing services include compatibility, functionality, interoperability, performance, Q.A., stress, and certification testing. LANQuest Labs' tests are nationally known and have appeared in such publications as Network World, Internet Week, LAN Technology, MacWEEK and Personal Computing. LANQuest also develops and markets network testbench software for use in Engineering and QA labs as well as network preventative maintenance and diagnostics tools for large Enterprise Networks. LANQuest's clients include most leading network vendors, including 3Com, Cisco Systems, Hewlett-Packard, IBM, Intel, Lucent, Madge, Microsoft, Novell, SMC, Olicom, and many others. Visit our website at http://www.lanquest.com.
Notice
This report is based on testing conducted by LANQuest Labs. Every effort has been made to ensure that the results described herein are fair and accurate. This report may be reproduced and distributed, as long as no part of this report is omitted or altered.