Team 3 Webpage

FINAL DRAFT FOR THE FINAL REPORT
(Dalat pulled the final parts together)

TEST TOOL DEVELOPMENT FOR ACCESS CONCENTRATOR DEVICES

CSC402/ECE470 - (Inter)Networking Technology & Projects

Team 3:

Dalat Bui

Chris Brezovec
Shadi Eskamaei

Tyler Humphrey

K. Fritz Lehr

North Carolina State University

November 20, 2002

EXECUTIVE SUMMARY (Tyler)
(to Table of Contents)

Our general over-arching goal this semester is to define a test methodology for Access Concentrators. To test and verify our methodology, we run the tests against many Access Concentrator brands being used in the field today. Examination of these results and recommendations for the future are also to be done. While examining the tests, we determine the ranges where these particular Access Concentrator brands fail and succeed during the test.

While researching various Access Concentrators, we are looking for new features that the vendors are advertising, that might make a good and reasonable test for the future. Based on this and results from the tests, we will proceed to make recommendations for following semesters.

TABLE OF CONTENTS (Dalat)

I. Cover page

II. Executive Summary

III. Table of Contents

IV. Introduction

V. Problem Description

Vendor Specific Functionality
Session Capacity Test Problem Description
Session Establishment Rate Problem Description
Throughput Problem Description
Session Stability Test Problem Description

VI. Approach to Solution

Vendor Specific Functionality
Throughput
Session Capacity
Session Establishment Rate
Access Concentrator Stability Tests

VII. Results

VIII. Verification and Validation

Vendor Specific Functionality
Session Capacity Tests
Session Establishment Rate Tests
Throughput Tests
Access Concentrator Stability Tests

IX. Schedule

X. References

XI. Appendices

Introduction
Session Capacity
Session Establishment Rate
Throughput
Access Concentrator Stability

INTRODUCTION (Tyler)
(to Table of Contents)

Owing to the success of the Internet, the number of connected hosts is growing at an exponential rate [10]. As a result, service providers and carriers continually have to respond to the increasing demand. They look for ways to expand their network while reducing the operational costs. One solution these service providers and vendors will most likely consider is an Access Concentrator that can handle their current needs and allow them to expand their network in the future.

An Access Concentrator is an intelligent networking device that sits at the edge of the IP cloud between the Internet Service Provider (ISP) backbone network and the end users (or end user private networks). Its main job is to handle all the connections between the two parties. An AC, which can act as both a router and a switch, is built to handle many connections and to provide many IP services such as Virtual Private Networks (VPN), management control, and security services.

The popular protocol used by many AC's to set up and maintain connections is the Point-to-Point Protocol (PPP). This protocol is used as a result of the early dial-up connections. Originally, Serial Line Internet Protocol (SLIP) was used to network two computers by using a telephone line. However, as networking technology developed and the medium bandwidth increased, the SLIP was replaced with the PPP. This is why PPP is now the stardard protocol used by ISP's in a point to point connection. To expand their service and increase the number of customers, PPP is run over Ethernet (PPPoE), over ATM (PPPoA), and over both Ethernet and ATM (PPPoEoA). In this project we are mainly dealing with PPPoE because it is more popular and simpler than the other two.

PROBLEM DESCRIPTION (Shadi)
(to Table of Contents)

Many AC vendors such as Redback, Cisco, netCom, Nortel, Cosine, and Juniper are working competitively to claim a larger share of the market. This sometimes pushes them to over advertise their products. For example, Redback may claim that it's SMS10000 to be able to 100,000 concurrent connections (www.redback.com). In reality, it may never be able to handle that many concurrent connections unless some special techniques are applied. Through our research and a close examination of an Access Concentrator, we derive four major aspects that should be tested: subscriber session capacity, session establishment rate, throughput, and device stability. The test on each of the aspects will be designed to simulate situations that are as close to real life situations as possible. In addition, these tests will determine the operational ranges of each of these aspects.

Vendor Specific Functionality (Tyler)

In studying and researching the various vendors of these Access Concentrator devices, it became evident that each vendor advertises different attributes including but not limited to maximum session capacity, maximum session establishment rate, maximum throughput, and stability. Stability is in other words, what causes the DUT to reach an unstable state. Many other attributes of each vendor's Access Concentrator are advertised that are rather difficult to test. However, the problem is the aforementioned quantitative attributes can be either overestimated to make their particular product seem better than it actually is or underestimated in order to assure that an unstable state will never be reached. Tests will be run to test the actual capabilities of each vendor's product to determine how close the advertised specifications come to the actual values.

Session Capacity Test Problem Description (Fritz)

The first test performed is session capacity, since it affects all subsequent tests performed. Session Capacity is the maximum number of sessions that a DUT (Device Under Test) can establish and maintain. The established sessions will be validated by the use of Echo-Requests sent to the DUT, after which the DUT will respond with an

Echo-Reply indicating the session is up. The methodology for determining this should produce repeatable and consistent results when used on different Access Concentrators. Once Session Capacity is accurately determined, the following tests including Session Establishment Rate, Throughput, and Session Stability, which use these results, can be performed.

Session Establishment Rate Problem Description (Dalat)

To provide services to more customers, an Access Concentrator must be able to handle more connections. To do this, it must be able to establish sessions at a higher rate. This is mainly for reliability purposes. For example, when an AC comes back on from a disaster, something that is very likely to happen, it is bombarded with establishment connection requests. If the AC is not able to handle these requests at a high enough rate, the AC will crash, resulting in customer complaints. Therefore, being able to establish connections at a high rate is very important. As a part of this project, we develop tests to determine how fast an Access Concentrator can establish connections. Once it has been determined exactly how many sessions can be successfully established, the Throughput test can be run under the assumption that the sessions have been successfully established and that data is ready to be transmitted in an effort to determine maximum throughput of the device.

Throughput Problem Description (Shadi)

Access Concentrator throughput tests check how well an AC network device transmits and receives data. Vendors are able to report a single value that has been proven to have use in the marketplace from the throughput figure determined by the tests. It is useful to know the actual maximum data rate that a device can support, since even the loss of one frame in a data stream can cause significant delays while waiting for the higher level protocols to time out. In addition to testing the rate of transmission and reception of data, throughput tests should ensure the data was transmitted and received correctly. Despite the importance of Capacity, Establishment, and Throughput tests, the overall stability of the Access Concentrator can not be determined without making sure that the device can recover from a highly unstable state, which is determined by the Session Stability Test.

Session Stability Test Problem Description (Chris)

There are many situations where too much might be asked of an Access Concentrator. The traffic patterns that cause the Access Concentrator to reach such an unstable state need to be defined and measured. After this state is reached, the severity of the state needs to be determined. This is done by measuring the time it takes for the Access Concentrator to become stable again on its own. The longer it takes, the more sever the instability.

The following situations are times when an Access Concentrator may reach an unstable state:

The number of valid requests and the number of current sessions out number the maximum number of sessions.
The sessions that are up and working start to be torn down when they should not be.
The throughput is under 10% of the requested throughput (This assumes that the requests are not unreasonable, and nobody is trying to send beyond the maximum throughput.)
The jitter of the sessions is varying by more then 20%

Many Access Concentrators may pass individual tests in the lab, but in a “real world” traffic setting, the Access Concentrator may reach an unstable state due to a combination of requests. Since the goal of the Access Concentrator, is to be installed, and never be touched again, unless some new configuration is wanted, the Access Concentrator needs to be stable. If an unstable state is reached, the Access Concentrator must become stable in a very short amount of time. Anytime the Access Concentrator is unstable in the field, it costs the buyer money.

APPROACH TO SOLUTION (Chris)
(to Table of Contents)

The general approach to the solution that our group took started out with everybody reading and investigating exactly what an Access Concentrator was. We looked at what makes it an Access Concentrator instead of a router or a switch. Based on this definition we, with the help of our mentors determined the certain key elements that make and Access Concentrator and that should be tested.

The five major elements that we decided on were session capacity, throughput, sessions establishment rate, Session Stability, and Session Forward Stability. We each picked on of the topics and began research and started to write a section of what will be the RFC. Once we met with Kumar to go over the first drafts of the RFCs, we decided that through the research that was done, and based on general session knowledge with PPP, that Session Stability and Session Forward Stability were not really important tests. The work that was done for those tests was scratched, and we replaced the both of them with Access Concentrator Stability as of November 7th, 2002. Chris and Tyler both worked on the Access Concentrator Stability, however Chris toke more of the visual aspects of the test. To make up for Tyler not having anything to put in the RFC draft, he is now responsible for helping Chris with the Access Concentrator Stability, but he also is creating the definitions section of the RFC and possible changes that we might want to add based on functionality that is being advertised by the different providers.

Vendor Specific Functionality (Tyler)

In order to test each vendor's product, test methodologies and algorithms to run these tests have been designed. The following details each individual test and a way to quantitatively compare the vendor's "estimated" specifications to the actual specifications found by running these tests. In order to test the device, the AC is set up to send and receive data in each of a number of sessions. See Appendix A Section 0.3 for more information on Test Setup.

Throughput (Shadi)

Conceptually measuring throughput is a simple task that involves issuing a certain number of requests, counting the number of replies (bits or bytes) received and dividing that number by the time (seconds) it took to complete the test. To simulate real data transmission accurately, we test for both sequential and non-sequential data transmission. Therefore, the size of individual IP packets as well as the size of bursts during non-sequential traffic transmission should be accounted for as set-up parameters.

The software being used to count the number of packets received also evaluates each packet for content and bit count, making sure that no faulty packets have been received.

Also, as with any other test, in order to get a quantitative idea of the robustness of a particular measurement, it is necessary to run the same test several times. Especially with throughput if only one throughput estimate is computed for the entire test, the variations that may occur at time scales shorter than that of the entire test will be hidden.

The algorithm devised allows many packets to be sent in a sequence without synchronizing between the sender and the receiver, resulting in the pseudo code throughput algorithm located in the Appendix. This approach to a solution was taken because the algorithm ensures getting a quantitative idea of the robustness of a particular measurement resulting in accurate calculations of throughput measure on the Access Concentrator device.

Session Capacity (Fritz)

To measure Session Capacity one must determine the maximum number of sessions the Access Concentrator can handle and maintain traffic on. The test methodology must produce the same Session Capacity results for different machines of the same type. This means that the methodology must be repeatable, and all of the parameters must be clearly defined. This does not mean that the Session Capacity determined by this methodology will be the same as that listed in the Manufacturer’s Access Concentrator specifications.

The methodology needs to define an established session. The session establishment is defined in the Appendix A under Session Capacity. Once an established session is connected, the methodology should test the connection. Sending an echo-request to the Access Concentrator and receiving back an echo-reply will do this. The echoes will be sent encapsulated in a LCP packet.

Parameters that must be controlled are keep-alives and network traffic. Typically the server, the ACCESS CONCENTRATOR in this case, sends a keep-alive and creates traffic. Since the ACCESS CONCENTRATOR doesn't need this information for this test, it will be turned off. Also, eliminate all traffic other than what is needed for session establishment, and echo requests to the ACCESS CONCENTRATOR, and echo replies from the ACCESS CONCENTRATOR. Other parameters that need to be varied are the session establishment rate, It can affect Session Capacity and must be fine tuned to achieve maximum capacity.

Session Establishment Rate (Dalat)

To measure the session establishment rate, two tests will be done. One is to send out a finite number of PADI packets at a constant rate. The other test is to send out a number of PADI packets at a bursty rate. As soon as the last PADI is sent, a time-out clock is started. When the time is up, we will measure how many connection acknowledgement packets come back. Based on the number of connection acknowledgement packets that come back comparing to the number of PADI sent, we can tell whether the ACCESS CONCENTRATOR is able to establish the connection requests packets at the rate. By adjusting the rate at which the PADI's are sent, we can determine the maximum rate at which the ACCESS CONCENTRATOR can handle.

The reason we will do two tests is that different types of connection request traffics can have a large impact on the establishment session rate. In the real world, the connection request traffics can be unpredictable. However, most of those traffics can be categorized into two ideal rate patterns, either constant or bursty.

Access Concentrator Stability Tests (Chris)

In order to write a test methodology for Access Concentrator Stability, one must know how to measure instability. This was the first thing that was looked at was what it meant for an Access Concentrator to be in an unstable state. Eventually it was decided that the instability of an Access Concentrator should be measured based on how long it takes for it to reach a stable state. Once an unstable state is reached, this is defined as a state where things are happening that should not be, the Access Concentrator is configured in such a way that it backs off to a known stable state whatever was being done to push the Access Concentrator. The severity of the instability is measured by the time it takes for the Access Concentrator to recover without outside interference. An example of an unstable state is where sessions are being torn down that should not be. Stability would be regained when the Access Concentrator can once again accept and maintain all of its connections and requests.

Once instability and stability were defined and quantified, one must now consider how to get the Access Concentrator into an unstable state. In order to test the stability, one must be able to get into an unstable state. Nothing is perfect, the Access Concentrator does not have infinite memory, or infinite bandwidth at its command. The Access Concentrator, just like all systems have their breaking points. The breaking points need to be found and used to test instability.

These breaking points are defined as:

Sessions are no longer being created.
Sessions are being torn down early.
Session throughputs have been reduced below an acceptable range.
Jitter associated with each of the existing sessions becomes outside of tolerated limits.

For more on how to test the instability and the various ways to get the Access Concentrator unstable, please refer to Appendix A and the section pertaining to Access Concentrator Stability for a more in-depth examination on how to do all of this.

Another situation an Access Concentrator may reach instability is in an internet based traffic model. This is where the traffic being simulated is being simulated as if it were on the internet. Many DUTs may be able to pass through the lab with no problems, however, the internet based traffic is much different then the traffic patterns normally used. One of the key differences is the length of time it takes to run a test. Normally these types of tests run for at least a day. Testing various traffic patterns that may be experienced during the day at the time they might be witnessed.

For more information of this, please refer to Appendix A and the section pertaining to Access Concentrator Stability.

RESULTS (Tyler)
(to Table of Contents)

Our results after following the previous approach are listed in Appendix A. We created a test methodology document that consists of these tests and pseudo-code. The pseudo-code is used as a logical frame work on how to run the tests. Our group did not get to actually write the scripts for the test tools themselves. Our time was spend defining and fine tuning this methodology document. To verify that these results and assumptions that are made are valid, we ran the test by hand instead of automating them. These results are in the Appendix and the Verification section of this document.

One of the tasks was to research different vendor specific functionality and make recommendations on what tests might be useful to create in the future. Many of the vendors give a quantity for attributes of their access concentrator such as net.com's claim that their product (SCREAM 100) has a "maximum throughput of 5Gb per second per Network Data Plane (NDP)" and "can handle up to 20 Gb per second of bursty traffic per NDP" (net.com). Depending on which devices are available for testing, many of these company's products will be tested using the following algorithms in order to compare actual findings to vendor advertised quantities.

The Access Concentrators to be studied included:

The Redback SMS 1800 and SMS 10000
The Nortel Shasta 5000
net.com's SCREAM 50, 100, and 200
Cisco's 10K and 6400
Cosine IPSX
and Juniper's AC

All of the tests that were performed by our group were reasonable and made good test items. Testing device stability is somewhat more advanced and difficult to perform. The reasons for this is that it is impossible to simulate "real-life" traffic. Different aspects of the DUT could be put to test in an attempt to cause an unstable state, however, when placed in a real world situation the DUT is put through a much wider variety of traffic than can be reasonably tested.

Other tests that should be included could be issues such as the DUT's security features and it's routing protocols. Adding new IP services such as voice over IP , virtual private networks , and IP Multicast create a considerable strain on existing resources and network designs. Often, users make significant trade-offs with throughput, capacity, or service mix. These choices can lower expected operating margins and create inconsistent service characteristics. This may be difficult to test but again would provide the most information as to how the DUT would react in a real world situation.

VERIFICATION AND VALIDATION (Fritz)
(to Table of Contents)

This part of the project will determine the validity and accuracy of our methodology. There will be two parts to this. First, the mentors will review our drafts and discuss changes and modifications needed. Once they have approved our drafts, we will do some testing on Access Concentrators using this methodology. The tests will help validate the accuracy of our methodology or determine other parameters or procedures need to be defined or followed.

Vendor Specific Functionality (Tyler)

Through the use of Spirent Com’s Adtech Network Tool v4.31 and the algorithms designed for each attribute to test, the actual specifications will be found. These specifications will be compared individually to the specifications advertised by the vendor for the DUT. The comparisons and any conclusions drawn from this comparison will be reported when completed.

Session Capacity Tests (Fritz)

To make sure the methodology is valid, several tests will be run and repeated to check for accuracy and consistency. As of the writing of this document, the only Access Concentrator that can be tested is the Cisco7200VXR using the Adtech software. Tests will look for:

The maximum Session Capacity established and maintained as specified by methodology.
Effects of changing the session establishment rate.
Testing variables or parameters either not defined or incomplete affecting capacity.
Using results of this test to improve methodology

Once the results are in from this test, they can be compared with the assumptions made by the test. The comparisons can then be used to make recommendations or modifications to the methodology for future testing.

Session Establishment Rate Tests (Dalat)

To make sure that the establishment rate part of the methodology is valid, we will need to run a sanity test on many different Access Concentrators. Due to the current unavailability of the different AC devices, we will just test our results on a Cisco 7200VXR by using the program Adtech Network Tool v4.31. There are several parameters we will examine:

Number of PADI packets sent.
The amount of time sending out the PADI packets.
Number of connection acknowledgement packets.
The time it took to set up each connection.
The number of packets dropped.

We will run the test several times to make sure that the result we get are coherent. In addition, the results will be in the form of graphs. There will be two separate graphs: One for constant back to back PADI rate and the other one for the bursty PADI rate. The x-axis is the Establishment Request Rate, and the y-axis is the corresponding percentage of successful established sessions. The point where the Establishment Request Rate is highest and the percentage of successful connection is still 100% is the maximum establishment session rate.

The result of the test will be done later and it will be in the Appendices section.

Throughput Tests (Shadi)

In order to verify that the throughput test methodology that was defined for Access Concentrator is correct, the test should be run, using Spirent Com’s Adtech Network Tool v4.31, on many different Access Concentrator brands including, Redback SMS 1800 and SMS 10000, Nortel Shasta 5000, net.com SCREAM 50/100/200, Cisco 10K and 6400, Cosine IPSX, and Juniper. Exactly which brands will be tested depends on machine availability. More importantly, when running the test methodology the following key points of interest are studied to determine if tests are valid:

Successful generation of a pre-determined data stream in accordance with specific application’s encoding rules.

At the receiver end, protocol analysis software can correctly assess the throughput.
Software correctly counts the number of packets received.
Evaluates each packet for content and bit count.
Returns error if defects are found in packets received.

Also, modifications to the algorithm that simulate the effects of adverse operating conditions such as signal jitter, noise, delay, and amplitude changes can be programmed to further evaluate and assess the system under study.

Access Concentrator Stability Tests (Chris)

In order to verify that the methodology that was defined for Access Concentrator stability was correct, we need to test out our results on many different Access Concentrators. In order to test stability, first we tested the methodology on a Cisco 7200VXR. Time permitting we will be testing the methods on the Cisco 10K and other Access Concentrators that Spirent has in their lab.

When running the methodology the following are things that we looked for to determine that everything was going right:

Run a sanity test.
- Created 10 sessions.
- Ran traffic through these 10 sessions at 5% line rate.
  - Looked at the number of sessions created and the memory associated with each.
  - Looked at the keep alive responses.
  - Looked at the packets on the other side of the throughput sanity test.
- Teardown all of the sessions and wait a few minutes for memory to free up.
Started to run the specific methodology.
- Viewed the packets going out of the test device.
- Viewed the packets received on the other side.
Collect results from the test.
Re-run the test the recommended number of times.
Created graphs based on data.
Analyzed these graphs based on expect data.
Reported any differences to be looked at later.

The methodology and the expected results are in the Access Concentrator Stability section of Appendix A.

Results

Session Capacity Test Results (Fritz)
[This part will be here soon !]

Session Establishment Rate Test Results (Dalat)
[This part will be here soon !]

Throughput Test Results (Shadi)
[This part will be here soon !]

Access Concentrator Stability Test Results (Chris)

Our group is current in the process of going through the data collected. Explanation will be here soon, along with the results in the Appendix.

SCHEDULE (Dalat)
(to Table of Contents)

REFERENCES (Dalat)
(to Table of Contents)

Bradner, S. "RFC 1242: Benchmarking Terminology for Network Interconnection Devices." Network Working Group. July 1991.

Bradner, S. "RFC2026: The Internet Standards Process -- Revision 3." Network Working Group. October 1996.

Bradner, S. and McQuaid, J. "RFC 2544: Benchmarking Methodology for Network Interconnect Devices." Network Working Group. March 1999.

Gross, G., Kaycee, M., Lin, A., Malis, A., and Stephens, J. "RFC2364: PPP Over AAL5." Network Working Group. July 1998.

"Journal of Internet Test Methodologies." RouterTester Solution. Agilent Technologies. November 2002.

Mamakos, L., Lidl, K., Evarts, J., Carrel, D., Simone, D., and Wheeler, R. "RFC 2516: A Method for Transmitting PPP Over Ethernet (PPPoE)." Network Working Group. February 1999.

Mandeville, R. "RFC 2285: Benchmarking Terminology for LAN Switching Devices." Network Working Group. February 1998.

Mandeville, R. and Perser, J. "RFC 2289: Benchmarking Methodology for LAN Switching Devices." Network Working Group. August 2000.

Simpson, W. "RFC 1661: The Point-to-Point Protocol (PPP)." Network Working Group. July 1994.

APPENDICES (Dalat and Chris)
(to Table of Contents)

Appendix A

Access Concentrator Test Methodology

0 Overview (Tyler)

0.1 Introduction

This document provides methodologies for the performance benchmarking of access concentrators. It provides methodologies in four areas: subscriber session capacity, session establishment rate, session throughput, and device stability.

In addition to defining the tests, this document also describes specific formats for reporting the results of the tests.

0.2 Definitions

The following section includes definitions of terms used throughout this document.

0.2.1 Definition Format

Definition: Specific definition of term

Discussion: A brief discussion about the term, its application and any restrictions on measurement procedures.

Measurement Units: The units used to report measurements of this term, if applicable.

0.2.2 Subscriber Session Capacity

Definition: The maximum number of PPPoE and PPPoEoA sessions a DUT can
establish and maintain.

Discussion: Session capacity is a function of memory and should be consistent from machine to machine.

Measurement Units: Number of sessions

0.2.3 Throughput

Definition: The maximum rate at which none of the offered frames are dropped by the device under test.

Discussion: The throughput will be tested with different numbers of sessions, packet lengths, and different sized packet bursts. Data will be sent to and from the DUT during all sessions.

Measurement Units: Gb per second as well as packets per second.

0.2.4 Session Establishment Rate

Definition: A session establishment rate is the rate at which a DUT can establish connections without dropping any connection request packet as well as maintaining all current sessions.

Discussion: For these tests this is the maximum rate at which PPP session establishment requests can be sent to the DUT resulting in successful sessions.

Measurement Units: Session Capacity over time

0.2.5 Device Stability

Definition: Device’s ability to handle traffic patterns. Some will cause the DUT to reach an unstable state meaning that this is beyond the threshold of the DUT’s stability.

Discussion: There are many ways to test the stability of an access concentrator. The following RFC uses five test cases to test stability.

Measurement Units: Time

0.3 Test Setup

Setting-up tests described below involves a DUT and a tester with both transmitting and receiving ports as shown in the figure below. Connection requests originate from the sending ports of the tester and are destined to the receiving ports of the DUT. Connection acknowledgements and replies are sent from the sending ports of the DUT back to the tester.

0.3.1 Configuration being simulated (logical)

0.3.2 Configuration in the lab (physical)

1 Session Capacity (Fritz)

1.1 Objective

Determine the maximum number of PPPoE and PPPoEoA sessions a DUT can establish and maintain.

1.2 Definition

Find the maximum number of sessions an Access Concentrator can establish and test the connections to determine if it can maintain these connections. The methodology in this draft should produce repeatable and consistent results on different machines of the same type since session capacity is a function of memory. In this experiment, traffic MUST be
controlled. You will also have to adjust the session establishment rate, to find the optimal rate. This will help with benchmark testing because unnecessary traffic that may affect results. The rate can slow the processor or fill the memory buffer and affect results.

1.3 Discussion

PPPoE session establishment details are specified in RFC 2516. This sets up a client-server type relationship between the host and the Access Concentrator. Session establishment begins with the discovery stage, where a PADI packet is broadcast to the Access Concentrator requesting service. The Access Concentrator responds to the PADI with several different PADO packets containing session information. The host then chooses one PADO packet and sends a PADR packet. The Access Concentrator generates a unique Session-ID once it receives the PADR packet and responds with a PADS packet. This brings us to the next phase, the optional authentication phase.

The authentication protocol details are specified in RFC 1661. A Config-Req message is sent from another peer through the Access Concentrator requesting the client identify itself. The client then responds to with a Config-Ack that contains the identification. The other peer looks at the Config-Ack and decides whether it should grant the permission or not. Once authenticated, the session is established and this is what we will define as an established session.

1.4 Setup Parameters and Assumptions

1.4.1 Session Establishment Rate

Establish sessions at different rates to see if this has any effect on the maximum number of sessions that can be established. First, establish a set number of connections one at a time in a specified amount of time. Second, establish a set number of connections in several bursts in a specified amount of time. Start with small bursts, about 5 per second and increment up to see if there are changes in capacity.

1.4.2 Established Connection Testing

Test currently established connections to see if the Access Concentrator can maintain all of the established connections. This test will be performed in conjunction with the establishment test. Sending Echo Request and Echo Reply messages through LCP packets can test this. Typically, server sends Echo Request, in this case that would be the DUT. Since the keep-alives are off and the Access Concentrator will not send Echo Request, the test machine will have to send an Echo Request and the DUT will send an Echo Reply.

1.4.3 Traffic

The keep-alives will be turned off, so the DUT will not send echo-requests and generate this traffic. The line traffic will be controlled, only traffic used to establish sessions and check session connection will be allowed. Traffic and keep-alives can affect results so these parameters MUST be controlled for the results to be repeatable.

1.4.4 Physical Hardware

It is important to note the line rate and trunk rate when determining Session Capacity. In most cases it is difficult to achieve line rate, and will assume the access port will not achieve line rate. Also, take note of the trunk rate and be sure this will not be a limiting parameter. It is assumed that you can not exceed the trunk rate which would affect the results

1.5 Procedure

1.5.1 Session Capacity Variables

These are the initial variables; you may want to start the low Number and Number variables at a higher number if your DUT Specifications are significantly higher. Set the high Number variable to a number at or above the manufacturer's specification or at a number you believe session capacity cannot achieve. The rate is an initial starting point that you will adjust after you have found an initial value for the session capacity. You will want to adjust this to see if changes occur in session capacity.

highNumber = *You Specify
lowNumber = 1000 or specify appropriate higher number
Rate = 5 sessions/sec
Number = 2000

1.5.2 Session Capacity Algorithm

When performing this test, each time you start testing the Session Capacity test at a different level, be sure to check between tests and validate that memory is free since this can affect results.

While ( highNumber != lowNumber )

test = null
establishSession(Number)

if ( ALL sessions are established )
test = true
else
test = false

if ( Test = = false)
highNumber = Number
Number = (highNumber-lowNumber)/2 + lowNumber
establishSession( Number )

if ( Test = = True)
lowNumber = Number
Number = (highNumber-lowNumber)/2 + lowNumber establishSession(Number)

Once this result is determined, increase the session establishment rate by 5 to see if there is an increase in session capacity. Repeat this several times and determine the rate that yields the highest session capacity.

1.6 Measurements

Depending on rate, the results can change. The rate can be too fast and the processor may reach capacity causing delays and retries that affect the results. Also, between tests, if the memory is not free it will have a negative affect on the results since capacity is determined by memory.

Insert more here about specific measurements

1.7 Reporting Format

Insert text here

2 Throughput (Shadi)

2.1 Objective

To determine the maximum rate at which none of the offered frames are dropped by the DUT.

2.2 Definition

When speaking in terms of computer technology, throughput can be informally defined as the amount of work a computer can do in a given amount of time. In terms of data transmission, throughput is the amount of data successfully transmitted from one location to another in a given amount of time. More formally defined, throughput is the maximum rate at which the number of frames transmitted to the DUT is equal to the number of frames received by the DUT.

2.3 Discussion

To measure the fastest rate at which frames can be transmitted successfully, traffic generator device should send traffic from one or more of the tester's source ports to the DUT. The number of ports the traffic is sent to and from depends on the number of sessions already established (this assumption is made before the throughput test is run). If the number of frames received is less than the number of frames transmitted, then packet loss has occurred, in which case, the test is repeated using a reduced offered load. If the number of frames received is equal to the number of frames transmitted, then the test is repeated with an increased offered load. The test is repeated with varying offered loads until a maximum frame rate with no loss is determined.

2.4 Setup Parameters and Assumptions

•Frame Size-->

The following IP Frame sizes should be used on PPPoE(Include the maximum and minimum frame sizes permitted by the Ehternet standard and a selection of sizes between the two extreams: 64,128, 256, 512, 1024, 1492 bytes

•Direction of Traffic-->

Can be downstream or upstream. Each direction will be handled by performing the reverse operations of eachother.

•Packet Count-->

This parameter will help determine the duration of the test. Packet count should be a multiple of the number of user sessions established so that the packets are distributed evenly between user sessions.

•Number of Sessions-->

The number of established sessions. Throughput could possible be affected by this parmeter, for example, a large number of sessions might cause the throughput rate to decrease.

2.5 Procedure

•Create traffic with N streams as uni-directional flows from one or more of the test device's source ports, through the DUT

•Transmit the initial load of frames (this value is half of what the max load is claimed to be) at a specific rate through the DUT from all sources to all destinations. Once an iteration of transmission has been completed, measure the number of received frames. It is recommended that Echo_Request packets are sent between iterations in order to confirm that the sessions are still up.

•If the count of offered frames is greater than the count of the received frames, then fewer frames are received than were transmitted. The rate of the offered stream is reduced lower than the failed rate by half the difference between previous load value and current load value.

• The third trial and subsequent trials make use of the same algorithm to continue to increase or decrease the offered load.

•If all the transmitted frames are received by the receiving port, no further trials are attempted and maximum frame rate is recorded as throughput.

Each trial iteration results in a record of the total number of packets transmitted and received. For the successful and last trial, the packet rate and percentage of maximum rate for a given packet size is recorded.

2.5.1 Algorithm

CONSTANT

Test_Duration = Time the test is to run; determines the max number of frames sent.

Packet_Size = Certain size of packets.

Resolution_percent = The percentage of traffic within which the acceptable

loss percentage is still met.

Maximum_Loss = Test can be adapted to find the optimal throughput rate with an acceptable packet loss ratio defined by the user.

2.6 Measurements

Fill this in

2.7 Reporting Format

The results of the throughput test SHOULD be reported in graph format where the x coordinate SHOULD be the frame size, and the y coordinate SHOULD be the frame rate. There SHOULD be at least two lines, actual and theoretical on the graph. Text accompanying the graph SHOULD indicate the protocol, data stream format, and type of media used in the tests.

3 Session Establishment Rate (Dalat)

3.1 Objective

To determine the maximum establishment rate of a DUT.

3.2 Definition

The maximum session establishment rate is the maximum rate at which the PADI's are sent to the DUT and still results in 100% successful sessions.

3.3 Discussion

To establish a PPP connection between a user device and an Access Concentrator, the two devices have to go through four states: discovery stage, data link layer setup, authentication, and network layer setup. In the discovery stage, a user device starts with broadcasting an initiation packet (PADI) to look for an Access Concentrator (AC). The AC then replies using an offer packet (PADO). Upon receiving the PADO, the user device and the AC communicate to provide the device a session ID and a MAC address. After this stage, the two peers proceed to the data link layer setup. Here, the two peers exchanges LCP config-req, config-nak, config-rej, and config-ack packets to negotiate data link layer options (see RFC 1661 for more detailed information). Once the data link layer is setup, the peers have to go through an authentication process. In this process, the AC verifies to see if the other peer is a valid user or not. Finally, when the authentication is successful, the two peers exchange IPCP packets to set up the network layer. Once this network layer is set up, the peers can start transferring data.

3.4 Setup Parameters and Assumptions

The following parameters affect the outcome of the test and they must be defined.

N - Is the number of PADI packets to be sent to the DUT. This number should be initialized to a known number that the DUT is able to handle and should be smaller than the maximum number of connections the DUT is claimed to be able to establish in one second. For this test, this N is initialized to 5.

Time-Out - This is the time length the tester should wait before it assumes that the PADI is dropped by the DUT. This time-out should be set to one minute.

Sending Time - This is the total transmission time spent to send all the N number of PADI's. This number should be set to 1 second.

Request Rate - This is the rate at which the PADI's are sent. A certain number of PADI's should be sent within a fixed period of time. The PADI's may be sent in a constant or in a bursty fashion.

Buffer - The DUT should have a finite amount of buffer to store the incoming request packets. In general, the more memory is assigned to the buffer, the less memory is available for the CPU. For this test, the buffer space is set to a fixed value, i.e. 5 kb.

3.5 Procedure

Two types of tests will be performed. One is to determine the maximum session establishment rate when the PADI's are sent at a constant rate. The other one is to determine the maximum session establishment rate when the PADI's are sent in a bursty fashion.

3.5.1 Measuring Session Establishment Rate with Constant PADI's

From the test port, initiate the sessions by sending out an N number of PADI packets at a constant rate for one second.

As soon as the last PADI packet is sent, start the Time-Out timer.

When the timer expires, record the number of successful established sessions by counting the number of of IPCP Config-Ack packets received from the DUT.

If the number of established sessions is equal to N, increase N by 5 and restart the test.

If the number of established sessions is less than N, decrease N by 1 and restart the test. Keep decreasing N by 1 and restart the test until the number of established session is equal to N. The final N is the maximum number of connections the DUT can establish in one second.

3.5.2 Measuring Session Establishment Rate with Busty PADI's

From the test port, initiate the sessions by sending out an N number of PADI packets for one second in a bursty fashion. There should be 5 packets in each burst. The space between each burst is 1 microsecond.

As soon as the last PADI packet is sent, start the Time-Out timer.

When the timer expires, record the number of successful established sessions by counting the number of of IPCP Config-Ack packets received from the DUT.

If the number of established sessions is equal to N, increase N by 5 and restart the test.

3.6 Measurements

Establishment Request Rate - This is the N number of PADI sent in 1 sec.

Percentage of Successful Sessions - This number is obtained by dividing the number of successful sessions by the number of PADI sent.

3.7 Reporting Format

The format should be in the form of graphs. There should be two separate graphs: One for constant back to back PADI rate and the other one for the bursty PADI rate. The x-axis is the Establishment Request Rate, and the y-axis is the corresponding percentage of successful established sessions. The point where the Establishment Request Rate is highest and the percentage of successful connection is still 100% is the maximum establishment session rate.

4 Access Concentrator Stability (Chris)

4.1 Objective

The objective of the following tests is to measures the instability of the DUT with various traffic patterns.

After this state is reached, the test measures the severity of the state by the time it takes for the DUT to become stable again on its own. The longer it takes, the more sever the instability.

4.2 Definition

The instability of an Access Concentrator is measured based on how long it takes for it to reach a stable state. Once an unstable state is reached, this is defined as a state where things are happening that should not be, the Access Concentrator is configured in such a way that it backs off to a known stable state whatever was being done to push the Access Concentrator. The severity of the instability is measured by the time it takes for the Access Concentrator to recover without outside interference.

4.3 Discussion

There are 5 test cases to run to test the stability of the DUT. Four of the five test cases test different forms of instability. The fifth test case tests the DUT's stability with an internet/"real-world" traffic model.

4.3.1 Types of Traffic That Will Be Used

4.3.1.1 LCP Traffic

New Session Establishments will be used to create sessions and also when testing the results of invalid session requests. These session establishment requests can be valid, or invalid based on the parameters selected. See Appendix A for more details.

Session Teardowns will be used to teardown sessions and also when testing the results of invalid session teardown requests. These session teardown requests can be valid or invalid based on the parameter, bursty, steady state requests, or be due to keep-alive timeouts. See Appendix A for more details.

Re-Negotiation of Established Sessions will be used to test how the Access Concentrator handles re-negotiation under different circumstances. These session Configure-Requests can be valid or invalid based on the parameters, bursty, or steady state requests.

4.3.1.2 NCP Traffic

To test how the sessions handle traffic flowing on them run varying amounts of valid and invalid NCP traffic on established sessions. Starting at the minimum frame size up to the maximum frame size. Varying the throughput on all established sessions upstream and downstream.

4.4 Setup Parameters and Assumptions

Blah

4.5 Procedure

4.5.1 Test case 1

4.5.1.1 Objective

This test measures the unstable state of the DUT where sessions are no longer being created.

4.5.1.2 Definition

Once an unstable state is reached, the test measures the severity of the state by the time it takes for the DUT to become stable again on its own. The longer it takes, the more sever the instability.

4.5.1.3 Discussion

When a DUT reaches a state where the number of valid requests and the number of current sessions out number the maximum number of sessions, the behavior of the Access Concentrator is unknown, and therefore unstable since some of these connections will have to be denied.

4.5.1.4 Setup Parameters and Assumptions

The following parameters MUST be defined with the listed considerations:

v Packet Size: The packets being sent are LCP packets, so the size would reflect such. Suggested sizes to reach are listed in RFC 2544 and Appendix B.

v Maximum Number of Sessions: The maximum number of sessions that the DUT can handle is required for this test. If it is not known run the Session Capacity test above, and wait a cool off period so that all of the memory is freed before running the test.

v Maximum Rate To Create Sessions: The maximum number of session rate that the DUT can handle is required for this test. If it is not known run the Session Establishment Rate test above, and wait a cool off period so that all of the memory is freed before running the test.

v Number of Sessions not being created: This will fluctuate as time goes on. It should start at zero, once it gets to denoted limit before the back off is performed and the value should go back to zero.

v Test Duration: The test should run until instability is reached and recovered from. We assume that instability will eventually be reached and recovered from.

Other packet options for creating sessions are in Appendix A.

4.5.1.5 Procedure

Goal - Create Sessions so fast that some the sessions no longer get created that are requested

Run the Max Establishment Script
Once the limit is found, back off to the maximum value that allowed traffic through.
Start time, at the first back off.
Stop time, when all sessions are created and torn down as thought.

4.5.1.5.1 Script

VARIABLES
VALUES [2] = {MAX_SESSIONS, MAX_RATE}

BEGIN
VALUES = MAX_ESTABLISHMENT_SCRIPT(NO_COOL_DOWN)

WHILE(SESSIONS_ARE_NOT_BEING_CREATED)
BEGIN
TEARDOWN so the number does not exceed MAX_SESSIONS
CREATE sessions at MAX_RATE

TIME = TIME+1;
END
END

4.5.1.6 Measurements

v Total Number of Sessions Created

v Maximum Rate To Create Sessions

v Number of Drops Before Back Off

v Time To Get Back To A Stable State

4.5.1.7 Reporting Format

Record the time it takes to get back to a steady state. You SHOULD run this test 20 times to get a good average of time. Plot the session creation rate vs time. You MAY also run this test and plot the number of sessions created vs. time (getting rid of max limit).

The graphs SHOULD look something like this:

NOTE: This is an approximation

The ridge in the graph shows where stability was lost. We backed off to the max right before that, so the rate should go back up to this max, and be a straight line the rest of the time. The difference in the x-axis is the recovery time.

4.5.2 Test Case 2

4.5.2.1 Objective

This test measures the unstable state of the DUT where sessions are being torn down early.

4.5.2.2 Definition

When an unstable state is reached, the test measures the severity of the state by the time it takes for the DUT to become stable again on its own. The longer it takes, the more sever the instability.

4.5.2.3 Discussion

When a DUT reaches a state where the sessions that are up and working start to be torn down when they should not be, the behavior of the Access Concentrator is unknown, and therefore unstable since some of these sessions are in use and now need recreated.

4.5.2.4 Setup Parameters and Assumptions

The following parameters MUST be defined with the listed considerations:

v Packet Size: The packets being sent are NCP packets, so the size would reflect such. Suggested sizes to reach are listed in RFC 2544 and Appendix B.

v Line Rate: The speed if packets were sent back to back on the line ignoring errors. This is a function of the media being used.

v Number of Sessions Dropped: This will fluctuate as time goes on. It should start at zero, once it gets to denoted limit before the back off is performed and the value should go back to zero.

v Test Duration: The test should run until instability is reached and recovered from. We assume that instability will eventually be reached and recovered from.

Other packet options for creating sessions are in Appendix A.

4.5.2.5 Procedure

Goal - Bog down the line, so the keep-alives get lost, and sessions start to get torn down early.

Create 10% of the maximum number of sessions on the DUT
Start with a small amount of traffic on these sessions
Increase traffic as time goes on until sessions start to get dropped
(Restart/Create a new session for each one that gets dropped)
Back off to the point right before this happened
Start timer
Stop timer when the sessions are not dropping

4.5.2.5.1 Script

VARIABLES
NUMBER_OF_SESSIONS = 10% MAX SESSIONS (defined earlier)
CURRENT_TRAFFIC = 5% LINE RATE
BEGIN
UNTIL(SESSIONS_ARE_DROPPED)
BEGIN
CURRENT_TRAFFIC = CURRENT_TRAFFIC+ 5% LINE RATE
END
CURRENT_TRAFFIC = CURRENT_TRAFFIC - 5% LINE RATE
WHILE(SESSIONS_ARE_DROPPED)
BEGIN
CREATE session to equal NUMBER_OF_SESSIONS
TIME = TIME+1;
END
END

4.5.2.6 Measurements

v Traffic rate

v Maximum Number of sessions

v Number of Sessions Dropped

v Burstiness

v Time To Get Back To A Stable State

4.5.2.7 Reporting Format

Record the time it takes to get back to a steady state. You SHOULD run this test 20 times to get a good average of time. Plot the traffic rate vs time. You MAY also run this test and plot burstiness vs. time..

The graphs SHOULD look something like this:

NOTE: This is an approximation

4.5.3 Test case 3

4.5.3.1 Objective

This test measures the unstable state of the DUT where session throughputs have been reduced below an acceptable range.

4.5.3.2 Definition

When an unstable state is reached, the test measures the severity of the state by the time it takes for the DUT to become stable again on its own. The longer it takes, the more sever the instability.

4.5.3.3 Discussion

When a DUT reaches a state where the throughput is under 10% of the requested throughput, the behavior of the Access Concentrator is unknown, and therefore unstable since some of these connections will have to be denied. This assumes that the requests are not unreasonable, and nobody is trying to send beyond the maximum throughput.

4.5.3.4 Setup Parameters and Assumptions

The following parameters MUST be defined with the listed considerations:

v Packet Size: The packets being sent are NCP packets, so the size would reflect such. Suggested sizes to reach are listed in RFC 2544 and Appendix B.

v Line Rate: The speed if packets were sent back to back on the line ignoring errors. This is a function of the media being used.

v Maximum Throughput: The max speed packets can be sent at before drops start to occur.

v Number of Sessions Dropped: This will fluctuate as time goes on. It should start at zero, once it gets to denoted limit before the back off is performed and the value should go back to zero.

v Test Duration: The test should run until instability is reached and recovered from. We assume that instability will eventually be reached and recovered from.

Other packet options for creating sessions are in Appendix A.

4.5.3.5 Procedure

Create 50% of MAX Sessions
Send invalid requests, and so valid requests to the DUT to be processed
Send traffic throughout at a rate of 5% max throughput
Increase the throughput until, the DUT can not handle all of the data on the line, and the throughput drops down dramatically.
Back off the throughput to the MAX accepted value.
State timer
Stop timer when the throughput is within acceptable ranges again

4.5.3.5.1 Script

VARIABLES
NUMBER_OF_SESSIONS = 50% MAX SESSIONS (defined earlier)
CURRENT_TRAFFIC = 5% MAX THROUGHPUT
BEGIN
UNTIL(THROUGHOUT < 50% of CURRENT_TRAFFIC)
BEGIN
CURRENT_TRAFFIC = CURRENT_TRAFFIC+ 5% LINE RATE
END
CURRENT_TRAFFIC = CURRENT_TRAFFIC - 5% LINE RATE
WHILE(THROUGHOUT < 50% of CURRENT_TRAFFIC)
BEGIN
TIME = TIME+1;
END
END

4.5.3.6 Measurements

v Traffic rate

v Burstiness

v Max throughput

v Current throughput

v Time To Get Back To A Stable State

4.5.3.7 Reporting Format

Record the time it takes to get back to a steady state. You SHOULD run this test 20 times to get a good average of time. Plot the throughput vs time.

The graphs SHOULD look something like this:

NOTE: This is an approximation

4.5.4 Test case 4

4.5.4.1 Objective

This test measures the unstable state of the DUT where jitter associated with each of the existing sessions becomes outside of tolerated limits.

4.5.4.2 Definition

After this state is reached, the test measures the severity of the state by the time it takes for the DUT to become stable again on its own. The longer it takes, the more sever the instability.

4.5.4.3 Discussion

When a DUT reaches a state where the jitter of the sessions is varying by more then X%, the behavior of the Access Concentrator is unknown, and therefore unstable since some of these connections will have to be denied.

4.5.4.4 Setup Parameters and Assumptions

Same as throughput

4.5.4.5 Procedure

Same as throughput, except this time measure jitter instead of throughput

4.5.4.5.1 Script

Same as throughput, except this time measure jitter instead of throughput

4.5.4.6 Measurements

v Traffic rate

v Burstiness

v Max throughput

v Current throughput

v Amount of jitter

v Time To Get Back To A Stable State

4.5.4.7 Reporting Format

The results should look very similar to the throughput test case.

4.5.5 Test case 5

4.5.5.1 Objective

This test measures the stability of the DUT under "real world" traffic circumstances. The DUT should never reach an unstable state, but if it does, the severity should be very low.

4.5.5.2 Definition

“Real World” traffic circumstances are when sessions are constantly being torn down and being initialized. There are a few peek times in the day as can be witnessed on the graph further down.

4.5.5.3 Discussion

Many DUTs can pass individual tests in the lab, but in a "real world" traffic setting, the DUT may reach an unstable state. Since the goal of the DUT, is to be installed, and never be touched again, unless some new configuration is wanted, the DUT needs to be stable. If an unstable state is reached, the DUT MUST has to become stable in a very short amount of time. Anytime the DUT is unstable in the field, it costs the buyer money.

4.5.5.4 Setup Parameters and Assumptions

Goal is to mimic Internet traffic. Internet traffic, from the perspective of sessions, and the DUT, will look like this:

The x-axis is time of day starting at midnight and ending at midnight.

4.5.5.5 Procedure

The general procedure is to allow 10% of the links to time out randomly because of no, purposeful, keep-alives. This is simulating users logging off, but the session not getting torn down until the keep-alive response is missed for a while.

We will traverse time, creating sessions and tearing them down based on the given chart. The traffic running on these sessions will be a variety of small queries like e-mail to big downloads. The size of the traffic will be randomly generated.

Run the test for 24 hrs, to see at what points the DUT becomes unstable and/or unable to recover.

Record the traffic patterns being sent at the time of instability so that more research can be done later on this.

4.5.5.5.1 Scripts

Each of the traffic patterns will have its own script running in parallel. The scripts will share the clock, and all the actions will be based on the time of day the system thinks it is.

EVERY MINUTE:
Do not reply to 10% of the keep-alives
Create or Destroy number of sessions required to get to the current level for this time
Renegotiate 10% of the active sessions parameters
Send random mix of traffic over the sessions
Varying Sizes
Valid/Invalid Protocols
Send random mix of incorrect LCP traffic to the DUT
Varying Codes and options

4.5.5.6 Variables

v Number of Connections

v Number of Connection retries

v Burstiness of source

v Throughput on sessions

v Jitter through the sessions

v Time that stability was lost

v Time stability was regained

4.5.5.7 Reporting Format

Plot Variables vs. time for the entire 24 hr period. You MAY wish to run this test case 7 times back to back to simulate an entire weeks worth of traffic.

Appendix A (Chris)

A Types of Traffic (Specific)

A.1 LCP Traffic

New Session Establishments –
To configure a new sessions, the host will send out a Configure-Request Packet. Before this is done, the host needs to find out some information about the server. This is called the discovery stage of PPP. PPP will send out a PADI waiting for a PADO response. After the response it will send out a PADR packet waiting for a PADS packet to respond.

To communicate you need to very important things, a MAC Address (for Ethernet), and you need a Session ID number. These packets are to get just those. When sending bad data, you can look at changing these two fields.

The following are the Discovery Stage packet formats and Options:

The following are PPP session stage packets and options:

Parts of the Packet:

Code -Which LCP Packet is being sent:

Identifier - Must be changed when the data inside of request changes, or a response with that number is received. Used to match requests with replies.

Options - List of Configurations option:

Session Teardowns -
There are three ways to teardown a session:

1. If you the host does not respond to the keep alive message

2. If a PADT is sent (Discovery Stage)

3. If a Terminate Request and Terminate Ack are sent/received

Everything is the same as in Session Establishment, except there are no options, and the codes are as follows:

Re-Negotiation of Established Sessions -
This is the same as the Session Establishment PPP session stage. Send the following LCP packets with the appropriate codes and options to re-negotiate different parameters.

A.2 NCP Traffic:

Normal Network Layer Traffic -