Calculating statistical confidence levels for error-probability estimates

How can you be sure that your design meets the required reliability specs? These steps will steer you in the right direction.

JUSTIN REDD, Maxim

It is important to transmit enough bits through the system to ensure that P'(e) is a reasonable approximation of the actual P(e) (i.e., the value to be obtained if the test could proceed for an infinite amount of time). For a reasonable limit on test time, therefore, we must know the minimum number of bits that yields a statistically valid test.

In most cases, it is sufficient to prove that P(e) is less than some upper limit (versus proving its exact value). For example, the P(e) required by many telecommunications systems is 10^-10 or better (meaning that it must be less than or equal to the upper limit of 10^-10). The primary advantage of associating a confidence level with an upper limit on P(e) is that test time can be reduced to the minimum possible while maintaining any selected level of confidence in the test results.

where P[ ] indicates probability and CL is the confidence level. Because confidence level is a probability by definition, the possible values range from 0% to 100%.

After computing the confidence level, we can say we have CL percent confidence that the P(e) is better than g. As another interpretation, if we repeat the bit-error test many times and recompute P'(e) = e/n for each test period, we expect P'(e) to be better than g for CL percent of the measurements.

Equation 3 gives the probability that k events (i.e., bit errors) occur in n trials (i.e., n bits transmitted), where p is the probability of event occurrence in a single trial (i.e., a bit error) and q is the probability that the event does not occur in a single trial (i.e., no bit error). Note that the binomial distribution models events that have two possible outcomes, such as success/failure, heads/tails, or error/no error. Thus, p + q = 1.

Graphical representations of equations 3 and 4, along with some of their properties, are summarized in Figure 1.

We can then use equation 4 to determine the probability P(e>N ^ph), based on ph, that more than N bit errors will occur when n total bits are transmitted. If, during actual testing, less than N bit errors occur (even though P(e>N ^ph) is high), then one of two conclusions can be made: (a) We just got lucky or (b) the actual value of p is less than ph. If we repeat the test over and over and continue to measure less than N bit errors, then we become more and more confident in conclusion (b).

The quantity P(e>N ^ph) defines our level of confidence in conclusion (b). If ph = p, we have a high probability of detecting more bit errors than N. When we measure less than N errors, we conclude that p is probably less than ph, and we define as the confidence level this probability that our conclusion is correct. In other words, we are CL% confident that P(e) (the actual probability of bit error) is less than ph.

where CL is the confidence level in terms of percent.

As noted above, when using the confidence-level method we generally choose a hypothetical value of p (^ph) along with a desired confidence level (CL), and then solve equation 5 to determine how many bits (n) we must transmit through the system (with n or fewer errors) to prove our hypothesis. Solving for n and n can prove difficult unless approximations are made.

Note that the second term in equation 8 equals zero for N = 0, and for that case, the equation is easily solved. Solutions to equation 8 are more difficult for N >0, but they can be obtained empirically using a computer. We can now determine the total number of bits that must be transmitted through the system to achieve a desired confidence level. Following is an example of this procedure:

1. Select ph, the hypothetical value of p. This value is the probability of bit error that we would like to verify. For example, if we want to show that P(e)<10^-10, then we set p in equation 8 equal to Ph = 10^-10.
2. Select the desired confidence level. Here, we are forced to trade confidence for test time. To minimize test time, choose the lowest reasonable confidence level. The tradeoff between test time and confidence level is proportional to -ln(1-CL).
3. Solve equation 8 for n. In most cases, this task is simplified by assuming that no bit errors will occur during the test (i.e., N = 0).
4. Calculate the test time. The time required to complete the test is n/R, where R is the data rate.

To develop a standard P(e) test for the MAX3675 and MAX3875, we might select the test time corresponding to N = 3 from Table 1. Using a bit-error-rate tester (BERT), we transmit 1011 bits through each of the two chips. The test time for 1011 bits is 2 min and 41 sec at 622 Mbits/sec or 40.2 sec at 2.5 Gbits/sec. At the end of the test time, we check the number of detected bit errors (e). If (e) is <3, the device has passed, and we are 99% confident that P(e)<10^-10.

Dan Wolaver has documented a method for reducing test time by stressing the system.3 It is based on an assumption that the dominant cause of bit errors is thermal (Gaussian) noise at the input of the receiver. (Note that this assumption excludes jitter and other potential causes of error.) For systems in which this assumption is valid, the signal-to-noise ratio (SNR) can be reduced by inserting a fixed attenuation in the transmission path (i.e., the attenuation applies to the signal only, not the dominant noise source). In the previous example (MAX3675 and MAX3875), it was determined that jitter effects and nonlinear gain in the input limiting amplifier violated the key assumptions of this method, so it was not employed.

where Q(x) is the complementary error (or the Q function included in many communications textbooks6). A variety of other sources for this data are available, including the NORMDIST function in Microsoft Excel. Key values for the complementary error function are listed in Table 2.

Equation 9 shows that the probability of bit error increases as the SNR decreases. If a fixed attenuation (a) is inserted in the transmission path, then the signal power (PS) is reduced by the factor a, while the noise power (PN) is unchanged. The SNR is therefore reduced from SNR = PS/PN to SNR = PS/aPN. The corresponding P(e) is increased by a factor that can be calculated using equation 9 and Table 2.

Table 3 (vs. Table 1) illustrates the reduction in test time achievable using an optical signal-attenuation factor of a = 1.132, which reduces the SNR from 12.7 to 11.2. In this example, ph is increased from 10^-10 to approximately 10-8 (calculated using equation 9). This factor-of-100 increase in ph results in a corresponding decrease in test time. Extrapolation of the results in Table 3 to other SNRs retains the same confidence level of 99%.

Statistical confidence levels can be used to establish the quality of an estimate. Application of this idea allows a tradeoff of test time versus the level of confidence desired in the test results.

Justin Redd, Ph.D. E.E., is a senior member of the technical staff for the high-frequency/fiber communications group at Maxim Integrated Products (Beaverton, OR).

1. A. Papoulis, Probability, Random Variables, and Stochastic Processes, New York, McGraw-Hill, 1984.
2. K.S. Shanmugan and A.M. Breipohl, Random Signals: Detection, Estimation, and Data Analysis, New York, John Wiley and Sons, 1988.
3. D.H. Wolaver, "Measure Error Rates Quickly and Accurately," Electronic Design, pp. 89-98, May 30, 1995.
4. J.G. Proakis, Digital Communications, New York, McGraw-Hill, 1995.
5. J.M. Senior, Optical Fiber Communications: Principles and Practice (second edition). Englewood Cliffs, NJ, Prentice Hall, 1992.
6. B. Sklar, Digital Communications: Fundamentals and Applications. Englewood Cliffs, NJ, Prentice Hall, 1988.