Customer Found Defect
Index：
Customer Found Defect (CFD)　 The Negative Impact of CFD　 Way to reduce CFD　 Comparison of Lab Test, Field Test and Replay Test

Customer Found Defect (CFD)
Customer Found Defect (abbreviated as CFD) is defined as the product defects founded in the real network environment at end-user side. CFD example 1: when you play on-line game via wireless devices but got disconnected often or download files via P2P but the speed got very slow. CFD example 2: connecting internet via Ethernet switch in school dormitory to experiment the software of packet producing in Network Security Class but resulted in disconnecting the internet access for others, or running the system with 10/100Mbps even via Gigabit Ethernet. CFD example 3: IT blocks MSN with security products, but web mail is also blocked while some still can look up the stock market but others can’t.
Would you, as a customer, expect to buy a product with defects? No, you would not!
Would you, as a manufacture/vendor, expect your product to be found with defects at end-user side? No, you would definitely not!


The Negative Impact of CFD

When the defects are found at end-user/customer side, it would generate a direct harm to the products and the company image. In order to save the reputation of both product and company, solving CFD reaches the highest priority. However, it costs a lot. We can discuss from the 3 steps of solving CFD. First is to clarify the problems and specifically describe the problematic situation, including knowing the product set-up features, user set-up features, application software used, duration of each application used, how long the problem occurred, and the frequency of the problem occurred…etc. Nevertheless, users often would not reply too many details to the manufacture. For instance, there is a function which is for users to “push a button” to reply the problems occurred as illustrated above. However, an informal survey in a seminar, there are only 2% of users would click on report back this defect in the pop-up dialog. As a result, it is very difficult for customers to report back the product defects. Second is to reproduce the CFD, however, as the limited information gained in the first step, test technologist need to reproduce the defects occurred at the customer side by trying various possible combinations, some would take days and some would not be able to be reproduced. Then, when the CFD could not be reproduced, the third step will be taken – technologist to visit the site (where the customer is) in person to clarify, reproduce and solve the problem/defect. In general, the first step (Identify) is time consuming both internally and externally. The second step will waste much time on reproduce problems. The third step (on-site debugging) would by costly on the travelling spending, especially for overseas case.
Besides the cost issue, CFD would also cause problems within the company. Take the example from the above chart, when CFD occurred, customer returns the product as not satisfying the quality, and the salesperson would be complained and also lost the revenue. Then FAE would not be able to completely clarify the product defects, and QA can not reproduce the problem at internal lab. As the result RD is not capable to exactly solve the problem, and the product quality would not be improved, then the project would be a total disaster to PM. From the company’s point of view, this is a negative circle and people would blame each other and cause separation within the company.


Comparison of Lab Test, Field Test and Replay Test

If to categorize tests type to Lab Test, Field Test and Replay Test, the following table outlines the characteristics for each test type. NBL’s data on the investment of equipment and personnel is also stated for reference. However, test in real flow environment is the only solution to reduce CFD occurrence.


Way to reduce CFD

Since the company needs to pay so much for CFD, then how to reduce CFD? From the viewpoint of testing, NBL considers how to provide a real testing environment to easily Identify, Reproduce and Debug, and provide solutions from the three aspects: Testing Environment, Technology and Application. The first aspect (testing environment) is the fundamental problem, how could we find a group of users to try the testing products in their general daily usage. We name this group of users as “Beta Site” where to test the prospective products. The current “Beta Site” which NBL builds on NCTU Computer & Network Center and NCTU Student Dormitory contains 1,300 students. This indicates that there will be 1,300 users to use the testing network products. The framework of Beta Site is illustrated as the chart at the right. There are 6 testing “zones” and the applicable testing products include: End-user software, Ethernet L2/L3 Switch, Wireless AP, Core Router, UTM, IPS, Anti-Virus, QoS Firewall, Network Forensic, Anti-Malware/Botnet, SOHO Router, Home Gateway, Broadband Gateway, DSL Router, IAD Gateway. The second aspect (technology) is to develop the technology of capturing and replaying real network traffic. This is for those products which are not suitable for testing on Beta Site due to certain reasons, such as flow speed and environment feasibility…etc. The technology used needs to be able to capture the network traffic in high speed and not to lose any packets. NBL applies special software and hardware to meet this requirement. For flow replaying, NBL uses Open Source tools (i.e. tcpreplay and tomahawk) and commercial testing tools (i.e Spirent Avalanche and Karalon Traffice IQ Pro). NBL also develops its own replay tools, such as natreplay and socketreplay. The third aspect (application) is to develop related applications based on the technology of network traffic capture and replay. NBL at the present has developed two applications: PCAP Library and In-Lab Live Test (ILLT). PCAP Library is the database of network flow which can be applied on figuring out the False Negative and False Positive of Network Security products. While In-Lab Live Test (ILLT) is to applied to test Home Gateway products on stability issues, such as crash, re-boot, software crash down, slow speed and easily disconnected.


Lab Test Field Test Replay Test
Reality of Network Flow Low High Medium
Test Type Functionality, Performance, Conformance, Inter-operability (IOT) Stability, Compatibility, Accuracy, Performance Stability, Accuracy, Performance
Contribution to reduce CFD Low High High
Participant QA Personnel RD Personnel QA or RD Personnel (depending the application)
Proportion of Equipment Investment 100 10 1
Proportion of Personnel Investment 50% 25% 25%