Security assessment: white or black box?

White or Black: Which is the best choice? Is white box testing cheating? To me, the answers seem trivial. Nevertheless, recent discussions highlighted that it is not clear for everybody.

White box testing means that the security assessor has access to the entire documentation, the source code and even sometimes an instrumented target of evaluation (TOE). Black box testing means that the security assessor has only access to the TOE and publicly available information. Thus, black box testing mimics the configuration encountered by hackers. Of course, hackers will use any possible means to collect related non-public information, for instance, using social engineering.

Many people believe that black box evaluation is the solution of choice. As it is similar to the hackers’ configuration, it should provide a realistic assessment of the security of the TOE. Unfortunately, this assumption is wrong. The main issue is the asymmetry of the situation. On one side, the black box evaluation is performed by one evaluator or a team of reviewers. On the other side, a legion of hackers probes the product. They may outnumber the evaluators by several magnitudes. They spend more time to discover vulnerabilities. Mathematically, their chance to find an exploit is higher than the likelihood of the evaluators to find the same vulnerability.

Do you evaluate your security to know whether an attacker may breach your system? According to Law 1, this will ineluctably happen. Then, if your evaluation team has not found any problem, you may only conclude that there were no blatant vulnerabilities. You have no idea whether (or rather when) a hacker will find a vulnerability.

Alternatively, do you evaluate the security to make your product more secure, i.e., with fewer vulnerabilities? In that case, you will give the evaluators the best tools to discover vulnerabilities. Therefore, white box testing will be your choice. For the same effort, white box testing will find more security issues than black box testing. Furthermore, the team will pinpoint the faulty part of the system whereas black box testing will disclose how the attacker may succeed but not where the actual issue is. Furthermore, let us assume that the white box assessment discovered a vulnerability through code review. The white box tester has just to explain the mistake and its consequences. For the same vulnerability, the black box tester has to blindly explore random attacks until he/she finds the vulnerability. Then, the evaluator has to write the exploit that demonstrates the feasibility of the attack. The required effort is too big. Thus, in term of Return On Investment (ROI), white box testing is far superior to black box testing. More discovered vulnerabilities for the money!

Fixing a vulnerability discovered by white box testing may also be cheaper. It is well known that the earlier a bug is found during the development cycle, the cheaper it is to fix. The same goes for vulnerabilities. As some white box security testing can occur before final integration (design review, code review…), fixing it is earlier thus cheaper than for black box testing which occurs after the final integration.

White box security testing is compliant with Kerckhoffs’s law. The selection of new cryptographic primitives uses white box testing. The algorithm is published for the cryptanalysts to break. ISO27001 is a kind of white box evaluation.

When a company claims that a third party audited the security of its product, in addition to the identity of this third party, it would be great to disclose whether it was white or black box testing. I would favor the first one.

Thus, when possible, prefer white box security testing over black box testing. This is the wise, winning choice. The bounty programs are the only exception. They operate as black box testing. They should be a complement to white box testing but never a replacement. Their ROI is high as you pay only when successful.




Leave a Reply

Your email address will not be published. Required fields are marked *