In a recent post, at TechRepublic, Chad Perrin argued that open source is definitely a better security solution than proprietary solution because it complies with Kerckoffs Law.
Although, it is true most of the time, it is not an absolute truth in security (as usual). It depends on the trust model of the security system. I will take an example: OpenSSL. The trust model of SSL is that Alice and Bob trust each other and they want to avoid that Eve spies them or tampers their messages. Thus, OpenSSL uses cryptographic algorithms. The OpenSSL cryptographic toolbox is well studied and perfect. But only under the above mentioned trust model.
Let us now suppose that Alice wants to control the access of Bob to an information stored on Bob’s computer. She does not trust Bob. Thus, she will cipher the information with a secret key and gives a decryption program to Bob. Nevertheless, for obvious reasons, she wants to keep the secret key secret from Bob. She cannot use the cryptographic toolbox of OpenSSL (although it is good and has no flaws) because Bob, being a good hacker, will easily extract the secret key by knowing where and when it is used in OpenSSL.
This example is a simplified illustration of the problem of DRM. Therefore, it is impossible to design an open source DRM for B2C or B2B applications. The final user is not trusted. It may have sense in a C2C model (Consumer To Consumer).
Open source is perfect if the trust model of the system assumes that the “owner” or “operator” of the corresponding software is trusted. If it is not the case, then Open source is not the right answer. Then, we enter in the realm of secure coding and tamper resistant software which is another story.
Nevertheless, even for proprietary implementations, it is recommended to use well known and studied algorithms and protocols. Here, security by obscurity is bad. For implementation issues, it is another story (remember AACS hack)