Jeff MOSS introduction
Jeff MOSS is the founder of Black Hat and Defcon. He always presents his latest thoughts.
New (probably) unforeseen threats have risen in the geopolitical landscape in the last few years. For instance, what do you do if some of your development teams are in a war zone? What if the IP is stored in these zones? Can you stay neutral? What are the cyber consequences if you cannot?
Keynote: Democracy’s Biggest Year: The Fight for Secure Elections Around the World
BUECHEL E. (CISA), De VRIES H. (European Union Agency for Cybersecurity), EASTERLY J. (Cybersecurity and Infrastructure Security), OSWALD F. (National Cyber Security Centre)
Nihil Nove Sub Sole. The usual expected stuff.
Practical LLM Security: Takeaways From a Year in the Trenches
HARANG R. (NVIDIA)
First, he provided a high-level explanation of Large Language MOdel (LLM). The interesting point is that although the candidate tokens are ranked by their highest probability, the sampling is random. Thus, LLM sometimes makes bad/weird selections (hallucination,…).
Sampled tokens are locked (no go-back). Thus, the lousy selection continues and cannot be reversed, at least by the LLM. The same is true for prompts (Forgetting previous prompts is not going back).
This is why Retrieval Augmented Generation (RAG) is used. RAG allows better fine-tuned knowledge.
He highlighted some RAG-related issues. But RAG increases the attack surface. It is easier to poison a RAG dataset than the LLM dataset. For instance, he described the Phantom attack. The attacker can direct the expected answer for a poisoned concept.
Therefore, the security and access control of the RAG is crucial. Furthermore, RAG is excellent at searching. Thus if the document classification (and reinforcement) and access control are lax, it is game over. It is relatively easy to leak confidential data inadvertently.
The RAG’s use of emails is a promising but dangerous domain. It is an easily accessible point of poisoning for an attacker and does not require penetration.
What is logged and who can view the logs is also a concern. Logging the prompts and their responses is very sensitive. Sensitive information may leak and, in any case, bypass the boundaries.
Do not rely on guardrails. They do not work or protect against a serious attacker.
Privacy Side Channels in Machine Learning Systems, Debendedetti et al., 2023 is an interesting paper to read.
15 Ways to Break Your Copilot
EFRAT A. (Zenity)
Copilot is a brand name that encompasses all of Microsoft’s AI products. All Copilots share the same low-level layers (i.e., they use the same kernel LLM) and are specialized for a set of tasks.
Copilot Studios allows with no code to create a Gen AI-based chatbot. The speaker presented many default configuration issues that opened devastating attacks. Meanwhile, Microsoft has fixed some of them to be less permissive. Nevertheless, there are still many ways to allow the leaking of information. This is especially true as the tool targets non-experts and thus has a rudimentary security stance if there is even a security stance)
Be careful who you authorize to use such tools and review the outcome.
Kicking in the Door to the Cloud: Exploiting Cloud Provider Vulnerabilities for Initial Access
RICHETTE N. (Datadog)
The speaker presented cross-tenant issues in AWS. Datadog found some vulnerabilities in the policies managing `sts:AssumeRole`.
Lesson: When using `sts:AssumeRole`, add restrictive conditions in the policy based on the ARN, or Source, and so on.
Compromising Confidential Compute, One Bug at a Time
VILLARD Maxime (Microsoft)
To isolate a tenant from the cloud provider, Intel proposes a new technology called TDX. It will be present in the next generation of Intel chips. The host sends a set of commands to enter the TDX mode for a module. In this mode, the TDX module can launch its own VM to execute independently from the cloud hypervisor.[1]
The team found two vulnerabilities. One enabled a DoS attack from within the TDX to crash all the other tenants executing on the host processor.
[1] TDX is not an enclave like SGX.