Execution security in the Spectre era

A. Mambretti

Since early 2018 with the release of new attacks such as Meltdown and Spectre the search fornew attack surfaces left the software domain and reached the microarchitectural world. This new type of vulnerabilities exploits bugs, or performance optimizations within the CPU to carry out information disclosure even across privilege domains. This new class of vulnerabilities, referred broadly as transient execution, presents a unique challenge because of the lack of details in the microarchitectural realm and tools to study such behaviors. While high level views are available, internal CPU implementations are highly variable from the vendors and the CPU families and often covered by patents. In this thesis, I provide research into understanding the impact of transient execution attacks in the field of system security. My contributions focus on two specific problems, improving the analysis of transient execution attacks, and understanding their impact on the security of modern systems-i.e., the effects of these attacks on the current existing threat models. First, I provide a new debug-like technique to study transient execution attacks and reverse engineer the microarchitecture. Leveraging the CPU Performance Monitor Counters (PMCs), I show how it is possible to deterministically observe the side effects of transient execution. I integrate such principle in a new tool, Speculator, that provides the infrastructure to easily build tests to shed light on the microarchitecture internals. Using Speculator, I provide, as results, insights in the microarchitecture internal behaviors, the study of a new Spectre variant called Split Spectre and, two new side-channel gadgets-i.e., the Branch Target Buffer (BTB) and the instruction cache (i-cache)- that can be used as alternative to the common data cache. Second, I provide insights on the effects of transient execution attacks in existing threat models. My effort towards solving this problem is twofold. On the one hand, I study a subset of the Spectre family of attacks, the SPEculative ARchitectural control flow hijacks (SPEAR) and their effect on current memory corruption mitigations-i.e., the Stack Smashing Protection (SSP), the Control Flow Integrity (CFI), and the stack protections in memory safe languages. I show how these mitigations, while mitigating memory corruption vulnerabilities, extend the attack surface in the context of transient execution attacks. My results indicate the need for such mitigations to be re-designed to include transient execution attacks in the threat model. On the other hand, I present the first study of the transient execution vulnerability checkers. I provide insights in current methodologies-i.e., their strengths and weaknesses-, and show how these tools are not adequate to understand the security stance of a system against transient execution attacks. As a result, I propose a new hybrid tool, called GhostBuster that overcomes the issues of the state-of-the-art and provides results that are threat model aware.--