Publications / FLASH
FLASH · SIGOPS ATC 2026
FLASH: Fault-Led, Attested, Secure Hybrid-Key Checkpoint for Confidential Inference
2026 ACM SIGOPS Annual Technical Conference (SIGOPS ATC 2026). To appear.
How can a confidential LLM service save its GPU state without pausing inference for seconds at every checkpoint? FLASH combines encrypted snapshots with GPU write-fault tracking to make recovery practical on a CPU–GPU confidential computing stack.
Why checkpointing is difficult
A long-running LLM service accumulates more than model weights. Its key-value (KV) cache holds the context needed to continue generating responses. Saving this state lets a service recover from failures without rebuilding all of that work.
Confidential computing changes the cost of a snapshot. GPU memory must leave the device encrypted. A conventional checkpoint path then decrypts it inside the CPU confidential VM and encrypts it again for storage. In the paper’s H100 measurements, the confidential transfer path reaches about 7.3 GB/s, versus about 40 GB/s without confidential computing; copying a full 80 GB device takes roughly 11 seconds. Repeating that work while inference is stopped creates a substantial serving interruption.
How FLASH works
FLASH brings three mechanisms together. It preserves ciphertext that the GPU has already produced, tracks pages that change during a snapshot, and avoids repeatedly copying allocations that remain unchanged.
Keep GPU ciphertext
Store the GPU copy engine’s encrypted output directly. Wrap its session key with the checkpoint key, alongside encrypted CPU state and authenticated metadata.
Track writes while serving
Copy pages while inference continues. Mark copied 2 MB pages read-only; the first subsequent write faults, records the page as dirty, and restores write access.
Pause for the final delta
Briefly stop inference and copy the remaining dirty state. Never-written allocations, such as model weights, are omitted from repeated copies while still being monitored.
- Hybrid-key snapshots. GPU memory stays under the GPU’s data-transfer key. CPU state and the wrapped GPU key use a separate checkpoint key. The snapshot retains authentication tags, initialization-vector counters, and an authenticated manifest so that the saved pieces can be validated together.
- Fault-led dirty tracking. A modified NVIDIA Unified Virtual Memory (UVM) driver supplies the write tracking that incremental checkpointing needs. The approach does not require changes to LLM kernels or application source, but it does require driver support.
- Freshness outside the cloud operator’s control. A trusted remote attestation service keeps checkpoint key and freshness state. This supports rejection of stale snapshots as well as tampered data.
Simplified checkpoint path. Preserving ciphertext removes redundant cryptography during checkpoint creation; recovery still decrypts saved state and transfers it under a new GPU session key.
What the experiments show
The evaluation runs vLLM 0.8.5 on an NVIDIA H100 with 80 GB of memory, paired with an Intel TDX confidential VM. It covers Llama-3.2-1B, Qwen2.5-7B, and Llama-2-13B under short chatbot and longer reasoning workloads. Checkpoint images are stored in DRAM-backed tmpfs; these results do not include remote-storage latency.
- About 1.2 seconds of stop-and-copy time on average. At a one-minute checkpoint interval, FLASH reduces this pause by up to 9.4× relative to the PhoenixOS-style baseline and 7.8× relative to the GCR-style baseline (§7.2). The paper reimplements those systems’ checkpoint data paths for the confidential-computing setting.
- Under 19 seconds to restore the evaluated state. The three models’ roughly 66–67 GB GPU footprints restore in 17.1–18.7 seconds (§7.4). Restore time is separate from the pause incurred when creating a checkpoint.
- Serving latency remains workload-dependent. Write faults and page management add overhead. Some short-chatbot settings see worse tail latency than the baselines, even though the final checkpoint pause is much shorter (§7.3).
Security and scope
FLASH protects checkpoint confidentiality, integrity, and freshness against a malicious cloud operator controlling the host, hypervisor, storage, and PCIe path. It trusts the CPU confidential VM, the GPU TEE and its secure transfer mechanism, and the remote attestation service.
The prototype targets a single GPU. Globally consistent multi-GPU checkpoints, denial-of-service prevention, and hiding access patterns or checkpoint timing and size are outside its scope. The main result is a shorter secure checkpoint pause, rather than a claim that every workload becomes faster.
Paper and artifact
The full paper describes the threat model (§3), checkpoint design and dirty-page tracking (§5), and evaluation (§7). The archived artifact provides the ATC 2026 evaluation release.
Related: ARGUS studies a complementary question—checking LLM execution when the GPU itself is untrusted. See also our work on trusted execution and confidential computing.
Abstract
GPU Confidential Computing (CC) enables secure LLM inference in production, combining a CPU TEE with NVIDIA’s GPU TEE to protect models and user data from a malicious cloud operator. As long-running LLMs are failure-prone, traditional cloud systems rely on backups, and live migration using checkpoint and restore to deal with failures. In particular, during LLM serving, GPUs build up a massive key-value (KV) cache that accelerates the decode process. But when using GPU CC, these recovery methods fail, as the GPU’s memory state, such as the KV cache, can’t be efficiently checkpointed. Checkpointing GPU memory requires encrypting the GPU data and transmitting to the CPU over the PCIe, which the CPU then decrypts using its shared key with the GPU. This encryption and decryption sequence over the PCIe leads to a 5.5× bandwidth penalty. The decrypted data is then re-encrypted with a new set of keys shared by the CPU and a destination server which adds overhead to the already-long checkpoint latency. LLM serving is stalled for the entire duration of checkpointing; a secure full-dump of an 80 GB H100 memory takes roughly 11 s. Alternatively, tracking write operations and selectively checkpointing these written pages is a preferable path as only a small fraction of the KV cache is updated after each decode round. Current GPUs, however, lack the ability to track dirty pages.
We present FLASH, a checkpoint/recovery (C/R) system for GPU CC that is both secure (providing confidentiality, integrity, and freshness against a malicious operator) and low latency. FLASH eliminates needless decryption of GPU state during checkpoint creation. FLASH preserves the GPU-encrypted HBM data as is without decryption. The CPU then creates a heterogeneous encryption package: (1) the CPU’s own memory state and the key used to encrypt the GPU CC is encrypted into a single bundle using keys shared by the CPU and a destination server that stores the checkpoint. (2) The GPU-encrypted HBM data is preserved as a second bundle within this heterogeneous package. This approach eliminates redundant decrypt-then-reencrypt overheads for GPU CC state. FLASH also presents a novel approach to track dirty pages to enable incremental checkpointing. FLASH enhances the GPU’s unified virtual memory (UVM) driver to downgrade any GPU page that is already checkpointed to a read-only page. It then uses the write fault mechanism to track modified pages at a 2 MB granularity. This approach requires no source code modifications to the LLM code base. Lastly, FLASH classifies never-written allocations, such as model weights, and removes them from repeated checkpoint copies while continuing to track them for correctness. On an H100 with Intel TDX running vLLM, FLASH reduces checkpoint stall latency up to 9.4× over state-of-the-art GPU C/R systems.
Based on the linked paper. Read the full paper for the design, threat model, and evaluation. Lab members and advisees are shown in bold.