The ups and downs of running enclaves in production

Cian Butler SRE@Evervault

Some Context

Evervault offers tooling for data security and compliance.

Encryption Proxy
Secure Serverless Functions
Embeddable UI components
Enclaves

Relay

The Problems

Piles of custom automation.
Nascent Libraries
Observability
Userspace Networking
Time slippage

Nascent Libraries

pub struct NsmConnection(i32);
impl NsmConnection {
    pub fn try_new() -> Result<Self, NsmConnectionError> {
        let nsm_fd = nitro::driver::nsm_init();
        if nsm_fd < 0 {
            return Err(NsmConnectionError::InitFailed);
        }
        Ok(Self(nsm_fd))
    }
}
impl std::ops::Drop for NsmConnection {
    fn drop(&mut self) {
        nitro::driver::nsm_exit(self.fd());
    }
}

Observability

Loadtest Graphs showing performance improvements moving from k1 to r1 curve

Loadtest Graphs showing performance improvements moving from m5 to c5 nodes

Our Encryption service load our users encrypted keys once it has been fully attested by our KMS. Given we require full attestation we are unable to run in debug mode (Not that we ever would want to do this) and produce any logs from our service.

To begin to get metrics out of service we had started blackbox monitoring it. We ran loadtests using artillery against it and instrumented the clients to produce metrics that would tell us how the service would perform under different workloads and scenarios.

This worked to help us fund bottlenecks in the service and optimise our workload. But this only got us so far for monitoring real production. When we wanted to actually debug real users queries we were missing vital details about what was happening.

For this we had to actually implement a concept of traces into the enclave. The Service would create a trace for each request store the finished trace and all its meta in a buffer that would periodically be flushed to a reporter on the host node for forwarding on to the central trace infra.

Userspace Networking

Graph showing performance difference show running a network request in a vm vs an enclave. Enclave is much lower then vm

Time slippage

There is no NTP in enclaves.

Evervault Enclaves

evervault.com/primitives/enclaves

Putting it all together we created Evervault Enclaves as a platform for others to deploy there services and don't have to worry about any of the issues we've run into.

Evervault Enclaves build from a regular docker container and are deployed into a enclave run in evervault infra. The enclaves have :

fully attestable connections.
TLS termination
Ability to integrate with other evervault encryption services.
And takes care of all the scaling issues

We want to take the pain that we experience out of running enclaves.