Opened 6 months ago

Closed 10 days ago

#14397 closed Bug/Something is broken (fixed)

update kvm-manager to pass host entropy through to the guest via virtio rng

Reported by: Daniel Kahn Gillmor Owned by: Jamie McClelland
Priority: Medium Component: Tech
Keywords: kvm entropy kvm-manager Cc: Jamie McClelland
Sensitive: no

Description (last modified by Daniel Kahn Gillmor)

As guests move to using more getrandom() system calls, it's possible for the guests to be starved of entropy. The MF/PL KVM hosts should offer to share their entropy with the guests by way of a virtio RNG. It's up to the guests whether they want to rely on this or not, of course, but it's good to make it available.

This can probably be done in a simple way just by merging in the latest head of kvm-manager -- both systemd or master branch incorporate a basic workaround .

Two risks here:

  • the guests have no way of vetting the quality of the entropy they get from this device. we don't want to risk feeding the guests entropy from an uninitialized RNG, which would be the case if we source the virtio device from /dev/urandom. My current patch uses qemu's defaults, which is to source the virtio device from /dev/random
  • we don't want the guests to starve the hosts of entropy via this interface, which is why i'm limiting it by default to 128 octets per second per host.

see qemu documentation about virtio RNG and kvm-manager bug report, and my upstream bug report asking to change the default source to the `getrandom()` syscall, if available.

I'm hoping that someone else will resolve this ticket, to double-check my reasoning here, and so that i'm not the only one thinking about these questions.

Change History (8)

comment:1 Changed 6 months ago by Daniel Kahn Gillmor

Description: modified (diff)

comment:2 Changed 6 months ago by JaimeV

Owner: set to JaimeV
Status: newassigned

I think I understand the issue, but still reading up on the details.

So basically:

  • there is an issue with the way guests currently source random numbers through /dev/random
  • what we really want is for upstream qemu to change the way the guests receive random numbers, i.e. using 'getrandom()'
  • for now upgrading kvm-manager will include a workaround that attempts to limit the possibility that the guest could starve the host and other guests of entropy.

Is that right?

comment:3 Changed 6 months ago by JaimeV

Cc: Jamie McClelland added

and copying jamie here.

comment:4 Changed 6 months ago by Jamie McClelland

Just a few clarifications to Jaime's summary:

  • The way KVM guests currently source randomness is fine, but with buster, it will change because of the use of a new system call, getrandom(), which will block if it doesn't have enough randomness
  • This problem is compounded by a problem with systemd in which it is not carrying over randomness when it reboots. So when it boots up, there is no randomness (previously, it seems that the randomness accumulated during a period in which the guest was running prior to reboot was available at boot) and the boot process has to stop and wait for enough randomness to become available before continuing.
  • The problem dkg seeks to fix with qemu won't solve this bigger problem. We will still need to pass a source of randomness from the host to the kvm guest to solve the problem. However, fixing qemu will mean we don't need to limit the amount of randomness a guest can suck down from the host

I reviewed all of dkg's notes and references and it all seems to make sense to me.

I would like us to switch to dkg's latest version in the systemd branch, however, that will require some fiddling (his latest branch places the kvm@.service file in /etc/systemd/system instead of /lib/systemd/system) and some testing.

We are setting up two brand new physical servers this week - so I suggest we test these changes on those servers and if all goes smoothly we roll them out to the rest of our servers.

comment:5 Changed 6 months ago by Jamie McClelland

Owner: changed from JaimeV to Jamie McClelland

comment:6 Changed 6 months ago by Daniel Kahn Gillmor

fwiw, i'm *not* convinced that the way KVM guests currently source randomness is fine.

In particular, most userspace tools that source randomness (e.g. those derived from OpenSSL) will pull from /dev/urandom, which returns data even when the CSPRNG is not yet adequately initialized.

What will change with buster is that OpenSSL (and gcrypt, and other libraries and userspace utilities) will start actually using the getrandom() system call instead of /dev/urandom. getrandom() is strictly better than /dev/urandom -- in the normal system state (when the kernel's CSPRNG is initialized) it produces the same output. But during early boot (before the CSPRNG is initialized) it blocks. This is also strictly better that /dev/random, fwiw.

So in the current situation, if the freshly-booted KVM guest needed entropy, it was getting it from an unintitialized source, which means it is potentially predictable. this is seriously bad, but subtle and difficult to notice, because everything seems to be working, and you need to read Heninger et al or be an attacker to know that this is an issue.

Once guests are running buster, the initial reads will block -- this is not subtle: you'll notice that programs/processes aren't running because they can't get the entropy that they need.

One way to solve that is to offer the KVM guests a virtualized "hardware" random number generator. the guest kernels can pull from that to seed their own CSPRNG, which will make the guest userspace tools not block when they call getrandom().

This requires that the KVM host itself has an initialized CSPRNG, though, since the virtualized hardware needs some source of entropy.

If you source the entropy from the host's /dev/urandom, then you might be passing through uninitialized entropy to the guest. but if you pass the host's /dev/random through to the guest, then the guest might get to starve the host (and the other guests) of entropy just by depleting the pool, even if the host kernel's CSPRNG is already initialized.

So the right long-term fix is to get the virtio RNG to source from the host's getrandom() syscall, to avoid starvation and to avoid sending uninitialized data. in the meantime, we can source from the host's /dev/random (to ensure it's initialized) but constrain how much any particular guest can pull over time (to avoid starvation).

comment:7 Changed 2 months ago by updater

Sensitive: set

Changed to sensitive as part of leadership decision to make all tickets sensitive.

comment:8 Changed 10 days ago by Jamie McClelland

Resolution: fixed
Sensitive: unset
Status: assignedclosed

I just pushed a small patch to include dkg's kvm-manager change on the systemd branch and I've now incorporated these changes int our puppet.

The next time we push our puppet changes to physical servers, this change will be possible.

And, I can confirm that buster guests take forever to start openssh without this change - thanks dkg for seeing this before we hit the problem so I knew exactly how to fix it!

Does this ticket need to stay open to address the long term solution or do you think we can close it?

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.