<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="en-RO" link="#0563C1" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> Hi all,<o:p></o:p> <o:p></o:p> This RFC is a continuation of a longer kernel patch thread<o:p></o:p> <a href="https://lkml.org/lkml/2021/3/8/677">https://lkml.org/lkml/2021/3/8/677</a> where we originally thought such<o:p></o:p> a mechanism belongs. Ultimately, consensus there was that this mechanism<o:p></o:p> would be better suited in userspace, so systemd was an obvious first choice.<o:p></o:p> <o:p></o:p> Current proposal:<o:p></o:p> <ul style="margin-top:0cm" type="disc"> <li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo3">As GitHub Issue here: <a href="https://github.com/systemd/systemd/issues/19269">https://github.com/systemd/systemd/issues/19269</a><o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo3">An example PoC here: <a href="https://github.com/acatangiu/sysgenid-dbus">https://github.com/acatangiu/sysgenid-dbus</a><o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo3">Described in this email as follows:<o:p></o:p></li></ul> <o:p></o:p> # SysGenID: a system generation id provider<o:p></o:p> <o:p></o:p> ## Background and problem<o:p></o:p> <o:p></o:p> The System Generation ID feature is required in virtualized or<o:p></o:p> containerized environments by applications that work with local copies<o:p></o:p> or caches of world-unique data such as random values, uuids,<o:p></o:p> monotonically increasing counters, cryptographic nonces, etc.<o:p></o:p> Such applications can be negatively affected by VM or container<o:p></o:p> snapshotting when the VM or container is either cloned or returned to<o:p></o:p> an earlier point in time.<o:p></o:p> <o:p></o:p> Solving the uniqueness problem strongly enough for cryptographic<o:p></o:p> purposes requires a mechanism which can deterministically reseed<o:p></o:p> userspace PRNGs with new entropy at restore time. This mechanism must<o:p></o:p> also support the high-throughput and low-latency use-cases that led<o:p></o:p> programmers to pick a userspace PRNG in the first place; be usable by<o:p></o:p> both application code and libraries; allow transparent retrofitting<o:p></o:p> behind existing popular PRNG interfaces without changing application<o:p></o:p> code; it must be efficient, especially on snapshot restore; and be<o:p></o:p> simple enough for wide adoption.<o:p></o:p> <o:p></o:p> ## Solution<o:p></o:p> <o:p></o:p> Introduce a mechanism that standardizes an API for<o:p></o:p> applications and libraries to be made aware of uniqueness breaking<o:p></o:p> events such as VM or container snapshotting, and allow them to react<o:p></o:p> and adapt to such events.<o:p></o:p> <o:p></o:p> The System Generation ID is meant to help in these scenarios by<o:p></o:p> providing a monotonically increasing u32 counter that changes each time<o:p></o:p> the VM or container is restored from a snapshot.<o:p></o:p> <o:p></o:p> The `sysgenid` service exposes a monotonic incremental System Generation<o:p></o:p> u32 counter via the DBus `com.RFC.sysgenid` accessible at<o:p></o:p> `/com/RFC/sysgenid`. It provides asynchronous SysGen<o:p></o:p> counter update notifications, as well as counter retrieval and<o:p></o:p> confirmation mechanisms.<o:p></o:p> The counter starts from zero when the service is started and<o:p></o:p> monotonically increments every time the system generation changes.<o:p></o:p> <o:p></o:p> Userspace applications or libraries can (a)synchronously consume the<o:p></o:p> system generation counter through the provided DBus interface, to<o:p></o:p> make any necessary internal adjustments following a system generation<o:p></o:p> update.<o:p></o:p> <o:p></o:p> The provided DBus interface operations can be used to build a<o:p></o:p> system level safe workflow that guest software can follow to protect<o:p></o:p> itself from negative system snapshot effects.<o:p></o:p> <o:p></o:p> System generation changes are driven by userspace software through a<o:p></o:p> dedicated DBus method.<o:p></o:p> <o:p></o:p> ### Warning<o:p></o:p> SysGenID alone does not guarantee complete snapshot<o:p></o:p> safety to applications using it. A certain workflow needs to be<o:p></o:p> followed at the system level, in order to make the system<o:p></o:p> snapshot-resilient. Please see the "Snapshot Safety Prerequisites"<o:p></o:p> section below.<o:p></o:p> <o:p></o:p> ## SysGenID DBus interface<o:p></o:p> <o:p></o:p> #### Terminology<o:p></o:p> - `watcher` - a client using the SysGenID service _watching_ for system generation changes.<o:p></o:p> - `untracked watcher` - default state for all clients. For a client to be tracked it has<o:p></o:p> to explicitly opt-in by confirming back to the service the correct _system generation<o:p></o:p> counter_.<o:p></o:p> - `tracked watcher` - a client that is tracked by the service. Such a watcher is considered<o:p></o:p> `up-to-date` only after confirming back to the service the correct<o:p></o:p> _system generation counter_.<o:p></o:p> Once tracked, a client is only _untracked_ when closing its connection to the DBus bus.<o:p></o:p> - `outdated watcher` - a _tracked_ client that whose tracking has lived through a system<o:p></o:p> generation change, but has not (yet) confirmed back to the service the correct _system<o:p></o:p> generation counter_.<o:p></o:p> <o:p></o:p> **Methods:**<o:p></o:p> - `GetSysGenCounter` - returns latest system generation counter.<o:p></o:p> - `AckWatcherCounter` - marks the client/watcher to be tracked for ACKs, is also<o:p></o:p> used by the watcher to confirm/ack the correct _sys gen counter_ to the service after<o:p></o:p> every generation change so the service keeps correct track of it as `outdated` or<o:p></o:p> `up-to-date`.<o:p></o:p> Will error if client/watcher confirms/acks the wrong _sys gen counter_.<o:p></o:p> - `CountOutdatedWatchers` - returns the number of current number of<o:p></o:p> _outdated tracked watchers_.<o:p></o:p> A value of `zero` can be interpreted as the system being fully re-adjusted after a<o:p></o:p> generation change.<o:p></o:p> - `TriggerSysGenUpdate` - triggers a generation update (should be a privileged operation).<o:p></o:p> <o:p></o:p> **Signals:**<o:p></o:p> - `NewSystemGeneration` - system generation change notification, also carries new<o:p></o:p> _sys gen counter_.<o:p></o:p> - `SystemReady` - notification sent out when all tracked watchers have _acked_ the new<o:p></o:p> _sys gen counter_. In other words, when all tracked software has adjusted to the new<o:p></o:p> environment.<o:p></o:p> <o:p></o:p> The service can keep track of watchers by DBus connections<o:p></o:p> (`org.freedesktop.DBus.NameOwnerChanged`).<o:p></o:p> <o:p></o:p> **Exported read-only file used for memory mappings:**<o:p></o:p> <o:p></o:p> The service also exports the current _sys gen counter_ through a simple file.<o:p></o:p> The file contains only 4 bytes of data at offset 0, representing the u32 value<o:p></o:p> of the system generation counter.<o:p></o:p> This file is meant to be mapped by other software in the system and be used as<o:p></o:p> a low-latency generation counter probe mechanism in critical sections.<o:p></o:p> This mmap() interface is targeted at libraries or code that needs to<o:p></o:p> check for generation changes in-line, where an event loop is not<o:p></o:p> available or in cases where DBus calls are too expensive.<o:p></o:p> In such cases, logic can be added in-line with the sensitive code to check the<o:p></o:p> counter and trigger on-demand/just-in-time readjustments when changes are<o:p></o:p> detected on the memory mapped file.<o:p></o:p> <o:p></o:p> Users of this interface that plan to lazily adjust most likely don't need to<o:p></o:p> also use the DBus interface, since tracking or waiting on them doesn't make sense.<o:p></o:p> <o:p></o:p> ### Service interface DBus XML specification<o:p></o:p> ```xml<o:p></o:p> <node name="/com/RFC/sysgenid"><o:p></o:p> <interface name="com.RFC.sysgenid"><o:p></o:p> <method name="AckWatcherCounter"><o:p></o:p> <arg name="watcher_counter" type="u" direction="in"/><o:p></o:p> <arg name="sysgen_counter" type="u" direction="out"/><o:p></o:p> </method><o:p></o:p> <method name="CountOutdatedWatchers"><o:p></o:p> <arg name="outdated_watchers" type="u" direction="out"/><o:p></o:p> </method><o:p></o:p> <method name="GetSysGenCounter"><o:p></o:p> <arg name="sysgen_counter" type="u" direction="out"/><o:p></o:p> </method><o:p></o:p> <method name="TriggerSysGenUpdate"><o:p></o:p> <arg name="min_gen" type="u" direction="in"/><o:p></o:p> </method><o:p></o:p> <signal name="NewSystemGeneration"><o:p></o:p> <arg name="sysgen_counter" type="u"/><o:p></o:p> </signal><o:p></o:p> <signal name="SystemReady"><o:p></o:p> </signal><o:p></o:p> </interface><o:p></o:p> <interface name="org.freedesktop.DBus.Introspectable"><o:p></o:p> <method name="Introspect"><o:p></o:p> <arg name="xml_data" type="s" direction="out"/><o:p></o:p> </method><o:p></o:p> </interface><o:p></o:p> </node><o:p></o:p> ```<o:p></o:p> <o:p></o:p> ## Snapshot Safety Prerequisites and Example<o:p></o:p> <o:p></o:p> If VM, container or other system-level snapshots happen asynchronously,<o:p></o:p> at arbitrary times during an active workload there is no practical way<o:p></o:p> to ensure that in-flight local copies or caches of world-unique data<o:p></o:p> such as random values, secrets, UUIDs, etc are properly scrubbed and<o:p></o:p> regenerated.<o:p></o:p> The challenge stems from the fact that the categorization of data as<o:p></o:p> snapshot-sensitive is only known to the software working with it, and<o:p></o:p> this software has no logical control over the moment in time when an<o:p></o:p> external system snapshot occurs.<o:p></o:p> <o:p></o:p> Let's take an OpenSSL session token for example. Even if the library<o:p></o:p> code is made 100% snapshot-safe, meaning the library guarantees that<o:p></o:p> the session token is unique (any snapshot that happened during the<o:p></o:p> library call did not duplicate or leak the token), the token is still<o:p></o:p> vulnerable to snapshot events while it transits the various layers of<o:p></o:p> the library caller, then the various layers of the OS before leaving<o:p></o:p> the system.<o:p></o:p> <o:p></o:p> To catch a secret while it's in-flight, we'd have to validate system<o:p></o:p> generation at every layer, every step of the way. Even if that would<o:p></o:p> be deemed the right solution, it would be a long road and a whole<o:p></o:p> universe to patch before we get there.<o:p></o:p> <o:p></o:p> Bottom line is we don't have a way to track all of these in-flight<o:p></o:p> secrets and dynamically scrub them from existence with snapshot<o:p></o:p> events happening arbitrarily.<o:p></o:p> <o:p></o:p> ### Simplifying assumption - safety prerequisite<o:p></o:p> <o:p></o:p> **Control the snapshot flow**, disallow snapshots coming at arbitrary<o:p></o:p> moments in the workload lifetime.<o:p></o:p> <o:p></o:p> Use a system-level overseer entity that quiesces the system before<o:p></o:p> snapshot, and post-snapshot-resume oversees that software components<o:p></o:p> have readjusted to new environment, to the new generation. Only after,<o:p></o:p> will the overseer un-quiesce the system and allow active workloads.<o:p></o:p> <o:p></o:p> Software components can choose whether they want to be tracked and<o:p></o:p> waited on by the overseer by using the marking themselves as tracked<o:p></o:p> watchers.<o:p></o:p> <o:p></o:p> The sysgenid service standardizes the API for system software to<o:p></o:p> find out about needing to readjust and at the same time provides a<o:p></o:p> mechanism for the overseer entity to wait for everyone to be done, the<o:p></o:p> system to have readjusted, so it can un-quiesce.<o:p></o:p> <o:p></o:p> ### Example snapshot-safe workflow<o:p></o:p> <o:p></o:p> 1) Before taking a snapshot, quiesce the VM/container/system. Exactly<o:p></o:p> how this is achieved is very workload-specific, but the general<o:p></o:p> description is to get all software to an expected state where their<o:p></o:p> event loops dry up and they are effectively quiesced.<o:p></o:p> 2) Take snapshot.<o:p></o:p> 3) Resume the VM/container/system from said snapshot.<o:p></o:p> 4) Overseer will trigger generation bump using<o:p></o:p> `TriggerSysGenUpdate` method.<o:p></o:p> 5) Software components which have the DBus `NewGeneration` signal in<o:p></o:p> their event loops are notified of the generation change.<o:p></o:p> They do their specific internal adjustments. Some may have chosen to<o:p></o:p> be tracked and waited on by the overseer, others might choose to do<o:p></o:p> their adjustments out of band and not block the overseer.<o:p></o:p> Tracked ones *must* signal when they are done/ready by confirming the<o:p></o:p> new sys gen counter using the `AckWatcherCounter` DBus method.<o:p></o:p> 6) Overseer will block and wait for all tracked watchers by waiting on<o:p></o:p> the `SystemReady` DBus signal. Once all tracked watchers are done<o:p></o:p> in step 5, the signal is sent by `sysgenid` service and overseer will<o:p></o:p> know that the system has readjusted and is ready for active workload.<o:p></o:p> 7) Overseer un-quiesces system.<o:p></o:p> 8) There is a class of software, usually libraries, most notably PRNGs<o:p></o:p> or SSLs, that don't fit the event-loop model and also have strict<o:p></o:p> latency requirements. These can take advantage of the<o:p></o:p> _exported read-only file used for memory mappings_. They can map the<o:p></o:p> file and check sys gen counter value in-line with the critical section<o:p></o:p> and can do so with low latency. When they are called after un-quiesce,<o:p></o:p> they can just-in-time adjust based on the updated mapped value.<o:p></o:p> For a well-designed service stack, these libraries should not be<o:p></o:p> called while system is quiesced. When workload is resumed by the<o:p></o:p> overseer, on the first call into these libs, they will safely JIT<o:p></o:p> readjust.<o:p></o:p> Users of this lazy on-demand readjustment model should not use the<o:p></o:p> DBus interface or at least not enable watcher tracking since doing so<o:p></o:p> would introduce a logical deadlock:<o:p></o:p> lazy adjustments happen only after un-quiesce, but un-quiesce is<o:p></o:p> blocked until all tracked watchers are up-to-date.<o:p></o:p> </div> Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005. </body> </html>