We read the first, most idealistic exokernel paper [1]. Also of interest are the followon paper on an x86 exokernel [2], which includes a discussion of the XN system for securely multiplexing the disk, and a later journal paper [3], whose Section 8 (Discussion) has some really interesting observations.
The big exokernel questions are: Is the exokernel architecture a viable alternative for operating systems design? And whether or not the architecture is viable, which of the mechanisms used to build an exokernel OS are suitable for other contexts? My answer to the first question is No, outside of limited research contexts.
Build higher performance applications by giving applications more flexible, extensible access to OS primitives.
“Hardcoding the implementation of these abstractions is inappropriate for three main reasons: it denies applications the advantages of domain-specific optimizations, it discourages changes to the implementations of existing abstractions, and it restricts the flexibility of application builders, since new abstractions can only be added by awkward emulation on top of existing ones (if they can be added at all).” [p1, 1]
Motivated by a perceived stagnation in operating system design.
At the time Microsoft was closed (and, for good and bad reasons, looked down upon by the OS research community [this is much less true today]), Apple was irrelevant, commercial Unix was fragmented, and the now-robust open Unix community was nascent. It felt like a dark time.
The goal is not any single optimization, but to provide flexibility sufficient for any optimization an application could desire.
“It is important to note that a sufficiently motivated kernel programmer can implement any optimization that is implemented in an extensible system. … Extensible systems (and we believe exokernels in particular) make these optimizations significantly easier to implement than centralized systems do.” [p9, 2]
“[A previous exokernel file system design] exhibited what we have come to appreciate as an indication that applications do not have enough control: the system made too many tradeoffs.” [p5, 2]
Separating protection from management (still frequently cited for this reason).
Of course the devil is in the details: how is this done? And something in a later section complicates this ideal: “An exokernel takes the elimination of policy one step further by removing ‘mechanism’ whenever possible. This process is motivated by the insight that mechanism is policy, albeit with one less layer of indirection.” Huh.
Secure bindings (“A secure binding is a protection mechanism that decouples authorization from the actual use of a resource” [p4, 1]).
Secure binding implementation techniques: hardware mechanisms, software caching, and downloading application code.
“Downloading code into the kernel has two main advantages. The first is obvious: elimination of kernel crossings. The second is more subtle: the execution time of downloaded code can be readily bounded.” [p5, 1]
Bounded execution time is a good technique to remember.
“Currently, Aegis dispatches exceptions in 18 instructions.” [p8, 1] Wow!
“Part of the reason for this improvement [>5x faster dispatch than Ultrix] is that Aegis does not use mapped data structures, and so does not have to separate kernel TLB misses from the more general class of exceptions in its exception demultiplexing routine.” OK: but if Aegis were to mature, might it need to map its data structures?
DPF is a very interesting system ([p10, 1], and the referenced paper on DPF). Dynamic code generation can be a very powerful technique in systems as well as languages.
The optimizations in the Cheetah HTTP/1.0 server ([p10, 2] and the referenced paper): a merged file cache and retransmission pool, knowledge-based packet merging, and particularly HTML-based file grouping.
Some undersupported claims about the exokernel architecture: “Furthermore, secure multiplexing does not require complex algorithms; it mostly requires tables to track ownership.” [p3, 1]; “Therefore, the implementation of an exokernel can be simple.” [p3, 1]; “Additionally, as is true with RISC instructions, the simplicity of exokernel operations allows them to be implemented efficiently.” [p3, 1]
“Finally, the number of kernel crossings in an exokernel system can be smaller, since most of the operating system runs in the address space of the application.” [p3, 1]
“Simple security precautions such as only allowing a trusted server to install filters can be used to address this problem.” [p5, 1]
Hasn’t the baby just been thrown out with the bathwater? How would you implement, say, two mutually distrustful servers on an exokernel, with different library OSes, that both speak IPv4—which features fragments? Aren’t we being driven towards a microkernel architecture here?
“One of the key features of an ASH [Application-specific Safe Handler] is that it can initiate a message.” [p5, 1]
How broadly applicable is ASH message initiation? Is there a limit to the complexity of protocols implementable as ASHes?
Visible Resource Revocation and The Abort Protocol: These are indicated as equal-importance contributions to secure bindings, but they seem somewhat complex and untested. How complex is the abort protocol? (“…a second stage of the revocation protocol in which the revocation request (‘please return a memory page’) becomes an imperative (‘return a page within 50 microseconds’). However, if a library operating system fails to respond quickly, the secure bindings need to be broken ‘by force.’ … [I]f a library operating system fails to comply with the revocation protocol, an exokernel simply breaks all existing secure bindings to the resource and informs the library operating system. To record the forced loss of a resource, we use a repossession vector. … [T]he library operating system receives a ‘repossession’ exception so that it can update any mappings…The simplest way to deal with [vital bootstrap information] is to guarantee each library operating system … resources that will not be repossessed (e.g., five to ten physical memory pages). If even those must be repossessed, some emergency exception that tells a library operating system to submit itself to a ‘swap server’ is required.” [p6, 1]
Watch the tenses closely. How much of the abort protocol are we sure got implemented?
“ExOS implements … several network protocols (ARP/RARP, IP, UDP, and NFS).” [p6, 1]
What’s missing? (In their defense, TCP is far more complex than UDP. On the other hand, OS-enforced congestion control is arguably important for TCP.)
“It is important to note that Aegis and ExOS do not offer the same level of functionality as Ultrix. We do not expect these additions to cause large increases in our timing measurements.” [p6–7, 1]
cough
“A crucial property of [the linear vector time slice representation] is position, which encodes an ordering … [it] can be used to meet deadlines and to trade off latency for throughput. For example, a long-running scientific application could allocate contiguous time slices in order to minimize the overhead of context switching, while an interactive application could allocate several equidistant time slices to maximize responsiveness.” [p7, 1]
And if both are running at the same time?
“Applications pay for each excess time slice consumed by forfeiting a subsequent time slice. If the excess time counter exceeds a predetermined threshold, the environment is destroyed.” [p7–8, 1]
Have fun debugging! It continues: “In a more friendly implementation, Aegis could perform a complete context switch for the application.” This would be more friendly; but isn’t it also less exokernel?
“[A]pplications are prevented from directly modifying the [Xok] page table and must instead use system calls. Although these restrictions make Xok less extensible than Aegis, they simplify the implementation of libOSes (see Section 9) with only a small reduction in application flexibility.” [p7, 2]
Then either the exokernel argument was exaggerated, or Xok’s “reduction in application flexibility” is less “small” than claimed. Which do you think it is?
“We attempt a crude comparison of our protected control transfer operation to the equivalent operation on L3…. For Table 6, we scaled the published L3 results (5 microseconds) by the SPECint92 rating of Aegis’s DEC5000 and L3’s 486…. Aegis’s trusted control transfer mechanism is 6.6 times faster.” [p9, 1]
It’s not easy to do comparisons across architectures, but this simplistic scaling is easy to poke holes in.
Figure 2 [1]. What is really being shown here? And does it have anything to do with the exokernel?
“To measure the costs of all protection we ran the benchmarks … without … any of the extra system calls. This reduces the total number of Xok system calls from 300,000 to 81,000, but only changes the total running time from 41.1 seconds to 39.7 seconds. Real workloads are dominated by costs other than system call overhead.” [p9, 2]
This is very true. It is also very true for most non-exokernel operating system workloads. It is important work to find the particular workloads that have system call overhead problems and then develop improved system calls for those workloads.
Good performance may not be the hard part of OS design. A usable interface that also gets good performance is harder. Many of the interfaces in this paper and its follow-ups are not usable, except in the sense that they can be replaced.
Microbenchmarks vs. macrobenchmarks! The first exokernel paper [1] is all about microbenchmarks: the performance of system calls, the performance of IPC. (So was Liedtke’s L3 paper.) The second exokernel paper [2] points out that microbenchmarks don’t matter for good performance. (“The main benefit of an exokernel is not that it makes primitive operations efficient, but that it gives applications control over expensive operations such as I/O.” [p13, 2]) Some of the cool systems that motivated the exokernel work are quite microbenchmark-focused; for example, Massalin and Pu’s Synthesis kernel featured optimizations that greatly improved the performance of reading or writing one byte at a time. In the real world if one-byte reads or writes are causing a performance problem, applications will use buffered I/O.
Some researchers believe that virtual machines are exokernels, and that the success of virtual machines therefore demonstrates the success of the exokernel idea. This has some merit: certainly exokernels were influential, and influential projects impact people’s thinking in unexpected ways. However, whether or not VMMs are exokernels, they certainly are low-performance relative to other kernel designs running on bare hardware.
In what ways are exokernels and usability in opposition? An exokernel designer would argue that cooperating library operating systems can provide just as friendly and forgiving a programming environment as a monolithic kernel. I would argue with this. First, libOSes share an address space with their applications, making them vulnerable to corruption from memory errors. If libOSes cooperate using shared memory, one buggy application can threaten an entire libOS ecosystem. (LibOSes can be programmed defensively, but this is tedious and unfriendly; sharing via IPC can be expensive.) Second, the exokernel design expects that some users want to program their own libOSes, or at least parts of their own libOSes; it is not clear how new libOSes would cooperate with existing ones, and I don’t know of any good examples.
Compare and contrast an exokernel to the early implementations of Unix. For instance, in early Unix, the file pointer was stored in the application. Once true multitasking was introduced, it was shifted to the kernel to facilitate sharing. How would an exokernel do this?
Compare and contrast secure bindings to mechanisms available on conventional operating systems. How is a secure binding like a capability (and how isn’t it)? How is a secure binding like a file descriptor? How is a file descriptor like a capability? How do they differ?
Compare and contrast Aegis/DECstation’s TLB handling mechanisms (guaranteed mappings, large software TLB, application TLB handler) to what an x86 machine requires. Section 5.1 of [2] describes the authors’ approach briefly; puzzle out how they really did it.
“A few of our benchmarks are extremely sensitive to instruction cache conflicts. In some cases the effects amounted to a factor of three performance penalty. Changing the order in which ExOS’s object files are linked was sufficient to remove most conflicts.” [p7, 2]
!!!
“For instance, the LRU policy of pagers on top of the virtual machine can conflict with the paging strategy used by the virtual machine monitor.” [p13, 1]
A real problem: remember it for later, when we read the paper on ESX Server!
Dawson R. Engler, M. Frans Kaashoek, and James O’Toole Jr., “Exokernel: An Operating Systems Architectrue for Application-Level Resource Management”, in Proc. 15th SOSP, Dec. 1995, pp251–266. (ACM Digital Library)
M. Frans Kaashoek, Dawson R. Engler, Gregory R. Ganger, Héctor M. Briceño, Russell Hunt, David Mazières, Thomas Pinckney, Robert Grimm, John Jannotti, and Kenneth Mackenzie, “Application Performance and Flexibility on Exokernel Systems”, Proc. 17th SOSP, Oct. 1997, pp.52–65 (ACM Digital Library)
Gregory R. Ganger, Dawson R. Engler, M. Frans Kaashoek, Héctor M. Briceño, Russell Hunt, Thomas Pinckney, “Fast and flexible application-level networking on exokernel systems”, ACM TOCS 20(1), Feb. 2002, pp49–83. (ACM Digital Library)