This is not the current version of the class.

# Lecture 8: Wait queues

## Motivation

Scheduling is a fundamental kernel responsibility. Kernels strive to efficiently and fairly divide all computer resources, including CPU, among jobs according to OS policy. There’s a ton of effort in every real kernel to do this—and all that work eventually boils down to efficiently and correctly putting tasks to sleep and waking them up again.

Think of it this way. Processes often wait for external events. Your editor waits for key presses; your browser waits for interactions (and/or animations); web servers wait for network connections. Efficiency demands that processes engaged only in waiting should use as little CPU as possible—ideally zero.

A blocking system call informs the kernel that a process is waiting, since the system call returns to its caller only when a specific external event (or interruption) occurs. But in Chickadee so far, waiting processes take far more than 0% CPU. Here’s a natural implementation of sys_msleep:

while (long(want_ticks - ticks) > 0) {
yield();
}


This implementation polls by repeatedly checking whether the desired time has arrived. The kernel task corresponding to the process remains runnable the whole time! In the p-testmsleep process, the tasks corresponding to blocked processes run more than a hundred thousand times. That wasted time would be better spent on actual work.

## Sleep-wakeup

We seek a coordination mechanism that allows a kernel task to block—to become non-runnable—until a condition becomes true. Generalizing a little bit to abstract “threads” rather than kernel tasks:

• Let $$x$$ be some subset of program state. $$x$$ can be as much as the whole program or as little as a single integer.

• A sleeping thread has blocked until a condition $$f(x)$$ becomes true. It is an error for the sleeping thread to remain blocked after $$f(x)$$ becomes true.

• A waking thread is a thread that modifies $$x$$ in a way that might cause a sleeping thread to unblock and become runnable.

The inherent difficulty of sleep–wakeup is that sleeping and waking threads must synchronize. If they do not, a sleeping thread might remain blocked forever! We’re worried about this interleaving of events, which is a classic lost wakeup or sleep–wakeup race:

computes f(x) == false;
decides to block;
modifies x so f(x) becomes true;
checks if any other thread is blocked (there are none);
does something else;
blocks forever;

How to fix this? A lock, such as a spinlock, can prevent bad interleavings, but blocking makes this bad interleaving tricky! Normally we’d associate x (the shared state) with a lock, say x_lock, and place lock calls on entry to the critical section and unlock calls after it:

locks x_lock;
computes f(x) == false;
decides to block;
tries to lock x_lock (blocks…)
blocks forever, holding x_lock;

This won’t work at all. It’s immoral to block while holding a lock. Doing so can deadlock the system, as it will here: the waking thread never wakes up the sleeping thread because the waking thread can’t acquire x_lock.

But moving “unlock x_lock” before the block point just recreates the original synchronization problem.

locks x_lock;
computes f(x) == false;
decides to block;
unlocks x_lock;
locks x_lock;
modifies x so f(x) becomes true;
checks if any other thread is blocked (there are none);
blocks (forever);
unlocks x_lock;
does something else;

So we need a lock for synchronization; we must not hold the lock while blocking; and we must not release the lock before blocking! What is to be done?

## Condition variables

In user-level programming, this problem is typically solved with synchronization objects called condition variables. The magic of condition variables is that they offer an atomic operation, cond_wait, that both releases a lock and blocks the calling thread. Here’s how the running example would work with cond_wait:

locks x_lock;
computes f(x) == false;
decides to block;
calls cond_wait(&x_cond, &x_lock);
locks x_lock;
modifies x so f(x) becomes true;
calls cond_signal(&x_cond), waking up the sleeping thread;

Since cond_wait releases the lock and blocks in one atomic step, there is no way for the waking thread to sneak in at the wrong place. Either the waking thread runs entirely before the sleeping thread acquires the lock, or the the waking thread runs entirely after cond_wait. And either run order has the right result. If the waking thread runs first, the sleeping thread won’t block; if the sleeping thread runs first, the waking thread will observe the sleeping thread and wake it up.

But we’re programming a kernel. If we want a condition variable, we will have to program it ourselves—and perhaps in kernel context a different abstraction will be preferred.

## How other kernels do it

OS/161, the previous CS 161 teaching kernel, uses this function:

void wchan_sleep(struct wchan *wc, struct spinlock *lk);

which atomically blocks the current thread (a later call to wchan_wake…(wc) will wake it) and releases the spinlock lk. Sound familar? This is a condition-variable wait operation! In implementation, wchan_sleep calls a more basic function, thread_switch, which handles all voluntary kernel context switches. The need for atomic unlock-and-block infects that foundational function too:

/*
* If NEWSTATE is S_SLEEP, the thread is queued on the wait channel
* WC, protected by the spinlock LK. Otherwise WC and Lk should be
* NULL.
*/
static void thread_switch(threadstate_t newstate, struct wchan *wc, struct spinlock *lk);

Since the foundational thread switching function uses wchans, any blocking primitive you think up must in OS/161 be expressed using wchans.

The xv6 teaching operating system has a similar design, except that “wait channels” are left undefined (they are opaque pointers).

// Atomically release lock and sleep on chan.
// Reacquires lock when awakened.
void sleep(void *chan, struct spinlock *lk);

This resembles the FreeBSD operating system’s general sleep function, _sleep. Like xv6 sleep, FreeBSD _sleep takes a void* ident (an opaque identifier for a wait channel) and a struct lock_object* lock. But years of use and feature development have added more stuff:

/*
* General sleep call.  Suspends the current thread until a wakeup is
* performed on the specified identifier.  The thread will then be made
* runnable with the specified priority.  Sleeps at most sbt units of time
* (0 means no timeout).  If pri includes the PCATCH flag, let signals
* interrupt the sleep, otherwise ignore them while sleeping.  Returns 0 if
* awakened, EWOULDBLOCK if the timeout expires.  If PCATCH is set and a
* signal becomes pending, ERESTART is returned if the current system
* call should be restarted if possible, and EINTR is returned if the system
* call should be interrupted by the signal (return EINTR).
*
* The lock argument is unlocked before the caller is suspended, and
* re-locked before _sleep() returns.  If priority includes the PDROP
* flag the lock is not re-locked before returning.
*/
int _sleep(void *ident, struct lock_object *lock, int priority,
const char *wmesg, sbintime_t sbt, sbintime_t pr, int flags);

All these operating systems implement sleep-wakeup by pushing the atomic unlock-and-block step to the core of the operating system. But that’s not the only way. Rather than joining unlocking and blocking into a single atomic step, we can separate blocking into two steps, enqueuing and scheduling. With blocking unpacked in this way, sleep–wakeup can be implemented with less core scheduler involvement. Linux, and Chickadee, adopt this design, using synchronization objects called wait queues.

## Wait queues

Chickadee wait queues are used like this.

wait_queue waitq; ...
function_that_might_sleep(proc* p) { ...
waiter w(p);
while (1) {
w.prepare(&waitq);
if (p should wake up) {
break;
}
w.block();
}
w.clear();
}


Here’s what these functions do.

• waiter::prepare(waitq)—This function prepares the waiter to sleep on the named wait queue, and implements the “enqueue” part of blocking. Specifically, the function:
• Locks the waitq data structure.
• Sets p->state_ to proc::blocked. This means the state is blocked even though the associated kernel task is running!
• Adds the waiter to a linked list of waiters associated with the waitq.
• Unlocks the waitq data structure.
• waiter::clear()—This function cleans up the waiter after the process has woken up, by undoing the effect of any preceding waiter::prepare. Specifically, the function:
• Locks the waitq data structure.
• Sets p->state_ to proc::runnable.
• Removes the waiter from the linked list, if it is currently linked.
• Unlocks the waitq data structure.
• waiter::block()—This function implements the “block” part of blocking. Specifically, the function:
• Calls p->yield(). If the process still has state_ == blocked, then the process will block.
• When the yield call returns, it calls waiter::clear() to clean up the waiter.

It’s very important that interrupts be disabled throughout this whole procedure, or an ill-timed timer interrupt could cause the kernel task to block forever.

A wakeup function will often look like this:

wait_queue waitq; ...
auto irqs = waitq.lock_.lock();
while (auto w = waitq.q_.pop_front()) {
w->p_->wake();
}
waitq.lock_.unlock(irqs);


This code uses a new proc::wake() function, which should set the current process’s state to runnable and enqueue it on the relevant cpustate’s run queue. Here’s ours:

inline void proc::wake() {
auto irqs = cpus[cpu_].runq_lock_.lock();
if (state_ == blocked) {
state_ = runnable;
cpus[cpu_].enqueue(this);
}
cpus[cpu_].runq_lock_.unlock(irqs);
}


## Synchronization

The separation of enqueuing and blocking is key to synchronization here. Let’s enumerate how sleeping and waking up can interleave in this design; we’ll see that every interleaving works OK.

1. The waking thread can run before waiter::prepare. No problem: the sleeping thread will not block (it will call break to exit the while loop).
2. The waking thread can run after waiter::prepare, but before waiter::block. No problem: the waking thread resets the sleeping thread’s proc::state to runnable, so the sleeping thread will not in fact block (its yield call will keep it on the run queue).
3. The waking thread can run after waiter::block. No problem: the waking thread will wake up the sleeping thread as intended.

The wait queue design implements its own synchronization (the wait_queue has an embedded spinlock), and works correctly even if the sleeping and waking threads don’t otherwise synchronize. But of course sleeping and waking threads will often have other synchronization requirements and manipulate other spinlocks, such as ptable_lock.