Distributed Systems Infrastructure

CS 239-1 Winter 2008

Lectures:      MW 2-3:30pm, La Kretz 120
Instructor:      Eddie Kohler
Office hours:      T 10am-12pm or by appointment

Week 1 Mon 1/7 Course introduction
Wed 1/9 Distributed data structures
"Scalable, Distributed Data Structures for Internet Service Construction", Steven D. Gribble, Eric A. Brewer, Joseph M. Hellerstein, and David Culler, Proc. 4th OSDI, Oct. 2000, pp.319-332
"Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications", Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, and Hari Balakrishnan, Proc. 2001 SIGCOMM, Mar. 2001, pp.149-160
Further information:
The Chord/DHash Project web site
MIT TR: supposedly most up-to-date description of the Chord algorithm
"Chord: A Scalable Peer-to-Peer Lookup Protocol for Internet Applications", Ion Stoica, Robert Morris, David Liben-Nowell, David R. Karger, M. Frans Kaashoek, Frank Dabek, and Hari Balakrishnan, IEEE/ACM Transactions on Networking 11(1), Feb. 2003, pp.17-32: probably the most up-to-date description of the Chord algorithm
Week 2 Mon 1/14 DHT geometry
"The Impact of DHT Routing Geometry on Resilience and Proximity", K. Gummadi, R. Gummadi, S. Gribble, S. Ratnasamy, S. Shenker, and I. Stoica, Proc. 2003 SIGCOMM, Aug. 2003
"Bandwidth-efficient management of DHT routing tables", Jinyang Li, Jeremy Stribling, Robert Morris, and M. Frans Kaashoek, Proc. 2nd NSDI, May 2005
Further information:
p2psim, P2P simulator used by, and implementing, Accordion
"Non-Transitive Connectivity and DHTs", Michael J. Freedman, Karthik Lakshminarayanan, Sean Rhea, and Ion Stoica, Proc. 2nd Workshop on Real, Large Distributed Systems (WORLDS '05), Feb. 2005
Wed 1/16 OpenDHT
"OpenDHT: A Public DHT Service and Its Uses", Sean Rhea, Brighten Godfrey, Brad Karp, John Kubiatowicz, Sylvia Ratnasamy, Scott Shenker, Ion Stoica, and Harlan Yu, Proc. 2005 SIGCOMM, Aug. 2005
"A Case Study in Building Layered DHT Applications", Yatin Chawathe, Sriram Ramabhadran, Sylvia Ratnasamy, Anthony LaMarca, Scott Shenker, and Joseph Hellerstein, Proc. 2005 SIGCOMM, Aug. 2005
"Fixing the Embarrassing Slowness of OpenDHT on PlanetLab", Sean Rhea, Byung-Gon Chun, John Kubiatowicz, and Scott Shenker, Proc. 2nd Workshop on Real, Large Distributed Systems (WORLDS '05), Feb. 2005
Week 3 Mon 1/21 Martin Luther King, Jr. Day holiday
Wed 1/23 The Google Infrastructure
"The Google File System", Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, Proc. 19th SOSP, Oct. 2003
"Bigtable: A Distributed Storage System for Structured Data", Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber, Proc. 7th OSDI, Nov. 2006
Further reading on GFS and Bigtable:
Hadoop: an open-source GFS clone (they're working towards Bigtable as well)
Google papers
Week 4 Mon 1/28 MapReduce
"MapReduce: Simplified Data Processing on Large Clusters", Jeffrey Dean and Sanjay Ghemawat, Proc. 6th OSDI, Dec. 2004, pp.137-150
"MapReduce: A major step backwards" (aka "You Damn Kids!"), David J. DeWitt and Michael Stonebraker, The Database Column, Jan. 17, 2008
Further reading on MapReduce:
Hadoop includes a MapReduce implementation
QtConcurrent has a non-distributed MapReduce implementation
Wed 1/30 Infrastructure for Your Infrastructure
PlanetLab Information
PlanetLab Bibliography
"Experiences Implementing PlanetLab", Larry Peterson, Andy Bavier, Marc Fiuczynski, and Steve Muir, Proc. 7th OSDI, Nov. 2006
PlanetLab Tutorial
"Events Can Make Sense", Maxwell Krohn, Eddie Kohler, and M. Frans Kaashoek, Proc. 2007 USENIX Annual Tech. Conf., June 2007
Tame and Tamer
Tame v2 (download/documentation)
Tamer (download/documentation)
sfslite (libraries, including for asynchronous networking, asynchronous RPC, and crypto)
"A Toolkit for User-Level File Systems", David Mazières, Proc. 2001 USENIX Annual Tech. Conf. (describes the base of sfslite)
Week 5 Mon 2/4 Minitransactions vs. Key-Value Pairs
"Sinfonia: a new paradigm for building scalable distributed systems", Marcos K. Aguilera, Arif Merchant, Mehul Shah, Alistair Veitch, and Christos Karamanolis, Proc. 21st SOSP, Oct. 2007
"Dynamo: Amazon's Highly Available Key-value Store", Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels, Proc. 21st SOSP, Oct. 2007
Wed 2/6 Paxos: Distributed Consensus
"Paxos Made Simple", Leslie Lamport, ACM SIGACT News 32(4), Dec. 2001, pp.51-58
"Paxos Made Live: An Engineering Perspective", Tushar Chandra, Robert Griesemer, and Joshua Redstone, Proc. ACM PODC 2007
More on Paxos

Much, much more information about Paxos can be found on the Internet. Here's a couple links.

"Paxos Made Practical", David Mazières, tech report, Jan. 2007
"The ABCD's of Paxos", Butler Lampson, Proc. ACM PODC 2001: a paper that demonstrates the relationships among several Paxos variants, using a painful specification
Week 6 Mon 2/11 Initial Project Presentations
Using Paxos
"The Chubby lock service for loosely-coupled distributed systems", Mike Burrows, Proc. 7th OSDI, Nov. 2006
Wed 2/13 Byzantine Fault Tolerance
"Practical Byzantine Fault Tolerance", Miguel Castro and Barbara Liskov, Proc. 3rd OSDI, Feb. 1999
"Zyzzyva: Speculative Byzantine Fault Tolerance", Ramakrishna Kotla, Lorenzo Alvisi, Mike Dahlin, Allen Clement, and Edmund Wong, Proc. 21st SOSP, Oct. 2007
Week 7 Mon 2/18 Presidents Day Holiday
Wed 2/20 Network Coordinates
"Vivaldi: A Decentralized Network Coordinate System", Frank Dabek, Russ Cox, Frans Kaashoek, and Robert Morris, Proc. 2004 SIGCOMM, Aug. 2004 (corrected)
"Network Coordinates in the Wild", Jonathan Ledlie, Paul Gardner, and Margo Seltzer, Proc. 4th NSDI, Apr. 2007, pp299-311
Week 8 Mon 2/25 Network Coordinates II
"Network Coordinates in the Wild", Jonathan Ledlie, Paul Gardner, and Margo Seltzer, Proc. 4th NSDI, Apr. 2007, pp299-311
Wed 2/27 Consensus Abstractions
"Attested Append-Only Memory: Making Adversaries Stick to their Word", Byung-Gon Chun, Petros Maniatis, Scott Shenker, and John Kubiatowicz, Proc. 21st SOSP, Oct. 2007
Week 9 Mon 3/3 Boxwood
"Boxwood: Abstractions as the Foundation for Storage Infrastructure", John MacCormick, Nick Murphy, Marc Najork, Chandramohan A. Thekkath, and Lidong Zhou, Proc. 6th OSDI, Dec. 2004
Wed 3/5 Coral
"Democratizing Content Publication with Coral", Michael J. Freedman, Eric Freudenthal, and David Mazières, Proc. 1st NSDI, Mar. 2004
Week 10 Mon 3/10 TBD
Wed 3/12 TBD