Created: 2011-07-10 19:27
Updated: 2015-11-05 05:08



Migrate herds of servers

Design assumptions

  • systems of 1-100 servers
  • posix

Design constraints

  • Migrations are written in ruby
    • With helpful APIs for package manipulation, init.d management, etc
  • Migrations are written against a single Role
  • A Host is tagged with a list of Roles
  • Migrations assumes a basic rolling pattern: leave load, update, rejoin
  • Migrations are ignorant of specific hosts or number of hosts
    • Instead blocks are written against roles
      • eg roles[:database].each do |server| ...
    • If actually needed, can emulate via Roles
  • Hosts are identified by UUIDs
    • But users never deal in UUIDs
  • Migrations are identified by git sha1
  • State of a host is the sha1 of the last migration completed
  • Migrations form a strict chain with no skipping, diverging or merging
    • Tempting to allow more, but erring on the side of easily predicted behavior
  • Migrations are stored in source control
  • Pull from git model, not push from checkout like capistrano
  • Usable for both CI and Production to give confidence
  • Hosts communicate their state by setting references in git
    • Migrations can wait on some condition of other hosts by polling on git
  • Can add a host to cluster by cloning a VM and changing UUID
  • Can restore a host by loading backup of vm and running migration

Open problems

  • Handling coordination of divergence
    • restoring a host from an old migration state might require different waiting behavior
  • How to handle host specific configuration like setting IP, hostname, etc

Simple model

  • Hosts, Roles, Migrations, Changes
  • Hosts are identified by uuid
  • Hosts are tagged with one or Roles
  • Migrations are written to iterate over Roles
  • Migrations can be one time or reusable
  • Migrations can be be written against tests/specs
  • Applying a migration to a cluster creates a Change
  • Hosts poll on a git repo
  • Hosts register their migration state as references in git
Cookies help us deliver our services. By using our services, you agree to our use of cookies Learn more