The order in which James Serves the empty chairs is also the order in which the former guests died.
The hypothesis is a bit flimsy for the last three but from the way James plays ‘Sir Toby’ can you imagine Sir Toby not volunteering to drink a beverage on someone else’s behalf?