This bug was first noticed an hour into a small test session with a handful of players. We noticed that Seedlings were managing to execute fewer and fewer activities - they struggled to even start an activity, for example cooking a meal or sitting in a chair. Sometimes they would do something but more and more often they just didn’t do anything.
QA decided to do a "soak test". That’s when you load up a system with a maximum amount of agents and let it stew or soak in its own juices, so to speak. We loaded up 250 Seedlings, and left the game to run for 24 hours, coming back to a completely broken game. First time round we sped up time, then for a real 24 hour period.
When a Seedling does something, this consists of several steps behind the scenes. For example, when they want to sit down on a chair, the first step would be to "discover" the chair. Next comes claiming the chair - the chair now gets reserved and can't be discovered by other Seedlings anymore. Next comes actually using the chair by sitting on it for a while. Eventually, the Seedling will leave and the chair is open to be discovered again.
Turns out, the "reservations" never got cleaned up anymore: so Seedlings who felt like sitting down would spot the chair, make their way to it and try to use it. This failed, because the chair was still registered as being reserved by another Seedling - so the second Seedling couldn't sit down, even though the chair technically wasn't in use. This was not just happening with chairs, but with every object in the game world, which created a nightmarish scenario for Seedlings in which they couldn’t interact with anything at all.
The soak test made the bug apparent: Over time no activities were successful anymore. Every single object in the world was now "claimed" and not used, and all Seedlings kept trying to live their lives and interact with their environment. Initially this just meant fewer and fewer jobs got finished, but since the Seedlings didn't stop trying, and every try triggered another error message, a domino effect eventually killed the whole deployment.
There is a moral to this story folks: claiming a spot by the pool with a towel is wrong and will cause the world to end eventually.
Also, it would be remiss not to point out that this particular bug was fixed by Alessia.