Daybreak Update on Broken Expeditions

[Fanra]

https://forums.daybreakgames.com/eq/.../#post-3693514

Test had 4 different issues with zones:
1. The machine that runs zones on Test runs out of memory regularly. Because it is on a different environment from Live, there is not as much memory available. This issue is ongoing, attempts have been made to fix it including hardware upgrades.

2. There was an initial zone crash issue that was fixed thanks to helpful information from Test players. Zones that crashed were exacerbating its memory issues.

3. When 2 was fixed we continued seeing reports of crashing zones. It wasn't clear if it was due to 2 not actually being fixed, 1, or something else. It ended up being caused by a totally different bug fix (my fault) that was attached to the fix for 2.

4. There was an issue (fixed today) that was causing HAs to go into an infinite loop. All HAs that were created were stuck and eating memory, stopping players from getting other instances as well (like raids and missions). This one wasn't discovered until it was already Live, mainly due to the depth of other zone crash problems that occurred in this patch. It was like a perfect storm of zone crashes, one on top of the other, that has made you guys miserable and I am very sorry for it.

Like all issues in EQ, for some there are ideas on how to fix them, others have definite, known causes that are not trivial to solve, others are evasive and without any leads, it is hard to find time to figure out how to fix them. Choices have to be made on what to fix first, and balanced with implementation of new features. The highest priority is always things that have broken due to a new change (something that breaks with the patch). I will say that the EQ team is always working as hard as we can trying to fix problems.

[Firewyne]

As an IT professional for more decades than some of you have been on earth, if new software goes in and breaks the system, the first thing you do is back it out. As part of any implementation, a back-out plan should be included. I get that the test system is not the same as the live system and there is always some risk when deploying into a new environment. But that is why one should be prepared to back-out changes. I was a mainframe (old systems and software) so perhaps back-outs are harder on newer platforms and software. However, Daybreak implementations are notoriously buggy.