March 9 Weasyl outage
Posted by
skylerbunny
Greetings! SkylerBunny here from the development and operations team.
Weasyl suffered an outage from March 9-10. The symptoms were variable and not immediately clear, but thanks to several members of our team (including Ikani, Hendikins, Charmander, and Kyra), collectively we identified the root cause.
As many of you may have noticed, Weasyl was found recently by spammers. We’ve been looking into best methods of how to eliminate the spam before it starts, including identifying spam traffic, deleting it, and blocking this traffic by username or IP address.
Several of these spam accounts exploited the fact that we did not have upper limits on some of our text fields, to throw enormous journal and comment entries into the database. Prior to yesterday, the primary problem was simply ‘garbage journal entries’, but this was escalated to ‘huge journals and comments’. The largest was 1 million characters long. Several of these in concert were sufficient to grind Weasyl to a halt.
Our developer team has set up new truncation limits for Weasyl text fields. If you’re curious, you can see the change at https://github.com/Weasyl/weasyl/pull/324 . To make a long story short, many fields that had limits that were too high (or no upper limit at all) now do; then we deleted all comments causing the issue, of which there were dozens. This should eliminate the reason the site went down.
Ops and the Weasyl developers will be watching the site closely, and will be working on additional strategies to counter the journal and comment spam problems.
Thank you very much for your patience!
Link
Uluri
I hope you don't mind me commenting to test if comments work on these.
Thank you weasyl team for working on this site and keeping it alive. Much appreciated.