March 9 Weasyl outage
Posted by skylerbunny
Greetings! SkylerBunny here from the development and operations team.
Weasyl suffered an outage from March 9-10. The symptoms were variable and not immediately clear, but thanks to several members of our team (including Ikani, Hendikins, Charmander, and Kyra), collectively we identified the root cause.
As many of you may have noticed, Weasyl was found recently by spammers. We’ve been looking into best methods of how to eliminate the spam before it starts, including identifying spam traffic, deleting it, and blocking this traffic by username or IP address.
Several of these spam accounts exploited the fact that we did not have upper limits on some of our text fields, to throw enormous journal and comment entries into the database. Prior to yesterday, the primary problem was simply ‘garbage journal entries’, but this was escalated to ‘huge journals and comments’. The largest was 1 million characters long. Several of these in concert were sufficient to grind Weasyl to a halt.
Our developer team has set up new truncation limits for Weasyl text fields. If you’re curious, you can see the change at https://github.com/Weasyl/weasyl/pull/324 . To make a long story short, many fields that had limits that were too high (or no upper limit at all) now do; then we deleted all comments causing the issue, of which there were dozens. This should eliminate the reason the site went down.
Ops and the Weasyl developers will be watching the site closely, and will be working on additional strategies to counter the journal and comment spam problems.
Thank you very much for your patience!
I hope you don't mind me commenting to test if comments work on these.
Thank you weasyl team for working on this site and keeping it alive. Much appreciated.
Thanks for testing! :D
Thank you for your work! DA account not working any more.
Too bad this site's still full of spammers...
Oh my god, I've been seeing those bots everywhere lately! They've even been spamming journals too.
What's worse, they fill their own descriptions and comment sections to the brim with links to these movie-watching websites. Sometimes you have to scroll down so much before you reach the bottom because of that.
It's insane. I can't even look at art without worrying that it's just stolen from another website just to hide links to movie-watching websites.
You guys have to do something with these spammers. The front page is getting really cluttered with illegal movie site links. Not only the general submission section, but also the character section, journal and critique.