March 9 Weasyl outage

avatar of skylerbunny

Posted by skylerbunny

10 March 2018 at 14:33:22 MST

Greetings! SkylerBunny here from the development and operations team.

Weasyl suffered an outage from March 9-10. The symptoms were variable and not immediately clear, but thanks to several members of our team (including Ikani, Hendikins, Charmander, and Kyra), collectively we identified the root cause.

As many of you may have noticed, Weasyl was found recently by spammers. We’ve been looking into best methods of how to eliminate the spam before it starts, including identifying spam traffic, deleting it, and blocking this traffic by username or IP address.

Several of these spam accounts exploited the fact that we did not have upper limits on some of our text fields, to throw enormous journal and comment entries into the database. Prior to yesterday, the primary problem was simply ‘garbage journal entries’, but this was escalated to ‘huge journals and comments’. The largest was 1 million characters long. Several of these in concert were sufficient to grind Weasyl to a halt.

Our developer team has set up new truncation limits for Weasyl text fields. If you’re curious, you can see the change at https://github.com/Weasyl/weasyl/pull/324 . To make a long story short, many fields that had limits that were too high (or no upper limit at all) now do; then we deleted all comments causing the issue, of which there were dozens. This should eliminate the reason the site went down.

Ops and the Weasyl developers will be watching the site closely, and will be working on additional strategies to counter the journal and comment spam problems.

Thank you very much for your patience!

Comments

  • Link

    I hope you don't mind me commenting to test if comments work on these.
    Thank you weasyl team for working on this site and keeping it alive. Much appreciated.

    • Link

      Thanks for testing! :D

  • Link

    It was probably those Indian spam bots. Ever since the Tumblr purge they've been doing the same thing on DeviantArt and probably Fur Affinity and it's lead to me being banned unfairly from DeviantArt and my ban appeal ignored. Shows just how much DA staff really care about their members tbh.

  • Link

    To assess the capacity of our models to separate new occasions, we physically clarified an irregular example of hopeful events for every classification. We at that point thought about coursework service uk the model's predic-tions against human decisions so as to quantify precisionand review.