New tools to fight the spam in Site Updates (Josh)

Hey everyone,

First, I’ll acknowledge the elephant in the room. The spam the last few months has been completely insane, exhausting and inexcusable. I’m sorry. There were days that the entire front page was taken up by spam, and it made me sick to see. There’s only so much playing Whac-a-Mole with spammers can really solve, as these folks are always adjusting and becoming more sophisticated. Well, enough is enough.

Over the last two months two developers and I have been piloting a new set of tools under the hood to detect and remove the most egregious spammers automatically without any need for users to flag them or for anyone to manually review lists of users for deletion. I wanted whatever we built to be robust, easy to tweak on the fly, and not require much if any human interaction. I also wanted it to work as close to instantly as possible so most users never see any spam at all. And, most importantly, so that the front page is never taken over like that again.

Before I dig in, know that I won’t get too in the weeds on how all of these tools work under the hood, because we’re still refining them and because frankly I don’t want the spammers to know. I’m constantly tweaking to try and find the right balance between real-world users and the crazy spammers, and there are a bunch of safeguards in place to ensure normal users don’t get swept up. But things happen. If you get caught up in any of these, please reach out and I’ll get you sorted.

Big note up front:
All of the limitations I’m about to talk about about are limited to Public and Member’s Only entries. You wanna spam your friends or yourself? Yolo, I guess.

Alright, let’s go down the list:

New User Restrictions
New users are now unable to post Public entries for three weeks. This should eliminate the instant gratification these spammers have been getting. Might expand this to include Member’s Only as well, but historically that hasn’t been an issue. Limits on commenting coming soon.

Post Frequency Limits
If a user tries to post more than one entry per [very short time period], well.. they can’t anymore. Tough break. They’ll get a message telling them as much. If they’re human, ideally they’ll read it and chill out. If they don’t, and they try to post more than a certain number of times over a certain time period, they’ll get added to a “hmmmm” list. Not a big deal yet, we’re just keeping an eye on them. People make mistakes. But if they keep trying, even with the system politely yelling at them not to, they’ll need to complete a captcha before they can post again. If they keep trying to break the rule after that, they’re auto-banned from posting for two weeks, added to the “possible baddie” list, and their posts are hidden from the front page entirely. Poof, you’re in time out. They do this twice (as in, they’re banned for a full four weeks and still keep trying to break the rules), then their journal is deleted.

In addition to that, there is now a post frequency limit over a longer time period. This is the one that’s required the most careful tweaking, because the spammers are super crafty and we don’t want normal users caught up in the dragnet. I won’t get into specifics here, but just know that if you get close to exceeding this, you’ll get plenty of warnings letting you know about it before any repercussions happen. It’s pretty tough to get auto-banned unless you really really want it or, yaknow, have an automated script running.

I know the above all seems like a lot of steps before getting banned, but think of how quickly these jerks post. With any luck, they’ll trigger an auto-ban almost immediately and most users will never see them at all.

Keyword Auto-Flagging
I won’t get too into the nuts and bolts of this, but I’ve noticed (as I’m sure you all have) some commonalities between some of the spammers when it comes to usernames, entry titles, and book names. There’s still a big manual component to this one because there’s a delicate balance, but it’s already helping.

THE PURGE
One of the great things about having set policies in place around posting is that you can then go back retroactively and find users that violated them in the past. I’ve pulled those stats and used that information to put together lists I’m in the process of very carefully reviewing. To err on the safe side, I’m only looking at users that have joined in the last three years, and only egregious offenders. I’m treading really carefully here to make sure a user that, for example, was having a Sudden Life Situation and posting extremely frequently doesn’t get accidentally put on the naughty list. I also have a couple of community members helping me with the review to triple-check everything before pulling the trigger. Once we have this locked down, the purge begins.

[horn sound]

Alright, all that said, there’s still a TON of work to do in the war against spam. For some of these accounts, they’ve been lying dormant for two full years before switching on and posting a single entry.. per year. I do have some ideas for how to nail these suckers, but boy is that one tough. Just know that banning users does help me get a sense of who the true troublemakers are.

Alright, that’s it for now.

The lights continue to stay on, and we live to fight another day.

Mochi tax:

<3,
-J