An update and an apology in Site Updates (Josh)

Hey everyone,

The last few days have been a whirlwind! Had a few things I wanted to update everyone about.

We’ve been steadily pushing out updates for the last 48 hours attempting to iron out some of the performance issues we’ve been seeing. We’re to the point now where we’re feeling pretty good about things, so for the next bit we’ll be closely monitoring things and only pushing out critical fixes.

Here’s a quick and dirty rundown of what’s been keeping us busy:
- Fixed Javascript errors with auto-save on the create entry screen
- Upgraded primary and database servers to dedicated setup and higher CPU
- Upgraded the back-end and turned on some new more modern database functionality that seems to be helping

Next, I wanted to touch on the resurgence of the spam on the front page: we’re looking into it, and we appreciate everyone that’s been reporting the entries. My hypothesis is that somewhere along our upgrade journey, the spam filters got a bit messed up, and a bunch slipped through. Fixing this is high on the priority list after making sure the server is stable, so please be patient. We’ll take a hatchet to these jerks once again very soon.

Finally, in the interest of transparency, I wanted to share and apologize for something truly unfortunate that happened the other night after the server upgrade.

What happened?
There was a glitch overnight Tuesday where all comments — even private ones — were visible for any logged in users that had access to an entry. Private entries were not visible, and if a user didn’t have permission to view an entry, they weren’t able to see anything. This was the case for about 8 hours, from 10:30 PM to 7:50 AM CST, and there were about 160 people online at that time. The moment Buffy woke up, he checked the site, saw a comment pointing out what was happening, and quite literally ran to his computer (pre-coffee) and took the site down to fix things.

The glitch was caused by a broken filter when we added comment pagination. What’s truly frustrating is that we ran the beta for months, and this never happened. It somehow slipped through the cracks.

What now?
Going forward, we’re putting automated testing in place to make sure this kind of thing can never happen again. We’re also adding a “Report Bug” feature so you can quickly alert us if anything seems off.

We both want to be very very clear: your privacy is incredibly important to both of us, and we’re committed to keeping it protected. We are truly sorry that this happened. We screwed up, and we’ll do better going forward.

Alright, I’ll leave it there. Feel free to ask any questions and we’ll do our best to answer.

<3,
-J