The server is extremely fixed now in Site Updates (Josh)

So, funny story.

I’m traveling at the moment (hello from Denver!) and my email checking has been intentionally rare. Trying to get off the grid for a few days. But let’s be honest: PB I can’t quit you.

I see an email come in this morning from SM offering his congratulations on the incredible speed he’s been seeing on PB today. Confused, I hit the site and oh my god it’s so speedy. I check the server stats and my jaw drops. After the deslothing a few weeks back, CPU usage dropped from 90% to the mid 40s. Slow, but manageable.

This morning? 4.5%. That is a dot. FOUR POINT FIVE!! That is extremely bananas.

I reach out to the developers and ask what witchcraft they performed overnight. They’ve been doing a TON of work on server optimization this week. And they have no clue what I’m talking about.

We have a mystery on our hands. Servers don’t just fix themselves.

So we dig in. We check the latest code that’s been pushed. Nope. We check the maintenance schedule of the host we use. Nada. We dig into analytics. Normal. Maybe we were being attacked for a few months and the attacker got bored? Unlikely.

My running theory for a few hours is “ghosts”, and that’s just as reasonable as anything else.

Fed up, I reach out to our host to very politely yell WTF and file a support ticket. They get back to me a few hours later and say that they did in fact patch a recently discovered vulnerability on the machine that PB lives on. Surprise!

Why this particular vulnerability has had such an effect on performance remains a bit of a mystery, but between this random fix, the upgrades we’ve done, and the ongoing work we’ve been doing to optimize things the site feels GREAT right now. And that looks to be permanent!

So, we did it. It’s done. PB feels like a real website again, and I can’t tell you how proud I am of the work we’ve done in the last six (!) months. It’s been a journey.

I really appreciate how patient all of you have been with the unexpected downtime and crashes lately, I know it’s been rough.

I can’t wait for what comes next.

(Oh, and PS comments are fixed now too!)

<3,
-J

Last updated September 23, 2018