@@ 0,0 1,94 @@
+---
+title: Rate Limiting
+date: "2023-03-29T23:55:56-05:00"
+draft: false
+---
+
+In October, my public MoinMoin wiki started getting hammered with rapid
+requests. It wasn't a major concern of mine at first.
+ 1. It was nowhere near the scale of a DOS attack.
+ 2. MoinMoin has built-in surge protection that always allows logged-in users
+ access.
+ 3. As a principle, I'm in favor of public documents remaining truly *public*.
+ Even to potentially bad actors.
+
+But then it started impacting my uptime percentage.
+*Oh boy* do I love seeing that perfect, bright-green 100%.
+And it turns out, that's all the motivation I need to start walking back a
+principle.
+
+----
+
+I've been using [upptime](https://github.com/upptime/upptime) for quite a while
+as my uptime monitor/tracker.
+It's an exceedingly simple system that can be hosted freely on GitHub Pages.
+Well, as long as you are willing and able to tweak with the CI/CD limits.
+It ships with some defaults that are a touch too far for an unpaid account.
+
+The major advantage to upptime is that my monitor is 100% isolated from my true
+production servers.
+The CI/CD does all maintenance automatically.
+Certificate expirations, host uptime, and end user error are all neatly folded
+out of the equation.
+
+The extra advantage is GitHub Issues as a web admin ticketing system.
+For the most part, I host services by and for myself.
+It would never be worthwhile to spin up a ticketing system just for server
+administration.
+But there are still *lessons learned* and *personal notes* that I would ideally
+staple to incident reports, even just for myself.
+GitHub Issues is a clunky beast, but it's more than enough for this use case.
+
+This is all to say, you can find a timeline of events and a conversation I had
+with myself at [#52](https://github.com/dricottone/upptime/issues/52).
+
+----
+
+I did *briefly* wonder if upptime was in error, and reporting a false positive.
+I certainly could not replicate the issues.
+And it wouldn't be the first time I needed to tweak the CI/CD due to GitHub's
+race to the bottom on unpaid account limits.
+As I now know, my inability to replicate was a result of MoinMoin's clever
+surge protection.
+
+I did a trial run with Uptime Robot but quickly realized that it reported the
+same issues.
+And if an uglier, paywalled monitor (*that has no ticketing system!*) was going
+to give me the same outcomes, why would I give it another thought?
+Haven't logged in since.
+
+I then did some exploration into the realm of building my own monitor on
+another cloud platform (*GCP?*).
+That didn't make it even as far as a trial run.
+
+----
+
+While this all began in October, it took me several months to take this
+seriously as a threat (to my uptime streak).
+
+My first approach was to try to understand the spam.
+I added more and more thorough connection logging, and even some HTTP headers.
+I monitored logs in the hope that I could predict incidents--even intervene.
+I tried to parse metrics in the aftermath of incidents to better predict the
+next one.
+
+This didn't really amount to anything.
+As far as I can tell, I've randomly become the latest client (*victim?*) of a
+particularly zealous spider.
+
+My second approach was sticky rate limiting.
+I have always sat my services behind an instance of HAProxy, because I'm a huge
+fan of premature optimization and building for scale that never actualizes.
+This made the introduction of rate limiting (and the aforementioned logging)
+incredibly simple.
+
+I started with limits that mirrored MoinMoin's own surge protection, figuring
+that someone more intelligent that I probably came up with those defaults.
+Those turned out to be a bit too loose, so I spent the next few months playing
+a back-and-forth tuning game.
+
+In the end, I've landed on a configuration that has staved off spam and
+maintained uptime since mid-January.
+(*Leaving aside a brief certificate expiration... oops.*)
+I'm extremely pleased with this result, and my uptime streak is so beautiful.
+